Brexit secretary met Labour MPs championing second referendum