Construction workers think Brexit will hit government infrastructure projects and ‘intensify skills crisis’, research shows