The City was never going to lose that many jobs to Brexit