Boris Johnson leaves Foreign Office mansion