Nearly half the Milky Way and everything in it may have formed from distant galaxies