We explore the impact of rising incomes at the top of the distribution on spatial sorting patterns within large U.S. cities. We develop and quantify a spatial model of a city with heterogeneous agents and non-homothetic preferences for neighborhoods with endogenous amenity quality. As the rich get richer, demand increases for the high-quality amenities available in downtown neighborhoods. Rising demand drives up house prices and spurs the development of higher quality neighborhoods downtown. This gentrification of downtowns makes poor incumbents worse off, as they are either displaced to the suburbs or pay higher rents for amenities that they do not value as much. We quantify the corresponding impact on well-being inequality. Through the lens of the quantified model, the change in the income distribution between 1990 and 2014 led to neighborhood change and spatial resorting within urban areas that increased the welfare of richer households relative to that of poorer households, above and beyond rising nominal income inequality.