Why are job polarization and income inequality higher in large U.S. cities? I offer a new explanation: when house prices grow faster in large cities, middle-income households increasingly cannot afford to own a house there. They move to smaller cities and the middle of the income distribution in large cities hollows out, making them more polarized and unequal. I document that (1) cities with higher price growth experienced larger polarization and increase in inequality since 1980 and (2) middle-income households migrate more often to cheaper locations for housing-related reasons than low- or high-income households. Using a spatial equilibrium model with tenure choice and skill heterogeneity, I find that excess growth of prices relative to incomes and rents in large cities accounts for nearly all of the gap in polarization and almost one-half of the gap in inequality growth between large and small cities from 1980 to 2019.