Using a class of endogenous growth models that exhibit international spillovers, we show that most of the cross-country differences in output per worker are explained by barriers to the accumulation of rival factors (physical and human capital) rather than by barriers to the accumulation of knowledge. This is shown theoretically, by comparing models with exogenous and endogenous TFP, and quantitatively by using a carefully calibrated version of the model. The main finding is that barriers to the accumulation of physical and human capital explain up to 64% of income gaps relative to the US.