If the population does not grow above replacement level, that means the average age of the population will get older and older, like in Japan where the average age is now ~48 years old. That means a greater percentage of the population is retired, i.e. does not work. This is economic doom. That's why countries worry a lot about birth deficits.