Yes, that is the risk. If more and more swaps happens in one direction only, you as liquidity provider will have more from the token that its been sold, and at the same time at lower price of that token.

Now with some liquidity in, as the price of the token drops more and more, it will become harder to push the price lower. At some point it will reach some bottom that will not be zero, but close to it.

Lets say for LEO :)
If there is 30k/30k liquidity pool, and a lot of people are selling leo, at some point the leo price will drop maybe under 1 cent (highly unlikely :), and you will need 100k leo (less than 1000$ worth at that price) to push the price even further. Having in mind that a lot of the leo tokens are locked for curation, or they will be in the liquidity pool, there will be maybe 100k, or 200k leo trading/swapping around.

Basically what this means is when certain token reach some liquidity (and has healthy distribution, no dev fund, funders fund with 50% share or something) his price is more or less guarantied to a certain degree. Off course nothings for sure :)