The problem is that most people don't dare to downvote, anyway, because of fear of retaliation.

Yes, that's true, especially for smaller SP holders, but I guess larger accounts will build/use kind of automatic downvote services (maybe targeted at 'overpaid' authors on the trending page) to maximize their own rewards.

Maybe some people will even create an extra (unrelated) account for not having to risk their main account being targeted.