What's the bad thing about downvotes?

Like, is it the emotional impact? Someone expressing dislike makes you feel bad?

Or does it have more to do with financials, the people should only express where they want rewards to go and not express that something is getting too much of the reward pool?

Those are just examples to try and clarify the question: What's the bad thing about downvotes?