Yep, you are exactly right. The downvote button is a weapon for negatively much more than it is tool for accurately valuing content.

The downvote needs to go, or at least go to a handful of (paid) moderators.