That only happens because of too few downvotes. If there are 50 people or 500 people downvoting, they're not going to retaliate against all of them.

Alternately, smaller stakeholders can stick with downvoting smaller stakeholders (when appropriate). Leave it to the larger stakeholders to downvote the other larger stakeholders. But at least by downvoting something you are maximizing your influence instead of throwing away 20% of it.