Upvotes should be enough of an incentive to produce quality but we need downvotes to stop abusive language, spam, trolling and harassment. I should also have the right to downvote or remove comments on my post.

I agree that downvoting for abuse of rewards is a grey area but hey if you want to act like that, go ahead.