I don't think it would be beneficial to eliminate downvotes, but there are various ways we can minimize the harm caused by their misuse. In the past I suggested payout recommendations as a way to reduce the need for a flag in certain cases, although linearization of the reward curve may be good enough for that case.
Shield and Counter Votes
One way to minimize the harm of downvotes is to make them easier to counter. I am proposing two related concepts, Shield Votes and Counter Votes.
A "Shield Vote" is a statement to the network saying "I am protecting this account from downvotes". Until cancelled, whenever the target account gets downvoted, as well as applying the negative r-shares of the downvote and consuming the voting power of the voter as normal, it also applies r-shares from the shielding account, and consumes the voting power of the shielding account up to the necessary amount. A "Shield Vote" shouldn't be able to apply more r-shares than the user would be able to otherwise, but it would just make it happen automatically without the need for human intervention, or a bot continually applying defensive votes.
A "Counter Vote" is a statement to the network saying "I am countering downvotes made by this account". It would work pretty much the same as a "Shield Vote", but instead of applying to the account receiving downvotes, it applies to an account making downvotes. It counters any downvotes made by that account, if it can, and consuming the required voting power to do so.
It should be noted that while similar and serving mostly the same purpose, the two concepts could be implemented independent of each other.