Yeah I think the only way is to over time encourage more good actors join the platform to positively upvote to reduce the power of of malicious downvoters.

I wouldn't advocate for removing anyone's stake or downvoting abilities.