In Turkish we have an expression, a staff with both ends smeared in shit.

That is what we have with downvotes, yes someone can use it to suppress vote expression and that isn't a good thing. However unlikely that it is, it is there.

But the other end of the stick isn't any better. How are you going to stop someone that powers up massive amount and start spam voting their own comments?

When you think about it, spam voting comments happened in the history of this chain. We can't disregard them as something that would not happen.

And there are already some accounts that are farming autovotes without adding anything of value. (Like their last comment on this chain were in 2020!)

It is a tragedy of commons.

So in a sense we have to choose a "lesser of both evils" to protect the commons. (It isn't really evil, just the saying fit here.)

Some problem just don't have easy solutions.