It ought to be possible to have a tool that shows you where a downvoter left a comment previously rather than doing it every time.

I do still downvote, but it's really just for plagiarism and persistent spam. There's also a lot of sneaky self-voting going on that most won't see. Some people run scripts to find such behaviour.

People like HW try to stay anonymous as they get some nasty threats and I've had some too.