it's getting kind of annoying to try and explain downvotes to each and every person who feels they are used unfairly

Just

stop

explaining.

Mute/ignore all complaints and vote as you see best.