HaHa, well said "Damned if you do, and damned if you don't".

It is agitators who agitate and then have to (want to?) deal with an agitated field.
In every case of a downvote there was an initial. I think it would be wise, if one receives one, not to react immediately, if at all. The immediate reaction is registered, of course, and in the further counter-reaction something is carried on that would probably not have gained energy at the beginning through healthy ignorance.
I perceived vindictiveness or "I'll show you a lesson now" and I see little difference in the respective camps where I observed it happening.
Making it an open stage, you can question that, find it annoying or stupid (which I found it sometimes but then there is something other to it ... ).
The fact is that it is happening.
I don't have a cemented position on downvotes (I just don't use them myself). It seems to be quite proxy conflicts that people seem to have privately but don't make public themselves.

One can be watchful of where one gets drawn into. Without realising it, one gets pulled into a position to which one has not willingly positioned oneself. However, I myself do not need to interpret anything into statements in which my username appears, but I would think that if people with quick guns take any appearance of my username as a guilt of contact with a chosen enemy, that such is nonsense. The same applies, of course, to the reverse case of me being counted as one of the "good fellows". That can only mean that I am being torn in two. Good thing this is a virtual me, isn't it? If that was, what you meant with "Damned if you do...."

If I had received a big downvote, I would probably tell my man and he might shrug his shoulders and say: "don't take it seriously, hun. Do what you think you want to do and your downvoter will probably calm down again." Even this fictitious conversation with my hubby would be good advice for me.

I think it's not the worst thing to keep the playing field uncertain about "which side" one belongs to.

Although, I suppose conflict is always good for something.

Good luck with creative muse in the next year.