These things aren't so bad, what's worse is when others go way and beyond trying to paint you in a bad light just cause of their irrational fear of downvotes or try to make things political and attempt to drag you down with them. That can get quite annoying and disappointing all in one go.