RE: I got downvoted because I don't downvote people
They can't.
Your rep went down because Bullionstackers and The-Reef flagged you, both with higher rep than you.
While I don't know why you were flagged, I know Bullionstackers is a comment farmer and a troll.
RE: I got downvoted because I don't downvote people