Your reputation score cannot be harmed by downvotes from anyone with a lower reputation than your own. It sucks when you see rewards plummeting, but the core of the chain is that no payout is guaranteed until a week has passed. Just remember, at least a decentralized system is harder for bad actors to completely take over.