I would say downvote those who give malicious downvotes because that's the only way the “innocent” people will feel a bit of justice. Also shouldn't there be a yardstick for downvotes?