It's still draining rewards out of the pool and therefore hurting the ability to reward real contributions (unless N=0 of course).

I do agree there is no reason to waste downvote mana and take things too far but even a lot of small payouts can add up to a lot. If they aren't adding value then the small amount isn't really a reason to give a pass.