A flag would lead to a counter-flag

This model fails when the original downvoter (flags don't exist any more) isn't depending on rewards (either self-voting milking type rewards or legitimate rewards).

There is nothing that requires all stakeholder voters who have at least some modest interest in growing their investment by policing the reward pool from being milked to death to themselves be either milkers or bloggers earning large rewards and many are not (and will not be).

I think we may indeed still see not enough downvotes since they represent uncompensated effort, but fear of retaliation won't be the main reason.