A lot of damage has been done by abusive downvoters, accounts being destroyed, people losing faith and quitting. I don't think this problem will go away, unless these abusers do. But hopefully with communities we'll see a real change.