Downvotes make sense as a way of indicating bad content. It's more an issue I think when people get into fights and start using voting as a way to try to punish people they don't like, less than the mechanism itself.