It's hard to put limits on what people can do without reducing our freedom. Who would get to decide what is allowed? Anyway, people can downvote less than they can upvote, so the damage is limited.