If ones witness vote helps a witness to rise in the rankings, why can't you downvote them too?

I was just thinking, as I was looking through and reading about the witnesses that there are some I love and some I wouldn't let anywhere near Steem. Why can't we downvote them to make them move lower in the rankings? You can upvote and downvote a post or comment, so why not a witness?