It's automatically against a person, because you can't know his reason to vote "positive" in the first place. Maybe the upvoter disagrees with the post, but upvotes to promote discussion? Why would you then downvote? Right, because you don't like him. Which is a stupid reason to vote.
RE: Our Corrupt Sense of Fairness