I see downvoting as a last resort sort of thing. If I see spam or scammers, I downvote. If I see activity I don't condone, I don't upvote. If I see repeated activity I don't condone, I talk about it.

Excellent example for others to (hopefully) follow.