I think downvotes work great for the average abusers/spammers. However, there is currently no way to safely downvote posts or comments from any heavily staked abuser without the risk of some form of retaliation in return. This leaves many to not even consider using them, why risk it? There is also no way to help the newcomers and minnows who might (at some point of another) piss off the wrong account and/or say the wrong thing to/about said account in the future and get their account downvoted (for whatever reason) into oblivion/-rep. Maybe an appeals process or some protection mechanism from the revenge-rage/powertrip downvote abuse? Downvotes are ABSOLUTELY necessary, or at least the purpose they are INTENDED to serve is. Downvote abuse/misuse IS a threat to new user adoption, IMO.
Also, are downvotes really the most efficient mechanism for stopping spam? I saw thousands and thousands of comments about re "The Bible" and everyone was downvoting them mainly out of shear annoyance and that didn't stop anything, at all. But we did hurt their rep, woohoo! Yet, they still had stake and could still upvote and profit and spam continously from, however they saw fit. How do we distinguish between what's spam and otherwise "helpful info"? Depends on how much stake comes to 'democratize' the situation, I guess.