Are these spammers really that significant group of authors that I upvote?

I know nothing about the spammers you talk about... I would say they are just a small group in contrast with the amount of other authors I upvote.