The consensus is all the stuff that doesn't get downvoted (or is downvoted less).

Someone can arbitrarily downvote a few things, but they have to choose what to downvote. No one has enough downvotes to blanket everything, or even most (maybe more like 1% in extreme cases).