It works as part of his self-delusion to think and promote me as the only one actively clicking the downvote button when, in fact, there are multiple people choosing to click it for themselves rather than trailing my vote.