I believe the logic into the idea that privilege does not exist is the same as the logic going into flat Earth thought.
Standing low on the ground you can't see very far and so the curves aren't apparent. The Earth is so big it seems like it encompasses everything there is. When you go exploring to new places it looks like it's all part of one flat plane. The only places you can get a full view of the Earth are machines and other people. If they don't conform to the worldview they don't matter though, because you can see the Earth stretch forever right out in front of you and you can see if has no curve. In some places, like the ocean, you can actually see the evidence of the curve. It's faint and in some far away place you aren't used to, but if you look closely the curve is there. The problem is that it is just one view. Every place you go to, except for the highest mountains, you still can't observe a curve.
Privilege is the same. You won't see it in your white middle and upper class neighbourhoods, it's just not part of your world. It's almost understandable. It would seem like privilege doesn't exist when the only people you regularly interact with are privileged.