That definitely could be a factor. It is also the case of the fact that (and I was surprised about this) a great percentage of the Mexican population looks an awful lot like me. Often at a restaurant someone fires off rapid fire Spanish to me and seems genuinely surprised when I respond with very basic Spanish or simply tell them that I don't speak Spanish.