A gray man is typically defined as a person who blends into their surroundings by largely matching the culture and preferences of the situation, making themselves largely unremarkable and forgettable typically as a way to not draw attention to themselves.

Uh-huh! seems like a "gray man" is what many people these days understand as Sigma males.

My motivation has mostly been that I just want to be left alone — and I have been pretty successful at that

Well, everything has advantages and disadvantages. But I reckon your choice of social and behavioral traits have a lot more advantages than disadvantages. :)