Initially I lived in The Shitamachi area of Tokyo, downtown, in the East. Then I was in Ikebukero - central North West of Tokyo. What probably made the difference is that I rarely hung out with non-Japanese people or played in any of the gaijin-towns.