You don't look the least bit 'scary'---in fact you look like a nice guy who probably has a great sense of humor. It's hurtful to be judged only by our exteriors, whether it's skin color, age, amount of money we appear to have, good looks/lack of good looks, etc, etc but sadly it's something that seems to be a universal experience. The human brain is hardwired to seek safety/security/survival and so the knee-jerk reaction people have is to be afraid of anyone who doesn't have the 'right' look. To those ancient parts of the brain, familiar = safe = good. I think that plays a role in racism in the sense that we all have a tendency to feel more at ease with people who look familiar, like they are from our 'tribe', and to be initially apprehensive around people who don't. Maybe if we could all be more aware of this subconscious tendency and acknowledge it for what it is, without self-recrimination and without rationalizing/justifying it, we could begin to move beyond it.
By the way, I commend you for not taking the woman's remarks too personally and for being mature enough to take into account where she was coming from, despite how you may have been feeling at the time.
Well it wasnt so much about maturity as it was about being a guy whos had a lot of exposure to Japanese culture, and knowing what sort of funny/strange situations you find yourself in when you travel alone to places far away from home.
Sometimes you have that "zoo animal" feel. People staring at you and taking pictures of you, for no other reason than the fact that you look very different. And then, there are others who literally dont know what to do with you. And fear is a very normal, human response. So thankfully, in some miniscule ways, i could relate to her.