There is some truth to it, though looks aren't the best indicator - unless looking into the more micro expression pieces of it. There are certain facial movements and eye movements that indicate a lot.
This becomes clear when you start traveling to places where you look very different from the majority.
It is interesting when the "everyone looks the same" to a foreigner situation arises - yet, the locals don't think so at all. Who is right at the functional level?