We always subconsciously judge people based on their appearance, because the truth is people who look the same generally act the same. There are always exceptions to this rule, but unfortunately those unique people end up getting clumped in with everyone else with a similar appearance.
This becomes clear when you start traveling to places where you look very different from the majority. Looking past that initial judgement requires conscious thought and mental effort, and very few people are willing to do that.
There is some truth to it, though looks aren't the best indicator - unless looking into the more micro expression pieces of it. There are certain facial movements and eye movements that indicate a lot.
It is interesting when the "everyone looks the same" to a foreigner situation arises - yet, the locals don't think so at all. Who is right at the functional level?