You got very close to dominant views in (social) psychology regarding this topic. The central presumption is always that people are limited in the amounts of energy and time they can invest in relationships. Another one is the fact that people want to increase their chances of survival by being social animals and grouping together. Therefore, to efficiently reduce the time needed to indentify the other's attitudes and intentions, we create categories of people based on a few traits. If those do not match with yours, why invest time in someone who is likely to oppose you down the road?
Another issue with individualism is the fact that reference groups are always needed to express yourself, as you said. You will only gain from standing out as an individual if the groups you associate with are seen as somewhat exclusive and beneficial. An individual disconnected with everyone will only be called a weirdo.
Judgment calls just are what they are and will not go away. Neither will discrimination. Mind you, this is not to say that population-level issues such as immigration are all to be regarded as bad and given up on. By mingling and repeated exposure to other people's ways, social barriers will eventually be lowered. It just takes time and energy.