The more of society that agrees on certain behavior norms, the more costly it becomes to violate those norms. The most universally demanded behaviors will be against violence, theft, and fraud. The price for violating these norms will be to pay full restitution or be completely outcast from society.
Or skin color, or gender, or sexual orientation.
Sadly, these are not choices made by the person to whom the demands are made, putting them in an unjust situation.
This, to me, is one of the most useful functions of government: to keep the majority of people/businesses/shops/establishments from setting rules like "no black people are allowed to eat in my restaurant". The majority is almost always wrong in big and fundamental ways (see also: the inquisition), and sometimes need to be prevented from doing harm and being jerks.
The question for me, then, is which is worse: violent coercion to prevent such assholery, or permitting the fundamental injustice by the majority?