If you were hungry, and a rabbit happened upon your path, you could do whatever you desired. Let the bunny run, or use physical force, to satisfy your hunger.
If you were confronted with a cruel person who wouldn't leave you alone, and said terrible things to you, you could ask that person to stop. If they didn't, what's stopping you from smacking them and telling them to shut up, believing yourself righteous? Sure, they might hit back, but the point is to achieve happiness.
Is a game of "nya nya I'm not touching you" a game you would tolerate, if a person was being invasive or cruel?
You must be aware of all legal consequences, of course, for humans have crafted laws to punish violent aggressors in many cases. But if you tease a bear, instead of a rabbit, the bear may wield its claws.
On the other hand, you could just move out. Yet, is that righteous?
It's up to you, and your personal judgement. Not some lofty philosophical idealism.
Unless and until that person actually commits a trespass, or there is clear evidence of an imminent act of trespass (having a gun pointed at you, for example, or pulling back to swing a fist at you), you don't have any ethical grounds to strike them.
The point isn't to achieve happiness. The point is to achieve peace. To not be aggressed against. Hitting people for words you find vile or cruel opens the door to others doing the same to you for words they consider vile or cruel, even if only they feel those words are. It maximizes conflicts.
We're not bears. We're not base animals that lack the ability to reason. It certainly may seem that way sometimes, and there will always be sociopaths, but humanity is distinguished from other animals by our ability to reason. I'm all for reciprocating garbage speech with ostracism and exclusion, but to argue that physical force is an ethically acceptable response to words allows the worst kind of thing: violence against people for unpopular speech, not just cruel or vile speech.