I am not sure that anyone is advocating taking away the Nazi's right to free speech. I have not seen anything like that anyway, so I do not see Nazi free speech as being threatened.
You have not seen the videos of demonstrators getting punched and pepper sprayed, or of the direct death threats being made?
I think most everyone agrees that such behavior is illegal, and wrong. Those folks at the demonstrations are emotionally charged and lost their temper. That is a different case. I don't see anyone saying it is illegal to simply dress like a Nazi, or even admit to being a racist.
Rather, many people just disagree with the Nazi message. That is not illegal either. It is also well protected, free speech to denounce Nazi behavior in every way possible if you feel it appropriate.
Kinda like this guy born in Austria, 1947!
Arnold Schwarzenegger message for President Trump and Neo- Nazis on Charlottesville violence.
Now he should address the Communists as well.