I definitely agree that there are grey areas to this sometimes, that need to be worked out. That said, basing the foundational system on individual self-ownership would at least make this a morally legitimate endeavor.
I did see the Vice documentary, and yes, the alt-right crowd appeared to almost want violence at many points.
That said, ANTIFA is committing actual, direct violence out there and the MSM doesn't seem to care.
I didn't hear any direct threats from the alt-right folks on that video, I don't think, but I may have missed some. The weapons they were carrying were legal, so that technically cannot be construed as one (a threat). I don't know if any of them drew their weapons.
I still feel the difference between free speech and direct threat is pretty clear, at the end of the day.
Both groups are brainwashed statist zombies, at the end of the day, though, don't get me wrong, and I do believe there are grey areas at times.