We need walls for the same reason we need walls to our homes. :) As you said. To protect us from bad people. Our homes also protect us from the elements.
They give us security.
How secure would your home be without a wall? (and yet, walls are amoral..., go figure.)