If we had nice crops in our front yard, the crazy neighbors across the street would get jealous and come over and start taking our food. They would also come out and boss us around when we were trying to work in our garden.
Lawns provide a nice buffer zone between a person and his neighbor. They take relatively little time to maintain and are an alternative to a fence. There is something to the saying that "Good fences make good neighbors".
I wonder what needs your neighbors have that aren't being met. I've dealt with "crazy" neighbors before. It can be a great way to find your own emotional buttons and learn how easily they can be pushed. I don't see why someone would stay off a lawn but walk into a garden.