For a lot of those towns that don't have large businesses to bring in lots of employees and/or customers, it can be difficult to grow. The people who live there often live there because it's where they were. If there aren't jobs, they could move away to find better work. Eventually we could see people moving back to the smaller towns, but it would take a major change in our way of life to get that to happen.
Looks like a nice town. That market seems like it would be a good activity if you were in the area!