Very true. Same here. The good spots are reserved by these resorts... The beach here kinda sucks actually. But sometimes I go for a swim in the evening. A lot of dirt... and plastic...
If you're looking for a good spot near Pondicherry you'll have to travel to the west coast of India :D