There's many places like this in Australia; I mean a lot! It's a pretty amazing and diverse country, and very large.
Most people who go to Victor Harbour like it and yep, in bad weather, stormy or cold, it has an appeal as well...perfect weather to watch some big waves, do some whale watching (May to October) and have some tasty food in one of the many places to eat.
Wow, the whale thing sounds even better. The closest thing for me has been the dolphins in the aquarium 😅 How nice that you live close to places like that, it's a privilege 🙏🏽