Don't let it stop you. I've had mine for maybe four years now and this is the first time I've ever encountered anyone that had a problem with it. Back in England people would just ask about it and move on. Nobody in Georgia has ever mentioned it. And I've flown it several times already in Armenia, even last year.
And if you're in the mountains or forestry then you'll always be fine providing there's no signs saying not to.
It's probably the best purchase I've ever made, a stupid amount of fun that completely changes the way you interact with a space and see the world. This one event could never stop me from pursuing more of that feeling.
Yep, flying it in the mountains or natural locations would probably be my go-to, so perhaps low chances of getting in trouble as you said. Thank you for the encouraging words and I hope you won't ever get into such situations again!