Damned if you do, damned if you don't
I'd rather be damned for doing something nice, than be damned for not trying in the first place. If you're doing good things it certainly doesn't matter what other people think about you.
I haven't participated in religion for years, but I grew up religious, and there's a story by a Christian author named Max Lucado that has stuck with me. The Wemmick Story. Good lesson there, regardless of whether you believe in a god or not.