You know, I've been in this position and turned the offer down to speak on the "women's panel" because I felt I was being ghettoized for my gender when they offered the person who forwarded the offer to me, a fellow make executive, with a full 30 minute speaking slot. I pointed out their actual creation of separation in trying to unify for a cause and they rectified quickly.
There is no one answer. We need to change culture.