That sucks. I took part only in Holi 2 times, but did not notice any untoward behavior. A bit of rowdiness sometimes verging on craziness. But no real misbehavior.
Then again, knowing what I know about the relations between men and women in India, you're comment is not surprising.
It depends on which economic level of society you are interacting with.