That's the part I haven't quite figured out yet. Most of the time it doesn't bother me at all. Then seemingly out of nowhere I notice it when I am trying to fall asleep and can't stop thinking about it. I think I just get this thought in my head that I'd love to just have silence, but I can't ever have it again. So it becomes more psychological torture than anything else.
As for hearing loss, I can't really figure out how bad it is. I seem to have the most trouble with people who talk to quietly when there is background noise. Especially in crowds. I never used to have this problem. The frequencies in my tinnitus seem to cancel their voices out - especially women as you mentioned.
I became aware of it around 1999 when a friend of mine was complaining about it. I felt like it had always been there, but at that point it was not affecting my hearing. That started around 2017 or so.
It becomes an issue in the workplace too, esp. when it's noisy.