Is it racist to assume someone has privileges because of the colour of their skin? Not considering what personal circumstances of theirs may negate any assumed privileges. Yeah, that's racism.
Is it sexist to assume someone has privileges because of their gender? It's a bizarre generalization. And that's what prejudice is.
Accepting some human conditions as more marginalized in particular spheres does not have to mean pointing the finger at people we think have it better. That's just projecting negative feelings about our lives on symbolic oppressors.
Everyone was born into the world having no say what happened in history to form civilization and the world as it is today. And the oppressions and hardships that happened between and to certain groups of people are forgotten while others we are reminded about.
How many people know what is the Holodomor? the Holocaust? Does it matter which was worse? No.