Privilege, be it racial, caste-based, nationality or religion based, is always a hard pill to swallow. People would rather lash out and question their accuser's motives than face facts. And the simple fact of the matter is that there do exist social frameworks that benefit us all. So pervasive is the influence of these frameworks sometimes, that we start considering them natural. But as long as there are people who exist outside these frameworks - either because they lack access or because they were deliberately kept out - we need to accept and appreciate the advantages that we have had in life.
Doing so makes you a better person.