Race is not a social construct, it is geography. It tells us where our African ancestors settled after diasporic events. Given a few millennia, race became a rumor. (Somebody said there was a place where us had no color and looked like something called snow. My cousins uncles neighbor said there were places where us was the color of rich bottomland.) More millennia, divergent adaptations, isolation and not us turns into them.
Unfortunately, us or them meant life or death for reproductive males. Any bipedal male hominid not family was likely to kill you, take your females, slay your young and maybe dine on your carcass. Obviously, you need to know who is approaching across your territory. Display behavior might provide silhouette identification but closer up, its the way feathers hang, designs are painted, skin is colored, weapons are shaped, etc. By the time we have tribes, states, and so-called civilizations, everybody is trying to lord it over somebody regardless of place, race or lineage.
So, how come we claim or blame color privilege? Dont kid yourself there is not multi-hued privilege outside the white world. A Chinese physicist once explained to me that white people were barbarians incapable of higher thought. Who is superior depends on who is the biggest rooster in a rigid chicken hierarchy. Power is the determinant, skin color is minuscule. In this framework white should just be another color, but it is not. The sun has yet to set on white hegemony and how did those few Spaniards conquer the Inca Empire?
Today, on the continent of Africa, albinos are killed as profitable talismans of good or evil, depending on who is selling the parts. In North America, native peoples will travel across the continent to see a sacred white buffalo. In the 15th century when people knew, not believed, knew there were demons, witches, gods, goblins, fairies, prophecies, mediums, curses and so much more; rare albinos of any species were tangible proof of whatever you needed tangible proof for. Could be salvation or the vengeance of gods you cant count for wrongs you didnt know were committed.
Also in the 15th century, 95% of the population in North and South American had suddenly died from unknown causes and the few survivors were more or less just trying to pick themselves up off the ground. Locals would be certain they have offended some god or other but dont know which or what to do about it.
Imagine yourself, 600 years ago, trying to cope with a disaster of unbelievable proportions: government barely functions, not enough people to bury the dead, food shortages, and on and on. What if you suddenly saw an albino human-like creature covered in strange armor sitting atop, and controlling, giant four-legged monsters that spew fire and kill from a distance? You would know its a god, just not whether its an evil or good one. While you hesitate, the Spaniards take over.
Albinos are significant, one way or the other, all over the world. Encountering these powerful albino creatures on horses would have been as terrifying as it would be if you woke up to a mob of Terminators kicking in your door. White privilege was and is a con job. It is finally being challenged and that is decidedly uncomfortable for gooses and ganders. I seriously doubt, when the pecking is over, that whoever ends up superior will keep in mind that it is privilege and power that corrupts, not skin color.