Put another way, if you are non-white, then you are classified (politically) as a black person.
Yellow = Black
Brown = Black
Red = Black
Black = everyone else that is not white.
Of course this causes all kinds of internal torment for some, and for very obvious reasons, but hey, it's either true or it's not - individual emotions notwithstanding. Don't take issue with the logic, your issue is with the system of racism/white supremacy.
Having a difficult time accepting that Red = Black?
Most accept that the Native Americans are "Red" people.. yet:
"The Apalachee are a Native American people who historically lived in the Florida Panhandle... under the state's binary racial segregation laws passed at the end of the nineteenth and early twentieth centuries, they were classified as "colored" or "black."
So yes, in the system of racism, it really is as simple as black and white.
Replace Racism with a better system (conceivably Justice) and all of the above becomes immediately moot.
Until then, setting aside emotion, let's deal with reality and devise a plan to produce a better system.