I think you're combining ideas about racism and racial prejudice. They aren't the same thing. Most people use the word racism when they are talking about racial prejudice. People in general are a prejudiced species. It's a fundamental part of our pattern-recognizing minds, that filter out things that don't fit previous narratives (more and more as we get older) and amplify our notice of things that do.

All in all I agree with you though. Racial prejudice is one of the most nefarious types of prejudice, and anyone can do it. Dismantling the power structures of racism is a different thing though. When you talk about defunding the police, that's actually fighting racism, not prejudice. It does more than that too, but it actually takes a bite out of the power that is behind racism if it is to be racism.