It's time we take responsibility for allowing flimsy justifications for racism.