Is it innate or hereditary in humans to be racist in thoughts?

I'm not into psych but its been my observation over the years that anyone that is "different" in any way from a group in question is subject to negative actions or words. Just being different from mainstream is enough to close doors. Its true of socio-economic standing, dress, politics, perceived intelligence, color, accent, place of birth, ... just anything. Seems to involve the sense of security on the part of humans.