This is a very reasonable response, and that is exactly what most researchers feared would often be the case decades ago. But tons of research in both the lab (experiments with strangers) and the field (studies of real friendships and real work relationships) showed that contact is not as fragile as most people feared or assumed. Another way to say this is the human need to like and understand those that we actually get to know is more even more powerful than racism, which is undeniably very powerful.
To be sure, contact can sometimes backfire, but it appears to work much more often than it fails. I was very skeptical of the power of contact myself until people I came to know and trust began getting these powerful results.
If you worked where I work, I assume we'd sit down and have lunch and compare notes. And I bet we'd learn a lot from each other. I can't believe I woudn't come to admire you. I already do from reading your very real and honest response.
Having said that, I wonder if you and I agree that changing laws and dismantling racist systems might be the most powerful way of all to change racism. I think racism lives in leaders and in political and economic systems more than it lives in ordinary people.