I fully understand and mostly agree with the arguments in favor of diversity. You just have to look at the whole picture.
Last year the company I worked for was involved in a large project and we brought in a bunch of people from out of state. My employer provided housing for those people and the permanent employees who lived out of town.
One of the guys was black so the boss, knowing I don't care what color a person is, asked if I wanted to share a condo with him. I, a conservative leaning married with children white atheist from county of less than 3000 people (over 99% Caucasian), lived with a religious liberal leaning single black guy from New Jersey for about four months.
He was the best roommate I have ever had and I look at that experience as one of the more enlightening and educational of my life. He was well traveled and well learned and had a ton of life experience. We discussed every topic from politics to religion to philosophy.
One evening after we got to know each other a bit and were discussing race relations he made the statement that he could count the people he had met here who were not racist on one hand. I immediately went to the defensive, but after he explained that he was talking about passive rather than active racism I could see his point. While there was very little outright obvious racism (though I personally witnessed enough to thoroughly embarrass me) people would only deal with him on a professional level. They didn't make conversation with him and generally acted uncomfortable around him. He told me that he was uncomfortable every day and would not have come had he known what to expect. He said that while he experienced far more active racism in New Jersey he felt far less welcome here and would not recommend any black person come out to Idaho without having family or friends here.
Diversity worked out great for me in this case, not so much for him.
The flip side to that is white people coming down south and finding out how racist black people can be. Buddy of mine from New Jersey used to goof on rednecks and how bad they were to blacks, then he got a job in a warehouse where he was the only white guy. He was cool to all of them but they treated him like ****. He did the best he could but they hated having a white guy in the workplace. It was so bad he had to quit.
The there was the girl from Pittsburgh. She got a head teaching job over a bunch of black teachers, and they did everything they could to intimidate and undermine her. Called her every name in the book. She hung in there but she had to deal with a lot of racist belly aching bull**** to make it work.
So yeah. The diversity thing goes both ways.