As someone who grew up and worked in classical music I have some ideas.
Cutbacks in public school music programs. Children who don't have parents who are interested in and play music were introduced to instrumental music in school band/orchestra programs. If you had real talent, you were encouraged and many schools provided private instruction as well as ensemble classes. Lots of my classmates in conservatory entered the study of an instrument through public school education...both black and white. Were the numbers of black classmates in proportion to the general population? No, but I think it was a little better than today.
Cost, and cost is subjective. If a young student pursues private lessons with a good teacher and accomplished professional it can be expensive. A decade ago, when my child was in high school and studying with a member of a major symphony orchestra lessons were about $60-$70 a pop. Expensive? Yes, but I saw many of his classmates' families spend much more on elite traveling baseball and AAU basketball teams. So, I guess we spend on what we find important.
Orchestras, which I know best, try everything they can to do effective outreach to minority communities with varying degrees of success.
I'm a white person, so I can't speak for minorities, but think about black kids and baseball. When I was a kid, everyone played baseball. By the time my son was a kid it was rare to see a black child on a baseball team he played on. They had all shifted to football and basketball. So maybe it's just partly what interests them.
What I can say with some certainty is that there is no hostility in orchestra management or among orchestra members to minority participation. In fact, they try very hard to increase numbers.
Why don't black kids and adults listen to classical/concert music? Probably the same reason not many white kids and families don't listen. It's a niche interest in America.
I don't understand it. Good music is good music. I have an interest in lots of music. A Brahms symphony is just as big a rush to me as the Beatles, Miles Davis, or a great bluegrass band. All can make me want to rock or bring me to tears.