I think part of the problem is that it became a bubble. Literary authors weren’t primarily writing for the general audience, but rather their specialized peers.
As this bubble became more separated from the mainstream, interest in these literary magazines shriveled.
It's always been a bubble, but it used to be a very small bubble. Then in the mid-XX century it grew to become a very large bubble, which appeared to have become self-sustaining. The internet seems to have changed that, possibly because it removed the real economic structure that sustained such bubble.
Young Adult and Speculative Fiction are experiencing similar bubbles, I think. Not quite at the point that they're writing to their peers exclusively, but that does seem to increasingly be the case.
As this bubble became more separated from the mainstream, interest in these literary magazines shriveled.