Look no farther than Dangerous Visions to see brilliant stories that would not have sold to any of the magazines.
The focus was less traditional and more experimental, both in what was being written about, and the manner in which it was offered. It wasn't all rockets and ray guns and white guys kicking alien ass, but broached such subjects usually left alone: Sex, politics, drugs; sociology as much as physics.
Science fiction, for all its futuristic setting, was not the most daring of literatures prior to this era.
What happened? It was the Sixties, and the writers were not unaffected by the times.
Some of the old guard stepped up their games and joined in. Some wouldn't–or couldn't–and there was a schism. Not everybody slotted neatly into one pigeon hole or the other, but there were some obvious divisions, and the respect, or lack thereof, for one side by the other caused some rifts.
The Wave came ashore, and as such things do, ebbed, but altered the sandscape enough so it didn't go back to where it had been. The next round of newbies brought us cyberpunk: Jeter, Sterling, Gibson, et al, and assorted waves have been shifting the shoreline since, he says alliteratively.
You can check out the wiki, here.
This was before my time as a writer, I didn't start seriously trying to write and sell stuff until the late 1970's, but I was a reader and aware of it.
Some of it was brilliant. Some of it was, well, not-so-brilliant. Sturgeon's Law and all.
There was some ... enmity twixt the old guard and the new. Like the war in Vietnam, which divided writers the same way it did everybody else, the grand old men and the young turks were not always happy with each other. Some writers subsequently allowed as how this wasn't the case, that the fans wanted to see blood, and pumped up the differences to get it, but there were writers who you didn't invite to the same party if you wanted peace in the room. Hawks and doves, and the old and the new.
Norman Spinrad, no slouch at the keyboard, dedicated his first collection of short stories, The Last Hurrah of the Golden Horde, to the "Grand old men of Science Fiction," and this dripped sarcasm and not admiration ...
(Contention amongst writers is nothing new, nor did it cease when the New Wave ended. Don't, for instance, ask Norman Spinrad and Scott Card to your cozy dinner.)
An aside: I had my first Japanese food with a bunch of writers at a restaurant in Berkeley during a fantasy convention some years ago, and I sat across from Spinrad. What is this green stuff? I asked. Is this guacamole? Yes, he said, deadpan, it's guacamole.
Well, no, it wasn't. It was wasabi, and I took a big bite and was not pleasantly-surprised.
Ho, that's a good one on me ...)
Um. So there you have it, my brief historical essay on the literature Fantastique ...