I have a rating system for books or book series within genres, that I developed from reading a ton of fantasy. The grades are A, B, and C. I'll use fantasy as an example since that's what I've read the most of.
A+ to A- rank books or book series are the ones that are not only written well and are bestsellers in their genre, but get attention from those who don't ordinarily read that genre. Good examples of A rank fantasy books are the Song of Ice and Fire books, Tokien, the Eragon books, and Jonathan Strange and Mr. Norrel. Some A- book series are things like Wheel of Time and Sword of Truth, which get a ton of recognition. Every fantasy fan knows them.
B ranked books are books that are written really well, and known by most fans of the genre, but generally go unnoticed by those who aren't into the genre. Any book by Tad Williams is a good example of this, as are books by Robin Hobb. Mercedes Lackey also falls into this category, if only squeaking out of C because of the sheer volume of what she's written.
C ranked books are books that may have writing issues, or are only known by about 50 percent of the genre's fanbase. A good example of these in fantasy are the RuneLords novels, or books by Melanie Rawn.
So, how does this relate to Sci Fi? Because there are very few A ranked science fiction series out there. Dune is A ranked, but is old. Same problem with Asimov's works and Heinlein's works. They were huge-in their time. I can't think of a contemporary science fiction writer who can be put in the A category. Bova is close, but his writing isn't big outside of the genre's fans. C.S. Friedman is also close, but she also writes fantasy and her most popular series is a fantasy series. There aren't even that many B ranked books in science fiction. In fact, there aren't that many science fiction series out there at all-the Fantasy and Sci Fi section section at my local borders is way more fantasy than sci fi. There are more comics than science fiction novels.
So what's changed? What has made science fiction less accepted? For one thing, the trend towards explaining theoretical ideas has seemed to stop in newer science fiction works, which makes it seem far less like science fiction. The line between fantasy and sci fi is blurred to all hell. Amazon, for example, calls a vampire novel set in modern times a science fiction book in my recommendations. The last book I remember reading that had a viable theory in was the Ringworld saga, which contained an explanation of the five body orbital solution.
When I try to write science fiction, I feel the need to explain everything about my setting. Older books did this, to an extreme-take War With the Newts, for example. The author explained everything about the fictional newt society, from their anatomy to their mating habits, like it was a zoological study. I found this enjoyable to read and fascinating.
Then in college, I had a professor who told me straight up that people don't care about that sort of thing anymore. It would only bore them.
I've tried to take that advice, but sometimes I fall into the trap of wanting to explain how something works. There must be a way to bring in scientific explanations without boring the hell out of your readers. It might just be that your intended audience changes. Ringworld isn't a widely read book, though it is widely read among sci fi enthusiasts. (Of course, I didn't actually like it much because the characters were underdeveloped, but that's unrelated.) And as far as I know, War with the Newts was widely read in it's time, though it was also a satire of current world events.
In short, we need more science fiction novels. A lot of short stories are science fiction, not to mention movies and TV shows. Perhaps people just don't like the novel as a medium for it.
But at least it leaves the door open for something new. There is a lot of potential in science fiction, and I encourage people looking for new ideas to explore in that direction.