It's all to do with the ubiquity of Markets. Everything, I've been told by a young writer who'd been to the UEA writing club (sorry school) and published one book, has to be judged by its market value. Books too. If it doesn't sell, it's not what's wanted. I don't know how anyone other than an accountant (and a illiterate accountant at that) could believe that. In any case, publishers have to pay to get books on to bookshop tables and prefer to spend their marketing money on surefire bestsellers, so how are people to see new books that don't have decent marketing budgets? Remember browsing? When you wandered around a bookshop and found out what had been published that month by picking up the books on the table and looking at them? Bookshops were pleased to display interesting as well as popular fiction. And remember when writing mattered enough to be regarded as worth subsidising through best sellers rather than racking up profits because shareholders had to have their dividends? Perhaps you don't. Anyway, once publishers were part of the literary world rather than subsidiaries of vast money making concerns. That meant that writers didn't get vast advances, but they were encouraged to work at their craft and given time to develop. I don't think that is happening very much now. Young writers and established writers get dumped when their sales figures drop or don't live up to expectations.
So in the world of books as well as everywhere else, capitalism has triumphed. And don't believe what they say about all worthwhile writers seeing the light of day. People who are not deeply concerned with good writing don't necessarily recognise it, or simply reject it as unsellable. They're looking for something else. And that's what they find, and what readers get.