What exactly is Google supposed to do here? They're returning what's available. Beyond that it goes from search engine to active curation and recommendation - and they are actively moving in that direction by replacing results with "answers", which I find to be much worse.
Have you read any of Google's advice to site owners? It amounts to "provide your users with what they're looking for, and do it with a fast and easy to use site". If you think SEO is an easy hack to get any site to the position 1 for any search, you're just ignorant of the landscape. All the "SEO" crap you see about keyword density or whatever, is just an industry trying to sound different and like they've got the key to position 1.
> They could start penalizing sites that show different things to googlebot and a real browser again.
Google is ridiculously good at this. Would be keen to see examples of a site that does this in a meaningfully negative way that ranks well for searches that matter.
I don't fully agree with all of your other points but I can see why you'd mention them.
Nothing, really. It's the "what's available" that's the problem. Quality paid services can't gain market share because free is a anticompetitive. Consumers will almost always choose it if it's not complete garbage. This throws all the metrics off for search because quality paid services become a statistical anomaly.
