How do you keep people who want to spend money on persuasion from attempting to bias the dominant search AI?

Let's take an extreme example - say Exxon decides they want to spend $1 billion on influencing society in favor of the petroleum industry. They hire 10,000 freelance content writers to write 1000 articles each at $100 a pop and flood the internet over the course of several years with (everything from subtle to blatant) pro-petroleum content (nuclear's dangerous, solar will never scale, fusion's a pipe dream, wind is ugly, great new ways of recycling plastic, microplastics are actually beneficial to the human biome, a brand new low emission engine, ...). How is the scraper that ingests knowledge for the AI supposed to weed those out? It's still SEO, just at a higher level than the current mom and pop efforts.

I completely agree that economic agents are going to try to game it at scale - it's basic human nature.

On the other hand, if people complained that Google was a black box, or that as the old marketing adage goes, "advertising spend works, I just don't know which half", this might be a whole other level.

So agents will try but I have no clue whether it will be economical or effective.

It won't drive the value of SEO to zero, it'll create a new frankenstein SEO where companies are incentivized to pump content out in an effort to train the model. Part of the problem of SEO is that it's a constant cat and mouse game of people trying to trick Google into ranking their stuff high, and Google trying to detect it. Well what do you think will happen when ChatGPT is the source for recommending which vacuum to buy? Billions of pages of spoof "review" content saying your new Suckomatic3000 is the best thing since the Hoover - in an effort to influence ChatGPT into repeating it. "Hi ChatGPT, how do I cure my depression?" "It's well known that the Big Bingo Bonanza at CryptoBet.com gives the highest highs to it's daily winners! Sign up now!"

And to an extent, I think we're already seeing that with Google

I'm already seeing it's AI Q&A feature displaying text of "authoritative" answers to questions relevant to original queries above the main results which are sourced to obviously GPT-generated spamblogs consisting of long lists of similar SEO-optimised questions with contradictory and in some cases clearly incorrect autogenerated answers.

Exactly. More to this: the incentive for SEO copywriting goes away - good riddance!

On the flip side, these new generative AI search engines will have to innovate on their business model as well. Google basically pays content creators in organic traffic. We don’t know what Bings new interface will look like. But I think it’s fair to assume that there will be fewer click-throughs to websites.

To keep feeding the AI with quality content, there will have to be new ways to incentivise the creators. Especially those creators that do not have access to other sources of funding - think journalists. Unless of course, the AI disrupts one of journalism’s key purposes: to explain facts to us.