The economic puzzle behind AI giants’ doomsday warnings

Artificial intelligence researchers are not only racing to develop super-intelligent models, they’re also vying to out-do each other in making ever more apocalyptic predictions.

But do all these warnings about the dangers of AI from big AI firms and their staff strike you as a bit rich?

Last week the top scientist at OpenAI warned “no one is prepared for the consequences of a continued rapid rise in machine intelligence,” then days later Anthropic researchers claimed the AI giant was worried the technology could wipe out humanity.

These are merely the latest cases of AI doom-saying. We’ve also been told by AI firms that there should be a global freeze on development of the most powerful systems, or that some of their models are too dangerous to publicly release .

While some of these claims are probably just hype, many of the people making these warnings no doubt mean it.

Even so, there is a glaring tension: the people who are raising the alarm are often employed by (or have recently quit) the very companies that are driving the AI craze.

If AI firms hold genuine concerns about the future of humanity, thanks to a technology they are rushing to develop, surely they would be the best people to do something about it by tapping on the brakes, right?

But we know the firms won’t voluntarily do this, because there’s just too much money to be made from the AI mania. Anthropic is reportedly eyeing a stockmarket float next month, and there have been unverified reports of a $US2 trillion ($2.8 trillion) valuation. There have also been unconfirmed reports OpenAI is also keen for a $US1 trillion valuation when it ultimately floats.

Economics and other social sciences have a phrase for describing this sort of puzzle – where businesses often do things even if they know it’s self-destructive. It’s known as a “collective action problem.” It’s a useful way to look at the AI mania and what might be done to contain the risks, if policymakers can act quickly and decisively enough.

Collective action problems are situations where businesses would be better off working together – such as agreeing to pause the development of these high-risk AI models – but no one wants to do so out of fear of losing the competitive race. They’re a downside of competition, and we’ve seen them appear in all sorts of situations before.

During the banking royal commission in 2018, for example, there were cases where banks continued to engage in dubious practices because they were afraid of losing business to rivals. The solution was for the government to step in and ban the dodgy practice, to save the firms from themselves.

In the AI race, greater regulation would also surely help to deal with some of the more catastrophic (if unlikely) risks AI insiders are warning of. But it’s much harder to put the brakes on AI development than it is to ban banks from selling certain products.

One reason is harder is that it’s not only AI companies that are competing with each other, but also nations. So even if one country were willing to rein in its AI sector, it might not actually do so if rival nations don’t also agree to tap the brakes.

It’s a diabolically tricky problem.