As AI companies get closer to ‘recursive self-improvement,’ this physics professor calls full autonomy ‘the worst idea in the history of humanity’

Once a distant ambition for technology researchers, the prospect of artificial intelligence models teaching themselves autonomously to be more efficient and capable appears ever closer to reality.

As the technology advances, developers say it is approaching “recursive self-improvement,” or RSI, in which AI models find ways to improve themselves and build their successor.

It could bring the promise of advances in science and medicine, tech company executives say, but also risks.

The uncertainty over where it all could lead is at the heart of growing fears about AI evading human control , and possible threats to humanity, which led several AI moguls to join last weekend in a call to slow down the technology’s pace of growth.

Anthropic this week detailed how its model Claude is helping the company to develop the next, more intelligent version of itself.

Claude is now leading 26% of Anthropic’s model research and development, which the company said means it can complete most of a given task “end-to-end from a high-level prompt” while still being under human supervision.

The models are not working completely autonomously — at least not yet.

Here are some key points about recursive self-improvement.

What is recursive self-improvement? Leading AI companies have different definitions for recursive self-improvement.

Some define it as when there is any feedback from AI on model improvement, while others define it as AI working toward that goal fully autonomously.

Autonomous recursive self-improvement essentially means AI that can improve itself designing the next version of the system, then the next version, and so on, said Anthony Aguirre, president and CEO of the nonprofit Future of Life Institute and a physics professor at the University of California, Santa Cruz.

“The really important thing here is that as AI is doing more of it, it gets faster, because AI operates just much, much more quickly than the humans do,” he said.

The fear around RSI is based largely on a runaway superintelligence emerging from that process, said John Thickstun, an assistant professor of computer science at Cornell University who studies methods that control the behavior of AI models.

But he said a more grounded view suggests a kind of recursive self-improvement has been going on in AI development for a while now.