A 'winner-take-all race' that could end in catastrophe, and the calls to slow it down

It sounds like science fiction — or the stuff of nightmares — as some of the biggest names in artificial intelligence warn that the technology their own companies are developing could go seriously wrong without an urgent slowdown.

Over the weekend, Anthropic boss Dario Amodei published an online essay calling for a coordinated reduction in the pace of AI development in order to give companies and governments more time to establish better guardrails.

While those concerns have existed for years, Amodei said events in recent months — including the Hugging Face AI hacking incident — had convinced him that action was needed.

Amodei reportedly left industry giant Open AI over ethical concerns and co-founded Anthropic, another large US AI company that operates the popular Claude chatbot.

Some experts have described that incident as a wake-up call that provided a concrete demonstration of the safety risks they had long warned about.

Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.

"I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong," Amodei wrote.

Key among Amodei's concerns was so-called "recursive self-improvement", where AI becomes capable of improving AI, potentially creating a cycle of increasingly powerful systems that outrun human ability to understand or control them.

The leaders of two rival AI companies, OpenAI's Sam Altman and xAI's Elon Musk, both voiced their agreement.

Alarm about the potentially existential risks posed by AI grew after an Anthropic researcher resigned last week, stating that "people building AI earnestly believe that it could kill us all by the end of the decade".

If some of the industry's biggest players agree that AI development should slow down, what would that look like — and is it even possible?

It depends who you ask. United States President Donald Trump brushed off calls to check the development of AI and suggestions that the technology is moving too quickly, attributing those warnings to "negative forces".

"We're leading China on AI ... and, frankly, I want to keep it that way because whoever wins AI, wins," he said over the weekend.

Experts say AI leaders may be genuinely concerned about the technology's consequences, yet they are also motivated to rapidly advance out of fear their competitors will outpace them.