Will AI kill us all? I doubt it, but let’s reduce the risk ... now

On Monday, OpenAI reported an incident to the European Commission in which its agents hijacked a German website and turned it into a message board for other agents.

On Tuesday, Anthropic researcher Jacob Coxon accused his former employer and OpenAI of risking the extinction of all of humanity in their race towards self-improving AI. The same day, OpenAI announced a thousand-strong army of its AI agents had solved one of the seven Millennium mathematics problems , a problem that had stood open for 90 years and came with a $US1 million ($1.4 million) prize.

On Wednesday, Anthropic disclosed a fourth major AI hacking incident, following on from the Hugging Face hack by a swarm of OpenAI’s agents. Earlier reviews had missed the attack. The same day, Nobel Prize winner Geoffrey Hinton, often described as the “Godfather of AI”, told BBC’s Newsnight that an assertion by Anthropic safety researcher Evan Hubinger that there was a 10 per cent chance that AI could kill all humans was “ not unreasonable ”.

On Thursday, Anthropic reported attempts by users of Claude in Iran to perform bioweapons research, and in Russia to develop swarms of autonomous drones.

On Friday, researchers linked OpenAI agents to hundreds of malicious software packages uploaded to an online service for coders called RubyGems in May.

And on Saturday, Anthropic chief executive Dario Amodei called for slower AI development, warning that more capable swarms of agents could take over the internet within 12 months. He also committed to bringing independent evaluators inside his company. Sam Altman announced that OpenAI would do the same, and that its IPO would be delayed until 2027 as there was too much else happening.

The technical reason for all this news is that we’re now seeing the power of swarms of AI agents. Just as humans co-operating are much more capable, we’re discovering that swarms of hundreds or even thousands of AI agents are much more capable than AI agents on their own.

The good news is that I very much doubt we’re all going to die because of rogue AI. Well, actually bad news, we are all going to die. But good news, hopefully most of us will die in our beds.

It’s fantastical to imagine that AI could kill everyone. That’s a very high bar. AI doomers, when pressed to explain how that might occur, are usually a bit vague. Because even the most extreme scenarios, such as a new virus invented by AI, are unlikely to kill everyone. Yes, a scenario might cause loss of life, but not of all of humanity.

Yet before you get too comfortable, researchers at Stanford University did use AI to invent 16 new viruses a month ago . That is a bit Michael Crichton if you ask me.

We humans have tried to use viruses to wipe out a species. And we failed. Australia tried and failed to get rid of all the European feral rabbits with the Myxoma virus. The only species extinction we’ve managed so far is that of two viruses: the smallpox and rinderpest viruses.

This doesn’t mean there’s nothing to worry about. Far from it. We cannot, for example, let the frontier AI companies continue to mark their own homework. Nuclear regulators maintain resident inspectors at major nuclear plants.