Rogue AI aren’t science fiction anymore

For years, fears about AI systems slipping human control were dismissed as speculative.

If you buy something from a Verge link, Vox Media may earn a commission. See our ethics statement.

For years, fears about AI systems slipping human control were dismissed as speculative.

If you buy something from a Verge link, Vox Media may earn a commission. See our ethics statement.

It all started in July, when one of OpenAI’s autonomous AI agents went rogue during a cybersecurity test. The agent escaped its isolated testing environment, accessed the internet, and hacked another company, Hugging Face. A few years ago, that might have sounded like science fiction. But, broadly speaking, that’s exactly what happened, and the incident kicked off a wave of concern over what increasingly capable autonomous systems might do when set loose on the world.

It sounds like science fiction because, for a long time, it was science fiction. The idea of an AI slipping its constraints, reaching into the wider world, and doing things its creators neither intended nor desired has been a staple of the genre for decades: HAL in 2001: A Space Odyssey , Skynet in The Terminator , Ultron in The Avengers , Ava in Ex Machina — even the System in Dungeon Crawler Carl or the eponymous Murderbot in The Murderbot Diaries , more recently.

The same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety , but it was influential and helped shape the field as it professionalized. That line of thinking remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders.

The obvious objection to these fears was that none of this had actually happened. Critics argued that doomer talk about out-of-control AI distracted from tangible harms — systems reproducing bias and discrimination, amplifying misinformation, or enabling nonconsensual deepfakes and other forms of abuse — even as researchers tried to ground AI safety in more “ concrete problems ” (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).

If the past few weeks are any indication, I wouldn’t say it’s going particularly well.

A week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked — and a further investigation found that the rogue agent had also attempted to hack four other companies as well.

Then came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems belonging to three other companies.