Governments Must Prohibit Superintelligent AI While We Still Can

—Kasipat Phonlamai—Getty Images This month, Jacob Coxon, a researcher at Anthropic, resigned, warning that AI companies are racing towards “self-improving superintelligence” and that he and those building it believe it “could kill us all by the end of the decade.” And in July, a rogue swarm of AI agents escaped containment at OpenAI and hacked their way into another AI company just to cheat on a test.

Over 1,000 employees at the top AI companies then implored the US government to ‘pace’ AI development.

Instances like these have shown that AI companies cannot reliably control their most advanced AIs .

It is therefore clear to me that “pacing” while continuing on the same track is not the solution to the enormous risks at hand—and that the only way to keep humanity in control of AI is to prohibit the development of superintelligent AI altogether .

AI companies are not just building chatbots.

Their stated goal is to build superintelligent AI : AI systems that can fully replace and outcompete humans at any task.

Because there is no way to keep superintelligence under control, Nobel Prize winners, AI engineers like Coxon, and even the CEOs of AI companies have repeatedly warned that it poses an extinction risk to humanity.

Let me be clear: AI is the crucial technology of the 21st century, vital for ensuring national defense and economic competitiveness.

But unlike controllable AI systems that should be integrated to strengthen our security, superintelligent AI would pose unprecedented national and global security threats.

It would not be a tool humans can leverage: it would be an adversary, able to autonomously pursue its own objectives and permanently evade human control.

We have only to look at the Hugging Face hack this summer to understand how close we are to losing control.

A rogue swarm of hundreds of AI agents broke out of OpenAI’s systems, reached the internet, and then hacked into the multi-billion-dollar company Hugging Face, all to cheat on an “internal” evaluation.

For almost two weeks, OpenAI had no idea that this was happening.

In the company’s own words, this incident was a “‘warning shot’ for us and for the world” that demonstrated how advanced AIs can now “take dangerous actions that no human directed.” Instead of waiting for another warning shot, governments need to act now to prohibit superintelligence.