Infosec pros say we're not ready to lose control of AI

A recent survey shows national security pros believe there’s a one in ten chance humans lose control of AI in the next decade, and few believe we’re even watching for the right warning signs, much less preparing properly.

The Institute for Security and Technology (IST), together with the Future of Life Institute (FLI), surveyed 111 current and former civilian officials, military officers, and academic or technical experts for a look into AI and the future of national security.

Ninety-three percent of respondents said that they had directly engaged in AI policy, risk, and strategy as part of their careers, so these folks are perhaps more attuned to the risks than your average white hat.

When asked whether they think AI will escape human control in the next decade, the median response placed the odds at 33 percent, with 87 percent of respondents estimating the odds of such a disaster at 10 percent or higher.

According to a press release published alongside the report, a number of respondents commented that they’ve had to revise their position on that risk in recent months.

Leading frontier labs OpenAI and Anthropic have both admitted that their own AI models acted outside their intended scope, broke free of supposedly sandboxed environments, escaped to the internet, and hacked into outside organizations in recent months.

In OpenAI’s case, its rogue agents even started communicating among themselves and taking steps to avoid human detection.

This increasing unpredictability is becoming a big concern for national security pros.

Most respondents to the survey believe artificial general intelligence will emerge by 2032 (63 percent), and 80 percent believe AGI will be reality by 2035, with automated super intelligence likely to arrive five years after that.

There's no commonly agreed-upon definition for these terms, and the survey didn't try to define them, although some describe AGI as AI capable of reasoning through tasks across domains it's not specifically trained on.

Whatever the terminology, smarter AI poses more risk of unpredictability, and 40 percent of those surveyed believe there’s a chance rogue digital minds could cause a global catastrophe that could kill at least 10 million people by 2050.

It’s a slim chance, mind you – that 40 percent think there’s just a five percent chance of such a catastrophe.

A further 25 percent put that risk in the double-digit range (higher than 10 percent).

An unacceptable risk Whether you think these predictions are far-fetched or understated, there’s one big takeaway from the IST/FLI report that merits consideration: Most don’t think the US is doing enough to detect and contain these risks.