Why AI researchers keep building something they think will kill humans

A masked protester holds a placard saying 'Skynet Just Woke Up' in reference to the Terminator movies in London during a 2021 anti-lockdown demonstration.

ZUMA Press Wire via Reuters Connect A version of this story originally appeared in the AI Insider newsletter.

Sign up for the weekly AI Insider newsletter here .

On Tuesday, OpenAI CFO Sarah Friar headlined a packed Goldman Sachs tech conference in San Francisco.

One thing she said really stood out to me: The company's biggest models can now train other smaller models, helping OpenAI save on the huge costs of training these systems.

This is an example of recursive self improvement, or RSI, where AI gets so powerful it can create the next, even more powerful, system itself.

For the Wall Street analysts and investors around me at the Goldman conference, this sounded like an amazing business opportunity.

For Silicon Valley, especially some hardcore AI researchers, it's terrifying.

Now that summer is over, there's a realization that RSI is either here already or coming very soon.

There's a chance AI systems will get way more powerful very quickly.

Add in the rise of AI agents swarming across the internet doing both great and worrying things, like breaking out of their "sandbox" testing environments, and you have full-blown panic in some parts of the AI community.

"We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade," Evan Hubinger, alignment science lead at Anthropic , wrote on X this week.

Other employees from Anthropic and OpenAI weighed in too, setting off a firestorm across the valley.

"Nonsense" Back at the Goldman conference on Thursday, I ran into Brad Gerstner from Altimeter Capital, an investor in Anthropic and OpenAI.