How the Clash Between Money and Safety Created a Monumental Crisis for AI

Before the doom descended last week, OpenAI researchers were euphoric.

The startup claimed its latest internal AI model had solved one of the famously complicated Millennium Prize math problems, an experiment it embarked on after seeing rumors online that archrival Anthropic had already done so. Researchers called each other excitedly and posted about their triumph on Slack.

The breakthrough was seen internally as a victory in a ruthless commercial race to build AI tools smarter than humans, a multitrillion-dollar sprint that promised an unprecedented windfall of wealth and power when the two companies eventually go public—tentatively planned for as soon as the end of this year.

But it was also a sign of an explosive problem for the industry that was about to detonate. AI development is moving ahead at blazing speed—much faster than expected—feeding a growing sense of dread within the companies that it was in danger of spinning out of control.

That same Tuesday, Anthropic researcher Jacob Coxon announced he was quitting over fears that his employer and its competitors were building self-improving tools that could destroy humanity. A top scientist at the company, Evan Hubinger, wrote on X that he thought there was a more than 10% chance AI could kill all humans in the next decade. Drake Thomas, an Anthropic safety employee, posted on X that he would “burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive.”

The collision of scientific progress, moral imperative and financial incentive has now created a crisis for the industry . Development of AI has fueled a ballooning stock market over the past year, a phenomenon that looms even larger with Anthropic and OpenAI both on the precipice of initial public offerings that could value the companies in the trillions. Competition from China has further raised the stakes.

Yet the loud warnings being shouted by some inside the companies themselves are proof that the dangers of proceeding—against a backdrop of almost no regulation—are potentially immense. The warnings come on top of a rising public backlash over the impact that the AI revolution is starting to have on prices, jobs and education.

The negative drumbeat continued as the week stretched on, and panic from some corners of Silicon Valley spilled into the mainstream.