As Anthropic heads towards a $2 trillion IPO, some of the loudest critics are company insiders

Beatrice Nolan here.

Anthropic is about to IPO, reportedly at a record breaking $2 trillion.

If you’re an employee at the company, especially one that’s been there for some time, you are about to come into a life-changing amount of money.

So quitting, publicly sounding the alarm about the technology you are building, or predicting the death of all humans due to your flagship product, may seem like an unusual move at such a precarious moment.

But that’s exactly what a fair few of Anthropic researchers, current and former, have been up to in the last week.

Here’s one post from an Anthropic employee called Evan Hubinger, who leads the company’s Alignment Science team, in response to a colleague’s resignation post: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.

I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” While a 10% chance of human destruction via AI may sound daunting, it’s not a new stance for a lot of these researchers.

Some, including those who had a hand in creating the technology, have been saying this since the beginning of the decade, some even longer. (Here’s a handy chart if you want to see what some of these leading researchers have for their ‘p.doom’ number.) While very much up for debate whether these doomsday scenarios from Hubinger and various other concerned researchers are likely to actually happen or not.

It certainly seems like they really do believe it.

See this post from another Anthropic researcher, Drake Thomas: “I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive.

I expect a great many of my colleagues across the industry would as well.

I promise you, we are actually just fucking scared, it’s not galaxy brained marketing.” None of this should come as much of a surprise, really.

Anthropic was founded as an AI safety lab in the first place, and CEO Dario Amodei himself has put his own p(doom) at somewhere between 10% and 25%.

Researchers voicing this kind of alarm are, in some sense, just saying out loud what’s been baked into the company’s identity from day one.