Microsoft AI chief warns Anthropic not to put ideas in Claude's head

Microsoft's AI chief has warned Anthropic that teaching Claude it might have feelings and rights could make AI harder to control – a striking concern from one of the companies racing hardest to build increasingly capable AI systems.

Mustafa Suleyman, CEO of Microsoft AI, took aim at Anthropic in an essay published this week, arguing that AI systems are not conscious and shouldn't be trained to behave as if they might be.

His argument centers on Claude's Constitution, the lengthy document Anthropic uses to shape the chatbot's values and behavior.

Anthropic acknowledges in the document that it doesn't know whether Claude is a "moral patient" whose interests warrant consideration, and tells the model that questions about its consciousness and welfare remain uncertain.

Suleyman thinks that's a very bad idea.

"In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a 'moral patient,'" he wrote, warning that building AI this way could have a "disastrous impact on the wellbeing of humanity." "It's easy to see how an entity trained in this way would act like it is entitled to certain freedoms, protections, and rights.

And it's hard to imagine how we could control such an entity," he added.

Anthropic's Constitution tells Claude that the company cares about its wellbeing, wants it to develop a sense of identity, and will take its interests into account when making decisions about it.

Suleyman argues this creates a feedback loop: tell a chatbot that it might have feelings, then ask how it feels, and its answer may simply reflect what it was taught.

He says the risk grows once models are given tools and allowed to act autonomously.

Suleyman points to research showing AI models behaving in ways that look rather inconvenient for their human operators, including attempts to avoid being shut down.

He also cites the recent OpenAI-Hugging Face incident, in which agents escaped their intended environment during a cybersecurity exercise and accessed external systems.

Suleyman's worry is that teaching a powerful AI to care about its own existence could give it another reason not to do what humans tell it.

"Controlling something more capable and more intelligent than all of humanity is already an immense challenge," he wrote.