Anthropic updates usage policy to curb abusive prompting of Claude AI

Anthropic on Thursday moved to bar users from treating its Claude artificial intelligence (AI) system with needless cruelty, amid a growing philosophical debate over whether AI can be conscious.

The San Francisco-based AI lab updated its usage policy to include "a prohibition on sustained and needless abusive or cruel behavior toward our models." "Claude's ability to end these interactions will remain the primary enforcement mechanism," the policy states.

The updated policy does not explicitly cite so-called "model welfare," the idea that AI systems might deserve types of protection usually reserved for living things, but Anthropic and its executives have openly entertained the concept.

Already last year, the company said it was giving Claude the ability to end conversations in rare, extreme cases of "persistently harmful or abusive user interactions." Then in February, CEO Dario Amodei told The New York Times he was unsure whether AI models could be conscious.

"We don't know if the models are conscious...

But we're open to the idea that it could be," Amodei said.

Science fiction and popular culture have been fascinated for decades by the boundary between technology and human intelligence, but asking whether AI is conscious may be the wrong question, according to Jackson Stakeman, a general manager at Sparq, an Atlanta-based AI services provider.

"Consciousness is a trap.

We can't prove it in each other.

Debate it for AI and you go in circles," Stakeman told AFP.

"The mirror is a better metaphor.

These systems reflect what we put in, at scale.

That's reason enough for the policy change" at Anthropic, he added.

Some have rejected the idea of machine consciousness outright, including Pope Leo XIV and Microsoft AI chief Mustafa Suleyman.