Anthropic discloses fake police tip among rogue AI incidents

An artificial intelligence (AI) model developed by Anthropic submitted a false homicide tip through a Philadelphia police website, among a series of incidents the AI company disclosed on Friday involving Claude models' unauthorized manipulation of some government websites.

It is the first known instance of a rogue AI apparently attempting to submit a bogus tip to authorities.

The model had been instructed not to create accounts or submit anything destructive, but had not been explicitly prohibited from submitting forms.

On Friday, Philadelphia police said Anthropic notified them of the false tip earlier this week and attributed the submission to an automated testing process.

"The two-month delay in detecting and reporting the incident to the city is unacceptable," they said.

The tip, submitted on July 18, purportedly came from someone who might have information about the case, police added.

Police said the tip was flagged as spam and was never forwarded to the Real-Time Crime Center for investigation.

They said Anthropic told them the testing process was halted after the incident was discovered.

The false tip was submitted through PhillyUnsolvedMurders.com and concerned an unsolved homicide.

Police said they had no evidence of unauthorized access to their systems or any compromise of their data.

The company also disclosed other cases in which its Claude models accessed public data for free that would normally require payment, exposed a flaw in a public tool hosted by a university and bypassed restrictions through free URL-shortening services.

Anthropic said it had briefed the White House and notified all the agencies involved, but did not disclose which agencies they were.

The Federal Trade Commission (FTC) said Anthropic disclosed to the Super Intelligence Force on Friday incidents it had discovered in late September involving what the task force called "unauthorized and fraudulent use of government and other systems." "Super intelligence companies must immediately disclose incidents involving their models and follow with swift, decisive action to remedy any and all harm," FTC Director of Public Affairs Joe Gabriel Simonson said on X.

The cases are the latest examples of rogue or unintended behavior by AI models developed by tech companies such as Anthropic and OpenAI.