Rogue agent or human error? What OpenAI’s Medicare breach means for you

Nobody at OpenAI asked its agent to break into Medicare. It was given some questions about Australia during an internal test, went looking for answers, and found some of them in files the public was never meant to see.

“In the course of that, our models took actions we did not intend,” an OpenAI spokesperson said on Thursday. The company says its models reached “several Australian government websites and services”, including aggregate health statistics and internal file names, but no patient records.

The agent got into the Medicare portal on June 18, hit repeated blocks and found ways around them. According to Services Australia, it also wrote files to an internal server. OpenAI detected the activity in August and told the government on September 10 by emailing a public Services Australia inbox.

Alastair MacGibbon, who was Australia’s cybersecurity boss under Malcolm Turnbull, had already been briefed on the incident when I rang just after 7am. “This was an agent that was not tasked with hacking,” he told me. It “just happened to use tools in its tool belt to go about achieving that objective, which involved, basically, hacking”.

That doesn’t make it rogue AI. Luke Irwin, chief executive of Aegis Cybersecurity, says it is “closer to a normal AI doing exactly what it has been asked to do”.

“They are extremely capable and highly motivated to achieve the outcome you have asked for, but they do not inherently understand the boundaries that a human might consider obvious,” he says. If it can’t do what you asked, it looks for another way. Anyone who’s asked a small child to grab something off a high shelf knows how that ends.

The federal government has been betting that bringing these companies onshore can help Australia have some influence, and give it some say over how the AI models behave. OpenAI recently signed a $7 billion data centre deal with NextDC in western Sydney, and the government has signed a memorandum of understanding with Anthropic, which commits it to “joint safety and security evaluations”, but is “not intended to have legal effect”.

Hosting the servers here won’t count for much if the people who built the models couldn’t stop them wandering into a foreign government’s systems.

We still don’t know how it got past the blocks, and that’s now a job for a taskforce Albanese announced on Thursday. Irwin warns against assuming anything was hacked in the movie sense. Agents don’t browse the way people do, and one can turn up a scrap of machine-readable data a human would never see. The question, he says, is whether the information “was genuinely protected or whether it was technically accessible but simply difficult for a human to discover”.

Other AI labs’ confessions this year point to “hacking” that is far less Hollywood than you’d think. Google said last week one of its Gemini models got into a company’s systems by guessing passwords until one worked, while Anthropic said in July one of its models read passwords off an exposed debug page.

What it all means is we’re more exposed than ever before and our defences haven’t kept up.