eSafety Commissioner Julie Inman Grant warns AI deepfake tools are weaponised against Australian schoolchildren

AI-powered apps capable of generating sexualised deepfake images are being downloaded about 200,000 times a month by Australians, the nation’s online safety watchdog has told a parliamentary inquiry.

eSafety Commissioner Julie Inman Grant revealed the figure on Tuesday as she warned artificial intelligence was being weaponised to produce child sexual abuse material and target Australian schoolchildren.

Appearing before the Joint Select Committee on Artificial Intelligence in Sydney, Ms Inman Grant said her office had taken enforcement action against eight AI-powered nudify services.

“These services represent about 200,000 downloads by Australians per month,” she said.

The commissioner warned the rapidly developing technology was creating increasingly sophisticated opportunities for criminals to exploit vulnerable Australians.

“We are already seeing AI weaponised to create child sexual exploitation and abuse material with increasing reports of deepfake image-based sexual abuse across Australian schools,” she said.

New research from eSafety also found one in five Australian children had experienced potentially inappropriate or harmful interactions with AI chatbots or companions.

One in three children had shared personal or potentially sensitive information with the technology.

Ms Inman Grant said organised criminals were using voice cloning and face-swapping technology in sextortion schemes targeting young Australian boys and men.

She also raised concerns about AI-powered companions designed to encourage dependency among children, alongside the technology’s potential to facilitate coercive control and domestic abuse.

The commissioner pointed to a recent investigation in Canada involving an artificial intelligence model allegedly created by pedophiles to generate graphic synthetic child sexual abuse material using childhood images of a survivor.

“These are real harms affecting real people right now. They’re not just existential threats,” she said.

Ms Inman Grant warned governments risked repeating the mistakes made during the rise of social media, when regulation struggled to keep pace with emerging online harms.

“For years, regulation played catch-up with social media harms, after the damage was visible in children’s mental health, in election integrity, and trust in our information ecosystem,” she said.