AI experts say there’s a chance AI could wipe us out. How are those odds calculated?

As AI systems grow more powerful, some of their creators are attempting to estimate the odds of the technology's darkest possible outcomes: human extinction.

Evan Hubinger, who leads alignment research at AI company Anthropic, recently said he believed there was a greater than 10 per cent chance AI "could kill all humans" within the next decade.

His comments followed the resignation of Anthropic researcher Jacob Coxon, who reportedly warned AI companies were racing towards self-improving superintelligence without adequate safeguards.

Hubinger is far from alone. Several prominent AI researchers have publicly estimated the likelihood that advanced AI could cause catastrophic harm.

Nobel Prize-winning computer scientist Geoffrey Hinton has previously put the chance of AI causing human extinction within the next 30 years at between 10 and 20 per cent.

Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.

A newly released survey of 1,580 researchers who had published at leading AI events found respondents assigned an average 18 per cent chance that future AI advances could lead to human extinction or similar permanent or disempowerment of humanity. The median estimate was 10 per cent.

But the increasingly alarming figures raise an obvious question: how do you calculate the probability of something that has never happened?

Dr Chaitanya Joshi, a senior lecturer in statistics at Adelaide University, said the figures should not be confused with probabilities calculated from historical data.

"In fact, it is a type of situation where we will probably never have data to estimate a probability."

Joshi said that meant people should consider not only an expert's estimate, but also the reliability of their expertise and the level of uncertainty surrounding it.

Associate Professor Michael Noetel from the University of Queensland, who studies how AI experts assess catastrophic risks, said the subjective nature of these estimates did not however render them meaningless.

"Whenever we're dealing with risk, the kind of right way to manage risk is to try to get some estimate of the chances it could happen," he told SBS News.

"It matters if it's one in a million or a one in 10 chance of catastrophe."