Investigation reveals key flaws that caused catastrophic Telstra outage in July

Telstra’s catastrophic national outage was a result of the telco’s failure to manage a “critical capability” and worsened by “insufficient technical expertise”, external investigators have found.

The July 8 outage impacted customers across the country for about 12 hours, leaving millions of Australians without access to mobile phone coverage and businesses left without digital payments.

An external investigation into the outage was conducted by the Technology Audit Partners (TAP), with the findings released on Wednesday morning.

They found the outage was caused by an incorrect date propagated through parts of the telco’s mobile network following planned management.

However, the investigation found the outage was “primarily a result of (Telstra) not treating network timing as a critical capability within the network … requiring the highest levels of oversight and protection”.

TAP found the telco had flaws in its network timing system as the “ownership of this function was not sufficiently defined”, leading to “weakness in this area and not treating it as a high-risk function of the network for the purposes of design, operations and resource prioritisation”.

The investigations claimed there was “insufficient technical expertise” in network timing systems, as well as inadequate end-to-end oversight of the evolving architecture supporting the network timing.

The findings determined there was a need for “stronger competency and more curiosity to investigate when something looked wrong”.

“There were shortcomings in process discipline including change management, configuration control, incident management, documentation and follow through,” the TAP findings read.

On Wednesday Ms Brady – who was overseas on holiday at the time of the outage – said it “should not have happened”.

“We let our customers down and we fell short of what Australians rightly expect from us,” Ms Brady said.

“In the days after the outage, I committed to being transparent about what happened, what we’d learn from it, and what we’d do differently.”

She said while modern networks were “complex”, the complexity was “not an excuse”.

“Our customers expect us to operate reliable and resilient networks. They expect us to learn when things go wrong. And they expect us to be transparent about both,” she said.