You have reached your maximum number of saved items.
The only two Telstra engineers with deep expertise in the system that failed in July were on a mandatory break when the national mobile outage they had unwittingly triggered took hold, an external investigation has found.
The outage cut hundreds of people off from Triple Zero, halted trains in Victoria and NSW, and brought down eftpos systems across the country. It has triggered a Senate inquiry and an investigation by the communications regulator that could result in Telstra being fined up to $30 million.
The pair had replaced a faulty power supply in a timing chassis in Melbourne in the early hours of July 8. When it came back online at 2.50am, a GPS card inside reset and began telling the mobile network the date was 2006. The card was missing a firmware update for a predictable fault the supplier had already flagged in bulletins, confirming this masthead’s July reporting.
The report by Technology Audit Partners found alarms on the timing servers were checked only during business hours by a small number of staff, and did not appear in the monitoring tools used by round-the-clock support teams. When things went wrong, nobody on duty could find records of the servers, the work done that night, or the people to call.
The outages have heaped fresh scrutiny on Australia’s Triple Zero system and the reliability of the nation’s telecommunications networks more broadly.
Chief executive Vicki Brady said the company accepted every finding.
“At its core, this came down to the network timing system not being given the priority that a critical network capability requires,” she said on Wednesday. “That is a miss on our side.”
Communications Minister Anika Wells said the report had unearthed preventable failures.
“This was an outage that should not have happened, and it caused an extraordinary disruption across the country,” she said. “Australians rightly expect to reach Triple Zero when they need it most, and telcos are legally obligated to ensure that connection occurs.”
Brady also revealed Telstra had missed eight failed Triple Zero calls during the outage and never carried out the welfare checks required to find out whether the people who made them were safe. She blamed a manual processing fault and a monitoring fault.
The checks have since been done. One caller got through on another network, and another found a different way to connect. Five either needed no help or had misdialled. Police visited the home of the unidentified eighth caller.
“It shouldn’t have happened. We should have picked those eight up,” Brady said. “We know how important Triple Zero is.”
The eight are on top of the hundreds of unsuccessful Triple Zero calls Telstra has already disclosed.
The telecommunications company also revised the start of the outage to 2.50am on July 8, almost 50 minutes earlier than it had said. The previous time reflected when a maintenance ticket was closed, rather than when the fault began.
