Separate cores for wireless and internet and AI-enhanced reliability are coming, said the CEO
Rogers Communications said it will spend to spend CA$10 billion (US$7.78 billion) over the next three years to avoid another outage like the one that took out services across Canada in early July. The company plans to leverage advances in artificial intelligence (AI) to assure service reliability and is making many other changes besides, company CEO Tony Staffieri revealed in a letter to customers, and while responding to questions from committee members during a recent session at Canada’s House of Commons.
Canadian consumers, businesses, banks and emergency services were left without voice or data communications for about 19 hours when the country’s largest mobile network went offline. It was the second major outage for Rogers in less than two years.
To keep 911 services operating, Rogers is working on a formal agreement between carriers that will transfer 911 calls to each others’ networks in the event of a widespread outage.
Any crippling outage comes at an inopportune time for telcos, but especially this one. Rogers is already under a microscope with Canadian regulators over its protracted effort to acquire broadband services provider Shaw Communications.
Rogers’ acquisition of Shaw has been in the works since March of 2021, but Canadian regulators are holding up the show over competition concerns. Rogers, along with Bell and Telus, carve up the lion’s share of the Canadian mobile market. Rogers and Shaw’s concession to divest regional mobile network operator Freedom Mobile to Quebecor, and Quebecor’s stated intention to make a fourth nationwide carrier network, hasn’t been enough to satisfy Canada’s Commissioner of Competition.
Rogers executives revealed to Canadian authorities that the outage was caused by a planned, multistage update to the company’s core network which went awry. Updated code deleted a routing filter, which caused a series of cascading failures that crippled both Rogers’ mobile and landline network.
Rogers CEO Tony Staffieri told Canadian officials that the Loonie stops with him, when it comes taking ownership for the outage. But behind the scenes, Rogers is shaking up the tree. Chief Technology Officer Jorge Fernandes is out, replaced by Rogers Business President Ron McKenzie, who Rogers hired in 2019 after a brief retirement from Shaw.
McKenzie addressed committee members to explain the technical details behind the failure. He said that ultimately, networking equipment sourced from different vendors reacted unpredictably during the outage because of the way each vendor implemented specific networking standards.
“And so it was that sequence that caused, essentially, the event that flooded the core network,” McKenzie said.
Rogers’ network operations staff ultimately needed to disconnect equipment causing trouble and redirect traffic before the company could bring services back online.
As part of it mitigation efforts, Rogers said it will separate wireless and wireless core networks. Staffieri called those changes, and others intended to add redundancy and improve network resiliency, “urgent fixes” that the company has allocated a quarter billion dollars to get right.
Staffieri also promised transparency. He said the company is working with unspecified “leading technology firms” to do a full network review.
“We will share lessons with our industry for the benefit of every Canadian,” he pledged.
