When it comes to real-world examples of how operators have leveraged automation based on network data, here are two: Nordic operator Elisa in Finland and its “lights-out” network operations center (NOC), and Rakuten Mobile (both in terms of utilizing automation itself, and helping other operators automate through its Rakuten Symphony platform provider).
According to Elisa Polystar (formerly Polystar), which supports the NOC, Elisa began its transition toward automation in the NOC all the way back in 2009. First, the company undertook “an extensive review of operational costs, taking into account the customer impact of faults, the volume of issues that arise, and the cost of resolving themmanual addressing of faults.” The company systematically identified key processes, reviewed the legacy technologies that were in place, classified its workloads and identified the main problems, coming away with the conclusion that the Radio Access Network should be the key target area for process automation because effectively managing RAN issues automatically would drive down costs and improve customer satisfaction.
Elisa then analyzed and identified candidate processes for automation, and ranked them by importance; the carrier also did a breakdown of its most common RAN problems, which ones had the biggest impact, which ones required the most resources, and what steps had to be taken to remedy them. That last part was particularly important, Elisa Polystar said in a white paper outlining its work, because while some faults could be fixed with just one action, some would have multiple, alternative options that needed to be offered up in an escalating fashion. Radio cells produce health status reports with “predictable frequency,”and in general, a wide range of notifications could be collected that could indicate trends that might lead to service issues. So Elisa built algorithms that would take appropriate action based on such network data, and then verified the resulting performance. Ultimately, the company said, 69% of alarms that have history are predicted, and Elisa has seen a 79% improvement in resolution time. And it is having fundamentally fewer issues because of predictive capabilities: a 71% reduction in the number of incidents, and a 15% drop in customer complaints.
Elisa Polystar announced in late 2022 that it is working with Hutchison Drei Austria to enable automated RAN operations, and other customers include T-Mobile Czech Republic and Slovak Telecom.
Greenfield Japanese operator Rakuten Mobile has always done things in a markedly different way as compared to legacy telcos, with very much a software-first focus and network equipment and software tools that it is now offering up via its Rakuten Symphony spin-off. Geoff Hollingworth, Rakuten Symphony CMO, recently said in an RCR Wireless News interview that while Rakuten Mobile essentially started with the challenge of having to build from the ground up, “the opposite is true of any existing operator,” Hollingworth pointed out. “They have 20 years of very complex instrumentation of a network that’s gone through many generations. All of that instrumentation is required at the time [and] is there for a reason. It’s important to understand if the reason still exists or not. … Also, an organization exists, that is designed around that architecture and those domains, and those silos. … It’s important to start with where the existing operator is today, and to then understand that there [are] existing levels of automation.
Automation within telecom operators “isn’t a new thing,” he acknowledged, but “it tends to be sub-optimized, and it tends to be held into more individual domains and domain controllers. … What we’re moving into when you go into … full, AI-driven [automation] is cross-domain.”
He offered up a specific case from Rakuten Symphony involving a large operator wanting to retire legacy IT technology stacks around site building, site management and site commissioning. “I think if you look at the annual reports of pretty much every operator, one of the challenges is retiring legacy tooling,” Hollingworth added. Most legacy tools have been created organically within an operator and may be called a “tool” but is essentially a spreadsheet or simple web app, a “daisy chain of many different applications and somehow it works through cultural knowledge.” It is “very hard to actually change and retire those,” he said. “And some of the companies we’ve worked with have tried multiple times and failed quite often, because the incentives of many people aren’t aligned to the actual outcome of that retirement.” There can also be built-in incentives around work that has been outsourced, and it’s in the interest of people who are doing the outsourced work to keep the legacy tools in place. But back to the example of the site technology stack. The process of designing, redesigning and deploying sites across a large country can involve “hundreds of thousands of people touching that process every day, as they build thousands of sites or refresh thousands of sites in parallel.” Rakuten Symphony offers a digitalized approach to this that Hollingworth said “accommodates existing processes as much as possible, but retires what doesn’t make sense anymore. … Everybody that is working in that system suddenly is contributing to a common database of understanding.”
With that common database in place, he explained, operators can start to run AI models on it. “You can actually start to ask which actual outsourced vendors, which turf vendors are performing the best? Where is the time taken [that is] slowing people down in different markets? … you can start to do almost automated vendor selection, automated suggestions on process improvement—because everything is timestamped and all interactions are recorded. You can imagine, then, that there are some serious insights you can generate from that, that are very hard to understand if the data is distributed and disparate across an organization,” he said.
He also offered up a specific example of how Rakuten has applied this to its own operations.
“Whenever there’s an outage, as part of the root-cause analysis, if we haven’t managed to automate it, we do ask the question: ‘Can this be automated?’ If it can be automated, we immediately put it in as a candidate for automation,” he said. “When we have a problem in the network, in the actual management console, we actually recommend potential automation to solve that–but we don’t automatically trigger them. But there is a checkbox there that says, ‘Do you want to automatically trigger this?’” That tells the humans involved that the issue has been recognized and that it has happened X number of times before with X result. That means that confidence is high that 1) such issues can be identified when they happen and 2) the proper response can also be identified, offered and executed. “But at all times, the operations people can decide how slow or fast they want to move on that, and everything is recorded,” Hollingworth noted.
Looking for more insights on telecom operators and leveraging automation and AI in network operations, testing and assurance? Download the RCR editorial report “Test, Measure, Monitor, Learn: How operators can make sense of data in a new era of observability” and watch the accompanying editorial webinar featuring Verizon and Viavi Solutions.
