A central office flood and a 2019 bombing prompted AT&T to rethink resilience
When AT&T was in the running for the FirstNet contract to run a national network for first responders, Chris Sambar—now AT&T’s president of network—was the head of AT&T’s FirstNet team. He said he had two main concerns about meeting first responders’ high expectations: AT&T’s coverage footprint wasn’t as extensive as responders wanted, and they also worried that the network wouldn’t be available during the natural disasters and emergencies when they would need it most.
AT&T as a company had to wrap its head around resiliency and availability expectations that Sambar said were “kind of shocking” compared to what a network operator would typically consider sufficient. But it went to work expanding its coverage (and upgrading its existing sites at the same time) and now claims a footprint of more than 2.81 million square miles and a 50,000-square-mile coverage lead. FirstNet has also been a driving force in transforming AT&T’s approach to network resiliency. (Read Part 1, focusing on FirstNet and the use of a new climatological model in disaster response, here.)
“The most interesting thing we’ve found over time is, whenever there’s an issue somewhere, if we figure out where the first responders are or are going, and we get the network up there [and] that becomes our priority—it fixes everything. It only makes sense,” Sambar says. “Logically, they’re going to the worst-hit places, they’re going to where people need them the most.” So if AT&T gets its network up in those areas, they already have dealt with the most pressing network issues. And Sambar said that AT&T has enough critical mass of FirstNet subscribers and a “tight relationship” now, that responders call the carrier with specific information about what is down or damaged and needs to be prioritized—and AT&T has specific regional FirstNet employees who are tasked with being ready for those calls as a storm or fire hits, and in the aftermath. “As a result, our response is typically a level about everyone else,” he said. In some cases, the responders just bring AT&T equipment along with them—in one recent case, Sambar said that responders towed FirstNet deployables behind their own trucks when heading to a wooded location where no operators had any coverage. “We’ve taken resiliency to a whole new level,” he said.
But two incidents in the last few years that brought the need for a broader network strategy and better planning information into stark relief, according to Sambar: A major flooding incident at an AT&T central office that was in a 500-year flood plain were flooding risk was supposed to be extremely low; and the Christmas Day bombing of its Nashville facility in 2020, when a Tennessee man blew up a recreational vehicle in front of an AT&T central office.
The central office flood was very expensive evidence that existing flood maps and other such information wasn’t enough to help AT&T accurately assess and plan for risk to infrastructure, especially in the long term. The Nashville bombing prompted the company to dedicate one of its VPs strictly to network resilience, who spent the following year talking to groups across the company, putting together a budget and drawing up plans that took into account worst-case scenarios like Nashville. AT&T spent $120 million this year and will spend another $120 million next year just on network resiliency, Sambar said—and its investments don’t just include actual network improvements and people, but better information on which to base its decisions. The company now has a new, highly sophisticated long-term climatological model, developed by working with the U.S. Department of Energy’s Argonne National Laboratory and its Center for Climate Resilience and Decision Science, that it uses to inform its disaster response as well as network investments and hardening. AT&T announced last week that, in partnership with the Federal Emergency Management Agency and Argonne National Lab, it is making the underlying data available to emergency managers and other local, tribal state and federal officials through the Climate Rick & Resilience (ClimRR) data portal so that they can use it for their own response and resiliency planning.
The model can forecast out on a 30-year timeframe with a have “a pretty good level of sophistication and high probability of accuracy,” as Sambar explained it. The main information elements that the carrier finds most useful, he says, are its modeling on inland and coastal flooding, high-intensity wind risks and wildfires. The model is showing the company where many of its buildings and infrastructure are facing more frequent or intense flooding or other increased climate-related risks: More of its buildings are at higher risk of power outages, for example, because the local power infrastructure is now in a flood zone when it wasn’t before. As it looks at the modeling and risk assessment, Sambar said, “We’re not just looking at our equipment or our offices, but we’re looking at others’ as well.”
The model’s information and the company’s planning for increased resiliency is being used down to the level of individual site construction plans.
“As we’re doing our network planning, in some cases it informs what the structure looks like that we build. So on a cell site, for example, we may elect to elevate the shelter—where in the past, using old climate models, we wouldn’t have elected to elevate the shelter,” Sambar said. “We’ll elevate it because we’re worried about flooding. We may do additional work at the top of the tower because we’re worried about hurricane damage or wind damage. When it comes to drought zones, the heat and the [heating, ventilation and air conditioning] that we put into the shelters and into the buildings, in some cases we may want to do ‘N+1’—meaning that we may want to have a secondary set of HVAC equipment. [If] this is an area that seems to be getting hotter than we expected, faster than we expected—it’s going to be at risk due to climatological changes, so let’s do an N+1 on the HVAC system.
“It informs everything that we do in the network now,” he concluded, adding that the company is at a point where it is using the model for increasing detailed resiliency work. At its central offices, for example, which typically have two connections to the network backbone/core (one at each end of the building), AT&T has made the expensive investment to add a third connection—and is now looking at the individual routes of each to identify which are aerial, which cross bridges and which are primarily underground so that every office has one route that the company considers “extremely hardened.” AT&T has added flood gates and retaining walls around buildings with increased flood risks, and put equipment for backup satellite connectivity on hundreds of its buildings, Sambar said. The company is also refining its supply chain and spare part inventory and locations so ensure that new equipment can quickly get to where it’s needed, whether that is for disaster recovery or a run-of-the-mill outage.
“What the model is telling us, which we saw in previous years, is that there may be the same number of storms, but the storms are getting more intense. And so they hit hard,” Sambar said. The telecom equipment itself probably can’t be hardened much more than it already is—and there’s only so much that can be done when winds are strong enough to bend steel towers, Sambar knows. “But what you can do is, you can be better prepared to respond faster.
“It’s kind of common sense—like, having an amphibious vehicle, or a few of them, in our inventory isn’t rock science. But nobody else has it,” he said. “It’s really a matter of, what’s your dedication to a specific activity. And in our case, because of FirstNet, we have said to public safety and the country … that we’re going to be the most resilient network in America. Not that we’ll never go down, but we will recover first, and we will be the most resilient. And so as a result of that we invested in things like a very complex and advanced climatological model, and then we invest in the tools that we need and the infrastructure that we need to make sure that we are the most resilient.
“The weather events, based on predictive modeling, may get stronger and more intense,” Sambar said. “We just have to get smarter about how we respond, the equipment we respond with and being able to get in there faster and recover the network faster.”
