Staying connected when the world is falling apart: How carriers keep networks going

To Mike Muniz, an area manager for AT&T’s network disaster recovery team, witnessing the aftermath of Hurricane Michael was like entering a war zone.

On Oct. 10, 2018, two days after forming over the Caribbean Sea, Michael made landfall in the Florida Panhandle. The most powerful hurricane to hit the US since Andrew in 1992, the Category 5 Michael killed 45 people, left 700,000 residents across Florida, Georgia and Alabama without power and caused $25 billion in damage.

Muniz arrived in Mexico Beach, Florida a couple of days later to help restore the area’s cell service, which the storm had wiped out.

“I look back, I think it was worse than Puerto Rico [after Hurricane Maria in 2017],” Muniz says. “I remember seeing people just wandering around.”

Following disasters that topple cell phone towers or knock entire networks offline, wireless providers need to be on top of their game when repairing them, especially as more Americans ditch landlines completely for their smartphones. Beyond providing a vital way for survivors to stay connected to loved ones and contact 911, reliable networks are also critical for receiving emergency alerts and staying informed of local conditions and recovery efforts. Likewise, emergency personnel need to plan and coordinate efforts to save lives and rescue people in danger.

While exact times will vary based on each situation, the company plans to have services restored within hours of being mobilized.

Either way, Muniz and his colleagues face exhausting work when managing disaster recovery, and they’re likely to be busy in the weeks ahead. The 2020 Atlantic hurricane season, which has broken records with 25 named storms as of early October, won’t officially close until Nov. 30. Hurricane Delta, currently in the Gulf of Mexico, is the most recent storm of the season and could make landfall in the US on Friday. Throw in the ongoing COVID-19 pandemic and the deadly wildfires in California and it makes for a scary mix.

During his days in Mexico Beach, Muniz recalls how residents were able to phone loved ones to let them know they were alive once AT&T’s crew had restored service using a portable Satellite Cell on Light Truck (or SAT COLT, as it’s known). One person was able to talk to a family member after not being in touch for three days.

“They were very, very thankful,” he says.

The SAT COLT is just one example of the emergency equipment that the country’s third-largest carrier keeps in its four disaster response centers across the country. (It has an additional facility in the UK to support its global operations.) Earlier this summer, I visited one of AT&T’s facilities located a few hours outside New York City to see what was inside. 

An ever-evolving toolkit

Hurricanes and wildfires are the most high profile disasters wireless carriers need to manage, but missing persons, terrorist attacks and, yes, pandemics are also on the list. Whatever hits, they have a fleet of tools and vehicles at their disposal, ready to respond.

“We are the last line of defense for AT&T,” Muniz says. “We can’t have assets that go out into the field and don’t work.”

The third-largest wireless carrier in the country with over 92 million users (as of its second-quarter earnings), AT&T also operates a broadband network that supplies home internet access to nearly 14 million users. 

Its core solutions are the COLTs, SAT COLTs and Cell on Wheels, or COWs. All three can quickly be driven into areas to boost service in the place of cell towers that might be destroyed, or while engineers repair them. AT&T wouldn’t break down the cost of the individual tools, but the company says it has spent over $650 million on its NDR resources over the past three decades.

The COLTs are outfitted with cell phone transmitters that connect to a fiber line or existing wireless service, while SAT COLTs connect to AT&T’s network through satellites. Both look like big semi-trucks, have their own generators on board for power and feature retractable masts that can extend up to 60 feet in the air to broadcast the wireless signals. Once deployed to a location, Muniz says it takes roughly an hour to get its latest SAT COLTs operational in the field and begin providing service from the truck.

The COWs are smaller, trailer-sized units that can be hooked up and towed by a car or pickup truck. They don’t provide the same broadcasting range as a COLT (the exact range of all equipment varies based on terrain) but are easier to transport. All three can be configured to use AT&T’s FirstNet, which uses a special band of the carrier’s 4G LTE network to prioritize first responder communication.

Blimps, drones, boats and more

For areas that a truck can’t reach, AT&T added new tools such as a blimp called Aerostat, or “FirstNet One,” and drones called Flying COWs (for “cell on wings”). When deployed — it can climb to 1,000 feet off the ground — the Aerostat blimp can cover 100 square miles, or about the same area as three SAT COLTs, and can stay aloft for two weeks. Using it frees up the trucks to go somewhere else.

Unlike recreational drones that can fly freely, the Flying COW and Aerostat are tethered to the ground by fiber and power wires. The drones hover no higher than 400 feet in the air and can provide coverage for three to five miles, depending on the terrain. Unlike the blimp, the Flying COWs need to land regularly for maintenance.

Past experiences with floods, which the company particularly experienced when Hurricane Harvey hit Texas in 2017, have also influenced AT&T’s toolkit. “One of the improvements that came out of Harvey was we ran into difficulty getting through flooded areas,” says Muniz. 

A solution was to purchase dual-purpose amphibious vehicles that can drive on land or operate as boats on water. Looking like mini-Jeeps or bigger ATVs, these vehicles don’t have cell radios but are used to transport people or equipment, such as the Flying COWs, through different terrain. During its response to Hurricane Laura, the carrier used some of its amphibious vehicles to move engineers around so they could restore a fiber connection.

It also deployed SAT COLTs for both FirstNet as well as regular customers, says Kevan Parker, an incident commander on the NDR team who went from Atlanta to Louisiana to help coordinate the carrier’s response. On Sept. 4, AT&T also launched the blimp over Cameron Parish, Louisiana where Laura made landfall.

Hurricanes, tornadoes, COVID and more

While hurricanes can provide several days’ notice before they make landfall — Parker and his team began prepping for Laura five days before it hit — other disasters, such as earthquakes, come without warning. The nature of their jobs requires the response teams to be prepared at a moment’s notice to go into the field, with 2020 proving the adage of expecting the unexpected.

When the COVID-19 pandemic hit the New York area hard earlier this year, Muniz and his team brought COLTs to Jones Beach on Long Island to help boost coverage at a COVID testing center. Similar trucks were deployed to testing centers in Newark and Bergen County, New Jersey, so that officials could quickly share test results and detail problems.

Restoring communications is also key for first responders looking to navigate new towns and cities that are thrust into a crisis, particularly in rural areas. Jim Milsap, a technical division chief in Fulton County, Georgia, was part of the team responding to tornadoes in Alabama and Georgia last year. He said phones and cell coverage are vital for using maps and GPS.

“These people that are coming in from out of town and out of the county, all of a sudden they show up and they need that cell phone coverage just as much as anybody else,” he says. “If you don’t have coverage, you can’t even find the town.”

AT&T isn’t alone in these efforts. Each of the three major US carriers has its own tools and teams, with Verizon and T-Mobile also operating fleets of COLTs and COWs, mobile command centers and relief trailers alongside other tools that are ready to be deployed in case of emergencies. Verizon even experimented with drones that connect to its cell network back in 2016.

Although each provider’s tools are designed for their own networks, they can work together. The National Coordinating Center for Communications, a part of the Department of Homeland Security’s Communications Information Sharing and Analysis (CISA) division based in Washington, DC, helps coordinate efforts between government agencies and wireless carriers before and during disasters. The individual carriers, despite being rival networks, can also allow customers on other networks to take advantage of whatever coverage is available in the area.

Billy Bob Brown, Jr., CISA’s associate director of priority telecom services sub-division, says that the group has regular coordination meetings with its partners in the government and the private sector, but can meet daily in advance of a big storm or even twice a day for more severe events. All three major wireless carriers are listed as industry representatives on CISA’s website for the NCC as are federal agencies. including the Department of Justice, Federal Emergency Management Agency (FEMA) and the Federal Communications Commission (FCC). 

“The best time to trade business cards is not during the middle of the hurricane,” says Brown Jr., noting a saying. “You trade business cards during a blue sky day when you can start talking about, you know, good, coordinated planning together.” 

“So day-to-day coordination, that’s where it’s really critical to develop good relationships.”

A volunteer workforce monitoring the whole way

Although AT&T has a team focused on maintaining its emergency equipment, the group that actually goes into the field also includes trained volunteers from AT&T’s other divisions, including its retail stores.

Patty Daughtery works on the incident command team, which arranges the crews, equipment and other logistics required to respond to a crisis. She says hundreds of employees have signed up to be volunteers and are paid their regular salaries when deployed.

Once a crew is in the field, Daughtery’s team remotely monitors them with GPS while overseeing the recovery process from beginning to end. While Parker was on the ground in Louisiana following Laura, he had support from teams in Austin, Atlanta, Cleveland and Bedminster, New Jersey.

For trips that may require days or weeks of support, the company rolls in trailers to use as a mobile command center for coordinating with public officials, as well as their own teams in the field. Mobile bunks with DirecTV connections, bathrooms and showers are also brought in to house AT&T staffers so the carrier doesn’t need to find housing in an area where newly homeless residents are already crowding hotels and evacuation centers.

With the importance of social distancing in the age of the coronavirus, the company is restructuring how it handles support for its staff in the field. One solution AT&T is considering, Daughtery says, is putting people into “pods,” or groups, to limit their exposure to others. Groups of four to six people will head down and deploy equipment together, eat together and work and sleep in the same areas.

How 5G and LEOs can make a difference

As 5G and other wireless technologies evolve, the emergency equipment that supports the new networks need to change as well. AT&T’s 5G rollouts are still ongoing — the carrier currently has a low-band 5G network that covers over 205 million people — but it’s evaluating how it will add the new faster wireless technology to its fleet of response vehicles.

Beyond the Aerostat blimp, Steve Poupos, AT&T’s director of advanced technology support and the person who oversees AT&T’s NDR program, says the company may go even higher with low Earth orbit satellites, or LEOs, to boost coverage, particularly in rural areas.

A number of companies have been experimenting with using LEOs to deliver internet service from orbit, including SpaceX for its Starlink program and Amazon with its Project Kuiper offering. SpaceX is expected to offer Starlink to consumers later this year, while Amazon recently received approval from the Federal Communications Commission to deploy 3,200 satellites.

“We’re looking at companies like Amazon and all of the companies that are looking to the LEO race if you will, to who’s going to be the first,” Poupos says. “We’re keeping close tabs on that.”

Since they beam the internet down from space, partnering with a company that operates an LEO network would allow the NDR team to help establish cell coverage particularly in remote or rural areas that don’t have strong fiber internet already running on the ground. Poupos believes the technology could be particularly useful for search and rescue missions.

Adding 5G to its tools is similarly on Poupos’ radar, though AT&T is still evaluating which flavors of the technology would work best for its needs. The carrier’s low-band 5G network doesn’t bring a significant speed boost compared to a good 4G LTE connection. It also has a much faster millimeter-wave connection in parts of 35 cities, but the range on that signal is often limited to just sections of a few city blocks.

At a medical triage station, for example, a millimeter-wave connection on a SAT COLT could allow for quickly sending large image files to doctors located remotely to view in real-time.

“We’re trying to establish which will be the most effective technology going forward,” says Poupos. “To have the flexibility and the speed of 5G is going to be really, really important. So that’s why we need to keep cutting edge and we need to continue to improve the program.”