FCC study pins all blame on AT&T for massive mobile outage

　　The errors and foibles that led to AT&T's massive mobile network outage in February that impaired millions of devices and blocked millions of voice calls were "all attributed" to the carrier, the FCC's Public Safety and Homeland Security Bureau concluded in its damning assessment of the outage.

　　The FCC's 29-page report (PDF) found that the size and scope of the outage resulted from a litany of factors that fell at the feet of AT&T, including a configuration error, a lack of adherence to the company's internal procedures, a lack of peer review, inadequate lab testing and insufficient safeguards and controls to ensure approval of changes that impact AT&T's core network.

　　The FCC's report arrives five months after a misconfiguration of AT&T's network during a routine maintenance window in the wee hours of February 22 sparked the outage.

　　Misconfiguration led to a massive outage

　　Per the FCC's description of the event, the misconfiguration delivered by an employee – and done without proper peer review – was not detected before it was introduced into the AT&T network. That caused AT&T's network to enter "protect mode" and trigger "an automated response that shut down all network connections to prevent the traffic from propagating further into the network," the FCC explained.

　　The outage occurred at 2:45 a.m. Central Time, roughly three minutes after the misconfigured network element was put into production, the FCC said.

　　"The shutdown isolated all voice and 5G data processing elements from the wireless towers and switching elements, preventing these services from being available," the FCC explained in the report.

　　Based on info provided by AT&T, the FCC said the outage affected more than 125 million registered devices, blocked more than 92 million voice calls, and prevented more than 25,000 calls to 911 call centers across all 50 US states, Washington, DC, Puerto Rico and the US Virgin Islands. The outage also impacted FirstNet subscribers, which AT&T prioritized when it moved to restore service.

　　The FCC noted that it took nearly two hours for AT&T to roll back the network change and at least 12 hours before full service was restored in part because AT&T's device registration systems were overwhelmed as re-registration requests flooded the network.

　　In addition to the lack of peer review, the FCC said AT&T asserted that another procedure error – namely, involving post-installation testing – occurred. Additionally, AT&T's own lab testing also did not discover the improper configuration. "Any such testing should have identified the issue prior to the occurrence of the outage," the FCC report said.

　　Next steps

　　The FCC said AT&T has "taken numerous steps to prevent a reoccurrence," noting that the carrier implemented additional technical controls in its network within 48 hours of the outage.

　　"In addition, post-outage, AT&T has implemented additional steps for peer review and adopted procedures to ensure that maintenance work cannot take place without confirmation that required peer reviews have been completed," the report added.

　　The FCC said it plans to release a public notice, based on this week's report and other recent outages, reminding service providers of the importance of implementing relevant industry-accepted best practices.

　　The Commission has also passed the matter to the Enforcement Bureau based on the Public Safety and Homeland Security Bureau's investigation for "potential violations" of FCC rules.

　　Following the outage, AT&T said it would issue $5 credits to customers, costing the company up to $140 million, according to one estimate.

　　AT&T is scheduled to announce Q2 2024 results Wednesday morning (July 24).