Innovation & Tech

Microsoft says a July bug and reduced capacity drove global outage

Microsoft restored service by Saturday after a 24-hour global outage. A July networking defect, reduced server capacity and heavy U.S. traffic compounded the failure, Microsoft said.

By Amara Osei

3 min read

Microsoft identifies months-old bug as a root cause of global outage
Microsoft identifies months-old bug as a root cause of global outageTho-Ge / pixabay

What's News

  • Microsoft restored global service by Saturday after a 24-hour blackout.
  • The disruption began Thursday and spread worldwide the next day, affecting millions.
  • A July latent defect surfaced during the peak U.S. traffic window, Charles Lamanna said.
  • Over 14% of Microsoft’s server capacity was unavailable because of an unrelated issue.
  • Microsoft issued its initial report Monday and plans a final report with more information.

By Amara Osei

Microsoft said it restored service by Saturday after a 24-hour global outage that crippled Outlook, Exchange, Office 365 and other software. The disruption began Thursday with interrupted Outlook access in the United States and spread the next day, when millions worldwide reported losing email and broader Office 365 access.

Microsoft’s initial report, issued Monday, identified a months-old bug malfunctioning connections between internal systems as a root cause. Microsoft executives said each factor might have been resolved on its own, but their combined effect produced a major outage that was harder to mitigate.

What triggered Microsoft’s global outage?

Charles Lamanna, Microsoft’s executive vice-president of Copilot, Agents, and Platform, traced the failure to a latent defect dating from July. In a Monday post on X, he said the defect caused Microsoft’s servers to become overwhelmed and unresponsive.

Microsoft typically has two available networking gateways, the entry and exit points that connect separate systems and allow data to pass between them. One was not working for Microsoft Teams, which overwhelmed the primary gateway, Lamanna wrote.

Over 14% of Microsoft’s server capacity was also unavailable because of an unrelated issue from the previous day. The reduced capacity and increased load then exposed the July defect, according to Lamanna.

“Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July,” Lamanna wrote. “This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.”

Microsoft’s account presents the failure as a chain rather than one isolated fault. Teams traffic weighed down the primary gateway, an unrelated problem removed over 14% of capacity, and the older defect then blocked servers as requests accumulated. The chain also ran through the peak traffic window in the United States, Lamanna said.

Why did service recovery take so long?

The July defect predated the incident by months, so Lamanna said it would not be easy to undo by rolling back a recent change. Recovery took longer because servers returning to service were immediately overwhelmed by waiting traffic and became unresponsive again.

Microsoft’s normally automated recovery processes were not designed to restore service from that broad unresponsive state. The company’s normal rollback, flighting and feature flags were also unable to quickly revert the change, according to Lamanna.

Lamanna said the combination defeated three standard responses at once. Rollback, flighting and feature flags could not quickly revert a months-old defect, while automated recovery could not handle servers that remained broadly unresponsive.

Microsoft plans to publish a final report with more information, Lamanna said. The company issued an apology after restoring service and did not attribute the blackout to a cybersecurity breach.

What was the outage’s wider impact?

Millions of people worldwide said they could not access email in any capacity, as well as other Office 365 services. Microsoft said the disruption reached essential services far beyond corporate email:

  • Jails could not release prisoners.
  • Hospitals turned to paper records.
  • Schools lost contact with parents.
  • Police departments could not receive crime reports.

The report’s analysts said the global outage highlighted the need for companies to examine how much they rely on a single operating system. Matt Hull, vice-president of cyber intelligence and response at NCC Group, said: “This is a prime example of the importance of not only cyber resilience, but business resilience.”

Hull said reports of affected hospitals, schools and other essential services demonstrated the wide impact of a system outage regardless of cause. His warning covered operational continuity as well as cybersecurity, after Microsoft said no cybersecurity breach caused the blackout.

Microsoft’s final report is the next point of scrutiny because it promises more detail on a failure that defeated rollback, flighting, feature flags and automated recovery. The company will need to explain how it prevents months-old defects from colliding with reduced capacity and peak U.S. traffic.

Original: nccgroup.com

Share this article:

More from Amara Osei

Amara Osei

Show full bio

Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.

2,136 articles

Related articles

« Previous articleNext article »