Innovation & Tech

Five Outages in Three Days: Network Change, Not Conspiracy

Outlook, air traffic control and major banks all failed within three business days in mid-November. The cause wasn't a conspiracy — it was unmanaged network change, and AI is about to make it worse.

By Amara Osei

5 min read

5 major network outages in three days: It’s not a conspiracy; it’s change
5 major network outages in three days: It’s not a conspiracy; it’s changeClickerHappy / pixabay

What's News

  • Email, air travel and banking suffered major outages within three business days in mid-November, including a U.S. ground stop on Friday the 13th.
  • A nationwide air traffic ground stop on Friday, December-like November 13 stranded travelers; bank customers lost account access on Monday, November 16.
  • The time between vulnerability discovery and exploitation has shrunk from a quarter or a year to mere hours, per the analysis.
  • Much of the world's critical infrastructure is 30-40 years out of date and faces forced modernization.
  • Microsoft's team reportedly spent days rebuilding its environment after configuration-triggered failures.

Email, air travel and banking — three pillars of the American economy — all crashed within the span of just three business days in mid-November. A Microsoft Outlook outage left businesses unable to send email. A nationwide air traffic ground stop on Friday the 13th stranded travelers at airports like Fort Lauderdale, where hotel rooms vanished within hours. On Monday, November 16, customers at several major banks found they couldn't reach their accounts.

The reaction was predictable. Conspiracy theories multiplied on social media and, as the analysis puts it, "in the dark corners of the internet." People who don't know a router from a switch were suddenly debating network infrastructure.

The real explanation is less dramatic but more daunting. "The infrastructure that runs the world is changing faster than ever, and change always brings risk," the analysis states. When organizations fail to manage changes in their network infrastructure, the result, as the country saw that week, is chaos.

What caused the air traffic failure?

The issue at the root of network outages is almost always change. In the air traffic control case, the network that carries communications between the major control centers is decades old and in the middle of a long modernization — moving from legacy protocols and transmission to modern fiber and new equipment.

At one step of that transition, something went wrong when new network devices were introduced. A misconfiguration or a misbehaving device sent traffic into the network that other devices couldn't handle. Those devices began to fail, and the failure cascaded. Shortly after, the rush for hotel rooms began.

The Outlook outage had the same origin: configuration changes, unintended consequences, knock-on failures. Some of the triggering configurations were already months old and sat quietly until a condition in the network finally exposed them.

Why can't networks just heal themselves?

Networks are supposed to route around damage. In theory, they should. In practice, change shakes up a network infrastructure in unpredictable ways. A network is, by definition, a set of interacting systems: change one element and you change how its neighbors behave, and how their neighbors behave after that. When a change is bad, it gets worse as it travels.

The teams who run critical networks practice serious discipline before any change: pre-change testing, a mapped-out plan for every step, and a rollback plan. The analysis is confident the people involved that week did all of it. But the interconnections inside a large network are so vast that no team can test for every eventuality. "That's not carelessness. It's the nature of complex, interconnected systems," the author writes. It could happen to even the most prepared team.

Redundancy helps, and these networks have layers of it. But redundancy does not remove complexity — in some ways, it adds to it.

Why does recovery take so long?

If a change caused the problem, the simple fix is to undo the change. It's not that easy. The same interconnectedness that caused the failure makes it hard to reverse: a problem becomes exponentially more difficult to fix as it cascades and tangles itself into a web of systems. Recovery requires three steps:

  • Diagnosis. When effects cascade across the network, it's often not obvious which change caused which symptom. An accurate picture of the network — what's connected to what, how each element should be configured, and what's happening right now — makes the difference between a fast fix and one that takes many times longer while the team works in the dark.
  • Remediation. Rollback is just the start. By the time a cause is found, the problem has usually triggered other failures, and network elements have to talk to one another again in a careful order so applications return cleanly. Microsoft's team reportedly spent days painstakingly rebuilding its environment to avoid further knock-on effects.
  • Regaining confidence that the problem won't come straight back. The initial fix doesn't always work. When lives depend on the answer, caution is always the right call.

The author compares the process to heart surgery on a global stage. Teams get it right 99.99% of the time, but almost nobody notices. When there is an outage, they do delicate, high-stakes work while the public watches the departure boards.

Why outages will keep coming

The rate of change in the world's infrastructure is accelerating for three reasons:

  • Decades-old systems are hitting their limits and must be modernized, as with air traffic control. Modernization requires change, and change always involves risk.
  • AI is putting new demands on networks. Agents and AI-powered applications are doing real work, and that work runs over infrastructure that can't adequately carry it without growth and adaptation.
  • AI brings new security implications — the most urgent factor. The industry now talks about the "patchpocalypse": AI is uncovering vulnerabilities faster than ever, some hidden for years or decades. The time between discovery and exploitation has shrunk from a quarter or a year to mere hours. Every patch is a change, and every change carries risk.

Put together, these factors produce infrastructure that's 30 or 40 years out of date and needs to change almost immediately. The networks aren't necessarily more fragile, but their criticality, complexity and necessity for change are growing faster than teams and funding can keep up.

What IT leaders should do now

Many of the world's most important networks are still managed largely by hand, with limited data, limited visibility and limited tooling for governing change. That approach isn't sustainable anymore. The analysis calls for two investments:

  • Give teams real data about their infrastructure. They should know what the network is, how it's interconnected, how it's meant to be configured and what's happening on it right now. In a crisis, they should see at a glance the impact, the affected workloads, the change that likely caused it and the downstream effects.
  • Govern change before it reaches the network. Change drove all the high-profile outages in mid-November. Modern patch windows are measured in hours rather than months, so teams need discipline built into tools and processes.

The people running these networks are doing extraordinary work under extraordinary pressure with very little recognition. The November outages showed how much of the world's economy depends on them — and until they get the data and tools they need, expect the shakiness to continue.

Source: Fast Company

Share this article:

More from Amara Osei

Amara Osei

Show full bio

Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.

2,503 articles

Related articles

« Previous article