I've drawn the diagram a hundred times. Two data centers, a load balancer, N+1, tidy little redundant boxes with tidy little redundant arrows. Ship it, sleep fine.
Every one of those diagrams was quietly lying to me about the same thing -- and last July, that exact lie took Pennsylvania's statewide 911 system down. Not a slice of it. The core, in four places, at once. No hacker. No ransomware. A bug in an operating system.
Let me walk the mechanism, because it's one most of us have shipped, and then let me pick a fight about it.
My unpopular opinion: most of what we call "high availability" is redundancy theater. We count instances and call it resilience. The count is almost never the thing that saves us, and Pennsylvania is a clean, public proof.
What actually happened
Pennsylvania's Next-Generation 911 runs on Next Generation Core Services (NGCS) -- the components that take a 911 call, resolve the caller's location, and route it to the right PSAP (the 911 center). The state runs four NGCS instances across two physically separate data centers. One vendor, statewide contract.
On July 11, around 2 p.m., calls started failing intermittently: dropped connections, calls landing with no callback number or location, quiet stretches then bursts. About 85 minutes in, the state pushed a Wireless Emergency Alert telling people to use county non-emergency lines. A majority of calls kept going through, and no statewide failure count was ever published. Restored around midnight.
Eleven days later, the official finding: "a defect in an operating system," "not believed to be" a cyberattack. No component named, no confirmed trigger.
One wrinkle worth your attention: in the days after, the state told at least two outlets the problem involved a firewall operating-system error. The formal write-up dropped that word. So the thing that failed may have been part of the security stack itself -- which makes "not a cyberattack" a strange kind of comfort.
Why four copies didn't help
Here's the part that should bother you. Four instances. Two data centers. And one defect degraded all of it across a state.
Redundancy buys availability only when the redundant units fail independently. Four boxes behind a balancer save you from one box dying, one rack losing power, one site flooding -- uncorrelated, mostly physical failures. They do almost nothing against a fault in something all four share: the same build, the same config, the same change pushed to all of them in the same window. That's a common-mode failure, and it's the blind spot "just add another copy" creates.
The whole idea in eight lines:
# "Redundancy" only helps if this is False for every shared layer:
shared_fate = (
same_build # all instances run the identical artifact
or same_config # all pull from one config / secret source
or same_deploy # one push updates all of them at once
or same_upstream # shared DB / DNS / auth / control plane
)
# If shared_fate is True anywhere:
# N copies of one flaw = ONE failure domain, not N.
# Your "replication factor" is a load-balancing number, not a resilience one.
NIST SP 800-160 says the quiet part out loud: redundancy is susceptible to common-mode failure, and the mitigation it points to is diversity -- making the copies genuinely different -- not more identical copies.
A second data center survives losing a place. It does not, by itself, survive losing a piece of logic that every place is running.
I'll be honest about the evidence: the public record doesn't prove an identical defect sat on every node. A shared config, a shared update path, or a synchronization mechanism could produce the same correlated failure. Doesn't matter for the argument. If the failure domains aren't independent, the instance count is theater.
And before you write this off as my hot take: Pennsylvania's own after-action review landed here too. When the state's outside reviewer briefed the 911 board in December, one recommendation was multi-vendor diversity -- explicitly not building your redundancy on one vendor and one software lineage. A board member pushed on the obvious objection (doesn't multi-vendor create its own interop and coordination mess?), and the reviewer's answer was the honest one: yes, it costs more, but it's the structural fix for the outage they'd just eaten. Hold onto that "yes, it costs more," because it's the whole tension in the next section.
What this does NOT solve
Here's where most write-ups stop and start feeling smug. So let me do the part that actually earns the argument -- what recognizing this does not fix.
- Diversity isn't free. Version skew, ringed deploys, an instance deliberately kept a release behind -- these cut common-mode risk and hand you a new steady-state problem: config drift, a testing matrix that multiplies, and 2 a.m. pages that open with "why is node C different." You're trading a rare correlated failure for a constant low-grade tax. Sometimes that's the right trade. Sometimes it isn't.
- It moves common-mode risk, it doesn't kill it. Skew your builds all you want; you may still share a control plane, a DNS zone, an auth service, or one tired operator running the same bad command on all four. Independence is a spectrum, not a checkbox.
- It doesn't resolve the trigger. We still don't publicly know what set Pennsylvania's off. The day-one "not a software update" and the later "a vendor update triggered it" were never reconciled. You can architect perfectly and still be guessing at causes.
- It can't reach the thing that actually blinds you: disclosure. Here's the fair version. The follow-up work isn't buried -- the vendor summarized its root-cause analysis to the state's 911 board in September, and in December the board got the outside reviewer's findings, recommendations, and a corrective-action update (better multi-channel outage notifications to PSAPs among them). What's still missing is the granular technical cause: which OS, which component, what triggered it, how it propagated -- even though the same vendor runs statewide 911 cores in eight other states. You can design flawless failure domains and still be blind, because the one thing you'd actually use to check your own exposure is the part that stayed in the room.
The two levers that aren't code
If you run or buy anything life-safety-adjacent, two contract terms matter as much as the architecture. Know exactly where the vendor's responsibility ends -- Pennsylvania's plan draws it at the demarcation gear in each center -- and pin down the incident-disclosure timeline. There's a federal floor: FCC rules now require covered 911 providers to notify affected centers within 30 minutes of discovering an outage. APCO wanted fifteen. Your contract can demand better than the floor.
And if you think this is a one-off: a 2025 NENA/Carbyne survey of 1,379 professionals found 88% of emergency communications centers had some kind of technology outage in the past year. The mechanism isn't exotic. It's Tuesday.
The fight I actually want to have
Intentional diversity -- version skew, ringed config, a deliberately-lagging instance -- is the textbook fix for common-mode failure. It's also a real operational tax, and I've watched teams cause more incidents chasing drift than they ever prevented. So where's your line? Is deliberate diversity worth it in your stack, or is it a cure worse than the disease it's vaccinating against?
And the one I really want to hear: has shared fate ever bitten you in a way your redundancy was supposed to prevent? What was the shared layer nobody drew on the diagram?