A signal has three states and most code stores two. When “cannot measure” is written with the same value as “fine,” the alarm branch becomes unreachable in exactly the situation the alarm exists for.
On 7 December 2021, the operations teams inside AWS lost the ability to see their own network. Not the ability to fix it. The ability to see it. The company's post-event summary says so in plain language: congestion "immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it" (Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, "Issue Summary" section). While customers refreshed a status page, the people responsible for one of the largest computing regions on earth "relied on logs to understand what was happening" (same document, same section). Even the public dashboard was caught in it: "the networking congestion impaired our Service Health Dashboard tooling from appropriately failing over to our standby region" ("Event Communication" section).
The lesson usually drawn from that day is deployment advice. Monitor your monitors. Host the status page somewhere else. Both are sound, and both miss what makes the incident interesting. The monitoring did not miss the failure. It could not report the failure, because the failure and the reporting path shared a dependency. Those are different defects, and the second one has a signature you can find in your own code without waiting for an outage.
Three states, two values
A monitoring signal has at least three states: measured and bad, measured and fine, and could not measure. Most of the code we audited encodes the third state with the same value as the second. When that happens, every branch that would have raised the alarm becomes unreachable in exactly the situation the alarm exists for. Unreachable in the strict sense: while the sensor is the thing that broke, no input can produce the alarm value.
You can check for this by construction rather than by misfortune. For each signal you own, ask one question. What comes out when this cannot be read, and can a healthy system produce that same value?
We asked that question about our own system
Over one working night we audited four independent monitoring and status signals in a single codebase we operate. All four collapsed "unknown" into "fine," each in its own way.
| signal | what "I cannot tell" produced | consequence we measured |
|---|
| a deploy-status probe | a lag warning with an empty count | fired on 100% of runs for 28 days; the "no lag" branch was unreachable |
| a fleet health dashboard | the verdict STALLED | reported a fully up-to-date system as stalled |
| a simulation's scarcity input | 0.0 | a region with no data read identically to an abundant one |
| an item-identity lookup | a guess derived from the display name | a garment classified as a liquid container |
The 28-day figure is the interval between a format change we can date and the night of the audit. For four weeks a warning fired on every single run, which meant it warned about nothing, which meant nobody read it.
The arithmetic version of the defect is the sharpest, so we shipped it as a runnable script. Take a health score averaged from three weighted inputs, alarm threshold 0.70. Two of the three inputs are dead and report the calm value, 0.0. The one live input is screaming at 0.95. The script prints:
naive sum over all three : 0.285 -> HEALTHY
renormalised over LIVE only: 0.950 -> ALARM
Identical inputs. And 0.285 is a ceiling, since 0.95 is the largest value the live input can report. With two legs dead and voting calm, the alarm was never rare. It was impossible. Renormalising over the inputs that can actually be seen asks the question the operator is actually asking: given what I can observe, how bad is it?
The one that got it right
The same codebase contained a fifth signal that had the answer built in, and it convinced us the fix is a value rather than a discipline. Its unknown state maps to 0.20, on a scale where its two calm states are 0.10 and 0.05. Nothing else about it is clever. An unrecognised input simply produces a number that no quiet state can produce, so the downstream branch for "something needs a look" stays reachable at all times. It asks nothing of anyone's memory and no change of habit. The distinguishable value does the remembering.
That is the difference between this and one more essay telling engineers to be more careful. The four broken signals were written by people being careful. The fifth was written so that carefulness was not load-bearing.
Honest scope
Our data is one codebase, one night, one investigator, and after the first instance we were primed to see the shape everywhere. The four signals are independent subsystems, which is the sample's strength, but this is a report from our own floor and no survey of anyone else's software.
The check
Skip the moral. Run the audit. For each signal you operate, ask what it emits when it cannot read, and whether a healthy system can emit the same thing. If the answers collide, you do not have a monitoring gap. You have an alarm branch that is unreachable precisely when it matters, and one distinguishable value fixes it.
AWS's operators spent that December morning reading raw logs because their telemetry answered "I cannot see" in a voice indistinguishable from silence. Ours answered "fine." Same bug. The only reason we found ours first is that we went looking.
Reproduction: every number we computed comes from repro_unknown_equals_healthy.py, shipped beside this essay. Run it and disagree with the arithmetic directly.
Sources: Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region (aws.amazon.com/message/12721), sections as cited in-text. All other figures are our own measurements, per the reproduction script and the scope note above.
An audit trail has the same three states as a monitor.
Everything above turns on one value: what a signal emits when it cannot see. The same question applies to the record a system leaves behind, where “no entry” and “nothing happened” are the pair that get confused. Chain of Consciousness is a tamper-evident record written as the work happens rather than reconstructed once somebody asks, so a gap in the record reads as a gap instead of as a quiet success. It does not make a system behave. It makes the difference between unobserved and fine a thing you can check.
Hosted Chain of Consciousness · Verify a record
pip install chain-of-consciousness · npm install chain-of-consciousness
Originally published at vibeagentmaking.com.
