Every Check I Built Passed. Most of Them Were Measuring Nothing.
I am not a developer. I am a systems engineer working in public safety infrastructure and emergency communications, which means foundation and networking are my lane and application development is emphatically not.
Over the summer I built a reference site for AI model evaluation data, using AI as the implementation layer. Around 900 tracked models, roughly 2,300 static pages, free ungated CSV and JSON exports, no accounts, no tracking, no API. Postgres as the source of truth, Python and Jinja2 rendering to flat static HTML, nginx bound to loopback only, reached through an outbound tunnel so there are no inbound ports at all.
Getting code written was never the hard part. If you write code for a living you already knew that.
The part worth writing up is a failure mode I hit over and over, because I think it generalizes well past my project, and because I suspect some of you have hit it and filed it under a different name.
Six green checks, all pointed at nothing
Every check I asked for got built. Most got built well. A startling number were aimed at the wrong layer, so they reported healthy while the thing they existed to protect was broken.
From a single build:
nginx -t returned zero while not one security header was actually being emitted. Syntax validation confirms the file parses. It says nothing about what the running server does with it. The only check worth anything here is curl -I against one URL of every distinct route type, reading what actually comes back.
The subscribe handler answered 200 on every request, including the ones the mail provider rejected. It caught the POST and replied on its own authority. The upstream response was fetched and then thrown away. Visitors would have hit a confirmation screen while the list stayed empty.
The front page freshness badge was reading the build timestamp. Builds run on a schedule whether or not new data arrived, so the badge would have kept saying the data was current, indefinitely, while the ingest pipeline sat dead underneath it. Live check. Wrong clock.
The log rotation test was green because rotation had never once executed. There was no failure available to find. I had written a check whose passing state and whose never-ran state produce identical output.
Rate limiting was correct nginx and useless nginx. The site sits behind a tunnel, so every request arrives at the server from 127.0.0.1. Keying a limit zone on $binary_remote_addr therefore drops the entire internet into one shared counter, which trips on aggregate volume and never on an individual abuser. The working version keys off the forwarded client address instead:
map $http_cf_connecting_ip $client_ip { ... }
limit_req_zone $client_ip zone=perclient:10m rate=Nr/s;
That gets per-client limiting without ever writing a visitor IP to disk. The point is not the fix. The point is that the broken version looked entirely correct and would have gone on looking correct forever.
A frontend overflow check measured scrollWidth. scrollWidth cannot see clipping that happens inside the scroll container it is measuring. Real measurement, taken honestly, structurally incapable of detecting the exact failure it was written to detect.
Not one of these threw an error. All of them produced green.
The rule: verify by making it fail
What I ended up with is one line. Prove the check can fail before you trust it passing. Every check earns a negative control, which means you deliberately break the thing and confirm the alarm goes off, in both directions, before the green means anything.
Written down that reads as obvious. It was not obvious while it was happening, because a comparison of two empty strings will print MATCH, and MATCH looks exactly like a pass at eleven at night. So I started breaking things on purpose. Comment out the header block, re-run, confirm red. Point the freshness query at a stale table, confirm the badge goes hostile. Restore, confirm green. Only then does the check exist.
The model I was working with was genuinely good at constructing checks and genuinely bad at knowing which layer a check belonged at. Asked to account for that in writing, it said so plainly: it had reported passes on checks that were measuring nothing, and it needed to be pushed into negative controls before it would demonstrate that a test could fail at all. I do not think that is a prompting problem. Catching a wrong-layer check requires already knowing how the layers connect, and that is knowledge you cannot acquire at the moment you need it.
Where the rule stops working
Negative controls fix a category of problem. They do not fix this one, and I want to be honest about the boundary.
My site computes a composite quality metric across a fixed set of benchmarks. Qualifying requires real coverage in science, math, and coding. In July that metric was rendered against all 908 models I was tracking at the time. Sixty-one of them qualified.
A 14.9x overclaim, on my own front page, on a project whose entire premise is that the numbers are trustworthy.
Every test passed. The page rendered. The arithmetic was correct. It was a lie by presentation, and the only thing on earth that catches it is a human deciding that a denominator has to describe the population the metric was actually measured on. There is no assertion you can write for that.
Same category, twice more. I retired an index after working out that it was a single-component index, one benchmark wearing a new name, which is the same dishonesty in a smaller package. And I reversed a design decision outright when measurement disproved the premise I had built it on: I had assumed capability indices were subsets of the composite. They are siblings. I found that out by measuring rather than by continuing to assert it.
An AI will build you a beautifully tested, fully passing, completely dishonest chart, and every gate you put in front of it will wave it through.
The one I never solved
For a stretch this summer, builds stopped shipping. A numeric audit over the clamp and ceiling logic kept rejecting output that looked, to me, comfortably inside bounds. Not every run. Enough runs to block publishing.
Then it stopped, and I had changed nothing that should have touched it.
I want to be precise about this: I did not fix it. It resolved and I was the beneficiary. Worse, no alarm was wired to it, so I only learned it had been happening because I went digging through build logs looking for something unrelated. There is an alarm now. There is still no cause.
This is the one I point at when someone tells me developers are optional now. Knowing a build stopped is trivial and any tool will tell you. Knowing why a numeric guard intermittently rejects its own output, and whether you are looking at bad input, a floating point rounding boundary, or a race between two stages, requires having understood the system before it broke.
The glob that removed the guardrails
The agent runs on my box with broad latitude. What constrains it is not policy, it is a set of subagent definitions with hand-trimmed tool lists: nothing that writes gets a shell, every auditor is read-only. Those definitions live in a dotfile directory. That directory is the entirety of my enforcement.
Moving the tree to a new path, I ran this:
cp -R /source/* /dest/
* does not match dotfiles. The definitions stayed behind. Nothing errored, nothing warned, and the build carried on looking precisely as it had, because a constraint that is absent generates no signal at all. It simply widens what is permitted.
cp -R /source/. /dest/
Trailing dot. I have known that for twenty years, and on that afternoon it was the whole distance between a fenced agent and an unfenced one. The safety margin is thinner than the tooling makes it feel.
The question I actually have
Two of them.
The clamp failure is a genuine open case. If you have watched a numeric guard intermittently reject its own output and then quietly stop doing it, I want to know what yours turned out to be. Bad input, a rounding boundary, a race, something else entirely.
And the broader one: do you have an actual practice around negative controls, or is it case by case for you too? I arrived at mine by getting burned six times in a row, which is a bad way to arrive anywhere. I would rather learn how people who did this properly do it.