Most of the bugs that taught me something weren't crashes. The system worked, the checks were green, and the property I cared about was still broken. Here are four of them, from three different projects.
1. The prompt-injection test that passed while the attack worked
In LLM-Council, three models answer a question, rank each other's answers blind, and a chairman model writes the final synthesis. Output from one stage becomes input to the next, so I had to fence untrusted model output off from the instructions around it.
My first version used static delimiters. I wrote tests that checked where the markers were placed and how the string was built. They passed.
The problem: those tests checked that the Python string was assembled correctly. An attacker doesn't care about that. A hostile model output can simply reproduce the static closing delimiter, and the next model reads the untrusted block as finished.
The fix was a fresh random nonce in the delimiters for every prompt, so untrusted output can't predict the real closing marker. I found a second boundary that needed the same treatment: the rankings going into the chairman stage were untrusted too.
Then I broke the fix on purpose. I replaced the random nonce with a static one and checked that the tests went red. I removed the ranking fences and checked that those tests failed as well.
One limit worth stating: the nonce protects the boundary against delimiter forgery. It doesn't make the content inside trustworthy, and it doesn't solve prompt injection in general.
Full write-up: I wrote a test for prompt injection. It passed while the attack worked.
2. Retrieval that returned valid results for the wrong decision
MonferrinoAI is a RAG pipeline (Firecrawl, Voyage AI embeddings, Supabase with pgvector) that prepares weekly content briefings for a local business website. A human still writes and publishes.
I tested retrieval against the real database and found that a keyword-only query could rank the wrong page above the one that was actually relevant. Nothing was technically broken: every result was a valid row. Adding context to the query, the keyword cluster and the target page, fixed the ranking.
The same project has a security side. Web pages are untrusted input and can contain text that looks like an instruction. So the page bodies never reach the final briefing: the handoff only carries URLs, headings and similarity distances.
3. Telemetry that carried more than I expected
Agentic-OS collects Claude Code telemetry through an OpenTelemetry Collector, with Prometheus and Grafana for metrics and a private Loki pipeline for logs. The point is to see what an agent actually did, not what it says it did.
Testing against the real client showed that it can send identity attributes, including an email address. My monitoring pipeline was a privacy boundary and I hadn't treated it as one.
Now attributes go through an allow-list, and the tests run against the real telemetry shape. Anything unknown is dropped instead of stored.
4. A strict CSP that the build quietly broke
My website uses a strict hash-based Content Security Policy without unsafe-inline. Syntax highlighting added inline styles that the policy would block.
The quick fix was to loosen the CSP. I removed the incompatible behavior instead, and added CI tests that run against the built HTML, the files that actually ship, checking inline scripts, hashes and style attributes. Some of those problems only exist after the build, so testing the source wouldn't see them.
The question I ask now
In all four cases a check existed and looked reasonable. It just wasn't looking at the property I cared about.
So before I trust a control, I ask: what should turn red if this property stops being true? Then I break the property and see if it does.
This post grew out of my Developer Story on CoderLegion, written by Mehadi Hasan, which covers the projects and the path behind them in more detail.
