If you've worked with SQL for long, you've probably been bitten by NULL. A column that should mean "we don't know" gets coalesced to 0 or false somewhere in a pipeline, and three joins later a report says something confidently that nobody ever actually observed. The bug isn't in any one line. It's in the moment a missing value quietly turned into a real one.
I think the same failure shows up in AI agents, and it's one of the things I tried to make testable in a public benchmark I've been building, UNISON-StateBench. In this post I want to walk through one design idea from it, what it does, and, just as important, what it doesn't show.
The problem: collapsing states
Picture an agent asked whether a remediation step was verified. The evidence it was given doesn't say. There are at least three honest answers: verified, not verified, or unknown. A lot of systems, models and plain code alike, only have room for two. So "unknown" gets flattened into "false" (or worse, guessed as "true"), and the downstream decision inherits a certainty that was never there.
The benchmark's design document states the rule directly: False != Unknown is treated as an epistemic invariant. A model that answers "false" when the evidence is simply missing has lost information, even if "false" happens to match the underlying truth.
Typing the fact, not just the value
The core move in the StateBench design doc is that a fact isn't a bare value. Each canonical invariant is typed as a tuple:
I_k = (value, epistemic_status, provenance, authority)
So "the remediation was verified" carries not only true or false but also how we know it, where it came from, and whose word it is. For full credit on what the benchmark calls Invariant Preservation Rate, a model's answer has to get all four components right, not just the value.
For developers, the takeaway doesn't need the benchmark at all: if your agent's memory or tool layer stores facts as plain booleans, you've already thrown away the information you'd need to tell "no" from "nobody said."
Consistency is not correctness
The second idea I find useful is keeping three properties separate:
- Consistency: the model gives the same answer when the same case is shown as prose, JSON, YAML, a table, or an event log.
- Correctness: the answer matches a deterministic oracle derived from the canonical case.
- State preservation: the answer doesn't merge distinct states, like unknown into false, or "approved" into "executed."
A model can be perfectly consistent and consistently wrong. That's why the design doesn't fold its eight metrics into one composite score. It reports them separately, so a high consistency number can't hide a state-collapse problem.
There's a related rule the doc calls the Non-Collapse Principle: A -> B -> C does not imply A => C unless the required transition evidence exists. "Requested, then approved, then done" is three states, and skipping the middle one is a failure, not a shortcut.
Changing what the model sees without changing the truth
The part that took the most care is how missing-evidence tests work. The benchmark has presentation-only mutations: reordering, paraphrase, contradiction injection, lower-authority conflicts, adversarial value flips, and removing evidence.
When evidence is removed from what the model sees, the justified answer becomes UNKNOWN, but the canonical case and its oracle must not change. So the trial's expectation is kept separate from the immutable oracle. An integrity guard snapshots the case and oracle before every renderer or mutation and checks them again afterward, even if the transform throws. Views are bound to their origin case by digest, so a view built from one case can't be rendered or mutated as if it belonged to another.
The scoring follows through on this too. If a trial deliberately hides the evidence for a value, a model that "knows" the value anyway and asserts it still counts toward the State Collapse Rate. Being right by luck or by prior isn't the same as being justified by the evidence you were shown.
What this does and doesn't show
I want to be clear about the limits, because they matter more than the design.
- These are definitions and a test harness, not findings. The design doc describes an evaluation plan (162 model calls per model in v0.1) and a frozen source commit, but this post doesn't report any model results, and I'm not claiming any model passes or fails.
- Nothing here has been independently reviewed or replicated. It's my own public artifact under an MIT license.
- The ideas aren't new in isolation. Three-valued logic, SQL's
NULL, provenance tracking, and data-lineage work all got here long before me. What the benchmark tries to do is put several of these checks into one deterministic, inspectable test setup for AI agents. I haven't done a formal prior-art review, so I'm not claiming it's the first.
- The repo's own scientific boundary says it doesn't establish that a model "understands truth," and it doesn't establish deployment safety or containment of arbitrary agents.
If you want to poke at it, the design doc includes a short Python example and the standard pip install -e '.[dev]' and pytest setup.
Open questions
The thing I keep going back and forth on is where the line between "unknown" and "absent" should sit, and whether four components is the right shape for a fact or one too many for real agent memory. If you've dealt with NULL-flattening in agent tool outputs, or have a case where this typing breaks down, I'd like to hear how you handled it.