As coding agents become more capable, code generation is no longer the whole system.
A little while ago, we published Before a Paper Becomes Code, where we shared one part of how our team has been working with AI on longer research and engineering projects.
The discussion afterward was useful precisely because several people pushed on the weak points. Writing invariants, provenance rules, or execution stages into the context of a probabilistic model does not make those rules deterministic, and a carefully governed extraction process still cannot recover information that disappeared before the representation reached the model. Both objections were fair.
They also happened to overlap with a messier problem we were already dealing with elsewhere.
Our coding agents were taking on larger pieces of work, and the difficult questions were no longer confined to whether the generated code compiled or whether the tests were green. We were arguing with our own workflow about whether the agent should have touched a file in the first place, whether a successful command had produced the state we thought it had, whether a verifier was measuring the right boundary, and, eventually, whether a PASS meant anything beyond “this particular check returned the result we expected.”
The contract was not “make the process disappear somehow.” We were trying to show that a specific request traversed the lifecycle:
So this is another part of our working method that we have decided to put in the open. It is not a proposed standard. Some of it is plainly too expensive for small or low-risk changes, and pieces of it are still changing as we find new failure modes.
Use whatever is useful and discard whatever is not.
For the article itself we have stayed close to things we could inspect: our own development records, public documentation, and external repositories whose relevant behavior we could verify. When the evidence allowed only a narrow claim, we kept the claim narrow.
The question underneath all of it is narrower than “how should coding agents be governed?” We are interested in what changes once governance and verification stop being activities performed after the agent has finished and begin leaking into the machinery through which the agent is allowed to work.
The controls arrived because the failures did
A coding assistant that proposes a function while a developer keeps their hands on the keyboard can remain fairly informal. The human still controls the write, the command, the merge and usually the interpretation of the result.
That arrangement becomes unstable once an agent can inspect a repository, edit several files, invoke a shell, run tests, touch a database, start or stop processes, reach the network and continue across several steps without a human deciding every transition.
We noticed the change gradually.
Better models let us delegate larger units of work. Larger units of work meant that a mistaken assumption could survive for longer before anybody noticed it, and the agent's own explanation of what it had done became less useful as evidence because the explanation was being produced by the same system that had selected the implementation path.
The temptation is to respond with a better instruction: do not touch files outside scope, make sure the tests really cover the requirement, verify everything before you finish. We still use instructions like that. They help. They also have an obvious ceiling.
A path restriction written in natural language is being interpreted by the same probabilistic model that is trying to finish the task. A path guard implemented outside that model can refuse the write.
Those two controls can sit next to each other and even express the same policy, but they are not equivalent. The difference became harder to ignore as the tasks became less disposable.
One failure made that painfully concrete.
In late August 2026, while testing a remote /close operation against one of our local runtime gates, a live falsifier exposed a PowerShell problem under StrictMode. The defect itself was narrow enough; we patched it and added an actual PowerShell 7 regression test.
The next part was worse because it looked, for a moment, like success.
During an isolated probe, another script rebound a runtime path and dispatched a shutdown to the active stack, which then stopped.
That was the desired end state. It was also useless as evidence for the claim we were trying to establish.
The contract was not “make the process disappear somehow.” We were trying to show that a specific request traversed the lifecycle:
REQUESTED
→ DISPATCHED
→ CLOSING
→ CLOSED
The probe had killed the process by another route. Nothing about the dead process established that the requested causal chain worked.
We now had two defects to think about: the original /close problem and a test-isolation problem that had managed to produce the right external state for the wrong reason. Counting the second event as a successful /close would have made the report cleaner. It would also have been false.
A different experiment produced the same discomfort without anything crashing.
During a mid-August 2026 RS-A/RS-B verification pass in our Reasoning Recovery work, we were testing whether a score intervention changed candidate content and obtained content_changed = 0. That was exactly the value we hoped to see. The result was reproducible.
An audit then found that the snapshot had been taken after the governance boundary specified in the contract.
We had a stable number from the wrong observation point. We moved the snapshot, and the next audit found another hole: the selection artifact recorded that twelve candidates had entered but did not record their identities.
Counts matched, which looked reassuring until we asked what would happen if one experimental arm disappeared.
A later falsifier showed that an entire arm could be removed and the verifier could still compare the surviving data against itself.
Nothing dramatic happened on screen. The verifier simply continued to agree with itself.
That case changed the way we thought about clean results. Zero difference, twelve entries, a green command, a dead process. None of these observations was intrinsically weak.
Their value depended on what had produced them, where they had been observed and whether the mechanism doing the checking was capable of noticing the failure we were excluding.
By then, “run the tests again” was no longer a serious response to every failed gate. The defect might sit in the implementation or in the test itself, while changes in the environment could invalidate both.
On a few occasions, the verifier was the component we had to break deliberately before trusting its result.
That started changing the contract we gave the agent
Our AGENTS.md-style files used to look much more familiar: repository conventions, local commands, constraints on changes, coding practices.
The verification failures made that increasingly difficult to defend. The arm-removal case above was enough to make the problem concrete: telling the model to “verify carefully” would not have fixed the measurement boundary that allowed the verifier to agree with itself.
We needed the contract to distinguish between what the agent was being asked to respect and what another mechanism could actually check.
The version below is a shortened operational example based on the direction we are using now. Most of the contract is still semantic guidance. The executable part shown here is a scope check that compares the task boundary declared before mutation with the Git-visible changes it can observe. CI, independent review, runtime verification and release authority only become stronger controls when a project binds them to mechanisms outside the implementation agent.
If the scope check cannot decide, the work does not advance.
# Agent Operating Contract
This file is an operating contract, not an enforcement engine.
instruction != enforcement
execution != correctness
test pass != verification
verification != independent verification
release eligibility != release
record of a check != the check having happened
## Enforcement Grades
DETERMINISTIC
Produced by a mechanism whose result the implementation
agent cannot silently rewrite.
SEPARATE_CONTEXT
Produced by a reviewer or verifier process that
did not author the implementation.
SELF_REPORTED
Produced by the implementation agent itself,
including files it can write.
INDEPENDENTLY_VERIFIED cannot be SELF_REPORTED.
RELEASED cannot be claimed below DETERMINISTIC.
## Scope
Before mutation, record the scope declaration in:
.agent/task.json
Include:
task_id
base_ref
allowed_write_paths
allowed_delete_paths
authorized_protected_paths
network
external_writes
risk
At SCOPED time, record the SHA-256 of:
.agent/task.json
as:
state.json.task_digest
The scope check recomputes that digest on every run.
digest mismatch
-> TASK_DIGEST_MISMATCH
-> SCOPE_VIOLATION
-> exit 2
Run:
python3 .agent/scope_check.py
0 = SCOPE_OK
2 = SCOPE_VIOLATION
3 = SCOPE_CHECK_ERROR -> HELD
An absent verdict is never SCOPE_OK.
## Working Evidence
For material work, retain:
.agent/
task.json
state.json
evidence/<task_id>/
manifest.json
diff.txt
commands.jsonl
review.md
falsification.md
These records are not automatically trusted merely
because the implementation agent wrote them.
## Verification
For every material claim, record:
CLAIM
ARTIFACT
PROCEDURE
OBSERVATION_BOUNDARY
EVIDENCE
AUTHORITY
GRADE
RESULT
A correct final state reached through the wrong causal
path does not verify the claim.
## Falsification
For consequential changes, ask:
Would this check fail if the implementation were wrong?
Use an appropriate negative control, mutation,
break-the-test probe, or held-out fixture.
If a required falsifier is inconclusive:
HELD
## Independent Review
A same-context second pass is not independent review.
If no actual separate reviewer or verifier route exists:
independent_review = NOT_RUN
Do not claim INDEPENDENTLY_VERIFIED.
## Failure Handling
Classify before retrying:
IMPLEMENTATION_DEFECT
TEST_DEFECT
ENVIRONMENT_DRIFT
VERIFIER_DEFECT
SCOPE_VIOLATION
EVIDENCE_GAP
AUTHORITY_GAP
DEPENDENCY_FAILURE
UNKNOWN
Do not repeat an unchanged failing operation simply
to obtain a different outcome.
## State
REQUESTED
-> SCOPED
-> IMPLEMENTED
-> TESTED
-> VERIFIED
-> INDEPENDENTLY_VERIFIED
-> RELEASE_ELIGIBLE
-> RELEASED
Any pre-release state:
missing evidence
ambiguous evidence
scope violation
missing authority
|
v
HELD
HELD returns to the stage that owns the resolved defect.
RELEASE_ELIGIBLE is not RELEASED.
This is an operational example, but it is not self-enforcing.
The scope gate can check what Git can expose to it. It observes changes already committed on the branch as well as staged, unstaged and untracked work.
Write authority and delete authority are evaluated separately. Permission to modify a path does not silently become permission to remove it.
Changes to AGENTS.md, the scope checker, CI configuration, .gitignore, CODEOWNERS and related enforcement files are treated as control-plane mutations. If the checker cannot produce a trustworthy verdict, the result is HELD.
Its observation boundary is narrower than the agent's possible effect on the machine:
- Git-visible changes are checkable. Git-ignored writes, symlink targets, parts of submodule behavior and writes outside the repository are not fully covered.
- Agent-written records remain agent-written records.
state.json, manifests and falsification reports help coordinate the workflow, but a VERIFIED or DETERMINISTIC label inside one of those files does not authenticate itself.
- Independent authority has to be bound externally. A reviewer is only independent when a genuinely separate reviewer route exists. A release state has operational authority only when something outside the implementation agent controls release.
- The grade field itself is not a security boundary. Nothing in this operational example prevents an implementation agent from typing
DETERMINISTIC into state.json. Until an external mechanism corroborates that grade, the field remains SELF_REPORTED.
That last limitation matters because deterministic computation and independent authority are different properties. A checker can produce the same verdict from the same inputs every time while still sitting inside a control plane that the implementation agent is able to rewrite.
The scope checker gives this example one executable boundary. It does not turn the rest of the contract into trusted infrastructure.
There is some conceptual overlap here with supply-chain provenance systems such as SLSA and in-toto. They also force a distinction between a recorded statement and a statement whose producer, artifact and trust boundary can be established. The grades above are not meant to replace those systems. They are a narrower vocabulary for asking who or what produced a claim during an agentic coding workflow, and how much authority that claim should carry inside the next transition.
The contract therefore does not try to compress every green-looking result into the same status.
A test can pass without establishing runtime behavior, while a genuine runtime observation can still come from the wrong causal path. An independent reviewer has another failure mode: the review can be sincere and technically competent while examining an artifact that is no longer the one scheduled for release.
Approval introduces a separate boundary. Permission to perform an operation says nothing about whether that operation was later carried out correctly. Once software starts consuming these states and deciding what happens next, the distinctions stop being documentation trivia.
HELD exists for the cases where nothing has been disproven but the available evidence is still insufficient to advance. Missing provenance or an unobserved runtime transition can leave a claim unresolved without turning the implementation itself into a failure.
An unavailable independent verifier creates the same kind of gap because sometimes the evidence simply runs out. Forcing that state into PASS or FAIL would create exactly the kind of closure this contract is meant to resist.
The same pressure is showing up in very different systems
The repositories we found do not form one school of thought and they should not be flattened into one. Their concerns range from security enforcement to independent review, persisted-state verification and formal methods.
The interesting part is that several of them move some important decision away from the free-form judgment of the agent that performed the work.
| System | Control it introduces | Failure pressure behind it |
|---|
| Sondera Coding Agent Hooks | deterministic reference monitor, Cedar policy, trajectory-aware enforcement | a model instruction is not an execution boundary |
| deep-review | isolated reviewer context and cross-model evaluation | the author of a change should not be its only evaluator |
| revmux | supervised find → synthesize → verify review with archived runs | review itself can be opaque, unstable and unauditable |
| skeptic-audit | independent audit with a break-the-test probe | a test can pass because it is incapable of detecting the defect |
| AgentSkeptic | read-only persisted-state verification and Outcome Certificate | a successful trace does not prove the intended state exists |
| lemmafit | Dafny verification before bounded generated logic proceeds | some correctness questions need not terminate in another LLM opinion |
The table keeps the six main control patterns separate. The LocalLLM hidden-contract example [14], discussed below under falsification, is treated here as a related held-out-oracle pattern rather than a seventh control category.
1. From instructions to enforceable boundaries
Sondera Coding Agent Hooks is one of the cleaner examples of the difference between asking a model to respect a boundary and moving the boundary outside the model.
It intercepts shell, file and web actions, normalizes them into events and evaluates deterministic Cedar policies that can allow, deny or escalate the action. The hook fails closed if the enforcement harness cannot be reached.
It can also use trajectory history, which matters because an operation that is harmless in isolation can become unsafe after an earlier operation has exposed a credential or changed the information available to the agent.
The repository is primarily about action security, not about every question of software correctness that concerns us.
Still, it demonstrates something we had been learning locally: semantic restraint and executable restraint solve different problems. You can use both. Pretending the first has the authority of the second is where the trouble begins.
2. From self-review to evaluator separation
The pressure behind deep-review 5 is different.
Its documentation is explicit that the agent which wrote a change has a structural blind spot when it reviews the same work. The evaluator operates in a separate context and receives the review payload rather than the originating session's reasoning, intentions and assumptions.
In Claude Code it can use isolated Codex reviewer sessions; under Codex the corresponding roles can run as history-free native subagents.
This is not perfect independence. The reviewers still share code, requirements, often the same model families and whatever biases are embedded in the review criteria.
But it removes one obvious contamination path: the implementation narrative does not automatically become the reviewer's starting explanation for why the implementation makes sense.
Its surrounding deep-work project was more interesting to us for another reason.
The workflow extends beyond “ask another model to review this diff.” It includes specification and planning phases, TDD enforcement, receipts, mutation testing, phase guards, worktree restrictions and exit gates. Some of those constraints are inferential; others are computational and can block actions.
That mixture is close to the problem we were wrestling with after the discussion around our previous article: where should semantics remain probabilistic, and where should a rule stop being prose and become software?
3. From review output to auditable review
revmux 6 attacks the review process itself. It supervises Claude and Codex subprocesses through a fixed find → synthesize → verify pipeline, but the more useful detail is what survives afterward.
The archive contains composed prompts, verbatim model output, intermediate findings, retry and stall events, runner information and a manifest. The verifier is not simply handed one giant review conversation either; finding groups can be isolated so one verification task is less likely to anchor on neighboring findings.
That does not make the review true. It makes the review inspectable.
This distinction tends to disappear in agent demonstrations because the output is the thing being shown. In a long project, the invisible process that produced the output starts to matter.
A review that existed only inside yesterday's conversational context is hard to challenge next week. If a subsequent agent disagrees, you may not even know whether the difference came from the code, the prompt, the reviewer roster or the context.
Once those inputs become artifacts, the disagreement has somewhere to attach.
4. From testing code to testing the test
The small skeptic-audit 7 repository does less, but one of its checks cuts straight into a problem that had already cost us time: it performs a break-the-test probe.
Instead of accepting a test because it passes against the implementation, the auditor asks whether the test fails when the relevant assumption is deliberately violated.
We had arrived at roughly the same requirement from our self-comparison bug. If a verifier reports “no difference,” we want to know what happens when a controlled difference is inserted.
If the result remains “no difference,” the clean result has just collapsed as evidence. The verifier may still be useful for something else, but it is not measuring what we thought it was measuring.
A small LocalLLM project [14] takes a related route with a deterministic hidden-contract oracle that the worker agent cannot see.
Visible tests can be modified, hard-coded around or accidentally weakened. A held-out behavioral condition is harder to optimize directly because the worker does not know the check it is trying to satisfy.
This does not eliminate gaming. The hidden oracle can be wrong, incomplete or stale. The design still becomes much easier to understand once the agent is capable of editing the evaluator it can see.
5. From traces to actual state
AgentSkeptic 8 is concerned with a different gap.
An agent can report that it executed the expected operation and still leave the persistent system in the wrong state. AgentSkeptic reconstructs the expected persisted state, reads the actual database through read-only queries, compares the two and emits an Outcome Certificate.
One limitation in that design is more important than the certificate itself: its claim is deliberately narrow.
A matching SQL state at verification time does not prove deep causality, prove the absence of every side effect, or retroactively make the agent's trace trustworthy. It says something more modest about the observed state.
That restraint is easy to underestimate. Verification gets dangerous when the confidence of the surrounding prose expands faster than the authority of the measurement.
6. From LLM review to a non-LLM verifier
At the other end of the spectrum, lemmafit 9 moves a bounded class of correctness questions out of LLM review altogether.
Claude Code writes effect-free business logic or state machines in Dafny, a daemon runs dafny verify, and only verified logic is compiled for use by the surrounding TypeScript application.
Obviously this does not generalize to most of a production stack. Real systems leak through network calls, databases, queues, deployment state, timing assumptions and APIs that will not politely turn themselves into proof obligations.
It still marks an extreme point on the same map.
There are places where “ask another agent whether this looks correct” is simply not the strongest verifier available.
What made the survey more interesting for us was seeing weaker forms of the same separation in tools much closer to everyday coding-agent use: prevent the action, require another context, inspect actual state, preserve evidence after the run.
The boundary is moving into the runtime
Claude Code's hook system 10 allows intervention at points such as PreToolUse and Stop. A tool call can be blocked before execution. Completion can be refused.
Agent-based hooks can invoke another context with access to repository files and tools, allowing it to inspect actual state or run checks before returning a decision.
This is flexible machinery rather than a complete governance model, and a badly designed hook can create a new failure surface just as easily as it closes an old one.
The architectural change is nevertheless hard to miss: the model's assertion that it is done and the runtime's decision to let it stop no longer need to be identical.
OpenAI's description of how it operates Codex internally 11 exposes another version of that split. Sandboxing provides technical constraints, approval policy governs actions beyond those constraints, network access is managed, and agent-native telemetry preserves a record that can be inspected separately from the model's narrative of what occurred.
OpenHands 12 also separates action-risk analysis from confirmation policy and supports deterministic analyzers alongside model-based analysis.
We would not combine those systems into one claim beyond that. Their threat models, enforcement mechanisms and product goals are different.
The useful signal is narrower: control surfaces around the model are becoming normal enough that the interesting question is no longer just whether an agent can call a tool. The harder questions concern what governs the call, what evidence remains afterward, and what the system is permitted to do with the result.
There is a secondary problem hiding inside that progress.
Once hooks, permission systems and verifiers become trusted control infrastructure, bugs in those controls matter more. A DEV Community article about Claude Code hooks [13] makes the point in very practical terms.
Synthetic events can cheaply test deterministic hook behavior. The author then recommends a second tier using a real headless Claude session in a throwaway repository, where assertions depend on side effects the model cannot merely claim away. For example, whether a protected secret appeared or whether an unwanted commit actually exists.
That is the same unpleasant recursion we found locally.
We added a verifier because we did not fully trust the agent. Then we had to ask why we trusted the verifier.
There is no obvious final layer.
Why the resemblance matters
The projects above do not constitute one coordinated movement. Some are primarily security systems, some are review harnesses, one is concerned with persisted state, and another pushes a narrow class of logic into formal verification. There is no evidence that these developers are trying to converge on one standard.
What repeats is the pressure that preceded the design.
A natural-language restriction proves too weak for a hard boundary and the restriction becomes executable. An implementation agent is too close to its own reasoning and a second context appears.
Review output becomes difficult to reconstruct, so the review inputs and decisions begin surviving as artifacts. A green test cannot demonstrate that its oracle works, so somebody deliberately breaks the thing being tested. Elsewhere the trace says success but persistent state disagrees, and the verification moves to the state itself.
These are related responses, but forcing them into one architecture would erase important differences.
“Convergent evolution” is useful to us only at that level. Software has repeatedly produced similar structures when different teams encounter the same class of failure. Transactions, consensus mechanisms and least-privilege boundaries are obvious historical examples, but the analogy should not be pushed into a prediction that every coding agent will acquire the same governance stack.
We are too early for that.
These are related responses, but forcing them into one architecture would erase important differences. Separation-of-authority controls face a different problem because better reasoning does not remove the need to divide authority. Cost will eliminate another class of controls when maintaining the assurance system becomes more burdensome than the failures it was meant to catch.
The cost question matters. Every independent reviewer consumes time and tokens. Held-out tests become another thing that can rot. Receipts and digests introduce state that can go stale. A release workflow can become so defensive that developers simply find a route around it.
We already have tasks where the full machinery would be absurd. A typo does not deserve the evidentiary burden of a release-capable runtime change.
Any serious version of agent governance will eventually have to answer the proportionality problem.
“Harness Engineering” gives the surrounding system a useful name
The recurring pressure above does not make all of these projects one discipline, but a separate body of work has started treating the machinery around the model as an engineering object in its own right.
Böckeler 1 describes guides and sensors around the coding agent: mechanisms that shape what the agent sees and does, and mechanisms that expose what happened so behavior can be corrected.
The useful part for us is what follows from that. A repeated model failure does not have to remain a sentence repeated in every new conversation. If the same failure keeps returning, some part of the response can migrate into the repository itself. The response can become a test, a linter, a hook, a local rule, a tool or another structural constraint that survives after the original session has disappeared.
Zhong and Zhu 2 push the surrounding-system view further. They treat software-engineering capability as a property of a model–harness–environment system and enumerate components including task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing and intervention recording.
Agent runs in that framing can become auditable episode packages rather than transient interactions.
The architecture is considerably broader than our current workflow, but the places where it accumulates structure feel familiar: authority before execution, state during execution, evidence afterward, and some mechanism deciding whether the evidence is sufficient to advance.
Lin et al. 3 get close to another issue we encountered. Harness modifications are paired with predictions and later checked against subsequent task outcomes, which makes a harness edit something that can fail rather than merely become the new configuration.
Their ablation result is also worth noticing. In the reported experiments, improvements localized to tools, middleware and long-term memory, while the system prompt alone did not carry the same gain.
That does not make prompts obsolete. Our own AGENTS.md exists because semantic guidance still has work to do, especially around interpretation, scope and decisions that are difficult or foolish to encode as deterministic policy.
But a rule that repeatedly proves useful does not necessarily have to remain a sentence forever.
Repeated rules can become tests, path guards or artifact checks. Decisions that lose important context when formalized should remain judgment calls.
We do not yet know where that line should settle.
The architecture we are experimenting with now
The following sequence is not something we found in an emerging standard. It is simply a compact representation of where our own work has ended up after those failures and after looking at the systems above.
Intent
→ Scoped Authority
→ Agent
→ Tool Boundary
→ Sandbox / Policy
→ Execution
→ Runtime Evidence
→ Reviewer
→ Verifier
→ Provenance
→ Release Gate
Even written that way it looks cleaner than the real work.
Actual tasks skip stages, loop backward, become HELD before implementation is finished, invalidate yesterday's evidence after a small edit, or reveal that the environment cannot support the claim we wanted to make in the first place.
The arrows name responsibilities. They are not a transcript of debugging.
What we are trying to preserve is risk-sensitive depth.
A small local edit may need only a bounded patch, targeted tests and a diff review. A database migration can justify replay against a real fixture and inspection of resulting state.
Security-sensitive work may justify a fresh reviewer, a mutation probe or tighter execution controls. Once an agent has authority to release something, the bar changes again because its own interpretation of success can now produce an external consequence without another person necessarily intervening.
At that point AGENTS.md, hooks, skills, sandbox policy and orchestration begin to look less like miscellaneous agent customization and more like parts of a control plane, although even that phrase makes the present systems sound tidier than they are.
Some of our policy still exists only in prose because we have not found a safe or useful deterministic representation for it.
Some of our deterministic checks later turned out to encode the wrong assumption.
Moving a rule outside the model changes the failure mode. It does not make the rule correct.
PASS has become conditional language for us
After enough of these incidents, PASS stopped feeling like a state that could stand comfortably by itself.
We still use the word constantly in ordinary tests, fixture checks and contract gates.
Whenever the word appears alone, we now ask what actually passed, which artifact and observation boundary the result refers to, which environment produced it, and whether a controlled negative case could have made the check fail.
Whenever the word appears alone, we now ask what actually passed, which artifact and observation boundary the result refers to, which environment produced it, and whether a controlled negative case could have made the check fail.
We also ask whether the evidence came from something able to rewrite it, whether the artifact has changed since verification, and what the result actually authorizes next.
Our current vocabulary is one attempt to keep those questions from collapsing:
REQUESTED
→ SCOPED
→ IMPLEMENTED
→ TESTED
→ VERIFIED
→ INDEPENDENTLY_VERIFIED
→ RELEASE_ELIGIBLE
→ RELEASED
None of these labels deserves standardization because we happen to use it. The point is to avoid assigning one state to evidence that means different things.
TESTED says something about tests. VERIFIED says that a defined claim was checked through a defined procedure.
INDEPENDENTLY_VERIFIED adds a separation the preceding state does not have. RELEASE_ELIGIBLE still does not mean the thing has been released.
When a human reads every result and decides every next action, these distinctions can remain fuzzy because human judgment fills the gaps.
When software consumes the state and chooses what happens next, the gaps themselves become executable.
We are less certain what the “agent” will mean a year from now
A large part of AI coding discussion still revolves around model capability: which model writes the better patch, which one solves the benchmark, which one can keep more of a repository in context.
Those questions remain important, but they describe a shrinking fraction of what we now spend time debugging.
We debug the model's decision, then the hook that constrained it, then the environment in which the hook executed. We find that a test passed against one artifact while the next command used another.
A reviewer can be independent in context and still depend on the same faulty fixture. A deterministic verifier can eliminate model variance while faithfully enforcing the wrong contract.
Provenance can tell us exactly where a bad decision came from without making the decision any better.
Calling the whole thing “the coding agent” starts to feel imprecise.
What we actually interact with increasingly resembles a collection of components with incomplete authority: something proposes and edits, something limits actions, something observes runtime state, something challenges the result, something records which artifact was checked, and somewhere near the end another mechanism decides whether the available evidence is enough to expose the result to the outside world.
The organizational metaphor is tempting. We can name an implementer, reviewer, verifier, policy engine, runtime and evidence store. The metaphor still has limits.
Two separately named agents running the same model against the same faulty fixture can agree very confidently. A verifier maintained through the same workflow can inherit the assumptions of the system it is supposed to challenge.
A receipt can be perfectly authentic and document a perfectly bad run.
We have not solved those problems.
What changed is that we no longer treat them as peripheral to coding-agent design. We began with an agent that wrote code and discovered, defect by defect, that a large part of the engineering burden had moved into deciding what should be allowed to count as evidence around that code.
Looking outside our own projects did not reveal a canonical architecture. It showed us that other builders were entering the same territory from different edges.
The next step cannot simply be adding every available control until the agent can barely move. We still need to work out which boundaries deserve deterministic enforcement, where probabilistic judgment remains useful, how independent a verifier really needs to be, and how much assurance a particular task can afford before assurance becomes the larger system.
The contract was not “make the process disappear somehow.” We were trying to show that a specific request traversed the lifecycle:
We do not have a clean answer to that yet.
That is the more useful place to stop.
References
- Birgitta Böckeler. 2026. Harness engineering for coding agent users. MartinFowler.com / Thoughtworks.
Hailin Zhong and Shengxin Zhu. 2026. AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents. arXiv:2605.13357.
Jiahang Lin et al. 2026. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. arXiv:2604.25850.
Sondera AI. 2026. Coding Agent Hooks. GitHub repository.
Sungmin Cho. 2026. deep-review: Independent Evaluator for AI Coding Agents. GitHub repository. Related project: deep-work.
umputun. 2026. revmux: Multi-agent code review, supervised and auditable. GitHub repository.
Prudai. 2026. skeptic-audit. GitHub repository.
J. Kavanagh. 2026. AgentSkeptic — State vs Trace. GitHub repository.
Midspiral. 2026. lemmafit: Make agents prove that their code is correct. GitHub repository.
Anthropic. 2026. Claude Code Hooks Reference. Claude Code Documentation.
OpenAI. 2026. Running Codex safely at OpenAI. OpenAI, May 8, 2026.
OpenHands. 2026. Security Architecture. OpenHands SDK Documentation.
Otto. 2026. Stop your coding agent from cat-ing .env: a Claude Code hooks cookbook. DEV Community, August 11, 2026.
sai_vineeth98. 2026. A coding agent in 5 files, plus a skeptic that catches fake fixes — runs local (Ollama/vLLM). r/LocalLLM, July 23, 2026.