Back in September I published Stop bloating your AGENTS.md: reference your conventions instead of pasting them — and advice I still stand behind: keep AGENTS.md a thin index, move the details into per-topic docs, reference them from one place. The argument was about context hygiene — your conventions stay detailed, your context stays clean. What that argument never claimed to do was enforcement.
It was a solid argument. It was also half an argument, and the second half bit me within two weeks of real agent work.
Because here's the uncomfortable truth I glossed over: a lean AGENTS.md is still just prose. ESLint enforces boundaries because CI turns red. AGENTS.md enforces itself because... nothing does. The agent reads it, nods, and the next time a task touches an edge case the doc doesn't cover, the doc becomes fiction. It doesn't drift loudly. It drifts quietly, one merge at a time, until you open it a month later and realize it describes a codebase that doesn't exist.
So this article is the missing half: how I turned AGENTS.md from advice the agent reads into a contract CI can enforce — the eval-suite moment from part 4 of the JobCopilot series, but for the doc that runs the project. And per house rules of this series, I'm leading with the ways it broke, not the ways it worked.
1. The lie we tell ourselves: "the doc is enforced because it's true"
Before the fix, the honest inventory of what my "lean" doc could guarantee:
- Lint boundaries — enforced. CI.
- Type checks — enforced. CI.
- "New scenario UI goes in
src/features/<feature>/ui/, never app/" — suggested.
- "Never commit to main" — enforced-ish, by branch protection, but invisible in the doc.
- "Use pnpm only" — suggested, until a second lockfile quietly appeared.
The pattern is: everything tooling can express gets enforced; everything that needs a sentence of prose gets vibes. And agents read prose the way juniors read style guides on Friday afternoon.
The fix wasn't writing more prose. The fix was giving every sentence of prose in that doc an ID and a hook that a script can execute.
2. Give every rule an address: <!--check: id -->
The doc now looks like this (real shape from the repo, trimmed):
## Package manager
**<!--check: single-lockfile -->**
This repo uses pnpm ONLY. pnpm-lock.yaml is the single lockfile.
A second lockfile (package-lock.json, yarn.lock) in the repo = CI red.
## Data
**<!--check: no-ui-fetch -->**
src/features/**/ui/** and app/** must NOT call fetch/axios directly.
Data access belongs to feature logic.
Two design decisions worth pausing on:
The marker sits next to the rule, not in it. The prose stays human-readable; the marker stays invisible in rendered Markdown. Whoever edits the doc — human or agent — sees the rule, not plumbing.
Every rule carries a check marker, and every marker is mapped. If a rule appears in the doc without a marker, it's a suggestion. If a marker exists but no check is implemented, that's not a TODO — it's a bug: somebody thought they'd promised the machine something.
The one sentence that made this work: "If CI is red, fix the code OR the rule — never bypass the check." Both directions are legitimate edits. Silencing the check is not.
3. The drift-check: a doc that can fail like a test
The checker is a small Node script — pure Node, no shell tools, because the repo has to verify identically on Windows, Linux and CI. Core of the real file, abbreviated (the npm-era shape — story 3 below renames these ids to cmd-pnpm-*):
import { readFileSync } from "node:fs";
/** AGENTS.md text — the contract being verified. */
const md = readFileSync("AGENTS.md", "utf8");
/** Rule ids from <!--check: id--> markers, de-duplicated, in document order. */
const ids = [
...new Set([...md.matchAll(/<!--check:\s*([\w-]+)\s*-->/g)].map((m) => m[1])),
];
/** Map: AGENTS.md rule id -> verification function.
Return contract: true = pass, false = generic fail, string[] = offending files. */
const checks = {
/** package-lock.json present — npm is the real package manager. */
"cmd-npm-install": () => existsSync("package-lock.json"),
"cmd-npm-lint": () => hasScript("lint"),
/** UI layers must do no direct fetching. */
"no-ui-fetch": () => {
const bad = [...hits("app"), ...hits("src/shared/ui"), ...featureUiHits()];
return bad.length ? bad : true;
},
};
// Two-way binding: doc -> map, and map -> doc.
for (const id of ids)
if (!(id in checks))
problems.push(`rule ${id} has a marker but no check is mapped`);
for (const id of Object.keys(checks))
if (!ids.includes(id))
problems.push(`check ${id} exists in the map but no marker in AGENTS.md`);
Then every rule runs, and any failure exits 1. On a green repo the whole suite takes under a second:
✓ AGENTS.md OK: 11 rules verified · 0.0s
And when things drift — rename a mapped file, add an unmapped marker, plant a fetch in a UI file, all exit-1 cases we actually reproduced in the first week:
✗ rule some-new-rule has a marker but no check is mapped
✗ drift detected: 1 rule(s), 1 binding issue(s)
Since this series lives in Python, here is the same two-way-binding loop as a ~25-line Python twin — the logic is the point, not the language:
import re
from pathlib import Path
# Every rule in AGENTS.md carries an HTML marker next to it.
MARKER = re.compile(r"<!--check:\s*([\w-]+)\s*-->")
md = Path("AGENTS.md").read_text(encoding="utf-8")
# de-duplicated, in document order
ids = list(dict.fromkeys(MARKER.findall(md)))
# Your map: rule id -> function that returns True / list of offending files
checks: dict = {} # {"single-lockfile": ..., "no-ui-fetch": ...}
problems = [
f"rule {i}: marker in AGENTS.md, no check mapped"
for i in ids if i not in checks
]
problems += [
f"check {c}: exists in the map, no marker in AGENTS.md"
for c in checks if c not in ids
]
print(f"AGENTS.md: {len(ids)} rules")
for p in problems:
print(f" ! {p}")
raise SystemExit(bool(problems))
Fed a real 11-rule AGENTS.md with an empty checks map it exits 1 with eleven "no check mapped" lines — exactly the failure you want the first time you wire the markers in. Same discipline as the part-4 evals: the contract has an exit code.
In fsaa-next it runs as its own GitHub workflow — .github/workflows/agents-md.yml, triggered on [pull_request, push] — parallel to ci.yml. The doc stops being advisory.
4. What broke in practice
The part I promised. Four failures, in the order they were discovered.
1. The parser eats itself. I wrote a doc section explaining the marker system and — being an idiot — pasted a literal example: "add a marker like <!--check: my-rule -->". The regex found it, treated my-rule as real rule #12, two-way binding flagged it, suite red. The doc's explanation of the contract violated the contract. Fix: explanations in the doc must never contain a live marker; paraphrase. (The irony was noted and savored.)
2. The check that green-lit too easily. The first version of the import-boundary rule checked whether eslint.config.mjs contained the boundary config — the words, not the effect. The rule was checking the check, not reality: relax or mangle the ESLint rules and the drift-check stayed green while actual cross-slice imports crept in. The rewrite scans the real files for real imports that shouldn't exist (@features/ inside src/shared/, upward imports anywhere), and keeps the config-word check as a second, independent guard. Lesson that cost the rewrite: a check can inspect the enforcement, which is not the same as inspecting the code. Always ask what the worst violating diff looks like — and whether your check catches that, or merely catches a keyword.
3. The package-manager migration nobody scoped. Switching npm to pnpm broke three things, and only one of them was on my list: (a) pnpm/action-setup@v4 conflicts with an explicit version: input when packageManager is pinned in package.json — red in 4 seconds on every push; (b) the old npm ci sitting in ci.yml, a different file, easy to forget; (c) the drift-check's own rule ids, still named cmd-npm-*. Three files, one decision. What saved it from becoming permanent archaeology: the contract rules (single-lockfile, cmd-pnpm-*) were updated in the same PR as the migration — because the contract lives next to the code, the PR diff itself shows all three.
4. Rule rot — the doc version of drift. A refactor renamed app/layout.tsx; the file-paths check went red on the next run, before anything downstream noticed. That's the quiet win of the whole setup: the contract fails exactly like a test nobody had time to write.
5. Verdict: is this overkill?
For a weekend project — yes, probably. Lean prose gets you 80% of the value, and I'll defend that in the next "bloat" debate.
But the 20% gap is where agents live. Every convention not expressible as CI is a convention an agent will eventually violate with total confidence, generating a code review you then do by hand. The drift-check replaces that review with an exit code. On an 11-rule contract it costs a few hundred lines of boring Node/Python and one workflow file — cheaper than a single forgotten lockfile incident.
The honest revision of my September advice: keep AGENTS.md lean, reference your conventions — and then make every remaining sentence falsifiable. A rule that nothing can break isn't a rule; it's a suggestion the next agent will politely decline.
JobCopilot returns next week, with the graph wiring and the negative-test story. If an AGENTS.md or CLAUDE.md of yours has gone quietly fictional, I want that war story — comments open. Follow if you want the rest of the series.
Related from this series: part 1 — the LLM output layer that doesn't break · part 3 — my first LangGraph graph
