Every SHA, every raw number and every recovered file below is traceable to the repo. Where something broke, the commit that fixed it is named. Where something is still open, it says so.
On September 24, a commenter named Tom Jones dropped a table into the discussion under our bootstrap-pipeline article: identical retriever, store, query and ranking — only the returned unit varied. Ten chunks: 9/14. Whole document: 5/14. No retrieval: 6/14. And the one count that survived: his chunk ranker's top hit landed on the answer-bearing document 0 times out of 14.
We ran his protocol on our index. This is the story of everything that had to break first — over nine days and 143 commits — before a single one of those numbers could be trusted.
Act I. The Rollback That Started It (Early August)
Months before any of this, in the thread under Tom's "A stable aggregate is not a stable measurement", he told the story that set our methodology: a config change measured 3/4 before against 2/10 after, production was rolled back on it — and the rolled-back config then measured 4/10. It had been noise the whole way. His answer was a noise-floor gate that refuses and states the n it would take, because "3 of 4 against 2 of 10" means nobody has measured anything yet. We adopted the rule wholesale: no claim below the noise floor, ever. Everything that follows was built to survive that gate — including, as you will see, our own published numbers.
Act II. Linking Tests to Functions (Sept 15–22)
The bootstrap pipeline needed one impossible edge: which function does a test actually execute? Name matching died at 0 out of 109 (d74785d4). A custom sys.settrace plugin traced all 1,727 tests: 1,551 (89.8%) execute ≥1 src function, 10.1 per test on average — "1 test = 1 function" was a myth, so edges became the full trace, no ranker (8a94026d, Sept 16).
Then the confounds queued up. Tarantula ranking as a selector: refuted (rank≤3 covered 22.6%). coverage.py as a faster driver: refuted (+19.96% vs +13.6%). Static signals: 70% union recall, companion only. Portability held on foreign repos (97.3–100%). E17 wired TESTS edges into search (c1fb2363, Sept 22, PR #39) — but the pilot first returned 120/120 Error 500, because methods are Class.method and the naive matcher looked for def name. Fixed with an AST extractor. Then the v1 judge's "completeness B>A, p=0.0046" turned out to be a judge-context artifact — the judge had seen each arm's own tests. Fix: identical reference across arms, effect erased to p=1.00, and a standing rule was born: the judge's reference must be identical across arms.
Act III. September Burns (Sept 19–20)
E10 (full-text embeddings) was refuted on a clean reindex (hit@1 0/10 fast, 2/10 quality) — code reverted to HEAD the same day (15ce15d7). E11 whispered a signal: +2 hit@5 at +6ms. E11 stayed a behind-flag candidate (A-prepend, N=10) — not shipped, not dead, awaiting a bigger panel. Then the crash: a full reindex died at 330K chunks after ~9 hours, losing everything, because run() held 1.3 GB of embeddings for one final bulk write. Fix (0fcb0f33): flush every 32 files, resume from known hashes, integrity check at start — and the rule that governs this project to this day: no experiment runs without a frozen baseline. E12 confirmed the 8-hour reindex was physics, not a bug. E13 refuted doc-chunks (hit@5 12.5% vs code's 50%/40%) — the first sighting of the two populations.
Act IV. The Week Everything Lied (Sept 22–25)
F4 said deepseek scored 0/3 and was "invalid as a model." It was an undercooked reasoning budget. Pinned rerun: 3/3, verdict corrected, Pitfall #19 logged. Held-out control #14 (email spacing) fired ~100% — it was a semantic twin of a catalogue sentence; replaced with an out-of-domain Docker tag, validator scripted. The frozen 16-item symptom list lived in %TEMP% and was deleted by a temp cleanup — recovered from the session database (opencode.db, write rid=75331, verified against read rid=75864), SHA256 a6f719df…, frozen in-repo with a guard test so it can never happen again. Sept 25 brought a permanent reindex deadlock (fixed by splitting locks, ledger added, 858.5s completion) and a foreign MCP server squatting our fixed ports :8080/:8081 — hence the standing rule: чужое не трогаем.
Act V. The Four Arms (Sept 26)
Pre-registered (5f158ece), harnessed (02b0781a), judged (0a1c212f, 516 files, manifest with SHA256 per artifact). Frozen 16 (8+8, sha e048aa12…), index snapshot 10106/716/14099, reader longcat-2.0, blind judge qwen3.7-plus, 10 trials, strict >50% majority:
| Arm | Judged (n=160) | Code (n=80) | Prose (n=80) |
|---|
| A top-k chunks | 26 (16.3%, CI 0.11–0.23) | 5 (6.3%, CI 2.7–13.9) | 21 (26.3%, CI 17.9–36.8) |
| B whole document | 55 (34.4%, CI 0.27–0.42) | 40 (50%, CI 39.3–60.7) | 15 (18.8%, CI 11.7–28.7) |
| C oracle | 156 (97.5%) | 76 (95%) | 80 (100%) |
| D closed book | 0 (160× "I don't know") | 0 | 0 |
Objective half: A hit@1 and B top-1-is-gold both 5/16 — the unit did not move retrieval. Judged: code 8× on non-overlapping CIs (the unit acts on the reader); prose A>B as direction only (overlapping CIs, the +6 sits on two queries that flip between snapshots, majority a 2:2 tie) — but 18.8% vs closed-book 0% means the strong prose-collapse prediction did not reproduce here. Majority strict: A 2/16, B 5/16 (one B point rested on a 5/5 tie and was counted out — stated, not hidden), C 16/16, D 0/16. Judge non-unanimous 6/64; repro t5→t10 within noise.
Act VI. Side Quests (Sept 27)
NodeRAG (153cdc6d): frozen 10 rule-queries + 3 positive + 3 NONE controls against our two long diary files — chunked TF-IDF 8/10 at 302K tokens vs graph BFS 7/10 at 170K, all controls clean. Verdict: REFUTED. Every graph miss returned zero files — no seed symbol, no traversal.
The logits murder mystery. A production-retriever run started losing targets at the reranker: MMR 10→10, reranker 10→2, raw scores like −6.88 and −0.99. Cause: llama.cpp returns raw logits (≈[−11,+11]) while MIN_RERANK_SCORE=0.3 was calibrated for [0,1] — the filter ate 70–97% of results. Fix: sigmoid in the llama_cpp branch only, threshold untouched, 7 regression tests plus a mutation check. Post-mortem sting: the P3 target normalizes to 0.271 < 0.3 — the model itself rates the true file at logit −0.99, so the threshold was never the root cause. P3 stays open: the sigmoid fixed the scale contract, not the recall — the model itself rates the true file below threshold, which is a ranking problem no filter fix can solve. Two traps logged: measuring the filter on survivors (P-002), and landing the sigmoid in the ONNX block first (P-003, caught by diff review).
The janitor work nobody photographs. A cp1251 mojibake from a PowerShell write (lesson: never write Russian markdown through PS). A privacy scrub — 49 tracked files with usernames, 115 with local paths — before answering Tom (PR #49). A pre-commit hook that failed twice on xdist worker crashes while the suite was green, passing on the third attempt with zero code changes. A merge (d86a4a45, PR #50) that conflicted in three diary files, silently dropped 7 origin entries in an earlier commit (restored), and needed a 429→204-line archival of known issues to satisfy the size gate. The lint commit (0cec4a63) fixed 4 ruff errors, two of which — an unreachable allowlist line after return, a blind except — had been sitting in HEAD blocking every commit.
Behind the Scenes: Three Days, One Laptop
Three days, one personal Windows machine, a single live index serving everything — no separate lab copy. Harness through opencode CLI at 4–8 threads: 640 reader calls plus the same in judge calls, then the trials=5 pilot on top. Public repo, so nothing is rewritten — invalid runs and snapshots sit on disk next to the real ones. No history rewrites, no cleaned-up reruns presented as first attempts. 143 commits since September 1; every number above names its file.
Anthropic's Contextual Retrieval (2024) cut retrieval failures 49% (67% with rerank) on their evals including codebases, via per-chunk LLM context prepended before embedding — compatible with, not identical to, our arm-B reader effect. Jina's late chunking (2024) conditions chunk embeddings on full-document context with gains growing on longer docs in their BEIR eval — same direction, prose/long-doc only. Neither splits code vs prose; our split is ours, pilot-scale (n=16), reader-specific.
Red Team
E17: 10 threads × 100 calls, 0 errors; 234-test hub at 16.1ms median; empty queries → 0 rows (null-behavior checks); mid-call close → empty results, no errors. F5 self-attacks: the 5/5 tie restated strict (B prose 1/8, B ALL 5/16 — hidden tie-breaks are how published tables mislead); "0 invalid" true for t10, false for the t5 pilot (F5S-03/B); gold-as-document overstates prose miss (7/8 prose queries had gold in neither arm, yet the reader scored via duplicate translations); t5 retrieval contexts overwritten and unrecoverable — verdicts reproduce, contexts do not; cheap reader bounds our numbers, not readers.
What Could Go Wrong
- n=16 pilot scale. Prefer CIs over bare rates; every % above carries Wilson 95%.
- Index snapshotted, not frozen. Drift documented, not eliminated — freeze before the next reader run.
- Judged noise band not re-run. t5→t10 is reproducibility, not noise measurement.
- No per-arm token costs. Context chars only — no cost claims attached.
graph_score = 0.4 shipped as an operating point (MRR held); ablation vs BM25/reranker not run.- Cap experiment on our own logs not run. Tom's numbers stay his.
- TESTS-signal limits carry over (Python-only, +15.3% wide-panel overhead, test nodes without line numbers, 10.2% mock blindness).
Acknowledgments
A huge thank you to everyone who engages with these posts in the comments. Your feedback, real-world observations, counter-examples, and benchmark numbers directly shape these experiments. Tom Jones handed over the full 4-arm protocol with controls and a falsifiable prediction for our index. This kind of open technical critique is what keeps engineering honest.
A note on how this was written
Every experiment, bug, failure, and idea here is mine — I earned them the hard way, in production, in public. AI worked as my editor: it helped me structure thoughts and polish my English. It did not invent the facts, because it has none of its own.
No AI detectors were consulted in the making of this disclosure. They have enough trouble agreeing on what I am.
Disclaimer & Status: draft (source-material for the article). This is not a "feature advertisement", but an honest engineering story: figures are reproducible, weak points are named, and unaddressed risks are listed in the "What Could Go Wrong" section.