How do you benchmark a context tool without grading it on its own terms?
TokenCap's benchmarks/fair/ uses 100 real commits from nine repos (hoppscotch, flask, httpx, gin, cobra, ripgrep, gson and others). Task = commit subject. Answer key = the non-test source files the commit changed. Repo checked out at the parent commit.
All 100 tasks:
- Pack: 87% of needed files, 9,997 tokens avg
- Grep every match: 96%, 382,258 tokens
- Median saving: 18.9×
Held-out half: 88%, 18.4× median.
Limits we publish with it: grep-everything finds more, small repos gain little (1.6× median), and no model is called, so it is not an answer-quality score.
Method at tokencap.vansharora.app