We Retracted Our Own 12.2× Benchmark. Here Is What Was Wrong With It.

Question

TokenCap's v1.6 claim, "12.2× fewer tokens, same answer", is retracted. Here is why:

• No model was called, so "same answer" was never checked.
• The naive baseline included TokenCap's own TOKENCAP.md output.
• A Windows temp path (C:\Users\...) made one task match every file.
• One capsule per repo was reused for all five tasks.
• 12.2× was a mean of ratios; 94% was a ratio of averages.

Fixed, the small repo's capsule (8,275 tokens) was about twice the naive read (3,880).

The replacement, benchmarks/fair/, uses 100 real commits and grades against the files each commit changed. Across all 100 tasks: 87% of needed files present, median 18.9× fewer tokens than reading every matching file.

Details at tokencap.vansharora.app