Claims such as “90% less output” or “50% less context” are useful diagnostics, but they do not answer the question that matters for a coding agent: did the complete, verified task become cheaper?
I ran a small matched experiment to see how different that answer can be from a local compression percentage.
The comparison
The task was to rewrite the Rust CLI eza as a behavior-compatible Python implementation and satisfy 52 harness assertions. I used GPT-5.6-sol High with Codex CLI 0.144.1 and ran each condition twice.
| Condition | Mean score | Observed tokens | Modeled cost | Mean rounds | Mean time |
|---|
| No plugin | 78.85% | 6.660M | $5.282 | 62.5 | 895s |
| Ponytail | 80.77% | 7.56% lower | 8.87% lower | roughly equal | 13.51% longer |
| RTK | 76.92% | 13.20% higher | 7.18% higher | 44% higher | 40.69% longer |
Ponytail looked cheaper in these two runs, but it took longer. RTK used more tokens, cost, rounds, and time. The important caveat is that n=2 is nowhere near enough to claim a causal plugin effect.
The within-condition cost range divided by the mean was 43.25% without a plugin, 51.69% for Ponytail, and 30.78% for RTK. Those swings are much larger than the apparent 7–9% cost differences between conditions.
Why the denominator changes the conclusion
A separate audit of 140 Codex CLI runs contained 901,608,531 tokens and $680.34 in modeled API cost. The distribution was:
| Token class | Share of tokens | Share of cost |
|---|
| Cached input | 96.46% | 63.91% |
| New uncached input | 3.16% | 20.94% |
| Model output | 0.38% | 15.14% |
If a tool compresses only model output by 90%, the direct effect on total token volume is at most about 0.34 percentage points in this dataset. Extra tool calls, retries, or quality loss can easily erase that saving.
That does not mean compression tools are useless. It means their benefit should be measured at the trajectory level, not by the size of one prompt fragment or one command result.
What I would measure
For any token-saving agent tool, I would record at least:
- total billed cost per verified successful task;
- final task quality;
- number of agent rounds;
- elapsed time;
- retries and failed attempts;
- variance across repeated runs of the same task.
The full methodology, source breakdown, assumptions, and limitations are in the analysis. The implementation and benchmark materials are in the Tura repository.
Disclosure: I maintain Tura and wrote the linked analysis. I am sharing the n=2 limitation and the unfavorable RTK result because the measurement method matters more than making one tool look good.
How are you measuring agent efficiency today: per prompt, per run, or per verified task?
