How to measure whether a token optimization saved anything

Every tool in this space, ours included, can show you a number that says it saved tokens. Most of those numbers are produced by the tool being measured. A method that cannot be wrong is not a measurement.

This is the procedure we use for our own benchmarks and the one we would ask you to use on us. It takes an afternoon for one workload and it settles arguments that dashboards cannot.

Why tool counters mislead

Three things we have published, all from our own runs:

• A local token counter reported 96.2 million tokens saved. Paired trials on the same work found a higher provider bill. The counter measured bytes removed from payloads; the bill measured what the provider charged after retries, cache misses and re-injected rules. Same session, opposite sign.
• A benchmark printed a 26.5 percent reduction. Only nine task pairs held quality, the bootstrap interval crossed zero, and output tokens had risen 37 percent. The report marked itself unpublishable and we kept it out of every claim.
• A 998-task replay gained seven net passes while 125 individual outcomes flipped. The aggregate was flat. The distribution was not.

Each of these looked like an answer for a day. The method below is what caught them.

The method

1. Pair the tasks

Agent workloads vary enormously. Comparing a hard investigation on one arm with an easy lookup on the other mostly measures task selection. Run the same task, from the same starting state, on both arms. Three repetitions per arm per task is the minimum that gives you a spread.

Rotate the order. Cache state and provider load drift over an afternoon, and if arm A always goes first it always pays the cold start.

2. Use the provider's counters

Take the numbers from the response, or from the agent's own usage log if it records provider counts. Do not use a local tokenizer. Sum the three input-side counters:

input_tokens + cache_read_input_tokens + cache_creation_input_tokens

Record output tokens separately. Record the cache buckets separately too, because a change that shifts tokens from fresh input to cache reads is a real saving in dollars and a zero in raw token count. Weight by price when you have the rates.

3. Put a correctness gate on every run

A cheaper wrong answer is not a saving. Every run needs a check the task can fail: an exact expected value, a test suite, a structured answer compared to a key. Runs that fail the gate count as zero savings at held quality. They are not excluded, because excluding them is how a small result becomes a large claim.

4. Count the overhead

Whatever the optimization adds, count it from the first request. Rule files injected into the prompt. Recovery calls and the input they trigger. Tool definitions the optimization installs. If a tool saves tokens on a large payload and costs tokens on every small one, the session number is the only honest one.

5. Report the spread as well as the mean

Cluster by task and bootstrap the aggregate. If the interval crosses zero, say so. If one task went negative, show the row. Our pinned Claude Code benchmark reports 33.2 percent with an interval of 14.6 to 48.5 percent, and an HTML case at minus 9.9 percent. The spread is the finding.

Label what you measured

A number without its basis gets promoted the moment someone pastes it into a slide. We use four labels and do not let them convert into each other by wording.

A local estimate stays inferred no matter how plausible it looks. A paired benchmark stays counterfactual no matter how large. Nothing on your laptop mints verified.

Run it with caveman trial

The CLI wraps the procedure for a real coding session. It runs the task with and without the local proxy, keeps provider counts, and prints the paired comparison with the labels attached.

caveman trial -- claude
caveman trial report

It measures Caveman. The same procedure, by hand, measures anything else in the path, including a gateway's caching injection or a router's model choices. If a vendor's number cannot survive it, that is worth knowing before the invoice does.

Questions people ask

Why not trust the token counter in my agent?

Because it usually counts something other than what you pay for. Local counters estimate with an offline tokenizer, ignore cache pricing, and do not see retries. We published a case where a counter reported 96.2 million tokens saved while paired trials showed a higher provider bill.

How many runs do I need?

Three per arm per task is the floor. It gives you a spread and lets a single slow provider response show up as noise rather than a result. More is better on tasks with high variance.

Should I exclude runs that failed?

No. Count them as zero savings at held quality and leave them in the table. Excluding failures is the most common way a modest result becomes a large claim.

What does verified mean in Caveman?

Provider-causal evidence on live traffic, with both arms observed, signed into a ledger. Local tools never produce it. Benchmarks never produce it. It starts at zero dollars and stays there until a change is active on real requests.

Can I use this method on LiteLLM or another gateway?

Yes. Nothing in it is specific to Caveman. Pair the tasks, take provider counters, gate on correctness, count overhead, report the spread. Then label the result by how it was produced.

Sources

• How removing tokens raised the bill.
• A benchmark printed 26.5 percent. We refused the result.
• The average stayed flat. 125 tasks changed outcome.
• CaveBench Wrap benchmark, the method in full.
• How numbers are counted, the four labels.