Headroom vs Caveman for prompt compression, measured

Headroom and Caveman solve the same problem from the same position. Both sit between a coding agent and its provider, both shrink tool output before it re-enters the context window, and both let the model fetch the original back. LiteLLM's cost guide recommends Headroom. We build Caveman. So the only useful thing we can offer is a comparison you can rerun.

Everything below comes from one pinned benchmark, with the method, fixtures, and raw counts published. It is controlled evidence, not production traffic, and not a savings claim about your bill.

The result

Caveman won 15 of 18 paired runs against direct. Headroom's three YAML runs failed the exact-answer gate. Those runs stay in its row rather than being dropped, which is why its reduction is small and its interval touches zero. Caveman's HTML case cost more than direct and stays in ours.

Per case

Each case ran three times per arm. Input is the sum of fresh, cache-read and cache-write tokens as Claude Code's own usage counters reported them.

The HTML row is the one to read first. No compressor matched that payload, so Caveman paid its own overhead, the skill rules and the recovery tooling, and won nothing back. We publish it because a benchmark that hides its red rows is not a benchmark.

How they differ

Both tools take the same seat. The differences are in what happens when compression is uncertain and where the originals live.

Headroom, as described in LiteLLM's guide and its own documentation, runs as a sidecar or guardrail that rewrites tool outputs and payloads, and exposes a retrieve_headroom tool the model can call to get content back. It reports 60 to 95 percent savings on the compressible portions of a payload. Its LiteLLM integration registers as a pre-call guardrail.

Caveman runs as a loopback proxy on the developer's machine. One compressor per content shape: JSON, logs, code, diffs, search results, tables, prose. Before any lossy view is shown to the model, the original bytes are written to a local SQLite store keyed by content hash, and the model receives a ccr_ handle. If the store write fails, the parse fails, or the result is not smaller, the original goes upstream unchanged. Recovery with an empty query is byte exact. The wrap also installs the output-shortening skill, which is included in the arm above and in its overhead.

We have not measured Headroom's behaviour on parse failure or its recovery fidelity beyond the exact-answer checks in this run, so we do not describe them.

What "compressible portions" hides

A vendor figure like 60 to 95 percent, or our own 70 to 90 percent target for JSON, describes what a compressor does to a payload it recognises. A session is a mix of payloads it recognises and ones it does not, plus the fixed overhead of running the tool at all. The benchmark measures the mix. That is why the whole-session figure, 33.2 percent, is far below any per-type target, and why one case went negative.

When you compare compression tools, ask for the session-level number, the quality gate, and the red rows. If any of the three is missing, the number is a ceiling, not a result.

Rerun it

Fixtures, oracle, and the run harness are in the public repository. Claude Code was pinned at 2.1.223, the model at Sonnet 5, and each arm called each fixture exactly once per run. Recovery calls and their follow-up input stayed counted. Skill prompt overhead was counted from the first request.

If your rerun disagrees with ours, open an issue with the raw counts. We will add the result to the write-up.

Questions people ask

Is Headroom or Caveman better for Claude Code?

On the six tool-heavy tasks in our pinned benchmark, Caveman reduced input tokens by 33.2 percent with 18 of 18 exact answers, and Headroom by 6.7 percent with 15 of 18. That is one benchmark, three runs per case, and both tools will behave differently on your payload mix. Run both on your own tasks before deciding.

Why did Headroom score so low?

Three of its 18 runs, all on the YAML case, failed the exact-answer check. Failed runs count as zero savings at held quality rather than being excluded, which is the rule we apply to our own rows too. Its passing runs saved more.

Does Caveman ever make things worse?

Yes. On the HTML dashboard case it cost 9.9 percent more than direct Claude Code, because no compressor applied and the fixed overhead stayed. That row is in the published total.

Can I run either tool behind LiteLLM?

Yes. Headroom registers as a LiteLLM guardrail. Caveman sits in front of LiteLLM as a local proxy, or behind it as a deployment api_base. The LiteLLM guide covers the Caveman side.

Are these numbers a savings claim?

No. The basis is benchmark_counterfactual: paired runs under a pinned method. It is not production traffic, not a provider invoice, and not what Caveman labels verified.

Sources

• CaveBench Wrap benchmark, method, intervals, per-run counts.
• LiteLLM, 5 ways to cut Claude Code costs, Headroom section.
• Recoverable compression, how the handle and store work.