---
title: "Prompt caching and compression: reduce agent cost without losing reuse"
description: "Look at your cache read tokens before you shorten anything. A 10,000 token prefix the provider is already reusing can cost less than the 2,000 token version that misses the cache on every request."
canonical: https://caveman.ac/guides/prompt-caching
last-updated: 2026-09-16
---

# Prompt caching and compression: reduce agent cost without losing reuse

Look at your cache read tokens before you shorten anything. A 10,000 token prefix the provider is already reusing can cost less than the 2,000 token version that misses the cache on every request.

Caching and compression pull against each other. Caching pays you for sending the same bytes again. Compression changes the bytes. Edit the front of a prompt and the discount resets, which is how a shorter prompt arrives with a larger bill.

## Read cache reads and cache writes before you change a prompt

Pull provider usage for one typical task and look at three fields: fresh input, cache reads, cache writes. That gives you the decision.

| What you see | What it means | What to do next |
| --- | --- | --- |
| Cache reads are most of your input | The prefix already works | Shrink the tail, meaning tool results and history, and leave the head alone |
| Cache reads are zero, the prompt barely changes | Something is breaking the prefix | Find it before you compress anything |
| Cache reads are zero, every request is genuinely different | There is nothing to reuse | Compression is your lever, not caching |

## Say which of the three caches you mean

| Cache | What gets reused | What to check |
| --- | --- | --- |
| Provider prompt cache | Work already done on a repeated prefix | Provider usage: reads, writes, retention, model rules |
| Gateway response cache | A previous answer, so no model call happens | Freshness, who is allowed to see it, how it is invalidated |
| Application retrieval cache | A tool or search result | Source freshness. The model still reads the result and still pays for it |

Two dashboards can both say "cache hit" about different work. Name the layer in the experiment write-up, every time.

## Everything before the first changing byte is what gets reused

| Prompt starts with | Cache result |
| --- | --- |
| `Session 8f21c, 2026-09-08 14:02:11`, then the system prompt | Nothing is reused. Every request is a new prefix |
| System prompt, tool definitions, then the timestamp beside the user turn | The stable head is reused, only the tail is fresh |

The same trap sits in code you did not write. Reordering tool definitions rewrites the prefix. A framework that renumbers or rewrites earlier turns rewrites the prefix. The visible question looks identical and the cache is gone.

Rules differ by provider, model and endpoint. [OpenAI's caching guide](https://developers.openai.com/api/docs/guides/prompt-caching) covers prefix matching and minimum lengths. [Claude's guide](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) covers cache controls and separate read and creation usage. [Gemini's guide](https://ai.google.dev/gemini-api/docs/caching) separates implicit from explicit caching. Check the model you actually run instead of copying a threshold from a different provider.

## Count the write, not just the read

The first request pays to create the cache and can cost more than an uncached one. The savings arrive on the requests after it, if there are any.

For Claude's documented contract, total input is fresh input plus cache reads plus cache creation. Other APIs report cached tokens as a subset of total input. Normalize to the source contract and do not add a subset twice. A framework that reports no cache field has not told you there was no cache. That stays unknown.

## Run four arms and keep cold apart from warm

| Arm | Context | Cache state |
| --- | --- | --- |
| A | Original | First use |
| B | Original | Repeated stable prefix |
| C | Candidate compression | First use |
| D | Candidate compression | Repeated candidate prefix |

Averaging A with B hides the whole effect. Compare A against C for the cold cost, B against D for the warm cost, then weight them by how often your workload is actually warm.

Hold the model, reasoning setting, tool definitions and task input fixed inside the comparison. Confirm each arm reached the path you intended by reading its usage, not by trusting the config. Repeat with the gap your application really has between runs: a tight local loop and a task resumed an hour later behave differently.

## The shorter request that costs twice as much

Take an illustrative price where uncached input costs ten times cached input. These are invented units for the arithmetic, not current provider prices:

```text
Original: 10,000 cached tokens × 1 unit = 10,000 units
Candidate: 2,000 uncached tokens × 10 units = 20,000 units
```

The candidate sends 80% fewer input tokens and pays twice as much. Over the next few turns it may build its own stable prefix and win, or it may keep editing the prefix and never recover. That is decided across the whole task, so price the whole task: output, cache writes, retries and recovery included. [Measure agent cost](/guides/measure-agent-cost) has the boundary.

## Check which mode Caveman is in before you read a number

The supported wrappers compress by default. A standalone listener with no configuration starts in record mode, where nothing is being compressed and a flat result is the correct result.

The local proxy also has provider-cache planning behavior on supported paths. Planner metadata and changes to what the model sees are separate operations, and mixing them in one trial makes the result unreadable. [Proxy configuration](https://docs.caveman.so/docs/proxy/configure) lists the controls. Change one.

Local Caveman numbers stay `inferred`. Do not attach a local ratio to a provider invoice.

## Decide from your reuse pattern, not from the ratio

One-off tasks are dominated by cold cost, so compression usually wins there. Long sessions live on a growing stable prefix, so protecting it usually wins. Repeated identical queries belong to the gateway response cache, where correctness and invalidation matter more than tokens.

Do not compress a prefix that is already being reused just because the local ratio looks good. Point compression at the expensive parts that are safe to shrink, verify the recovery path in [prompt compression](/guides/prompt-compression), and keep whichever arm finished more accepted work per dollar under your real reuse pattern.