Cost and quality
How to reduce LLM costs in an agent that already works
Pick the one change worth testing first from your own usage data. A symptom to experiment map for caching, compression, tool output, model choice, and retries.
- 01Group by task
- 02Read the symptom
- 03Change one thing
- 04Keep what passed
Make one change at a time, and pick it from your own usage data rather than from a list of best practices.
The constraint is arithmetic. Switch models, shorten the system prompt and turn on caching in the same week, watch the bill fall, and you have learned nothing you can act on next month. You cannot roll back the change that hurt quality, because you cannot tell which one that was.
Start from the cost of one finished task, not the token price
A cheap model that retries four times costs more than an expensive model that gets it right once. A 30,000 token prompt is cheap when the provider is already reusing 28,000 of it. Both facts disappear if you look at price per million tokens.
Group your spend by something a person recognizes: a support case closed, a pull request merged, one extraction accepted. Put failed attempts in the same group as the task they failed at. Measure agent cost has the boundary rules and the field list.
No task IDs in your logs? Take twenty runs whose edges you can reconstruct by hand. A small honest sample beats a dashboard that quietly mixes three workloads.
Find your symptom in this table, then run only that row
| What the usage data shows | The one thing to try first | Guide |
|---|---|---|
| Cache reads near zero on a prompt that barely changes | Stabilize the prefix. Do not shorten anything yet | Prompt caching |
The same 40,000 token log or grep result resent every turn | Recoverable compression, or a smaller tool response | Prompt compression |
| Thousands of tokens of tool definitions before any work starts | Remove the MCP servers this agent never calls | MCP token overhead |
| Trivial tasks always go to the strongest model | Fix one cheaper model on one task family, no router yet | Model routing |
| Two or three tool repair turns per task | Fix the tool contract and its error messages | Below on this page |
| Long final answers nobody reads | Ask for a shorter output format, then check nothing was lost | Coding agent setup |
| You cannot answer any of the rows above | You have a visibility problem, not a cost problem | Agent observability |
Pick the row with the largest reliable signal. One row. The others are still there next month, and they will be easier to read once this one is settled.
Make the tool return the answer, not the corpus
Most agent context is tool output the model never asked for, and shrinking it later is more work than not sending it.
| Tool | Returns today | Should return |
|---|---|---|
search_docs | 40 whole documents | The five best matches, each with the matching paragraph and an ID for fetching the rest |
read_logs | 2,000 lines, mostly health checks | Every error line, plus a count of what was skipped |
query_table | The whole table | The first page, labeled as the first page |
Do not let a filter hide the one line that mattered. If the next question might need what you dropped, leave a way to fetch it: pagination, a follow-up query, or Caveman's recoverable compression for the parts that are safe to shrink.
Fix the retries before you make each retry cheaper
Read ten failed runs end to end. Count the turns that produced no progress: the same tool called with the same broken arguments, a schema error with no hint about what was wrong, a loop that ends when the context window fills.
Give each tool one clear error message that contains what the model needs for a single correct retry. Make side-effecting calls safe to repeat where your application needs that. Then rerun the failure path against disposable fixtures and confirm the agent either recovers or stops.
A shorter prompt makes a broken loop cheaper per turn. It does not make the loop stop.
Try the local Caveman path on a task you can check
For a supported coding agent:
npm install -g @caveman-ai/cli
caveman setup --install
caveman doctor claude
caveman claudeRun the same task through your normal launcher in a separate fresh workspace, with the prompt, model, reasoning setting and tools held fixed. Check the patch and run the tests before you look at any token count.
Read the local numbers with:
caveman stats --jsonThose numbers are inferred. They describe what the local tools did to your context, and they are not a provider invoice. Do not subtract them from a subscription fee. If all you want is shorter replies, that is the skill, and it is a different change: see the coding agent guide.
Three patterns that waste a quarter
The five changes shipped together. The bill moves, quality moves, and nobody can attribute either. Spot it in your change log: more than one entry between two measurements. Revert to one change and keep the previous path callable.
The retry loop that looks cheap. Cost per call drops, cost per finished task rises. Spot it by counting attempts per accepted task, not dollars.
The shorter prompt that costs more. You cut the system prompt and lost the cached prefix, so every request now pays full price for fresh input. Spot it in cache read tokens before and after. Prompt caching has the arithmetic.
Ship it to the task family that passed
Run the same acceptance test on both arms, built the way agent evaluations describes, and report the sample count, the spread and the cases the candidate lost. Total spend includes every failed attempt.
Roll out to the family that passed and leave the direct path callable. Repeat the comparison when the model, the agent version, the tool output format or provider behavior changes. If your first row was the gateway itself, the comparison index maps tools to the layer you would be changing.
Try it on work you know
Keep your agent. Add Caveman.
Install the local tools, run a task you can check, and compare the result.
Open the quickstart Explore comparisons ↗