Cost and quality
How to measure AI agent cost per completed task
Count every attempt, including the failed ones, into one number. Set the task boundary, record the right fields per attempt, and divide by the tasks that passed.
- 01Define finished
- 02Fix the boundary
- 03Record every attempt
- 04Divide by passes
Cost per finished task is everything you spent on a job divided by the number of times the job actually got done. Not cost per call. Not dollars per million tokens.
The constraint behind that: agents do not finish on the first try. They fail, get re-prompted, repair a tool call, read back text they compressed, and a human nudges them twice. All of that is the price of one finished task. Leave any of it out and every optimization looks better than it is.
Decide what finished means before you run anything
Pick an outcome you can check without asking the agent how it went.
| Task | Too loose | Checkable |
|---|---|---|
| Code change | The agent produced a patch | The patch applies, the test suite passes, and no test file was edited |
| Log triage | The agent explained the outage | The answer names incident INC-4471 and the 04:12 restart |
| Extraction | Valid JSON came back | Every field equals the saved expected value |
Write the rejections down too. A patch that passes by deleting the failing assertion is a failure. A summary that drops the only warning line is a failure. Use the same grader for both arms and do not edit it halfway through.
The boundary starts at the first prompt and ends at the accepted artifact
This is the part that decides whether your number means anything.
| Inside the total | Outside the total |
|---|---|
| Every model call in every attempt, including attempts that produced nothing | Calls belonging to a different task in the same session |
| Tool repair turns and schema retries | Your own reading time |
| Recovery calls that read back compressed text | One-off exploration you would not run in production |
| Paid judge or verifier calls that run in production | Benchmark runs that are not the task itself |
| The second session after a human re-prompted the same task | Time spent waiting on a rate limit, which is latency |
If you cannot tell where one task ended and the next began, you do not have a boundary yet. Run one task per fresh session until you can, then widen.
Run baseline and candidate in separate copies of the same checkout
Record the revision, the starting files, the prompt, the model ID, the reasoning setting, the agent version, and the tool configuration. Then change exactly one thing.
# Baseline, fresh session
claude
# Candidate, another fresh session
caveman claudeUse the same task text in both. If an output-shortening skill is installed, enable it in both arms or declare it as the candidate change. Otherwise you are testing two changes and cannot tell which one moved the number.
Never let the second arm run in the first arm's workspace. It inherits the patch, the generated files and a warm test cache, then looks fast and cheap for reasons that have nothing to do with your change.
Record these fields per attempt, not per task
| Field | Why it is there |
|---|---|
| Task ID and arm | Pairs the two runs |
| Attempt number | Failed attempts need their own row |
| Model, provider, settings | Says what you actually bought |
| Fresh input, cached input, cache writes, output | Four different prices |
| Retries and recovery calls | Keeps the overhead inside the result |
| Elapsed time | Catches a cheap path nobody can wait for |
| Pass or fail | Stops a cheap failure from looking like a win |
| Cost source | A provider charge is not an estimate |
Some APIs report cached tokens inside the input total, others report them beside it. Read that provider's contract before you add anything up, or you will count the same tokens twice. Missing usage stays unknown, never zero.
Divide by the tasks that passed
attempt cost = sum of all model-call charges in that attempt
cost per completed task = total spend across all attempts / accepted completionsIf nothing passed, the result is undefined. Not zero. Report the spend and the failure rate instead.
Keep failed attempts in the numerator when you compare arms. Drop them and you have built a metric that rewards an agent for giving up early.
Three ways this number comes out too low
The retry loop that looks cheap. Cost per call falls, cost per task climbs, because the cheap call now runs four times. Spot it by counting attempts per accepted task in each arm. Compare that ratio before you compare dollars.
The task a human finished. The agent got most of the way and someone wrote the last function, and it is filed as a success. Spot it by checking the accepted artifact against what the agent actually produced. Score human-repaired tasks as failures, or give them their own row.
The subscription that did not move. Fewer tokens in a session does not lower a fixed monthly fee or buy back a quota. Report the usage change on its own and do not convert it into cash.
Check the number before you trust it
Alternate the run order so one arm does not always get the quiet hour. Cover the payloads you really run: two line answers, a 40,000 token log, structured data, code changes that need every detail. Do not pick only the inputs a compressor is good at.
Read the per-task rows before you average them. A candidate that wins on long logs and loses on short requests earns a narrow rollout, not a headline. Report the sample count and the spread.
Local Caveman reports stay inferred. A paired benchmark is evidence about that benchmark. Provider-confirmed savings on live traffic need provider evidence, and that difference has to survive into your dashboard and into anything you send a customer.
Read how Caveman counts, local proxy reporting, and the CaveBench methodology. Once you have the number, pick which change to test first and build the grader.
Try it on work you know
Keep your agent. Add Caveman.
Install the local tools, run a task you can check, and compare the result.
Open the quickstart Explore comparisons ↗