---
title: "Agent observability: join a trace to money and to finished work"
description: "Stamp every model call with the ID of the task it belongs to, and stamp the task with whether it passed. Cost per completed task is that join, and your tracing tool cannot supply either fact on its ow"
canonical: https://caveman.ac/guides/agent-observability
last-updated: 2026-10-12
---

# Agent observability: join a trace to money and to finished work

Stamp every model call with the ID of the task it belongs to, and stamp the task with whether it passed. Cost per completed task is that join, and your tracing tool cannot supply either fact on its own.

The constraint: a trace records calls. Only your application knows when the job was actually done. Without both facts on the same row you can total tokens forever and still not say what one merged pull request cost.

## Your tracer counts calls. Your application knows when the work finished.

A provider request is smaller than the job someone asked for. One support answer is a retrieval, three model calls, a tool that failed, a retry and a final check. One coding task can run across three sessions before a patch is accepted.

So give the job a stable ID and give each attempt its own, then attach both to every call underneath. Do not group requests because they happened near each other in time; that merges two users into one invented task.

The verdict is the other half and it is the half teams skip. When the task ends, write down whether the result was accepted: the patch merged, the reply sent, the schema validated. A verdict written six hours later by a human still counts, as long as it lands on the task ID.

## The join answers questions the trace alone cannot

| What you want to know | A trace on its own | The same trace with a task ID and a verdict |
| --- | --- | --- |
| What one merged pull request cost | Tokens for forty requests, no way to tell which belong together | Every call in the attempt that produced the merge, summed |
| Whether retries are expensive | A retry count | The share of spend that went to attempts nobody accepted |
| Whether the cheaper model is fine | A lower cost per call | A lower cost per call, next to the acceptance rate it came with |

## Record these fields or the join will not close

| Field family | Why it has to be there |
| --- | --- |
| Task, attempt, trace and parent IDs | Rebuild one job without merging two separate attempts |
| Agent, environment and owner | Attribute the spend to someone who can act on it |
| Provider, model and deployment | Identify the inference path and which price applies |
| Start, finish, status and error class | Explain latency and failure |
| Input, output, cache and reasoning usage | Keep the provider's own accounting |
| Final artifact and acceptance | Separate work attempted from work completed |

Collect message content only when an investigation needs it and your data policy allows it. Timing and usage answer most cost questions without copying prompts or source code into another store. If you record both the raw tool output and the smaller model-facing view, label which stage each one is.

## Never turn a missing number into a zero

Providers do not all mean the same thing by input tokens. Cache reads are a subset of input in one API and a separate field in another. Reasoning tokens are sometimes already inside output. Read the contract before you sum anything.

Unknown is a legitimate value. A missing cache field is not a measured zero, a model that is not in your price catalog stays unpriced rather than free, and a request with partial usage does not get a cost that looks precise to four decimals.

This is a reporting shape, not a Caveman ingestion API:

```json
{
  "task_id": "support-case-42",
  "attempt_id": "attempt-2",
  "usage_source": "provider_response",
  "accounting_status": "incomplete",
  "cost_usd": null,
  "acceptance": "passed"
}
```

The task passed and the accounting is incomplete. Both are true at once, so store both instead of letting one overwrite the other.

## Reconcile one hour of traffic before anyone sees a dashboard

Take a small window and line up three sources: your application records, your gateway logs and the provider's own usage. Explain every difference. Then report coverage as a number, such as how many calls carried complete usage and how many carried a known task ID, because a partial ledger is not the bill.

## Catch the three numbers that lie by default

**The double span.** Two wrappers sit around the same provider client, so one call arrives twice. Spot it when your request count is roughly double the provider's. Fix: one recording layer per client, with a test that fails if a second one is registered.

**The retry below your callback.** The SDK retries inside the client, the provider bills three calls and your application logs one. Spot it when provider usage exceeds yours by about the retry rate. Fix: count at the transport, or treat the provider's usage as the source of truth and your own as a label.

**The average that hides the forty dollar task.** Mean cost per task looks stable while one runaway loop carries the month. Fix: report p95 alongside the mean, and always name the single most expensive completed task.

## Investigate the most expensive completed task, not the most expensive call

Start at the accepted result and walk backwards. Look for the largest repeated input, the branch that ran twice, the attempt that failed silently before the one that worked.

Then write a hypothesis a test can reject. "This search result is repeated on six turns; a smaller view should keep the required answer with fewer total input tokens" is specific enough to run. "Context is too big" is not.

Where the answer usually goes: a long tool result points at [compression](/guides/prompt-compression) or a narrower tool query, repeated validation failures point at a schema or tool contract, easy tasks on an expensive model point at [a routing test](/guides/model-routing), and duplicated research points at orchestration.

## Turn that task into an evaluation case before you change anything

Keep the permitted input, the environment, the expected result and the grader, with a reference back to the trace it came from. Use disposable fixtures for tools with side effects. Run baseline and candidate under the same acceptance rule, count the failed attempts too, and rerun before you believe the delta. [The evaluation guide](/guides/agent-evaluations) covers case design; [the measurement guide](/guides/measure-agent-cost) covers cost per accepted completion.

## Pick tools by whether they can carry your task ID

[Langfuse](/compare/langfuse), [LangSmith](/compare/langsmith), [Braintrust](/compare/braintrust), [Phoenix](/compare/arize-phoenix), [AgentOps](/compare/agentops) and [PromptLayer](/compare/promptlayer) differ in ingestion, evaluation, prompt handling and runtime dependencies. Judge each on whether your task ID, attempt ID and verdict survive the trip and can be queried together afterwards.

Caveman's public local tools reduce the parts of context that are safe to shrink while any of those stay in place, and their local reports are `inferred`.

## Caveman Platform is built for this join, and it is not open yet

The join described above is the product. One workspace over all your AI traffic, where the dollars resolve per member, per key, per workflow and per model. That spend is provider-reported usage priced against the public model catalog: a subtotal, not an invoice. A model the catalog does not price stays visibly unpriced instead of being guessed, averaged or counted as zero, which is the rule from the section above enforced in the ledger.

Savings climb a ladder and a number never skips a rung. `inferred` is modelled from your own traffic. `replayed` is the change re-run against the provider on real requests. `verified` means the provider's own billing shows the change, signed into the ledger, and `verified_savings = $0.00` until the change is active on live traffic. Receipts are Ed25519 signed and hash chained, and export is manual today.

Caveman Cloud is open for sign-up with a Free plan. Without it, the join is yours to build with the fields above, and your current tracer is fine as the collection layer. Either way, prove one complete path end to end before you widen collection. [What Caveman Cloud does](/products/platform) has the rest.

## You have this working when you can answer one question

"How much did the last ten merged pull requests cost, and how many attempts did each take?" If that takes one query and no spreadsheet, the join is real. If it takes an afternoon, you have a trace collection, not agent observability.