---
title: "Caveman vs Braintrust: evals plus token compression"
description: "Braintrust is built around one question: did this change make the product better or worse? Caveman Middleware is a change you will want to ask that about. It shortens large tool results before each mo"
canonical: https://caveman.ac/compare/braintrust
last-updated: 2026-10-04
---

# Caveman vs Braintrust: evals plus token compression

Braintrust is built around one question: did this change make the product better or worse? Caveman Middleware is a change you will want to ask that about. It shortens large tool results before each model call, which cuts input tokens, and that only counts if the answers hold up. So the pairing is natural: keep Braintrust for logging and evals, add the Caveman adapter, and let a Braintrust experiment decide whether it stays.

## What is Braintrust?

Braintrust is an AI observability and evaluation platform. Teams including Cloudflare, Vercel, Notion, Replit and Retool use it to log production traces, score outputs and compare prompts and models against versioned datasets.

What it does well:

- **Evals as code.** `Eval()` in TypeScript or Python takes data, a task and scorers, and every run becomes an experiment you can compare with the last one.
- **Scorers.** LLM, code or human scoring. The open-source `autoevals` library (MIT, about 1k GitHub stars) ships ready-made scorers such as `Factuality`.
- **Tracing.** Wrappers and auto-instrumentation for the OpenAI SDK, the Vercel AI SDK, LangChain and others log prompts, responses, tool calls and token metrics.
- **Loop and Discover.** Loop is an agent that generates prompts, scorers and datasets toward a goal you set. Discover classifies production traces by topic and surfaces recurring issues.
- **Brainstore**, Braintrust's own trace database, and an **AI gateway** at `gateway.braintrust.dev` that routes provider calls through one endpoint and can cache responses.

Starter is free with 1 GB of processed data, 10k scores and 14 days of retention. Pro is $249 a month with 5 GB, 50k scores and 30 days. Both include unlimited users. Enterprise adds on-prem or hosted deployment. The JavaScript SDK is Apache 2.0; the platform itself is a hosted product.

## What does Caveman add to Braintrust?

Caveman makes the model calls Braintrust traces carry fewer input tokens. Caveman Middleware wraps the provider client or framework you already use. For each call it replaces eligible tool results (a repo file read, a big API response, command output) with a shorter copy in the outbound request, and adds a `caveman_retrieve` tool so the model can fetch the exact original if the short copy is not enough. Your app keeps its original history. User messages, system prompts and errors are left alone, and inference goes to your provider as before.

For scale, the Caveman docs show a 400-line worker log going from 8,399 tokens to roughly 180 in the copy the model received. That is an inferred `o200k_base` count on one synthetic log, not a billing result. The adapters are alpha, in TypeScript (`@caveman-ai/middleware`) and Python (`caveman-middleware`), for the OpenAI, Anthropic and Google SDKs, the Vercel AI SDK, LangChain, Mastra, Pydantic AI and more.

## Which copy of the request does a Braintrust trace show?

That depends on which Braintrust integration you use, because the two tools hook in at different layers.

| Integration | Input on the LLM span | Token metrics on the span |
| --- | --- | --- |
| `wrapOpenAI` / `wrap_openai` + Caveman OpenAI tool-loop adapter | Your original messages | Provider usage for the shortened request |
| AI SDK 7 `braintrustAISDKTelemetry` + Caveman `withCaveman` | Your original messages | Provider usage for the shortened request |
| `BraintrustCallbackHandler` + Caveman LangChain adapter | The shortened copy the model read | Provider usage for that copy |

Braintrust's OpenAI wrapper emits a span per `chat.completions.create` call, and Caveman's OpenAI adapter rewrites the HTTP request underneath that method. So the span keeps the messages you passed, while `prompt_tokens` comes from the response for what was actually sent. The AI SDK works the same way: version 7 passes telemetry the call's messages before the wrapped model applies Caveman's transform. With LangChain, Caveman's agent middleware hands the chat model the shortened copy, so the LLM span shows it, headed `[caveman: shortened; exact original via caveman_retrieve …]`.

Two rules for the raw OpenAI path. Use the tool-loop form (`withCavemanOpenAITools`, or `with_caveman_openai_tools` in Python), which registers `caveman_retrieve`; the plain wrapper has no recovery tool, so it sends every call unchanged. And Caveman returns a new client, so apply `wrapOpenAI` to that one.

## Braintrust vs Caveman: feature comparison

| | Braintrust | Caveman Middleware |
| --- | --- | --- |
| Main job | Evaluate and observe AI apps | Fewer input tokens from tool output |
| Changes the request | No | Yes, an outbound copy per call |
| Experiments and diffs | Yes | No, use Braintrust's |
| Scorers | LLM, code, human; `autoevals` | None |
| Production tracing | Yes | No |
| Gateway | Braintrust gateway | No; wraps your client instead |
| Original recovery | Not applicable | `caveman_retrieve` per conversation |
| Self-hosting | On-prem on Enterprise | Runtime beside your app |
| License | Apache 2.0 SDK, hosted platform | MIT client, BSL runtime |
| Status | Generally available | Alpha |

## How do you check Caveman did not hurt quality in Braintrust?

Run one `Eval()` twice over the same data, with the Caveman client in `record` mode and then `compress`, and compare the two experiments. `record` sends every byte unchanged through the same code path, so any score difference comes from compression.

The dataset decides whether the test means anything. Use cases where the answer depends on a detail inside a large tool result: a failing test name in a long CI log, one price in a 20k-token catalog response. Score with `Factuality` or your own scorer, then compare three numbers: scores per example, total tokens per task, and how many `caveman_retrieve` tool spans appear. Recovery calls are real round trips, so they count toward the total. [The switch guide](/switch/braintrust) has a working Eval.

## When should you use Braintrust alone?

Use Braintrust alone when your model calls carry little tool output. With short prompts and small tool results there is nothing for Caveman to shorten, and its reports will say `skipped` with `no_candidate`.

Stay with Braintrust alone too while you are still finding out where tokens go. Its logs may show the cost sits in retries, long system prompts or a scorer you run on every request. Caveman Middleware is alpha, so pin versions and pass an Eval before production. Caveman has no scorers, trace store or dataset tools, and Caveman Platform, the hosted workspace, is not generally available.

## FAQ

**Is Caveman an alternative to Braintrust?**
No. Braintrust measures quality and cost; Caveman changes the request to lower tokens. You use Braintrust to judge Caveman.

**Does Caveman work behind the Braintrust gateway?**
There is no gateway-level adapter. Wrap your provider client or framework with Caveman and keep its base URL pointed at `https://gateway.braintrust.dev/v1`.

**Why do my Braintrust inputs still show the full tool output?**
With `wrapOpenAI` or AI SDK telemetry, the span records the messages your code passed, which are the originals. The token metrics reflect the shorter request. Log Caveman's `onReport` status as metadata if you want the decision on the span.

**Does Caveman see my Braintrust data?**
No. Caveman's adapter only talks to the runtime you run, on localhost by default. Braintrust receives what its integration already sends.

## Sources

- [Braintrust homepage](https://www.braintrust.dev)
- [Braintrust pricing](https://www.braintrust.dev/pricing)
- [Braintrust OpenAI integration](https://www.braintrust.dev/docs/integrations/ai-providers/openai)
- [Braintrust Vercel AI SDK integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/vercel)
- [autoevals on GitHub](https://github.com/braintrustdata/autoevals)
- [Caveman middleware docs](https://docs.caveman.so/docs/sdk/middleware) and [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks)

Next: [add Caveman to a Braintrust setup](/switch/braintrust), [agent evaluations](/guides/agent-evaluations), [prompt compression](/guides/prompt-compression), and [Caveman Middleware](/products/caveman-middleware).