Caveman vs Braintrust: evals plus token compression

Braintrust is built around one question: did this change make the product better or worse? Caveman Middleware is a change you will want to ask that about. It shortens large tool results before each model call, which cuts input tokens, and that only counts if the answers hold up. So the pairing is natural: keep Braintrust for logging and evals, add the Caveman adapter, and let a Braintrust experiment decide whether it stays.

What is Braintrust?

Braintrust is an AI observability and evaluation platform. Teams including Cloudflare, Vercel, Notion, Replit and Retool use it to log production traces, score outputs and compare prompts and models against versioned datasets.

What it does well:

• Evals as code. Eval() in TypeScript or Python takes data, a task and scorers, and every run becomes an experiment you can compare with the last one.
• Scorers. LLM, code or human scoring. The open-source autoevals library (MIT, about 1k GitHub stars) ships ready-made scorers such as Factuality.
• Tracing. Wrappers and auto-instrumentation for the OpenAI SDK, the Vercel AI SDK, LangChain and others log prompts, responses, tool calls and token metrics.
• Loop and Discover. Loop is an agent that generates prompts, scorers and datasets toward a goal you set. Discover classifies production traces by topic and surfaces recurring issues.
• Brainstore, Braintrust's own trace database, and an AI gateway at gateway.braintrust.dev that routes provider calls through one endpoint and can cache responses.

Starter is free with 1 GB of processed data, 10k scores and 14 days of retention. Pro is $249 a month with 5 GB, 50k scores and 30 days. Both include unlimited users. Enterprise adds on-prem or hosted deployment. The JavaScript SDK is Apache 2.0; the platform itself is a hosted product.

What does Caveman add to Braintrust?

Caveman makes the model calls Braintrust traces carry fewer input tokens. Caveman Middleware wraps the provider client or framework you already use. For each call it replaces eligible tool results (a repo file read, a big API response, command output) with a shorter copy in the outbound request, and adds a caveman_retrieve tool so the model can fetch the exact original if the short copy is not enough. Your app keeps its original history. User messages, system prompts and errors are left alone, and inference goes to your provider as before.

For scale, the Caveman docs show a 400-line worker log going from 8,399 tokens to roughly 180 in the copy the model received. That is an inferred o200k_base count on one synthetic log, not a billing result. The adapters are alpha, in TypeScript (@caveman-ai/middleware) and Python (caveman-middleware), for the OpenAI, Anthropic and Google SDKs, the Vercel AI SDK, LangChain, Mastra, Pydantic AI and more.

Which copy of the request does a Braintrust trace show?

That depends on which Braintrust integration you use, because the two tools hook in at different layers.

Braintrust's OpenAI wrapper emits a span per chat.completions.create call, and Caveman's OpenAI adapter rewrites the HTTP request underneath that method. So the span keeps the messages you passed, while prompt_tokens comes from the response for what was actually sent. The AI SDK works the same way: version 7 passes telemetry the call's messages before the wrapped model applies Caveman's transform. With LangChain, Caveman's agent middleware hands the chat model the shortened copy, so the LLM span shows it, headed [caveman: shortened; exact original via caveman_retrieve …].

Two rules for the raw OpenAI path. Use the tool-loop form (withCavemanOpenAITools, or with_caveman_openai_tools in Python), which registers caveman_retrieve; the plain wrapper has no recovery tool, so it sends every call unchanged. And Caveman returns a new client, so apply wrapOpenAI to that one.

Braintrust vs Caveman: feature comparison

How do you check Caveman did not hurt quality in Braintrust?

Run one Eval() twice over the same data, with the Caveman client in record mode and then compress, and compare the two experiments. record sends every byte unchanged through the same code path, so any score difference comes from compression.

The dataset decides whether the test means anything. Use cases where the answer depends on a detail inside a large tool result: a failing test name in a long CI log, one price in a 20k-token catalog response. Score with Factuality or your own scorer, then compare three numbers: scores per example, total tokens per task, and how many caveman_retrieve tool spans appear. Recovery calls are real round trips, so they count toward the total. The switch guide has a working Eval.

When should you use Braintrust alone?

Use Braintrust alone when your model calls carry little tool output. With short prompts and small tool results there is nothing for Caveman to shorten, and its reports will say skipped with no_candidate.

Stay with Braintrust alone too while you are still finding out where tokens go. Its logs may show the cost sits in retries, long system prompts or a scorer you run on every request. Caveman Middleware is alpha, so pin versions and pass an Eval before production. Caveman has no scorers, trace store or dataset tools, and Caveman Platform, the hosted workspace, is not generally available.

FAQ

Is Caveman an alternative to Braintrust? No. Braintrust measures quality and cost; Caveman changes the request to lower tokens. You use Braintrust to judge Caveman.

Does Caveman work behind the Braintrust gateway? There is no gateway-level adapter. Wrap your provider client or framework with Caveman and keep its base URL pointed at https://gateway.braintrust.dev/v1.

Why do my Braintrust inputs still show the full tool output? With wrapOpenAI or AI SDK telemetry, the span records the messages your code passed, which are the originals. The token metrics reflect the shorter request. Log Caveman's onReport status as metadata if you want the decision on the span.

Does Caveman see my Braintrust data? No. Caveman's adapter only talks to the runtime you run, on localhost by default. Braintrust receives what its integration already sends.

Sources

• Braintrust homepage
• Braintrust pricing
• Braintrust OpenAI integration
• Braintrust Vercel AI SDK integration
• autoevals on GitHub
• Caveman middleware docs and supported frameworks

Next: add Caveman to a Braintrust setup, agent evaluations, prompt compression, and Caveman Middleware.