Add Caveman to Braintrust: run it as an Eval

Nothing moves out of Braintrust. You wrap your AI SDK call with Caveman's adapter, keep Braintrust's telemetry registered, and run the same Eval() twice: once as a baseline and once with compression on. If the scores hold and the tokens drop, keep it. This guide uses TypeScript and the Vercel AI SDK 7. For how the two products relate, see Caveman vs Braintrust.

1. Install and start the runtime

The Caveman adapter accepts ai >=7.0.94 <8. The CLI that installs the local runtime needs Node.js 22.13 or newer.

npm install ai @ai-sdk/openai braintrust autoevals @caveman-ai/middleware @caveman-ai/sdk
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

From another shell:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'

mode must be compress. Plain caveman start runs in record and changes nothing.

2. Wrap the call and write the Eval

withCaveman takes the options you pass to generateText or streamText and returns a new object: your model wrapped with the AI SDK's wrapLanguageModel, plus a caveman_retrieve tool. Braintrust's AI SDK 7 telemetry is registered once, as in Braintrust's docs.

import * as ai from "ai";
import { openai } from "@ai-sdk/openai";
import { Eval, braintrustAISDKTelemetry } from "braintrust";
import { Factuality } from "autoevals";
import { MiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";
import { cases, tools } from "./fixtures"; // your cases and your existing AI SDK tools

ai.registerTelemetry(braintrustAISDKTelemetry());

const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "record") as "off" | "record" | "compress"; // "off" = rollback
const runtime = new MiddlewareRuntime({
  endpoint: "http://127.0.0.1:8787",
  mode,
  onReport: report => console.log(report.status, report.reason),
});

Eval("heavy-tool-output", {
  experimentName: `caveman-${mode}`,
  data: cases, // [{ input: "Which test failed in run 4812?", expected: "test_refund_rounding" }, ...]
  task: async (input: string) => {
    if (mode !== "off") await runtime.ready(); // ready() throws in off mode
    // One scope per conversation. Recovery handles only resolve under the same four values.
    const scope = { namespace: "eval", session_id: crypto.randomUUID(), branch_id: "main", cache_epoch: "0" };
    const result = await ai.generateText({
      ...withCaveman({ model: openai(process.env.OPENAI_MODEL!), tools }, { runtime, scope }),
      prompt: input,
      stopWhen: ai.stepCountIs(8),
    });
    return result.text;
  },
  scores: [Factuality],
}).finally(() => runtime.close());

Run it twice with Braintrust's eval CLI:

CAVEMAN_CLIENT_MODE=record bt eval trial.eval.ts
CAVEMAN_CLIENT_MODE=compress bt eval trial.eval.ts

record sends every byte unchanged through the same wrapper, which makes it a fair baseline. Factuality calls an LLM to grade, so it needs a provider key; swap in an exact-match or code scorer if your answers are structured.

Choose cases where the answer sits inside a big tool result: a test name in a 3,000-line CI log, one field in a 20k-token JSON response, a figure deep in an extracted PDF. Those are the cases a shortened view could get wrong, and they are the ones worth 30 minutes of eval time.

3. Check it worked

Open the caveman-compress experiment in Braintrust and compare it with caveman-record. Look at:

• Scores per example. Read every example whose score dropped. An average that holds can hide one important regression.
• Token metrics. Compare the task's total prompt tokens across all LLM spans, not only the first call. A caveman_retrieve call is one more round trip.
• Recovery tool spans. Count caveman_retrieve spans. A few mean the model is using recovery when it needs detail. Recoveries on most examples mean the short view is missing what your tasks need.
• The console. The onReport lines should read applied or reused on heavy calls. skipped no_candidate means that call had no eligible tool result.

One thing that looks wrong but is correct: the LLM span's input shows your full tool output. AI SDK 7 hands telemetry the call's messages before the wrapped model applies Caveman's transform, so Braintrust records your originals, while the token metrics come from the provider's response for the shorter request. If you want the decision visible on the trace, store the onReport status as span metadata.

Using wrapOpenAI instead

With a raw OpenAI client you own the tool loop, so use withCavemanOpenAITools. It adds caveman_retrieve to your tool definitions and returns a dispatch table that runs it. Plain withCavemanOpenAI on chat.completions.create has no recovery executor, and the runtime will not shorten text it cannot give back, so those calls go out unchanged. Put Braintrust's wrapper on the outside, because Caveman returns a new client built with withOptions:

import OpenAI from "openai";
import { wrapOpenAI } from "braintrust";
import { withCavemanOpenAITools } from "@caveman-ai/middleware/openai";

const loop = withCavemanOpenAITools(new OpenAI({ fetch: globalThis.fetch }), {
  runtime, scope, fetch: globalThis.fetch,
  protocol: "openai-chat", tools: chatTools, functions, // functions: { [tool name]: (args) => result }
});
const client = wrapOpenAI(loop.client);
const response = await client.chat.completions.create({ model, messages, tools: loop.tools });
// Run every tool call through loop.functions[name](JSON.parse(args)), caveman_retrieve included.

Pass the same fetch you built the client with, and send loop.tools on every request; a call without the recovery tool goes out uncompressed. The same trace rule applies: inputs show your originals, token metrics show the shorter request. To route through the Braintrust gateway, set baseURL: "https://gateway.braintrust.dev/v1" and your Braintrust key on the OpenAI client as usual; Caveman wraps the client and leaves the base URL alone.

Roll back

• Fastest: run with CAVEMAN_CLIENT_MODE=off, which creates the runtime with mode: "off". Calls pass through untouched.
• Code: replace ...withCaveman({ model, tools }, { runtime, scope }) with model, tools. withCaveman returns a new object and mutates nothing, so the call is back to exactly what it was.
• Runtime: restart it without CAVEMAN_MODE=compress and it returns to record.

Braintrust needs no change. Your experiments stay, including the two from this trial, which are a useful record of why you kept or dropped Caveman.

More: Caveman middleware docs, Vercel AI SDK adapter, supported frameworks, Braintrust eval quickstart, and the agent evaluations guide.