---
title: "Add Caveman to Braintrust: run it as an Eval"
description: "Nothing moves out of Braintrust. You wrap your AI SDK call with Caveman's adapter, keep Braintrust's telemetry registered, and run the same Eval() twice: once as a baseline and once with compression o"
canonical: https://caveman.ac/switch/braintrust
last-updated: 2026-10-04
---

# Add Caveman to Braintrust: run it as an Eval

Nothing moves out of Braintrust. You wrap your AI SDK call with Caveman's adapter, keep Braintrust's telemetry registered, and run the same `Eval()` twice: once as a baseline and once with compression on. If the scores hold and the tokens drop, keep it. This guide uses TypeScript and the Vercel AI SDK 7. For how the two products relate, see [Caveman vs Braintrust](/compare/braintrust).

## 1. Install and start the runtime

The Caveman adapter accepts `ai >=7.0.94 <8`. The CLI that installs the local runtime needs Node.js 22.13 or newer.

```bash
npm install ai @ai-sdk/openai braintrust autoevals @caveman-ai/middleware @caveman-ai/sdk
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

From another shell:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'
```

`mode` must be `compress`. Plain `caveman start` runs in `record` and changes nothing.

## 2. Wrap the call and write the Eval

`withCaveman` takes the options you pass to `generateText` or `streamText` and returns a new object: your model wrapped with the AI SDK's `wrapLanguageModel`, plus a `caveman_retrieve` tool. Braintrust's AI SDK 7 telemetry is registered once, as in Braintrust's docs.

```ts
import * as ai from "ai";
import { openai } from "@ai-sdk/openai";
import { Eval, braintrustAISDKTelemetry } from "braintrust";
import { Factuality } from "autoevals";
import { MiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";
import { cases, tools } from "./fixtures"; // your cases and your existing AI SDK tools

ai.registerTelemetry(braintrustAISDKTelemetry());

const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "record") as "off" | "record" | "compress"; // "off" = rollback
const runtime = new MiddlewareRuntime({
  endpoint: "http://127.0.0.1:8787",
  mode,
  onReport: report => console.log(report.status, report.reason),
});

Eval("heavy-tool-output", {
  experimentName: `caveman-${mode}`,
  data: cases, // [{ input: "Which test failed in run 4812?", expected: "test_refund_rounding" }, ...]
  task: async (input: string) => {
    if (mode !== "off") await runtime.ready(); // ready() throws in off mode
    // One scope per conversation. Recovery handles only resolve under the same four values.
    const scope = { namespace: "eval", session_id: crypto.randomUUID(), branch_id: "main", cache_epoch: "0" };
    const result = await ai.generateText({
      ...withCaveman({ model: openai(process.env.OPENAI_MODEL!), tools }, { runtime, scope }),
      prompt: input,
      stopWhen: ai.stepCountIs(8),
    });
    return result.text;
  },
  scores: [Factuality],
}).finally(() => runtime.close());
```

Run it twice with Braintrust's eval CLI:

```bash
CAVEMAN_CLIENT_MODE=record bt eval trial.eval.ts
CAVEMAN_CLIENT_MODE=compress bt eval trial.eval.ts
```

`record` sends every byte unchanged through the same wrapper, which makes it a fair baseline. `Factuality` calls an LLM to grade, so it needs a provider key; swap in an exact-match or code scorer if your answers are structured.

Choose cases where the answer sits inside a big tool result: a test name in a 3,000-line CI log, one field in a 20k-token JSON response, a figure deep in an extracted PDF. Those are the cases a shortened view could get wrong, and they are the ones worth 30 minutes of eval time.

## 3. Check it worked

Open the `caveman-compress` experiment in Braintrust and compare it with `caveman-record`. Look at:

- **Scores per example.** Read every example whose score dropped. An average that holds can hide one important regression.
- **Token metrics.** Compare the task's total prompt tokens across all LLM spans, not only the first call. A `caveman_retrieve` call is one more round trip.
- **Recovery tool spans.** Count `caveman_retrieve` spans. A few mean the model is using recovery when it needs detail. Recoveries on most examples mean the short view is missing what your tasks need.
- **The console.** The `onReport` lines should read `applied` or `reused` on heavy calls. `skipped no_candidate` means that call had no eligible tool result.

One thing that looks wrong but is correct: the LLM span's input shows your full tool output. AI SDK 7 hands telemetry the call's messages before the wrapped model applies Caveman's transform, so Braintrust records your originals, while the token metrics come from the provider's response for the shorter request. If you want the decision visible on the trace, store the `onReport` status as span metadata.

## Using wrapOpenAI instead

With a raw OpenAI client you own the tool loop, so use `withCavemanOpenAITools`. It adds `caveman_retrieve` to your tool definitions and returns a dispatch table that runs it. Plain `withCavemanOpenAI` on `chat.completions.create` has no recovery executor, and the runtime will not shorten text it cannot give back, so those calls go out unchanged. Put Braintrust's wrapper on the outside, because Caveman returns a new client built with `withOptions`:

```ts
import OpenAI from "openai";
import { wrapOpenAI } from "braintrust";
import { withCavemanOpenAITools } from "@caveman-ai/middleware/openai";

const loop = withCavemanOpenAITools(new OpenAI({ fetch: globalThis.fetch }), {
  runtime, scope, fetch: globalThis.fetch,
  protocol: "openai-chat", tools: chatTools, functions, // functions: { [tool name]: (args) => result }
});
const client = wrapOpenAI(loop.client);
const response = await client.chat.completions.create({ model, messages, tools: loop.tools });
// Run every tool call through loop.functions[name](JSON.parse(args)), caveman_retrieve included.
```

Pass the same `fetch` you built the client with, and send `loop.tools` on every request; a call without the recovery tool goes out uncompressed. The same trace rule applies: inputs show your originals, token metrics show the shorter request. To route through the Braintrust gateway, set `baseURL: "https://gateway.braintrust.dev/v1"` and your Braintrust key on the OpenAI client as usual; Caveman wraps the client and leaves the base URL alone.

## Roll back

- **Fastest:** run with `CAVEMAN_CLIENT_MODE=off`, which creates the runtime with `mode: "off"`. Calls pass through untouched.
- **Code:** replace `...withCaveman({ model, tools }, { runtime, scope })` with `model, tools`. `withCaveman` returns a new object and mutates nothing, so the call is back to exactly what it was.
- **Runtime:** restart it without `CAVEMAN_MODE=compress` and it returns to `record`.

Braintrust needs no change. Your experiments stay, including the two from this trial, which are a useful record of why you kept or dropped Caveman.

More: [Caveman middleware docs](https://docs.caveman.so/docs/sdk/middleware), [Vercel AI SDK adapter](https://docs.caveman.so/docs/sdk/middleware/vercel-ai-sdk), [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks), [Braintrust eval quickstart](https://www.braintrust.dev/docs/start/eval-sdk), and the [agent evaluations guide](/guides/agent-evaluations).