How to add token compression to the Vercel AI SDK

You keep the Vercel AI SDK. Adding Caveman is one wrap around the options you already pass to streamText, generateText or a ToolLoopAgent, plus a small local runtime. For the why, read Caveman vs Vercel AI SDK.

What you need

• ai >=7.0.94 <8 and @ai-sdk/provider >=4.0.11 <5. Run npm ls ai @ai-sdk/provider to check. Outside that range the adapter reports unsupported_version and passes calls through untouched.
• Node.js 22.13 or newer for the caveman CLI that runs the runtime.
• A place for the runtime to live. Locally that is your machine. On serverless functions it has to be a host you keep running, reached over HTTPS; see a runtime on another host.
• One agent task whose tools return a lot of text: a log, a file read, a big JSON API response.

Step 1: Record a baseline

Run your chosen task a few times as it is today. Save the final answer, whether it passed your check, and the provider usage from result.totalUsage. You will compare against this later, so keep the model, prompt and tools fixed.

Step 2: Install the packages

npm install @caveman-ai/middleware @caveman-ai/sdk ai

@caveman-ai/middleware (0.1.0-alpha.2) holds the adapters and declares no framework dependency of its own. @caveman-ai/sdk (1.1.0) holds the runtime client.

Step 3: Start the local runtime

The runtime does the compression and stores the originals. It answers on 127.0.0.1:8787.

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

From another shell, confirm it is in compress mode:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'

You want "mode": "compress" and "recovery": true. Plain caveman start runs in record mode, which measures and changes nothing.

Step 4: Wrap the call

Create the runtime once, build a scope per conversation, and wrap your options. This is the core of the docs example, trimmed to the parts you add:

import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const runtime = createMiddlewareRuntime({
  endpoint: "http://127.0.0.1:8787",
  mode: "compress",
  onReport: report => console.error("Caveman:", JSON.stringify(report)),
});
const scope = { namespace: "support", session_id: conversationId, branch_id: "main", cache_epoch: "0" };

if (runtime.mode !== "off") await runtime.ready(); // ready() throws in off mode
const result = streamText(withCaveman(options, { runtime, scope }));

Get these right:

• Keep the scope stable. The four scope fields key the stored originals. A recovery handle from an earlier turn only resolves under the same four values, so derive session_id from your own conversation id.
• Keep the tools withCaveman returned. Replacing options.tools after the wrap removes caveman_retrieve, and the model loses its way back to the original.
• Set stopWhen with room for one more step. Compression starts on the call after a tool returns, and a recovery is one extra tool call. Set stopWhen explicitly, for example stepCountIs(8), and if your limit is tight, raise it by one.

withCaveman returns a new object and mutates nothing, so your existing options stay usable as the baseline.

Step 5: Check it worked

onReport prints one line per provider call. A healthy run looks like this:

• First call: skipped with no_candidate. There is no tool result yet.
• The call after the first tool returns: applied, with a transform_ids entry such as caveman.engine.log.v1.
• Later calls: reused. The same shortened bytes are resent, so the provider's cached prefix stays warm.

Then check the things that matter:

1. Your app's own tool result is still the original. The docs' example asserts that result.steps holds the full output.
2. The task still passes the same check as the baseline.
3. Provider-reported input tokens in result.totalUsage are lower for the same task. Count recovery calls and retries in the total.

If nothing shrank, the report reason tells you why. runtime_unavailable means the endpoint did not answer. recorded means the runtime is in record mode. recovery_unavailable means the runtime cannot store originals, so it refuses to shorten anything. The Vercel AI SDK middleware page has the full symptom table.

To test the mechanism without paying for model calls, the docs ship verify.mjs, which runs the example in off, record and compress against a local fixture server.

How to roll back

Pick whichever is fastest:

• Delete the withCaveman(...) call and pass options directly. Nothing else changed.
• Or set mode: "off" on the runtime. The adapter reports disabled and sends the original bytes.

Call runtime.close() at shutdown. Closing the client does not stop the runtime process, and stored originals stay resolvable for 24 hours after their last use, so conversations already in flight can still recover.

Next steps

Measure a handful of finished tasks, not one request. How to measure agent cost walks through that, and what prompt compression does to your bytes explains how to check the shortened copy. The supported frameworks table lists every adapter if you also call a provider SDK directly.