---
title: "How to add token compression to the Vercel AI SDK"
description: "You keep the Vercel AI SDK. Adding Caveman is one wrap around the options you already pass to streamText, generateText or a ToolLoopAgent, plus a small local runtime. For the why, read [Caveman vs Ver"
canonical: https://caveman.ac/switch/vercel-ai-sdk
last-updated: 2026-10-04
---

# How to add token compression to the Vercel AI SDK

You keep the Vercel AI SDK. Adding Caveman is one wrap around the options you already pass to `streamText`, `generateText` or a `ToolLoopAgent`, plus a small local runtime. For the why, read [Caveman vs Vercel AI SDK](/compare/vercel-ai-sdk).

## What you need

- `ai >=7.0.94 <8` and `@ai-sdk/provider >=4.0.11 <5`. Run `npm ls ai @ai-sdk/provider` to check. Outside that range the adapter reports `unsupported_version` and passes calls through untouched.
- Node.js 22.13 or newer for the `caveman` CLI that runs the runtime.
- A place for the runtime to live. Locally that is your machine. On serverless functions it has to be a host you keep running, reached over HTTPS; see [a runtime on another host](https://docs.caveman.so/docs/sdk/middleware/deployment).
- One agent task whose tools return a lot of text: a log, a file read, a big JSON API response.

## Step 1: Record a baseline

Run your chosen task a few times as it is today. Save the final answer, whether it passed your check, and the provider usage from `result.totalUsage`. You will compare against this later, so keep the model, prompt and tools fixed.

## Step 2: Install the packages

```bash
npm install @caveman-ai/middleware @caveman-ai/sdk ai
```

`@caveman-ai/middleware` (0.1.0-alpha.2) holds the adapters and declares no framework dependency of its own. `@caveman-ai/sdk` (1.1.0) holds the runtime client.

## Step 3: Start the local runtime

The runtime does the compression and stores the originals. It answers on `127.0.0.1:8787`.

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

From another shell, confirm it is in compress mode:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'
```

You want `"mode": "compress"` and `"recovery": true`. Plain `caveman start` runs in `record` mode, which measures and changes nothing.

## Step 4: Wrap the call

Create the runtime once, build a scope per conversation, and wrap your options. This is the core of the docs example, trimmed to the parts you add:

```ts
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const runtime = createMiddlewareRuntime({
  endpoint: "http://127.0.0.1:8787",
  mode: "compress",
  onReport: report => console.error("Caveman:", JSON.stringify(report)),
});
const scope = { namespace: "support", session_id: conversationId, branch_id: "main", cache_epoch: "0" };

if (runtime.mode !== "off") await runtime.ready(); // ready() throws in off mode
const result = streamText(withCaveman(options, { runtime, scope }));
```

Get these right:

- **Keep the scope stable.** The four scope fields key the stored originals. A recovery handle from an earlier turn only resolves under the same four values, so derive `session_id` from your own conversation id.
- **Keep the tools `withCaveman` returned.** Replacing `options.tools` after the wrap removes `caveman_retrieve`, and the model loses its way back to the original.
- **Set `stopWhen` with room for one more step.** Compression starts on the call after a tool returns, and a recovery is one extra tool call. Set `stopWhen` explicitly, for example `stepCountIs(8)`, and if your limit is tight, raise it by one.

`withCaveman` returns a new object and mutates nothing, so your existing `options` stay usable as the baseline.

## Step 5: Check it worked

`onReport` prints one line per provider call. A healthy run looks like this:

- First call: `skipped` with `no_candidate`. There is no tool result yet.
- The call after the first tool returns: `applied`, with a `transform_ids` entry such as `caveman.engine.log.v1`.
- Later calls: `reused`. The same shortened bytes are resent, so the provider's cached prefix stays warm.

Then check the things that matter:

1. Your app's own tool result is still the original. The docs' example asserts that `result.steps` holds the full output.
2. The task still passes the same check as the baseline.
3. Provider-reported input tokens in `result.totalUsage` are lower for the same task. Count recovery calls and retries in the total.

If nothing shrank, the report reason tells you why. `runtime_unavailable` means the endpoint did not answer. `recorded` means the runtime is in record mode. `recovery_unavailable` means the runtime cannot store originals, so it refuses to shorten anything. The [Vercel AI SDK middleware page](https://docs.caveman.so/docs/sdk/middleware/vercel-ai-sdk) has the full symptom table.

To test the mechanism without paying for model calls, the docs ship `verify.mjs`, which runs the example in `off`, `record` and `compress` against a local fixture server.

## How to roll back

Pick whichever is fastest:

- Delete the `withCaveman(...)` call and pass `options` directly. Nothing else changed.
- Or set `mode: "off"` on the runtime. The adapter reports `disabled` and sends the original bytes.

Call `runtime.close()` at shutdown. Closing the client does not stop the runtime process, and stored originals stay resolvable for 24 hours after their last use, so conversations already in flight can still recover.

## Next steps

Measure a handful of finished tasks, not one request. [How to measure agent cost](/guides/measure-agent-cost) walks through that, and [what prompt compression does to your bytes](/guides/prompt-compression) explains how to check the shortened copy. The [supported frameworks table](https://docs.caveman.so/docs/sdk/middleware/frameworks) lists every adapter if you also call a provider SDK directly.