Add Caveman to Vercel AI Gateway with the AI SDK

This guide adds Caveman Middleware to an AI SDK app that calls models through Vercel AI Gateway. The gateway, your API key, budgets, provider order and fallbacks all stay. You wrap the gateway model with withCaveman, run one agent task twice, and compare the two requests in the AI Gateway logs. Background is in Caveman vs Vercel AI Gateway.

Where will the Caveman runtime run?

Decide this first, because it shapes everything else. The middleware calls a Caveman runtime that does the compression and keeps originals on disk. The runtime is a long-running process, so it can't live inside a Vercel Function.

• A Node service, container or queue worker: run the runtime beside it on 127.0.0.1:8787. This is the easy path.
• Vercel Functions: run the runtime on a host you keep up and connect over HTTPS with a token. The setup is at the end of this page.
• Your laptop: fine for the first test. Point the AI SDK at the gateway with your AI_GATEWAY_API_KEY as usual.

Step 1: Start the runtime

npm install -g @caveman-ai/cli && caveman setup --install
caveman start

This starts in record mode, which measures calls and changes no bytes. That is your baseline. Check it:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'

Step 2: Wrap the gateway model

Install the adapter and the runtime client next to ai:

npm install @caveman-ai/middleware @caveman-ai/sdk ai

The adapter accepts ai >=7.0.94 <8. Outside that range it passes calls through untouched and reports unsupported_version.

Then wrap the model and tools you already pass to generateText or streamText:

import { gateway, generateText, stepCountIs } from "ai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({
  endpoint: process.env.CAVE_RUNTIME_URL ?? "http://127.0.0.1:8787",
  mode,
  onReport: report => console.error("caveman", JSON.stringify(report)),
});
const scope = { namespace: "repo-agent", session_id: conversationId, branch_id: "main", cache_epoch: "0" };
if (mode !== "off") await runtime.ready();

const result = await generateText({
  ...withCaveman({ model: gateway("anthropic/claude-sonnet-5.5"), tools: { readFile, searchIssues } }, { runtime, scope }),
  messages,
  stopWhen: stepCountIs(8),
  providerOptions: { gateway: { order: ["vertex", "anthropic"] } },
});

Three details matter here.

Use gateway("..."), not a bare model string. The adapter wraps a model object, and gateway() gives you one that still routes through AI Gateway.

Don't replace tools after the spread. withCaveman adds caveman_retrieve to your tools and installs step callbacks that register it. If you have your own onStepStart or onStepFinish, put them inside the object you pass to withCaveman so both run. Overriding the tools or callbacks leaves calls without recovery.

Keep session_id stable across the conversation. A recovery handle only resolves under the scope that created it.

providerOptions.gateway goes to AI Gateway unchanged, so provider order and fallback models behave as before.

Step 3: Run a baseline and a candidate

With the runtime still in record, run the task once. The client asks for compression, but a record runtime changes nothing, so you get a clean baseline through the same code. Then restart in compress mode and run the task again with a new conversationId:

CAVEMAN_MODE=compress caveman start

Same model, same messages, same tool data. If you want a second model in the test, do that as a separate experiment, or you won't know which change did what.

How do you check it worked?

Start with the onReport lines. applied means at least one tool result was replaced. reused means a replacement from an earlier step was kept, which keeps the prompt prefix stable for provider caching. skipped comes with a reason, and all skipped with no_candidate means the task had no eligible tool output.

Then open AI Gateway logs, which can take up to 90 seconds to show a request. Compare input tokens and cost for the two runs, step by step. Count any caveman_retrieve round trips in the candidate, since recovery sends the original back and costs tokens.

Last, check the result the way you normally would. Fewer tokens with a wrong answer is a regression. The cost measurement guide walks through fair comparisons, including failed attempts.

How do you connect Vercel Functions to a remote runtime?

Run the runtime on a host you keep running, bound to a network address, with an inbound token:

CAVEMAN_AUTH_TOKEN="$(openssl rand -hex 24)" CAVEMAN_LISTEN=0.0.0.0:8787 CAVEMAN_MODE=compress caveman start

Put it behind TLS, then give the function the same token and explicit consent to send content off the machine:

const runtime = createMiddlewareRuntime({
  endpoint: "https://caveman.internal.example",
  token: process.env.CAVE_RUNTIME_TOKEN,
  allowRemoteContent: true,
  mode: "compress",
});

Without https: and allowRemoteContent: true, the client refuses to start with remote_content_not_enabled. Tool output now travels to that host, so treat it like any service that handles your data. If the runtime is unreachable, the adapter reports runtime_unavailable and sends the request as the AI SDK built it. The deployment docs cover scope, retention and the circuit breaker.

How do you roll back?

Remove the wrapper. withCaveman returns a new object and mutates nothing, so going back to model: gateway("anthropic/claude-sonnet-5.5"), tools: { readFile, searchIssues } restores the previous behaviour exactly.

To switch it off without a code change at the call site, set CAVEMAN_CLIENT_MODE=off. The snippet passes it as mode on the runtime client and skips ready(), which throws in off mode. To keep measuring but stop compressing, restart the runtime with plain caveman start.

Where to go next

• Caveman Middleware and supported frameworks
• AI Gateway provider for the AI SDK and AI Gateway getting started
• Add Caveman to the Vercel AI SDK and Caveman vs Vercel AI Gateway