---
title: "Add Caveman to Vercel AI Gateway with the AI SDK"
description: "This guide adds Caveman Middleware to an AI SDK app that calls models through Vercel AI Gateway. The gateway, your API key, budgets, provider order and fallbacks all stay. You wrap the gateway model w"
canonical: https://caveman.ac/switch/vercel-ai-gateway
last-updated: 2026-10-04
---

# Add Caveman to Vercel AI Gateway with the AI SDK

This guide adds Caveman Middleware to an AI SDK app that calls models through Vercel AI Gateway. The gateway, your API key, budgets, provider order and fallbacks all stay. You wrap the gateway model with `withCaveman`, run one agent task twice, and compare the two requests in the AI Gateway logs. Background is in [Caveman vs Vercel AI Gateway](/compare/vercel-ai-gateway).

## Where will the Caveman runtime run?

Decide this first, because it shapes everything else. The middleware calls a Caveman runtime that does the compression and keeps originals on disk. The runtime is a long-running process, so it can't live inside a Vercel Function.

- A Node service, container or queue worker: run the runtime beside it on `127.0.0.1:8787`. This is the easy path.
- Vercel Functions: run the runtime on a host you keep up and connect over HTTPS with a token. The setup is at the end of this page.
- Your laptop: fine for the first test. Point the AI SDK at the gateway with your `AI_GATEWAY_API_KEY` as usual.

## Step 1: Start the runtime

```bash
npm install -g @caveman-ai/cli && caveman setup --install
caveman start
```

This starts in `record` mode, which measures calls and changes no bytes. That is your baseline. Check it:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'
```

## Step 2: Wrap the gateway model

Install the adapter and the runtime client next to `ai`:

```bash
npm install @caveman-ai/middleware @caveman-ai/sdk ai
```

The adapter accepts `ai >=7.0.94 <8`. Outside that range it passes calls through untouched and reports `unsupported_version`.

Then wrap the model and tools you already pass to `generateText` or `streamText`:

```ts
import { gateway, generateText, stepCountIs } from "ai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({
  endpoint: process.env.CAVE_RUNTIME_URL ?? "http://127.0.0.1:8787",
  mode,
  onReport: report => console.error("caveman", JSON.stringify(report)),
});
const scope = { namespace: "repo-agent", session_id: conversationId, branch_id: "main", cache_epoch: "0" };
if (mode !== "off") await runtime.ready();

const result = await generateText({
  ...withCaveman({ model: gateway("anthropic/claude-sonnet-5.5"), tools: { readFile, searchIssues } }, { runtime, scope }),
  messages,
  stopWhen: stepCountIs(8),
  providerOptions: { gateway: { order: ["vertex", "anthropic"] } },
});
```

Three details matter here.

Use `gateway("...")`, not a bare model string. The adapter wraps a model object, and `gateway()` gives you one that still routes through AI Gateway.

Don't replace `tools` after the spread. `withCaveman` adds `caveman_retrieve` to your tools and installs step callbacks that register it. If you have your own `onStepStart` or `onStepFinish`, put them inside the object you pass to `withCaveman` so both run. Overriding the tools or callbacks leaves calls without recovery.

Keep `session_id` stable across the conversation. A recovery handle only resolves under the scope that created it.

`providerOptions.gateway` goes to AI Gateway unchanged, so provider order and fallback models behave as before.

## Step 3: Run a baseline and a candidate

With the runtime still in `record`, run the task once. The client asks for compression, but a `record` runtime changes nothing, so you get a clean baseline through the same code. Then restart in compress mode and run the task again with a new `conversationId`:

```bash
CAVEMAN_MODE=compress caveman start
```

Same model, same messages, same tool data. If you want a second model in the test, do that as a separate experiment, or you won't know which change did what.

## How do you check it worked?

Start with the `onReport` lines. `applied` means at least one tool result was replaced. `reused` means a replacement from an earlier step was kept, which keeps the prompt prefix stable for provider caching. `skipped` comes with a reason, and all `skipped` with `no_candidate` means the task had no eligible tool output.

Then open AI Gateway logs, which can take up to 90 seconds to show a request. Compare input tokens and cost for the two runs, step by step. Count any `caveman_retrieve` round trips in the candidate, since recovery sends the original back and costs tokens.

Last, check the result the way you normally would. Fewer tokens with a wrong answer is a regression. The [cost measurement guide](/guides/measure-agent-cost) walks through fair comparisons, including failed attempts.

## How do you connect Vercel Functions to a remote runtime?

Run the runtime on a host you keep running, bound to a network address, with an inbound token:

```bash
CAVEMAN_AUTH_TOKEN="$(openssl rand -hex 24)" CAVEMAN_LISTEN=0.0.0.0:8787 CAVEMAN_MODE=compress caveman start
```

Put it behind TLS, then give the function the same token and explicit consent to send content off the machine:

```ts
const runtime = createMiddlewareRuntime({
  endpoint: "https://caveman.internal.example",
  token: process.env.CAVE_RUNTIME_TOKEN,
  allowRemoteContent: true,
  mode: "compress",
});
```

Without `https:` and `allowRemoteContent: true`, the client refuses to start with `remote_content_not_enabled`. Tool output now travels to that host, so treat it like any service that handles your data. If the runtime is unreachable, the adapter reports `runtime_unavailable` and sends the request as the AI SDK built it. The [deployment docs](https://docs.caveman.so/docs/sdk/middleware/deployment) cover scope, retention and the circuit breaker.

## How do you roll back?

Remove the wrapper. `withCaveman` returns a new object and mutates nothing, so going back to `model: gateway("anthropic/claude-sonnet-5.5"), tools: { readFile, searchIssues }` restores the previous behaviour exactly.

To switch it off without a code change at the call site, set `CAVEMAN_CLIENT_MODE=off`. The snippet passes it as `mode` on the runtime client and skips `ready()`, which throws in off mode. To keep measuring but stop compressing, restart the runtime with plain `caveman start`.

## Where to go next

- [Caveman Middleware](https://docs.caveman.so/docs/sdk/middleware) and [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks)
- [AI Gateway provider for the AI SDK](https://ai-sdk.dev/providers/ai-sdk-providers/ai-gateway) and [AI Gateway getting started](https://vercel.com/docs/ai-gateway/getting-started)
- [Add Caveman to the Vercel AI SDK](/switch/vercel-ai-sdk) and [Caveman vs Vercel AI Gateway](/compare/vercel-ai-gateway)