Caveman vs Vercel AI SDK: tool-result compression

The Vercel AI SDK runs your model calls, tool loop and streaming. Caveman Middleware plugs into it and shrinks large tool results before each call goes out. They do different jobs, so the usual answer is to keep the AI SDK and add the Caveman adapter where tool output is heavy.

What is the Vercel AI SDK?

The AI SDK is Vercel's free, open-source TypeScript toolkit for building AI apps and agents, from the team behind Next.js. It is Apache 2.0 licensed and sits at about 27k stars on GitHub. It comes in three parts: AI SDK Core (one API for text, structured objects, tool calls and agents), AI SDK UI (hooks for chat and generative interfaces) and AI SDK Harnesses (a uniform API for running established agent harnesses).

The boring parts are done well, which is why so many TypeScript agents start here:

• One call shape across providers through packages like @ai-sdk/openai, @ai-sdk/anthropic and @ai-sdk/google.
• ToolLoopAgent, which runs the model, calls tools and feeds results back until a stop condition. It stops after 20 steps by default.
• prepareStep, which lets you change the model, tools or messages between steps, plus a pruneMessages helper for simple compaction.
• Streaming straight into React, Next.js, Svelte, Vue and plain Node.
• A language model middleware system: wrapLanguageModel with transformParams, wrapGenerate and wrapStream.

That last one matters here, because it is the seam Caveman uses.

Does Caveman work with the Vercel AI SDK?

Yes. Caveman ships a native adapter at @caveman-ai/middleware/ai-sdk for ai >=7.0.94 <8 and @ai-sdk/provider >=4.0.11 <5. Its withCaveman function takes the options you already pass to streamText, generateText or a ToolLoopAgent and returns a new object: your model wrapped with the SDK's own wrapLanguageModel, plus one extra tool called caveman_retrieve.

+import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
+import { withCaveman } from "@caveman-ai/middleware/ai-sdk";
+
+const runtime = createMiddlewareRuntime({ endpoint: "http://127.0.0.1:8787", mode: "compress" });
+const scope = { namespace: "support", session_id: conversationId, branch_id: "main", cache_epoch: "0" };
+
-const result = streamText(options);
+const result = streamText(withCaveman(options, { runtime, scope }));

On each provider call, the adapter copies the outbound request, sends eligible tool-result text to a local Caveman runtime, and swaps in the shorter version it gets back. Your messages array and result.steps keep the original bytes. If the model needs a line that got cut, it calls caveman_retrieve and reads the original back. Your provider, retries, streaming and tools stay where they were.

Here is what that looks like on a real shape of data. The middleware docs send a 400-line worker log as a tool result. The runtime counted 8,399 tokens before and 177 to 187 after across sixteen runs. Those are inferred counts from the o200k_base tokenizer on one synthetic log, not a provider bill. The model sees the first and last lines, a note saying how many lines were elided, and a handle it can use to fetch the rest.

User messages, system prompts, reasoning, errors and image or PDF bytes are left alone. Only eligible tool-result text is touched.

Vercel AI SDK vs Caveman: feature comparison

When should you use the Vercel AI SDK alone?

Use it alone when your tool results are small. A weather lookup or a row count has nothing worth compressing, and the first call of any loop carries no tool result at all, so Caveman reports no_candidate and does nothing.

Skip Caveman too if your cost problem is long model output. Caveman shortens input; if output dominates, change the response format instead.

You also need a runtime you can reach. That matters for AI SDK apps on serverless functions: the runtime is a process with a store on disk, so it cannot ride along inside a function. You would point the function at a runtime on a host you keep running, over HTTPS with a token, or run without compression.

And if you already prune aggressively with prepareStep and the model still answers well, you may not need more. Measure first.

When does adding Caveman pay off?

It pays off in agents whose tools return a lot of text: file reads, API calls that return big JSON, shell commands, document extraction. In a tool loop every earlier result gets sent again on every later step. A 20k-token JSON response read at step 2 of a 10-step run rides along on eight more calls.

Caveman shortens that result on each resend. After the first cut it resends the same shortened bytes, reported as reused, so the provider's cached prefix stays byte identical from call to call.

It also stacks with what you have. pruneMessages decides which messages go out. Caveman shrinks the tool results in the messages that stay. The two do not fight, because the adapter works on the request that is about to be sent.

If the runtime is down, the call goes out with the original input and the report says runtime_unavailable. After three failures in a row the adapter stops asking for 30 seconds. Your agent keeps working either way.

How to add Caveman to the Vercel AI SDK

The short version:

1. npm install @caveman-ai/middleware @caveman-ai/sdk ai
2. Start the local runtime with CAVEMAN_MODE=compress caveman start.
3. Wrap your options with withCaveman(options, { runtime, scope }).
4. Log onReport and look for applied and reused.

The step-by-step switch guide covers the baseline, scope, rollback and how to read the reports. The Vercel AI SDK middleware docs have a complete runnable example and a local fixture test that makes no provider calls. For measuring the change on finished tasks, see how to measure agent cost.

FAQ

Does Caveman replace the Vercel AI SDK? No. The AI SDK still makes every model call, runs the tool loop and streams to your UI. Caveman only changes the tool-result text inside the outbound copy.

Does it work with streamText and ToolLoopAgent? Yes. withCaveman wraps the options object for streamText, generateText or a ToolLoopAgent. Keep the tools it returns, because replacing them after the wrap removes caveman_retrieve.

Does it rewrite my stored messages? No. The docs' test asserts that the application's own messages and tool results keep their original bytes. Only the copy sent to the provider shrinks.

Is this the same as Vercel AI Gateway? No. AI Gateway routes model calls to providers. Caveman shrinks what those calls carry. See Caveman vs Vercel AI Gateway.

What does it cost? The client and adapters are MIT licensed and the runtime runs on your own machine under BSL terms. Inference is still billed by your provider as before.

Sources

• AI SDK introduction
• AI SDK language model middleware
• AI SDK loop control
• vercel/ai on GitHub
• Caveman middleware: Vercel AI SDK
• Caveman middleware: supported frameworks