---
title: "Caveman vs Vercel AI Gateway: fewer tokens per call"
description: "Keep Vercel AI Gateway. Caveman works with it, and the fit is especially clean if you use the AI SDK. The gateway decides where a call goes and who pays for it. Caveman Middleware changes how many inp"
canonical: https://caveman.ac/compare/vercel-ai-gateway
last-updated: 2026-10-12
---

# Caveman vs Vercel AI Gateway: fewer tokens per call

Keep Vercel AI Gateway. Caveman works with it, and the fit is especially clean if you use the AI SDK. The gateway decides where a call goes and who pays for it. Caveman Middleware changes how many input tokens the call carries, by shortening the big tool results an agent keeps resending. You wrap the gateway model with one function and everything else, budgets included, keeps working.

## What is Vercel AI Gateway?

Vercel AI Gateway is a managed gateway for calling models across providers from any infrastructure. Your app doesn't have to run on Vercel. You authenticate with an AI Gateway API key, or with OIDC on a Vercel deployment, and call models by `provider/model` slug.

What you get is mostly operational. Credentials live in one place, and BYOK lets you use your own provider accounts. Request logs show model, provider attempts, latency, token usage, status and cost. Budgets can be set per team, project, API key or team member. Requests fail over across providers, and you can configure ordered provider and model fallbacks. The AI SDK docs describe access to 200+ models, across text, image, video, speech, embeddings and more.

It speaks several protocols: the AI SDK, OpenAI Chat Completions, the OpenAI Responses API and Anthropic Messages. In the AI SDK, a plain string like `'openai/gpt-6-astra'` routes through the gateway by default, and `gateway()` from `ai` gives you the provider instance. For coding agents, `npx vercel@latest ai-gateway setup` connects tools like Claude Code in one command.

Pricing is simple. Vercel charges no markup and no platform fee on tokens, including with BYOK. There is a free tier with a subset of models and lower rate limits, and buying credits opens the full catalog. Some add-ons cost extra, such as team-wide zero data retention and custom reporting.

## Does Caveman work with Vercel AI Gateway?

Yes, and there are two ways in.

With the AI SDK, use the native adapter. `withCaveman` from `@caveman-ai/middleware/ai-sdk` wraps the model and tools you pass to `generateText` or `streamText`. It adds a `caveman_retrieve` tool and applies AI SDK language-model middleware, so the gateway model becomes a wrapped gateway model. Pass `gateway('anthropic/claude-sonnet-5.5')` rather than a bare string, because the adapter wraps a model object. The accepted range is `ai >=7.0.94 <8`.

With an OpenAI client pointed at `https://ai-gateway.vercel.sh/v1`, wrap the client with `withCavemanOpenAITools` in TypeScript or `with_caveman_openai_tools` in Python and leave the base URL alone.

Either way, before each call Caveman finds eligible tool results (file reads, API responses, command output, extracted documents) and replaces them with a shorter copy from a local runtime. Your app's message history keeps the originals. The model can call `caveman_retrieve` to get the full text back. System prompts, user messages, reasoning, errors and image or PDF bytes are left as they are.

## Vercel AI Gateway vs Caveman: feature comparison

| Capability | Vercel AI Gateway | Caveman Middleware |
| --- | --- | --- |
| One key across many providers | Yes, 200+ models | No, keeps your gateway |
| Token markup | None, including BYOK | Not a billing layer |
| Budgets per team, project, key, member | Yes | No |
| Provider and model fallbacks | Yes | No, your client keeps its retries |
| Request logs with cost | Yes | Per-call decision reports only |
| AI SDK integration | Default provider | `withCaveman` adapter |
| Shrink a large tool result | No | Yes, original recoverable |
| Runs in a Vercel Function | Yes, it is a hosted API | Client yes, runtime needs a kept host |
| Status | Live | Alpha |

## Can Caveman run on Vercel Functions?

The client code can. The runtime can't live inside a function. It is a process with a store on disk, and a serverless invocation ends before either is useful. Caveman's deployment docs are direct about it: point the function at a runtime on a host you keep running, over HTTPS with an auth token and `allowRemoteContent: true`, or run the function without compression. With no reachable runtime, requests go out exactly as the AI SDK built them.

That makes long-running Node services, containers and background workers the easiest place to start. An agent running in a queue worker next to a Caveman container is the simplest production shape.

## What do Vercel's logs show with Caveman in the path?

They show the shortened request. The middleware runs in your process before the call reaches the gateway, so AI Gateway logs, token usage and cost all describe the smaller body. Your app still holds the full tool output.

That is what makes the gateway a good place to measure. Run one task with the Caveman runtime in `record` mode and once in `compress`, then compare the two requests' input tokens in the logs. Add any `caveman_retrieve` round trips to the candidate, because recovery costs tokens too.

Budgets count spend billed through gateway system credentials (BYOK is metered separately), so smaller requests draw a budget down more slowly. Caveman doesn't set or enforce limits itself.

## How much can Caveman save through Vercel AI Gateway?

It depends on how much of each request is tool output. For a sense of scale, the middleware docs show a 400-line worker log going from 8,399 tokens to between 177 and 187 in compress mode, counted with `o200k_base` and labelled inferred. The middleware product page reports 13 synthetic samples, also inferred. Your gateway logs are where the real number comes from.

## When should you use Vercel AI Gateway alone?

Use it alone when your calls are short and carry few tool results. A chat UI sending short turns has nothing for Caveman to compress.

Use it alone when all your model calls run in Vercel Functions and you don't want to operate a runtime host. Compression needs that host.

Use it alone while you are still choosing models. Changing a slug and watching cost and latency in the logs is quick, and adding compression at the same time muddies the comparison.

## When is it worth adding Caveman?

Add it when the gateway logs show input tokens growing call after call inside an agent loop. A `streamText` agent that reads repo files, calls a search API or loads a big JSON record, then keeps that output in context for ten more steps, is the case Caveman is built for.

If the heavy traffic is a coding agent on your laptop, such as Claude Code, the [Caveman Proxy](/products/caveman-proxy) is the simpler fit. Caveman also has a hosted gateway, [Caveman Cloud](/products/platform), open for sign-up with a Free plan.

## How to add Caveman to Vercel AI Gateway

Start the runtime, wrap your gateway model with `withCaveman`, and compare two runs in the gateway logs. The [switch guide](/switch/vercel-ai-gateway) has the code, the checks and the rollback. The [Caveman vs Vercel AI SDK](/compare/vercel-ai-sdk) page covers the adapter itself in more depth.

## FAQ

**Is Caveman an alternative to Vercel AI Gateway?**
No. Caveman doesn't route, bill or log. It shrinks tool output in your app, and the gateway handles everything after that.

**Does `withCaveman` work with a model string like `'openai/gpt-6-astra'`?**
Wrap `gateway('openai/gpt-6-astra')` instead. The adapter wraps a model object, and `gateway()` gives you one that still routes through AI Gateway.

**Do provider options and fallbacks still work?**
Yes. `providerOptions.gateway` settings such as `order` and `models` go to the gateway as before. Caveman only changes eligible tool-result text in the outbound copy.

**Is Caveman Middleware production ready?**
It is alpha. The client and adapters are MIT and the runtime is BSL. Test one workflow and read the decision reports before widening it.

## Sources

- [Vercel AI Gateway overview](https://vercel.com/docs/ai-gateway) and [getting started](https://vercel.com/docs/ai-gateway/getting-started)
- [AI Gateway pricing](https://vercel.com/docs/ai-gateway/pricing)
- [AI Gateway provider for the AI SDK](https://ai-sdk.dev/providers/ai-sdk-providers/ai-gateway)
- [Caveman Middleware](https://docs.caveman.so/docs/sdk/middleware), [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks) and [deployment](https://docs.caveman.so/docs/sdk/middleware/deployment)
- Related: [choosing an AI gateway](/guides/choose-ai-gateway), [measuring agent cost](/guides/measure-agent-cost)