AI gateways
Caveman vs Vercel AI Gateway: fewer tokens per call
Vercel AI Gateway gives one key, budgets and fallbacks across 200+ models. Caveman Middleware wraps the AI SDK model to shrink tool output. How they fit.
- 01Keep the gateway model
- 02Wrap it with withCaveman
- 03Read tokens in gateway logs
Keep Vercel AI Gateway. Caveman works with it, and the fit is especially clean if you use the AI SDK. The gateway decides where a call goes and who pays for it. Caveman Middleware changes how many input tokens the call carries, by shortening the big tool results an agent keeps resending. You wrap the gateway model with one function and everything else, budgets included, keeps working.
What is Vercel AI Gateway?
Vercel AI Gateway is a managed gateway for calling models across providers from any infrastructure. Your app doesn't have to run on Vercel. You authenticate with an AI Gateway API key, or with OIDC on a Vercel deployment, and call models by provider/model slug.
What you get is mostly operational. Credentials live in one place, and BYOK lets you use your own provider accounts. Request logs show model, provider attempts, latency, token usage, status and cost. Budgets can be set per team, project, API key or team member. Requests fail over across providers, and you can configure ordered provider and model fallbacks. The AI SDK docs describe access to 200+ models, across text, image, video, speech, embeddings and more.
It speaks several protocols: the AI SDK, OpenAI Chat Completions, the OpenAI Responses API and Anthropic Messages. In the AI SDK, a plain string like 'openai/gpt-6-astra' routes through the gateway by default, and gateway() from ai gives you the provider instance. For coding agents, npx vercel@latest ai-gateway setup connects tools like Claude Code in one command.
Pricing is simple. Vercel charges no markup and no platform fee on tokens, including with BYOK. There is a free tier with a subset of models and lower rate limits, and buying credits opens the full catalog. Some add-ons cost extra, such as team-wide zero data retention and custom reporting.
Does Caveman work with Vercel AI Gateway?
Yes, and there are two ways in.
With the AI SDK, use the native adapter. withCaveman from @caveman-ai/middleware/ai-sdk wraps the model and tools you pass to generateText or streamText. It adds a caveman_retrieve tool and applies AI SDK language-model middleware, so the gateway model becomes a wrapped gateway model. Pass gateway('anthropic/claude-sonnet-5.5') rather than a bare string, because the adapter wraps a model object. The accepted range is ai >=7.0.94 <8.
With an OpenAI client pointed at https://ai-gateway.vercel.sh/v1, wrap the client with withCavemanOpenAITools in TypeScript or with_caveman_openai_tools in Python and leave the base URL alone.
Either way, before each call Caveman finds eligible tool results (file reads, API responses, command output, extracted documents) and replaces them with a shorter copy from a local runtime. Your app's message history keeps the originals. The model can call caveman_retrieve to get the full text back. System prompts, user messages, reasoning, errors and image or PDF bytes are left as they are.
Vercel AI Gateway vs Caveman: feature comparison
| Capability | Vercel AI Gateway | Caveman Middleware |
|---|---|---|
| One key across many providers | Yes, 200+ models | No, keeps your gateway |
| Token markup | None, including BYOK | Not a billing layer |
| Budgets per team, project, key, member | Yes | No |
| Provider and model fallbacks | Yes | No, your client keeps its retries |
| Request logs with cost | Yes | Per-call decision reports only |
| AI SDK integration | Default provider | withCaveman adapter |
| Shrink a large tool result | No | Yes, original recoverable |
| Runs in a Vercel Function | Yes, it is a hosted API | Client yes, runtime needs a kept host |
| Status | Live | Alpha |
Can Caveman run on Vercel Functions?
The client code can. The runtime can't live inside a function. It is a process with a store on disk, and a serverless invocation ends before either is useful. Caveman's deployment docs are direct about it: point the function at a runtime on a host you keep running, over HTTPS with an auth token and allowRemoteContent: true, or run the function without compression. With no reachable runtime, requests go out exactly as the AI SDK built them.
That makes long-running Node services, containers and background workers the easiest place to start. An agent running in a queue worker next to a Caveman container is the simplest production shape.
What do Vercel's logs show with Caveman in the path?
They show the shortened request. The middleware runs in your process before the call reaches the gateway, so AI Gateway logs, token usage and cost all describe the smaller body. Your app still holds the full tool output.
That is what makes the gateway a good place to measure. Run one task with the Caveman runtime in record mode and once in compress, then compare the two requests' input tokens in the logs. Add any caveman_retrieve round trips to the candidate, because recovery costs tokens too.
Budgets count spend billed through gateway system credentials (BYOK is metered separately), so smaller requests draw a budget down more slowly. Caveman doesn't set or enforce limits itself.
How much can Caveman save through Vercel AI Gateway?
It depends on how much of each request is tool output. For a sense of scale, the middleware docs show a 400-line worker log going from 8,399 tokens to between 177 and 187 in compress mode, counted with o200k_base and labelled inferred. The middleware product page reports 13 synthetic samples, also inferred. Your gateway logs are where the real number comes from.
When should you use Vercel AI Gateway alone?
Use it alone when your calls are short and carry few tool results. A chat UI sending short turns has nothing for Caveman to compress.
Use it alone when all your model calls run in Vercel Functions and you don't want to operate a runtime host. Compression needs that host.
Use it alone while you are still choosing models. Changing a slug and watching cost and latency in the logs is quick, and adding compression at the same time muddies the comparison.
When is it worth adding Caveman?
Add it when the gateway logs show input tokens growing call after call inside an agent loop. A streamText agent that reads repo files, calls a search API or loads a big JSON record, then keeps that output in context for ten more steps, is the case Caveman is built for.
If the heavy traffic is a coding agent on your laptop, such as Claude Code, the Caveman Proxy is the simpler fit. Caveman also has a hosted gateway, Caveman Cloud, open for sign-up with a Free plan.
How to add Caveman to Vercel AI Gateway
Start the runtime, wrap your gateway model with withCaveman, and compare two runs in the gateway logs. The switch guide has the code, the checks and the rollback. The Caveman vs Vercel AI SDK page covers the adapter itself in more depth.
FAQ
Is Caveman an alternative to Vercel AI Gateway? No. Caveman doesn't route, bill or log. It shrinks tool output in your app, and the gateway handles everything after that.
Does withCaveman work with a model string like 'openai/gpt-6-astra'?
Wrap gateway('openai/gpt-6-astra') instead. The adapter wraps a model object, and gateway() gives you one that still routes through AI Gateway.
Do provider options and fallbacks still work?
Yes. providerOptions.gateway settings such as order and models go to the gateway as before. Caveman only changes eligible tool-result text in the outbound copy.
Is Caveman Middleware production ready? It is alpha. The client and adapters are MIT and the runtime is BSL. Test one workflow and read the decision reports before widening it.
Sources
Try it on work you know
Keep your stack. Add the adapter.
Wrap the client or framework you already call, run one agent task you can check, and compare the tokens sent.
Open the middleware docs Explore comparisons ↗