AI gateways
Caveman vs Cloudflare AI Gateway: add compression
Cloudflare AI Gateway gives you caching, rate limits and logs at the edge. Caveman Middleware compresses agent tool output before it gets there. Use both.
- 01Keep the gateway URL
- 02Wrap the client with Caveman
- 03Watch tokens in gateway analytics
Keep Cloudflare AI Gateway. Caveman works with it. The gateway controls traffic at the edge: caching, rate limits, fallbacks, spend limits, logs. Caveman Middleware works one step earlier, inside your app, and shrinks the large tool results an agent keeps resending. Point the client at Cloudflare as you do today, wrap it with Caveman, and both do their jobs on the same call.
What is Cloudflare AI Gateway?
Cloudflare AI Gateway is a proxy between your app and model providers, run on Cloudflare's network. You change a base URL and your calls to OpenAI, Anthropic, Google Gemini, Workers AI, Replicate and others go through it. It is available on all Cloudflare plans.
The feature list is long, and most of it is free. Caching serves repeated requests from Cloudflare's cache. Rate limiting caps request volume. Retries and model fallback handle provider failures. Dynamic routing lets you build routing flows in the dashboard without code changes, and Auto Router picks a model per request when you set the model to cloudflare/auto. Spend limits track cumulative cost against a budget. Guardrails moderate content, DLP catches sensitive data, BYOK stores provider keys, and analytics plus logging show requests, tokens and cost.
For OpenAI-style clients there is a compatible endpoint, https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/compat, with models named provider/model, for example openai/gpt-5.2.
Cloudflare's pricing page says the core features, including analytics, caching and rate limiting, are free. Guardrails are billed as Workers AI inference, Unified Billing adds a 5% fee on purchased credits, and log storage has plan limits. Gateways created on or after September 24, 2026 follow Workers Logs pricing and retention instead of the older limits.
Does Caveman work with Cloudflare AI Gateway?
Yes. There is no Cloudflare-specific adapter, because none is needed. Caveman Middleware wraps the provider client or framework in your code, and the client keeps its base URL. For an OpenAI SDK client pointed at the /compat endpoint, that is withCavemanOpenAITools in TypeScript or with_caveman_openai_tools in Python. If you use the Vercel AI SDK, LangChain, Mastra or another supported framework, wrap that instead. The frameworks page lists every adapter.
Before each call, the adapter finds eligible tool results (file reads, API responses, command output, extracted documents) and replaces them with a shorter copy from the Caveman runtime. The original stays in your app's history, and the model gets a caveman_retrieve tool to read it back when it needs the detail. System prompts, user messages, reasoning, errors and images or PDF bytes are not touched. The request then goes to Cloudflare with the same headers, including cf-aig-authorization.
Cloudflare AI Gateway vs Caveman: feature comparison
| Capability | Cloudflare AI Gateway | Caveman Middleware |
|---|---|---|
| Where it runs | Cloudflare's edge | In your app, with a runtime beside it |
| Response caching | Exact match on the whole request | No |
| Rate limits and spend limits | Yes | No |
| Retries and model fallback | Yes | No, your client keeps its own retries |
| Model routing | Dynamic routing, cloudflare/auto | No |
| Guardrails and DLP | Yes | No |
| Logs and cost analytics | Yes | Per-call decision reports only |
| Shrink a large tool result | No | Yes, original recoverable |
| Price | Core features free | MIT client, BSL runtime, no account needed |
| Status | Live on all Cloudflare plans | Alpha |
Why does the Cloudflare cache miss on agent traffic?
Cloudflare's cache matches the entire request exactly. Any difference in messages, tools or parameters makes a separate entry. That is perfect for a FAQ bot answering the same question 500 times, and a poor fit for an agent, where turn six carries everything from turn five plus a new tool result. The body is new every time, so the cache rarely hits.
The cost in agent work is the history each turn drags along. A 30,000-token crawl result read on turn two is paid for again on turn three, four and five. Caveman targets that content directly. The two mechanisms don't compete: cache what repeats, compress what doesn't.
One interaction to know about. Within a conversation, Caveman reuses the choice it made on an earlier turn, so the prefix stays byte stable. A new conversation gets a fresh recovery handle, so two separate conversations with the same tool output won't produce identical bodies. If you depend on exact-match hits for a repeated request that also carries tool output, test that path with Caveman on.
What does Cloudflare see when Caveman is in the path?
Cloudflare sees the shortened copy. Compression happens in your process before the request leaves, so the gateway's logs, token analytics and cost numbers all describe the smaller request. That is what you want for measuring the change.
It also means guardrails and DLP inspect the shortened tool output, since that is what arrives. If a DLP rule has to scan every byte of a tool result, test that rule with Caveman enabled before you roll out.
When should you use Cloudflare AI Gateway alone?
Use it alone when your traffic repeats. A cache hit costs nothing and beats any compression ratio.
Use it alone when your prompts are short and carry little tool output. Rate limits, spend limits and logs are the value there, and Caveman has nothing to shrink.
Use it alone when your app is a Worker and you want no extra process anywhere. Caveman's runtime is a process with a store on disk. It can't live inside a Worker, so a Worker would need to call a runtime on a host you keep running over HTTPS. Caveman's docs cover Node processes and containers, and we have not published a Workers-tested setup.
When does adding Caveman pay off?
Add it when gateway analytics show input tokens climbing turn after turn in agent runs. Coding agents, research agents that paste search results, and support agents that load order histories are the usual suspects. Cloudflare will show you the before and after in the same analytics view, which makes it a good place to test.
For coding agents like Claude Code or Codex on a laptop, the Caveman Proxy is the simpler tool. Middleware is for applications you build.
Caveman's own hosted gateway, Caveman Cloud, is open for sign-up with a Free plan. Cloudflare's is also free to start, so try whichever fits the traffic you have.
How to add Caveman to Cloudflare AI Gateway
Start the runtime on the same host as your app, wrap the client, keep the /compat base URL. The switch guide has the code, a baseline run, how to read the result in Cloudflare analytics, and the rollback.
FAQ
Is Caveman a Cloudflare AI Gateway alternative? No. Caveman doesn't cache, rate limit or log traffic. It compresses tool output inside your app, and Cloudflare stays the gateway.
Can the Caveman runtime run inside a Cloudflare Worker? No. It needs a process and a disk. A Worker can call a runtime you host elsewhere over HTTPS with an auth token, or run without compression.
Does Caveman work with Cloudflare Auto Router? Caveman doesn't pick models, so they don't overlap. The recovery tool is a normal function tool, so any model Auto Router picks needs to handle tool calls for recovery to work.
Will my Cloudflare logs show the full tool output?
They show the shortened copy that was sent. Your app's history keeps the original, and the model can fetch it with caveman_retrieve.
Sources
Try it on work you know
Keep your stack. Add the adapter.
Wrap the client or framework you already call, run one agent task you can check, and compare the tokens sent.
Open the middleware docs Explore comparisons ↗