AI gateways
How to choose an AI gateway for agents
A decision path instead of a feature matrix. Name the one problem you have now, key sprawl, model access, spend limits, caching or context size, then buy for that.
- 01Name the one problem
- 02Rule out a gateway
- 03Replay your hardest request
- 04Recheck in thirty days
Buy the gateway that fixes the one problem you have this week, and ignore the rest of its feature page. Every gateway looks the same in a matrix because they all list the same twelve features.
Here is the constraint behind that. A gateway is a chokepoint: all your agent traffic runs through one process you now operate. Every feature you did not need is a new way for that process to fail, and every feature you did need has to work on your exact requests, not on a chat demo.
Say the problem out loud before you open any comparison page
There are five reasons teams actually buy a gateway. One of them is hurting right now. Pick it.
| What is going wrong | What the gateway has to do | What you can ignore |
|---|---|---|
| Six services each hold a copy of the same provider key | Issue per-caller keys you can revoke without a redeploy | Model count, routing policy |
| You need a model your current provider does not host | Reach it without a second billing relationship and a second SDK | Guardrails, budgets |
| One agent loop spent the month's budget on a Saturday | Stop the caller at a hard ceiling, mid-run, not in tomorrow's report | Caching, prompt management |
| The same system prompt and file tree ship on every turn | Add provider cache hints without changing the bytes the model sees | Virtual keys, fallbacks |
| Requests are large and getting larger | Nothing. Read the next section. | All of it |
If two rows apply, fix the top one and stop. Changing key custody and adding caching in one release means that when quality drops you cannot say which change did it.
A gateway moves requests. It does not make them smaller.
If your bill climbs because a 40,000 token log file rides along on every turn, a new gateway routes those same 40,000 tokens down a different pipe. The spend is in the payload, not the path.
One gateway is being built to do both. Caveman Platform is the byte-safe LLM gateway: point traffic at one URL and caching, compression and routing run eval gated. Caching is the only one of the three producing verified dollars today, and it adds provider-native cache hints to the upstream request while the bytes the model sees stay as they were. Caveman Cloud is open for sign-up with a Free plan; routing is still in development.
Test that separately, against the gateway you already run. Start with prompt compression for tool output, and MCP token overhead if your tool definitions load on every request. Caveman's local reports are labelled inferred; treat them as a reason to run a provider-side test, not as money saved.
Replay your hardest request before you shortlist anything
"OpenAI compatible" covers text. It does not promise that your tool calls, your structured output schema, your streaming event names and your image parts all survive the trip. Take the single most complicated request your agent sends and send that one.
Three results decide it:
- The tool call comes back with the argument shape your parser already expects, not a JSON string where an object used to be.
- Cancelling the stream halfway stops the upstream call, and your client still recognizes the events.
- A request for a model the caller is not allowed to use gets refused.
The refusal test is the one everyone skips. A candidate that answers a forbidden request correctly has failed.
Replace feature-page criteria with numbers from your own traffic
| Weak criterion | What to measure instead |
|---|---|
| "Supports 300 models" | The four models you pin today, at the versions you pin |
| "Adds caching" | A cold run and a warm run of one real task, read from the provider's cache fields |
| "Sub-millisecond overhead" | Your p95 at your concurrency, with your payload size and streaming on |
| "Full observability" | One expensive task traced end to end, joined to whether the task was accepted |
| "Automatic failover" | A forced provider failure, checking whether any tool ran twice |
Watch for the three ways this choice goes wrong
The catalog shortlist. You choose on 300 models and then call four of them forever. Spot it when nobody can name the second model in the list. Fix: shortlist on what you pin now plus the one model you expect to want next quarter.
The gateway that becomes the outage. Every request now depends on one hop you own. Spot it by killing the gateway in staging: if your agent hangs until the request timeout instead of failing fast, you bought a new incident. Fix: set a short connect timeout at the client and decide in advance whether a gateway failure falls back to the provider or stops the run.
The savings number built from two different caches. Provider prefix caching and gateway response caching have different correctness risks and different hit conditions. Fix: measure them apart using prompt caching, and never publish one blended percentage.
Read the tradeoffs for each candidate, then commit
LiteLLM, OpenRouter, Portkey, Helicone, Cloudflare AI Gateway, Vercel AI Gateway and Bifrost each have a page covering what they own and where they stop.
Put Caveman Cloud on the same shortlist with its status written next to it: open for sign-up, with a Free plan for one person and routing still in development. Try it on one workload before you widen it.
Check the edition as well as the product. A control described on an enterprise pricing page may not exist in the self-hosted package you were planning to install, and the difference usually shows up after the migration.
Know within thirty days whether you chose right
You chose right when the problem you named in the first table is gone, your error rate and p95 match the old path, and cost per completed task has not quietly risen because retries moved somewhere you stopped counting.
Keep the old path configured and reachable until that day. Gateway migration has the rollback sequence and the equivalence checks to run first.
Try it on work you know
Keep your stack. Add the adapter.
Wrap the client or framework you already call, run one agent task you can check, and compare the tokens sent.
Open the middleware docs Explore comparisons ↗