Caveman vs Bifrost: fast AI gateway, fewer tool tokens

Bifrost makes the hop to your provider fast and reliable. Caveman makes the request smaller before it takes that hop. They stack cleanly: keep Bifrost for routing, failover, virtual keys and caching, and add Caveman Middleware in your app where agents send large tool output. A 20,000-token log goes through Bifrost in microseconds either way; the difference is how many of those tokens the provider bills you for.

What is Bifrost?

Bifrost is an open-source AI gateway from Maxim AI, written in Go and licensed Apache 2.0. It puts 20+ providers, including OpenAI, Anthropic, Bedrock and Vertex, behind one OpenAI-compatible API, and it is built for speed: its docs report 11 µs of added overhead per request at 5,000 requests per second. The GitHub repo has about 8,500 stars, and Maxim says more than 1,000 teams use it.

The open-source version already covers a lot:

• Failover between providers and models, and load balancing across API keys.
• Virtual keys with their own permissions, budgets, rate limits and routing.
• Semantic caching, with an exact-hash lookup first and an embedding match second, switched on per request with a cache key.
• An MCP gateway, including an opt-in agent mode where Bifrost runs approved tools itself.
• Go and WASM plugins, Prometheus metrics and OpenTelemetry.

Enterprise adds guardrails, adaptive load balancing, clustering, Okta and Entra sign-in, RBAC, VPC deployment and audit trails, with a 14-day trial. Getting started is one line: npx -y @maximhq/bifrost, then point an OpenAI client at http://localhost:8080/openai.

Does Caveman work with Bifrost?

Yes. Bifrost is a drop-in OpenAI-compatible endpoint, so Caveman wraps the client you already point at it. There is no Bifrost plugin and you do not need one: wrap the OpenAI SDK, the Anthropic SDK or your agent framework with Caveman Middleware and keep the base URL on Bifrost.

client = OpenAI(base_url="http://localhost:8080/openai", api_key=bifrost_virtual_key)

loop = with_caveman_openai_tools(client, runtime=runtime, scope=scope,
                                 protocol="openai-chat", tools=tools, functions=functions)

Before each call, the adapter copies the request, asks a local Caveman runtime to shorten eligible tool results, and sends the shorter copy to Bifrost. It adds a caveman_retrieve tool so the model can read any original back. Your messages list keeps the full text. User messages, system prompts, reasoning, errors and images are never changed.

TypeScript works the same way through the Vercel AI SDK, OpenAI, Anthropic, LangChain, Mastra or Strands adapters, and Python through LangChain, Pydantic AI, CrewAI, Agno and more. The Bifrost switching guide has a complete loop.

Where does each one save you money?

Bifrost saves in three ways: cache hits you never pay a provider for, failover that keeps a request from being thrown away, and budgets that stop spend at a limit. Its speed saves you latency, not tokens.

Caveman saves on input tokens from tool output. The published samples ran 13 synthetic tool results through the real runtime and counted tokens with o200k_base. An API response with 160 JSON records went from 7,202 tokens to 403. A worker log with one timeout went from 4,316 to 279. A git diff, rendered HTML and code search results came through unchanged. Those are inferred counts from single samples, not a bill, and the middleware page links the full set and the script that reproduces it.

The two add up because they hit different parts of the cost. A cached response costs nothing either way. An uncached one costs less when its tool results are shorter.

Bifrost vs Caveman: feature comparison

What about latency?

Caveman adds one call to a runtime on the same machine or pod before each request. That call has a deadline, 100 ms by default in the Python client. Outside strict mode, if the runtime misses the deadline or is down, the original request goes out unchanged, so the cost of a slow runtime is a short delay. Run the runtime close to the app and measure the added time on your own traffic. If your service lives on a microsecond budget and sends small prompts, the trade is not worth it.

Does Caveman see tools Bifrost runs in agent mode?

No. In Bifrost's MCP agent mode, which is off until you mark tools as auto-executable, Bifrost runs the tools and feeds results back to the model inside the gateway. Those results never pass through your app, so the middleware cannot shorten them. Caveman covers tool results your own code puts in the request, including MCP tools your app calls through the Caveman MCP adapter.

When should you use Bifrost alone?

Use Bifrost alone when requests are small, when the goal is uptime and routing, or when governance is the whole job. Caveman does none of that.

Use it alone if every component in the path has to be generally available today. Caveman Middleware is alpha, and Caveman's hosted gateway, Caveman Platform, is not generally available.

Add Caveman when an agent's tool output dominates its input: repo files, search results, test logs, JSON from internal APIs.

FAQ

Is Caveman a Bifrost alternative? No. It does not route, fail over, cache or hold keys. It shortens what you send through Bifrost.

Do I need to change my Bifrost config? No. Providers, virtual keys, plugins and cache settings stay as they are. The change is in your app.

Will Bifrost's semantic cache still hit? Within one conversation Caveman reuses the same shortened view for the same result, so identical requests stay identical. Watch your direct-hash hit rate during the test.

What does Bifrost record about the request? Whatever Bifrost logs or traces, it sees the shortened copy, because that is what it receives. Shortened tool results start with a [caveman: shortened; …] marker. The original stays in your app and the local Caveman store.

Sources

• Bifrost on GitHub
• Bifrost overview
• Bifrost with the OpenAI SDK
• Bifrost MCP agent mode and semantic caching
• Bifrost product page
• Caveman Middleware for Python and supported frameworks
• Choose an AI gateway