---
title: "How to add Caveman to Bifrost: step by step setup"
description: "This guide adds Caveman Middleware to an app that already calls Bifrost. Bifrost keeps doing everything it does now: providers, virtual keys, failover, caching, plugins. Caveman runs inside your app a"
canonical: https://caveman.ac/switch/bifrost
last-updated: 2026-10-04
---

# How to add Caveman to Bifrost: step by step setup

This guide adds Caveman Middleware to an app that already calls Bifrost. Bifrost keeps doing everything it does now: providers, virtual keys, failover, caching, plugins. Caveman runs inside your app and shortens large tool results in the outgoing copy of each request, so Bifrost forwards fewer tokens. Start with one agent. For the background, read the [Bifrost comparison](/compare/bifrost).

## What you need before you start

- A running Bifrost, for example `npx -y @maximhq/bifrost` on `localhost:8080`, and a virtual key.
- An app that calls it through the OpenAI SDK or a framework such as the Vercel AI SDK, LangChain or Pydantic AI.
- Node.js 22.13 or newer for the `caveman` CLI, and Python 3.13 or newer for Python code.
- One agent whose tools return big text, and a task with a known answer.

## Step 1: Start the Caveman runtime

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

It listens on `127.0.0.1:8787`, so it does not clash with Bifrost on 8080, and keeps originals on local disk. Check the mode:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, recovery}'
```

Run it on the same host or pod as your app, not next to Bifrost if those are different machines. The runtime sits on the app's side of the call.

## Step 2: Wrap the client you point at Bifrost

```bash
pip install caveman-sdk "caveman-middleware[openai]==0.1.0a1"
```

Keep the client exactly as Bifrost documents it and wrap it. `with_caveman_openai_tools` takes your tools and the functions that run them, and adds `caveman_retrieve` so the model can read an original back.

```python
import json, os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

client = OpenAI(base_url="http://localhost:8080/openai",
                api_key=os.environ["BIFROST_VIRTUAL_KEY"])
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()
scope = Scope("ci-agent", conversation_id)

loop = with_caveman_openai_tools(client, runtime=runtime, scope=scope,
                                 protocol="openai-chat", tools=tools,
                                 functions={"read_ci_log": read_ci_log})

for _ in range(8):
    response = loop.client.chat.completions.create(
        model="openai/gpt-5.5", messages=messages, tools=loop.tools)
    message = response.choices[0].message
    messages.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        result = loop.functions[call.function.name](json.loads(call.function.arguments))
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": result if isinstance(result, str) else json.dumps(result)})
```

`tools`, `messages` and `read_ci_log` are yours, and the model string uses Bifrost's `provider/model` naming, so swap in one your virtual key allows. Keep `conversation_id` fixed for the conversation, because recovery handles only resolve under the scope that issued them. The adapter needs `openai >=3.10 <4`.

In TypeScript with the Vercel AI SDK, point the OpenAI provider at Bifrost and wrap the call options:

```ts
import { stepCountIs, streamText } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const bifrost = createOpenAI({ baseURL: "http://localhost:8080/openai", apiKey: process.env.BIFROST_VIRTUAL_KEY });
const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({ endpoint: "http://127.0.0.1:8787", mode });
const scope = { namespace: "ci", session_id: conversationId, branch_id: "main", cache_epoch: "0" };

if (mode !== "off") await runtime.ready();

const result = streamText({
  ...withCaveman({ model: bifrost.chat("gpt-4o-mini"), tools }, { runtime, scope }),
  messages,
  stopWhen: stepCountIs(8),
});
```

Other frameworks follow the same pattern. The [framework list](https://docs.caveman.so/docs/sdk/middleware/frameworks) has the adapter and version range for each.

One limit to know: if you use Bifrost's MCP agent mode, tools Bifrost runs itself add their results inside the gateway, after Caveman has seen the request. Those results are not shortened. Tools your app runs are.

## How to check it worked

1. **The reports.** `on_report=print` logs one decision per call. You want `applied` or `reused` with a transform such as `caveman.engine.log.v1` on calls with big tool output. `no_candidate` means nothing was eligible; `unsupported_version` means your SDK version is outside the accepted range and the call went out unchanged.
2. **Input tokens.** Run the same task with `CAVEMAN_CLIENT_MODE=off` and with `CAVEMAN_CLIENT_MODE=compress` and compare `response.usage.prompt_tokens`. That count comes from the provider, through Bifrost.
3. **Latency.** Compare end-to-end time for both runs. Bifrost's overhead stays the same; the new cost is the local runtime call.
4. **Quality.** Check the answer, and count how often the model called `caveman_retrieve`. Lots of recovery calls eat into the saving.

If you use Bifrost's semantic cache, compare its hit rate between the runs too. The [measurement guide](/guides/measure-agent-cost) covers repeats and retries.

## How to roll back

Set `CAVEMAN_CLIENT_MODE=off` and restart the app. The snippets pass it as `mode` on the runtime client and skip `ready()`, so every request goes to Bifrost unchanged with no change at the call site. To remove Caveman fully, call `client` instead of `loop.client` with your own `tools`, or drop `withCaveman(...)` in TypeScript. Nothing in Bifrost needs undoing. Stop the runtime with Ctrl+C.

Sources: [Bifrost with the OpenAI SDK](https://docs.getbifrost.ai/integrations/openai-sdk), [Bifrost on GitHub](https://github.com/maximhq/bifrost), [Bifrost MCP agent mode](https://docs.getbifrost.ai/mcp/agent-mode), [Caveman Python middleware](https://docs.caveman.so/docs/sdk/middleware/python), [Caveman TypeScript middleware](https://docs.caveman.so/docs/sdk/middleware).