How to add Caveman to Bifrost: step by step setup

This guide adds Caveman Middleware to an app that already calls Bifrost. Bifrost keeps doing everything it does now: providers, virtual keys, failover, caching, plugins. Caveman runs inside your app and shortens large tool results in the outgoing copy of each request, so Bifrost forwards fewer tokens. Start with one agent. For the background, read the Bifrost comparison.

What you need before you start

• A running Bifrost, for example npx -y @maximhq/bifrost on localhost:8080, and a virtual key.
• An app that calls it through the OpenAI SDK or a framework such as the Vercel AI SDK, LangChain or Pydantic AI.
• Node.js 22.13 or newer for the caveman CLI, and Python 3.13 or newer for Python code.
• One agent whose tools return big text, and a task with a known answer.

Step 1: Start the Caveman runtime

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

It listens on 127.0.0.1:8787, so it does not clash with Bifrost on 8080, and keeps originals on local disk. Check the mode:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, recovery}'

Run it on the same host or pod as your app, not next to Bifrost if those are different machines. The runtime sits on the app's side of the call.

Step 2: Wrap the client you point at Bifrost

pip install caveman-sdk "caveman-middleware[openai]==0.1.0a1"

Keep the client exactly as Bifrost documents it and wrap it. with_caveman_openai_tools takes your tools and the functions that run them, and adds caveman_retrieve so the model can read an original back.

import json, os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

client = OpenAI(base_url="http://localhost:8080/openai",
                api_key=os.environ["BIFROST_VIRTUAL_KEY"])
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()
scope = Scope("ci-agent", conversation_id)

loop = with_caveman_openai_tools(client, runtime=runtime, scope=scope,
                                 protocol="openai-chat", tools=tools,
                                 functions={"read_ci_log": read_ci_log})

for _ in range(8):
    response = loop.client.chat.completions.create(
        model="openai/gpt-5.5", messages=messages, tools=loop.tools)
    message = response.choices[0].message
    messages.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        result = loop.functions[call.function.name](json.loads(call.function.arguments))
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": result if isinstance(result, str) else json.dumps(result)})

tools, messages and read_ci_log are yours, and the model string uses Bifrost's provider/model naming, so swap in one your virtual key allows. Keep conversation_id fixed for the conversation, because recovery handles only resolve under the scope that issued them. The adapter needs openai >=3.10 <4.

In TypeScript with the Vercel AI SDK, point the OpenAI provider at Bifrost and wrap the call options:

import { stepCountIs, streamText } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const bifrost = createOpenAI({ baseURL: "http://localhost:8080/openai", apiKey: process.env.BIFROST_VIRTUAL_KEY });
const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({ endpoint: "http://127.0.0.1:8787", mode });
const scope = { namespace: "ci", session_id: conversationId, branch_id: "main", cache_epoch: "0" };

if (mode !== "off") await runtime.ready();

const result = streamText({
  ...withCaveman({ model: bifrost.chat("gpt-4o-mini"), tools }, { runtime, scope }),
  messages,
  stopWhen: stepCountIs(8),
});

Other frameworks follow the same pattern. The framework list has the adapter and version range for each.

One limit to know: if you use Bifrost's MCP agent mode, tools Bifrost runs itself add their results inside the gateway, after Caveman has seen the request. Those results are not shortened. Tools your app runs are.

How to check it worked

1. The reports. on_report=print logs one decision per call. You want applied or reused with a transform such as caveman.engine.log.v1 on calls with big tool output. no_candidate means nothing was eligible; unsupported_version means your SDK version is outside the accepted range and the call went out unchanged.
2. Input tokens. Run the same task with CAVEMAN_CLIENT_MODE=off and with CAVEMAN_CLIENT_MODE=compress and compare response.usage.prompt_tokens. That count comes from the provider, through Bifrost.
3. Latency. Compare end-to-end time for both runs. Bifrost's overhead stays the same; the new cost is the local runtime call.
4. Quality. Check the answer, and count how often the model called caveman_retrieve. Lots of recovery calls eat into the saving.

If you use Bifrost's semantic cache, compare its hit rate between the runs too. The measurement guide covers repeats and retries.

How to roll back

Set CAVEMAN_CLIENT_MODE=off and restart the app. The snippets pass it as mode on the runtime client and skip ready(), so every request goes to Bifrost unchanged with no change at the call site. To remove Caveman fully, call client instead of loop.client with your own tools, or drop withCaveman(...) in TypeScript. Nothing in Bifrost needs undoing. Stop the runtime with Ctrl+C.

Sources: Bifrost with the OpenAI SDK, Bifrost on GitHub, Bifrost MCP agent mode, Caveman Python middleware, Caveman TypeScript middleware.