Add Caveman to OpenRouter: wrap the OpenAI client

This guide adds Caveman Middleware to an app that already calls OpenRouter through the OpenAI SDK. The key, base URL, model slugs and attribution headers stay the same. You wrap the client, pin one model while you measure, and compare the two runs in your OpenRouter usage. For the comparison itself, including Auto Router, see Caveman vs OpenRouter.

What do you need first?

An app using the OpenAI SDK against https://openrouter.ai/api/v1, and one agent task you can grade. Choose a task where a tool returns something large, such as a search result, a JSON API response or a file. Versions: TypeScript needs openai >=7.12 <8; Python needs Python 3.13+ and openai >=3.10 <4. Out of range, the adapter passes calls through untouched.

Step 1: Start the runtime

The middleware sends eligible tool output to a local Caveman runtime, which returns the short copy and keeps the original on disk.

npm install -g @caveman-ai/cli && caveman setup --install
caveman start

That runs in record mode: it measures and changes nothing. Check it:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'

Step 2: Wrap the OpenRouter client

npm install @caveman-ai/middleware @caveman-ai/sdk openai

Build the client as you do today, and pass the same fetch to both the client and the adapter:

import OpenAI from "openai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCavemanOpenAITools } from "@caveman-ai/middleware/openai";

const fetch = globalThis.fetch;
const client = new OpenAI({
  baseURL: "https://openrouter.ai/api/v1",
  apiKey: process.env.OPENROUTER_API_KEY,
  defaultHeaders: { "HTTP-Referer": "https://example.com", "X-OpenRouter-Title": "Research agent" },
  fetch,
});

const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({
  endpoint: "http://127.0.0.1:8787",
  mode,
  onReport: report => console.error("caveman", JSON.stringify(report)),
});
const scope = { namespace: "research-agent", session_id: conversationId, branch_id: "main", cache_epoch: "0" };
if (mode !== "off") await runtime.ready();

const loop = withCavemanOpenAITools(client, {
  runtime, scope, fetch,
  protocol: "openai-chat",
  tools: [webSearchTool],
  functions: { web_search: webSearch },
});

const history = structuredClone(messages);
for (let step = 0; step < 8; step++) {
  const response = await loop.client.chat.completions.create({
    model: process.env.OPENROUTER_MODEL,
    messages: history,
    tools: loop.tools,
  });
  const message = response.choices[0].message;
  history.push(message);
  if (!message.tool_calls?.length) break;
  for (const call of message.tool_calls) {
    const output = await loop.functions[call.function.name](JSON.parse(call.function.arguments));
    history.push({ role: "tool", tool_call_id: call.id, content: typeof output === "string" ? output : JSON.stringify(output) });
  }
}

loop.tools is your tool list plus caveman_retrieve, and loop.functions runs both, which is how the model gets an original back when it needs a detail. history keeps the full tool output. Only the copy sent to OpenRouter is shortened. Keep session_id stable for the conversation, because a recovery handle only resolves under the scope that made it.

In Python, the same pattern uses with_caveman_openai_tools(client, runtime=runtime, scope=scope, protocol="openai-chat", tools=[...], functions={...}) from caveman_middleware.openai, installed with pip install 'caveman-sdk==1.1.0' 'caveman-middleware[openai]==0.1.0a1'. The Cloudflare guide shows a full Python loop that differs only in the base URL.

Step 3: Pin a model and run both arms

OpenRouter can change the provider or the model between runs: provider load balancing, fallbacks, and openrouter/auto if you use it. Any of those can move cost more than compression does. For the test, set OPENROUTER_MODEL to one concrete slug and, if you want to be strict, pin the provider with OpenRouter's provider options (order plus allow_fallbacks: false) as described in its provider routing docs.

Run the task once with the runtime in record mode. That is your baseline, through the same code. Then restart in compress mode and run it again with a new conversationId:

CAVEMAN_MODE=compress caveman start

Same model, same prompt, same tool data.

How do you check it worked?

Read the onReport output first. applied means a tool result was replaced, reused means an earlier replacement was kept so the prefix stays byte stable, and skipped carries a reason such as no_candidate or not_smaller. If every call is skipped, this task had nothing to compress.

Then compare the two runs in your OpenRouter activity: input tokens and cost per request. Add any caveman_retrieve calls to the candidate total, because recovery costs tokens. Keep OpenRouter's billed numbers and Caveman's local estimates in separate columns; the local ones are inferred.

Then grade the answer. If the compressed run needed more retries or got the task wrong, it didn't save anything. The cost measurement guide covers the bookkeeping.

Can you turn Auto Router back on afterwards?

Yes. Once you've measured compression on a pinned model, switch the model back to openrouter/auto if that's how you run in production. Two things to set. Use allowed_models on the auto-router plugin to keep the pool to models that handle tool calls well, since caveman_retrieve is a function tool. And read the model field on responses so you know which model served each call when you look at cost later.

How do you roll back?

Set CAVEMAN_CLIENT_MODE=off and restart. The snippet passes it as mode on the runtime client and skips ready(), which throws in off mode. Nothing goes to the runtime, and OpenRouter receives requests exactly as your code built them. To remove Caveman entirely, call client with your own tool list instead of loop.client and loop.tools. The wrapper builds a new client with withOptions, so the original is untouched.

Restarting the runtime with plain caveman start stops compression but keeps measuring. If the runtime goes down, the adapter sends the original request, so your OpenRouter traffic keeps flowing.

Where to go next

• Caveman Middleware, Python adapters and supported frameworks
• OpenRouter quickstart and Auto Router
• Caveman vs OpenRouter and the model routing guide