Add Caveman to Helicone: compress tool output

This guide adds Caveman Middleware to an app that already sends its model calls through the Helicone AI gateway. Nothing about Helicone changes: same base URL, same key, same session headers. You wrap the OpenAI client, run one task twice, and read the answer off two Helicone sessions. If you want the background first, read Caveman vs Helicone.

What do you need before you start?

You need an app using the OpenAI SDK against https://ai-gateway.helicone.ai, and one agent task whose result you can check. Pick a task where a tool returns something big: a file read, a JSON API response, a long command output. A task with only short messages will show no change.

Versions matter because the adapters check them and pass the call through untouched when they are out of range. TypeScript needs openai >=7.12 <8. Python needs Python 3.13+ and openai >=3.10 <4. The frameworks page has the full table.

Step 1: Start the Caveman runtime

The middleware talks to a local runtime that does the compression and keeps the originals on disk. It is the Caveman proxy, installed by the CLI:

npm install -g @caveman-ai/cli && caveman setup --install
caveman start

Plain caveman start runs in record mode: it measures every call and hands the bytes back unchanged. That is your baseline. Leave it running and check it answers:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'

Step 2: Wrap the Helicone client

Install the packages next to your app:

npm install @caveman-ai/middleware @caveman-ai/sdk openai

Then wrap the client. The base URL stays on Helicone. The TypeScript adapter needs the same fetch the client was built with, so pass it explicitly:

import OpenAI from "openai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCavemanOpenAITools } from "@caveman-ai/middleware/openai";

const fetch = globalThis.fetch;
const client = new OpenAI({
  baseURL: "https://ai-gateway.helicone.ai",
  apiKey: process.env.HELICONE_API_KEY,
  fetch,
});

const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({
  endpoint: "http://127.0.0.1:8787",
  mode,
  onReport: report => console.error("caveman", JSON.stringify(report)),
});
const scope = { namespace: "support-agent", session_id: sessionId, branch_id: "main", cache_epoch: "0" };
if (mode !== "off") await runtime.ready();

const loop = withCavemanOpenAITools(client, {
  runtime, scope, fetch,
  protocol: "openai-chat",
  tools: [readOrderHistoryTool],
  functions: { read_order_history: readOrderHistory },
});

const response = await loop.client.chat.completions.create(
  { model: "gpt-4o-mini", messages: history, tools: loop.tools },
  { headers: {
    "Helicone-Session-Id": sessionId,
    "Helicone-Session-Path": "/support",
    "Helicone-Session-Name": "support-agent caveman",
  } },
);

loop.tools is your tool list plus caveman_retrieve. When the model calls a tool, dispatch it through loop.functions[name](args) so the recovery tool runs too. Keep session_id stable for the whole conversation, because a recovery handle only resolves under the scope that created it.

In Python the shape is the same:

python -m pip install 'caveman-sdk==1.1.0' 'caveman-middleware[openai]==0.1.0a1'

import json, os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

client = OpenAI(base_url="https://ai-gateway.helicone.ai", api_key=os.environ["HELICONE_API_KEY"])
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode, deadline_ms=500, on_report=print)
scope = Scope("support-agent", session_id, "main", "0")
if mode != "off":
    runtime.ready()

loop = with_caveman_openai_tools(
    client, runtime=runtime, scope=scope, protocol="openai-chat",
    tools=[read_order_history_tool], functions={"read_order_history": read_order_history},
)
headers = {
    "Helicone-Session-Id": session_id,
    "Helicone-Session-Path": "/support",
    "Helicone-Session-Name": "support-agent caveman",
}

for _ in range(8):
    response = loop.client.chat.completions.create(
        model="gpt-4o-mini", messages=history, tools=loop.tools, extra_headers=headers)
    message = response.choices[0].message
    history.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        output = loop.functions[call.function.name](json.loads(call.function.arguments))
        history.append({"role": "tool", "tool_call_id": call.id,
                        "content": output if isinstance(output, str) else json.dumps(output)})

Use with_caveman_openai_tools here, not the plain with_caveman_openai. The plain wrapper registers no recovery tool, and without one Caveman forwards every tool result unchanged.

Step 3: Run a baseline and a candidate

With the runtime still in record mode, run the task once. The client asks for compression, but a runtime in record changes nothing, so this is a clean baseline through the same code path. Name the session support-agent baseline.

Now restart the runtime in compress mode and run the same task again with a new session ID:

CAVEMAN_MODE=compress caveman start

Same model, same prompt, same tool data. Both arms cold, or both warm. Never reuse one session ID for both arms.

How do you check it worked?

Three places tell you.

The onReport lines in your console. applied means at least one tool result was replaced, reused means a replacement from an earlier turn was kept so the prefix stays stable, and skipped comes with a reason such as no_candidate or not_smaller. If every line is skipped, your task has nothing eligible to compress.

The Helicone session. Open both sessions side by side and compare total input tokens, the number of calls, and any caveman_retrieve tool calls in the candidate. Those recovery calls cost tokens and belong in the total.

The task result. A cheaper run that gets the answer wrong is a failed run. Check the output the same way you would without Caveman. The cost measurement guide covers counting retries and failed attempts.

One thing to expect: Helicone now logs the shortened tool result, because compression happens in your process before the request is sent. Your app's history keeps the original.

How do you roll back?

Set CAVEMAN_CLIENT_MODE=off and redeploy. Both snippets pass it as mode on the runtime client and skip ready(), which throws in off mode. Nothing is sent to the runtime and requests go to Helicone as your framework built them. To remove Caveman completely, call the original client instead of the wrapped one and delete the wrapper line. The TypeScript wrapper builds a new client with withOptions, so the original is untouched.

To keep measuring without changing anything, restart the runtime with plain caveman start. That puts it back in record.

If the runtime stops or becomes unreachable, the adapter sends the original request. Your Helicone traffic keeps flowing.

Where to go next

• Caveman Middleware docs, Python middleware and supported frameworks
• Helicone sessions and gateway quickstart
• Caveman vs Helicone and migrating between gateways