---
title: "Add Caveman to Helicone: compress tool output"
description: "This guide adds Caveman Middleware to an app that already sends its model calls through the Helicone AI gateway. Nothing about Helicone changes: same base URL, same key, same session headers. You wrap"
canonical: https://caveman.ac/switch/helicone
last-updated: 2026-10-04
---

# Add Caveman to Helicone: compress tool output

This guide adds Caveman Middleware to an app that already sends its model calls through the Helicone AI gateway. Nothing about Helicone changes: same base URL, same key, same session headers. You wrap the OpenAI client, run one task twice, and read the answer off two Helicone sessions. If you want the background first, read [Caveman vs Helicone](/compare/helicone).

## What do you need before you start?

You need an app using the OpenAI SDK against `https://ai-gateway.helicone.ai`, and one agent task whose result you can check. Pick a task where a tool returns something big: a file read, a JSON API response, a long command output. A task with only short messages will show no change.

Versions matter because the adapters check them and pass the call through untouched when they are out of range. TypeScript needs `openai >=7.12 <8`. Python needs Python 3.13+ and `openai >=3.10 <4`. The [frameworks page](https://docs.caveman.so/docs/sdk/middleware/frameworks) has the full table.

## Step 1: Start the Caveman runtime

The middleware talks to a local runtime that does the compression and keeps the originals on disk. It is the Caveman proxy, installed by the CLI:

```bash
npm install -g @caveman-ai/cli && caveman setup --install
caveman start
```

Plain `caveman start` runs in `record` mode: it measures every call and hands the bytes back unchanged. That is your baseline. Leave it running and check it answers:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'
```

## Step 2: Wrap the Helicone client

Install the packages next to your app:

```bash
npm install @caveman-ai/middleware @caveman-ai/sdk openai
```

Then wrap the client. The base URL stays on Helicone. The TypeScript adapter needs the same `fetch` the client was built with, so pass it explicitly:

```ts
import OpenAI from "openai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCavemanOpenAITools } from "@caveman-ai/middleware/openai";

const fetch = globalThis.fetch;
const client = new OpenAI({
  baseURL: "https://ai-gateway.helicone.ai",
  apiKey: process.env.HELICONE_API_KEY,
  fetch,
});

const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({
  endpoint: "http://127.0.0.1:8787",
  mode,
  onReport: report => console.error("caveman", JSON.stringify(report)),
});
const scope = { namespace: "support-agent", session_id: sessionId, branch_id: "main", cache_epoch: "0" };
if (mode !== "off") await runtime.ready();

const loop = withCavemanOpenAITools(client, {
  runtime, scope, fetch,
  protocol: "openai-chat",
  tools: [readOrderHistoryTool],
  functions: { read_order_history: readOrderHistory },
});

const response = await loop.client.chat.completions.create(
  { model: "gpt-4o-mini", messages: history, tools: loop.tools },
  { headers: {
    "Helicone-Session-Id": sessionId,
    "Helicone-Session-Path": "/support",
    "Helicone-Session-Name": "support-agent caveman",
  } },
);
```

`loop.tools` is your tool list plus `caveman_retrieve`. When the model calls a tool, dispatch it through `loop.functions[name](args)` so the recovery tool runs too. Keep `session_id` stable for the whole conversation, because a recovery handle only resolves under the scope that created it.

In Python the shape is the same:

```bash
python -m pip install 'caveman-sdk==1.1.0' 'caveman-middleware[openai]==0.1.0a1'
```

```python
import json, os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

client = OpenAI(base_url="https://ai-gateway.helicone.ai", api_key=os.environ["HELICONE_API_KEY"])
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode, deadline_ms=500, on_report=print)
scope = Scope("support-agent", session_id, "main", "0")
if mode != "off":
    runtime.ready()

loop = with_caveman_openai_tools(
    client, runtime=runtime, scope=scope, protocol="openai-chat",
    tools=[read_order_history_tool], functions={"read_order_history": read_order_history},
)
headers = {
    "Helicone-Session-Id": session_id,
    "Helicone-Session-Path": "/support",
    "Helicone-Session-Name": "support-agent caveman",
}

for _ in range(8):
    response = loop.client.chat.completions.create(
        model="gpt-4o-mini", messages=history, tools=loop.tools, extra_headers=headers)
    message = response.choices[0].message
    history.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        output = loop.functions[call.function.name](json.loads(call.function.arguments))
        history.append({"role": "tool", "tool_call_id": call.id,
                        "content": output if isinstance(output, str) else json.dumps(output)})
```

Use `with_caveman_openai_tools` here, not the plain `with_caveman_openai`. The plain wrapper registers no recovery tool, and without one Caveman forwards every tool result unchanged.

## Step 3: Run a baseline and a candidate

With the runtime still in `record` mode, run the task once. The client asks for compression, but a runtime in `record` changes nothing, so this is a clean baseline through the same code path. Name the session `support-agent baseline`.

Now restart the runtime in compress mode and run the same task again with a new session ID:

```bash
CAVEMAN_MODE=compress caveman start
```

Same model, same prompt, same tool data. Both arms cold, or both warm. Never reuse one session ID for both arms.

## How do you check it worked?

Three places tell you.

The `onReport` lines in your console. `applied` means at least one tool result was replaced, `reused` means a replacement from an earlier turn was kept so the prefix stays stable, and `skipped` comes with a reason such as `no_candidate` or `not_smaller`. If every line is `skipped`, your task has nothing eligible to compress.

The Helicone session. Open both sessions side by side and compare total input tokens, the number of calls, and any `caveman_retrieve` tool calls in the candidate. Those recovery calls cost tokens and belong in the total.

The task result. A cheaper run that gets the answer wrong is a failed run. Check the output the same way you would without Caveman. The [cost measurement guide](/guides/measure-agent-cost) covers counting retries and failed attempts.

One thing to expect: Helicone now logs the shortened tool result, because compression happens in your process before the request is sent. Your app's history keeps the original.

## How do you roll back?

Set `CAVEMAN_CLIENT_MODE=off` and redeploy. Both snippets pass it as `mode` on the runtime client and skip `ready()`, which throws in off mode. Nothing is sent to the runtime and requests go to Helicone as your framework built them. To remove Caveman completely, call the original `client` instead of the wrapped one and delete the wrapper line. The TypeScript wrapper builds a new client with `withOptions`, so the original is untouched.

To keep measuring without changing anything, restart the runtime with plain `caveman start`. That puts it back in `record`.

If the runtime stops or becomes unreachable, the adapter sends the original request. Your Helicone traffic keeps flowing.

## Where to go next

- [Caveman Middleware docs](https://docs.caveman.so/docs/sdk/middleware), [Python middleware](https://docs.caveman.so/docs/sdk/middleware/python) and [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks)
- [Helicone sessions](https://docs.helicone.ai/features/sessions) and [gateway quickstart](https://docs.helicone.ai/getting-started/quick-start)
- [Caveman vs Helicone](/compare/helicone) and [migrating between gateways](/guides/gateway-migration)