---
title: "Use Caveman with Not Diamond: add token compression"
description: "You don't have to move off Not Diamond to try Caveman. Not Diamond's chat router returns a model recommendation and leaves the call to you, which leaves a clean spot for compression: wrap the provider"
canonical: https://caveman.ac/switch/not-diamond
last-updated: 2026-10-12
---

# Use Caveman with Not Diamond: add token compression

You don't have to move off Not Diamond to try Caveman. Not Diamond's chat router returns a model recommendation and leaves the call to you, which leaves a clean spot for compression: wrap the provider client that makes that call. Routing stays exactly as it is.

This guide uses Python and the OpenAI SDK. The same pattern works with the Anthropic and Google clients, and in TypeScript with `@caveman-ai/middleware`. The [comparison page](/compare/not-diamond) covers when routing, compression or both make sense.

## What changes and what doesn't

Not Diamond still sees your full conversation, because you keep passing your own history to `select_model`. Caveman Middleware only edits a copy of the outbound request to the chosen model. It shortens eligible tool results, like a 20,000-token JSON response or a long test log, and registers a `caveman_retrieve` tool so the model can read the original back.

Your keys, your provider account and your retry logic stay where they are. User messages, system prompts, errors and images are never touched.

## Step 1: start the Caveman runtime

The runtime is a local process that does the compression and stores the originals. It needs Node.js 22.13 or newer.

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

Confirm it is in compress mode from another shell:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'
```

`mode` has to read `compress`. Plain `caveman start` runs in `record` mode, which measures and changes nothing.

## Step 2: install the Python packages

The middleware is alpha and needs Python 3.13 or newer. Pin the version.

```bash
pip install notdiamond caveman-sdk "caveman-middleware[openai]==0.1.0a1"
```

The OpenAI adapter accepts `openai >=3.10 <4`. Outside that range it declines and passes calls through untouched, so check `pip show openai` if nothing seems to happen.

## Step 3: route, then call through the wrapped client

```python
import json, os
from notdiamond import NotDiamond
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

nd = NotDiamond()  # reads NOT_DIAMOND_API_KEY
openai_client = OpenAI()
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            on_report=lambda r: print("caveman", r.status, r.reason))
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

CANDIDATES = [
    {"provider": "openai", "model": "gpt-5-2025-08-07"},
    {"provider": "openai", "model": "gpt-5-nano-2025-08-07"},
]

def run(history, conversation_id, tools, functions):
    pick = nd.model_router.select_model(messages=history, llm_providers=CANDIDATES, tradeoff="cost")
    model = pick.providers[0].model
    scope = Scope("support", conversation_id, "main", "0")
    loop = with_caveman_openai_tools(openai_client, runtime=runtime, scope=scope,
                                     protocol="openai-chat", tools=tools, functions=functions)
    messages = list(history)
    for _ in range(8):
        reply = loop.client.chat.completions.create(model=model, messages=messages, tools=loop.tools)
        message = reply.choices[0].message
        messages.append(message.model_dump(exclude_none=True))
        if not message.tool_calls:
            return message.content
        for call in message.tool_calls:
            output = loop.functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(output)})
    raise RuntimeError("tool step limit reached")
```

A few details worth knowing:

- `tools` are your native OpenAI function definitions and `functions` maps each tool name to a callable that takes the decoded arguments. The wrapper adds `caveman_retrieve` to both, so your loop runs recovery like any other tool.
- In `notdiamond` 1.7.0 the response has a `providers` list. The quickstart in Not Diamond's docs shows a single `provider` field, so check the version you install.
- `Scope` identifies one conversation. Keep its four values stable across turns, or a handle from an earlier turn won't resolve.
- If your candidates include Anthropic models, wrap an `anthropic.Anthropic` client with `with_caveman_anthropic` and pick the client from `pick.providers[0].provider`.

## Step 4: check that it worked

Run one task that reads something large, such as a big file or a verbose API response.

1. The `on_report` line should print `applied` for the call that carried the large tool result and `reused` on later calls in the same loop. `skipped` comes with a reason, for example `no_candidate` when there was no eligible tool result.
2. Compare the provider's own `usage.prompt_tokens` for the same task with `CAVEMAN_CLIENT_MODE=off` and `CAVEMAN_CLIENT_MODE=compress`. That is your measured number for that task. The runtime's own token counts are inferred estimates, so keep the two apart.
3. Your `history` list is unchanged after the call. The middleware copies it and never writes back.
4. Grade the answer. A smaller request only helps if the task still passes. The [agent evaluations guide](/guides/agent-evaluations) and [cost per task guide](/guides/measure-agent-cost) cover how.

## Coding agents on Not Diamond Code

If you run Claude Code through Not Diamond Code's local proxy, Caveman Middleware isn't the right layer. [Caveman Proxy](/products/caveman-proxy) is the coding-agent version, launched with `caveman claude`. Both run as local proxies next to the agent, and chaining the two proxies hasn't been tested, so trial them one at a time before combining.

Caveman Router, the Caveman option for picking subagent models in Claude Code, is not released yet. [Read about Router](/router).

## Roll back

Set `CAVEMAN_CLIENT_MODE=off` and restart the app. Reports then say `disabled`, nothing goes to the runtime, and calls go straight to the provider. To remove it completely, call `openai_client` directly instead of `loop.client` and drop the import. Your stored conversations never contained compressed text, so there is nothing to migrate back.

## Sources

- [Not Diamond chat router quickstart](https://docs.notdiamond.ai/docs/quickstart-routing)
- [Caveman Middleware, Python](https://docs.caveman.so/docs/sdk/middleware/python)
- [Supported frameworks and versions](https://docs.caveman.so/docs/sdk/middleware/frameworks)
- [Caveman quickstart](https://docs.caveman.so/docs/quickstart)