Use Caveman with Not Diamond: add token compression

You don't have to move off Not Diamond to try Caveman. Not Diamond's chat router returns a model recommendation and leaves the call to you, which leaves a clean spot for compression: wrap the provider client that makes that call. Routing stays exactly as it is.

This guide uses Python and the OpenAI SDK. The same pattern works with the Anthropic and Google clients, and in TypeScript with @caveman-ai/middleware. The comparison page covers when routing, compression or both make sense.

What changes and what doesn't

Not Diamond still sees your full conversation, because you keep passing your own history to select_model. Caveman Middleware only edits a copy of the outbound request to the chosen model. It shortens eligible tool results, like a 20,000-token JSON response or a long test log, and registers a caveman_retrieve tool so the model can read the original back.

Your keys, your provider account and your retry logic stay where they are. User messages, system prompts, errors and images are never touched.

Step 1: start the Caveman runtime

The runtime is a local process that does the compression and stores the originals. It needs Node.js 22.13 or newer.

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

Confirm it is in compress mode from another shell:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'

mode has to read compress. Plain caveman start runs in record mode, which measures and changes nothing.

Step 2: install the Python packages

The middleware is alpha and needs Python 3.13 or newer. Pin the version.

pip install notdiamond caveman-sdk "caveman-middleware[openai]==0.1.0a1"

The OpenAI adapter accepts openai >=3.10 <4. Outside that range it declines and passes calls through untouched, so check pip show openai if nothing seems to happen.

Step 3: route, then call through the wrapped client

import json, os
from notdiamond import NotDiamond
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

nd = NotDiamond()  # reads NOT_DIAMOND_API_KEY
openai_client = OpenAI()
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            on_report=lambda r: print("caveman", r.status, r.reason))
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

CANDIDATES = [
    {"provider": "openai", "model": "gpt-5-2025-08-07"},
    {"provider": "openai", "model": "gpt-5-nano-2025-08-07"},
]

def run(history, conversation_id, tools, functions):
    pick = nd.model_router.select_model(messages=history, llm_providers=CANDIDATES, tradeoff="cost")
    model = pick.providers[0].model
    scope = Scope("support", conversation_id, "main", "0")
    loop = with_caveman_openai_tools(openai_client, runtime=runtime, scope=scope,
                                     protocol="openai-chat", tools=tools, functions=functions)
    messages = list(history)
    for _ in range(8):
        reply = loop.client.chat.completions.create(model=model, messages=messages, tools=loop.tools)
        message = reply.choices[0].message
        messages.append(message.model_dump(exclude_none=True))
        if not message.tool_calls:
            return message.content
        for call in message.tool_calls:
            output = loop.functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(output)})
    raise RuntimeError("tool step limit reached")

A few details worth knowing:

• tools are your native OpenAI function definitions and functions maps each tool name to a callable that takes the decoded arguments. The wrapper adds caveman_retrieve to both, so your loop runs recovery like any other tool.
• In notdiamond 1.7.0 the response has a providers list. The quickstart in Not Diamond's docs shows a single provider field, so check the version you install.
• Scope identifies one conversation. Keep its four values stable across turns, or a handle from an earlier turn won't resolve.
• If your candidates include Anthropic models, wrap an anthropic.Anthropic client with with_caveman_anthropic and pick the client from pick.providers[0].provider.

Step 4: check that it worked

Run one task that reads something large, such as a big file or a verbose API response.

1. The on_report line should print applied for the call that carried the large tool result and reused on later calls in the same loop. skipped comes with a reason, for example no_candidate when there was no eligible tool result.
2. Compare the provider's own usage.prompt_tokens for the same task with CAVEMAN_CLIENT_MODE=off and CAVEMAN_CLIENT_MODE=compress. That is your measured number for that task. The runtime's own token counts are inferred estimates, so keep the two apart.
3. Your history list is unchanged after the call. The middleware copies it and never writes back.
4. Grade the answer. A smaller request only helps if the task still passes. The agent evaluations guide and cost per task guide cover how.

Coding agents on Not Diamond Code

If you run Claude Code through Not Diamond Code's local proxy, Caveman Middleware isn't the right layer. Caveman Proxy is the coding-agent version, launched with caveman claude. Both run as local proxies next to the agent, and chaining the two proxies hasn't been tested, so trial them one at a time before combining.

Caveman Router, the Caveman option for picking subagent models in Claude Code, is not released yet. Read about Router.

Roll back

Set CAVEMAN_CLIENT_MODE=off and restart the app. Reports then say disabled, nothing goes to the runtime, and calls go straight to the provider. To remove it completely, call openai_client directly instead of loop.client and drop the import. Your stored conversations never contained compressed text, so there is nothing to migrate back.

Sources

• Not Diamond chat router quickstart
• Caveman Middleware, Python
• Supported frameworks and versions
• Caveman quickstart