Use Caveman with Martian Gateway: compress tool output

Martian stays your gateway. You keep the key, the base URL and the model strings. The only change is a wrapper around the SDK client, which shortens large tool results in the outbound copy before the request leaves your process.

This guide uses Python and the OpenAI SDK against Martian's OpenAI-compatible endpoint. The Anthropic-compatible endpoint works the same way with the Anthropic adapter, and TypeScript users get the same adapters from @caveman-ai/middleware. For the tradeoffs, read the comparison.

Before you start

Write down what you send today: the base URL (https://api.withmartian.com/v1), the model string, and whether it uses the :optimized suffix. Pick one fixed model string for this test. If you change the model and add compression at the same time, you won't know which one moved the result.

Pick a task that carries real tool output, like an agent that reads a large file or pulls a verbose API response. A plain chat request has nothing for the middleware to shorten.

Step 1: start the Caveman runtime

The runtime is a local process that compresses content and stores the originals. It needs Node.js 22.13 or newer.

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

Check the mode from another shell:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'

It must say compress. Plain caveman start runs record, which measures and changes nothing.

Step 2: install the packages

The middleware is alpha and needs Python 3.13 or newer.

pip install caveman-sdk "caveman-middleware[openai]==0.1.0a1"

The OpenAI adapter accepts openai >=3.10 <4. Outside that range it declines and passes every call through untouched.

Step 3: wrap the client you point at Martian

import json, os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

martian = OpenAI(base_url="https://api.withmartian.com/v1", api_key=os.environ["MARTIAN_API_KEY"])
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            on_report=lambda r: print("caveman", r.status, r.reason))
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

def run(history, conversation_id, tools, functions, model="openai/gpt-4.1-nano"):
    scope = Scope("agent", conversation_id, "main", "0")
    loop = with_caveman_openai_tools(martian, runtime=runtime, scope=scope,
                                     protocol="openai-chat", tools=tools, functions=functions)
    messages = list(history)
    for _ in range(8):
        reply = loop.client.chat.completions.create(model=model, messages=messages, tools=loop.tools)
        message = reply.choices[0].message
        messages.append(message.model_dump(exclude_none=True))
        if not message.tool_calls:
            return message.content
        for call in message.tool_calls:
            output = loop.functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(output)})
    raise RuntimeError("tool step limit reached")

tools are your OpenAI function definitions and functions maps each name to a callable that takes the decoded arguments. The wrapper adds caveman_retrieve to both, so when the model asks for an original, your loop runs it like any other tool.

The model string is the example from Martian's OpenAI SDK guide. Use the one you recorded. Scope names one conversation: keep its four values the same across turns so earlier handles still resolve.

If you call Martian through LiteLLM

Caveman has a native LiteLLM adapter, CavemanLiteLLM, which runs as a LiteLLM callback while LiteLLM stays the only inference hop. Your Martian entries in model_list stay as they are. It only compresses on the SDK or Router path when you pass operator_recovery and your loop runs the caveman_retrieve tool; inside the LiteLLM proxy it forwards the original request, because nothing there can execute recovery. The LiteLLM guide covers that path, and the supported frameworks page lists the accepted LiteLLM versions.

Coding agents through Martian

Martian's Claude Code setup changes ANTHROPIC_BASE_URL and uses a Martian key. Caveman Proxy also works by changing the agent's base URL, and running it in front of Martian isn't documented yet. Test compression for coding agents on a direct provider route first, and keep the two changes apart.

Step 4: check that it worked

1. on_report prints applied on the call that carried the large tool result and reused on later calls in the same loop. skipped carries a reason, such as no_candidate when nothing was eligible or unsupported_version when the SDK is outside the accepted range.
2. Run the same task with CAVEMAN_CLIENT_MODE=off and CAVEMAN_CLIENT_MODE=compress and compare usage.prompt_tokens from the response, or the request history in the Martian dashboard. Provider usage is your measured number. Runtime estimates are inferred, so keep them in a separate column.
3. Confirm history is unchanged after the run. The middleware edits a copy only.
4. Grade the output. Fewer tokens only count if the task still passes; see agent evaluations and cost per completed task.

Once compression holds up on one fixed model, test a model change or :optimized on its own, with compression held steady.

Roll back

Set CAVEMAN_CLIENT_MODE=off and restart, and calls go straight to Martian with a disabled report. To remove it fully, call martian.chat.completions.create directly instead of loop.client and drop the imports. Nothing about your Martian account, keys or stored conversations changed, so there is nothing else to undo.

Sources

• Martian OpenAI SDK integration
• Martian LiteLLM integration
• Martian Claude Code integration
• Caveman Middleware, Python
• Caveman Middleware, supported frameworks