---
title: "Use Caveman with Martian Gateway: compress tool output"
description: "Martian stays your gateway. You keep the key, the base URL and the model strings. The only change is a wrapper around the SDK client, which shortens large tool results in the outbound copy before the "
canonical: https://caveman.ac/switch/martian
last-updated: 2026-10-04
---

# Use Caveman with Martian Gateway: compress tool output

Martian stays your gateway. You keep the key, the base URL and the model strings. The only change is a wrapper around the SDK client, which shortens large tool results in the outbound copy before the request leaves your process.

This guide uses Python and the OpenAI SDK against Martian's OpenAI-compatible endpoint. The Anthropic-compatible endpoint works the same way with the Anthropic adapter, and TypeScript users get the same adapters from `@caveman-ai/middleware`. For the tradeoffs, read the [comparison](/compare/martian).

## Before you start

Write down what you send today: the base URL (`https://api.withmartian.com/v1`), the model string, and whether it uses the `:optimized` suffix. Pick one fixed model string for this test. If you change the model and add compression at the same time, you won't know which one moved the result.

Pick a task that carries real tool output, like an agent that reads a large file or pulls a verbose API response. A plain chat request has nothing for the middleware to shorten.

## Step 1: start the Caveman runtime

The runtime is a local process that compresses content and stores the originals. It needs Node.js 22.13 or newer.

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

Check the mode from another shell:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'
```

It must say `compress`. Plain `caveman start` runs `record`, which measures and changes nothing.

## Step 2: install the packages

The middleware is alpha and needs Python 3.13 or newer.

```bash
pip install caveman-sdk "caveman-middleware[openai]==0.1.0a1"
```

The OpenAI adapter accepts `openai >=3.10 <4`. Outside that range it declines and passes every call through untouched.

## Step 3: wrap the client you point at Martian

```python
import json, os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

martian = OpenAI(base_url="https://api.withmartian.com/v1", api_key=os.environ["MARTIAN_API_KEY"])
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            on_report=lambda r: print("caveman", r.status, r.reason))
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

def run(history, conversation_id, tools, functions, model="openai/gpt-4.1-nano"):
    scope = Scope("agent", conversation_id, "main", "0")
    loop = with_caveman_openai_tools(martian, runtime=runtime, scope=scope,
                                     protocol="openai-chat", tools=tools, functions=functions)
    messages = list(history)
    for _ in range(8):
        reply = loop.client.chat.completions.create(model=model, messages=messages, tools=loop.tools)
        message = reply.choices[0].message
        messages.append(message.model_dump(exclude_none=True))
        if not message.tool_calls:
            return message.content
        for call in message.tool_calls:
            output = loop.functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(output)})
    raise RuntimeError("tool step limit reached")
```

`tools` are your OpenAI function definitions and `functions` maps each name to a callable that takes the decoded arguments. The wrapper adds `caveman_retrieve` to both, so when the model asks for an original, your loop runs it like any other tool.

The model string is the example from Martian's OpenAI SDK guide. Use the one you recorded. `Scope` names one conversation: keep its four values the same across turns so earlier handles still resolve.

## If you call Martian through LiteLLM

Caveman has a native LiteLLM adapter, `CavemanLiteLLM`, which runs as a LiteLLM callback while LiteLLM stays the only inference hop. Your Martian entries in `model_list` stay as they are. It only compresses on the SDK or Router path when you pass `operator_recovery` and your loop runs the `caveman_retrieve` tool; inside the LiteLLM proxy it forwards the original request, because nothing there can execute recovery. The [LiteLLM guide](/switch/litellm) covers that path, and the [supported frameworks page](https://docs.caveman.so/docs/sdk/middleware/frameworks) lists the accepted LiteLLM versions.

## Coding agents through Martian

Martian's Claude Code setup changes `ANTHROPIC_BASE_URL` and uses a Martian key. [Caveman Proxy](/products/caveman-proxy) also works by changing the agent's base URL, and running it in front of Martian isn't documented yet. Test compression for coding agents on a direct provider route first, and keep the two changes apart.

## Step 4: check that it worked

1. `on_report` prints `applied` on the call that carried the large tool result and `reused` on later calls in the same loop. `skipped` carries a reason, such as `no_candidate` when nothing was eligible or `unsupported_version` when the SDK is outside the accepted range.
2. Run the same task with `CAVEMAN_CLIENT_MODE=off` and `CAVEMAN_CLIENT_MODE=compress` and compare `usage.prompt_tokens` from the response, or the request history in the Martian dashboard. Provider usage is your measured number. Runtime estimates are inferred, so keep them in a separate column.
3. Confirm `history` is unchanged after the run. The middleware edits a copy only.
4. Grade the output. Fewer tokens only count if the task still passes; see [agent evaluations](/guides/agent-evaluations) and [cost per completed task](/guides/measure-agent-cost).

Once compression holds up on one fixed model, test a model change or `:optimized` on its own, with compression held steady.

## Roll back

Set `CAVEMAN_CLIENT_MODE=off` and restart, and calls go straight to Martian with a `disabled` report. To remove it fully, call `martian.chat.completions.create` directly instead of `loop.client` and drop the imports. Nothing about your Martian account, keys or stored conversations changed, so there is nothing else to undo.

## Sources

- [Martian OpenAI SDK integration](https://docs.withmartian.com/integrations/openai-sdk)
- [Martian LiteLLM integration](https://docs.withmartian.com/integrations/litellm)
- [Martian Claude Code integration](https://docs.withmartian.com/integrations/claude-code)
- [Caveman Middleware, Python](https://docs.caveman.so/docs/sdk/middleware/python)
- [Caveman Middleware, supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks)