---
title: "Use Caveman with RouteLLM: route, then compress"
description: "RouteLLM keeps making the strong-or-weak decision. Caveman goes on the call that follows it. The pattern is simple: ask RouteLLM for a model name, then make the request through Caveman's LiteLLM adapt"
canonical: https://caveman.ac/switch/routellm
last-updated: 2026-10-12
---

# Use Caveman with RouteLLM: route, then compress

RouteLLM keeps making the strong-or-weak decision. Caveman goes on the call that follows it. The pattern is simple: ask RouteLLM for a model name, then make the request through Caveman's LiteLLM adapter, which shortens large tool results in the outbound copy.

This fits RouteLLM well because it already speaks LiteLLM: its controller calls `litellm.completion` under the hood, and its model names are LiteLLM model names. For the tradeoffs, read the [comparison](/compare/routellm).

## Step 1: save what you run today

Record the router (`mf` for most people), the threshold, the strong and weak model names and the package version. Keep the calibration data and a set of tasks the router hasn't seen. You'll compare against this exact setup, so change nothing else during the test.

## Step 2: start the Caveman runtime

The runtime is a local process that compresses content and stores the originals. It needs Node.js 22.13 or newer.

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

Check the mode from another shell:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'
```

It must say `compress`. Plain `caveman start` runs `record`, which measures and changes nothing.

## Step 3: install the packages

The middleware is alpha and needs Python 3.13 or newer.

```bash
pip install routellm caveman-sdk "caveman-middleware[litellm]==0.1.0a1"
```

A dry-run install of these three on Python 3.13 resolved to `routellm` 0.2.0 and LiteLLM 1.103, inside the adapter's accepted range of `litellm >=1.100 <2`.

Use the LiteLLM adapter here, not the OpenAI one. RouteLLM pulls in LiteLLM, which currently requires `openai` below 3, and the OpenAI adapter only acts on `openai >=3.10`. Outside its range an adapter passes calls through untouched, so the OpenAI wrapper would quietly do nothing in this environment.

## Step 4: route once, call through Caveman

LiteLLM doesn't run your tools, so your loop carries the `caveman_retrieve` tool and executes it when the model asks. You bind it to the adapter with `operator_recovery`. That binding is required: every Caveman transform needs exact recovery, so without it the adapter forwards the original and reports `recovery_unavailable`.

```python
import json, os
from routellm.controller import Controller
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.litellm import CavemanLiteLLM

router = Controller(routers=["mf"],
                    strong_model=os.environ["STRONG_MODEL"],
                    weak_model=os.environ["WEAK_MODEL"])
THRESHOLD = 0.11593  # use the value you calibrated

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

def run_task(task, conversation_id, messages, tools, functions):
    model = router.route(prompt=task, router="mf", threshold=THRESHOLD)
    scope = Scope("agent", conversation_id, "main", "0")
    recovery = runtime.recovery(scope)
    retrieve_tool = {"type": "function", "function": {
        "name": recovery.name, "description": recovery.description,
        "parameters": recovery.input_schema}}
    handlers = {**functions, recovery.name: recovery.execute}

    with CavemanLiteLLM(runtime=runtime,
                        operator_recovery=lambda s: (recovery, None) if s == scope else None) as caveman:
        for _ in range(8):
            response = caveman.completion(scope=scope, model=model,
                                          messages=messages, tools=[*tools, retrieve_tool])
            message = response.choices[0].message
            messages.append(message.model_dump())
            if not message.tool_calls:
                return message.content
            for call in message.tool_calls:
                result = handlers[call.function.name](json.loads(call.function.arguments))
                messages.append({"role": "tool", "tool_call_id": call.id,
                                 "content": result if isinstance(result, str) else json.dumps(result)})
    raise RuntimeError("tool step limit reached")
```

`tools`, `functions` and `messages` are yours: native OpenAI-format tool definitions, a map from each tool name to a callable that takes the decoded arguments, and the conversation so far. `messages` keeps the full tool output; only the copy sent through LiteLLM is shorter. The Caveman wiring mirrors the tested pattern in the [LiteLLM guide](/switch/litellm).

The router picks the model once per task, from the user's task text. Two reasons. RouteLLM's routers score a single prompt, and the code notes they were trained on first-turn data, so the task text is what they were built to read. And switching models mid-task can throw away a warm prompt cache.

The `mf` router embeds each prompt with OpenAI's `text-embedding-3-small`, so `OPENAI_API_KEY` must be set even if neither routed model is from OpenAI.

`Scope` names one conversation. Keep its four values stable across turns, or a handle the model got earlier won't resolve.

## A setup to avoid

Don't point a Caveman-wrapped OpenAI client at the RouteLLM server instead. The server routes on the last message, which in an agent loop is often the tool result the middleware just shortened, so compression would change the routing decision.

## Step 5: check that it worked

1. `on_report` prints a status per call. `applied` or `reused` means content was shortened. `skipped` comes with a reason: `no_candidate` when the call had no eligible tool result, `recovery_unavailable` when the `operator_recovery` binding or the `caveman_retrieve` tool is missing, `unsupported_version` when LiteLLM is out of range.
2. Run the same held-out tasks twice with the same routed model, once with `CAVEMAN_CLIENT_MODE=off` and once with `CAVEMAN_CLIENT_MODE=compress`, and compare `usage.prompt_tokens` from the responses. That is your measured number. Runtime estimates are inferred, so keep them in their own column.
3. Grade the finished task, not the first reply. A weak model that needs three extra turns isn't cheaper. The [agent evaluations](/guides/agent-evaluations) and [cost per task](/guides/measure-agent-cost) guides cover the scoring.
4. Once compression holds on a fixed route, turn the router back on and recalibrate the threshold on your own queries, as the RouteLLM README recommends.

## Roll back

Set `CAVEMAN_CLIENT_MODE=off` and restart. Calls go straight through LiteLLM with a `disabled` report. To remove Caveman completely, go back to `router.chat.completions.create(model="router-mf-<threshold>", ...)` with your saved threshold and drop the imports. Your stored history never contained compressed text, so there is nothing to clean up.

Caveman Router, Caveman's own routing product for Claude Code subagents, is not released yet. [Read about Router](/router).

## Sources

- [RouteLLM README](https://github.com/lm-sys/RouteLLM)
- [RouteLLM controller source](https://github.com/lm-sys/RouteLLM/blob/main/routellm/controller.py)
- [Caveman Middleware, Python](https://docs.caveman.so/docs/sdk/middleware/python)
- [Caveman Middleware, supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks)