Use Caveman with RouteLLM: route, then compress

RouteLLM keeps making the strong-or-weak decision. Caveman goes on the call that follows it. The pattern is simple: ask RouteLLM for a model name, then make the request through Caveman's LiteLLM adapter, which shortens large tool results in the outbound copy.

This fits RouteLLM well because it already speaks LiteLLM: its controller calls litellm.completion under the hood, and its model names are LiteLLM model names. For the tradeoffs, read the comparison.

Step 1: save what you run today

Record the router (mf for most people), the threshold, the strong and weak model names and the package version. Keep the calibration data and a set of tasks the router hasn't seen. You'll compare against this exact setup, so change nothing else during the test.

Step 2: start the Caveman runtime

The runtime is a local process that compresses content and stores the originals. It needs Node.js 22.13 or newer.

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

Check the mode from another shell:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'

It must say compress. Plain caveman start runs record, which measures and changes nothing.

Step 3: install the packages

The middleware is alpha and needs Python 3.13 or newer.

pip install routellm caveman-sdk "caveman-middleware[litellm]==0.1.0a1"

A dry-run install of these three on Python 3.13 resolved to routellm 0.2.0 and LiteLLM 1.103, inside the adapter's accepted range of litellm >=1.100 <2.

Use the LiteLLM adapter here, not the OpenAI one. RouteLLM pulls in LiteLLM, which currently requires openai below 3, and the OpenAI adapter only acts on openai >=3.10. Outside its range an adapter passes calls through untouched, so the OpenAI wrapper would quietly do nothing in this environment.

Step 4: route once, call through Caveman

LiteLLM doesn't run your tools, so your loop carries the caveman_retrieve tool and executes it when the model asks. You bind it to the adapter with operator_recovery. That binding is required: every Caveman transform needs exact recovery, so without it the adapter forwards the original and reports recovery_unavailable.

import json, os
from routellm.controller import Controller
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.litellm import CavemanLiteLLM

router = Controller(routers=["mf"],
                    strong_model=os.environ["STRONG_MODEL"],
                    weak_model=os.environ["WEAK_MODEL"])
THRESHOLD = 0.11593  # use the value you calibrated

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

def run_task(task, conversation_id, messages, tools, functions):
    model = router.route(prompt=task, router="mf", threshold=THRESHOLD)
    scope = Scope("agent", conversation_id, "main", "0")
    recovery = runtime.recovery(scope)
    retrieve_tool = {"type": "function", "function": {
        "name": recovery.name, "description": recovery.description,
        "parameters": recovery.input_schema}}
    handlers = {**functions, recovery.name: recovery.execute}

    with CavemanLiteLLM(runtime=runtime,
                        operator_recovery=lambda s: (recovery, None) if s == scope else None) as caveman:
        for _ in range(8):
            response = caveman.completion(scope=scope, model=model,
                                          messages=messages, tools=[*tools, retrieve_tool])
            message = response.choices[0].message
            messages.append(message.model_dump())
            if not message.tool_calls:
                return message.content
            for call in message.tool_calls:
                result = handlers[call.function.name](json.loads(call.function.arguments))
                messages.append({"role": "tool", "tool_call_id": call.id,
                                 "content": result if isinstance(result, str) else json.dumps(result)})
    raise RuntimeError("tool step limit reached")

tools, functions and messages are yours: native OpenAI-format tool definitions, a map from each tool name to a callable that takes the decoded arguments, and the conversation so far. messages keeps the full tool output; only the copy sent through LiteLLM is shorter. The Caveman wiring mirrors the tested pattern in the LiteLLM guide.

The router picks the model once per task, from the user's task text. Two reasons. RouteLLM's routers score a single prompt, and the code notes they were trained on first-turn data, so the task text is what they were built to read. And switching models mid-task can throw away a warm prompt cache.

The mf router embeds each prompt with OpenAI's text-embedding-3-small, so OPENAI_API_KEY must be set even if neither routed model is from OpenAI.

Scope names one conversation. Keep its four values stable across turns, or a handle the model got earlier won't resolve.

A setup to avoid

Don't point a Caveman-wrapped OpenAI client at the RouteLLM server instead. The server routes on the last message, which in an agent loop is often the tool result the middleware just shortened, so compression would change the routing decision.

Step 5: check that it worked

1. on_report prints a status per call. applied or reused means content was shortened. skipped comes with a reason: no_candidate when the call had no eligible tool result, recovery_unavailable when the operator_recovery binding or the caveman_retrieve tool is missing, unsupported_version when LiteLLM is out of range.
2. Run the same held-out tasks twice with the same routed model, once with CAVEMAN_CLIENT_MODE=off and once with CAVEMAN_CLIENT_MODE=compress, and compare usage.prompt_tokens from the responses. That is your measured number. Runtime estimates are inferred, so keep them in their own column.
3. Grade the finished task, not the first reply. A weak model that needs three extra turns isn't cheaper. The agent evaluations and cost per task guides cover the scoring.
4. Once compression holds on a fixed route, turn the router back on and recalibrate the threshold on your own queries, as the RouteLLM README recommends.

Roll back

Set CAVEMAN_CLIENT_MODE=off and restart. Calls go straight through LiteLLM with a disabled report. To remove Caveman completely, go back to router.chat.completions.create(model="router-mf-<threshold>", ...) with your saved threshold and drop the imports. Your stored history never contained compressed text, so there is nothing to clean up.

Caveman Router, Caveman's own routing product for Claude Code subagents, is not released yet. Read about Router.

Sources

• RouteLLM README
• RouteLLM controller source
• Caveman Middleware, Python
• Caveman Middleware, supported frameworks