How to add Caveman to LiteLLM: SDK, Router and proxy

This guide adds Caveman Middleware to a Python app that already calls LiteLLM. Nothing moves off LiteLLM: it still makes every provider call, with your keys, model aliases, retries and fallbacks. Caveman shortens large tool results in the outgoing copy of each request. Start with one agent, check it, then decide. For the why, read the LiteLLM comparison.

What you need before you start

• Python 3.13 or newer and litellm >=1.100 <2.
• Node.js 22.13 or newer for the caveman CLI, which installs and runs the local runtime.
• One agent that calls tools returning big text: API responses, logs, file reads, test output.
• A task with a known answer, so you can check quality as well as tokens.

Step 1: Start the Caveman runtime

The runtime is a local process that does the compression and stores the originals on disk. No Caveman account is needed.

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

It listens on 127.0.0.1:8787. Plain caveman start runs in record mode, which changes nothing, so keep the variable. From another shell:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'

You want "mode": "compress" and "recovery": true.

Step 2: Install the LiteLLM adapter

pip install caveman-sdk "caveman-middleware[litellm]==0.1.0a1"

The middleware is alpha, so pin the version.

Step 3: Wrap one agent's LiteLLM calls

CavemanLiteLLM takes the same arguments as litellm.completion(), plus a scope that names the conversation. Because LiteLLM does not run your tools, your loop also carries the caveman_retrieve tool and runs it when the model asks. That is what lets Caveman shorten a result safely. Without it, the adapter forwards the original and reports recovery_unavailable.

import json, os
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.litellm import CavemanLiteLLM

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()

scope = Scope("orders-agent", conversation_id)
recovery = runtime.recovery(scope)
retrieve_tool = {"type": "function", "function": {
    "name": recovery.name, "description": recovery.description,
    "parameters": recovery.input_schema}}
functions = {"list_orders": list_orders, recovery.name: recovery.execute}

with CavemanLiteLLM(runtime=runtime,
                    operator_recovery=lambda s: (recovery, None) if s == scope else None) as caveman:
    for _ in range(8):
        response = caveman.completion(scope=scope, model="openai/gpt-5.5",
                                      messages=messages, tools=[*tools, retrieve_tool])
        message = response.choices[0].message
        messages.append(message.model_dump())
        if not message.tool_calls:
            break
        for call in message.tool_calls:
            result = functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": result if isinstance(result, str) else json.dumps(result)})

tools, messages and list_orders are yours. Keep conversation_id stable across turns: a handle the model got earlier only resolves under the same scope. messages keeps the full tool output; only the copy sent through LiteLLM is shorter. Use acompletion in async code, same arguments.

Using a LiteLLM Router

Pass your Router as the client. Aliases, load balancing and fallbacks stay with LiteLLM, and Caveman runs after the Router picks a deployment.

router = litellm.Router(model_list=model_list)

with CavemanLiteLLM(runtime=runtime, client=router,
                    operator_recovery=lambda s: (recovery, None) if s == scope else None) as caveman:
    response = await caveman.acompletion(scope=scope, model="fast",
                                         messages=messages, tools=[*tools, retrieve_tool])

Synchronous Router calls are supported for OpenAI and Anthropic deployments. The async path covers the rest.

Using the LiteLLM proxy

The adapter also runs inside the LiteLLM proxy as a callback. Create custom_callbacks.py next to your proxy config:

from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.litellm import CavemanLiteLLM

runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode="compress", deadline_ms=500)

def proxy_scope(key, data):
    # key is LiteLLM's authenticated UserAPIKeyAuth, never the request body
    return Scope(key.team_id or "default", key.token)

caveman = CavemanLiteLLM(runtime=runtime, proxy_scope=proxy_scope)

Then point LiteLLM at the instance:

litellm_settings:
  callbacks: custom_callbacks.caveman

Be clear about what this does today. Every request gets a Caveman decision scoped to the calling key, but the proxy has no tool loop, so the callback forwards originals with recovery_unavailable until your deployment binds a recovery executor through operator_recovery and callers send the caveman_retrieve tool. Use the proxy hook to see which traffic qualifies. Do the compressing on the SDK or Router side first.

How to check it worked

Run the same task twice: once with CAVEMAN_CLIENT_MODE=off, once with CAVEMAN_CLIENT_MODE=compress. The snippet passes that variable as mode on the runtime client.

1. The reports. on_report=print logs one line per call. You want status of applied or reused on calls that carry big tool output, with a transform such as caveman.engine.json.v1. skipped with no_candidate just means nothing was eligible.
2. Provider usage. Compare response.usage.prompt_tokens between the two runs. That number comes from your provider, so it is the one to trust.
3. The answer. Check the task result is still right, and count how often the model called caveman_retrieve. A smaller request that needs two extra turns is not a win.

The measurement guide covers repeat runs, cache hits and retries.

How to roll back

Set CAVEMAN_CLIENT_MODE=off and restart. The snippet passes it as mode on MiddlewareRuntime and skips ready(), which raises in off mode, so the adapter passes every call straight to LiteLLM without touching the call site. To remove it fully, call litellm.completion() or your Router directly again and drop the retrieve tool. On the proxy, delete the callbacks line and restart. Stop the runtime with Ctrl+C when nothing uses it.

Sources: Caveman Python middleware, supported frameworks, LiteLLM custom callbacks, LiteLLM Router.