---
title: "How to add Caveman to LiteLLM: SDK, Router and proxy"
description: "This guide adds Caveman Middleware to a Python app that already calls LiteLLM. Nothing moves off LiteLLM: it still makes every provider call, with your keys, model aliases, retries and fallbacks. Cave"
canonical: https://caveman.ac/switch/litellm
last-updated: 2026-10-04
---

# How to add Caveman to LiteLLM: SDK, Router and proxy

This guide adds Caveman Middleware to a Python app that already calls LiteLLM. Nothing moves off LiteLLM: it still makes every provider call, with your keys, model aliases, retries and fallbacks. Caveman shortens large tool results in the outgoing copy of each request. Start with one agent, check it, then decide. For the why, read the [LiteLLM comparison](/compare/litellm).

## What you need before you start

- Python 3.13 or newer and `litellm >=1.100 <2`.
- Node.js 22.13 or newer for the `caveman` CLI, which installs and runs the local runtime.
- One agent that calls tools returning big text: API responses, logs, file reads, test output.
- A task with a known answer, so you can check quality as well as tokens.

## Step 1: Start the Caveman runtime

The runtime is a local process that does the compression and stores the originals on disk. No Caveman account is needed.

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

It listens on `127.0.0.1:8787`. Plain `caveman start` runs in `record` mode, which changes nothing, so keep the variable. From another shell:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'
```

You want `"mode": "compress"` and `"recovery": true`.

## Step 2: Install the LiteLLM adapter

```bash
pip install caveman-sdk "caveman-middleware[litellm]==0.1.0a1"
```

The middleware is alpha, so pin the version.

## Step 3: Wrap one agent's LiteLLM calls

`CavemanLiteLLM` takes the same arguments as `litellm.completion()`, plus a `scope` that names the conversation. Because LiteLLM does not run your tools, your loop also carries the `caveman_retrieve` tool and runs it when the model asks. That is what lets Caveman shorten a result safely. Without it, the adapter forwards the original and reports `recovery_unavailable`.

```python
import json, os
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.litellm import CavemanLiteLLM

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()

scope = Scope("orders-agent", conversation_id)
recovery = runtime.recovery(scope)
retrieve_tool = {"type": "function", "function": {
    "name": recovery.name, "description": recovery.description,
    "parameters": recovery.input_schema}}
functions = {"list_orders": list_orders, recovery.name: recovery.execute}

with CavemanLiteLLM(runtime=runtime,
                    operator_recovery=lambda s: (recovery, None) if s == scope else None) as caveman:
    for _ in range(8):
        response = caveman.completion(scope=scope, model="openai/gpt-5.5",
                                      messages=messages, tools=[*tools, retrieve_tool])
        message = response.choices[0].message
        messages.append(message.model_dump())
        if not message.tool_calls:
            break
        for call in message.tool_calls:
            result = functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": result if isinstance(result, str) else json.dumps(result)})
```

`tools`, `messages` and `list_orders` are yours. Keep `conversation_id` stable across turns: a handle the model got earlier only resolves under the same scope. `messages` keeps the full tool output; only the copy sent through LiteLLM is shorter. Use `acompletion` in async code, same arguments.

## Using a LiteLLM Router

Pass your Router as the client. Aliases, load balancing and fallbacks stay with LiteLLM, and Caveman runs after the Router picks a deployment.

```python
router = litellm.Router(model_list=model_list)

with CavemanLiteLLM(runtime=runtime, client=router,
                    operator_recovery=lambda s: (recovery, None) if s == scope else None) as caveman:
    response = await caveman.acompletion(scope=scope, model="fast",
                                         messages=messages, tools=[*tools, retrieve_tool])
```

Synchronous Router calls are supported for OpenAI and Anthropic deployments. The async path covers the rest.

## Using the LiteLLM proxy

The adapter also runs inside the LiteLLM proxy as a callback. Create `custom_callbacks.py` next to your proxy config:

```python
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.litellm import CavemanLiteLLM

runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode="compress", deadline_ms=500)

def proxy_scope(key, data):
    # key is LiteLLM's authenticated UserAPIKeyAuth, never the request body
    return Scope(key.team_id or "default", key.token)

caveman = CavemanLiteLLM(runtime=runtime, proxy_scope=proxy_scope)
```

Then point LiteLLM at the instance:

```yaml
litellm_settings:
  callbacks: custom_callbacks.caveman
```

Be clear about what this does today. Every request gets a Caveman decision scoped to the calling key, but the proxy has no tool loop, so the callback forwards originals with `recovery_unavailable` until your deployment binds a recovery executor through `operator_recovery` and callers send the `caveman_retrieve` tool. Use the proxy hook to see which traffic qualifies. Do the compressing on the SDK or Router side first.

## How to check it worked

Run the same task twice: once with `CAVEMAN_CLIENT_MODE=off`, once with `CAVEMAN_CLIENT_MODE=compress`. The snippet passes that variable as `mode` on the runtime client.

1. **The reports.** `on_report=print` logs one line per call. You want `status` of `applied` or `reused` on calls that carry big tool output, with a transform such as `caveman.engine.json.v1`. `skipped` with `no_candidate` just means nothing was eligible.
2. **Provider usage.** Compare `response.usage.prompt_tokens` between the two runs. That number comes from your provider, so it is the one to trust.
3. **The answer.** Check the task result is still right, and count how often the model called `caveman_retrieve`. A smaller request that needs two extra turns is not a win.

The [measurement guide](/guides/measure-agent-cost) covers repeat runs, cache hits and retries.

## How to roll back

Set `CAVEMAN_CLIENT_MODE=off` and restart. The snippet passes it as `mode` on `MiddlewareRuntime` and skips `ready()`, which raises in off mode, so the adapter passes every call straight to LiteLLM without touching the call site. To remove it fully, call `litellm.completion()` or your Router directly again and drop the retrieve tool. On the proxy, delete the `callbacks` line and restart. Stop the runtime with Ctrl+C when nothing uses it.

Sources: [Caveman Python middleware](https://docs.caveman.so/docs/sdk/middleware/python), [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks), [LiteLLM custom callbacks](https://docs.litellm.ai/docs/proxy/call_hooks), [LiteLLM Router](https://docs.litellm.ai/docs/routing).