---
title: "Caveman vs LiteLLM: tool output compression in LiteLLM"
description: "Keep LiteLLM. Caveman is built to run inside it. The Python package caveman-middleware ships a native LiteLLM adapter, a CustomLogger callback that shrinks large tool results in the request LiteLLM is"
canonical: https://caveman.ac/compare/litellm
last-updated: 2026-10-04
---

# Caveman vs LiteLLM: tool output compression in LiteLLM

Keep LiteLLM. Caveman is built to run inside it. The Python package `caveman-middleware` ships a native LiteLLM adapter, a `CustomLogger` callback that shrinks large tool results in the request LiteLLM is about to send. LiteLLM stays the only hop to your provider and keeps its retries, fallbacks, streaming and spend tracking. Caveman changes one thing: how many tokens of tool output ride along on every call.

That part adds up fast in an agent loop. A tool returns a 7,000 token JSON response on turn two, and LiteLLM forwards it again on turns three, four and five, because it is part of the conversation. Caveman sends a short view instead and keeps the full text one tool call away.

## What is LiteLLM?

LiteLLM is an open-source library and gateway that lets you call 100+ LLM providers through one OpenAI-style interface. It comes in two shapes. The Python SDK translates `litellm.completion()` calls for OpenAI, Anthropic, Bedrock, Vertex and the rest, with retries and fallbacks across deployments. The proxy server turns that into a shared AI gateway for a platform team: virtual keys, per-key, per-team and per-user budgets, rate limits, an admin UI, and logging callbacks into tools like Langfuse and MLflow.

It is MIT licensed outside its `enterprise/` directory, has roughly 60,000 GitHub stars, and the company reports more than 240 million Docker pulls. Enterprise adds SSO, SCIM, audit logs and RBAC, with pricing through sales. LiteLLM also has an opt-in beta Rust gateway core that handles provider translation on some routes, while the Python proxy still runs auth, routing and callbacks.

If your team already routes through LiteLLM, it is doing its job. Nothing here asks you to move off it.

## Does Caveman work with LiteLLM?

Yes, natively. `caveman_middleware.litellm.CavemanLiteLLM` subclasses LiteLLM's own `CustomLogger` and accepts `litellm >=1.100 <2`. It covers three placements:

- **The LiteLLM SDK.** `caveman.completion()` and `caveman.acompletion()` take the same arguments as `litellm.completion()`, plus a `scope`.
- **A LiteLLM Router.** Pass `client=router` and your model aliases, load balancing and fallbacks keep working. Caveman hooks in after the Router picks a deployment.
- **The LiteLLM proxy.** The same class has proxy hooks. A `proxy_scope` function builds the Caveman scope from LiteLLM's authenticated key, never from the request body.

```python
from caveman_middleware.litellm import CavemanLiteLLM

with CavemanLiteLLM(runtime=runtime, operator_recovery=bind_recovery) as caveman:
    response = caveman.completion(scope=scope, model="openai/gpt-5.5",
                                  messages=messages, tools=[*tools, retrieve_tool])
```

On each call the callback copies the outbound request, sends eligible tool-result text to a local Caveman runtime, and swaps in the shorter version. Your `messages` list keeps the original bytes. User messages, system prompts, reasoning, errors and image or PDF bytes are never touched.

One detail matters, because it decides whether anything gets compressed at all. Caveman only replaces a tool result when the model can read the original back. LiteLLM calls the model and returns; it does not run your tools. So your loop registers the `caveman_retrieve` tool and runs it when the model asks, which takes about four lines. Without it the callback forwards the original request and reports `recovery_unavailable`. The [switching guide](/switch/litellm) has the full loop.

## How many tokens does Caveman save on LiteLLM traffic?

It depends on the shape of your tool output, and the published numbers say so plainly. Caveman ran 13 synthetic tool results through the real local middleware runtime and counted tokens with `o200k_base`. These are inferred counts from one sample each, not a provider bill:

- An API response with 160 repeated JSON records: 7,202 tokens before, 403 after.
- A worker log with one timeout in 160 health checks: 4,316 before, 279 after.
- ANSI test output with one failing assertion: 3,837 before, 176 after.
- TypeScript source with repeated handlers: 9,280 before, 3,243 after.
- A git diff, rendered HTML and repository search results: unchanged.

Each compressed request also carries a 62 token definition for the recovery tool. The [middleware page](/products/caveman-middleware) links the full measurements and the script that reproduces them. Task quality, later recovery calls and cache effects were not measured, so run your own task set before you count on a number.

## LiteLLM vs Caveman: feature comparison

| | LiteLLM | Caveman Middleware |
| --- | --- | --- |
| Main job | Call 100+ providers through one API | Shrink tool results before the call |
| Calls the model | Yes, SDK or proxy | No, LiteLLM still does |
| Retries, fallbacks, load balancing | Built in | Left to LiteLLM |
| Virtual keys, budgets, rate limits | Proxy feature | None |
| Cost and spend tracking | Built in | None; reports compression decisions |
| Tool-result compression | Via `compress()` or partner guardrails | Core feature, per content type |
| Originals after compression | `compress()` keeps a stub cache | Stored locally, exact recovery by handle |
| Runs where | Your process or your gateway | Local runtime next to your app |
| License | MIT, enterprise dir separate | MIT client, BSL runtime |
| Status | Mature, widely deployed | Alpha |

## How is this different from litellm.compress()?

LiteLLM already has compression options, and they are worth knowing about. `litellm.compress()` scores older conversation history for relevance to the latest query (optionally with embeddings), replaces low-relevance content with stubs, and hands you a `litellm_content_retrieve` tool plus a cache so the model can ask for the full text. On the proxy, LiteLLM also documents Headroom and Compresr as pre-call compression options. Compresr targets tool messages too, through its own hosted API.

Caveman works at a different level. It reads each tool result by type, so a log keeps its first and last lines plus the one error, a JSON array keeps the odd record out plus a summary of the ones it elided, and test output keeps the failing assertion. Compression runs in a local runtime, and recovery is exact and scoped to one conversation. If relevance ranking of old history is your main problem, `compress()` is a sensible first try. If big tool payloads are, Caveman targets that directly. For a head-to-head with Headroom, see [Caveman vs Headroom](/compare/headroom).

## When should you use LiteLLM alone?

Use LiteLLM alone when your prompts are already small. A retrieval app that sends 900 tokens per question has nothing worth shrinking, and a runtime call on the path only adds latency.

Use it alone when the problem is coverage, keys or budgets. A new model shipped today, finance wants spend per team, someone needs one key revoked: that is LiteLLM's work, and Caveman does none of it.

And stick with LiteLLM alone for now if you need a settled, production-hardened stack today. Caveman Middleware is alpha. Caveman's own hosted gateway, Caveman Platform, is not generally available yet.

## How to add Caveman to LiteLLM

Start a local runtime with `CAVEMAN_MODE=compress caveman start`, install `caveman-middleware[litellm]` on Python 3.13 or newer, and wrap one agent's calls with `CavemanLiteLLM`. Leave every other caller alone. The [LiteLLM switching guide](/switch/litellm) walks through the SDK, Router and proxy setups, how to check it worked, and how to turn it off.

Running a coding agent like Claude Code against LiteLLM instead? [Caveman Proxy](/products/caveman-proxy) compresses at the base URL and needs no code change.

## FAQ

**Does Caveman replace LiteLLM?**
No. LiteLLM still translates the call, picks the deployment, retries, and tracks spend. Caveman only shortens eligible tool results in the outbound copy.

**Does it work with the LiteLLM proxy?**
The adapter has proxy hooks and scopes each request by the authenticated virtual key. Compression there needs a recovery executor your deployment provides, so most teams start with the SDK or Router path.

**What does LiteLLM log after Caveman runs?**
Callbacks that read the request after Caveman see the shortened copy. Your application history keeps the originals, and the adapter turns off its own message logging.

**Is the recovered text exact?**
Yes. Every published sample that was compressed recovered byte for byte, checked by SHA-256.

**What does it cost?**
The client and adapters are MIT. The local runtime is under BSL terms and needs no Caveman account.

## Sources

- [LiteLLM documentation](https://docs.litellm.ai/docs/)
- [LiteLLM on GitHub](https://github.com/BerriAI/litellm) and [license](https://github.com/BerriAI/litellm/blob/main/LICENSE)
- [LiteLLM homepage](https://www.litellm.ai/)
- [LiteLLM prompt compression](https://docs.litellm.ai/docs/completion/prompt_compression), [Headroom](https://docs.litellm.ai/docs/proxy/headroom) and [Compresr](https://docs.litellm.ai/docs/proxy/guardrails/compresr) on the proxy
- [LiteLLM Rust gateway beta](https://docs.litellm.ai/docs/proxy/rust_gateway)
- [Caveman Middleware for Python](https://docs.caveman.so/docs/sdk/middleware/python) and [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks)
- [Measure an agent optimization](/guides/measure-agent-cost)