Caveman vs LiteLLM: tool output compression in LiteLLM

Keep LiteLLM. Caveman is built to run inside it. The Python package caveman-middleware ships a native LiteLLM adapter, a CustomLogger callback that shrinks large tool results in the request LiteLLM is about to send. LiteLLM stays the only hop to your provider and keeps its retries, fallbacks, streaming and spend tracking. Caveman changes one thing: how many tokens of tool output ride along on every call.

That part adds up fast in an agent loop. A tool returns a 7,000 token JSON response on turn two, and LiteLLM forwards it again on turns three, four and five, because it is part of the conversation. Caveman sends a short view instead and keeps the full text one tool call away.

What is LiteLLM?

LiteLLM is an open-source library and gateway that lets you call 100+ LLM providers through one OpenAI-style interface. It comes in two shapes. The Python SDK translates litellm.completion() calls for OpenAI, Anthropic, Bedrock, Vertex and the rest, with retries and fallbacks across deployments. The proxy server turns that into a shared AI gateway for a platform team: virtual keys, per-key, per-team and per-user budgets, rate limits, an admin UI, and logging callbacks into tools like Langfuse and MLflow.

It is MIT licensed outside its enterprise/ directory, has roughly 60,000 GitHub stars, and the company reports more than 240 million Docker pulls. Enterprise adds SSO, SCIM, audit logs and RBAC, with pricing through sales. LiteLLM also has an opt-in beta Rust gateway core that handles provider translation on some routes, while the Python proxy still runs auth, routing and callbacks.

If your team already routes through LiteLLM, it is doing its job. Nothing here asks you to move off it.

Does Caveman work with LiteLLM?

Yes, natively. caveman_middleware.litellm.CavemanLiteLLM subclasses LiteLLM's own CustomLogger and accepts litellm >=1.100 <2. It covers three placements:

• The LiteLLM SDK. caveman.completion() and caveman.acompletion() take the same arguments as litellm.completion(), plus a scope.
• A LiteLLM Router. Pass client=router and your model aliases, load balancing and fallbacks keep working. Caveman hooks in after the Router picks a deployment.
• The LiteLLM proxy. The same class has proxy hooks. A proxy_scope function builds the Caveman scope from LiteLLM's authenticated key, never from the request body.

from caveman_middleware.litellm import CavemanLiteLLM

with CavemanLiteLLM(runtime=runtime, operator_recovery=bind_recovery) as caveman:
    response = caveman.completion(scope=scope, model="openai/gpt-5.5",
                                  messages=messages, tools=[*tools, retrieve_tool])

On each call the callback copies the outbound request, sends eligible tool-result text to a local Caveman runtime, and swaps in the shorter version. Your messages list keeps the original bytes. User messages, system prompts, reasoning, errors and image or PDF bytes are never touched.

One detail matters, because it decides whether anything gets compressed at all. Caveman only replaces a tool result when the model can read the original back. LiteLLM calls the model and returns; it does not run your tools. So your loop registers the caveman_retrieve tool and runs it when the model asks, which takes about four lines. Without it the callback forwards the original request and reports recovery_unavailable. The switching guide has the full loop.

How many tokens does Caveman save on LiteLLM traffic?

It depends on the shape of your tool output, and the published numbers say so plainly. Caveman ran 13 synthetic tool results through the real local middleware runtime and counted tokens with o200k_base. These are inferred counts from one sample each, not a provider bill:

• An API response with 160 repeated JSON records: 7,202 tokens before, 403 after.
• A worker log with one timeout in 160 health checks: 4,316 before, 279 after.
• ANSI test output with one failing assertion: 3,837 before, 176 after.
• TypeScript source with repeated handlers: 9,280 before, 3,243 after.
• A git diff, rendered HTML and repository search results: unchanged.

Each compressed request also carries a 62 token definition for the recovery tool. The middleware page links the full measurements and the script that reproduces them. Task quality, later recovery calls and cache effects were not measured, so run your own task set before you count on a number.

LiteLLM vs Caveman: feature comparison

How is this different from litellm.compress()?

LiteLLM already has compression options, and they are worth knowing about. litellm.compress() scores older conversation history for relevance to the latest query (optionally with embeddings), replaces low-relevance content with stubs, and hands you a litellm_content_retrieve tool plus a cache so the model can ask for the full text. On the proxy, LiteLLM also documents Headroom and Compresr as pre-call compression options. Compresr targets tool messages too, through its own hosted API.

Caveman works at a different level. It reads each tool result by type, so a log keeps its first and last lines plus the one error, a JSON array keeps the odd record out plus a summary of the ones it elided, and test output keeps the failing assertion. Compression runs in a local runtime, and recovery is exact and scoped to one conversation. If relevance ranking of old history is your main problem, compress() is a sensible first try. If big tool payloads are, Caveman targets that directly. For a head-to-head with Headroom, see Caveman vs Headroom.

When should you use LiteLLM alone?

Use LiteLLM alone when your prompts are already small. A retrieval app that sends 900 tokens per question has nothing worth shrinking, and a runtime call on the path only adds latency.

Use it alone when the problem is coverage, keys or budgets. A new model shipped today, finance wants spend per team, someone needs one key revoked: that is LiteLLM's work, and Caveman does none of it.

And stick with LiteLLM alone for now if you need a settled, production-hardened stack today. Caveman Middleware is alpha. Caveman's own hosted gateway, Caveman Platform, is not generally available yet.

How to add Caveman to LiteLLM

Start a local runtime with CAVEMAN_MODE=compress caveman start, install caveman-middleware[litellm] on Python 3.13 or newer, and wrap one agent's calls with CavemanLiteLLM. Leave every other caller alone. The LiteLLM switching guide walks through the SDK, Router and proxy setups, how to check it worked, and how to turn it off.

Running a coding agent like Claude Code against LiteLLM instead? Caveman Proxy compresses at the base URL and needs no code change.

FAQ

Does Caveman replace LiteLLM? No. LiteLLM still translates the call, picks the deployment, retries, and tracks spend. Caveman only shortens eligible tool results in the outbound copy.

Does it work with the LiteLLM proxy? The adapter has proxy hooks and scopes each request by the authenticated virtual key. Compression there needs a recovery executor your deployment provides, so most teams start with the SDK or Router path.

What does LiteLLM log after Caveman runs? Callbacks that read the request after Caveman see the shortened copy. Your application history keeps the originals, and the adapter turns off its own message logging.

Is the recovered text exact? Yes. Every published sample that was compressed recovered byte for byte, checked by SHA-256.

What does it cost? The client and adapters are MIT. The local runtime is under BSL terms and needs no Caveman account.

Sources

• LiteLLM documentation
• LiteLLM on GitHub and license
• LiteLLM homepage
• LiteLLM prompt compression, Headroom and Compresr on the proxy
• LiteLLM Rust gateway beta
• Caveman Middleware for Python and supported frameworks
• Measure an agent optimization