How Caveman works with LiteLLM

Running LiteLLM for your team? Connect it to Caveman Platform through OpenTelemetry. Keep your existing gateway and provider keys. Bring traces, reported usage, and optional retained messages into Caveman for debugging, evaluations, and investigations. The local proxy options below are separate ways to add compression; they are not required for the platform connection.

LiteLLM is a gateway. It holds provider keys, maps a hundred provider APIs onto one shape, enforces budgets per virtual key, and retries. Caveman is an efficiency layer. It shortens what an agent writes, shrinks what it rereads, and measures the difference. The two do not compete for the same job, and the seams between them are small.

Start with the platform connection if you run LiteLLM for a team. The local skill and compression options below work independently. Platform traces and model routing use a Caveman project key.

Connect Caveman Platform through native traces

Your inference path stays app → LiteLLM → provider. Enable LiteLLM's native OpenTelemetry exporter, or add Caveman to your existing collector's export pipeline. Caveman receives the trace tree, models, token counts, timing, errors, and reported cost without becoming another inference hop.

Opt into message and tool-content capture when you need retained evidence for evaluation datasets, scenarios, and investigations. Capture requires storage and purpose consent; redaction, encryption, access controls, and retention apply. Metadata export works without content capture.

Connect LiteLLM to Caveman Platform for copyable settings and a live connection check. The setup is tested with LiteLLM 1.96.0, including streamed requests and provider errors. Trace export alone does not apply compression or prove savings.

What LiteLLM and the local Caveman tools own

For local compression, the optional Caveman proxy sits between the agent and LiteLLM. The platform connection above sends telemetry separately.

1. The skill needs nothing from the gateway

The Caveman skill is a rule file the agent loads. It changes how Claude, Codex or Gemini writes, so it is upstream of any gateway. Install it in the agent and LiteLLM never knows.

npx skills add JuliusBrussee/caveman

On ten measured coding prompts it cut output tokens by 65 percent. It adds roughly 1,000 to 1,500 input tokens per turn for its own rules, so on already terse sessions it can cost more than it saves. That trade does not change when a gateway is in the path.

2. Per call: point api_base at the local proxy

If your Python code calls litellm.completion directly, the smallest change is one argument. Start the local proxy, then aim the call at it.

npm install -g @caveman-ai/cli && caveman setup --install
caveman tools config set think.mode compress   # bare start defaults to record
caveman start                                  # listens on 127.0.0.1:8787

import os
import litellm

res = litellm.completion(
    model="openai/gpt-5.5",
    api_base="http://127.0.0.1:8787/openai/v1",
    api_key=os.environ["OPENAI_API_KEY"],
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)

For Anthropic models the mount is /anthropic/v1, for Gemini /gemini/v1beta. The providers page lists every route. The key travels on the request and the proxy forwards it. Nothing is stored except the originals of compressed payloads, in a local SQLite file, so the agent can recover exact bytes later.

The CLI prints this snippet for you, with the right mount for your provider:

caveman snippets litellm

3. Fleet-wide: put the proxy in config.yaml

If you run the LiteLLM proxy server, every client behind it inherits whatever you set on a deployment. Point the deployment at Caveman instead of the provider and compression applies to all of them.

model_list:
  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_base: http://127.0.0.1:8787/openai/v1
      api_key: os.environ/OPENAI_API_KEY

Two things to know before you do this on a shared host.

The Caveman proxy listens on loopback only. Run it on the same machine as the LiteLLM proxy, or in the same pod. It refuses non-local upstreams unless you add an exact host to its allowlist, which is a security property, not a limitation to work around.

Recovery has to live somewhere. When the agent runs on the same machine, it recovers originals through the Caveman MCP tool. A client on another machine behind LiteLLM has no such tool, so the proxy compresses only where it can handle recovery itself, which today means non-streaming API-key requests on supported adapters. Streaming sessions on this path go through unchanged. The wrap page has the exact rule.

Compression runs before LiteLLM's own budget accounting sees the response. LiteLLM will count what the provider reported, which is the compressed request. That is correct behaviour. It means your LiteLLM spend dashboard shows the post-compression bill, and the local caveman stats shows what was removed, labelled as an inferred estimate.

4. Claude Code through LiteLLM through Caveman

Teams often run Claude Code through LiteLLM already, for budgets and one set of keys. Adding Caveman keeps that arrangement and puts compression at the front.

Claude Code  →  Caveman proxy (loopback)  →  LiteLLM proxy  →  Anthropic

Declare your LiteLLM host as a compatibility mount, allow it as an upstream, give Claude Code the recovery tool, and point Claude Code at the mount:

# ~/.caveman/caveman.yaml
compat:
  litellm:
    base_url: http://litellm.internal:4000
    api_key_env: LITELLM_VIRTUAL_KEY

caveman tools config set think.mode compress
caveman tools mcp install claude --server caveman
CAVE_SSRF_ALLOWLIST=litellm.internal caveman start
ANTHROPIC_BASE_URL=http://127.0.0.1:8787/compat/litellm claude

Claude Code appends /v1/messages, and a compat mount carries Anthropic protocol on that path with the key in x-api-key. Caveman compresses tool results in Messages shape, then forwards to LiteLLM, which serves its Anthropic-format endpoint as it normally would. Your virtual key still enforces its budget window. Your cache breakpoints pass through untouched, because Caveman never rewrites the cache_control markers a client sent.

One limit: a Claude Pro or Max login cannot cross LiteLLM. This path needs an API-key backed virtual key. If you want subscription auth, drop LiteLLM from the chain and run caveman claude, which passes the OAuth token straight to Anthropic.

Measured on the pinned benchmark, direct against wrapped, Claude Code used 33.2 percent fewer provider-reported input tokens across 18 paired runs with all 18 exact-answer checks passing. One case, an HTML dashboard, got 9.9 percent more expensive because no compressor applied. The gateway hop does not change either number.

5. Model decisions inside LiteLLM: caveman-router

Everything above moves bytes. This section moves a decision.

caveman-router is a small MIT Python package that registers as a LiteLLM callback. On each chat request it asks Caveman which of your configured deployment aliases to use, then changes only the alias. LiteLLM keeps the provider keys, does the inference, streams the response, and handles retries exactly as before. Inference never passes through Caveman.

# in the Python environment that runs your LiteLLM proxy
python -m pip install .           # from the caveman-router package directory
export CAVE_API_KEY='project key with router:write'
caveman-router setup --config config.yaml
caveman-router check --config config.caveman.yaml
litellm --config config.caveman.yaml

setup writes a second file beside your existing one. It adds the callback first in litellm_settings.callbacks, so later callbacks see the chosen alias, and it leaves every provider setting alone. It discovers eligible chat deployments from model_list and refuses to proceed with fewer than two distinct models, because a router with one option is not a router.

What the callback will not touch:

• Requests with tools, response_format, reasoning_effort, thinking, audio, logit_bias, or a previous_response_id. Those depend on model capabilities the decision does not carry, so the request is left as sent.
• Requests that pin a provider: api_base, api_key, custom_llm_provider, Azure, Bedrock or Vertex fields, or an explicit deployment.
• Anything outside the models the virtual key or team is allowed to call. The callback intersects with LiteLLM's own scope before asking, then re-runs LiteLLM's authorization check on the answer.
• Any model whose context window or supported parameters, read from LiteLLM's local catalog, cannot hold the request.

The decision has a wall-clock deadline, 150 milliseconds by default. On a timeout, an HTTP error, or an unparseable answer, the request proceeds on the model the client asked for and the callback logs one warning. Prompts are truncated to 8 KB before they leave, tool results are omitted, and no headers or tenant identifiers are sent.

The decision service behind /v1/route is part of Caveman Platform. The package, its setup tool, and its conformance tests against a real LiteLLM server are public. If you have a Platform project key, it works as described; if you do not, the first four sections stand on their own.

Headroom inside LiteLLM

LiteLLM's guide recommends Headroom as its compression sidecar. We benchmarked both against direct Claude Code on the same six tool-heavy tasks, three runs each, with an exact-answer check on every run.

Headroom's three YAML runs failed the exact-answer gate and stay visible in its row rather than counting toward savings. Caveman's HTML case lost 9.9 percent and stays visible in ours. Method, intervals, and fixtures are in the benchmark write-up. Both tools slot into the same place in a LiteLLM deployment, so switching is a config change.

Questions people ask

Does Caveman replace LiteLLM?

No. LiteLLM holds provider keys, translates provider APIs, and enforces budgets per virtual key. Caveman shortens output, compresses tool results with recovery, and can supply a model decision through a callback. Run both. Put Caveman between the agent and LiteLLM.

Does compression break LiteLLM's spend tracking?

No. LiteLLM records what the provider reported, which is the compressed request. Its dashboard shows the bill you actually paid. The amount removed is reported separately by caveman stats as an inferred local estimate, and the two are never added together.

Can the router callback see my provider keys?

No. The callback changes the deployment alias on the request and nothing else. Keys stay in LiteLLM's config. The callback sends at most 8 KB of prompt text and the list of allowed aliases to the decision endpoint, with no headers, tool results, or tenant identifiers.

What happens if the Caveman decision service is down?

The request goes to the model the client asked for. The callback has a 150 millisecond deadline by default and treats a timeout, an HTTP error, or an invalid response the same way: keep the requested model, log one warning, count the reason.

Does the local proxy work with LiteLLM's prompt caching injection?

Yes. Caveman forwards cache_control markers unchanged. If LiteLLM injects them, they arrive at the provider as injected. If the client sends them, they arrive as sent. Compression changes tool-result bodies, which sit after the cached prefix in a normal agent request, so a stable prefix stays stable.

Sources

• LiteLLM integration recipe and caveman snippets litellm.
• Proxy providers and mounts, security.
• CaveBench Wrap benchmark, 54 runs.
• LiteLLM, 5 ways to cut Claude Code costs.