---
title: "Switch from LLMLingua to Caveman for tool output"
description: "This guide moves one job from LLMLingua to Caveman: compressing the results your agent's tools return. Your RAG step keeps LLMLingua. The example uses a Python LangChain agent, because LLMLingua users"
canonical: https://caveman.ac/switch/llmlingua
last-updated: 2026-10-04
---

# Switch from LLMLingua to Caveman for tool output

This guide moves one job from LLMLingua to Caveman: compressing the results your agent's tools return. Your RAG step keeps LLMLingua. The example uses a Python LangChain agent, because LLMLingua users often live in LangChain or LlamaIndex, and Caveman has native adapters for both. The [comparison page](/compare/llmlingua) explains why the split makes sense.

You need Python 3.13 or newer for `caveman-middleware` and Node.js 22.13 or newer for the CLI that runs the local runtime.

## Step 1: Sort what LLMLingua compresses today

Find every `compress_prompt` call and write down what goes into it. Two kinds of input usually show up.

Retrieved passages, transcripts and long documents are prose. Leave those on LLMLingua. It is good at them, and Caveman Middleware does not compress user or system messages anyway.

Tool results are the other kind: JSON from an API, a log file, test output, a search over the repo. A common pattern looks like this, with the tool pruning its own output before returning it:

```python
@tool
def read_log(path: str) -> str:
    """Read a service log."""
    raw = open(path).read()
    return llm_lingua.compress_prompt(raw, rate=0.33)["compressed_prompt"]
```

Those calls are the ones to move. Save a few real payloads from them so you can compare before and after.

## Step 2: Install and start the Caveman runtime

The runtime is the local Caveman proxy. It does the compression and holds the originals on disk. The CLI installs it:

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

`setup --install` downloads about 205 MB of signed Go binaries into `~/.caveman/bin/`. Plain `caveman start` runs in `record` mode and changes nothing, so set `CAVEMAN_MODE=compress` when you want compression. Check it from another shell:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'
```

You want `"mode": "compress"`, `"persistent": true` and `"recovery": true`.

Then install the Python packages. The middleware only has prereleases so far, so pin the version:

```bash
pip install caveman-sdk "caveman-middleware[langchain]==0.1.0a1"
```

## Step 3: Return raw output and wrap the agent

Take LLMLingua out of the tool so it returns the full text again:

```python
@tool
def read_log(path: str) -> str:
    """Read a service log."""
    return open(path).read()
```

Then wrap the agent. `with_caveman_agent` rewrites the keyword arguments you pass to `create_agent` and registers the recovery tool:

```python
import os
from langchain.agents import create_agent
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.langchain import with_caveman_agent

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(
    endpoint="http://127.0.0.1:8787",
    mode=mode,
    on_report=lambda report: print(report.status, report.reason),
)
scope = Scope("support", conversation_id, "main", "0")

if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable
agent = create_agent(**with_caveman_agent(
    {"model": model, "tools": [read_log]}, runtime=runtime, scope=scope))
result = agent.invoke({"messages": messages})
```

`scope` identifies one conversation. Its four parts key the stored originals, so a handle from an earlier turn only resolves under the same scope. Build `conversation_id` from your own authenticated IDs and never share a scope across users. For LangGraph, `scope_from_config(config, namespace=...)` derives it from the `RunnableConfig`.

On LlamaIndex, swap `FunctionAgent` for `CavemanFunctionAgent` from `caveman_middleware.llama_index`. Other frameworks are listed in [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks).

## Step 4: Check that it worked

Run one task that reads a large tool result. Each model call prints a report status:

- `applied` means at least one tool result was replaced with a shorter copy.
- `reused` means a replacement from an earlier turn was kept as is, so the provider's cached prefix stays identical.
- `skipped` comes with a reason such as `not_smaller`, `runtime_unavailable` or `unsupported_version`.

A short tool result usually skips as `no_candidate` or `not_smaller`. That is expected.

Next, prove recovery. Ask a follow-up question that needs a detail the short view left out, such as one line from the middle of the log. The transcript should show a `caveman_retrieve` call and the right answer. The docs' [verify program](https://docs.caveman.so/docs/sdk/middleware/python) runs the same check with a scripted model and asserts the recovered text matches the original exactly, at no provider cost.

Last, confirm your own `messages` list still holds the full tool output. The middleware compresses a copy of the request and leaves your history alone.

## Step 5: Compare the two setups

Run the saved payloads through the old path (LLMLingua in the tool) and the new one (raw tool output, Caveman on the request). Keep the model and prompts the same. Grade the final answers with exact values in the grader, and count provider input tokens, recovery calls and wall time. The [cost measurement guide](/guides/measure-agent-cost) has a format for paired runs.

Local token counts from Caveman are inferred estimates. Your provider's usage report is the number that matters for cost.

If Caveman does nothing for a particular tool, that tool probably returns prose. Put LLMLingua back on it.

## Roll back

Rollback takes one line. `with_caveman_agent` returns a new dict and mutates nothing, so building the agent from the original options restores the old behaviour exactly. You can also set `CAVEMAN_CLIENT_MODE=off` to stop compression without touching the agent code.

Put `compress_prompt` back in the tool if you want LLMLingua's pruning again. Leave the Caveman runtime running until in-flight conversations finish, because their handles point into its store.

## Where to go next

- [Caveman vs LLMLingua](/compare/llmlingua) for the full comparison
- [Caveman Middleware](/products/caveman-middleware) for the product overview
- [Python middleware docs](https://docs.caveman.so/docs/sdk/middleware/python) for every adapter
- [LLMLingua on GitHub](https://github.com/microsoft/LLMLingua) for the methods you are keeping