Switch from LLMLingua to Caveman for tool output

This guide moves one job from LLMLingua to Caveman: compressing the results your agent's tools return. Your RAG step keeps LLMLingua. The example uses a Python LangChain agent, because LLMLingua users often live in LangChain or LlamaIndex, and Caveman has native adapters for both. The comparison page explains why the split makes sense.

You need Python 3.13 or newer for caveman-middleware and Node.js 22.13 or newer for the CLI that runs the local runtime.

Step 1: Sort what LLMLingua compresses today

Find every compress_prompt call and write down what goes into it. Two kinds of input usually show up.

Retrieved passages, transcripts and long documents are prose. Leave those on LLMLingua. It is good at them, and Caveman Middleware does not compress user or system messages anyway.

Tool results are the other kind: JSON from an API, a log file, test output, a search over the repo. A common pattern looks like this, with the tool pruning its own output before returning it:

@tool
def read_log(path: str) -> str:
    """Read a service log."""
    raw = open(path).read()
    return llm_lingua.compress_prompt(raw, rate=0.33)["compressed_prompt"]

Those calls are the ones to move. Save a few real payloads from them so you can compare before and after.

Step 2: Install and start the Caveman runtime

The runtime is the local Caveman proxy. It does the compression and holds the originals on disk. The CLI installs it:

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

setup --install downloads about 205 MB of signed Go binaries into ~/.caveman/bin/. Plain caveman start runs in record mode and changes nothing, so set CAVEMAN_MODE=compress when you want compression. Check it from another shell:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'

You want "mode": "compress", "persistent": true and "recovery": true.

Then install the Python packages. The middleware only has prereleases so far, so pin the version:

pip install caveman-sdk "caveman-middleware[langchain]==0.1.0a1"

Step 3: Return raw output and wrap the agent

Take LLMLingua out of the tool so it returns the full text again:

@tool
def read_log(path: str) -> str:
    """Read a service log."""
    return open(path).read()

Then wrap the agent. with_caveman_agent rewrites the keyword arguments you pass to create_agent and registers the recovery tool:

import os
from langchain.agents import create_agent
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.langchain import with_caveman_agent

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(
    endpoint="http://127.0.0.1:8787",
    mode=mode,
    on_report=lambda report: print(report.status, report.reason),
)
scope = Scope("support", conversation_id, "main", "0")

if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable
agent = create_agent(**with_caveman_agent(
    {"model": model, "tools": [read_log]}, runtime=runtime, scope=scope))
result = agent.invoke({"messages": messages})

scope identifies one conversation. Its four parts key the stored originals, so a handle from an earlier turn only resolves under the same scope. Build conversation_id from your own authenticated IDs and never share a scope across users. For LangGraph, scope_from_config(config, namespace=...) derives it from the RunnableConfig.

On LlamaIndex, swap FunctionAgent for CavemanFunctionAgent from caveman_middleware.llama_index. Other frameworks are listed in supported frameworks.

Step 4: Check that it worked

Run one task that reads a large tool result. Each model call prints a report status:

• applied means at least one tool result was replaced with a shorter copy.
• reused means a replacement from an earlier turn was kept as is, so the provider's cached prefix stays identical.
• skipped comes with a reason such as not_smaller, runtime_unavailable or unsupported_version.

A short tool result usually skips as no_candidate or not_smaller. That is expected.

Next, prove recovery. Ask a follow-up question that needs a detail the short view left out, such as one line from the middle of the log. The transcript should show a caveman_retrieve call and the right answer. The docs' verify program runs the same check with a scripted model and asserts the recovered text matches the original exactly, at no provider cost.

Last, confirm your own messages list still holds the full tool output. The middleware compresses a copy of the request and leaves your history alone.

Step 5: Compare the two setups

Run the saved payloads through the old path (LLMLingua in the tool) and the new one (raw tool output, Caveman on the request). Keep the model and prompts the same. Grade the final answers with exact values in the grader, and count provider input tokens, recovery calls and wall time. The cost measurement guide has a format for paired runs.

Local token counts from Caveman are inferred estimates. Your provider's usage report is the number that matters for cost.

If Caveman does nothing for a particular tool, that tool probably returns prose. Put LLMLingua back on it.

Roll back

Rollback takes one line. with_caveman_agent returns a new dict and mutates nothing, so building the agent from the original options restores the old behaviour exactly. You can also set CAVEMAN_CLIENT_MODE=off to stop compression without touching the agent code.

Put compress_prompt back in the tool if you want LLMLingua's pruning again. Leave the Caveman runtime running until in-flight conversations finish, because their handles point into its store.

Where to go next

• Caveman vs LLMLingua for the full comparison
• Caveman Middleware for the product overview
• Python middleware docs for every adapter
• LLMLingua on GitHub for the methods you are keeping