Caveman vs LLMLingua: prompt compression compared

LLMLingua and Caveman both cut input tokens, and they cut different things. LLMLingua runs a small language model over your prompt and deletes the tokens it scores as least informative. Whatever it deletes is gone. Caveman looks at a tool result, works out whether it is JSON, a log, a diff or something else, rewrites it into a shorter view and stores the original first, so the model can pull the exact bytes back when it needs them.

For long prose and retrieved passages, LLMLingua is the stronger tool. For agent loops where one missing ID or stack frame breaks the task, Caveman fits better. Many applications could reasonably run both.

What is LLMLingua?

LLMLingua is Microsoft Research's open source prompt compression library: MIT licensed, written in Python, installed with pip install llmlingua. It uses a compact model, GPT2-small or LLaMA-7B in the reference setup, to find and remove non-essential tokens, and the project reports up to 20x compression with minimal performance loss on its benchmarks.

The name covers a family of methods, each with a paper behind it:

• LLMLingua (EMNLP 2023) scores tokens with the small model and prunes to a budget.
• LongLLMLingua (ACL 2024) makes compression question-aware and reorders retrieved documents. The project reports up to 21.4% better RAG performance using a quarter of the tokens.
• LLMLingua-2 (ACL 2024 Findings) replaces the scorer with a BERT-level token classifier distilled from GPT-4. It is 3x to 6x faster than the original and handles out-of-domain text better.

It plugs into LangChain, LlamaIndex and Microsoft Prompt flow, and the repo includes training code if you want to fit LLMLingua-2 to your own data. The latest PyPI release is 0.2.2, from April 2024. A typical call:

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
)
result = llm_lingua.compress_prompt(prompt, rate=0.33, force_tokens=["\n", "?"])

What does Caveman do differently?

Caveman compresses tool output by format and keeps the original. Caveman Middleware is a native adapter for the framework you already use: LangChain and LangGraph, LlamaIndex, Pydantic AI, the OpenAI and Anthropic SDKs, the Vercel AI SDK and more. Before each provider call it takes a copy of the outbound request, finds eligible tool results (file reads, API responses, command output, document extraction) and swaps each one for a shorter view.

Detection reads the bytes and calls no model. The runtime sorts a payload into JSON, terminal output, diff, HTML, table, code, log, search results, config or plain text, and each type has its own compressor. Before any lossy replacement, the original goes into a local store. The model gets a caveman_retrieve tool and a handle, so a later turn can page the exact bytes back. If the store write fails, the runtime refuses to shorten anything.

Your application keeps its own conversation history byte for byte. User messages, system prompts, reasoning, errors and image or PDF bytes are never touched. Inference still goes straight to your provider.

In the middleware docs, a 400-line worker log went from 8,399 tokens to about 180 in the view the model saw. That count is an inferred o200k_base estimate on repetitive sample output, not a bill figure.

For coding agents like Claude Code or Codex, the same engine runs as Caveman Proxy, a local proxy you start with caveman claude.

Is LLMLingua compression lossless?

No. LLMLingua deletes tokens and keeps no copy. Its README says GPT-4 can recover the key information from compressed prompts, which means the model infers what the missing words meant. Your program has no way to read the deleted text back.

The README's own example shows the trade: a worked math demonstration comes out with lines like "Sam bought 1 boxes x00 oflters". A model can usually still follow the reasoning, which is why this works well for prose. It works less well when the task depends on an exact value: a UUID, a version string, a file path, the line number in a stack frame.

Caveman's view is lossy too. The 396 log lines it folded are not in the prompt. The difference is that they sit on your disk under a handle, and the model can ask for them.

LLMLingua vs Caveman: feature comparison

When should you use LLMLingua?

Use LLMLingua when most of your context is prose. Research papers, meeting transcripts, support threads and retrieved passages have no JSON or log structure for Caveman to exploit. Caveman's text compressor is deliberately coarse: it keeps headings, the opening and closing sections and anything marked important, then folds the middle into a marker. LLMLingua works at the token level and can thin every paragraph.

It is also the better pick when you need a hard budget. Pass target_token=500 and LLMLingua aims for 500. Caveman shrinks by whatever the format allows, which on a dense payload can be nothing.

LongLLMLingua's question-aware ranking is a real advantage for RAG. It compresses retrieved documents relative to the question and moves the relevant ones out of the middle of the context. Caveman has nothing equivalent.

And if you study compression, LLMLingua ships papers, baselines and training code other people can reproduce.

When is Caveman the better pick?

Pick Caveman when your agent calls tools and the results are big: a 20k-token JSON API response, a 3,000-line test run, a repo-wide search. Those payloads are repetitive, the format compressors are built for them, and the details the model needs later stay reachable.

It also fits when you want no second model in the request path. LLMLingua runs its scorer on every call; Caveman's detection and transforms are deterministic code.

TypeScript teams have a native path through the Vercel AI SDK, Mastra, LangChain.js and the OpenAI or Anthropic clients. LLMLingua's official package is Python only.

Can you use LLMLingua and Caveman together?

Yes, if each one handles its own inputs. Run LLMLingua where you build the prompt, on retrieved passages or long documents a user pasted. Let Caveman Middleware handle tool results at call time. They rarely collide, because Caveman leaves user and system messages alone.

One rule: do not run LLMLingua over a tool result before Caveman sees it. Caveman can only store what it receives, so the "original" it recovers would be LLMLingua's pruned text.

How do you compare them on your own data?

Send one recorded payload through both with the model held fixed, and grade the finished answer on exact values like an error code. Charge LLMLingua for scoring time and Caveman for recovery calls. No matched public benchmark exists, so this page gives no head-to-head number. The measurement guide has a format, and prompt compression explained covers the category.

Ready to try it? The LLMLingua switching guide moves tool-result compression to Caveman and leaves your RAG step alone.

FAQ

Can the model recover text LLMLingua removed? No. The deleted tokens are not stored anywhere. The model can only infer what they said.

Does Caveman need a GPU? No. Format detection and compression are deterministic and run in a local runtime on CPU.

Does Caveman compress RAG documents? Only when they come back as a tool result, and prose gets the coarse section-folding compressor. For question-aware passage compression, LLMLingua is the better tool.

Is Caveman Middleware production ready? It is alpha: @caveman-ai/middleware 0.1.0-alpha.2 and caveman-middleware 0.1.0a1. Check the supported framework versions and test your own workload first.

Sources

• LLMLingua on GitHub
• LLMLingua project page
• llmlingua on PyPI
• Caveman Middleware docs
• Caveman supported frameworks
• Caveman compressors