Caveman vs LangGraph: compress tool results in LangGraph

LangGraph runs your agent as a durable, stateful graph. Caveman Middleware is an adapter that shrinks large tool results before each model call in that graph. It plugs in through LangChain's agent middleware, so you keep LangGraph, its checkpointer and your tools, and add one entry to the middleware list.

What is LangGraph?

LangGraph is LangChain's low-level orchestration framework and runtime for long-running, stateful agents. It is MIT licensed, available for Python and JavaScript, and the Python repo has about 42k GitHub stars. The docs name Klarna, Uber and J.P. Morgan among the companies using it.

What it does better than most frameworks:

• Durable execution. Agents persist through failures and resume where they stopped.
• Human in the loop. You can inspect and edit agent state at any point.
• Memory. Short-term working memory for a run and long-term memory across sessions, backed by checkpointers keyed by thread_id.
• Control. Deterministic, hand-coded steps and LLM-driven steps in the same graph.
• Debugging through LangSmith.

For a standard tool loop you do not have to write the graph yourself. LangChain 1.0's create_agent is built on LangGraph, and its middleware hooks (before_model, wrap_model_call, wrap_tool_call and others) are where you customize the loop.

LangGraph itself is free. LangSmith, the companion platform for tracing and deployment, has a free Developer plan with one seat and 5k base traces a month, a Plus plan at $39 per seat per month, and custom Enterprise pricing with self-hosted options.

Does Caveman work with LangGraph?

Yes, through the LangChain adapter. In Python, caveman_middleware.langchain supports langchain >=1.4 <2, langchain-core >=1.6 <2 and langgraph >=1.2 <2. In TypeScript, @caveman-ai/middleware/langchain supports langchain >=1.5 <2, @langchain/core >=1.2 <2 and @langchain/langgraph >=1.4 <2.

with_caveman_agent takes the keyword arguments you were going to pass to create_agent and returns a new dict with a Caveman middleware appended to your middleware list and a caveman_retrieve tool appended to tools. This is the Python docs example:

from langchain.agents import create_agent
from caveman_middleware.langchain import with_caveman_agent

def run(*, model, tools, messages, runtime, scope):
    if runtime.mode != "off":  # ready() raises in off mode
        runtime.ready()
    agent = create_agent(**with_caveman_agent(
        {"model": model, "tools": tools}, runtime=runtime, scope=scope))
    return agent.invoke({"messages": messages}, {"recursion_limit": 12})

The TypeScript version has the same shape:

import { createAgent } from 'langchain';
import { withCavemanAgent } from '@caveman-ai/middleware/langchain';
export async function run({ model, tools, messages, runtime, scope }) {
  if (runtime.mode !== "off") await runtime.ready(); // ready() throws in off mode
  const agent = createAgent(withCavemanAgent({ model, tools }, { runtime, scope }));
  return agent.invoke({ messages }, { recursionLimit: 12 });
}

The middleware works in wrap_model_call. For each model call it copies the request, sends eligible ToolMessage text to a local Caveman runtime, and swaps in the shorter version. The graph state and your checkpointer keep the original messages. If the model needs a detail that was cut, it calls caveman_retrieve and reads the exact original back.

The scope fits LangGraph's own identity model. scope_from_config(config, namespace=...) (scopeFromConfig in TypeScript) builds the Caveman scope from configurable.thread_id, so the thread your checkpointer resumes is the same session Caveman stores originals under.

For a sense of size: the Caveman docs send a 400-line worker log as a tool result, and the runtime counted 8,399 tokens before and 177 to 187 after across sixteen runs. Those are inferred o200k_base counts on one synthetic log, not a bill.

LangGraph vs Caveman: feature comparison

When should you use LangGraph alone?

Use it alone if your tools return short results. Caveman needs a sizable text tool result to work on, and reports no_candidate or not_smaller otherwise.

LangChain's built-in context tools may also be enough. SummarizationMiddleware compresses older history with a model call when you approach a token limit. Context editing with ClearToolUsesEdit replaces older tool outputs with placeholders. Both ship with LangChain. They are a good fit when old tool output truly stops mattering.

If your graph is hand-built without create_agent, note the limit: with_caveman_model wraps a single chat model for custom graphs, but it has no recovery executor. The recoverable cuts described here need the agent middleware path.

When does adding Caveman pay off?

It pays off when your agent keeps needing its tool output. A research agent that reads a 20k-token API response at step 2 and keeps reasoning over it for eight more steps sends that response eight more times. Clearing it loses the data. Summarizing it costs a model call and loses detail. Caveman shortens the resent copy and leaves the full text one caveman_retrieve call away.

After the first cut, Caveman resends the exact same shortened bytes on later calls, reported as reused, so the provider's cached prefix stays byte identical.

It also layers with what you have. Context editing can still clear stale results, and Caveman shrinks the ones that remain in the request.

If the runtime is unreachable, calls go out with the original messages. After three failures in a row the adapter skips the runtime for 30 seconds, and your graph keeps running.

How to add Caveman to LangGraph

1. pip install caveman-sdk "caveman-middleware[langchain]==0.1.0a1" on Python 3.13 or newer, or npm install @caveman-ai/middleware @caveman-ai/sdk.
2. Start the runtime with CAVEMAN_MODE=compress caveman start.
3. Pass your create_agent arguments through with_caveman_agent.
4. Log reports and look for applied and reused.

The LangGraph switch guide covers the steps, scope and rollback. The Python middleware docs include a runnable LangChain example that makes no provider calls. If you trace with LangSmith, Caveman vs LangSmith covers that side.

FAQ

Does Caveman replace LangGraph or LangChain? No. LangGraph still runs the graph, checkpoints state and calls the model. Caveman only shortens tool-result text in the request copy sent to the provider.

Does it change what my checkpointer stores? No. The graph state keeps the original messages. The docs' example asserts that application history is unchanged after a compressed run.

Does it work with a custom StateGraph? Partly. with_caveman_model wraps a chat model inside any graph, but without a recovery tool. For recoverable compression use create_agent with with_caveman_agent.

How is this different from SummarizationMiddleware? Summarization rewrites older history with a model call and the original detail is gone from the request. Caveman shortens tool results in a local runtime, and the model can fetch the exact original back when it needs it.

Sources

• LangGraph overview
• What's new in LangChain v1
• LangChain built-in middleware
• langchain-ai/langgraph on GitHub
• LangSmith pricing
• Caveman middleware: Python and LangChain