Add Caveman to LangSmith: setup and eval check

LangSmith stays exactly as it is. You add Caveman's LangChain middleware to the agent, LangSmith keeps tracing through its environment variables, and two LangSmith experiments over the same dataset tell you whether the shorter tool output changed any answers. This guide uses Python and create_agent. For the bigger picture, read Caveman vs LangSmith.

1. Install and start the runtime

The Python packages need Python 3.13 or newer. The Caveman CLI installs the runtime and needs Node.js 22.13 or newer.

pip install langsmith caveman-sdk "caveman-middleware[langchain]==0.1.0a1"
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

From another shell, confirm the mode:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'

It has to say compress. A runtime started without CAVEMAN_MODE=compress runs record and changes nothing.

Keep your LangSmith variables as they are: LANGSMITH_TRACING=true and LANGSMITH_API_KEY. LangChain tracing needs no code.

2. Add the middleware to the agent

with_caveman_agent returns a new copy of your create_agent arguments with Caveman's agent middleware and the caveman_retrieve tool appended. Anything already in middleware stays.

import os, uuid
from langchain.agents import create_agent
from langsmith import Client
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.langchain import with_caveman_agent

MODE = os.environ.get("CAVEMAN_CLIENT_MODE", "record")  # "record" = baseline, "compress" = candidate, "off" = rollback
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=MODE)
if MODE != "off":  # ready() raises in off mode
    runtime.ready()

def target(inputs: dict) -> dict:
    scope = Scope("eval", str(uuid.uuid4()), "main", "0")  # one scope per conversation
    agent = create_agent(**with_caveman_agent(
        {"model": model, "tools": tools},  # your existing model and tools
        runtime=runtime, scope=scope))
    result = agent.invoke(
        {"messages": [{"role": "user", "content": inputs["question"]}]},
        {"recursion_limit": 12},
    )
    messages = result["messages"]
    return {
        "answer": messages[-1].content,
        "recoveries": sum(1 for m in messages if getattr(m, "name", None) == "caveman_retrieve"),
    }

In production, key the scope to the real conversation. scope_from_config(config, namespace="support") builds it from your LangGraph thread_id, so a recovery handle from turn three still resolves on turn nine.

3. Run two experiments on one dataset

Create a LangSmith dataset of 20 to 50 examples where the answer sits inside a large tool result: a failing assertion in a long CI log, one field in a 20k-token JSON response, a clause on page 40 of an extracted contract. Put the expected answer in the reference output.

def correct(outputs: dict, reference_outputs: dict) -> bool:
    return reference_outputs["answer"].lower() in outputs["answer"].lower()

def recoveries(outputs: dict) -> dict:
    return {"key": "recoveries", "score": outputs["recoveries"]}

Client().evaluate(
    target,
    data="heavy-tool-output",
    evaluators=[correct, recoveries],
    experiment_prefix=f"caveman-{MODE}",
    max_concurrency=2,
)
runtime.close()

CAVEMAN_CLIENT_MODE=record python trial.py
CAVEMAN_CLIENT_MODE=compress python trial.py

record sends every byte unchanged through the same code path, which makes it a fair baseline. Use an LLM-as-judge evaluator instead of the substring check if your answers are prose.

4. Check it worked

In LangSmith, open the dataset, select both experiments, click Compare, and set caveman-record as the source experiment. Then check:

• Red rows. LangSmith marks examples that regressed against the source in red. Open each one. A single flipped answer on a task you care about outweighs a nice token number. For recoveries, set lower as better in the column settings.
• What the model read. Open an LLM run in the compress experiment. Shortened tool results begin with [caveman: shortened; exact original via caveman_retrieve handle=…]. The tool run above it still shows the full original.
• Tokens per task. Compare total input tokens across all LLM runs in a trace, not the first call. Every caveman_retrieve call is one more round trip.
• Recovery rate. A handful of recoveries is the system working. Recoveries on most examples mean the short view is missing what your tasks need.

To see the local decision per call, pass on_report=print to MiddlewareRuntime. You will see applied, reused, recorded, or skipped with a reason such as no_candidate.

If you trace a raw OpenAI client with wrap_openai

A raw client means you own the tool loop, so use with_caveman_openai_tools. It adds caveman_retrieve to your tool definitions and returns a dispatch table that runs it. The plain with_caveman_openai wrapper has no recovery executor, and the runtime will not shorten text it cannot give back, so on its own it sends every call unchanged. Wrap in this order:

from openai import OpenAI
from langsmith.wrappers import wrap_openai
from caveman_middleware.openai import with_caveman_openai_tools

loop = with_caveman_openai_tools(OpenAI(), runtime=runtime, scope=scope,
    protocol="openai-chat", tools=tools, functions=functions)  # functions: {tool name: callable(args dict)}
client = wrap_openai(loop.client)
response = client.chat.completions.create(model=model_name, messages=messages, tools=loop.tools)
# Run every tool call the model makes through loop.functions[name](arguments), caveman_retrieve included.

Caveman returns a new client, so wrap_openai has to go on loop.client. Send loop.tools on every request; a call without the recovery tool goes out uncompressed. The Python OpenAI adapter acts only on openai >=3.10 <4. In these traces the input shows your original messages, because Caveman rewrites the request below chat.completions.create, while usage_metadata comes from the provider's response for the shorter request.

Roll back

• Fastest: run with CAVEMAN_CLIENT_MODE=off, which sets mode="off" on MiddlewareRuntime. Calls pass through untouched and nothing reaches the runtime.
• Code: remove with_caveman_agent(...) and pass your original dict to create_agent. The wrapper built a new dict and changed nothing in yours.
• Runtime: restart it without CAVEMAN_MODE=compress to return to record.

Nothing in LangSmith needs undoing. Your datasets, experiments and traces stay as they were.

More: Python middleware docs, supported frameworks, LangSmith evaluation quickstart, comparing experiments, and the agent evaluations guide.