---
title: "Add Caveman to LangSmith: setup and eval check"
description: "LangSmith stays exactly as it is. You add Caveman's LangChain middleware to the agent, LangSmith keeps tracing through its environment variables, and two LangSmith experiments over the same dataset te"
canonical: https://caveman.ac/switch/langsmith
last-updated: 2026-10-04
---

# Add Caveman to LangSmith: setup and eval check

LangSmith stays exactly as it is. You add Caveman's LangChain middleware to the agent, LangSmith keeps tracing through its environment variables, and two LangSmith experiments over the same dataset tell you whether the shorter tool output changed any answers. This guide uses Python and `create_agent`. For the bigger picture, read [Caveman vs LangSmith](/compare/langsmith).

## 1. Install and start the runtime

The Python packages need Python 3.13 or newer. The Caveman CLI installs the runtime and needs Node.js 22.13 or newer.

```bash
pip install langsmith caveman-sdk "caveman-middleware[langchain]==0.1.0a1"
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

From another shell, confirm the mode:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery}'
```

It has to say `compress`. A runtime started without `CAVEMAN_MODE=compress` runs `record` and changes nothing.

Keep your LangSmith variables as they are: `LANGSMITH_TRACING=true` and `LANGSMITH_API_KEY`. LangChain tracing needs no code.

## 2. Add the middleware to the agent

`with_caveman_agent` returns a new copy of your `create_agent` arguments with Caveman's agent middleware and the `caveman_retrieve` tool appended. Anything already in `middleware` stays.

```python
import os, uuid
from langchain.agents import create_agent
from langsmith import Client
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.langchain import with_caveman_agent

MODE = os.environ.get("CAVEMAN_CLIENT_MODE", "record")  # "record" = baseline, "compress" = candidate, "off" = rollback
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=MODE)
if MODE != "off":  # ready() raises in off mode
    runtime.ready()

def target(inputs: dict) -> dict:
    scope = Scope("eval", str(uuid.uuid4()), "main", "0")  # one scope per conversation
    agent = create_agent(**with_caveman_agent(
        {"model": model, "tools": tools},  # your existing model and tools
        runtime=runtime, scope=scope))
    result = agent.invoke(
        {"messages": [{"role": "user", "content": inputs["question"]}]},
        {"recursion_limit": 12},
    )
    messages = result["messages"]
    return {
        "answer": messages[-1].content,
        "recoveries": sum(1 for m in messages if getattr(m, "name", None) == "caveman_retrieve"),
    }
```

In production, key the scope to the real conversation. `scope_from_config(config, namespace="support")` builds it from your LangGraph `thread_id`, so a recovery handle from turn three still resolves on turn nine.

## 3. Run two experiments on one dataset

Create a LangSmith dataset of 20 to 50 examples where the answer sits inside a large tool result: a failing assertion in a long CI log, one field in a 20k-token JSON response, a clause on page 40 of an extracted contract. Put the expected answer in the reference output.

```python
def correct(outputs: dict, reference_outputs: dict) -> bool:
    return reference_outputs["answer"].lower() in outputs["answer"].lower()

def recoveries(outputs: dict) -> dict:
    return {"key": "recoveries", "score": outputs["recoveries"]}

Client().evaluate(
    target,
    data="heavy-tool-output",
    evaluators=[correct, recoveries],
    experiment_prefix=f"caveman-{MODE}",
    max_concurrency=2,
)
runtime.close()
```

```bash
CAVEMAN_CLIENT_MODE=record python trial.py
CAVEMAN_CLIENT_MODE=compress python trial.py
```

`record` sends every byte unchanged through the same code path, which makes it a fair baseline. Use an LLM-as-judge evaluator instead of the substring check if your answers are prose.

## 4. Check it worked

In LangSmith, open the dataset, select both experiments, click **Compare**, and set `caveman-record` as the source experiment. Then check:

- **Red rows.** LangSmith marks examples that regressed against the source in red. Open each one. A single flipped answer on a task you care about outweighs a nice token number. For `recoveries`, set lower as better in the column settings.
- **What the model read.** Open an LLM run in the compress experiment. Shortened tool results begin with `[caveman: shortened; exact original via caveman_retrieve handle=…]`. The tool run above it still shows the full original.
- **Tokens per task.** Compare total input tokens across all LLM runs in a trace, not the first call. Every `caveman_retrieve` call is one more round trip.
- **Recovery rate.** A handful of recoveries is the system working. Recoveries on most examples mean the short view is missing what your tasks need.

To see the local decision per call, pass `on_report=print` to `MiddlewareRuntime`. You will see `applied`, `reused`, `recorded`, or `skipped` with a reason such as `no_candidate`.

## If you trace a raw OpenAI client with wrap_openai

A raw client means you own the tool loop, so use `with_caveman_openai_tools`. It adds `caveman_retrieve` to your tool definitions and returns a dispatch table that runs it. The plain `with_caveman_openai` wrapper has no recovery executor, and the runtime will not shorten text it cannot give back, so on its own it sends every call unchanged. Wrap in this order:

```python
from openai import OpenAI
from langsmith.wrappers import wrap_openai
from caveman_middleware.openai import with_caveman_openai_tools

loop = with_caveman_openai_tools(OpenAI(), runtime=runtime, scope=scope,
    protocol="openai-chat", tools=tools, functions=functions)  # functions: {tool name: callable(args dict)}
client = wrap_openai(loop.client)
response = client.chat.completions.create(model=model_name, messages=messages, tools=loop.tools)
# Run every tool call the model makes through loop.functions[name](arguments), caveman_retrieve included.
```

Caveman returns a new client, so `wrap_openai` has to go on `loop.client`. Send `loop.tools` on every request; a call without the recovery tool goes out uncompressed. The Python OpenAI adapter acts only on `openai >=3.10 <4`. In these traces the input shows your original messages, because Caveman rewrites the request below `chat.completions.create`, while `usage_metadata` comes from the provider's response for the shorter request.

## Roll back

- **Fastest:** run with `CAVEMAN_CLIENT_MODE=off`, which sets `mode="off"` on `MiddlewareRuntime`. Calls pass through untouched and nothing reaches the runtime.
- **Code:** remove `with_caveman_agent(...)` and pass your original dict to `create_agent`. The wrapper built a new dict and changed nothing in yours.
- **Runtime:** restart it without `CAVEMAN_MODE=compress` to return to `record`.

Nothing in LangSmith needs undoing. Your datasets, experiments and traces stay as they were.

More: [Python middleware docs](https://docs.caveman.so/docs/sdk/middleware/python), [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks), [LangSmith evaluation quickstart](https://docs.langchain.com/langsmith/evaluation-quickstart), [comparing experiments](https://docs.langchain.com/langsmith/compare-experiment-results), and the [agent evaluations guide](/guides/agent-evaluations).