Caveman vs Arize Phoenix: traces, evals and compression

Keep Phoenix. Caveman works with it. Phoenix shows you what happened inside an agent run and grades whether the answer was any good. Caveman Middleware changes one thing in that run: it sends the model a shorter copy of large tool results, like a 20,000-token JSON response or a whole file read, and keeps the original recoverable. The two sit at different layers, and Phoenix is the right place to check that Caveman did not cost you quality.

What is Arize Phoenix?

Arize Phoenix is an AI observability and evaluation tool you run yourself. It traces your app with OpenTelemetry and the OpenInference conventions, so each run becomes a tree of spans: model calls, retrieval, tool use and your own functions. On top of the traces it gives you LLM-as-judge and code evaluators, versioned datasets, experiments that rerun the same inputs after a change, a playground that can replay a traced call, and prompt management with versions and tags. Newer additions include PXI, an assistant for debugging traces, and a remote MCP server so Claude Code or Cursor can query your traces.

It is easy to start. pip install arize-phoenix then phoenix serve gives you a local UI on port 6006, and there are Docker images and a Helm chart for real deployments. Auto-instrumentation covers OpenAI, Anthropic, Google GenAI, Bedrock, LiteLLM, LangChain and LangGraph, LlamaIndex, CrewAI, DSPy, Mastra, the Vercel AI SDK and more, in Python, TypeScript and Java.

Phoenix is source available on GitHub under the Elastic License 2.0 (about 11.7k stars when we checked) and free to self-host. Arize sells a managed platform, Arize AX, with a free tier of 25k spans a month and a Pro tier at $50 a month.

Does Caveman work with Phoenix?

Yes. Phoenix has no Caveman adapter and does not need one. Caveman Middleware wraps the framework or provider client you already call, and Phoenix keeps instrumenting that same code. You change one line where the agent is built, and your register() call stays as it is.

Before each model call, the middleware finds eligible tool results in the outbound request (file reads, API responses, command output, extracted documents) and swaps them for a shorter copy made by a local Caveman runtime. Your application keeps its original message history. The model gets a caveman_retrieve tool so it can pull the full original back when the short copy is not enough. User messages, system prompts, reasoning, errors, and image or PDF bytes are left alone.

Adapters are published for TypeScript (@caveman-ai/middleware: Vercel AI SDK, OpenAI, Anthropic, Google Gen AI, LangChain, Strands, Mastra, MCP) and Python (caveman-middleware: OpenAI, Anthropic, Google Gen AI, LangChain and LangGraph, LiteLLM, CrewAI, Pydantic AI, AutoGen, LlamaIndex, Agno, Strands, MCP, ASGI). Both are alpha.

Which copy does a Phoenix trace record?

It depends on where Phoenix's instrumentation hooks in relative to Caveman. We read both codebases, and three cases matter.

OpenAI Python client: the span shows the shortened copy. The OpenInference OpenAI instrumentor patches the client's request method. Caveman's Python OpenAI adapter rewrites the body one step earlier, in the client's post. So the LLM span holds what the model actually received, recovery handle included.

LangChain and LangGraph: the model span is short, the tool span is full. Caveman runs as agent middleware and hands the model an overridden, shorter message list. The OpenInference LangChain tracer records the model's input from that list. The tool span still shows the full output your tool returned, because the graph state keeps the original.

Anthropic Python client: the span shows the original. OpenInference wraps Messages.create, and Caveman runs below it inside the SDK's request middleware. The message text on that span is what your app passed in, not what the model read.

You don't have to take our word for any of this. A shortened tool result starts with [caveman: shortened; exact original via caveman_retrieve handle=cmw_…]. Search your spans for caveman: shortened and you will see which copy your setup records.

Both copies are useful. The tool span shows what the tool returned, and the model span shows what the model saw. Check the model span before blaming the model for a missed detail.

Phoenix vs Caveman: feature comparison

How do you use Phoenix evals to check Caveman?

Run the same dataset through your agent twice and grade both runs with the same evaluators. Phoenix's run_experiment takes a dataset, a task function and a list of evaluators, and keeps each run as an experiment you can compare side by side.

Make the two arms differ in one thing only. Start the Caveman runtime in compress mode, then set the middleware client to record for the baseline and compress for the candidate. In record mode the runtime measures each call and sends the original bytes. Keep the model, tools, prompt version and dataset identical.

Then read three things. Did the evaluator scores hold? How many caveman_retrieve tool spans appeared, since each recovery carries the original back and costs tokens? And what did provider-reported input tokens do across the whole task, retries included?

Caveman's own published numbers do not answer the quality question. The docs show a 400-line worker log going from 8,399 tokens to between 177 and 187 across sixteen runs, counted with o200k_base and labelled inferred. The middleware page reports 13 synthetic samples through the real runtime, also inferred, and says task quality was not measured. Your Phoenix experiment is where quality gets measured, on your data.

When should you use Phoenix alone?

Use Phoenix alone when your requests are short. A chat app with no tool calls and 1,500-token turns gives Caveman nothing to compress.

Use it alone when the problem is answer quality. If retrieval returns the wrong document, a shorter copy of the wrong document fixes nothing.

Use it alone when you don't yet know where the tokens go. Trace first.

When does adding Caveman make sense?

Add Caveman when your Phoenix traces show large tool results riding along turn after turn. A coding agent that reads a 20,000-token file on step two and carries it to step fourteen is the classic case. So is a support agent holding a big order-history JSON in context.

For terminal coding agents like Claude Code or Codex, Caveman Proxy is simpler.

How to add Caveman to a Phoenix-traced app

Start the runtime, wrap your agent or client, keep register() unchanged, and run the two-arm experiment. The step-by-step version with code and a rollback is in add Caveman to Arize Phoenix. Adapter versions are on the middleware frameworks page.

FAQ

Is Caveman an Arize Phoenix alternative? No. Phoenix traces and evaluates. Caveman reduces the input tokens sent to the model. Most teams that want both keep Phoenix and add the middleware.

Will Caveman break my Phoenix traces? No. Spans still arrive and nest as before. Depending on the instrumentor, a model span may show the shortened copy, and caveman_retrieve calls show up as ordinary tool spans.

Does Caveman send my prompts anywhere? The middleware talks to a runtime you run, by default on 127.0.0.1:8787. Inference goes to your provider as before.

Does Caveman have its own trace UI? Caveman Cloud, a hosted workspace, is open for sign-up with a Free plan. If Phoenix already holds your traces, it stays a good place to read them.

Sources

• Phoenix documentation and OpenAI tracing setup
• Phoenix on GitHub and the phoenix-client README
• OpenInference instrumentation source
• Arize pricing
• Caveman Middleware docs, Python middleware and supported frameworks
• Related: agent evaluations, agent observability, prompt compression