Caveman vs Pydantic AI: token compression for agents

Pydantic AI is the Python framework your agent runs on. Caveman Middleware plugs into it as a capability that shrinks large tool returns before each model request. Keep Pydantic AI for typed agents, tools and outputs, and add CavemanCapability where tools return more text than the next request needs.

What is Pydantic AI?

Pydantic AI is a Python agent framework from the team behind Pydantic, the validation layer inside the OpenAI SDK, the Anthropic SDK and Google ADK. It is MIT licensed with about 20k GitHub stars. The pitch is "typed end to end": structured outputs, typed dependency injection and typed tools, so mistakes surface while you develop.

What it does well:

• Model agnostic. One Python API for OpenAI, Anthropic, Google, Bedrock, Azure, Groq, Mistral, xAI, Ollama and dozens more.
• Capabilities. Reusable, composable units that bundle tools, lifecycle hooks, instructions and model settings. You pass them in capabilities=[...].
• Observability. OpenTelemetry-native instrumentation that feeds Pydantic Logfire.
• Durable execution on engines like Temporal, DBOS and Prefect.
• Extras like MCP support, Pydantic Graph for typed control flow, and Pydantic Evals.

It also takes context seriously. The core Compaction capability uses provider-native compaction on OpenAI and Anthropic. Pydantic AI Harness, the official capability library, adds model-agnostic compaction (tool-result clearing, sliding windows, LLM summarization) and Tool Output Limits, which truncate, spill to a queryable file, or summarize oversized tool returns at the source. The ProcessHistory capability lets you rewrite message history yourself before each request.

Does Caveman work with Pydantic AI?

Yes. caveman_middleware.pydantic_ai is a native adapter for pydantic-ai-slim >=2.42 <3, installed with the [pydantic-ai] extra on Python 3.13 or newer. It is a capability, so it fits the way Pydantic AI already composes behaviour. This is the docs example:

from pydantic_ai import Agent
from caveman_middleware.pydantic_ai import CavemanCapability

def build(*, model, runtime, scope):
    return Agent(model, capabilities=[CavemanCapability(runtime=runtime, scope=scope)])

The capability does two things. It contributes a caveman_retrieve tool through its own toolset, and it wraps each model request through the wrap_model_request hook. On every request it copies the outbound messages, sends eligible ToolReturnPart text to a local Caveman runtime, and swaps in the shorter version. The agent's message history keeps the originals. If the model needs a detail that was cut, it calls caveman_retrieve and gets the exact original back.

It works on streamed runs too, and it accepts the regular synchronous MiddlewareRuntime, converting it to an async view for Pydantic AI's async internals. scope_from_run(ctx, namespace=...) builds the scope from the run's native conversation_id.

Two limits to know before a trial. The adapter compresses only when the model underneath is Pydantic AI's OpenAIChatModel or AnthropicModel; other model classes pass through untouched and report unsupported_provider. And it shortens only tool returns that are plain strings, so a tool that returns a dict or a Pydantic model is sent as is.

For a feel of the size change: the Caveman docs send a 400-line worker log as a tool result, and the runtime counted 8,399 tokens before and 177 to 187 after across sixteen runs. Those are inferred o200k_base counts on one synthetic log, not a bill.

Pydantic AI vs Caveman: feature comparison

When should you use Pydantic AI alone?

Use it alone when tool returns are short. Caveman reports no_candidate or not_smaller and sends the original when there is nothing worth cutting.

Use it alone when your agent mostly returns structured output. The capability only offers the recovery tool on plain text-output runs with automatic tool choice. With a structured output_type or a forced tool choice, it does not register recovery for that request, and without recovery the request goes out uncompressed.

And if Tool Output Limits already caps your big returns and answers stay good, that may be all you need. It runs at the source and needs no extra process.

When does adding Caveman pay off?

It pays off when the agent keeps needing its tool output. A support agent that pulls a 20k-token order history at step 2 and reasons over it for eight more steps sends it eight more times. Truncation loses the tail. Summarizing costs a model call and blurs detail. Clearing drops it. Caveman leaves the full return in the agent's history, sends a shorter copy, and keeps the exact text one caveman_retrieve call away.

After the first cut, Caveman resends the same shortened bytes on later requests, reported as reused, so the provider's cached prefix stays byte identical.

It stacks with the built-in capabilities. Compaction and history processors decide what history is sent. Caveman shrinks the tool returns inside the request that actually goes out.

If the runtime is unreachable, requests go out with the original messages. After three failures in a row the adapter skips the runtime for 30 seconds. The agent keeps running either way.

How to add Caveman to Pydantic AI

1. pip install caveman-sdk "caveman-middleware[pydantic-ai]==0.1.0a1"
2. Start the runtime with CAVEMAN_MODE=compress caveman start.
3. Add CavemanCapability(runtime=runtime, scope=scope) to capabilities.
4. Log reports and look for applied and reused.

The Pydantic AI switch guide walks through each step with scope and rollback. The Python middleware docs list every Python adapter, and how to measure agent cost covers judging the change on finished tasks.

FAQ

Does Caveman replace Pydantic AI? No. Pydantic AI still runs the agent, validates outputs and calls the model. Caveman only shortens tool-return text in the copy of the request sent to the provider.

Is it a capability or a model wrapper? Both exist. CavemanCapability is the one to use, because it registers the recovery tool. with_caveman_model wraps a model on its own with no recovery executor.

Does it change my message history? No. The adapter is copy-on-write: it builds a shortened view for the outbound request and leaves the agent's messages as they were.

How is this different from Tool Output Limits? Tool Output Limits acts on the tool return at the source. Caveman leaves the return as is in history and shortens only what is sent, with the exact original recoverable by the model.

Sources

• Pydantic AI overview
• Pydantic AI capabilities
• Pydantic AI message history
• pydantic/pydantic-ai on GitHub
• Caveman middleware: Python adapters
• Caveman middleware: supported frameworks