---
title: "Add Caveman to PromptLayer without touching prompts"
description: "This guide adds Caveman Middleware to a Python agent that gets its prompts from PromptLayer. The registry, release labels and request logs stay as they are. Caveman never touches the system prompt or "
canonical: https://caveman.ac/switch/promptlayer
last-updated: 2026-10-04
---

# Add Caveman to PromptLayer without touching prompts

This guide adds Caveman Middleware to a Python agent that gets its prompts from PromptLayer. The registry, release labels and request logs stay as they are. Caveman never touches the system prompt or user messages, so the version PromptLayer serves reaches the model unchanged. What gets shorter is the tool output your agent collects along the way. For the background, read [Caveman vs PromptLayer](/compare/promptlayer).

## What do you need before you start?

You need an agent that makes its own model calls with the OpenAI Python SDK and fetches its prompt from the registry. Calls made with `pl.run()` can't be wrapped, because PromptLayer's SDK builds its own client for them. The `pl.openai` proxy can't be wrapped either, since Caveman's adapter takes a plain `OpenAI` client. If you use either, this guide moves one agent to a direct call.

Pick an agent whose tools return a lot of text: JSON API responses, file reads, logs. `caveman-middleware` needs Python 3.13 or newer, and its OpenAI adapter accepts `openai >=3.10 <4`. Outside that range it passes calls through untouched.

## Step 1: Start the Caveman runtime

The middleware sends candidates to a local runtime that compresses them and keeps the originals on disk:

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

Plain `caveman start` runs in `record` mode and changes nothing. Keep the runtime in `compress` and set the baseline on the client, so the two arms differ in one setting.

## Step 2: Fetch the template, call a wrapped client, log the call

```bash
pip install promptlayer caveman-sdk "caveman-middleware[openai]==0.1.0a1"
```

`with_caveman_openai_tools` wraps a plain OpenAI client and adds `caveman_retrieve` to your tool list, with its executor in `loop.functions`. Dispatch every tool call through that table so recovery works. `openai_to_blueprint` is the converter from PromptLayer's [custom logging docs](https://docs.promptlayer.com/features/observability/request-logs/custom-logging).

```python
import json, os, time
from openai import OpenAI
from promptlayer import PromptLayer
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

pl = PromptLayer()  # reads PROMPTLAYER_API_KEY
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(mode=mode)
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

def answer(ticket_id, question):
    tpl = pl.templates.get("support-agent", {"version": 7, "input_variables": {"question": question}})
    request = tpl["llm_kwargs"]
    loop = with_caveman_openai_tools(
        OpenAI(), runtime=runtime, scope=Scope("support", ticket_id, "main", "0"),
        protocol="openai-chat", tools=TOOLS, functions=FUNCTIONS)  # your tools
    messages = list(request["messages"])
    while True:
        start = time.time()
        response = loop.client.chat.completions.create(**{**request, "messages": messages, "tools": loop.tools})
        log_input, log_output = openai_to_blueprint(messages, response)
        pl.log_request(provider="openai", model=response.model, input=log_input, output=log_output,
            request_start_time=start, request_end_time=time.time(),
            prompt_name="support-agent", prompt_version_number=tpl["version"],
            input_tokens=response.usage.prompt_tokens, output_tokens=response.usage.completion_tokens,
            tags=[f"caveman-{mode}"], api_type="chat-completions")
        message = response.choices[0].message
        if not message.tool_calls:
            return message.content
        messages.append(message.model_dump(exclude_none=True))
        for call in message.tool_calls:
            result = loop.functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id, "content": str(result)})
```

Three details matter here. The version is pinned (`"version": 7`) so a release can't land mid-test; go back to `"label": "prod"` afterwards. PromptLayer warns that `llm_kwargs` is provider-specific and can change shape, so keep an eye on it after SDK upgrades. And the `Scope` is per conversation: its four parts key the stored originals.

## Which copy ends up in the log?

`messages` is your own history, so the logged input shows the full tool output. `input_tokens` comes from `response.usage`, the provider's count of what was actually sent, which is the shortened request. A log showing 20,000 tokens of JSON with a much smaller input count is working as intended.

If you prefer `PromptLayer(enable_tracing=True)` over `log_request()` (it needs the `promptlayer[otel-genai-instrumentation]` extra for provider spans), you get the same split: its OpenAI instrumentor records the messages passed to `chat.completions.create`, and token counts come from the response. Don't run both on the same call, or you will log it twice.

## Step 3: Run both arms and score them

```bash
CAVEMAN_CLIENT_MODE=record python eval_run.py
CAVEMAN_CLIENT_MODE=compress python eval_run.py
```

Same inputs, same model, same prompt version. In the `record` run the runtime measures each call and sends the original bytes.

Create a PromptLayer Table, choose **Import request history**, and add the final responses from both tags as rows. Score them with one scorecard. Score these logged outputs rather than adding a prompt column, since a prompt column runs through PromptLayer and skips your app, and Caveman with it.

## How do you check it worked?

1. **Scores.** The `caveman-compress` rows should hold the `caveman-record` rows on your scorecard.
2. **Tokens.** Compare the summed `input_tokens` per conversation across both tags.
3. **Recovery calls.** Count calls where the model asked for `caveman_retrieve`. Each one carries an original back and costs tokens.
4. **Prompt version.** Every log in both arms should show the same `prompt_version_number`.

For the per-call decision, pass `on_report=print` to `MiddlewareRuntime`. `applied` or `reused` means a tool result was shortened; `skipped` includes a reason such as `no_candidate`.

## How do you roll back?

Replace `loop.client` with a plain `OpenAI()` client, `loop.tools` with your original `TOOLS` and `loop.functions` with `FUNCTIONS`. The wrapper mutates nothing, so that is the whole revert. To switch off without a deploy, set `CAVEMAN_CLIENT_MODE=off`. Registry, labels and past logs were never touched. Keep the runtime up until live conversations finish, because handles the model already holds need the stored originals. Grants last 24 hours from their last use.

## Where to go next

- [Caveman vs PromptLayer](/compare/promptlayer) for where each tool fits
- [Python middleware docs](https://docs.caveman.so/docs/sdk/middleware/python) for the Anthropic, LangChain and LiteLLM adapters
- [PromptLayer Tables](https://docs.promptlayer.com/features/tables/overview) for scorecards
- [Measure agent cost](/guides/measure-agent-cost) for a fair comparison