Add Caveman to PromptLayer without touching prompts

This guide adds Caveman Middleware to a Python agent that gets its prompts from PromptLayer. The registry, release labels and request logs stay as they are. Caveman never touches the system prompt or user messages, so the version PromptLayer serves reaches the model unchanged. What gets shorter is the tool output your agent collects along the way. For the background, read Caveman vs PromptLayer.

What do you need before you start?

You need an agent that makes its own model calls with the OpenAI Python SDK and fetches its prompt from the registry. Calls made with pl.run() can't be wrapped, because PromptLayer's SDK builds its own client for them. The pl.openai proxy can't be wrapped either, since Caveman's adapter takes a plain OpenAI client. If you use either, this guide moves one agent to a direct call.

Pick an agent whose tools return a lot of text: JSON API responses, file reads, logs. caveman-middleware needs Python 3.13 or newer, and its OpenAI adapter accepts openai >=3.10 <4. Outside that range it passes calls through untouched.

Step 1: Start the Caveman runtime

The middleware sends candidates to a local runtime that compresses them and keeps the originals on disk:

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

Plain caveman start runs in record mode and changes nothing. Keep the runtime in compress and set the baseline on the client, so the two arms differ in one setting.

Step 2: Fetch the template, call a wrapped client, log the call

pip install promptlayer caveman-sdk "caveman-middleware[openai]==0.1.0a1"

with_caveman_openai_tools wraps a plain OpenAI client and adds caveman_retrieve to your tool list, with its executor in loop.functions. Dispatch every tool call through that table so recovery works. openai_to_blueprint is the converter from PromptLayer's custom logging docs.

import json, os, time
from openai import OpenAI
from promptlayer import PromptLayer
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

pl = PromptLayer()  # reads PROMPTLAYER_API_KEY
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(mode=mode)
if mode != "off":
    runtime.ready()  # raises if the runtime is unreachable

def answer(ticket_id, question):
    tpl = pl.templates.get("support-agent", {"version": 7, "input_variables": {"question": question}})
    request = tpl["llm_kwargs"]
    loop = with_caveman_openai_tools(
        OpenAI(), runtime=runtime, scope=Scope("support", ticket_id, "main", "0"),
        protocol="openai-chat", tools=TOOLS, functions=FUNCTIONS)  # your tools
    messages = list(request["messages"])
    while True:
        start = time.time()
        response = loop.client.chat.completions.create(**{**request, "messages": messages, "tools": loop.tools})
        log_input, log_output = openai_to_blueprint(messages, response)
        pl.log_request(provider="openai", model=response.model, input=log_input, output=log_output,
            request_start_time=start, request_end_time=time.time(),
            prompt_name="support-agent", prompt_version_number=tpl["version"],
            input_tokens=response.usage.prompt_tokens, output_tokens=response.usage.completion_tokens,
            tags=[f"caveman-{mode}"], api_type="chat-completions")
        message = response.choices[0].message
        if not message.tool_calls:
            return message.content
        messages.append(message.model_dump(exclude_none=True))
        for call in message.tool_calls:
            result = loop.functions[call.function.name](json.loads(call.function.arguments))
            messages.append({"role": "tool", "tool_call_id": call.id, "content": str(result)})

Three details matter here. The version is pinned ("version": 7) so a release can't land mid-test; go back to "label": "prod" afterwards. PromptLayer warns that llm_kwargs is provider-specific and can change shape, so keep an eye on it after SDK upgrades. And the Scope is per conversation: its four parts key the stored originals.

Which copy ends up in the log?

messages is your own history, so the logged input shows the full tool output. input_tokens comes from response.usage, the provider's count of what was actually sent, which is the shortened request. A log showing 20,000 tokens of JSON with a much smaller input count is working as intended.

If you prefer PromptLayer(enable_tracing=True) over log_request() (it needs the promptlayer[otel-genai-instrumentation] extra for provider spans), you get the same split: its OpenAI instrumentor records the messages passed to chat.completions.create, and token counts come from the response. Don't run both on the same call, or you will log it twice.

Step 3: Run both arms and score them

CAVEMAN_CLIENT_MODE=record python eval_run.py
CAVEMAN_CLIENT_MODE=compress python eval_run.py

Same inputs, same model, same prompt version. In the record run the runtime measures each call and sends the original bytes.

Create a PromptLayer Table, choose Import request history, and add the final responses from both tags as rows. Score them with one scorecard. Score these logged outputs rather than adding a prompt column, since a prompt column runs through PromptLayer and skips your app, and Caveman with it.

How do you check it worked?

1. Scores. The caveman-compress rows should hold the caveman-record rows on your scorecard.
2. Tokens. Compare the summed input_tokens per conversation across both tags.
3. Recovery calls. Count calls where the model asked for caveman_retrieve. Each one carries an original back and costs tokens.
4. Prompt version. Every log in both arms should show the same prompt_version_number.

For the per-call decision, pass on_report=print to MiddlewareRuntime. applied or reused means a tool result was shortened; skipped includes a reason such as no_candidate.

How do you roll back?

Replace loop.client with a plain OpenAI() client, loop.tools with your original TOOLS and loop.functions with FUNCTIONS. The wrapper mutates nothing, so that is the whole revert. To switch off without a deploy, set CAVEMAN_CLIENT_MODE=off. Registry, labels and past logs were never touched. Keep the runtime up until live conversations finish, because handles the model already holds need the stored originals. Grants last 24 hours from their last use.

Where to go next

• Caveman vs PromptLayer for where each tool fits
• Python middleware docs for the Anthropic, LangChain and LiteLLM adapters
• PromptLayer Tables for scorecards
• Measure agent cost for a fair comparison