How to add Caveman to Portkey without changing configs

This guide adds Caveman Middleware to an app that already sends its LLM calls through Portkey. Portkey stays the gateway: same base URL, same headers, same configs and guardrails. Caveman shortens large tool results in the outgoing copy of each request, so Portkey forwards fewer tokens. Try it on one agent first. For the reasoning, read the Portkey comparison.

What you need before you start

• An app that calls Portkey through the OpenAI SDK, the Anthropic SDK, or a framework such as the Vercel AI SDK, LangChain or Pydantic AI.
• Node.js 22.13 or newer for the caveman CLI. Python code needs Python 3.13 or newer.
• One agent whose tools return big text, and a task with a known answer.

Step 1: Start the Caveman runtime

npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start

The runtime listens on 127.0.0.1:8787 and keeps originals on local disk. If you also run Portkey's open-source gateway on this machine, it wants port 8787 too: start Caveman with --port 8788 and use that in the endpoint below. Confirm it is in compress mode:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, recovery}'

Step 2: Wrap the client you point at Portkey

In Python, install the adapter for the SDK you use. For OpenAI:

pip install caveman-sdk "caveman-middleware[openai]==0.1.0a1" portkey-ai

Build the OpenAI client exactly as Portkey documents it, then wrap it. with_caveman_openai_tools takes your tool definitions and the functions that run them, and adds caveman_retrieve so the model can ask for an original back.

import json, os
from openai import OpenAI
from portkey_ai import createHeaders
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="https://api.portkey.ai/v1",
    default_headers=createHeaders({"apiKey": os.environ["PORTKEY_API_KEY"],
                                   "provider": "@openai-prod"}),
)
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()
scope = Scope("support-agent", conversation_id)

loop = with_caveman_openai_tools(client, runtime=runtime, scope=scope,
                                 protocol="openai-chat", tools=tools,
                                 functions={"search_tickets": search_tickets})

for _ in range(8):
    response = loop.client.chat.completions.create(
        model="gpt-5.5", messages=messages, tools=loop.tools)
    message = response.choices[0].message
    messages.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        result = loop.functions[call.function.name](json.loads(call.function.arguments))
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": result if isinstance(result, str) else json.dumps(result)})

tools, messages and search_tickets are yours; use your own provider slug and model. Keep conversation_id the same for the whole conversation, since recovery handles only resolve under the scope that issued them. The adapter needs openai >=3.10 <4.

In TypeScript with the Vercel AI SDK, keep your Portkey provider and wrap the call options:

import { stepCountIs, streamText } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const portkey = createOpenAI({
  baseURL: "https://api.portkey.ai/v1",
  headers: { "x-portkey-api-key": process.env.PORTKEY_API_KEY!, "x-portkey-provider": "@openai-prod" },
});
const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({ endpoint: "http://127.0.0.1:8787", mode });
const scope = { namespace: "support", session_id: conversationId, branch_id: "main", cache_epoch: "0" };

if (mode !== "off") await runtime.ready();

const result = streamText({
  ...withCaveman({ model: portkey.chat("gpt-5.5"), tools }, { runtime, scope }),
  messages,
  stopWhen: stepCountIs(8),
});

Other frameworks follow the same idea: configure the model to use Portkey as you do now, then add the Caveman adapter from the framework list.

How to check it worked

1. The reports. In Python, on_report=print logs one decision per call. Look for applied or reused with a transform like caveman.engine.json.v1 on calls that carry big tool output. unsupported_version means your openai package is outside the accepted range, and the call went out unchanged.
2. Portkey logs. Open the request in Portkey. The tool message should start with [caveman: shortened; exact original via caveman_retrieve handle=cmw_…]. Sending an x-portkey-trace-id header makes the test runs easy to find.
3. Tokens. Run the same task with CAVEMAN_CLIENT_MODE=off and with CAVEMAN_CLIENT_MODE=compress, and compare input tokens in Portkey's analytics or in response.usage. Those counts come from the provider.
4. Quality. Check the answer, and count caveman_retrieve calls. If the model keeps asking for originals, the saving shrinks.

Also re-run a guardrail you rely on with a known bad input, so you know it still fires on the shortened copy. The measurement guide covers repeats, caching and retries.

How to roll back

Set CAVEMAN_CLIENT_MODE=off and restart the app. The snippets pass it as mode on the runtime client and skip ready(), so the adapter sends every request to Portkey unchanged without touching the call site. To remove Caveman completely, call client instead of loop.client and pass your own tools again, or drop withCaveman(...) in TypeScript. Nothing in Portkey needs undoing.

Sources: Portkey request headers, Portkey gateway README, Caveman Python middleware, Caveman TypeScript middleware.