---
title: "How to add Caveman to Portkey without changing configs"
description: "This guide adds Caveman Middleware to an app that already sends its LLM calls through Portkey. Portkey stays the gateway: same base URL, same headers, same configs and guardrails. Caveman shortens lar"
canonical: https://caveman.ac/switch/portkey
last-updated: 2026-10-04
---

# How to add Caveman to Portkey without changing configs

This guide adds Caveman Middleware to an app that already sends its LLM calls through Portkey. Portkey stays the gateway: same base URL, same headers, same configs and guardrails. Caveman shortens large tool results in the outgoing copy of each request, so Portkey forwards fewer tokens. Try it on one agent first. For the reasoning, read the [Portkey comparison](/compare/portkey).

## What you need before you start

- An app that calls Portkey through the OpenAI SDK, the Anthropic SDK, or a framework such as the Vercel AI SDK, LangChain or Pydantic AI.
- Node.js 22.13 or newer for the `caveman` CLI. Python code needs Python 3.13 or newer.
- One agent whose tools return big text, and a task with a known answer.

## Step 1: Start the Caveman runtime

```bash
npm install -g @caveman-ai/cli && caveman setup --install
CAVEMAN_MODE=compress caveman start
```

The runtime listens on `127.0.0.1:8787` and keeps originals on local disk. If you also run Portkey's open-source gateway on this machine, it wants port 8787 too: start Caveman with `--port 8788` and use that in the endpoint below. Confirm it is in compress mode:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, recovery}'
```

## Step 2: Wrap the client you point at Portkey

In Python, install the adapter for the SDK you use. For OpenAI:

```bash
pip install caveman-sdk "caveman-middleware[openai]==0.1.0a1" portkey-ai
```

Build the OpenAI client exactly as Portkey documents it, then wrap it. `with_caveman_openai_tools` takes your tool definitions and the functions that run them, and adds `caveman_retrieve` so the model can ask for an original back.

```python
import json, os
from openai import OpenAI
from portkey_ai import createHeaders
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="https://api.portkey.ai/v1",
    default_headers=createHeaders({"apiKey": os.environ["PORTKEY_API_KEY"],
                                   "provider": "@openai-prod"}),
)
mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode,
                            deadline_ms=500, on_report=print)
if mode != "off":
    runtime.ready()
scope = Scope("support-agent", conversation_id)

loop = with_caveman_openai_tools(client, runtime=runtime, scope=scope,
                                 protocol="openai-chat", tools=tools,
                                 functions={"search_tickets": search_tickets})

for _ in range(8):
    response = loop.client.chat.completions.create(
        model="gpt-5.5", messages=messages, tools=loop.tools)
    message = response.choices[0].message
    messages.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        result = loop.functions[call.function.name](json.loads(call.function.arguments))
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": result if isinstance(result, str) else json.dumps(result)})
```

`tools`, `messages` and `search_tickets` are yours; use your own provider slug and model. Keep `conversation_id` the same for the whole conversation, since recovery handles only resolve under the scope that issued them. The adapter needs `openai >=3.10 <4`.

In TypeScript with the Vercel AI SDK, keep your Portkey provider and wrap the call options:

```ts
import { stepCountIs, streamText } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { createMiddlewareRuntime } from "@caveman-ai/sdk/middleware";
import { withCaveman } from "@caveman-ai/middleware/ai-sdk";

const portkey = createOpenAI({
  baseURL: "https://api.portkey.ai/v1",
  headers: { "x-portkey-api-key": process.env.PORTKEY_API_KEY!, "x-portkey-provider": "@openai-prod" },
});
const mode = (process.env.CAVEMAN_CLIENT_MODE ?? "compress") as "off" | "record" | "compress";
const runtime = createMiddlewareRuntime({ endpoint: "http://127.0.0.1:8787", mode });
const scope = { namespace: "support", session_id: conversationId, branch_id: "main", cache_epoch: "0" };

if (mode !== "off") await runtime.ready();

const result = streamText({
  ...withCaveman({ model: portkey.chat("gpt-5.5"), tools }, { runtime, scope }),
  messages,
  stopWhen: stepCountIs(8),
});
```

Other frameworks follow the same idea: configure the model to use Portkey as you do now, then add the Caveman adapter from the [framework list](https://docs.caveman.so/docs/sdk/middleware/frameworks).

## How to check it worked

1. **The reports.** In Python, `on_report=print` logs one decision per call. Look for `applied` or `reused` with a transform like `caveman.engine.json.v1` on calls that carry big tool output. `unsupported_version` means your `openai` package is outside the accepted range, and the call went out unchanged.
2. **Portkey logs.** Open the request in Portkey. The tool message should start with `[caveman: shortened; exact original via caveman_retrieve handle=cmw_…]`. Sending an `x-portkey-trace-id` header makes the test runs easy to find.
3. **Tokens.** Run the same task with `CAVEMAN_CLIENT_MODE=off` and with `CAVEMAN_CLIENT_MODE=compress`, and compare input tokens in Portkey's analytics or in `response.usage`. Those counts come from the provider.
4. **Quality.** Check the answer, and count `caveman_retrieve` calls. If the model keeps asking for originals, the saving shrinks.

Also re-run a guardrail you rely on with a known bad input, so you know it still fires on the shortened copy. The [measurement guide](/guides/measure-agent-cost) covers repeats, caching and retries.

## How to roll back

Set `CAVEMAN_CLIENT_MODE=off` and restart the app. The snippets pass it as `mode` on the runtime client and skip `ready()`, so the adapter sends every request to Portkey unchanged without touching the call site. To remove Caveman completely, call `client` instead of `loop.client` and pass your own `tools` again, or drop `withCaveman(...)` in TypeScript. Nothing in Portkey needs undoing.

Sources: [Portkey request headers](https://portkey.ai/docs/api-reference/inference-api/headers), [Portkey gateway README](https://github.com/Portkey-AI/gateway), [Caveman Python middleware](https://docs.caveman.so/docs/sdk/middleware/python), [Caveman TypeScript middleware](https://docs.caveman.so/docs/sdk/middleware).