Add Caveman to Cloudflare AI Gateway

This guide adds Caveman Middleware to a service that already calls models through Cloudflare AI Gateway's OpenAI-compatible endpoint. Your gateway, its cache settings, rate limits and spend limits all stay as they are. You wrap the client in code, run one task twice, and read the difference in Cloudflare's logs. For the why, see Caveman vs Cloudflare AI Gateway.

Where can the Caveman runtime run?

The runtime needs a process and a disk, so it runs next to a Node or Python service: same host, same pod, or the same container network. It can't run inside a Cloudflare Worker.

If your model calls come from a Worker, do the first test from a Node or Python service or your laptop against the same gateway. A Worker in production would need a runtime you keep running elsewhere, reached over HTTPS with an auth token and allowRemoteContent: true on the client. The deployment docs cover that setup. We haven't published a Workers-tested configuration, so treat it as your own experiment.

Step 1: Start the runtime

npm install -g @caveman-ai/cli && caveman setup --install
caveman start

This starts in record mode, which measures calls and changes nothing. You'll use it for the baseline. Check it answers:

curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'

Step 2: Wrap the client

Install the Python packages (Python 3.13 or newer):

python -m pip install 'caveman-sdk==1.1.0' 'caveman-middleware[openai]==0.1.0a1'

Then wrap the client you already point at the /compat endpoint. The base URL doesn't change:

import json
import os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

account = os.environ["CF_ACCOUNT_ID"]
client = OpenAI(
    base_url=f"https://gateway.ai.cloudflare.com/v1/{account}/my-gateway/compat",
    api_key=os.environ["CF_API_TOKEN"],
)

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode, deadline_ms=500, on_report=print)
scope = Scope("research-agent", conversation_id, "main", "0")
if mode != "off":
    runtime.ready()

loop = with_caveman_openai_tools(
    client, runtime=runtime, scope=scope, protocol="openai-chat",
    tools=[fetch_page_tool], functions={"fetch_page": fetch_page},
)

for _ in range(8):
    response = loop.client.chat.completions.create(model="openai/gpt-5.2", messages=history, tools=loop.tools)
    message = response.choices[0].message
    history.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        output = loop.functions[call.function.name](json.loads(call.function.arguments))
        history.append({"role": "tool", "tool_call_id": call.id,
                        "content": output if isinstance(output, str) else json.dumps(output)})

loop.tools is your tool list plus caveman_retrieve, and loop.functions runs both. Dispatching every call through loop.functions is what lets the model fetch an original back. history stays yours and keeps the full tool output; only the outbound copy is shortened.

If you pass provider keys with a cf-aig-authorization header, keep doing it. The adapter rewrites the request body and leaves headers alone.

In TypeScript the equivalent is withCavemanOpenAITools from @caveman-ai/middleware/openai, with the same fetch you gave the OpenAI client passed in its options. The middleware docs show the setup, and the Helicone guide has a full TypeScript example that only differs in the base URL.

Step 3: Run both arms with the cache off

Cloudflare's cache matches the entire request, and a cache hit never reaches the provider. A cached arm next to an uncached one tells you nothing about compression. Turn caching off for the test gateway, or skip it on both arms with the cf-aig-skip-cache header.

Run the task once with the runtime in record mode. That is the baseline, through the same code. Then restart the runtime in compress mode and run it again with a new conversation_id:

CAVEMAN_MODE=compress caveman start

Same model, same prompt, same tool data. If you want the two arms fully apart in analytics, create a second gateway with identical settings and point the candidate at its ID.

How do you know it worked?

Read the reports first. Each call prints a decision: applied means a tool result was replaced, reused means an earlier replacement was kept so the prefix stays stable, skipped carries a reason like no_candidate or not_smaller. All skipped means there was nothing eligible in this task.

Then open the gateway logs. Compare input tokens per request across the two arms, and look for caveman_retrieve calls in the candidate. A recovery call carries the original back and costs tokens, so add it in. Cloudflare's cost column is priced by Cloudflare and is a fair common scale for both arms. Caveman's local estimates are labelled inferred and belong in a separate column.

Finally, check the answer. A cheaper run with a worse result is a failed test. The cost measurement guide covers counting failed attempts and retries.

If you use DLP or guardrails on this gateway, look at what they flagged in the candidate arm. They now inspect the shortened tool output, since that is what the gateway receives.

How do you roll back?

Set CAVEMAN_CLIENT_MODE=off and redeploy. The snippet passes it as mode on the runtime client and skips ready(), which raises in off mode. Nothing goes to the runtime, and requests reach Cloudflare as your code built them. To remove Caveman entirely, call client instead of loop.client, pass your original tool list, and delete the wrapper.

Restarting the runtime with plain caveman start also stops compression while keeping the measurements.

If the runtime goes down, the adapter sends the original request instead, so gateway traffic keeps flowing. Re-enable your cache settings on the gateway if you changed them for the test.

Where to go next

• Caveman Middleware, Python adapters and supported frameworks
• Cloudflare's OpenAI-compatible endpoint and caching docs
• Caveman vs Cloudflare AI Gateway and how prompt caching works