---
title: "Add Caveman to Cloudflare AI Gateway"
description: "This guide adds Caveman Middleware to a service that already calls models through Cloudflare AI Gateway's OpenAI-compatible endpoint. Your gateway, its cache settings, rate limits and spend limits all"
canonical: https://caveman.ac/switch/cloudflare-ai-gateway
last-updated: 2026-10-04
---

# Add Caveman to Cloudflare AI Gateway

This guide adds Caveman Middleware to a service that already calls models through Cloudflare AI Gateway's OpenAI-compatible endpoint. Your gateway, its cache settings, rate limits and spend limits all stay as they are. You wrap the client in code, run one task twice, and read the difference in Cloudflare's logs. For the why, see [Caveman vs Cloudflare AI Gateway](/compare/cloudflare-ai-gateway).

## Where can the Caveman runtime run?

The runtime needs a process and a disk, so it runs next to a Node or Python service: same host, same pod, or the same container network. It can't run inside a Cloudflare Worker.

If your model calls come from a Worker, do the first test from a Node or Python service or your laptop against the same gateway. A Worker in production would need a runtime you keep running elsewhere, reached over HTTPS with an auth token and `allowRemoteContent: true` on the client. The [deployment docs](https://docs.caveman.so/docs/sdk/middleware/deployment) cover that setup. We haven't published a Workers-tested configuration, so treat it as your own experiment.

## Step 1: Start the runtime

```bash
npm install -g @caveman-ai/cli && caveman setup --install
caveman start
```

This starts in `record` mode, which measures calls and changes nothing. You'll use it for the baseline. Check it answers:

```bash
curl -s http://127.0.0.1:8787/caveman/v1/middleware/capabilities | jq '{mode, persistent, recovery, retention_seconds}'
```

## Step 2: Wrap the client

Install the Python packages (Python 3.13 or newer):

```bash
python -m pip install 'caveman-sdk==1.1.0' 'caveman-middleware[openai]==0.1.0a1'
```

Then wrap the client you already point at the `/compat` endpoint. The base URL doesn't change:

```python
import json
import os
from openai import OpenAI
from caveman_cloud.middleware import MiddlewareRuntime, Scope
from caveman_middleware.openai import with_caveman_openai_tools

account = os.environ["CF_ACCOUNT_ID"]
client = OpenAI(
    base_url=f"https://gateway.ai.cloudflare.com/v1/{account}/my-gateway/compat",
    api_key=os.environ["CF_API_TOKEN"],
)

mode = os.environ.get("CAVEMAN_CLIENT_MODE", "compress")
runtime = MiddlewareRuntime(endpoint="http://127.0.0.1:8787", mode=mode, deadline_ms=500, on_report=print)
scope = Scope("research-agent", conversation_id, "main", "0")
if mode != "off":
    runtime.ready()

loop = with_caveman_openai_tools(
    client, runtime=runtime, scope=scope, protocol="openai-chat",
    tools=[fetch_page_tool], functions={"fetch_page": fetch_page},
)

for _ in range(8):
    response = loop.client.chat.completions.create(model="openai/gpt-5.2", messages=history, tools=loop.tools)
    message = response.choices[0].message
    history.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        break
    for call in message.tool_calls:
        output = loop.functions[call.function.name](json.loads(call.function.arguments))
        history.append({"role": "tool", "tool_call_id": call.id,
                        "content": output if isinstance(output, str) else json.dumps(output)})
```

`loop.tools` is your tool list plus `caveman_retrieve`, and `loop.functions` runs both. Dispatching every call through `loop.functions` is what lets the model fetch an original back. `history` stays yours and keeps the full tool output; only the outbound copy is shortened.

If you pass provider keys with a `cf-aig-authorization` header, keep doing it. The adapter rewrites the request body and leaves headers alone.

In TypeScript the equivalent is `withCavemanOpenAITools` from `@caveman-ai/middleware/openai`, with the same `fetch` you gave the OpenAI client passed in its options. The [middleware docs](https://docs.caveman.so/docs/sdk/middleware) show the setup, and the [Helicone guide](/switch/helicone) has a full TypeScript example that only differs in the base URL.

## Step 3: Run both arms with the cache off

Cloudflare's cache matches the entire request, and a cache hit never reaches the provider. A cached arm next to an uncached one tells you nothing about compression. Turn caching off for the test gateway, or skip it on both arms with the `cf-aig-skip-cache` header.

Run the task once with the runtime in `record` mode. That is the baseline, through the same code. Then restart the runtime in compress mode and run it again with a new `conversation_id`:

```bash
CAVEMAN_MODE=compress caveman start
```

Same model, same prompt, same tool data. If you want the two arms fully apart in analytics, create a second gateway with identical settings and point the candidate at its ID.

## How do you know it worked?

Read the reports first. Each call prints a decision: `applied` means a tool result was replaced, `reused` means an earlier replacement was kept so the prefix stays stable, `skipped` carries a reason like `no_candidate` or `not_smaller`. All `skipped` means there was nothing eligible in this task.

Then open the gateway logs. Compare input tokens per request across the two arms, and look for `caveman_retrieve` calls in the candidate. A recovery call carries the original back and costs tokens, so add it in. Cloudflare's cost column is priced by Cloudflare and is a fair common scale for both arms. Caveman's local estimates are labelled inferred and belong in a separate column.

Finally, check the answer. A cheaper run with a worse result is a failed test. The [cost measurement guide](/guides/measure-agent-cost) covers counting failed attempts and retries.

If you use DLP or guardrails on this gateway, look at what they flagged in the candidate arm. They now inspect the shortened tool output, since that is what the gateway receives.

## How do you roll back?

Set `CAVEMAN_CLIENT_MODE=off` and redeploy. The snippet passes it as `mode` on the runtime client and skips `ready()`, which raises in off mode. Nothing goes to the runtime, and requests reach Cloudflare as your code built them. To remove Caveman entirely, call `client` instead of `loop.client`, pass your original tool list, and delete the wrapper.

Restarting the runtime with plain `caveman start` also stops compression while keeping the measurements.

If the runtime goes down, the adapter sends the original request instead, so gateway traffic keeps flowing. Re-enable your cache settings on the gateway if you changed them for the test.

## Where to go next

- [Caveman Middleware](https://docs.caveman.so/docs/sdk/middleware), [Python adapters](https://docs.caveman.so/docs/sdk/middleware/python) and [supported frameworks](https://docs.caveman.so/docs/sdk/middleware/frameworks)
- [Cloudflare's OpenAI-compatible endpoint](https://developers.cloudflare.com/ai-gateway/usage/chat-completion/) and [caching docs](https://developers.cloudflare.com/ai-gateway/features/caching/)
- [Caveman vs Cloudflare AI Gateway](/compare/cloudflare-ai-gateway) and [how prompt caching works](/guides/prompt-caching)