AI gateway migration: a reversible test plan for agent traffic

Run the new gateway beside the old one, move a single caller, and keep the change small enough to undo in a minute. Nothing here requires you to turn the old path off.

The constraint: you cannot prove a gateway is equivalent by reading its docs. A migration changes model naming, credential custody, policy enforcement, retries, cache behavior, stream events and the usage records your cost report is built on. You find out which of those moved by sending your own requests through both paths and comparing what comes back.

Write down the values you will type back before you change one

Rollback is retyping a short list, so write the list first: the caller's base URL, the model string exactly as the caller sends it, where the key comes from, the timeouts, and the gateway policy you rely on. That last one usually means allowed models, budgets, provider pins, fallbacks and cache policy.

Put it in the ticket, not in your shell history. Keep one reproducible task with a result you can check by looking at it, and use staging credentials with disposable tools for anything with a side effect.

The switching index covers the configuration boundaries for each specific alternative.

Give the candidate its own listener and leave production alone

Install the CLI and its companion binaries on the same host as the test caller:

npm install -g @caveman-ai/cli
caveman setup --install

Create a dedicated caveman-test.yaml:

label: gateway-migration-test
mode: record
listen: 127.0.0.1:8787
providers: {}
compat: {}

Start it in its own terminal on a machine where the port is free:

caveman start --config ./caveman-test.yaml

Record mode leaves the request unchanged. If something already serves 8787, move to an isolated environment rather than restarting shared infrastructure.

Send one real request through the candidate before you believe anything

Point the test caller at the listener and send the simplest request your application already sends. For an OpenAI Chat Completions app that is http://127.0.0.1:8787/openai/v1/chat/completions, with your key supplied through your usual secret mechanism and a model your account can actually use. It bills like any other request, and it proves exactly one thing: the bytes arrive.

That path goes straight to the provider. It carries none of your old gateway's per-caller keys, budgets or fallbacks. See provider mounts for other protocols and LiteLLM switching for its documented combined paths.

The Caveman destination that carries those controls is Caveman Cloud: one URL, with caching and compression running eval gated, and spend that resolves per member, per key, per workflow and per model. Routing is still in development. Cloud is open for sign-up with a Free plan; move one workload first rather than cutting everything over at once. The listener above is the part you can run with no account.

Test the shapes that break, not the one that works

Send the same task down both paths and diff the results. Chat text passes everywhere, which is why it proves nothing.

Decide which layer owns retries before you cut over. If the SDK and the gateway both retry, a tool with a side effect can run twice and nothing in your logs will say so.

Move one caller and leave it there for a week

Cut over one caller or one task family. Then compare request counts, model identity, input and output and cache token fields, error class, latency and how many tasks were accepted. Judge the result on cost per completed task, not on the first request's apparent discount.

Leave compression out of this migration. Changing the path and the payload together makes both unmeasurable; test that afterwards with the compression guide.

Rehearse the rollback before you need it

Write the trigger down before cutover: the error rate, the p95 and the acceptance rate that send you back. An unwritten trigger turns into an argument at 2am.

Then run the drill once in staging, timed:

1. Put back the saved base URL, model string and key source.
2. Start a fresh session. A process already running keeps the candidate's config, so verify on a new one.
3. Confirm a request reaches the expected provider and that the old policy still refuses what it used to refuse.
4. Leave the candidate's configuration in place. Delete it only once no request is in flight and nothing still points at text it stored.

If the drill takes longer than the incident you would be having, it is not a rollback.

Two mistakes turn a reversible migration into an outage

The migration that also changed the prompt. Someone tidies the system prompt during the move. Quality shifts, and the gateway gets blamed for a week. Fix: one variable per change, and the prompt is a separate release.

The migration that quietly dropped the policy. The new path answers correctly, so nobody notices that per-caller model restrictions stopped applying. Spot it by sending a deliberately forbidden model request on day one. Fix: treat a forwarded forbidden request as a failed migration, whatever the answer said.

Call the migration done when the old path is boring

You are finished when a week of traffic through the new path shows the same acceptance rate, no new error class, usage fields that reconcile against the provider, and a rollback you have actually executed once. Until then, keep the old endpoint, alias and credentials exactly where they are.

For a full replacement rather than a trial, the selection guide covers the deployment and operating-cost questions this plan does not answer.