When routing to a cheaper model actually saves money

Sending a request to a cheaper model saves money only when the request was expensive for a reason the cheaper model removes. Two things quietly eat the discount: the cache you break by moving the request, and the cheap attempt that fails and gets retried on the expensive model anyway.

The constraint is the unit. You are not comparing price per million tokens. You are comparing the cost of one finished task, and both of those effects are invisible until you count at that level.

Price the two traps before you build anything

The broken prefix. A long agent session keeps resending the same opening context: the system prompt, the tool definitions, the files it has read. Providers bill repeated context far below fresh input once it is warm. Move that request to a different model and the prefix is cold again, so you pay full rate for every token you used to get cheaply. On a long session the lost reuse can outweigh the whole list-price gap. The caching guide explains why fewer tokens and lower price are not the same thing.

The double payment. A cheap first attempt with an expensive fallback costs more than the expensive model alone, every time the cheap one fails. It pays only if it succeeds often enough to cover the losses, and the arithmetic includes the repair work the failure created.

attempt cost = routing cost + selected-model calls + fallback calls + repair calls

Routing cost is real. If your policy runs an embedding model or a small LLM call to decide, that call is part of the system you are pricing, in dollars and in latency.

Decide which models are allowed before you ask which is cheapest

Eligibility comes first and it is a filter, not a preference. A model that fails any row here is not a cheap option, it is not an option.

Never make a request fit a cheaper model by dropping a field the task needs. Keep one known permitted model as the answer for anything the router cannot safely judge.

Prove two fixed models can do the job before you write a router

Pick representative tasks. Run each candidate model with the same tools, the same starting state and the same acceptance tests. Record full usage, latency, retries and failures.

Include the hard cases and at least one task that hinges on an exact detail from earlier in the session. A model that clears a public benchmark can still fall apart on your tool schemas or a twenty step workflow.

Keep the tuning set and the acceptance set separate, or you will tune yourself a threshold that only works on the data you tuned it with. Agent evaluations covers that split and the quality floor.

Route whole tasks first, single calls later

Per-call routing gives you the most control and does the most damage to a running session: it is the pattern that breaks prefixes mid-conversation. Per-task routing keeps continuity and sometimes leaves an expensive model doing easy intermediate work.

Start at a boundary your application can name out loud. A task type or a stated capability requirement is auditable. A confidence score nobody can explain is not. Log why a model was allowed, and what happened when none was.

Three routing patterns look cheap on a dashboard and are not

The router that refuses hard work. It sends everything difficult to a refusal or a short apology. Cost per request falls, accepted completions fall further, and the product gets worse while the graph gets better. Grade on accepted completions.

Three components retrying at once. The client retries, the gateway retries, the router retries. You get more provider calls than anyone intended and no single component can see the total. Decide which one owns retries and fallbacks, and turn the others off.

Two experiments in one rollout. Routing changed and compression turned on in the same week. Hold the route fixed while you change context, check the finished task, then test the combination once both have evidence of their own.

Caveman Router is not released yet, and its savings stay inferred

Caveman Router routes inside the Caveman Platform request path. Each call goes to the most efficient model in your pool that passes your evals, and when nothing passes, the traffic stays where it is. Rollout is eval gated with automatic rollback on a regression, which is the eligibility table above enforced by the system rather than by a spreadsheet someone updates.

Its published figure is 39.6% lower spend on replayed-traffic benchmarks, modelled at list prices. Repeat the basis with the number: replayed traffic priced against a public catalog, not a customer invoice, and not a prediction for your workload. On your own traffic a Router number starts as inferred and stays there until a provider bill shows the change.

Router is live in Claude Code through the caveman CLI, and Caveman Cloud is open for sign-up; routing inside the Cloud request path is in development. The LiteLLM integration keeps inference and provider keys inside LiteLLM and adds a model-selection path on top. The callback can keep the original route for request shapes it should not touch, such as tools, structured output, reasoning or a provider pin, so a pilot does not automatically cover every request your agent sends.

Read Caveman for LiteLLM and use the configuration supplied for your pilot. There is no public anonymous Router endpoint to paste into an SDK, and what Router is meant to do covers the rest.

For alternatives, RouteLLM is an open-source routing experiment with published methods and thresholds, Not Diamond and Martian sell routing products, and gateways like LiteLLM and OpenRouter own fallback behavior you will have to account for either way.

Ship it behind a rollback, then recheck it when anything moves

Keep the original route callable. Test the candidate on held-out tasks before it touches a production default. Record which models were allowed, the model chosen, the baseline, the decision latency, the failure behavior and the final result.

For a live pilot, pick one task family and write down the rollback trigger in advance. Test the ugly cases: a model that is down, a decision that times out, missing usage numbers, a caller whose key cannot reach one candidate. The caller's contract holds in all of them.

A threshold belongs to one model pair, one dataset and one deployment. New model versions, edited prompts and different tool output all invalidate it. Rerun the fixed baselines when those change, use cost per completed task as the economic result, and keep quality, latency and policy failures on the same page as the dollars.