Caveman vs RouteLLM: open-source LLM router compared

RouteLLM is the open-source reference for learned model routing: you run it yourself, you own the threshold, and it costs nothing but compute. Caveman Router is a hosted service, not released yet, aimed at subagents in Claude Code. If you want a router you can read, modify and run today, RouteLLM is the one.

Caveman's live products don't route at all. Caveman Middleware makes each request smaller, which helps whichever model RouteLLM picks.

What is RouteLLM?

RouteLLM is a Python framework from LMSYS for serving and evaluating LLM routers. It sends each query to one of two models, a strong expensive one or a weak cheap one, based on a learned estimate of whether the strong model would win. It is Apache-2.0 and has about 5.6k GitHub stars.

It ships four trained routers. Matrix factorization (mf) is the recommended one; the others use weighted Elo, a BERT classifier and an LLM classifier, plus a random baseline. They were trained on GPT-4 and Mixtral-8x7B, and the README says they generalize to other strong and weak pairs without retraining.

You use it in one of two ways. The Python Controller is a drop-in for the OpenAI client. Or you launch an OpenAI-compatible server and point any client at it. Either way, the model name carries the router and its threshold, like router-mf-0.11593. A calibration script picks the threshold for a target share of strong-model calls, using Chatbot Arena data or a sample of your own queries. There's also an evaluation harness for MMLU, GSM8K and MT Bench.

The LMSYS launch post from July 2024 reported cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against GPT-4 alone, while keeping 95% of GPT-4's performance. Those are benchmark results on single questions, and the post is careful about where routers struggled.

Can you use RouteLLM for AI agents?

Yes, with care. Reading the source turns up three details that matter for agent workloads.

• It routes on the last message. The controller takes messages[-1]["content"] as the prompt. A comment in the code says the routers were trained on first-turn data only. In an agent loop the last message is often a tool result, so the router may be scoring a log file instead of the task.
• The mf router embeds every prompt with OpenAI. It calls OpenAI's text-embedding-3-small for each routing decision. That's an extra request per call, and the prompt text goes to OpenAI.
• The server's request schema is chat-shaped. It types tools as a flat dictionary, and a standard nested OpenAI function definition fails its validation (checked against the schema with pydantic 2.11). For tool calling, use the Python Controller, which passes arguments straight through to LiteLLM.

Development has also been quiet: the last commit on the main branch is from August 2024. None of this makes RouteLLM a bad choice. It means you should expect to own it.

What is Caveman Router?

Caveman Router picks the model for each subagent Claude Code spawns and leaves the lead agent alone. It keeps your claude.ai login, so Claude turns stay on your Pro or Max plan, and if it can't answer in time the spawn runs as proposed. The client is open source under Apache-2.0; the routing service is hosted and not released yet. Caveman hasn't published a savings methodology for it yet, so there's no number to compare with the LMSYS results.

RouteLLM vs Caveman: feature comparison

When should you use RouteLLM?

Pick RouteLLM when you want to own the routing method. The code, the thresholds and the training pipeline are all yours, and there's no vendor to wait on.

It also fits best when your pool really is two models and the traffic looks like single-turn questions. That's the shape it was trained and evaluated on. For research, the eval harness alone is worth the install.

Caveman Router won't help you there. It is hosted, it targets Claude Code subagents, and it isn't open yet.

Does Caveman work with RouteLLM?

Yes, with one design choice to make. Routing lowers the price per token, compression lowers the token count, and the two multiply.

Caveman Middleware wraps the client that makes the model call. It shortens eligible tool results, like a 20,000-token JSON response or a repo file read, in a copy of the outbound request, and your app's history keeps the original. User messages and system prompts are never touched.

The clean setup is to use RouteLLM for the decision and Caveman on the call. Controller.route(prompt, router, threshold) returns the strong or weak model name. Route on the task text, then make the calls through Caveman's LiteLLM adapter, which fits because RouteLLM already uses LiteLLM model names. Your loop carries the caveman_retrieve tool and runs it when the model asks, which is what lets the adapter shorten anything. The router scores what the user asked, and the model gets a smaller request. The RouteLLM setup guide has the code.

What to avoid: pointing a Caveman-wrapped client at the RouteLLM server in an agent loop. The router would then score the compressed last message, and the schema issue above applies anyway.

Both Middleware packages are alpha, and the runtime is local with no Caveman account. The only published figures are 13 synthetic samples through the real runtime, counted as inferred token estimates. They describe sample content, not your bill. See the Middleware page.

How to test both

Freeze the route first. Send one model everything, add compression, and measure cost per completed task. Then turn the router back on and recalibrate the threshold on your own traffic. Model routing for agents and measuring agent cost cover the method.

FAQ

Is Caveman an alternative to RouteLLM? Caveman Router is, for Claude Code subagents, but it's hosted and not released yet. For a self-hosted open-source router today, RouteLLM is the option.

Does compression change RouteLLM's routing decision? It can, if the router scores a message the middleware shortened. Route on the original task text and compress only the model call to keep them independent.

Does RouteLLM work with more than two models? Its routers choose between one strong and one weak model. For a larger pool you'd chain decisions or use another router.

Is RouteLLM still maintained? The last commit on its main branch is from August 2024. It is still installable from PyPI (version 0.2.0), but plan to maintain your own fork if you depend on it.

Sources

• RouteLLM on GitHub
• RouteLLM controller source
• RouteLLM launch post, LMSYS
• Caveman Router client README
• Caveman Middleware docs