AI gateways
Caveman vs Helicone: LLM logs plus token compression
Helicone logs, caches and routes LLM calls through its AI gateway. Caveman Middleware shrinks tool output before the call leaves your app. How the two fit.
- 01Keep the Helicone base URL
- 02Wrap the client with Caveman
- 03Compare two Helicone sessions
Keep Helicone. Caveman works with it. Helicone is where you find out which agent step cost the money, and Caveman Middleware is a small wrapper that makes the expensive part of that step smaller before the request reaches Helicone's gateway. They sit at different points of the same call, so there is very little overlap.
What is Helicone?
Helicone is an open source LLM observability platform with an AI gateway in front of it. You point an OpenAI SDK client at https://ai-gateway.helicone.ai, use a Helicone key, and every request is logged with its tokens, cost and latency. The gateway reaches 100+ models across OpenAI, Anthropic, Google and others, falls back automatically when a provider is down, and adds 0% markup to provider prices.
The part people stay for is the dashboard. Sessions group a whole agent run with three headers (Helicone-Session-Id, Helicone-Session-Path, Helicone-Session-Name), so a fourteen-call run shows up as one tree instead of fourteen loose rows. Caching is header driven: Helicone-Cache-Enabled: true turns it on, responses live in Cloudflare Workers KV, and the default TTL is seven days. There are also prompts, datasets, a playground, alerts and HQL for querying your own logs.
It is Apache 2.0 on GitHub (about 6.2k stars when we checked) and self hostable with Docker Compose. Hosted pricing starts with a free Hobby tier of 10,000 requests and seven days of retention, then Pro at $79 a month and Team at $799 a month.
One 2026 change is worth knowing. On March 3, 2026 Helicone announced it was joining Mintlify, and said the service stays live in maintenance mode: security updates, new models, and bug and performance fixes keep shipping. If you already run Helicone, nothing breaks. If you are picking a gateway for the next three years, factor that in.
Does Caveman work with Helicone?
Yes. Helicone has no Caveman adapter of its own, and it doesn't need one. Caveman Middleware wraps the provider client you already use, and the client's base URL stays pointed at Helicone. For an app that runs its own tool loop, that is withCavemanOpenAITools from @caveman-ai/middleware/openai in TypeScript, or with_caveman_openai_tools from caveman_middleware.openai in Python. Those variants register the recovery tool, and Caveman only shortens a tool result when the model can read the original back.
Before each call, the wrapper finds eligible tool results in the outbound request (file reads, API responses, command output, extracted documents) and swaps them for a shorter copy produced by a local Caveman runtime. Your application keeps its original message history. The model gets a caveman_retrieve tool so it can pull the full original back when the short copy is not enough. User messages, system prompts, reasoning, errors and image or PDF bytes are left alone.
The request then goes to Helicone as before. Same key, same headers, same model string. The adapter rewrites the JSON body and leaves your session and property headers as they were, so Helicone still groups the run correctly.
Helicone vs Caveman: feature comparison
| What you need | Helicone | Caveman Middleware |
|---|---|---|
| Log every request with cost and latency | Yes, core feature | No, reports its own decision per call |
| Group an agent run into one trace | Sessions via headers | No |
| One key for 100+ models | Yes, 0% markup | No, keeps your client and gateway |
| Provider fallbacks | Automatic | No |
| Cache identical responses | Header driven, Workers KV | No response cache |
| Shrink a 20k-token tool result | No | Yes, original recoverable |
| Keep original chat history in your app | Not its concern | Yes, only the outbound copy changes |
| Self host | Docker Compose, Apache 2.0 | Runtime runs next to your app |
| Status | Live, maintenance mode since March 2026 | Alpha |
| License | Apache 2.0 | MIT client and adapters, BSL runtime |
What does Helicone log when Caveman is in the path?
Helicone logs the compressed copy. The middleware runs inside your process before the HTTP request leaves, so by the time Helicone sees the body, the long tool result has already been replaced with the shorter version and its recovery handle. Your app's own history still has the original.
That is useful, and a little surprising at first. Helicone's token and cost columns now show the smaller request, which is the number you care about. But the first time someone opens a log and sees [caveman: shortened; exact original via caveman_retrieve handle=...] where they expected a full file, they will wonder what happened. Tell your team before they go looking.
A caveman_retrieve call shows up in the session tree as an ordinary tool round trip. Count those. A recovery call carries the original back and costs tokens, so a fair comparison adds it to the total.
How much does Caveman save on a Helicone session?
It depends on how much of your traffic is tool output, and nobody can give you your number in advance. What Caveman publishes is narrow. In the middleware docs, a 400-line worker log sent in compress mode went from 8,399 tokens to between 177 and 187 across sixteen runs, counted with o200k_base and labelled inferred. The middleware product page reports 13 synthetic samples through the real runtime, also inferred. Neither is a bill.
Helicone is the right tool for getting your real number. Run the same task twice, once with the runtime in record mode and once in compress, and compare the two sessions.
When should you use Helicone alone?
Use Helicone alone when your requests are short. A chat app sending 1,500 tokens a turn with no tool calls has nothing for Caveman to compress, and Helicone's logs, cache and fallbacks are the whole value.
Use it alone when your problem is visibility. If you can't yet say which agent or which customer is expensive, logging comes first. Compression without a baseline is a guess.
Use it alone when you replay the same prompt over and over, such as tuning a template in development. An exact cache hit costs nothing, which beats any compression ratio.
When does adding Caveman make sense?
Add Caveman when the Helicone session view shows the same big payload riding along on call after call. A coding agent that reads a 20,000-token file on step two and carries it through step fourteen is the textbook case. So is a support agent that pulls a large JSON order history from an API and keeps it in context.
For coding agents in a terminal, such as Claude Code or Codex, the separate Caveman Proxy is the simpler route. The middleware is for applications you build.
How to add Caveman to Helicone
Install the runtime, start it in compress mode, wrap your OpenAI client, and keep the base URL on Helicone. The full walkthrough, with code, a rollback and a way to check it worked, is in add Caveman to Helicone. The adapter list and version ranges are in the middleware frameworks docs.
Caveman also has a hosted gateway, Caveman Cloud, open for sign-up with a Free plan.
FAQ
Is Caveman a Helicone alternative? For most teams, no. Helicone logs and routes requests, and Caveman makes the request body smaller. Teams that want both keep Helicone as the gateway and add the middleware in the app.
Does Caveman break Helicone sessions?
It shouldn't. The adapter rewrites the request body and leaves headers alone, so Helicone-Session-Id and the rest arrive as sent. Check one run in the dashboard before rolling out.
Will Helicone's cache still hit? Within one conversation, Caveman reuses its earlier choice, so repeated turns stay byte stable. A new conversation gets a fresh recovery handle, so its body differs from an older one and an exact-match cache will miss.
Is Helicone shutting down? No. Its March 2026 post says the service stays live in maintenance mode, with security updates, new models and fixes still shipping.
Sources
Try it on work you know
Keep your stack. Add the adapter.
Wrap the client or framework you already call, run one agent task you can check, and compare the tokens sent.
Open the middleware docs Explore comparisons ↗