Caveman vs Portkey: AI gateway plus tool-token compression

Portkey and Caveman do different jobs, so most teams asking this should keep Portkey. Portkey is the gateway: it routes, retries, caches, enforces guardrails and budgets, and logs every call. Caveman Middleware sits one step earlier, inside your app, and shortens large tool results before the request leaves. Portkey then gets a smaller request and does everything it did before.

What is Portkey?

Portkey is an AI gateway and control plane for teams running LLM apps in production. Its open-source gateway routes to 1,600+ models through one API and ships more than 50 guardrails. The hosted platform adds observability, prompt management, an MCP gateway and governance: RBAC, budgets, audit logs. Portkey says more than 3,000 GenAI teams use it.

The open-source gateway is MIT licensed, written in TypeScript, and has about 13,000 GitHub stars. You can run it locally with npx @portkey-ai/gateway. The README says Portkey's enterprise gateway is merging into open source with the 2.0 release, now on a pre-release branch.

Hosted pricing has three tiers. Developer is free with 10k recorded logs a month. Production is $49 a month for 100k logs, with guardrails, caching, fallbacks and RBAC. Enterprise is custom and adds SSO, granular budgets and rate limits, VPC hosting and private cloud deployment.

The big 2026 news: Palo Alto Networks closed its acquisition of Portkey on May 29, and Portkey is now the core AI gateway inside Prisma AIRS. If your agent traffic has to pass a security review, that counts in its favor.

Does Caveman work with Portkey?

Yes. Portkey speaks the OpenAI API, and Caveman wraps the client you already point at it. There is no Portkey-specific adapter and none is needed: wrap the OpenAI SDK, the Anthropic SDK or your agent framework with Caveman Middleware and leave the base URL on Portkey.

client = OpenAI(api_key=..., base_url="https://api.portkey.ai/v1",
                default_headers=createHeaders({"apiKey": ..., "provider": "@openai-prod"}))

loop = with_caveman_openai_tools(client, runtime=runtime, scope=scope,
                                 protocol="openai-chat", tools=tools, functions=functions)

Your Portkey headers, configs and provider slugs pass through untouched. The adapter adds one tool, caveman_retrieve, and gives you its executor in loop.functions, so the model can pull back the full original when the short view is not enough. Your messages list keeps every byte; only the outgoing copy is shorter.

The same works from TypeScript with the Vercel AI SDK, OpenAI, Anthropic, LangChain, Mastra or Strands adapters, and from Python with LangChain, Pydantic AI, CrewAI, Agno and others. Point the framework's model at Portkey as you do today and add the adapter. The Portkey switching guide has a complete loop.

What does Portkey see after Caveman?

Portkey sees the shortened request, because Caveman runs before the request leaves your process. In practice:

• Logs. Portkey's request logs show the short version of each big tool result, starting with a line like [caveman: shortened; exact original via caveman_retrieve handle=cmw_…]. The full original stays in your app and in the local Caveman store.
• Guardrails. Input guardrails check the shortened copy. When the model asks for the original, it comes back as a caveman_retrieve tool result on the next request, and that request passes Portkey's guardrails like any other.
• Token and cost metrics. Portkey records provider usage for what was actually sent. That is the cleanest place to see the difference, because the count comes from your provider, not from Caveman.

How many tokens does Caveman save?

It depends on your tool output. Caveman's published samples ran 13 synthetic tool results through the real runtime and counted tokens with o200k_base. An API response with 160 JSON records went from 7,202 to 403 tokens. A 160-line test run with one failure went from 3,837 to 176. A git diff, rendered HTML and code search results came through unchanged. These are inferred counts from single samples, not a bill. The middleware page has the full set and the script that reproduces it.

Portkey vs Caveman: feature comparison

When should you use Portkey alone?

Use Portkey alone when your requests are small. A chatbot sending a few hundred tokens a turn gives Caveman nothing to shorten.

Use it alone when the problem is governance: who may call which model, under what budget, with which content checks. That is Portkey's core, and Caveman does none of it.

And stay with Portkey alone if everything in your request path has to be generally available today. Caveman Middleware is alpha, and Caveman's hosted gateway, Caveman Platform, is not generally available.

Add Caveman when agents keep resending big tool output: a 20k-token search result that rides along for ten more turns, repo files read once and carried forever, CI logs.

FAQ

Is Caveman a Portkey alternative? Not for gateway work. Caveman does not route, enforce policy or log requests. It sends Portkey smaller requests.

Do I need to change my Portkey config? No. Configs, provider slugs and guardrails stay as they are. The change is in your app code.

Will Portkey's cache still hit? Within one conversation Caveman reuses the same shortened view for the same result, so repeated requests stay stable. Check your hit rate during the test anyway.

Can I run Portkey's open-source gateway and Caveman on one machine? Yes, but both default to port 8787. Start Caveman with caveman start --port 8788 and pass that endpoint to the runtime.

Sources

• Portkey homepage and pricing
• Portkey open-source gateway on GitHub
• Portkey request headers
• Palo Alto Networks completes acquisition of Portkey
• Caveman Middleware for Python and supported frameworks
• Choose an AI gateway