How to evaluate an AI agent optimization before rollout

Write the checks that a cheaper wrong answer fails, and write them before you run the candidate.

An optimization rarely breaks an agent loudly. It drops the one clause that mattered: the caveat, the account ID, the second error line. The JSON still parses, the tests still pass, the summary still reads well, and the average score does not move. That is the whole difficulty. A suite a subtly wrong answer can pass is not an acceptance test, it is a smoke test.

Write the rejection list before you write the score

Start from how a cheaper answer would cheat, then write the check that catches it.

Every check is a hard pass or fail on one named requirement. A single blended score is how a real regression gets averaged into noise. Use a program wherever a program can decide. Use a judge or a human only for the parts that need judgment, with the rubric written down and versioned.

Include the cases your specific change can break

Include ordinary successful work as well. A suite built entirely from last week's incident will happily approve a fix that damages everything else.

Keep the cases you tuned on out of the acceptance set

Use one set to design the prompt, the routing policy or the compression settings. Keep another set sealed for the decision. Tune on the same cases you grade with and you will learn the suite instead of the task.

Version the graders and the expected outputs with the cases. Change a rubric and you rerun the baseline under the new rubric before you compare anything. Scores from two different judges do not belong on one line.

Keep past failures in as named cases. They exist to stop a specific thing from happening twice, not to nudge an average.

Freeze the starting state so the arms differ by one thing

Record the input, the repository or dataset revision, the model, the reasoning setting, the agent version, the tool configuration. Give baseline and candidate separate copies of the same starting workspace.

A resumed session needs its real history or checkpoint. A fresh session cannot stand in for a warm one when cache reuse or memory is part of the claim. For tools with side effects, use staging and disposable fixtures, and reset only the state you own. Do not replay production writes to make the traces match.

Alternate the order of the arms. Repeat enough tasks to see the variation your workload has, and record the sample count. If the candidate also changes the model, that is a second experiment and it cannot isolate the first.

Count everything that happens after the first call

Recovery reads, validation retries, repair turns, escalation to a stronger model, abandoned branches: all of it is the price of the answer. A router that picks a cheap model and then escalates on half the tasks has bought two attempts.

cost per accepted completion = spend on all attempts / accepted completions

When nothing passes, the denominator is zero and the number is undefined. Report the spend and the failures. Use provider charges where you have them, otherwise keep usage counts with a dated price source and label the result an estimate. Local Caveman reports stay inferred. Measure agent cost has the cache normalization and the subscription rule.

Read the losing cases before you read the average

Open the artifacts from both arms side by side on the tasks the candidate lost. Look for the three things an aggregate hides: work the baseline did that the candidate skipped, a required caveat that vanished, a different tool path that happened to pass this time.

A candidate that helps long logs and hurts short requests is a candidate for a narrower policy with an explicit eligibility check. It is not a reason to drop the short requests from the report.

For a generated fix, check its scope and rerun the cases it never saw while it was being written. A proposed patch stays a proposal until the sealed set passes.

Write the rollout rule before you see the result

Decide the quality floor, the latency ceiling, the cost threshold and the rollback trigger while you still have no idea which way the numbers go. Some workflows need every hard check green. Others can carry a measured error rate behind human review. Write down which one this is.

Then roll out to the task family that passed, keep the original path callable, and re-evaluate when the model, framework, agent version or tool output changes. Caveman's local tools can be graded with the tests and observability you already run. A trial of Caveman Cloud, which is open for sign-up with a Free plan, should show the same chain: evidence, the change, the checked result.

Keep enough to let someone else rerun it: baseline configuration, the candidate change, the cases, grader versions, raw outcomes, the command. Say what evidence is missing and what the conclusion does not cover. For a worked example, read CaveBench Wrap, including its losing cases. Its numbers are evidence about that benchmark and not a promise about your agents.