Did the change make it better?

The average says yes. Eleven cases that used to pass now fail — those are the reason to hold, and an average score cannot show them to you.

A case that passes 10 out of 10 and a case that passes 9 out of 10 are different products. Most tools show the last run and render them identically — which is how intermittent failures reach customers.

Where did it go wrong?

Every step here looks correct on its own. The agent retrieved the wrong customer's record at step 4, then reasoned flawlessly about the wrong person for the next 58 steps.

Where the graders don't agree

A human and a model reviewed the same answer and reached opposite conclusions. That means one of three things — the model judge is miscalibrated, the human is wrong, or the rule itself is ambiguous. All three are worth fixing, so Verdict never averages them into one score.

Eval sets

A set everything passes is not a good set — it is a set that stopped telling you anything. Verdict flags saturation instead of celebrating it.

smoke-12 has passed above 95% for six consecutive weeks. It no longer separates a good agent from a bad one — it only confirms the service is up. Verdict suggests retiring it or adding harder cases rather than letting it inflate the aggregate.

What "correct" currently means

Changing a rule changes what every score means. Verdict keeps old scores readable but refuses to compare across versions — a comparison between two definitions of correct is not a comparison.

What grading costs

Unpredictable spend is why teams stop running evals, and a tool nobody runs produces no quality signal. Every action shows its estimated cost before you click it, not on a bill at the end of the month.