Calibrated Confidence, Wider Coverage: Decision Model routers for Verifying Agent Traces at Scale

Every production interaction creates a trace of what an agent saw, which tools it called, how the environment responded, and what it told the user. At sufficient volume, those traces contain the clearest record of where an agent is reliable and where it fails.

A useful verification system has to get two things right. Judgment quality asks whether its judgments agree with reliable reference judgments: does it catch real failures without flooding the team with false alarms? Coverage asks how much production behavior receives those judgments: how many sessions, and how many relevant checks within each session, can we afford to evaluate?

These requirements matter together. A reliable verifier applied to a small sample can miss sparse failures. A weaker verifier applied everywhere can produce so much noise, or miss so many failures, that its coverage tells us little. We want trustworthy judgments across enough production behavior to understand what users actually experience.

Verification makes that behavior actionable. A trace records the conversation and tool calls; a verifier asks whether the agent understood the request, completed the workflow, recovered from an error, or respected a policy. Those judgments provide the signal for investigating failures, identifying recurring patterns, and turning production examples into representative evals.

The difficult cases often sit in the long tail: partial completions, quiet policy violations, and plausible-looking responses that still fail the user. They can be sparse within otherwise successful traffic. Evaluating more of that traffic gives teams more opportunities to find them, provided the judgments remain reliable.

Strong LLM judges can answer these semantic questions, but their cost can limit how much production behavior a team evaluates. This is the trade-off we investigate: can decision models help us preserve judgment quality while expanding the coverage a verification budget can support?

We explored two decision models, Jev from TypeSafe and OpenAI's Decisions model, as a first layer of verification.

A decision model takes the agent trace and a specific question, such as “Did the agent complete the requested task?”, and chooses from declared answers such as pass and fail. It also returns probabilities for those choices. The output gives us a verdict to use and a confidence signal to examine, without needing to extract a verdict from a free-form explanation.

The verdict and confidence answer different questions. The verdict says what the model thinks happened. Confidence expresses how strongly it favors that answer. A confident verdict can still be wrong.

That distinction is central to our experiment. First, we measure whether the verdicts match reference labels. Then we ask whether confidence helps identify errors. If uncertainty identifies judgments that a stronger model can improve, a decision model could answer broadly while reserving expensive judgment for the cases that need it.

We use Failure F1 to measure failure-detection quality. When examining confidence calibration, we test whether confidence distinguishes correct from incorrect answers well enough to guide escalation. This does not establish that every reported probability is an accurate estimate of correctness.

A router is the policy that decides which judge handles a piece of work. Here, it uses the decision model's confidence to allocate the verification budget.

For example, a policy might keep an answer above a confidence cutoff and ask a stronger judge to replace the answer below it. Raising the cutoff escalates more work and increases cost. Lowering it keeps more first-pass answers, potentially retaining more errors. The right balance depends on how reliable the first model is and whether the fallback actually improves its answers.

We compare two ways to make that decision.

Route one check at a time

The decision model answers each check on a session. A check below its cutoff goes to a stronger judge, whose verdict becomes the final answer for that check. Other checks retain the decision model's verdict.

This allows a session to receive stronger judgment on one difficult question while keeping the decision model's answers on the others. We compare one shared confidence cutoff with cutoffs fitted separately for each check.

Route the whole session

The decision model's confidence across all checks becomes a score for the session. That score selects one stronger model to answer every check in the session.

This tests a different hypothesis: whether uncertainty can identify sessions that need a more capable judge overall. It also commits more budget at once, since one difficult check can send all the session's checks to that model.

Both approaches use confidence to route verification work. We evaluate whether the resulting judgments justify their cost, and therefore how much production coverage each approach could support.

We evaluated 250 real production sessions against the same 11 pass-or-fail checks. Each model received the session's messages, tool calls, attached files, and the questions it needed to answer. No model saw reference labels or another model's answers.

The baselines were individual models grading every check. We compared them with decision models used as a first pass with escalation per check, and with decision models routing whole sessions.

We measure the trade-off in two ways:

• Judgment quality: Failure F1, the F1 score on the fail class, balances catching actual failures against false alarms. We also examine recall and precision to understand what changed.

• Coverage for a fixed budget: cost normalized to 900 sessions lets us compare how much traffic each approach could afford to evaluate. All approaches see the same experimental dataset; expanded coverage is inferred from cost, rather than measured in a larger deployment.

We randomly split sessions into calibration (100), validation (50), and test (100). We fitted cutoffs on calibration, checked choices on validation, and report test results unless stated otherwise. Each request ran three times to measure variance.

Reference labels came from the majority vote of GPT-6 Astra (OpenAI), Muse Spark 1.3 (Meta), and Claude Opus 5.5 (Anthropic), each judging every check three times. A person resolved panel splits. Of 1,080 test check cells, 811 had a unanimous panel, 241 a majority, and 11 a human decision. The remaining 17 were unresolved and excluded. These are reference judgments built primarily from model consensus, rather than independently established human ground truth.

Jev alone cost $0.55 per 900 sessions, but reached only 0.461 Failure F1, with failure recall of 0.396. It caught fewer than four in ten reference failures.

For comparison, Luna grading everything reached 0.695 Failure F1 at $1.05, while Sonnet 5 reached 0.799 at $61.17.

Jev's low cost made broad evaluation affordable, but its standalone verdicts missed too much of the signal we wanted. On this workload, the case for using Jev depended on a system that could retain useful answers and correct unreliable ones.

That brings us to confidence: could it tell the router where stronger judgment would help?

The relationship between confidence and correctness varied across checks. Check A shows the difficulty. On calibration, Jev was wrong 63% of the time, with average confidence of 0.50 when right and 0.45 when wrong. Those averages suggest little separation between correct and incorrect answers.

A routing policy cannot safely assume that confidence has the same meaning on every question.

A shared 0.70 cutoff sent 79% of Check A cases to Luna on validation. Fitting a cutoff for that check increased escalation to 95%.

Check C went the other way. Jev was right 87% of the time on calibration, with average confidence of 0.54 when right and 0.37 when wrong. Escalating to Luna fixed no more failure-detection errors than it introduced on calibration. The fitted policy therefore kept Jev's answers, reducing escalation from 80% under the shared cutoff to zero on validation.

The useful question was whether escalation improved a particular check. That required evidence about the first judge and its fallback, rather than confidence alone.

With a cutoff fitted per check, Jev → Luna reached 0.768 Failure F1 at $1.30 per 900 sessions. It scored 7.3 points above Luna alone for $0.25 more, and 3.1 points below Sonnet 5 at roughly 1/47 of the cost.

At those measured rates, a fixed budget could evaluate about 47 times as many sessions with calibrated Jev → Luna as with Sonnet 5 alone. That wider coverage comes with a modest reduction in measured failure-detection quality. It creates more opportunities to encounter sparse failures; the experiment does not establish how many additional rare failures would be found.

The combination also improved both recall and precision over Luna alone. Across three repeated test runs, it caught 262 of 303 failures, compared with 231 for Luna (recall 0.865 versus 0.762), and raised 117 false alarms, compared with 131 (precision 0.691 versus 0.638). These counts include repeated judgments of the same test sessions.

Most of the gain came from Checks B and C, where calibration retained more Jev answers. On Check C, Jev answered every case and raised 38 fewer false alarms than Luna, while missing three more failures. On Check B, it caught 21 more failures, with 13 more false alarms.

This explains why using a stronger model for every check did not necessarily produce a better overall result. Different judges contributed useful answers on different checks, and calibration helped determine which answers to keep.

Fitting cutoffs per check raised Jev → Luna's test Failure F1 by 10.2 points, with a 95% interval of 5.3 to 15.2, at the same cost. The cascade gained 12.8 points (8.1 to 17.6), while cost rose from $3.28 to $19.07. Jev → Sonnet 5 gained 2.4 points (−2.3 to 7.2), an inconclusive change.

The size of the gain remains uncertain. On the 50 validation sessions, Jev → Luna moved only from 0.674 to 0.676. Calibration also changed little for OpenAI's Decisions model as a first pass: Decisions → Luna moved from 0.710 to 0.713. Calibration did not improve every model or split equally.

Additional spend improved scores, but sharply increased the cost of coverage. The calibrated cascade reached 0.781 at $19.07, and calibrated Jev → Sonnet 5 reached 0.787 at $48.20. Those were gains of 1.3 and 1.9 points over Jev → Luna, at about 15 and 37 times its cost. Sonnet 5 alone remained higher at 0.799, for $61.17.

Those gains may justify the spend where a missed failure is especially costly. For broader monitoring, the first-pass combination offered a more affordable balance on this workload.

The session-router experiment asked whether decision-model uncertainty could choose the right stronger judge for an entire session.

The seven Jev routers scored 0.726 to 0.748 at $23 to $39 per 900 sessions. Decisions routers scored 0.716 to 0.733, each within 2.0 points of its matching Jev version.

Four of the five versions with searched thresholds sent 95 to 99% of sessions to Terra, which scored 0.726 at $32.92 alone. Even with thresholds searched on test itself, an optimistic upper bound, the router matched calibrated Jev → Luna's 0.768 at about $31, compared with $1.30.

For the approaches tested here, selecting a judge for the whole session was a less effective use of the budget. A session can contain one difficult check alongside several straightforward ones. Escalation per check allows the system to buy stronger judgment only where it helps, leaving more budget available to evaluate additional sessions.

For a team running agents, the aim is a reliable picture of production behavior: what succeeds, where workflows break down, and which failures deserve attention. That requires both judgment quality and coverage throughout the verification system.

Our results suggest starting with labeled examples to establish how each verifier performs. Keep exact checks in code. For semantic checks, evaluate whether a decision-model first pass provides useful judgments and whether its confidence helps identify cases a stronger judge can improve. Fit escalation policies per check and revisit them when traffic or the checks change.

On this workload, that approach reached 0.768 Failure F1 at $1.30 per 900 sessions, making much wider coverage affordable than Sonnet 5 alone. Whole-session routing cost 17 to 30 times as much and scored lower.

These results come from a small production dataset, primarily model-generated reference labels, and a calibration gain whose size varied between validation and test. They support a promising trade-off between judgment quality and cost. Expanded discovery of rare failures and downstream agent improvement remain to be measured.

The opportunity is to make verified behavioral signal available across more of production. Broader coverage can surface recurring failures that sampling overlooks, supply representative cases for regression evals, and help teams assess changes to their models, prompts, tools, and harnesses against real user workflows.

A useful verification system earns trust through the quality of its judgments and extends that trust through coverage. Calibrating where stronger judgment is needed can make both possible at production scale.