NeoSigma Verifier Router: Find 4x as many agent failures at 93% lower cost

Key takeaways

• NeoSigma Verifier Router is NeoSigma's hybrid verification engine. It uses a decision model's confidence score to escalate low-confidence verdicts to a stronger model, 32% of verdicts in this study.

• On a customer's production sessions, NeoSigma Verifier Router finds 3.8x as many failures as Claude Sonnet 5 at a 30% sample rate, at 93% lower cost.

Every production session records what your agent did for a user, and verifiers turn that record into a verdict on whether it went right. A frontier model gives good verdicts, but at frontier prices most teams set a sample rate and check only part of their traffic. At a 30% sample rate, 7 of every 10 sessions are never checked, so a failure that shows up in only one session usually goes unseen.

NeoSigma Verifier Router verifies every session. A decision model, a model that picks a verdict from a fixed set and returns its confidence, gives the first judgment for every verifier. When that confidence falls below the cutoff set for that verifier, NeoSigma Verifier Router escalates the judgment to a stronger model. Most judgments stay with the decision model, so the budget goes to the judgments where stronger review helps.

In NeoSigma, you can trace each failure NeoSigma Verifier Router finds to its root cause and turn it into an eval case. Then you test the next change to your agent's harness against that growing set.

The research behind this, including Jev against LLM judges, calibration and whole-session routing, is in Calibrated Confidence, Wider Coverage.

Here is one of the customer's held-out production sessions, checked by 11 verifiers. Jev 1.14, a decision model from TypeSafe, disagreed with the reference labels on 3 of its 11 first verdicts. Reference labels are the majority verdict of a panel of three frontier models. The 5 verdicts that fell below their cutoffs went to GPT-6 Luna, which fixed all 3 mistakes.

NeoSigma Verifier Router caught 262 of 303 failures for $1.30 per 900 sessions. Claude Sonnet 5 at a 30% sample rate catches about 68 for $18.35.

We scored NeoSigma Verifier Router on 100 held-out production sessions with the customer's 11 pass-or-fail LLM judge verifiers, three runs each. Reference labels came from a majority vote of three frontier models (GPT-6 Astra, Muse Spark 1.3 and Claude Opus 5.5), and a person settled the cases where they split. We count failures caught, the reference fail verdicts a setup also marks fail, and report Failure F1, F1 computed on fail verdicts, which balances failures caught against false alarms.

Measured on every session. NeoSigma Verifier Router caught 86% of failures across three runs, against 75% for Claude Sonnet 5 checking every session and 96% for GPT-6 Sol. Its Failure F1 was 0.768, against 0.799 for Sonnet 5 and 0.746 for Sol. Its verdicts were also more consistent. Across three runs of the same session, NeoSigma Verifier Router's verdict flipped between pass and fail on 2.0% of session and verifier pairs, against 4.4% for Sonnet 5. Jev alone caught 40% of failures.

Projected at a 30% sample rate. A random sample finds, on average, the same share of failures as the share of sessions it checks. Against Sonnet 5 at that rate, NeoSigma Verifier Router finds 3.8x as many failures at 93% lower cost. GPT-6 Sol catches about 87 for $8.31, so NeoSigma Verifier Router finds 3.0x as many at 84% lower cost. These are expected values from the measured runs, not a sampling deployment.

The trade-off is false alarms. NeoSigma Verifier Router raised 117, against about 12 for Sonnet 5 on a 30% sample, so about 3 in 10 of its fail verdicts were false alarms.

On the four rarest verifiers, with failures in only 9 of the 100 sessions between them, NeoSigma Verifier Router caught 20 of 27 failures across three runs. Sonnet 5 on a 30% sample catches about 6.

These results come from one customer's workload, and the reference labels are model consensus, not independent human ground truth. We did not measure how many of the extra failures teams go on to fix.

A decision model's confidence means different things on different verifiers. On Verifier A, Jev was wrong 63% of the time, and its average confidence was almost the same when it was right (0.50) and when it was wrong (0.45). On Verifier C, Jev was right 87% of the time, and sending its low-confidence answers to Luna fixed no more errors than it introduced.

One shared cutoff handles both verifiers badly. On a validation set, a cutoff of 0.70 sent about 80% of each verifier to Luna. With a cutoff per verifier, NeoSigma Verifier Router escalated 95% of Verifier A and none of Verifier C.

NeoSigma Verifier Router fits each cutoff on a labeled calibration set that is separate from the sessions it is scored on, and escalated 32% of judgments across the held-out sessions.

The size of this gain is uncertain. Per-verifier cutoffs raised Failure F1 from 0.667 to 0.768 over a shared 0.70 cutoff on the held-out sessions, but only from 0.674 to 0.676 on a smaller validation set.

The full method, with per-verifier confidence analysis, is in Calibrated Confidence, Wider Coverage.

Use it on the NeoSigma platform, where your sessions and verifiers already live. NeoSigma Verifier Router fits a cutoff for each verifier.

1. 1.

   Run it on your sessions and define your verifiers:

   Trace your agent with the NeoSigma SDK to bring its production sessions into NeoSigma. Then define a verifier for each thing you want to check: an LLM judge, a code verifier for exact business rules, or an agent verifier that uses tools to follow a session before it decides. NeoSigma Verifier Router runs your LLM judge verifiers.

2. 2.

   Label a sample:

   NeoSigma labels a sample of your sessions with a panel of three frontier models. You can also annotate sessions in NeoSigma or use labels you already have. Annotations, a score and an optional comment on any session or turn, sit next to the verifier verdicts, so your team also sees where a verifier disagrees with them.

3. 3.

   NeoSigma Verifier Router fits a cutoff for each verifier:

   It tests each cutoff on a separate test set, because the same confidence can mean different things on different verifiers.

4. 4.

   Verify every session:

   Run NeoSigma Verifier Router on all of your sessions with those cutoffs. Each verdict comes with its confidence score.

5. 5.

   Refit when things change:

   When you add a verifier or your traffic shifts, add a fresh labeled sample and refit the cutoffs.

You define the verifiers, so you decide what a good session is.

Agent failures are often sparse: a partial completion, a quiet policy violation, an answer that looks right and is not. Because NeoSigma Verifier Router verifies every session, a failure that falls outside a sample can still be found. With NeoSigma, your team can then act on it:

• Find failures that raised no error. Filter sessions by verifier result, or search them in plain language, such as "sessions where the refund tool timed out", to surface failures that never showed up as a system error.

• Group repeated failures. NeoSigma groups failing sessions into deduplicated issues, and "Find similar sessions" pulls up other sessions with the same failure.

• Turn failures into evals. Add failing sessions to a dataset as regression evals, each linked back to its source session. Run the next version of your agent against them.

• Draft a fix. Triage sends an agent to investigate an issue and open a draft pull request. It never merges on its own, so your team reviews the fix and decides what ships.

• Check each verifier against your annotations. Filter sessions by annotation and by verifier result to see where your team and a verifier disagree. Each disagreement points to a verifier to revisit and gives you a label for the next refit.

Use NeoSigma Verifier Router on your sessions in NeoSigma. Request a demo to get started.