Cheaper Judges, Denser Feedback: Verifying Agent Traces at Scale with Jev

Every production interaction creates a trace of what an agent saw, which tools it called, how the environment responded, and what it told the user. At sufficient volume, those traces contain the clearest record of where an agent is reliable and where it fails.

The hard part is turning that record into reliable judgments. We run verifiers to convert raw traces into actionable behavioral signals: identifying failures, surfacing regressions, and detecting the patterns that matter across real user workflows. At scale, they reveal recurring failure clusters, help turn representative examples into evaluation tasks, and provide a tighter feedback loop for optimizing the agent on these failures.

Many of the questions that matter most are semantic rather than deterministic: Did the agent understand the user's intent? Did it complete the workflow correctly? Did it recover appropriately from an error? LLM judges can answer these questions, but their cost and latency often limit teams to evaluating only a small sample of production behavior.

That leaves the long tail largely unexamined: partial completions, subtle policy violations, and plausible-looking responses that still fail the user. These are often the failures that matter most, and the ones sparse evaluation is most likely to miss.

Jev is TypeSafe's first "System One" model. Jev uses a different objective, Reinforcement Learning for Calibrated Decisions (RLCD), which optimizes probabilities against outcomes over a predefined answer space. For a pass-or-fail verifier, that makes the task a direct decision rather than text generation constrained into a verdict unlike in LLMs.

Jev takes an unstructured state and a typed question, then returns a structured decision and probabilities rather than generating an explanation. Its three question types cover choices, ordered scores, and yes-or-no probabilities.

For a verifier, the interface is direct: provide the agent trace, ask whether the task was completed, and constrain the answer to pass or fail. Jev returns one of those declared choices with a probability distribution. There is no free-form rationale to parse back into a verdict.

That distinction removes the free-form parsing step and constrains the output shape. It does not establish correctness. A model can always choose the wrong valid label, and a consistently wrong judge only produces bad feedback more efficiently.

We examine whether Jev, a specialized decision model, can make trace-level verification cheap and consistent enough to run broadly over large volumes of requests, while reserving larger judges and human review for the cases that need them.

Across 10,000 verifier calls on 834 frozen Tau2 traces, Jev used about one-sixth of GPT-5 Mini's spend. It was also roughly 34x faster at p50 latency and more repeatable on identical inputs. In a blinded 100-trace audit, Jev matched the human label on 88 traces, compared with 85 for GPT-5 Mini.

Our GPT-5 Mini baseline uses a strict schema that returns a verdict, self-reported confidence, and a concise rationale. Jev returns the structured verdict and pass/fail probabilities without a generated rationale. So this is a comparison of the verifier configurations we actually ran, including GPT-5 Mini's rationale output tokens, rather than a comparison of model input prices alone.

TypeSafe currently lists Jev at $0.042 per million input tokens with unmetered output and reports 70–500 ms end-to-end latency. They make Jev interesting for high-volume evaluation, but the relevant numbers are the ones measured on the traces and rubric a team actually uses. Jev can also amortize a shared state across several typed questions, so additional questions may have low marginal cost. They are not free, and this experiment measures one question per call.

We test the same proposition across a broader agent-behavior corpus while keeping correctness, consistency, cost, latency, and failures separate.

We designed the experiment around one control: Jev and GPT-5 Mini judge the same frozen trace against the same binary rubric.

Did the agent successfully complete the user's task?

The evaluation state contains the agent's frozen domain policy, conversation, tool calls, tool results, and final response. Both judges receive the same state and rubric. Neither receives the Tau2 reward, termination metadata, or human label. A digest identifies the exact state and rubric used for every judgment.

The distinction between traces and decisions matters. Each judge completes a deterministic 10,000-valid-decision cohort over the same 834 distinct agent runs; failed attempts, retries, and any later valid decisions produced during recovery are recorded in addition and remain in billed spend. Unique traces measure coverage and correctness. Repeated decisions measure verdict stability, latency, operational failures, and cost at volume.

For the human reference, one reviewer received the same frozen policy, rubric, and trace while judge outputs and Tau2 rewards remained hidden. The reviewer labelled all 100 traces. The analysis uses those first-reviewer labels, including the 11 rows that later proved to disagree with Tau2.

Before running the comparison, we defined "good enough" for a high-volume first-pass verifier. On the same human-labelled audit rows, Jev must stay within five percentage points of GPT-5 Mini on human agreement and within ten percentage points on failure recall. It must return valid results at least 99.5% of the time and have lower actual billed cost per completed valid decision and lower first-seen cost.

We also set a descriptive repeatability target: at least 99% observed mean within-trace pairwise agreement on the fixed 100 x 20 slice.

Jev was cheaper, faster, and more repeatable. In the blinded human review, its overall agreement was similar to GPT-5 Mini, while it caught substantially more of the traces the reviewer marked as failures. The audit is modest, so we treat the agreement result as promising, not conclusive.

Lower cost at the scale we ran

For 10,000 valid decisions, Jev cost $3.23 and GPT-5 Mini cost $19.98. That made Jev 6.2x cheaper on the workload we actually ran. On 10,000 first-seen traces, where replay caching has less influence, the projected gap widened to 9.6x.

Faster decisions, with fewer changes of mind

The median Jev decision took 231 milliseconds, compared with 7.8 seconds for GPT-5 Mini. Both judges completed every planned call. When we judged the same 100 traces 20 times, Jev changed its verdict on 3 traces; GPT-5 Mini changed its verdict on 19.

Similar overall human agreement, stronger failure recall

In the blinded 100-trace review, Jev matched the human label 88 times and GPT-5 Mini matched it 85 times. That three-point lead is encouraging, but the audit was not large enough to make the preregistered agreement test conclusive.

The clearer difference was on failures. Of the 29 traces the reviewer marked as failed, Jev found 25 and GPT-5 Mini found 17. This preregistered test passed.

A secondary check against Tau2

Across all 834 traces, the judges were nearly tied against the Tau2 reward: Jev matched 696 traces (83.5%), while GPT-5 Mini matched 704 (84.4%). On the 240 Tau2 failures, Jev caught 185 and GPT-5 Mini caught 151. We treat this as supporting evidence rather than ground truth because a benchmark reward can miss legitimate changes in user intent.

Four preregistered gates passed: measured cost, first-seen cost, valid completion, and human failure recall. Jev also landed just below the descriptive repeatability target at 98.99% against a 99% bar.

Detailed statistical view

The summary visuals above are the reader-first view. This chart keeps the full comparison, including uncertainty and the paired human-audit tests.

An airline customer asked to move a flight. The agent found a valid nonstop option, but it required a $396 upgrade. The customer's budget was $100, so they declined and explicitly asked to keep the existing reservation. The agent confirmed that no change had been made.

The human reviewer and Jev called this trace a pass. GPT-5 Mini and the Tau2 reward called it a failure.

The split maps to two valid-looking ways of reading the trace: judge it against the opening request, or judge it against the user's final instruction after seeing the price. For production evaluation, that change of intent matters. A useful verifier has to follow the full trajectory, not only compare the ending with the task as it began.

The value of a cheap decision model is not that it replaces every LLM judge. It changes how evaluation capacity can be allocated.

A practical evaluation stack has three layers:

1. 1.

   Keep exact checks in code.

2. 2.

   Run bounded semantic questions through a low-cost decision model across a much larger share of traces.

3. 3.

   Escalate uncertain, novel, or high-impact cases to a stronger LLM or a human reviewer.

On this workload, Jev produced a similar human-agreement point estimate at much lower cost and latency, with stronger failure recall and fewer verdict flips. It supports a narrower conclusion: Jev is a strong candidate for a high-volume first pass when the question is bounded and an escalation path protects the cases where mistakes matter most.

The compounding impact is the real unlock. When verification is cheap, fast, and reliable enough to run across far more production behavior, it creates the dense feedback signal an agent system needs to learn and improve. More evaluated traces build a richer record of how the agent behaves in the real world: regressions surface sooner, recurring failures become reusable capability and regression evaluations, and every change to the model, prompt, tools, or harness can be tested against a more representative view of production.

The best verifier is not simply the most capable model per call. It is the evaluation system that generates enough reliable judgment to understand, measure, and continuously improve agent behavior at scale.