JEV-as-a-Judge — Breakdown

The whole paper in 20 seconds

TL;DR

The problem: Reasoning judges like GPT-6 cost about $12 per 1,000 judgments, a bill that recurs with every re-evaluation.
The idea: Use a decision-only judge that returns label probabilities, accept its verdict when confident, and escalate when unsure.
The catch: It keeps pace where a verdict can be read off the text, and falls behind where it must be derived: math, code, logic.
The payoff: Escalating 31% of pairs beats GPT-6 alone by 0.9 points at 41% of its fee.

See it move

Six steps, from the cost of a reasoning judge to a cascade that beats it. Pause, step through, or open the paper behind each step.

STEP 1 OF 6

Reasoning judges are accurate, and expensive

GPT-6 costs about $12 per 1,000 judgments and 1.89 s each. JEV costs $0.044 and answers in 0.15 s.

Prefer to watch?

The whole breakdown as a short video: the bill, the gate, and when to trust a cheap judge.

SIFTY walks through the paper in 3:29. Every number in the video is sourced to the paper below.

Straight from the paper

Every number in the motion and the video, with the passage it comes from. Click any passage to open the PDF at that line.

Five things to remember

1
A judge 277× cheaper exists.
JEV returns only label probabilities: $0.044 per 1,000 judgments and 0.15 s median, ranking eighth of fifteen LLM judges.
2
Reading is cheap; deriving is not.
Within about two points of GPT-6 where the verdict can be read off the text; 7–28 points behind on knowledge, code, math and logic.
3
Confidence tells you when to trust it.
At confidence ≥ 0.9 the two judges are nearly tied (94.7% vs 94.4%), so one threshold can route the rest.
4
The cascade beats GPT-6 for less.
Escalating 31% of pairs beats GPT-6 alone by 0.9 points at 41% of its fee, and matched it exactly on two unseen workloads.
5
Validate the threshold locally.
Set the gate by a lower confidence bound on ~100 local labels; watch style-adversarial answers and open prose, where confidence misleads.

The judge and the gate

A decision-only judgeWe study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge.
Reasoning judges cost $12 per 1,000On our timing panel, GPT-6 at low reasoning effort costs about $12 per 1,000 judgments, or $12,000 per million, a bill that recurs with every new checkpoint, response set, and training corpus.
Accept when confident, escalate when unsureFigure 2: Accept when confident, escalate when unsure. JEV returns a decision and label probabilities; we take the largest probability as its confidence q. Confident decisions are accepted, and uncertain ones are escalated to a stronger LLM judge.

Cost and accuracy

Eighth of fifteen, 277× cheaperPooled over 4,850 public judgments, JEV ranks eighth of fifteen LLM judges, 1.7 points behind GPT-6, yet it is 277 times cheaper ($0.044 per 1,000 judgments) and answers in 0.15 seconds at the median.
$0.044 vs $12.182 per 1,000JEV's median latency is 0.15 seconds and its fee $0.044 per 1,000 judgments, against 0.55 seconds and $0.390 for GPT-4.1 mini and 1.89 seconds and $12.182 for GPT-6: about 9 and 277 times cheaper, and about 13 times faster than GPT-6.
Tied on RewardBenchOn the 1,500 RewardBench pairs JEV and GPT-6 both score 92.5% (paired difference 0.0 points).
Reading: on parJEV is on par with GPT-6, within about two points either way, wherever the verdict can be read off the text: chat quality (86.1% versus 83.9% on supplied labels), refusals of harmful requests (96.5% versus 94.5%), and evidence-grounded factuality (87.3% versus 88.4%).
Deriving: falls behindIt falls behind wherever the judge must derive or check a result: expert knowledge (−7.0 points), code (−12.9), math (−14.3), and above all logic puzzles (−27.6; 68.4% versus 95.9%).
78.6% vs 93.1% on JudgeBenchJudgeBench separates the judges. JEV scores 78.6% against 93.1% for both GPT-6 and GPT-5.6, a paired difference of −14.6 points.
Humans side with GPT-6, 57 to 1It is not label noise: the adjudication sides with GPT-6 on 57 of the 69 disputed items and with JEV on one.

The cascade

Confident items: nearly tiedWhere JEV's confidence (averaged over both orders for pairs) is at least 0.9, the two judges are nearly tied (94.7% versus 94.4% on 3,744 items).
Cascade: +0.9 points at 41% of the feeA two-order cascade with a threshold frozen before evaluation escalates 31% of 1,610 held-out pairs and beats GPT-6 alone by 0.9 points at 41% of its fee.
JudgeBench: 65% escalated, 94% of gap closedOn RewardBench it escalates 25% of pairs at 27% of GPT-6's fee; on JudgeBench, where JEV alone trails by 12 points, it escalates 65% and closes 94% of the gap.
0.27 s vs 2.10 s liveRun live on 510 of these pairs, it answers in 0.27 seconds at the median, against 2.10 for GPT-6 alone.
Pick the threshold by a lower boundChoosing the threshold by a lower confidence bound rather than a point estimate cuts the risk of losing more than two points from about 45% to about 5% with about 100 local labels, and to under 1% when each workload gets its own threshold.
New workloads: matches GPT-6 exactlyIn a pre-specified live test on 570 held-out pairs from two new correctness workloads, where JEV trails GPT-6 by 14 points, the cascade matched GPT-6's accuracy exactly while escalating 74% of pairs and saving a quarter of its fee.

Limits

Where confidence misleadsThe signal weakens where JEV is confidently misled: on style-adversarial pairs, where the AUROC of its confidence for detecting its errors is 0.76, against 0.84 on style-matched pairs, and on reference-free prose, where every LLM judge tested is near chance yet confident.

Paper: Li et al., arXiv:2609.26550 (2026), licensed CC BY 4.0. The PDF shown here is unmodified.