The paper "JEV-as-a-Judge: Accept When Confident, Escalate When Unsure" introduces a decision-only evaluation framework utilizing "JEV," a proprietary model that provides a verdict and label probabilities without generating a rationale. The research investigates the feasibility of using this model as an economical "first-pass" judge, only escalating to more expensive, reasoning-capable LLMs when JEV exhibits low confidence.

Core Methodology and Technical Framework

The methodology rests on a decision-only interface (TypeSafe JEV) that exposes a structured probability distribution $p_k$ over a set of target labels $k$. The judge provides an uncertainty signal, primarily derived from the maximum label probability $q = \max_k p_k$. The authors evaluate this framework across three main task types:

1. Pairwise Ranking: Assessing preference (RewardBench).
2. Evidence-Grounded Factuality: Fact-checking against provided text (HaluEval).
3. Correctness Adjudication: Complex reasoning, coding, and mathematical verification (JudgeBench).

The study employs a "frozen cascade" architecture where the decision to escalate is governed by a threshold mechanism. For a given input, if the judge's confidence $q$ falls below a learned threshold, the task is routed to a stronger, higher-cost LLM (e.g., GPT-6 Astra). The authors validate the calibration of $q$ using multiclass Brier scores, ten-bin Expected Calibration Error (ECE), and error-detection AUROC, where errors are treated as the positive class with $1-q$ as the score.

Performance and Findings

The authors find that JEV is highly competitive for ordinary preferences and evidence-grounded tasks, performing within 3% of the state-of-the-art GPT-6 judge while operating at approximately 0.36% of the cost. Specifically:

• Efficiency: JEV achieved a median latency of 0.152 seconds and a cost of roughly $0.044 per 1,000 judgments.
• Accuracy Limits: A significant performance gap emerges in "difficult correctness" domains (reasoning and coding), where JEV trails the strongest comparator by 9–20 percentage points. The gap is exacerbated when judges must resist "elaborately written wrong answers" or perform multi-step derivations.
• Calibration and Deferral: While JEV provides a reliable signal for routine decisions, the authors observe that temperature scaling, used for calibration (Guo et al., 2017), does not transfer universally across different workloads, necessitating task-specific validation.

Conclusions on Operational Strategy

The paper concludes that evaluation should not rely on a monolithic judge. Instead, an "operating envelope" approach is optimal:

• Acceptance: Use JEV for high-confidence, reference-grounded, or preference-based judgments.
• Escalation: Use stronger models when the evaluation requires reasoning, resistance to misleading styles, or when the decision is in a low-confidence regime ($q < \text{threshold}$).
• Limitations: The study demonstrates that all tested judges struggle with reference-free prose evaluation, suggesting that JEV's failures in this area represent a broader boundary for current LLM-as-a-judge systems rather than a flaw specific to decision-only interfaces. The final analysis confirms that a selective cascade retains 99% of the stronger model's accuracy at a significantly reduced computational budget.