⚖️ Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

UC Berkeley

* Equal contribution

Abstract

Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.

Methods

Overview of RaRE with two judges: the user prompt, rubric, and a policy rollout are scored by two LLM judges (GPT-4o-mini and DeepSeek-V3.1); each judge's top-5 logprobs give an expected rating and a confidence; the soft-Likert reward weights the expected ratings by confidence, and the drop-on-disagreement mask excludes rollouts where the judges disagree. — Overview of RaRE (Rubrics as Rewards with Ensembles), instantiated with two judges. Each judge scores the rollout against the rubric on a 1–5 Likert scale, and we read its top-5 logprobs to recover an expected rating $E_i$ and a per-judge confidence $C_i$ . **(A)** Our *soft-Likert reward* weights each judge by its inverse-variance confidence. **(B)** Our *drop-on-disagreement mask* excludes high-disagreement rollouts from the GRPO update.

RaRE replaces the single training judge with an ensemble of $K$ judges and targets two distinct sources of uncertainty. The two mechanisms can be used independently or composed.

Soft-Likert reward via judge logprobs

We read the top-5 logprobs at the rating-digit position to recover the full distribution $p_j(k)$ over ratings, and weight each judge’s expected rating $E_j$ by its inverse-variance confidence $c_j$ :

c_j = \frac{1}{1 + \mathrm{Var}_j}, \qquad r_{\mathrm{soft}} = \frac{1}{L} \cdot \frac{\sum_{j=1}^{K} c_j E_j}{\sum_{j=1}^{K} c_j}.

Drop on disagreement

Rollouts where the judges’ normalized rating spread exceeds $\tau$ are removed from the GRPO update by replacing their reward with the unmasked group mean, so that their group-normalized advantage is zero.

d = \frac{\max_{j} \hat{r}_j - \min_{j} \hat{r}_j}{L - 1}

Results

We train Qwen3-4B-Base on RaR-Medicine and RaR-Science with GRPO across five open and closed weight judges, and evaluate on HealthBench and GPQA Diamond.

Single-judge RaR is sensitive to judge

The spread across judges (0.296–0.339 on HealthBench) is comparable to the gap between RL and the untrained base (0.257). On GPQA Diamond, the single-judge spread is 0.345–0.406.

RaRE outperforms single-judge RaR

Our best RaRE configuration outperforms every single-judge RaR variant in both domains (0.364 vs. 0.339 best single judge on HealthBench; 0.412 vs. 0.406 on GPQA Diamond). Naive averaging alone does not outperform single judges; the gains come from filtering or reweighting.

Bar chart of HealthBench scores: baselines 0.257 and 0.308; single judges 0.296 to 0.339; ensembles Gem+DS mean 0.307, Gem+DS mean+drop 0.347, DS+Mini soft 0.330, DS+Mini soft+drop 0.364 (best). — `Gem+DS` pairs Gemini-2.5-Flash-lite with DeepSeek-V3.1; `DS+Mini` pairs DeepSeek-V3.1 with GPT-4o-mini.

Complementary judge strengths

Single judges win on disjoint HealthBench themes. RaRE inherits, rather than averages out, the constituent judges’ theme-level strengths.

	DeepSeek V3.1	Gemini-2.5 flash-lite	GPT-4o mini	DeepSeek +Gemini	DeepSeek+ GPT-4o-mini
Comm.	+0.086	+0.045	+0.082	+0.090	+0.099
Complex	−0.024	−0.029	+0.026	−0.001	+0.039
Context	+0.111	+0.051	+0.098	+0.112	+0.118
Emerg.	+0.075	+0.023	+0.060	+0.072	+0.089
Global	+0.102	+0.041	+0.097	+0.106	+0.131
Data	+0.069	+0.034	+0.077	+0.080	+0.088
Hedging	+0.093	+0.065	+0.098	+0.114	+0.134

Mean-reward delta on English-only HealthBench (n=3931) vs. Qwen3-4B-Base across themes (Comm. = communication; Emerg. = emergency referrals). Ensembles: DeepSeek+Gemini, naive mean; DeepSeek+GPT-4o-mini, soft-Likert; both with drop. Bold: row winner.

Configuration ablations

Threshold sweep ( $\tau$ ). $\tau=0.30$ is best on the HealthBench 1K subset (Gemini+DeepSeek pair).

$\tau$	drop %	Score	95% CI
none (mean only)	0%	0.314	[0.295, 0.335]
0.15	14%	0.251	[0.233, 0.271]
0.25	9%	0.301	[0.281, 0.323]
0.30	5%	0.330	[0.312, 0.351]
0.35	2%	0.273	[0.254, 0.294]

Confidence aggregation. Replacing inverse-variance confidence with simpler schemes drops the score by 12–14 pp.

Confidence + aggregation	Score	95% CI
inverse-var, weighted avg	0.336	[0.316, 0.355]
top-logit, weighted avg	0.214	[0.195, 0.233]
top-logit, winner-takes-all	0.198	[0.179, 0.217]

BibTeX

@misc{liu2026judgingjudgesconfidencedisagreement,
      title={Judging the Judges: Confidence and Disagreement in Rubrics as Rewards},
      author={Michael Tian-Yue Liu and Anne Harrington and Trevor Darrell and Yutong Bai},
      year={2026},
      eprint={2610.21179},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.21179},
}