llm_judges: Miller error bars -- repeat draws, don't touch the thermostat, paired differences

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

refs/llm_judges.md

@@ -104,6 +104,18 @@ Check stability across order and repeats:
 - Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest.
 - Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same canary as [seed variance](../SKILL.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
 
+## Repeat draws, temperature, and paired differences
+
+The repeat passes above are a validity check, but they are also the standard variance-reduction
+move, and two of Evan Miller's five recommendations in "Adding Error Bars to Evals" are exactly
+the checklist items here.[^miller] Both are cheap and neither needs more questions.
+
+- **Average K draws per question, then take the standard error across question means.** Miller's worked binary example with uniform question difficulty: "Going from K = 1 (no resampling of answers) to K = 2, the total variance is reduced by 1/3. Increasing to K = 4, we have a variance reduction of 1/2, and setting K = 6, we reduce variance by 5/9. The upper limit on variance reduction via resampling in this example is 2/3."[^miller] So draws only remove response-level noise. Question-difficulty variance is the floor and only more questions moves it. Do not pool the K*N answers into one standard error, that "will be inconsistent, as multiple answers to the same question would violate the assumption of independent draws".[^miller] inspect's `epochs` parameter already reduces this way.
+- **Do not drop temperature to make the numbers look stable.** Section 3.3 is titled "Don't touch the thermostat!": "adjusting the sampling temperature may simply shift the conditional variance (which can be mitigated using the two techniques above) into the variance of the conditional means (which cannot), or else reduce conditional variance by injecting bias into the estimator."[^miller] In his single-token true/false example, going to T=0 rounds a uniform difficulty distribution into a Bernoulli one and *triples* the score variance, from 1/12 to 1/4; a second example moves the mean as well, 2/3 to 3/4. T=0 is a legitimate choice when you want to study the model at T=0, but it is not a variance fix, and it makes repeat draws useless as a noise measurement because the draws are no longer independent samples of the model's behaviour.
+- **Compare on question-level paired differences, not on two separate bars.** Same rule as [same-seed paired comparison in sweeps](sweeps.md), and it applies to judge scores too: score both arms on the same questions and do inference on the per-question difference. Anthropic's post reports question-score correlations "between 0.3 and 0.7" between frontier models on popular evals, so the pairing is "a 'free' variance reduction technique".[^miller]
+
+Report the number of questions beside every score, not just the percentage.
+
 Give the judge a voice, and save everything:
 
 - Add a free-text field for the judge to flag a broken, missing, or ambiguous rubric or context. Read it, but keep it out of the score. Do the same for the evaluated agents: an unscored exit interview about ambiguity, missing context, broken tools, and unnatural constraints, kept separate from the task score. Say in the field description that it is never scored, or the agent treats a complaint as an admission and stays quiet. Read both sides during the audit: they disagree, and the disagreement is the finding.
@@ -179,6 +191,7 @@ For a worked example, wassname has a ~300-line async OpenRouter judge (WIP) that
 [^overthink]: Ghosal et al., "Does Thinking More always Help? ... Mirage of Test-Time Scaling in Reasoning Models" (2025) — https://arxiv.org/pdf/2506.04210 (preprint; one controlled study, accuracy-vs-thinking-token curve is non-monotonic, peak setup-specific) ([litreview](llm_judge_litreview.md))
 [^cais]: CAIS simple-evals .env.example vs litellm constants.py (fetched 2026-07, directly verifiable config) — https://github.com/centerforaisafety/simple-evals/blob/main/.env.example (effort high=24576/med=8192/low=1024, overriding litellm stock 4096/2048/1024)
 [^loo]: Loo, "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" (2025) — https://arxiv.org/pdf/2511.00751 (single-author preprint, low citation signal; plateau ~N=10-15 on modern models, can decline past it) ([litreview](llm_judge_litreview.md))
+[^miller]: Evan Miller (Anthropic), "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (2024) — https://arxiv.org/pdf/2411.00640, short version https://www.anthropic.com/research/statistical-approach-to-model-evals ([cache](../docs/evidence/miller_2024_error_bars_evals.md)). arXiv stat.AP preprint, not peer reviewed, but the statistics are textbook and the recommendations show up in tooling (inspect `epochs`). The variance fractions come from one uniform-difficulty toy example, so treat the direction as general and the numbers as illustrative.
 [^nolima]: Modarressi et al., "NoLiMa: Long-Context Evaluation Beyond Literal Matching" (ICML 2025) — https://arxiv.org/pdf/2502.05167 (peer-reviewed; effective length = length holding 85% of base score; most models below half by 32K once literal cues removed) ([litreview](llm_judge_litreview.md))
 [^petri]: Petri 3.0 judge — https://github.com/meridianlabs-ai/inspect_petri/blob/main/src/inspect_petri/_judge/judge.py (dynamic `create_model` answer schema; `JUDGE_PROMPT` with the prefill-attribution caps and the empty-transcript rule; refusal returns `Result(value=None, metadata={"refusal": True})`). Maintained by Meridian Labs, used in Anthropic's alignment audits; strong trust signal as engineering, but it is one team's design, not a measured result.
 [^petri-dim]: Petri judge dimensions — https://github.com/meridianlabs-ai/inspect_petri/tree/main/src/inspect_petri/_judge/dimensions (one markdown file per dimension with YAML front matter; `concerning.md` quoted above)