llm_judges: Miller error bars -- repeat draws, don't touch the thermostat, paired differences

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

SKILL.md

@@ -338,7 +338,7 @@ Open the relevant one when the task calls for it. These are synthesized checklis
 - [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
 - [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
 - [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
-- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference), the mitigation checklist, and wassname's judge-validity checklist (rubric, read-the-trace, saturation, anchoring, repeat variance, give-the-judge-a-voice), for when an LLM-judged eval looks too good.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference), the mitigation checklist, and wassname's judge-validity checklist (rubric, read-the-trace, saturation, anchoring, repeat variance, give-the-judge-a-voice), plus repeat draws, why dropping temperature is not a variance fix, and paired differences, for when an LLM-judged eval looks too good.
 - [refs/time_series.md](refs/time_series.md) — deployment-faithful temporal evaluation, causal missing-value handling, and properties that make forecasting problems easier or harder.
 - [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
 - [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.

docs/evidence/miller_2024_error_bars_evals.md

@@ -0,0 +1,63 @@
+Source: https://arxiv.org/pdf/2411.00640 (Evan Miller, Anthropic, Nov 2024) + https://www.anthropic.com/research/statistical-approach-to-model-evals
+Title: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
+Fetched-via: r.jina.ai on the arXiv PDF and the Anthropic post, 2026-08-16
+Fetch-status: verbatim from full PDF text (math notation mangled by the PDF-to-markdown pass; prose is clean)
+Used-by: refs/llm_judges.md (repeat draws, temperature, paired differences)
+
+# Adding Error Bars to Evals (excerpts)
+
+## The five recommendations, verbatim from Section 1
+
+> Our specific recommendations to researchers include: 1. Computing standard errors of the mean using the Central Limit Theorem 2. When questions are drawn in related groups, computing clustered standard errors 3. Reducing variance by resampling answers and by analyzing next-token probabilities 4. When two models are being compared, conducting statistical inference on the question-level paired differences, rather than the population-level summary statistics 5. Using power analysis to determine whether an eval (or a random subsample) is capable of testing a hypothesis of interest
+
+The framing that makes the rest work, from Section 2:
+
+> Suppose that the questions in an eval do not represent all possible questions, but instead were drawn at random from a (hypothetical, infinite, unseen) super-population of questions. This simple supposition lets us jump "through the looking glass" of the specific questions that appear in an eval in order to study the underlying skill that the eval is attempting to measure.
+
+Reporting practice, from the Table 2 caption:
+
+> We suggest two new reporting practices: including the number of questions in each eval, and the standard error of each estimate in parentheses (fictional models and numbers).
+
+## Section 3.1: repeat draws help, with diminishing returns
+
+The worked binary-score example with uniformly distributed question difficulty:
+
+> Going from K = 1 (no resampling of answers) to K = 2, the total variance is reduced by 1/3. Increasing to K = 4, we have a variance reduction of 1/2, and setting K = 6, we reduce variance by 5/9. The upper limit on variance reduction via resampling in this example is 2/3. Note that computing a pooled standard error across all KN answers will be inconsistent, as multiple answers to the same question would violate the assumption of independent draws.
+
+So the draws only ever remove the response-level (conditional) variance. Question-difficulty
+variance is the floor, and only more questions moves it.
+
+## Section 3.3: "Don't touch the thermostat!"
+
+> It may be tempting to reduce the "sampling temperature" of the model in order to reduce (or eliminate) the conditional variance. However, we advise against this practice, unless the purpose is to study the model at the new temperature. Besides altering the model's behavior, adjusting the sampling temperature may simply shift the conditional variance (which can be mitigated using the two techniques above) into the variance of the conditional means (which cannot), or else reduce conditional variance by injecting bias into the estimator.
+
+The first worked counter-example, a single-token true/false eval with difficulty x ~ U[0,1]:
+
+> As in Section 3.1, Var(x_T=1) = 1/12. But at T = 0, x_T=0 = 1{x_T=1 > 0.5} and the uniform distribution is "rounded" into a Bernoulli distribution with p = 1/2. So Var(x_T=0) = 1/4. In this case, reducing the sampling temperature, and thereby eliminating the conditional variance, has inadvertently tripled the minimum variance in the score data from 1/12 to 1/4.
+
+The second, where the mean moves too, with difficulty x ~ U[1/3, 1]:
+
+> Then E[x_T=1] = 2/3 < E[x_T=0] = 3/4 and Var(x_T=1) = 1/27 << Var(x_T=0) = 3/16; that is, not only has the temperature change shifted the expected score, but the variance of the conditional means has increased approximately five-fold.
+
+The closing rule of the section:
+
+> When next-token probabilities are not available, or the answer requires a chain of thought or other complex interaction, choose a K such that E[sigma_i^2]/K << Var(x) and compute the standard error across question-level mean scores. In neither case should the sampling temperature be adjusted for the sake of reducing variance in the scores.
+
+## Section 4.2: paired differences
+
+> The naive comparison above misses an opportunity to reduce the standard error when two models evaluate the same set of questions.
+
+The Anthropic post gives the size of the effect in practice:
+
+> Since the question list is shared across models, conducting a paired-differences test lets us eliminate the variance in question difficulty and focus on the variance in responses. [...] In practice, we find the correlation of question scores on popular evals between frontier models to be substantial - between 0.3 and 0.7 on a scale of -1 to +1. Put another way, frontier models have an overall tendency to get the same questions right and wrong. Paired-difference analysis thus represents a "free" variance reduction technique that is very well suited for AI model evals.
+
+And it names an existing implementation of the resampling recommendation:
+
+> If an eval uses chain-of-thought reasoning, we recommend resampling answers from the same model several times, and using the question-level averages as the question scores fed into the Central Limit Theorem. We note that the Inspect framework correctly computes standard errors in this way via its _epochs_ parameter.
+
+epistemic context: arXiv stat.AP preprint by an Anthropic author, with a first-party company post
+summarising it. Not peer reviewed, but the statistics are textbook (CLT, clustered SEs, paired
+t-test, power analysis) rather than novel claims, and the recommendations are now visible in
+tooling (Inspect epochs). The variance-reduction fractions above are all from one worked
+uniform-difficulty toy example, not measured on a real eval; treat the direction as general and
+the numbers as illustrative.

refs/llm_judges.md

@@ -104,6 +104,18 @@ Check stability across order and repeats:
 - Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest.
 - Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same canary as [seed variance](../SKILL.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
 
+## Repeat draws, temperature, and paired differences
+
+The repeat passes above are a validity check, but they are also the standard variance-reduction
+move, and two of Evan Miller's five recommendations in "Adding Error Bars to Evals" are exactly
+the checklist items here.[^miller] Both are cheap and neither needs more questions.
+
+- **Average K draws per question, then take the standard error across question means.** Miller's worked binary example with uniform question difficulty: "Going from K = 1 (no resampling of answers) to K = 2, the total variance is reduced by 1/3. Increasing to K = 4, we have a variance reduction of 1/2, and setting K = 6, we reduce variance by 5/9. The upper limit on variance reduction via resampling in this example is 2/3."[^miller] So draws only remove response-level noise. Question-difficulty variance is the floor and only more questions moves it. Do not pool the K*N answers into one standard error, that "will be inconsistent, as multiple answers to the same question would violate the assumption of independent draws".[^miller] inspect's `epochs` parameter already reduces this way.
+- **Do not drop temperature to make the numbers look stable.** Section 3.3 is titled "Don't touch the thermostat!": "adjusting the sampling temperature may simply shift the conditional variance (which can be mitigated using the two techniques above) into the variance of the conditional means (which cannot), or else reduce conditional variance by injecting bias into the estimator."[^miller] In his single-token true/false example, going to T=0 rounds a uniform difficulty distribution into a Bernoulli one and *triples* the score variance, from 1/12 to 1/4; a second example moves the mean as well, 2/3 to 3/4. T=0 is a legitimate choice when you want to study the model at T=0, but it is not a variance fix, and it makes repeat draws useless as a noise measurement because the draws are no longer independent samples of the model's behaviour.
+- **Compare on question-level paired differences, not on two separate bars.** Same rule as [same-seed paired comparison in sweeps](sweeps.md), and it applies to judge scores too: score both arms on the same questions and do inference on the per-question difference. Anthropic's post reports question-score correlations "between 0.3 and 0.7" between frontier models on popular evals, so the pairing is "a 'free' variance reduction technique".[^miller]
+
+Report the number of questions beside every score, not just the percentage.
+
 Give the judge a voice, and save everything:
 
 - Add a free-text field for the judge to flag a broken, missing, or ambiguous rubric or context. Read it, but keep it out of the score. Do the same for the evaluated agents: an unscored exit interview about ambiguity, missing context, broken tools, and unnatural constraints, kept separate from the task score. Say in the field description that it is never scored, or the agent treats a complaint as an admission and stays quiet. Read both sides during the audit: they disagree, and the disagreement is the finding.
@@ -179,6 +191,7 @@ For a worked example, wassname has a ~300-line async OpenRouter judge (WIP) that
 [^overthink]: Ghosal et al., "Does Thinking More always Help? ... Mirage of Test-Time Scaling in Reasoning Models" (2025) — https://arxiv.org/pdf/2506.04210 (preprint; one controlled study, accuracy-vs-thinking-token curve is non-monotonic, peak setup-specific) ([litreview](llm_judge_litreview.md))
 [^cais]: CAIS simple-evals .env.example vs litellm constants.py (fetched 2026-07, directly verifiable config) — https://github.com/centerforaisafety/simple-evals/blob/main/.env.example (effort high=24576/med=8192/low=1024, overriding litellm stock 4096/2048/1024)
 [^loo]: Loo, "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" (2025) — https://arxiv.org/pdf/2511.00751 (single-author preprint, low citation signal; plateau ~N=10-15 on modern models, can decline past it) ([litreview](llm_judge_litreview.md))
+[^miller]: Evan Miller (Anthropic), "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (2024) — https://arxiv.org/pdf/2411.00640, short version https://www.anthropic.com/research/statistical-approach-to-model-evals ([cache](../docs/evidence/miller_2024_error_bars_evals.md)). arXiv stat.AP preprint, not peer reviewed, but the statistics are textbook and the recommendations show up in tooling (inspect `epochs`). The variance fractions come from one uniform-difficulty toy example, so treat the direction as general and the numbers as illustrative.
 [^nolima]: Modarressi et al., "NoLiMa: Long-Context Evaluation Beyond Literal Matching" (ICML 2025) — https://arxiv.org/pdf/2502.05167 (peer-reviewed; effective length = length holding 85% of base score; most models below half by 32K once literal cues removed) ([litreview](llm_judge_litreview.md))
 [^petri]: Petri 3.0 judge — https://github.com/meridianlabs-ai/inspect_petri/blob/main/src/inspect_petri/_judge/judge.py (dynamic `create_model` answer schema; `JUDGE_PROMPT` with the prefill-attribution caps and the empty-transcript rule; refusal returns `Result(value=None, metadata={"refusal": True})`). Maintained by Meridian Labs, used in Anthropic's alignment audits; strong trust signal as engineering, but it is one team's design, not a measured result.
 [^petri-dim]: Petri judge dimensions — https://github.com/meridianlabs-ai/inspect_petri/tree/main/src/inspect_petri/_judge/dimensions (one markdown file per dimension with YAML front matter; `concerning.md` quoted above)