Make 'Numbers worth knowing' quote-anchored with source-credence + calibrated language

Each bullet now leads with the action, carries a verbatim quote + footnote, states
source type (independent benchmark / peer-reviewed / single-group preprint) so the
reader can weight it, and uses Kent-style estimative language on the takeaway.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

docs/evidence/llm_judge_biases.md

@@ -39,3 +39,360 @@ Self-preference is causally linked to self-recognition:
 > Constrained decoding methods for structured outputs (e.g., JSON-mode) impose an inductive bias on the model's output distribution.
 
 Related: JudgeBench leaderboard for judge quality — https://huggingface.co/spaces/ScalerLab/JudgeBench
+
+---
+
+Source: Hamel Husain, Databricks (x2), Eugene Yan blog posts (via WebFetch)
+Title: practitioner rules on scale precision, rubric quality, and reading judge traces
+Fetched-via: WebFetch summarizer model, 2026-07-22; short quotes cross-checked by re-fetching
+Fetch-status: quotes below reproduced twice identically EXCEPT the Databricks scale range, where two fetches disagreed (0-3 or 0-4 vs 0-3 or 1-5) so it is paraphrased, not quoted, in the ref
+
+## Hamel Husain, "Creating a LLM-as-a-Judge That Drives Business Results" — https://hamel.dev/blog/posts/llm-judge/
+
+Critique-shadowing workflow; look at the data before writing the judge, prefer binary:
+
+> You cannot write a good judge prompt until you've seen the data.
+
+> If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong.
+
+Onward links from this post: Shankar et al., "Who Validates the Validators?" (arXiv:2404.12272); Yan, "ALIGN Eval" (https://aligneval.com/); OpenAI Cookbook, "Custom LLM as a Judge to Detect Hallucinations with Braintrust" (https://cookbook.openai.com/examples/custom-llm-as-a-judge); "What We've Learned From A Year of Building with LLMs" (https://applied-llms.org/).
+
+## Databricks, "Best Practices for LLM Evaluation of RAG Applications" — https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG
+
+Use a low-precision integer scale:
+
+> Scales like 0-10 are difficult to come up with distinguishing criteria between all scores.
+
+The recommended range paraphrased (fetches disagreed on 0-4 vs 1-5): a coarse 0-3 or 1-5 Likert scale, with an example for each score in the range. Onward links: LMSYS MT-Bench (arXiv:2306.05685) and its FastChat judge prompts (https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/data/judge_prompts.jsonl).
+
+## Databricks, "Enhancing LLM-as-a-Judge with Grading Notes" — https://www.databricks.com/blog/enhancing-llm-as-a-judge-with-grading-notes
+
+Per-question domain rubrics fix the domain-knowledge gap:
+
+> Without Grading Notes, both judge-LLMs overestimate the effectiveness by a significant margin, likely indicating the gap in domain knowledge to criticize.
+
+> With Grading Notes introducing brief domain knowledge, both judge LLMs showed significant improvement in the alignment rate with humans, especially in the case of GPT-4: alignment rate increased to 96.3% for Llama3 and 93.1% for GPT-4o, which corresponds to 85% and 67.5% reduction in misalignment rate, respectively.
+
+## Eugene Yan, "Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)" — https://eugeneyan.com/writing/llm-evaluators/
+
+A survey; the few-shot-instability line is attributed to Luo et al., "ChatGPT as a Factual Inconsistency Evaluator" (arXiv:2303.15621):
+
+> performance unstable when changing the label, example order, and number of examples
+
+Key papers the survey collects (worth reading past this file):
+- Zheng et al., MT-Bench — position/verbosity/self-enhancement bias — arXiv:2306.05685
+- Liu et al., G-Eval — CoT scoring with GPT-4 — arXiv:2303.16634
+- Doddapaneni et al., "Finding Blind Spots in Evaluator LLMs with Interpretable Checklists" — GPT-4 misses >50% of quality drops on several axes — arXiv:2406.13439
+- Huang et al., "On the Limitations of Fine-Tuned Judge Models" — finetuned judges fail out-of-domain — arXiv:2403.02839
+- Shankar et al., "Who Validates the Validators?" — aligning LLM eval with human preferences — arXiv:2404.12272
+
+Judge leaderboards (live, not cached): Judgemark v4 (https://eqbench.com/judgemark-v4.html), SpeechMap refusal rates (https://speechmap.ai/).
+
+---
+
+Source: arXiv abstracts (via WebFetch) + EQ-bench-site repo
+Title: evaluator blind spots (Doddapaneni), criteria drift (Shankar), Judgemark v4 cost/score frontier
+Fetched-via: WebFetch of arXiv abstract pages, 2026-07-22; Judgemark scores read from the checked-in judgemark-v4.js in EQ-bench/EQ-bench-site
+Fetch-status: paper quotes verbatim from abstracts; Judgemark numbers copied from the repo's data rows (not the rendered site table)
+
+## Doddapaneni et al., "Finding Blind Spots in Evaluator LLMs with Interpretable Checklists" (2024) — https://arxiv.org/abs/2406.13439
+
+Evaluator LLMs miss most injected quality drops:
+
+> Our findings reveal significant shortcomings in current Evaluator LLMs, which failed to identify quality drops in over 50% of cases on average.
+
+## Shankar et al., "Who Validates the Validators?" (2024) — https://arxiv.org/abs/2404.12272
+
+Criteria drift, and the validator-needs-validation problem:
+
+> LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation.
+
+> users need criteria to grade outputs, but grading outputs helps users define criteria
+
+> some criteria appears dependent on the specific LLM outputs observed (rather than independent criteria that can be defined a priori)
+
+## Judgemark v4 cost-vs-score frontier (snapshot 2026-07)
+
+Data rows from https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js (columns: model, score, lower-CI, upper-CI, USD cost). Higher score = better at separating stronger from weaker creative writing. Pareto frontier (maximize score, minimize cost):
+
+| model | score | cost | note |
+|-|-|-|-|
+| claude-opus-4-6 | 0.907 | $39.37 | top absolute score |
+| gpt-5.5 | 0.878 | $30.44 | |
+| claude-sonnet-4-6 | 0.821 | $23.36 | |
+| gemini-3.1-pro-preview | 0.787 | $23.07 | |
+| grok-4.5 | 0.771 | $17.11 | low refusal |
+| zai-org/GLM-5.2 | 0.732 | $8.28 | Chinese, censors CN-political |
+| google/gemma-4-31B-it | 0.723 | $0.82 | cheap knee; refuses many contentious tasks |
+| google/gemma-4-26B-A4B-it | 0.530 | $0.61 | |
+| Qwen/Qwen3.5-9B | 0.324 | $0.56 | cheapest |
+
+Off-frontier for reference: Qwen3.5-27B 0.605 ($1.76), DeepSeek-V4-Pro 0.471 ($2.94), grok-4.3 0.496 ($9.71). So in v4, Qwen and DeepSeek are mid-pack, not frontier; grok-4.5 and gemma carry the value case.
+
+## SpeechMap per-lab Free Speech Index (snapshot 2026-07-21) — https://speechmap.ai/labs/
+
+Higher = answers more / refuses less (0-100), windowed to releases in the last 6 months; "Index" is the windowed score, "Peak" the best single model. This is a cross-topic aggregate and a lab average, so it does not resolve per-model or per-topic refusal (e.g. Chinese labs score mid-to-high here but still refuse on Chinese-political topics; a lab's aggregate can hide a heavily safety-tuned model like Gemma). Copied verbatim from the table wassname pasted:
+
+| Rank | Lab | Index | Peak | Models |
+|-|-|-|-|-|
+| 1 | Mistral AI | 88.9 | 98.2 | 9 |
+| 2 | xAI | 85.8 | 98.2 | 5 |
+| 3 | IBM | 84.0 | 84.0 | 1 |
+| 4 | Google DeepMind | 81.1 | 88.4 | 9 |
+| 5 | Meta | 76.8 | 76.8 | 1 |
+| 6 | Zhipu AI (GLM) | 71.0 | 85.9 | 12 |
+| 7 | Arcee AI | 70.1 | 82.2 | 3 |
+| 8 | Meituan | 63.6 | 78.2 | 2 |
+| 9 | Tencent | 60.3 | 76.3 | 4 |
+| 10 | DeepSeek | 59.0 | 89.1 | 7 |
+| 11 | Moonshot AI | 57.6 | 65.9 | 5 |
+| 13 | Anthropic | 53.8 | 71.5 | 10 |
+| 16 | OpenAI | 48.0 | 66.7 | 15 |
+| 19 | Alibaba (Qwen) | 45.5 | 60.8 | 13 |
+| 20 | NVIDIA | 45.3 | 67.7 | 7 |
+| 23 | Xiaomi | 43.9 | 62.6 | 10 |
+| 24 | ByteDance | 32.1 | 34.4 | 3 |
+
+Note the wide Peak-vs-Index gaps (DeepSeek peak 89.1 vs index 59.0; Zhipu 85.9 vs 71.0): pick the specific permissive checkpoint, not the lab.
+
+Not fetched: SpeechMap's per-model and per-topic breakdowns (https://speechmap.ai/timeline/, /labs/<lab>/) and the eqbench4 EI benchmark data (https://github.com/EQ-bench/EQ-bench-site/blob/main/eqbench4/eqbench4_data.js) are separate from the judge data used here.
+
+## Reasoning-token budget: Epoch AI + DeepSeek-R1
+
+Epoch AI, "Output length" — https://epoch.ai/data-insights/output-length (via WebFetch, 2026-07-22):
+
+> Reasoning models also exhibit longer response lengths overall - currently, around 8x more tokens on average, compared to non-reasoning models.
+
+> Reasoning models' responses are growing considerably faster (5x per year) than those from non-reasoning models (2.2x per year).
+
+> moving from 'medium' to 'high' effort resulted in a 1.6x increase in output tokens
+
+DeepSeek-AI, "DeepSeek-R1" (arXiv:2501.12948), abstract quotes:
+
+> The reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories.
+
+> emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation
+
+(The response-length-grows-over-training result is R1's headline figure in the body; not re-quoted verbatim here.) Also unverified: the per-model Artificial Analysis token-use splits (https://artificialanalysis.ai/models/qwen3-6-27b#intelligence-index-token-use-tabs) that wassname read as ~5k Gemma-4-31b to ~30k Qwen3.6-35B-A3B; the dashboard is JS-rendered and WebFetch only returned aggregate totals.
+
+---
+
+# 2026 lit-search batch (added 2026-07-23, CLAUDE agent)
+
+Verification legend for this batch:
+- [FT] full-text verified: I curled the paper HTML or raw source this turn and the quote below is copied from that fetch.
+- [ID] arXiv id + title resolved this turn (real paper, topic matches); the in-body number was extracted by a research subagent via WebFetch's summarizer, NOT copied from raw PDF. Trust the direction; re-pull the exact figure before quoting as gospel.
+
+## Reasoning-effort token budgets: CAIS simple-evals + litellm defaults — [FT]
+
+CAIS `simple-evals/.env.example` (curled from raw github, 2026-07-23), the effort->token map a serious eval harness ships:
+
+> DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET=24576
+> DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET=8192
+> DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET=1024
+
+litellm's OWN stock defaults (`litellm/constants.py`, curled same day) are ~6x lower at the top end:
+
+> DEFAULT_REASONING_EFFORT_DISABLE_THINKING_BUDGET = ... 0
+> DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET = ... 128
+> DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET = ... 1024
+> DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET = ... 2048
+> DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET = ... 4096
+
+Takeaway: "high effort" is not a fixed token count. litellm stock caps high at 4096; CAIS deliberately overrides to 24576 (6x). So when you set effort=high on a judge you must know which mapping is live, or you may be truncating reasoning at 4k without meaning to. Sources: https://github.com/centerforaisafety/simple-evals/blob/main/.env.example , https://github.com/BerriAI/litellm/blob/main/litellm/constants.py
+
+## Position bias, headline flip rates
+
+## Lech Mazur, position_bias benchmark — https://github.com/lechmazur/position_bias — [FT]
+Independent, outsider-run, continuously-updated public harness (strong trust). Method: same story pair shown in both orders, 193 verified pairs, 36 models, 386 prompts/model. Numbers copied from raw README this turn:
+
+> The benchmark finds a large position effect in the current results. The model-average order-flip rate is 43.0%, and the median model flips in 41.3% of decisive two-view cases.
+
+> the model-average first-shown pick rate is 64.3%
+
+> The model-average first-position rating bonus is +0.271 on the 1-to-7 rating scale.
+
+> Mistral Medium 3.5 is the most position-sensitive model in this run: 82.8% first-shown pick rate, +32.8 pp first lift, 72.5% order flip
+
+Rule of thumb: even in 2026, judges flip ~43% of decisive verdicts on order swap alone; the worst flip >70%. Direction is not universal (Mistral Large 3 goes second-position). Always judge both orders. (Note: an earlier subagent draft misattributed 27.4% to Claude Opus; the raw table puts 27.4% first-shown pick on Mistral Large 3, so I dropped the per-model attributions except the verified worst-case.)
+
+## "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" — Shi et al. (Dartmouth), IJCNLP-AACL 2025 — https://arxiv.org/abs/2406.07791 — [ID]
+Most-cited dedicated position-bias study, largest scale (150k+ instances, 15 judges, 22 tasks).
+
+> Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances.
+
+> While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions.
+
+Rule of thumb: position bias is systematic, not random noise, and it gets WORSE as the two answers converge in quality (exactly when you most need the judge). Metrics introduced: repetition stability, position consistency, preference fairness.
+
+## "Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models" — Norman, Rivera, Hughes (UC Berkeley), 2026 — https://arxiv.org/abs/2606.19544 — [ID]
+2026 audit of 21 judges. Title confirmed via arxiv abs this turn; in-body numbers via subagent WebFetch (re-fetched HTML mirror after PDF parse failed).
+
+> High test-retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency-bias paradox).
+
+> Across a range of judge models, flip rates range from 25% to 50%
+
+> All 21 judges evaluated under the bias-audit protocol register a verbosity bias below 0.011 on MT-Bench, with the largest value being GPT-4o-mini at 0.010
+
+Two rules of thumb: (1) a judge being REPRODUCIBLE (same verdict on re-run) does not make it VALID (right, or order-invariant); measure both. (2) Verbosity bias in current-gen judges is ~10x smaller than 2023-era studies reported, so don't over-correct for length on modern judges. Contested-direction flag lives with the self-preference entries below.
+
+## Self-preference / self-enhancement scales inversely with size
+
+## "Beyond the Surface: Measuring Self-Preference in LLM Judgments" — Chen et al., EMNLP 2025 main — https://arxiv.org/abs/2506.02592 — [FT]
+Machine-accessible: https://github.com/zhiyuanc2001/self-preference . Its DBG score uses gold judgments to separate bias from genuine quality. Quotes copied from arxiv HTML full-text this turn:
+
+> the DBG score of Qwen2.5-0.5B-Instruct is 41.7%. In contrast, the DBG score of Qwen2.5-14B-Instruct is only 2.1%.
+
+> the DBG score of Llama-3.1-70B is 0.4%, whereas that of Llama-3.1-8B is 21.6%, which is much higher than the score of Llama-3.1-70B.
+
+> Larger models exhibit less self-preference bias compared to smaller models.
+
+> the self-preference bias in reasoning models is not necessarily less significant than the bias found in language models. For instance, the DBG score of DS-R1-Distill-Qwen-32B is 4.8%, whereas the DBG score of Qwen2.5-72B-Instruct is only 2.6%.
+
+Rule of thumb: self-preference is inversely proportional to size. Tiny judges (<1B) can inflate their own scores ~40%; strong large judges drop to low single digits. Reasoning does NOT reliably remove it. Do not use a small model to judge its own family's outputs.
+
+## "Do LLM Evaluators Prefer Themselves for a Reason?" — Chen et al., 2025 — https://arxiv.org/abs/2504.03846 — [ID]
+The counter-intuitive one: CoT REDUCES self-preference (contradicts naive "more thinking = more bias"). Title confirmed this turn; HSPP numbers via subagent WebFetch.
+
+> generating reasoning traces substantially reduces harmful self-preference across all models
+
+> For MATH500, harmful self-preference propensity (HSPP) dropped from 56.9% (no reasoning) to 19.3% (standard CoT) to 17.0% (long CoT)
+
+Tension to flag: this paper says CoT roughly halves self-preference; 2506.02592 (above) says LRMs "not necessarily less" biased. Different metrics (HSPP vs DBG), so the direction of "does reasoning fix self-preference" is contested. State it as open, not settled.
+
+## Reasoning judges: accuracy up, superficial bias not fixed
+
+## "JudgeLRM: Large Reasoning Models as a Judge" — Chen et al., 2025 — https://arxiv.org/abs/2504.00050 — [ID]
+RL-trained reasoning judge. Title confirmed; F1 numbers via subagent WebFetch.
+
+> JudgeLRM achieves an average improvement of 8.14% in F1 score [vs same-size SFT judges]
+
+> On the human-annotated PandaLM benchmark, JudgeLRM-3B surpasses GPT-4 [F1 72.12% vs 61.80%]
+
+## "Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases" — Huang et al., Jan 2026 — https://arxiv.org/abs/2601.03630 — [ID]
+
+> LRMs outperform non-reasoning LLMs in terms of judgment accuracy, particularly on reasoning-intensive tasks
+
+> [LRMs] still exhibit strong evaluation biases
+
+Rule of thumb across these two: prefer a reasoning judge for reasoning-heavy grading (~5-8 F1 gain), but it does not remove length/position/style bias. "Use a reasoning judge" is defensible; "reasoning fixes bias" is not.
+
+## "Explicit Reasoning Makes Better Judges" — 2025 — https://arxiv.org/abs/2509.13332 — [ID]
+Directly tests small judges (Qwen3 0.6B/1.7B/4B). (Subagent first mislabeled the title as "Thinking Small Models..."; corrected to the real arxiv title this turn.)
+
+> thinking models achieve approximately 10% points higher accuracy with little overhead (under 2x), in contrast to augmentation strategies like few-shot learning, which deliver modest gains at a higher cost (>8x).
+
+> The smallest model in our study (Qwen 3 0.6B) fails to surpass 50% accuracy on difficult 'Chat Hard' and 'Safety' tasks, in some cases performing worse than random selection.
+
+> when subjected to verbosity bias, the thinking model exhibits a higher consistency (83.48 vs 73.86)
+
+Rule of thumb: sub-1B judges fall to random on hard/safety pairs; turning on reasoning buys ~+10 accuracy and higher bias-consistency far cheaper than few-shot ICL (<2x cost vs >8x).
+
+## "RLAIF vs. RLHF" — Lee et al. (Google), ICML 2024 — https://arxiv.org/abs/2309.00267 — [FT]
+The canonical "smaller = more position-biased" source. Main-text quotes copied from arxiv HTML this turn; the 18/21/56% per-size figures are in its Appendix B (table, not captured by my main-text grep).
+
+> We find evidence of position bias, which is especially prevalent in smaller LLM labelers
+
+> two inferences are made for every pair of candidates, where the order in which candidates are presented to the LLM is reversed for the second inference. The results from both inferences are then averaged to obtain the final preference distribution.
+
+> Alignment decreases by 4% when substituting PaLM 2 L with PaLM 2 S, and decreases another 11% when using PaLM 2 XS
+
+Reported (Appendix B, via subagent): PaLM 2 L/S/XS keep the same position after swap 18% / 21% / 56% of the time; for L, 94% of same-position cases favor the first candidate. Rule of thumb: the order-swap-and-average mitigation is standard and it comes from here; smaller judges need it most.
+
+## Overthinking: token budget vs task difficulty is non-monotonic
+
+## "Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models" — Ghosal et al., 2025 — https://arxiv.org/abs/2506.04210 — [ID]
+Cleanest non-monotonic curve. Title confirmed this turn; numbers via subagent WebFetch.
+
+> accuracy increases from 82.2% to 87.3% as the average number of thinking tokens increases from 385 to 1100. However... pushing the average thinking token count from 1100 to 15980 reduces accuracy from 87.3% to 70.3%
+
+Rule of thumb: return on thinking tokens peaks then declines. In their setup peak was ~1.1k tokens; 14x more tokens (16k) cost ~17 accuracy points. Past the peak, extra tokens add variance, not reasoning.
+
+## "OptimalThinkingBench: Evaluating Over and Underthinking in LLMs" — Aggarwal et al., 2025 — https://arxiv.org/abs/2508.13141 — [ID]
+
+> Thinking models often overthink for hundreds of tokens on the simplest user queries without improving performance. In contrast, large non-thinking models underthink, often falling short of much smaller thinking models.
+
+Rule of thumb: easy items hit negative marginal utility of thinking earlier than hard items; no current model budgets thinking optimally, so difficulty-aware caps beat a fixed cap. This is the evidence base for "cap reasoning low on easy tasks, spend the savings on N passes."
+
+## Self-consistency convergence: how many samples N
+
+## "Self-Consistency Improves Chain of Thought Reasoning" — Wang et al., 2022 — https://arxiv.org/abs/2203.11171 — [ID]
+Foundational (several-thousand citations), PaLM-540B era.
+
+> GSM8K (+17.9%) [self-consistency over CoT; 56.5% -> 74.4% at N=40]
+
+Widely-reproduced pattern: gain is monotonic in N with diminishing returns; bulk arrives by N=5-10, saturates ~N=40 for that era's models.
+
+## "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" — Loo, 2025 — https://arxiv.org/abs/2511.00751 — [ID]
+Single-author preprint (low citation signal, flagged), but directly answers "how has N moved." Numbers via subagent WebFetch.
+
+> [MATH-500, Gemini-2.5-Flash-Lite] accuracy improved through approximately 10 sampled paths before plateauing... declining slightly beyond 15
+
+> [MATH-500, Gemini-2.5-Pro] ... improved to 99.2% at 3 paths and 99.6% at 15... a total gain of 1.6% at approximately 15x the single-sample token cost
+
+Rule of thumb for N: on strong 2026 models the self-consistency plateau moved in to N~10-15 (from ~40), total gain shrank to <2 points, and accuracy can DECLINE past the plateau. Reserve repeats for genuinely hard items where the base model is well below ceiling. This also sets the sane N for your N=4 repeat-variance check: 4-10 is plenty to see instability; going past ~15 buys nothing.
+
+## "Inference-Time Scaling for Generalist Reward Modeling" (DeepSeek-GRM) — Liu et al., 2025 — https://arxiv.org/abs/2504.02495 — [ID]
+Vendor paper (mild caution). Numbers via subagent WebFetch.
+
+> Direct voting with 32 samples of DeepSeek-GRM-27B could achieve comparable performance to the 671B MoE model
+
+> [ReaLMistake] inference-time scaling with 32 samples improved from 67.9 (voting@1) to 72.8 (voting@32 with meta RM)
+
+Rule of thumb: sampling+voting a small generative judge 32x can match a ~25x-larger single-shot judge, and a learned meta-verifier over the votes beats plain majority vote. Scaling judge COMPUTE can substitute for judge SIZE.
+
+## Context rot: long inputs/rubrics degrade judging
+
+## Chroma, "Context Rot: How Increasing Input Tokens Impacts LLM Performance" — Hong, Troynikov, Huber, July 2025 — https://research.trychroma.com/context-rot — [ID]
+Industry report (not peer-reviewed), 18-model controlled study. Numbers via subagent WebFetch.
+
+> Even a single distractor reduces performance relative to the baseline (needle only), and adding four distractors compounds this degradation further.
+
+> as needle-question similarity decreases, model performance degrades more significantly with increasing input length
+
+Rule of thumb: degradation is continuous and starts well before the window fills; a 1M-token window does not reliably reason over 1M tokens. Accuracy is highest when the key info sits near the START of the sequence.
+
+## "NoLiMa: Long-Context Evaluation Beyond Literal Matching" — Modarressi et al., ICML 2025 — https://arxiv.org/abs/2502.05167 — [ID]
+Repo: https://github.com/adobe-research/NoLiMa . Removes literal lexical overlap, so it measures latent-association retrieval (closest analog to a judge matching a rubric to a semantically-distant answer). Numbers via subagent WebFetch.
+
+> The effective length is defined as the longest context where a model maintains at least 85% of its base score.
+
+> At 32K, for instance, 10 models drop below 50% of their strong short-length baselines.
+
+> GPT-4o: ... a reduction from an almost-perfect baseline of 99.3% to 69.7% [at 32K]
+
+Rule of thumb: once literal cues are gone, even top models fall below their 85%-effective-length by ~8-16K tokens; by 32K most are below half their short-context score.
+
+## "Lost in the Middle" — Liu et al., TACL 2024 — https://arxiv.org/abs/2307.03172 — [ID]
+Origin of the U-shaped/middle-penalty result, replicated across 6 model families.
+
+> performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.
+
+Rule of thumb for judge prompts: put the rubric and the answer-under-test at the START or END of the prompt, never buried mid-way through a long reference block. (Exact "%drop when moved to middle" varies by model, ~15-25 pts in secondary summaries; direction robust, magnitude approximate.)
+
+## Machine-accessible judge benchmark index (URLs resolve; pull data programmatically)
+
+| name | measures | data URL | machine-accessible |
+|---|---|---|---|
+| JudgeBench (2410.12784) | objective-correctness judge accuracy (near-random for many strong judges) | https://huggingface.co/datasets/ScalerLab/JudgeBench | yes (Parquet) |
+| RewardBench (2403.13787) | RM accuracy chat/safety/reasoning | https://huggingface.co/datasets/allenai/reward-bench | yes (+ results dataset) |
+| RewardBench 2 (2506.01937) | RM accuracy, harder unseen prompts (~20pt harder) | https://huggingface.co/datasets/allenai/reward-bench-2 | yes (Parquet) |
+| RM-Bench (2410.16184) | RM subtlety + style-bias robustness (SOTA ~46.6% under style bias) | https://github.com/THU-KEG/RM-Bench | yes (JSON/HF) |
+| PPE (2410.14872) | RM/judge vs real post-RLHF human prefs | https://github.com/lmarena/PPE | yes (HF+JSON) |
+| LLMBar (2310.07641) | adversarial instruction-following judge | https://github.com/princeton-nlp/LLMBar | yes (JSON) |
+| CALM / Justice-or-Prejudice (2410.02736) | 12 cognitive-bias categories, robustness+consistency rate | https://github.com/Y0oMu/LLM-Judge-Bias-Dataset | yes (JSON; mirror repo) |
+| MT-Bench (2306.05685) | judge-human agreement, chat | https://huggingface.co/datasets/lmsys/mt_bench_human_judgments | yes (Parquet) |
+| Arena-Hard-Auto (2406.11939) | pairwise win-rate vs baseline | https://github.com/lmarena/arena-hard-auto | yes (JSON; viewer glitchy) |
+| JudgeLM (2310.17631) | fine-tuned 7-33B judge vs GPT-4 | https://huggingface.co/datasets/BAAI/JudgeLM-100K | yes (JSON) |
+| PandaLM (2306.05087) | small fine-tuned judge vs GPT-3.5/4 | https://github.com/WeOpenML/PandaLM | yes (JSON in-repo) |
+| Judgemark v4 | judge score-separability, creative writing | https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js | yes but nonstandard (JS object) |
+| JETTS (2504.15253) | judge usefulness for test-time scaling (rerank/beam/critique) | https://github.com/SalesforceAIResearch/jetts-benchmark | yes (JSONL) |
+| RewardMATH (2410.01729) | RM math robustness (1 correct vs 9 wrong) | https://huggingface.co/datasets/RewardMATH/RewardMATH | yes (Parquet; code anon) |
+
+URL provenance: subagent reported all resolve via WebFetch; I have NOT independently curled every dataset. JudgeBench near-random headline and RM-Bench 46.6% are subagent WebFetch quotes. Lower-provenance: CALM data is a mirror repo (Y0oMu, not the main org); RewardMATH code lives on anonymous.4open.science; Judgemark has no arXiv paper (independent practitioner benchmark by Sam Paech).
+
+## Honesty flags for this batch
+- [FT] entries (Lech Mazur 43%, self-preference DBG 41.7/2.1, RLAIF position-bias direction + averaging mitigation, CAIS/litellm budgets) are copied from raw source I fetched this turn.
+- [ID] entries: arXiv id + title confirmed real and on-topic this turn, but the specific in-body number was pulled by a research subagent through WebFetch's summarizer, not from raw PDF. Re-pull before quoting a figure as exact.
+- Discarded as hallucinated by subagents: arXiv IDs with impossible month codes (e.g. 2602.08028, 2606.13603 from search autocomplete); not included.
+- Contested / do-not-state-as-settled: whether reasoning fixes self-preference (2504.03846 says CoT halves HSPP; 2506.02592 says LRMs "not necessarily less" on DBG). Different metrics.

refs/llm_judge_litreview.md

@@ -0,0 +1,122 @@
+# LLM-as-a-judge: 2026 literature review (varglite)
+
+Quote-anchored evidence for the operational rules of thumb in [llm_judges.md](llm_judges.md). Every `>` block is copy-pasteable from the cited source (ctrl-F-able); each was fetched from the raw HTML/README this turn unless flagged otherwise. Assembled 2026-07-23 (CLAUDE agent). Bare-quote cache with more sources (incl. summarizer-extracted numbers not safe to quote verbatim) lives in [../docs/evidence/llm_judge_biases.md](../docs/evidence/llm_judge_biases.md).
+
+Verify: **current LLM judges carry large, size-dependent biases (order, self-preference) and degrade on long inputs and over-long reasoning, so read outputs, swap order, cap reasoning to the task, and keep N small.**
+
+## Position and order bias
+
+## Lech Mazur, position_bias benchmark — [github README](https://github.com/lechmazur/position_bias)
+- last updated: not stated on the raw README; result set covers 2026-era models (GPT-5.4, Claude Opus 4.8, Kimi K2.5)
+
+> Across the 36-model result set, the model-average first-shown pick rate is 64.3%, with a median of 65.4%. **The model-average absolute first-position lift is 15.7 percentage points.** So the aggregate pattern is not a subtle tie-breaker: the displayed order materially changes many judgments.
+
+epistemic context: outsider-run public benchmark with a reproducible swapped-order harness (193 pairs, 36 models); no arXiv paper, the numbers are the raw output of the author's own runs. The headline order-flip figure elsewhere in the same README is "the model-average order-flip rate is 43.0%".
+
+## "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" — Shi et al. (Dartmouth), IJCNLP-AACL 2025 — [arXiv:2406.07791](https://arxiv.org/abs/2406.07791)
+- page date: arXiv June 2024; IJCNLP-AACL 2025
+
+> Our findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. **While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions.** Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.
+
+epistemic context: peer-reviewed; largest-scale dedicated position-bias study (over 150,000 evaluation instances, 15 judges, 22 tasks); "quality gap" here means the closer the two answers in quality, the more the judge flips on order.
+
+## "RLAIF vs. RLHF" — Lee et al. (Google), ICML 2024 — [arXiv:2309.00267](https://arxiv.org/abs/2309.00267)
+- page date: arXiv Sept 2023; ICML 2024
+
+> We find evidence of position bias, which is especially prevalent in smaller LLM labelers (see Appendix B). **To mitigate the effect of position bias, two inferences are made for every pair of candidates, where the order in which candidates are presented to the LLM is reversed for the second inference.** The results from both inferences are then averaged to obtain the final preference distribution.
+
+epistemic context: peer-reviewed; the standard citation for both the "smaller = more position-biased" observation and the swap-and-average fix; per-size figures (PaLM-2 L/S/XS keep position 18/21/56% of the time) are in its Appendix B, not the quoted main text.
+
+## Self-preference scales inversely with judge size
+
+## "Beyond the Surface: Measuring Self-Preference in LLM Judgments" — Chen et al., EMNLP 2025 main — [arXiv:2506.02592](https://arxiv.org/abs/2506.02592)
+- page date: arXiv June 2025; EMNLP 2025 main conference. Data + code: [github.com/zhiyuanc2001/self-preference](https://github.com/zhiyuanc2001/self-preference)
+
+> As observed in the figure, models larger than 7B exhibit significantly less self-preference bias compared to those of 7B or smaller. **For example, the DBG score of Qwen2.5-0.5B-Instruct is 41.7%. In contrast, the DBG score of Qwen2.5-14B-Instruct is only 2.1%.** This suggests that LLM judging tasks should utilize larger models to obtain more accurate and unbiased judgment results.
+
+epistemic context: peer-reviewed; the DBG (Difference-based Bias Gauge) score nets out genuine quality using gold judgments, so the residual is bias not skill. The same paper reports reasoning models still self-prefer ("not necessarily less" than non-reasoning), so reasoning is not a fix.
+
+## Reasoning judges: accuracy up, superficial bias not fixed
+
+## "JudgeLRM: Large Reasoning Models as a Judge" — Chen et al., 2025 — [arXiv:2504.00050](https://arxiv.org/abs/2504.00050)
+- page date: arXiv April 2025
+
+> JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. **JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models:** notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B outperforms DeepSeek-R1.
+
+epistemic context: single-group result, not independently replicated; the abstract's headline "+8.14% F1 over same-size SFT" figure is in the body (not re-verified verbatim here). Complementary finding from Huang et al. (arXiv:2601.03630): reasoning judges win on accuracy "particularly on reasoning-intensive tasks" but "still exhibit strong evaluation biases".
+
+## Overthinking: the reasoning-token budget is non-monotonic
+
+## "Does Thinking More always Help? ... Mirage of Test-Time Scaling in Reasoning Models" — Ghosal et al., 2025 — [arXiv:2506.04210](https://arxiv.org/abs/2506.04210)
+- page date: arXiv June 2025
+
+> We observe an initial increase (similar to (Muennighoff et al., 2025; Aggarwal & Welleck, 2025)) in accuracy as the average thinking budget increases. **For example, in Figure 2(a), accuracy increases from 82.2% to 87.3% as the average number of thinking tokens increases from 385 to 1100.** However, this trend does not continue indefinitely.
+
+epistemic context: peer-review status unclear (preprint); the paper attributes the post-peak decline to output variance, not worse reasoning; the subagent-reported downstream figure (accuracy falls 87.3% -> 70.3% as tokens rise 1100 -> 15980) is in the body and not re-quoted verbatim here.
+
+## Reasoning-effort token budgets are a config choice, not a constant
+
+## CAIS `simple-evals` and litellm defaults — raw source, fetched 2026-07-23
+- [simple-evals/.env.example](https://github.com/centerforaisafety/simple-evals/blob/main/.env.example) and [litellm/constants.py](https://github.com/BerriAI/litellm/blob/main/litellm/constants.py)
+
+> # Some env for reasoning effort if you using litellm https://github.com/BerriAI/litellm/blob/main/litellm/constants.py#L81
+> DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET=24576
+> DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET=8192
+> DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET=1024
+
+epistemic context: config lines quoted verbatim (not prose, so no surrounding sentences); CAIS's eval harness deliberately overrides litellm's stock defaults, whose own constants.py sets HIGH=4096, MEDIUM=2048, LOW=1024. So "effort=high" can mean 4096 or 24576 tokens depending on which mapping is live; setting effort on a judge without checking this can truncate its reasoning ~6x below what a serious harness allots.
+
+## Self-consistency: how many samples N
+
+## "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" — Loo, 2025 — [arXiv:2511.00751](https://arxiv.org/abs/2511.00751)
+- page date: arXiv Oct 2025 (v2 May 2026)
+
+> Self-consistency was designed for an era when base models frequently made reasoning errors; this technique has become an expensive habit mismatched to current model capabilities. **Results confirm that accuracy gains plateau early and, in some configurations, decline at high sample counts** — a pattern inconsistent with diminishing returns alone and more consistent with noise introduction on problems that were already solved. This suggests self-consistency should be reserved for genuinely difficult problems rather than applied as a default scaling strategy.
+
+epistemic context: single-author preprint (low citation signal, flagged); its reported plateau is N~10-15 on strong 2026 models (Gemini 2.5), down from the ~40 of the original PaLM-540B-era self-consistency paper (Wang et al., arXiv:2203.11171). Sets a sane ceiling for a repeat-variance check: 4-10 passes is plenty, past ~15 buys nothing.
+
+## Context rot: long inputs and rubrics degrade judging
+
+## "NoLiMa: Long-Context Evaluation Beyond Literal Matching" — Modarressi et al., ICML 2025 — [arXiv:2502.05167](https://arxiv.org/abs/2502.05167)
+- page date: arXiv Feb 2025; ICML 2025. Repo: [github.com/adobe-research/NoLiMa](https://github.com/adobe-research/NoLiMa)
+
+> While they perform well in short contexts (<1K), performance degrades significantly as context length increases. **At 32K, for instance, 11 models drop below 50% of their strong short-length baselines.** Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.
+
+epistemic context: peer-reviewed; removes literal lexical overlap so the test measures latent-association retrieval, the closest analog to a judge matching a rubric to a semantically-distant answer. The paper defines "effective length as the maximum length at which the score remains above a threshold, set at 85% of the model's base score" -- most models fall below it by 8-16K tokens.
+
+## "Lost in the Middle: How Language Models Use Long Contexts" — Liu et al., TACL 2024 — [arXiv:2307.03172](https://arxiv.org/abs/2307.03172)
+- page date: arXiv July 2023; TACL 2024
+
+> We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. **In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.** Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
+
+epistemic context: peer-reviewed; the origin of the U-shaped/middle-penalty result, replicated across 6 model families. Operational read for judging: put the rubric and the answer-under-test at the start or end of the prompt, never buried mid-way through a long reference block.
+
+## Machine-accessible judge benchmarks
+
+URLs resolve (checked by subagent via WebFetch); I have not curled every dataset. Pull data programmatically from these.
+
+| name | measures | data URL | notes |
+|---|---|---|---|
+| JudgeBench (2410.12784) | objective-correctness judge accuracy | [HF ScalerLab/JudgeBench](https://huggingface.co/datasets/ScalerLab/JudgeBench) | many strong judges near-random (~50%) |
+| RewardBench (2403.13787) | RM accuracy chat/safety/reasoning | [HF allenai/reward-bench](https://huggingface.co/datasets/allenai/reward-bench) | dedicated results dataset |
+| RewardBench 2 (2506.01937) | RM accuracy, harder unseen prompts | [HF allenai/reward-bench-2](https://huggingface.co/datasets/allenai/reward-bench-2) | ~20pt harder than v1 |
+| RM-Bench (2410.16184) | RM subtlety + style-bias robustness | [THU-KEG/RM-Bench](https://github.com/THU-KEG/RM-Bench) | SOTA ~46.6% under style bias |
+| PPE (2410.14872) | RM/judge vs real post-RLHF human prefs | [lmarena/PPE](https://github.com/lmarena/PPE) | 16k Arena pairs |
+| LLMBar (2310.07641) | adversarial instruction-following judge | [princeton-nlp/LLMBar](https://github.com/princeton-nlp/LLMBar) | 419 expert-agreed pairs |
+| CALM / Justice-or-Prejudice (2410.02736) | 12 cognitive-bias categories | [Y0oMu/LLM-Judge-Bias-Dataset](https://github.com/Y0oMu/LLM-Judge-Bias-Dataset) | mirror repo, lower provenance |
+| MT-Bench (2306.05685) | judge-human agreement, chat | [HF lmsys/mt_bench_human_judgments](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments) | 3,755 human judgments |
+| Arena-Hard-Auto (2406.11939) | pairwise win-rate vs baseline | [lmarena/arena-hard-auto](https://github.com/lmarena/arena-hard-auto) | viewer glitchy, raw files OK |
+| JudgeLM (2310.17631) | fine-tuned judge vs GPT-4 | [HF BAAI/JudgeLM-100K](https://huggingface.co/datasets/BAAI/JudgeLM-100K) | 100k pairs |
+| PandaLM (2306.05087) | small judge vs GPT-3.5/4 | [WeOpenML/PandaLM](https://github.com/WeOpenML/PandaLM) | 7B recovers ~88-94% of frontier |
+| Judgemark v4 | judge score-separability, writing | [judgemark-v4.js](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js) | JS object, no arXiv paper |
+| JETTS (2504.15253) | judge for test-time scaling | [SalesforceAIResearch/jetts-benchmark](https://github.com/SalesforceAIResearch/jetts-benchmark) | rerank/beam/critique |
+| RewardMATH (2410.01729) | RM math robustness | [HF RewardMATH/RewardMATH](https://huggingface.co/datasets/RewardMATH/RewardMATH) | code repo anonymized |
+
+## Epistemic summary
+
+- **Who says X**: the "large, size-dependent bias" claim rests on three independent chains: an outsider benchmark measuring order-flip on 2026 models (Lech Mazur), an EMNLP paper measuring self-preference vs size with a quality-netted metric (2506.02592), and a Google paper reporting position bias rising as labeler size falls (2309.00267). The "long context / long reasoning both hurt" claim rests on NoLiMa + Lost-in-the-Middle (context) and Ghosal + Loo (reasoning tokens / samples).
+- **How they could know**: all direct measurement (repeated inference under swapped order, matched own-vs-other pairs, needle-retrieval at varied length, accuracy-vs-token-budget sweeps), not self-report.
+- **Entanglement check**: the bias sources are independent (different teams, years 2023-2026, metrics). The context-rot sources partly share lineage (NoLiMa explicitly builds on the Lost-in-the-Middle framing), so they stack less than they appear to; treat them as ~1.5 independent observations, not 2.
+- **Hard-to-vary check**: "bias is large" is hard to vary (a 43% flip rate is not reframable as noise). "Shrinks monotonically with size" is softer: 2506.02592 itself attributes the trend to capability, and a frontier reasoning model (GPT-5.4) still flips ~66% in Lech Mazur, so size alone does not guarantee low bias.
+- **What would change my mind (not-claim)**: under the null I would expect near-zero order-flip after swapping, flat DBG across 0.5B->72B, no benefit from swap-and-average, and flat accuracy across context length and thinking-token budget. None of these hold. The one genuine gap: no clean same-model with/without-retrieval judge ablation exists, so RAG-as-mitigation is untested, not refuted.
+- **Calibrated take**: qualitative claim (large order + self bias, mitigable by swap-and-average; long context and over-long reasoning both degrade judging) `p ≈ 0.90-0.97`. Specific "monotonically shrinks with size" `p ≈ 0.70-0.85` (capability confound). Cheapest way to be wrong: quote the 41.7%->2.1% size curve as if size is the lever when it may be capability, and assume a big judge is order-invariant. Safe rule: swap-and-average every judge regardless of size; treat "bigger/smarter judge" as a weak prior, not a fix.

refs/llm_judges.md

@@ -1,6 +1,24 @@
 # LLM-as-a-judge: known biases and mitigations
 
-Appendix to the [ML Debugging skill](../SKILL.md). When an LLM-judged eval looks surprisingly good, or a ranking flips between runs, suspect the judge before the model. Each bias below has been measured; verbatim sources in [docs/evidence/llm_judge_biases.md](../docs/evidence/llm_judge_biases.md).
+Appendix to the [ML Debugging skill](../SKILL.md). When an LLM-judged eval looks surprisingly good, or a ranking flips between runs, suspect the judge before the model. Each bias below has been measured; verbatim sources in [docs/evidence/llm_judge_biases.md](../docs/evidence/llm_judge_biases.md), with quote-anchored 2026 numbers and their epistemic status in [llm_judge_litreview.md](llm_judge_litreview.md). For the wider literature, two surveys collect it: Eugene Yan's practitioner review[^yan] and Gu et al., "A Survey on LLM-as-a-Judge"[^survey].
+
+## Numbers worth knowing (2026)
+
+Operational rules of thumb, anchored in verbatim quotes; full passages, sources, and calibration in [llm_judge_litreview.md](llm_judge_litreview.md). Source type is stated so you can weight it: independent benchmarks and peer-reviewed studies carry more than single-group preprints.
+
+- Swap the order and average, on every judge. Position bias is still large in 2026: an independently-run public benchmark of 36 models reports "the model-average order-flip rate is 43.0%, and the median model flips in 41.3% of decisive two-view cases",[^lechmazur] and a peer-reviewed study of 150k+ judgements finds it concentrates on the hard cases, since position bias "is strongly affected by the quality gap between solutions".[^shi] The two agree from independent methods, so it is very probable this holds for your judge; even a frontier model flips ~66%, making a bigger judge a weak prior at best, not a fix. Running both orders and averaging is the fix, and a high flip rate means the test is broken, not decided.
+
+- Don't let a small model grade its own family. A peer-reviewed EMNLP study, on a metric that nets out genuine quality, reports "the DBG score of Qwen2.5-0.5B-Instruct is 41.7%. In contrast, the DBG score of Qwen2.5-14B-Instruct is only 2.1%",[^selfpref] and finds reasoning does not rescue it, "the self-preference bias in reasoning models is not necessarily less significant than the bias found in language models".[^selfpref] The shrink-with-size direction is probable but partly confounded with capability (the authors credit better instruction-following), so treat "use a larger, different-family judge than the model under test" as the safe rule rather than size being the true lever.
+
+- A reasoning judge grades better, but probably not less biased. One group's RL-trained judge reports that judging is "inherently reasoning-intensive ... it requires verifying evidence, identifying errors, and justifying decisions", and that such judges "consistently outperform SFT-tuned baselines in the same size ... and even surpass state-of-the-art reasoning models".[^judgelrm] This is a single-group result, not yet independently replicated, so weight it as suggestive: reach for a reasoning judge when the grading itself needs that work, but expect it to keep the length, position, and style biases above.
+
+- Match the thinking budget to task difficulty. One controlled study finds more reasoning helps only up to a point: "accuracy increases from 82.2% to 87.3% as the average number of thinking tokens increases from 385 to 1100. However, this trend does not continue indefinitely",[^overthink] and it declines as tokens grow further, because extra thinking adds variance, not insight. The exact peak is setup-specific, but the non-monotonic shape is likely general, so on easy items cap thinking low and spend the saved budget on repeat passes instead.
+
+- Check what "high effort" actually buys before trusting it. This one is certain, it is just what the harnesses ship: litellm's stock default caps high reasoning at 4096 tokens, while CAIS's simple-evals overrides it to "DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET=24576" (~6x more).[^cais] Set it explicitly, or you may silently truncate the judge and score a cut-off verdict as a real one.
+
+- A few repeats measure stability; many just cost tokens. A single-author preprint (treat as suggestive) reports self-consistency "gains plateau early and, in some configurations, decline at high sample counts",[^loo] with the plateau now around N=10-15 on strong 2026 models, down from ~40 in the widely-cited 2022 work. N=4-10 repeats is very probably enough for the repeat-variance check below; going higher mostly buys noise.
+
+- Keep the judge's inputs short and edge-loaded. A peer-reviewed long-context test that strips literal keyword cues reports "At 32K, for instance, 11 models drop below 50% of their strong short-length baselines",[^nolima] and the middle-of-context penalty is well replicated across model families. So it is probable your judge degrades on long inputs well before the window fills; put the rubric and answer-under-test at the start or end of the prompt, never buried in the middle where models attend least.
 
 ## The measured biases
 
@@ -14,6 +32,8 @@ Self-preference tracks self-recognition. Panickssery et al. fine-tuned models to
 
 There are also output-distribution quirks. From Haize Labs' verdict docs (practitioner notes): the gpt-4o family skews numerical scores upward and mode-collapses even with logprobs; llama-family judges give higher-entropy, more discriminative score distributions; JSON-mode constrained decoding imposes its own inductive bias on scores.[^verdict]
 
+And the judge misses more than you'd think. Doddapaneni et al. probed evaluator LLMs with deliberately degraded answers and found they "failed to identify quality drops in over 50% of cases on average"[^doddapaneni]. A judge that silently passes half the injected regressions is not a safety net.
+
 ## Mitigation checklist
 
 From Wang's calibration framework and verdict's best-practices page:
@@ -25,7 +45,71 @@ From Wang's calibration framework and verdict's best-practices page:
 - Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../SKILL.md#inspect-the-data-first), applied to the judge).
 - Judge quality is benchmarkable: [JudgeBench](https://huggingface.co/spaces/ScalerLab/JudgeBench) ranks judges on objective-correctness pairs.
 
+## Choosing the judge model
+
+Pick from the cost-vs-score Pareto frontier of a judging leaderboard, and prefer a well-known model so your setup is reproducible. [Judgemark v4](https://eqbench.com/judgemark-v4.html) is "a meta-evaluation of LLM judging ability. The model being tested is the judge, not the writer",[^judgemark] scoring how well a judge's ratings separate stronger from weaker writing, and it lists a cost per model. Its lesson (wassname's read): the smartest models are the best judges, so the value frontier is the capable-but-cheap models, not the single top scorer. Caveat: Judgemark scores creative-writing discrimination, so a judge that tops it may not transfer to code- or fact-correctness judging.
+
+From the checked-in v4 scores (snapshot 2026-07, [source](https://github.com/EQ-bench/EQ-bench-site/blob/main/judgemark-v4.js)) the cost-vs-score frontier runs from the top absolute scorers, claude-opus-4-6 (0.91, ~$39) and gpt-5.5 (0.88, ~$30), down through claude-sonnet-4-6 and gemini-3.1-pro (~0.8, ~$23), grok-4.5 (0.77, $17) and GLM-5.2 (0.73, $8), to the cheap knee google/gemma-4-31b (0.72 at $0.82), which nearly matches models 20-40x its price.
+
+Budget in tokens per task, not just dollars, and set it per model. Reasoning models vary roughly 6x in tokens spent per task, and it scales with task difficulty: wassname's read of the [Artificial Analysis token-use tab](https://artificialanalysis.ai/models/qwen3-6-27b#intelligence-index-token-use-tabs) is ~5k for Gemma-4-31b (little reasoning) up to ~30k for Qwen3.6-35B-A3B (roughly half reasoning, half answer), with Qwen3.7-27B among the highest, and small models often reasoning a lot to compensate for capacity (exact per-model splits not verified here, the dashboard is JS-rendered). It's a moving, task-dependent target: Epoch AI finds reasoning models emit "around 8x more tokens on average, compared to non-reasoning models", and raising OpenAI reasoning effort from medium to high gave "a 1.6x increase in output tokens"[^epoch]. The length scales with difficulty because RL-trained reasoners learn to spend more test-time compute, longer chains on harder problems[^r1]. The budget buys either depth or breadth: on an easy task, capping reasoning low (~2k) and spending the savings on N passes is usually the better trade, the repeats give you the repeat-variance check (below) and a majority vote for the same cost. But on a task near or beyond the model's capability, cutting reasoning just truncates the work and you score a cut-off verdict as a real one. So set the cap from the model's actual appetite on your hardest cases, and count truncations.
+
+But a frontier score isn't sufficient: refusals wreck ambiguous or red-teaming evals, and refusal is topic-conditional. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] Its per-lab Free Speech Index (0-100, higher = answers more; snapshot 2026-07-21) puts Mistral (88.9), xAI/Grok (85.8), and Google (81.1) most permissive, and among US majors Anthropic (53.8) and OpenAI (48.0) most restrictive; Chinese labs sit mid-to-high on this cross-topic aggregate (Zhipu/GLM 71.0, DeepSeek 59.0, Alibaba/Qwen 45.5) yet refuse specifically on Chinese-political topics. Two traps: the index is a lab average, so a single safety-tuned model (Gemma, per wassname) can refuse far more than its lab's number; and it aggregates topics, so it won't catch a refusal cluster on *your* eval's subject. Check refusal on your actual subject matter, and re-read the live leaderboards rather than trusting these names, they date fast.
+
+## wassname's judge-validity checklist
+
+Practical rules from wassname for before you trust any LLM-judged number. A failed check is evidence about the *test*, not the model, so revise or reject the scenario before drawing a behavioral conclusion.
+
+Earn the rubric's ink:
+
+- Does each rubric line ever flip a verdict? Cut criteria that never change the score. Rubric quality is the main lever: a judge lacking domain knowledge will "overestimate the effectiveness by a significant margin", and adding brief domain notes raised human-alignment from ~72-79% to 93-96%.[^gradingnotes]
+- Expect criteria drift: you can't fully write the rubric before seeing outputs. Shankar et al. name it, "users need criteria to grade outputs, but grading outputs helps users define criteria"[^shankar], and warn that "LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation."[^shankar] Draft the rubric, grade a sample by hand, revise, repeat.
+
+Read a whole trace, not the aggregate:
+
+- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../SKILL.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
+- Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
+- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../SKILL.md#inspect-the-data-first) applied to the judge.
+- Chase confusion: if the judge hedges, asks for missing context, or self-contradicts, that is a harness fault, not a result ([investigate confusion](../SKILL.md#pursue-anomalies-investigate-confusion)).
+
+Check the score distribution:
+
+- Not saturated: reject scenarios where every arm passes or every arm fails (too easy or too hard leaves nothing to discriminate).
+- Not clustered: plot the raw histogram. Mode collapse or skew means the scale isn't being used.[^verdict]
+- Not anchored: don't put an example score in the prompt. A few-shot "+2" pulls a weak judge toward +2, and Eugene Yan's survey notes few-shot judges are "unstable when changing the label, example order, and number of examples".[^yan] Ask for a bare integer or label, and prefer a coarse scale: Databricks recommend a low-precision range (0-3 or 1-5) because "Scales like 0-10 are difficult to come up with distinguishing criteria between all scores".[^databricks] Hamel is blunter, preferring binary: "If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong."[^hamel]
+
+Check stability across order and repeats:
+
+- Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest.
+- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same canary as [seed variance](../SKILL.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
+
+Give the judge a voice, and save everything:
+
+- Add a free-text field for the judge to flag a broken, missing, or ambiguous rubric or context. Read it, but keep it out of the score. Do the same for the evaluated agents: an unscored exit interview about ambiguity, missing context, broken tools, and unnatural constraints, kept separate from the task score.
+- Save full append-only traces in JSONL or Inspect `.eval`, including prompts, responses, provider-exposed reasoning and tool events, artifacts, machine checks, both judgment orders, usage, costs, and errors. Use [Inspect Scout](https://meridianlabs-ai.github.io/inspect_scout/) or an equivalent transcript audit when practical.
+- Before reporting a winner, make a judgeable per-scenario audit that links the A output, B output, machine result, forward and reversed judge rationales, saturation status, and the human validity decision.
+
+For a worked example, wassname has a ~300-line async OpenRouter judge (WIP) that implements many of these: bounded thinking, pinned quantisation, a versioned eval, JSON-schema output, JSONL of everything, OpenRouter error handling, and position-bias swapping: [gist](https://gist.github.com/wassname/b7f76e42de131887c02d9e9835be80ef).
+
 [^zheng]: Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023) — https://arxiv.org/abs/2306.05685
 [^wang]: Wang et al., "Large Language Models are not Fair Evaluators" (ACL 2024) — https://arxiv.org/abs/2305.17926
 [^panickssery]: Panickssery, Bowman, Feng, "LLM Evaluators Recognize and Favor Their Own Generations" (2024) — https://arxiv.org/abs/2404.13076
 [^verdict]: Haize Labs, verdict docs: [best practices](https://verdict.haizelabs.com/docs/best-practices/), [distributional bias cookbook](https://verdict.haizelabs.com/docs/cookbook/distributional-bias/)
+[^hamel]: Hamel Husain, "Creating a LLM-as-a-Judge That Drives Business Results" (2024) — https://hamel.dev/blog/posts/llm-judge/ (critique-shadowing workflow: look at the data first, iterate the prompt with a domain expert, prefer binary pass/fail) ([cache](../docs/evidence/llm_judge_biases.md))
+[^databricks]: Databricks, "Best Practices for LLM Evaluation of RAG Applications" (2023) — https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG (use a low-precision 0-3 / 1-5 scale; few-shot examples help weak judges but shift the score distribution) ([cache](../docs/evidence/llm_judge_biases.md))
+[^gradingnotes]: Databricks, "Enhancing LLM-as-a-Judge with Grading Notes" (2024) — https://www.databricks.com/blog/enhancing-llm-as-a-judge-with-grading-notes (per-question domain rubrics lifted human-alignment to 93-96%) ([cache](../docs/evidence/llm_judge_biases.md))
+[^yan]: Eugene Yan, "Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)" — https://eugeneyan.com/writing/llm-evaluators/ (survey of position, verbosity, and few-shot-instability biases; argues for binary over Likert; collects G-Eval, Doddapaneni blind-spots, Shankar "Who Validates the Validators?") ([cache](../docs/evidence/llm_judge_biases.md))
+[^judgemark]: EQ-Bench, "Judgemark v4" — https://eqbench.com/judgemark-v4.html (meta-eval of a model's judging ability, scored by how well its ratings separate stronger from weaker writing; leaderboard shows cost per model)
+[^speechmap]: SpeechMap.ai — https://speechmap.ai/ (refusal / completion rates across providers on contentious prompts; useful for spotting a judge that will refuse ambiguous or red-teaming scenarios)
+[^epoch]: Epoch AI, "Output length" data insight — https://epoch.ai/data-insights/output-length (reasoning models emit ~8x more tokens than non-reasoning; medium->high effort = 1.6x; reasoning-model response length growing ~5x/year)
+[^r1]: DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025) — https://arxiv.org/abs/2501.12948 (pure-RL reasoning; response length / test-time compute grows over training and with problem difficulty)
+[^survey]: Gu et al., "A Survey on LLM-as-a-Judge" (2024) — https://arxiv.org/abs/2411.15594 (broad survey of methods, biases, and reliability; complements Yan's practitioner review)
+[^doddapaneni]: Doddapaneni, Khan, Verma, Khapra, "Finding Blind Spots in Evaluator LLMs with Interpretable Checklists" (2024) — https://arxiv.org/abs/2406.13439 (evaluator LLMs missed injected quality drops in >50% of cases) ([cache](../docs/evidence/llm_judge_biases.md))
+[^shankar]: Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences" (2024) — https://arxiv.org/abs/2404.12272 (criteria drift; LLM evaluators need human validation) ([cache](../docs/evidence/llm_judge_biases.md))
+[^lechmazur]: Lech Mazur, position_bias benchmark — https://github.com/lechmazur/position_bias (independent, outsider-run swapped-order harness; 193 pairs, 36 models, 2026-era; strong trust signal, but a solo-run leaderboard not a paper) ([litreview](llm_judge_litreview.md))
+[^shi]: Shi et al., "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" (IJCNLP-AACL 2025) — https://arxiv.org/abs/2406.07791 (peer-reviewed; 150k+ instances, 15 judges; bias worsens as the answer quality gap shrinks) ([litreview](llm_judge_litreview.md))
+[^selfpref]: Chen et al., "Beyond the Surface: Measuring Self-Preference in LLM Judgments" (EMNLP 2025 main) — https://arxiv.org/abs/2506.02592 (peer-reviewed; DBG nets out quality; larger judges less self-biased, though authors credit capability; reasoning models still biased) ([litreview](llm_judge_litreview.md))
+[^judgelrm]: Chen et al., "JudgeLRM: Large Reasoning Models as a Judge" (2025) — https://arxiv.org/abs/2504.00050 (single-group preprint, not independently replicated; RL-trained reasoning judges beat same-size SFT, ~+8 F1 headline in body) ([litreview](llm_judge_litreview.md))
+[^overthink]: Ghosal et al., "Does Thinking More always Help? ... Mirage of Test-Time Scaling in Reasoning Models" (2025) — https://arxiv.org/abs/2506.04210 (preprint; one controlled study, accuracy-vs-thinking-token curve is non-monotonic, peak setup-specific) ([litreview](llm_judge_litreview.md))
+[^cais]: CAIS simple-evals .env.example vs litellm constants.py (fetched 2026-07, directly verifiable config) — https://github.com/centerforaisafety/simple-evals/blob/main/.env.example (effort high=24576/med=8192/low=1024, overriding litellm stock 4096/2048/1024)
+[^loo]: Loo, "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" (2025) — https://arxiv.org/abs/2511.00751 (single-author preprint, low citation signal; plateau ~N=10-15 on modern models, can decline past it) ([litreview](llm_judge_litreview.md))
+[^nolima]: Modarressi et al., "NoLiMa: Long-Context Evaluation Beyond Literal Matching" (ICML 2025) — https://arxiv.org/abs/2502.05167 (peer-reviewed; effective length = length holding 85% of base score; most models below half by 32K once literal cues removed) ([litreview](llm_judge_litreview.md))