llm_judges: verify the five abstract-quoted papers against raw PDF

NoLiMa effective length was wrong in both files: most models fall below the
85% threshold at 1-4K tokens, not 8-16K (only GPT-4o 8K, GPT-4.1 16K). NoLiMa
32K count was 11 models, not 10. The Loo quote's first sentence was stitched
from two places and is dropped. JudgeLRM's 8.14% is an Introduction number,
not an abstract one, and the abstract quote had lost /14B and its trailing
clause. Shi et al is 'The findings', not 'Our findings'. Body quotes now
replace abstract quotes where the number matters; tags [ID] -> [FT].

docs/evidence/llm_judge_biases.md

@@ -216,7 +216,7 @@ Independent, outsider-run, continuously-updated public harness (strong trust). M
 
 Rule of thumb: even in 2026, judges flip ~43% of decisive verdicts on order swap alone; the worst flip >70%. Direction is not universal (Mistral Large 3 goes second-position). Always judge both orders. (Note: an earlier subagent draft misattributed 27.4% to Claude Opus; the raw table puts 27.4% first-shown pick on Mistral Large 3, so I dropped the per-model attributions except the verified worst-case.)
 
-## "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" — Shi et al. (Dartmouth), IJCNLP-AACL 2025 — https://arxiv.org/abs/2406.07791 — [ID]
+## "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" — Shi et al. (Dartmouth), IJCNLP-AACL 2025 — https://arxiv.org/abs/2406.07791 — [FT]
 Most-cited dedicated position-bias study, largest scale (150k+ instances, 15 judges, 22 tasks).
 
 > Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances.
@@ -262,8 +262,8 @@ Tension to flag: this paper says CoT roughly halves self-preference; 2506.02592
 
 ## Reasoning judges: accuracy up, superficial bias not fixed
 
-## "JudgeLRM: Large Reasoning Models as a Judge" — Chen et al., 2025 — https://arxiv.org/abs/2504.00050 — [ID]
-RL-trained reasoning judge. Title confirmed; F1 numbers via subagent WebFetch.
+## "JudgeLRM: Large Reasoning Models as a Judge" — Chen et al., 2025 — https://arxiv.org/abs/2504.00050 — [FT]
+RL-trained reasoning judge. Full text re-pulled from raw PDF 2026-08-14; F1 numbers confirmed.
 
 > JudgeLRM achieves an average improvement of 8.14% in F1 score [vs same-size SFT judges]
 
@@ -323,8 +323,8 @@ Foundational (several-thousand citations), PaLM-540B era.
 
 Widely-reproduced pattern: gain is monotonic in N with diminishing returns; bulk arrives by N=5-10, saturates ~N=40 for that era's models.
 
-## "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" — Loo, 2025 — https://arxiv.org/abs/2511.00751 — [ID]
-Single-author preprint (low citation signal, flagged), but directly answers "how has N moved." Numbers via subagent WebFetch.
+## "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" — Loo, 2025 — https://arxiv.org/abs/2511.00751 — [FT]
+Single-author preprint (low citation signal, flagged; author states AI tools assisted drafting), but directly answers "how has N moved." Full text re-pulled from raw PDF 2026-08-14.
 
 > [MATH-500, Gemini-2.5-Flash-Lite] accuracy improved through approximately 10 sampled paths before plateauing... declining slightly beyond 15
 
@@ -352,18 +352,18 @@ Industry report (not peer-reviewed), 18-model controlled study. Numbers via suba
 
 Rule of thumb: degradation is continuous and starts well before the window fills; a 1M-token window does not reliably reason over 1M tokens. Accuracy is highest when the key info sits near the START of the sequence.
 
-## "NoLiMa: Long-Context Evaluation Beyond Literal Matching" — Modarressi et al., ICML 2025 — https://arxiv.org/abs/2502.05167 — [ID]
-Repo: https://github.com/adobe-research/NoLiMa . Removes literal lexical overlap, so it measures latent-association retrieval (closest analog to a judge matching a rubric to a semantically-distant answer). Numbers via subagent WebFetch.
+## "NoLiMa: Long-Context Evaluation Beyond Literal Matching" — Modarressi et al., ICML 2025 — https://arxiv.org/abs/2502.05167 — [FT]
+Repo: https://github.com/adobe-research/NoLiMa . Removes literal lexical overlap, so it measures latent-association retrieval (closest analog to a judge matching a rubric to a semantically-distant answer). Full text re-pulled from raw PDF 2026-08-14.
 
 > The effective length is defined as the longest context where a model maintains at least 85% of its base score.
 
-> At 32K, for instance, 10 models drop below 50% of their strong short-length baselines.
+> At 32K, for instance, 11 models drop below 50% of their strong short-length baselines.
 
 > GPT-4o: ... a reduction from an almost-perfect baseline of 99.3% to 69.7% [at 32K]
 
-Rule of thumb: once literal cues are gone, even top models fall below their 85%-effective-length by ~8-16K tokens; by 32K most are below half their short-context score.
+Rule of thumb: once literal cues are gone, most models fall below their 85% effective length by 1-4K tokens (median 2K, Table 3); only GPT-4o reaches 8K and GPT-4.1 16K. By 32K, 11 of 13 are below half their short-context score.
 
-## "Lost in the Middle" — Liu et al., TACL 2024 — https://arxiv.org/abs/2307.03172 — [ID]
+## "Lost in the Middle" — Liu et al., TACL 2024 — https://arxiv.org/abs/2307.03172 — [FT]
 Origin of the U-shaped/middle-penalty result, replicated across 6 model families.
 
 > performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.
@@ -394,5 +394,6 @@ URL provenance: subagent reported all resolve via WebFetch; I have NOT independe
 ## Honesty flags for this batch
 - [FT] entries (Lech Mazur 43%, self-preference DBG 41.7/2.1, RLAIF position-bias direction + averaging mitigation, CAIS/litellm budgets) are copied from raw source I fetched this turn.
 - [ID] entries: arXiv id + title confirmed real and on-topic this turn, but the specific in-body number was pulled by a research subagent through WebFetch's summarizer, not from raw PDF. Re-pull before quoting a figure as exact.
+- 2026-08-14 (CLAUDE agent): the five papers the litreview quotes (2406.07791, 2504.00050, 2511.00751, 2502.05167, 2307.03172) were re-pulled from raw PDF and moved [ID] -> [FT]. Two numbers here were wrong and are now fixed: NoLiMa effective length was "~8-16K", actually 1-4K for most models, and "10 models drop below 50%" at 32K was 11. Everything else in those five entries held up.
 - Discarded as hallucinated by subagents: arXiv IDs with impossible month codes (e.g. 2602.08028, 2606.13603 from search autocomplete); not included.
 - Contested / do-not-state-as-settled: whether reasoning fixes self-preference (2504.03846 says CoT halves HSPP; 2506.02592 says LRMs "not necessarily less" on DBG). Different metrics.

refs/llm_judge_litreview.md

@@ -16,9 +16,9 @@ epistemic context: outsider-run public benchmark with a reproducible swapped-ord
 ## "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" — Shi et al. (Dartmouth), IJCNLP-AACL 2025 — [arXiv:2406.07791](https://arxiv.org/abs/2406.07791)
 - page date: arXiv June 2024; IJCNLP-AACL 2025
 
-> Our findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. **While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions.** Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.
+> The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. **While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions.** Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.
 
-epistemic context: peer-reviewed; largest-scale dedicated position-bias study (over 150,000 evaluation instances, 15 judges, 22 tasks); "quality gap" here means the closer the two answers in quality, the more the judge flips on order.
+epistemic context: peer-reviewed; largest-scale dedicated position-bias study (over 150,000 evaluation instances, 15 judges, 22 tasks); "quality gap" here means the closer the two answers in quality, the more the judge flips on order. The abstract's counts moved across versions (v1-v3: 9 judges / 80,000 instances), and Section 3.1 of the current version still says "more than 100,000", contradicting its own abstract. Best judges in Table 2 reach position consistency 0.82, so ~18% of pairs flip on order even at the top.
 
 ## "RLAIF vs. RLHF" — Lee et al. (Google), ICML 2024 — [arXiv:2309.00267](https://arxiv.org/abs/2309.00267)
 - page date: arXiv Sept 2023; ICML 2024
@@ -41,9 +41,9 @@ epistemic context: peer-reviewed; the DBG (Difference-based Bias Gauge) score ne
 ## "JudgeLRM: Large Reasoning Models as a Judge" — Chen et al., 2025 — [arXiv:2504.00050](https://arxiv.org/abs/2504.00050)
 - page date: arXiv April 2025
 
-> JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. **JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models:** notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B outperforms DeepSeek-R1.
+> Empirical results demonstrate that JudgeLRM not only surpasses proprietary models like GPT-4 and DeepSeek-R1 but also outperforms SFT and RL baselines of comparable sizes, **with an average improvement of 8.14% in F1 score over SFT counterparts.**
 
-epistemic context: single-group result, not independently replicated; the abstract's headline "+8.14% F1 over same-size SFT" figure is in the body (not re-verified verbatim here). Complementary finding from Huang et al. (arXiv:2601.03630): reasoning judges win on accuracy "particularly on reasoning-intensive tasks" but "still exhibit strong evaluation biases".
+epistemic context: quoted from Section 1 (Introduction), v3; the 8.14% figure is not in any abstract version. Single-group result, not independently replicated. Table 3 backs the PandaLM claim: JudgeLRM-3B F1 72.12 vs GPT-4 61.80 on human ground truth, out of distribution. The abstract's own wording is "JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks", and it too has changed across versions (v1/v2 said 2.79%). Complementary finding from Huang et al. (arXiv:2601.03630): reasoning judges win on accuracy "particularly on reasoning-intensive tasks" but "still exhibit strong evaluation biases".
 
 ## Overthinking: the reasoning-token budget is non-monotonic
 
@@ -71,9 +71,11 @@ epistemic context: config lines quoted verbatim (not prose, so no surrounding se
 ## "Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs" — Loo, 2025 — [arXiv:2511.00751](https://arxiv.org/abs/2511.00751)
 - page date: arXiv Oct 2025 (v2 May 2026)
 
-> Self-consistency was designed for an era when base models frequently made reasoning errors; this technique has become an expensive habit mismatched to current model capabilities. **Results confirm that accuracy gains plateau early and, in some configurations, decline at high sample counts** — a pattern inconsistent with diminishing returns alone and more consistent with noise introduction on problems that were already solved. This suggests self-consistency should be reserved for genuinely difficult problems rather than applied as a default scaling strategy.
+> **Results confirm that accuracy gains plateau early and, in some configurations, decline at high sample counts** — a pattern inconsistent with diminishing returns alone and more consistent with noise introduction on problems that were already solved. This suggests self-consistency should be reserved for genuinely difficult problems rather than applied as a default scaling strategy.
 
-epistemic context: single-author preprint (low citation signal, flagged); its reported plateau is N~10-15 on strong 2026 models (Gemini 2.5), down from the ~40 of the original PaLM-540B-era self-consistency paper (Wang et al., arXiv:2203.11171). Sets a sane ceiling for a repeat-variance check: 4-10 passes is plenty, past ~15 buys nothing.
+> On MATH-500, Flash-Lite accuracy improved through approximately 10 sampled paths before plateauing and then declining slightly beyond 15, as shown in Figure 2. **This decline is notable: it suggests that once a model reliably solves most problems, additional samples introduce occasional wrong reasoning paths that the aggregator cannot fully suppress.**
+
+epistemic context: single-author preprint (low citation signal, flagged), and the author states AI tools assisted the drafting. Both quotes are from Section 1 and Section 4 of the raw PDF. The plateau is N~10-15 for Gemini-2.5-Flash-Lite on MATH-500, down from the ~40 of the original PaLM-540B-era self-consistency paper (Wang et al., arXiv:2203.11171); Gemini-2.5-Pro was only run to N=15 and did not decline. Sample sizes are small (Section 6: 250 rows for Flash-Lite). Sets a sane ceiling for a repeat-variance check: 4-10 passes is plenty, past ~15 buys nothing.
 
 ## Context rot: long inputs and rubrics degrade judging
 
@@ -82,14 +84,14 @@ epistemic context: single-author preprint (low citation signal, flagged); its re
 
 > While they perform well in short contexts (<1K), performance degrades significantly as context length increases. **At 32K, for instance, 11 models drop below 50% of their strong short-length baselines.** Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.
 
-epistemic context: peer-reviewed; removes literal lexical overlap so the test measures latent-association retrieval, the closest analog to a judge matching a rubric to a semantically-distant answer. The paper defines "effective length as the maximum length at which the score remains above a threshold, set at 85% of the model's base score" -- most models fall below it by 8-16K tokens.
+epistemic context: peer-reviewed; removes literal lexical overlap so the test measures latent-association retrieval, the closest analog to a judge matching a rubric to a semantically-distant answer. The paper defines "effective length as the maximum length at which the score remains above a threshold, set at 85% of the model's base score" (Table 3). Effective lengths are shorter than they sound: 1-4K tokens for most of the 13 models (median 2K), 8K for GPT-4o, and 16K for GPT-4.1 in the extended Table 10.
 
 ## "Lost in the Middle: How Language Models Use Long Contexts" — Liu et al., TACL 2024 — [arXiv:2307.03172](https://arxiv.org/abs/2307.03172)
 - page date: arXiv July 2023; TACL 2024
 
 > We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. **In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.** Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
 
-epistemic context: peer-reviewed; the origin of the U-shaped/middle-penalty result, replicated across 6 model families. Operational read for judging: put the rubric and the answer-under-test at the start or end of the prompt, never buried mid-way through a long reference block.
+epistemic context: peer-reviewed; the origin of the U-shaped/middle-penalty result, across 6 model families (MPT, LongChat, GPT-3.5, Claude, GPT-4, Llama-2; the last two are appendix-only subsets). Size of the effect, Section 2.3: "GPT-3.5-Turbo's multi-document QA performance can drop by more than 20% -- in the worst case, performance in 20- and 30-document settings is lower than performance without any input documents (i.e., closed-book performance; 56.1%)". Operational read for judging: put the rubric and the answer-under-test at the start or end of the prompt, never buried mid-way through a long reference block.
 
 ## Machine-accessible judge benchmarks