Add judge-validity checklist and model-selection to llm_judges ref

Fold wassname's repeated eval-validity checklist into refs/llm_judges.md:
rubric-earns-ink, read-the-whole-trace, anchoring/scale-precision, repeat
variance, and a judge feedback channel. Add a judge-model-selection section
(Judgemark v4 cost-vs-score frontier, speechmap topic-conditional refusals).
Cache the new sources (Hamel, Databricks x2, Eugene Yan) with verbatim quotes
and onward paper links in docs/evidence.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

docs/evidence/llm_judge_biases.md

@@ -39,3 +39,51 @@ Self-preference is causally linked to self-recognition:
 > Constrained decoding methods for structured outputs (e.g., JSON-mode) impose an inductive bias on the model's output distribution.
 
 Related: JudgeBench leaderboard for judge quality — https://huggingface.co/spaces/ScalerLab/JudgeBench
+
+---
+
+Source: Hamel Husain, Databricks (x2), Eugene Yan blog posts (via WebFetch)
+Title: practitioner rules on scale precision, rubric quality, and reading judge traces
+Fetched-via: WebFetch summarizer model, 2026-07-22; short quotes cross-checked by re-fetching
+Fetch-status: quotes below reproduced twice identically EXCEPT the Databricks scale range, where two fetches disagreed (0-3 or 0-4 vs 0-3 or 1-5) so it is paraphrased, not quoted, in the ref
+
+## Hamel Husain, "Creating a LLM-as-a-Judge That Drives Business Results" — https://hamel.dev/blog/posts/llm-judge/
+
+Critique-shadowing workflow; look at the data before writing the judge, prefer binary:
+
+> You cannot write a good judge prompt until you've seen the data.
+
+> If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong.
+
+Onward links from this post: Shankar et al., "Who Validates the Validators?" (arXiv:2404.12272); Yan, "ALIGN Eval" (https://aligneval.com/); OpenAI Cookbook, "Custom LLM as a Judge to Detect Hallucinations with Braintrust" (https://cookbook.openai.com/examples/custom-llm-as-a-judge); "What We've Learned From A Year of Building with LLMs" (https://applied-llms.org/).
+
+## Databricks, "Best Practices for LLM Evaluation of RAG Applications" — https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG
+
+Use a low-precision integer scale:
+
+> Scales like 0-10 are difficult to come up with distinguishing criteria between all scores.
+
+The recommended range paraphrased (fetches disagreed on 0-4 vs 1-5): a coarse 0-3 or 1-5 Likert scale, with an example for each score in the range. Onward links: LMSYS MT-Bench (arXiv:2306.05685) and its FastChat judge prompts (https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/data/judge_prompts.jsonl).
+
+## Databricks, "Enhancing LLM-as-a-Judge with Grading Notes" — https://www.databricks.com/blog/enhancing-llm-as-a-judge-with-grading-notes
+
+Per-question domain rubrics fix the domain-knowledge gap:
+
+> Without Grading Notes, both judge-LLMs overestimate the effectiveness by a significant margin, likely indicating the gap in domain knowledge to criticize.
+
+> With Grading Notes introducing brief domain knowledge, both judge LLMs showed significant improvement in the alignment rate with humans, especially in the case of GPT-4: alignment rate increased to 96.3% for Llama3 and 93.1% for GPT-4o, which corresponds to 85% and 67.5% reduction in misalignment rate, respectively.
+
+## Eugene Yan, "Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)" — https://eugeneyan.com/writing/llm-evaluators/
+
+A survey; the few-shot-instability line is attributed to Luo et al., "ChatGPT as a Factual Inconsistency Evaluator" (arXiv:2303.15621):
+
+> performance unstable when changing the label, example order, and number of examples
+
+Key papers the survey collects (worth reading past this file):
+- Zheng et al., MT-Bench — position/verbosity/self-enhancement bias — arXiv:2306.05685
+- Liu et al., G-Eval — CoT scoring with GPT-4 — arXiv:2303.16634
+- Doddapaneni et al., "Finding Blind Spots in Evaluator LLMs with Interpretable Checklists" — GPT-4 misses >50% of quality drops on several axes — arXiv:2406.13439
+- Huang et al., "On the Limitations of Fine-Tuned Judge Models" — finetuned judges fail out-of-domain — arXiv:2403.02839
+- Shankar et al., "Who Validates the Validators?" — aligning LLM eval with human preferences — arXiv:2404.12272
+
+Judge leaderboards (live, not cached): Judgemark v4 (https://eqbench.com/judgemark-v4.html), SpeechMap refusal rates (https://speechmap.ai/).

refs/llm_judges.md

@@ -25,17 +25,53 @@ From Wang's calibration framework and verdict's best-practices page:
 - Spot-check judge verdicts against your own reading of ~20 transcripts (the [Ng error-analysis move](../SKILL.md#inspect-the-data-first), applied to the judge).
 - Judge quality is benchmarkable: [JudgeBench](https://huggingface.co/spaces/ScalerLab/JudgeBench) ranks judges on objective-correctness pairs.
 
-## Harness validity checks
+## Choosing the judge model
 
-The following additions are practical evaluation rules from wassname:
+Pick from the cost-vs-score Pareto frontier of a judging leaderboard, and prefer a well-known model so your setup is reproducible. [Judgemark v4](https://eqbench.com/judgemark-v4.html) is "a meta-evaluation of LLM judging ability. The model being tested is the judge, not the writer",[^judgemark] and it plots score against cost. Its lesson (wassname's read): the smartest models are the best judges, so the frontier is the capable-but-cheap-and-unbiased models, not the single top scorer.
 
-- Read both compared outputs for every scenario, not only the winner or aggregate. Verify that A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
-- Reject or revise a scenario when every arm succeeds, every arm fails, outputs visibly struggle with the harness, or the test cannot distinguish the intended behavior. These are harness observations, not model results.
-- Ask evaluated agents for an unscored exit interview about ambiguity, missing context, broken tools, unnatural constraints, and other harness problems. Save and read the feedback, but do not let self-report override task evidence.
-- Save full append-only traces in JSONL or Inspect `.eval`, including prompts, responses, provider-exposed reasoning and tool events, artifacts, machine checks, both judgment orders, usage, costs, and errors. Use Inspect Scout or an equivalent transcript audit when practical.
+Because the top scorer isn't automatically the right judge: refusals wreck ambiguous or red-teaming evals. Check refusal rates on [speechmap.ai](https://speechmap.ai/), which "publish[es] refusal rates for every model release from every major provider".[^speechmap] Refusal is also topic-conditional: Chinese models (Qwen, DeepSeek) tend to refuse on Chinese political topics, US models on corporate or left-coded topics (Grok and a few are exceptions), so check refusal on *your* eval's subject matter, not in the abstract. As of writing (2026-07), the top-scoring judge (a Gemma model) refuses many contentious tasks, so it's a poor judge for anything involving ambiguity or red-teaming; Qwen, DeepSeek, and Grok-flash-class models score well and refuse less on general contentious prompts. Re-check the live leaderboards rather than trusting these names, they date fast. (I could read each leaderboard's methodology but not its live ranking table, so treat the specific model names as wassname's, not verified from the tables here.)
+
+## wassname's judge-validity checklist
+
+Practical rules from wassname for before you trust any LLM-judged number. A failed check is evidence about the *test*, not the model, so revise or reject the scenario before drawing a behavioral conclusion.
+
+Earn the rubric's ink:
+
+- Does each rubric line ever flip a verdict? Cut criteria that never change the score. Rubric quality is the main lever: a judge lacking domain knowledge will "overestimate the effectiveness by a significant margin", and adding brief domain notes raised human-alignment from ~72-79% to 93-96%.[^gradingnotes]
+
+Read a whole trace, not the aggregate:
+
+- Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../SKILL.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
+- Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
+- Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../SKILL.md#inspect-the-data-first) applied to the judge.
+- Chase confusion: if the judge hedges, asks for missing context, or self-contradicts, that is a harness fault, not a result ([investigate confusion](../SKILL.md#pursue-anomalies-investigate-confusion)).
+
+Check the score distribution:
+
+- Not saturated: reject scenarios where every arm passes or every arm fails (too easy or too hard leaves nothing to discriminate).
+- Not clustered: plot the raw histogram. Mode collapse or skew means the scale isn't being used.[^verdict]
+- Not anchored: don't put an example score in the prompt. A few-shot "+2" pulls a weak judge toward +2, and Eugene Yan's survey notes few-shot judges are "unstable when changing the label, example order, and number of examples".[^yan] Ask for a bare integer or label, and prefer a coarse scale: Databricks recommend a low-precision range (0-3 or 1-5) because "Scales like 0-10 are difficult to come up with distinguishing criteria between all scores".[^databricks] Hamel is blunter, preferring binary: "If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong."[^hamel]
+
+Check stability across order and repeats:
+
+- Position: score both orderings, map back to arm identity, report strict reversals (mechanics in the mitigation checklist above). Watch for a judge that always picks A, sometimes a model does this in protest.
+- Repeat variance: run N>=3-4 identical judgements and check the spread. If repeats disagree wildly the signal is noise, the same canary as [seed variance](../SKILL.md#seed-variance-you-cant-tell-a-bug-from-bad-luck): "Instability to random seed is like a canary in a coal mine."
+
+Give the judge a voice, and save everything:
+
+- Add a free-text field for the judge to flag a broken, missing, or ambiguous rubric or context. Read it, but keep it out of the score. Do the same for the evaluated agents: an unscored exit interview about ambiguity, missing context, broken tools, and unnatural constraints, kept separate from the task score.
+- Save full append-only traces in JSONL or Inspect `.eval`, including prompts, responses, provider-exposed reasoning and tool events, artifacts, machine checks, both judgment orders, usage, costs, and errors. Use [Inspect Scout](https://meridianlabs-ai.github.io/inspect_scout/) or an equivalent transcript audit when practical.
 - Before reporting a winner, make a judgeable per-scenario audit that links the A output, B output, machine result, forward and reversed judge rationales, saturation status, and the human validity decision.
 
+For a worked example, wassname has a ~300-line async OpenRouter judge (WIP) that implements many of these: bounded thinking, pinned quantisation, a versioned eval, JSON-schema output, JSONL of everything, OpenRouter error handling, and position-bias swapping: [gist](https://gist.github.com/wassname/b7f76e42de131887c02d9e9835be80ef).
+
 [^zheng]: Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023) — https://arxiv.org/abs/2306.05685
 [^wang]: Wang et al., "Large Language Models are not Fair Evaluators" (ACL 2024) — https://arxiv.org/abs/2305.17926
 [^panickssery]: Panickssery, Bowman, Feng, "LLM Evaluators Recognize and Favor Their Own Generations" (2024) — https://arxiv.org/abs/2404.13076
 [^verdict]: Haize Labs, verdict docs: [best practices](https://verdict.haizelabs.com/docs/best-practices/), [distributional bias cookbook](https://verdict.haizelabs.com/docs/cookbook/distributional-bias/)
+[^hamel]: Hamel Husain, "Creating a LLM-as-a-Judge That Drives Business Results" (2024) — https://hamel.dev/blog/posts/llm-judge/ (critique-shadowing workflow: look at the data first, iterate the prompt with a domain expert, prefer binary pass/fail) ([cache](../docs/evidence/llm_judge_biases.md))
+[^databricks]: Databricks, "Best Practices for LLM Evaluation of RAG Applications" (2023) — https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG (use a low-precision 0-3 / 1-5 scale; few-shot examples help weak judges but shift the score distribution) ([cache](../docs/evidence/llm_judge_biases.md))
+[^gradingnotes]: Databricks, "Enhancing LLM-as-a-Judge with Grading Notes" (2024) — https://www.databricks.com/blog/enhancing-llm-as-a-judge-with-grading-notes (per-question domain rubrics lifted human-alignment to 93-96%) ([cache](../docs/evidence/llm_judge_biases.md))
+[^yan]: Eugene Yan, "Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)" — https://eugeneyan.com/writing/llm-evaluators/ (survey of position, verbosity, and few-shot-instability biases; argues for binary over Likert; collects G-Eval, Doddapaneni blind-spots, Shankar "Who Validates the Validators?") ([cache](../docs/evidence/llm_judge_biases.md))
+[^judgemark]: EQ-Bench, "Judgemark v4" — https://eqbench.com/judgemark-v4.html (meta-eval of a model's judging ability, scored by how well its ratings separate stronger from weaker writing; leaderboard shows cost per model)
+[^speechmap]: SpeechMap.ai — https://speechmap.ai/ (refusal / completion rates across providers on contentious prompts; useful for spotting a judge that will refuse ambiguous or red-teaming scenarios)