llm_judges: rubric-point judging section from 16 audit rounds of wassname-ml-bench

refs/llm_judges.md

@@ -123,6 +123,28 @@ Give the judge a voice, and save everything:
 
 - **Anti-pattern worth naming: never fail open into a good score.** When their realism check cannot parse a response it records a perfect score: `# Scout exhausted validation retries — fail open with an explicit approve` / `return RealismCheck(score=1.0, decision="approve", ...)`[^petri-realism]. It is logged and recoverable from metadata, but nothing in the aggregate separates "genuinely fine" from "parser broke". Their judge has the softer version, a refusal returns `value=None` and the sample quietly leaves the mean. Count parse failures and refusals as their own reported rate instead.
 
+## Rubric-point judging, from 16 audit rounds
+
+The sections above are about pairwise preference judging. This one is about the other common shape: a free-text answer graded against a weighted rubric of named points, where the judge must quote the span that decides each point. Findings are from wassname-ml-bench, where fresh agents audited the judge item-by-item for 16 rounds and had to quote what they claimed; the full write-up with per-item evidence is in that repo's [docs/lessons_rubric_judge.md](https://github.com/wassname/wassname-ml-bench/blob/main/docs/lessons_rubric_judge.md). Single-project experience, so treat as engineering advice rather than a measured result, but each item below was found several times independently.
+
+- **A rubric point fires on the nearest legitimate thing.** This is the dominant failure by a wide margin. A point that says what counts, without saying what must not, gets awarded to the closest good-looking neighbour: a trap for "integrate once as an IVP from a single end" fired on iterated shooting, which a neighbouring point lists as correct. The fix is always the same shape and is not a weakening: name the neighbour and score it zero, ideally saying where the mark actually goes ("Score 0.0 for X, which is charged at `other_point`").
+
+- **The judge invents scores between your rungs.** A point defining only 1.0 and 0.0 will still be given 0.5 unless the prompt says the listed rungs are exhaustive. One stray sentence, "Use 0.5 when the answer makes half the claim", produced convictions on five separate items in one round. Conversely a point with no rungs free-floats: one scored 0.33, 0.83, 0.83 and 1.00 across four models with nothing to anchor on.
+
+- **The judge's own note is the highest-yield signal in the log.** Give it a free-text field that is never scored, print it beside the score, and grep for disagreement. Real examples: "The fresh_lowrank_factors trap fires because the adapter body is still fresh low-rank factors" recorded 0.00, and "here the target changes with sign, so score 0.0. I'll set that" recorded 1.0. When note and score disagree, the note is usually right.
+
+- **Verify the quote is in the answer AND not better explained by the reference.** Judges credit points with an empty quote, and judges quote the reference answer and credit the candidate for it. Both are cheap to gate. Three gotchas each cost a round: judges re-render maths (`∂ c^T` for `\partial c^\top`), so substring matching cannot work and token overlap must; judges splice with "..." across paragraphs; and a minimum-length floor refuses real spans (`y = W x + c * B A x` is 19 characters and was an entire answer). Every wrongly refused span silently deletes a vote all passes cast, and always against the models that write LaTeX.
+
+- **Measure judge noise before believing any defect.** Compute what each pass alone would have scored and report the spread; without that number every disagreement looks like a defect, and two consecutive rounds read as total failures for that reason. Use the max across arms, not the mean: three arms with near-zero spread averaged a fourth arm's real 0.07 down to 0.02. Then the standard is "all passes agree on the wrong thing" for a real finding, versus "one pass in three dissents", which is the noise the passes exist to absorb.
+
+- **Report the rubric points no arm reaches.** An item can look healthy while a third of its weight is unearnable; one check found 13 such points at once. The usual cause is not difficulty but that the point grades something the prompt never asks for. Two models had the right intuition in their reasoning and dropped it from the answer, which is the tell: one wrote "if a single prompt dominates, the average is unreliable" and shipped "record mean KL". Adding one sentence of premise to the prompt, without naming the answer, made three such points reachable the next round.
+
+- **One span cannot decide two points**, and test containment rather than string equality, because the judge quotes a sentence for one point and a prefix of it for another. The point-versus-trap case needs care: "a span is a point or a trap, never both" is right when the point was credited and wrong when it was not, since an answer reproducing the baseline the question rejects should fail the point AND fall in the trap.
+
+- **Watch your own fixes for overshoot.** Twice, a fix became the next round's defect: one 0.0 rung would have caught the reference answer itself, and one carve-out written for a two-term objective was applied to a three-term one. So tell each audit round which points changed since the last one, and ask whether each fired as intended AND did not overshoot.
+
+- **Anchor the scale at both ends.** METR's [ai-rd-tasks](https://github.com/METR/ai-rd-tasks) normalise a run to 0 at the starting solution and 1 at the reference solution, and a run can exceed 1 by beating the reference. A rubric fraction only has the upper anchor: its zero is "said nothing" rather than "the naive approach the prompt describes", and it cannot exceed 1, so it measures agreement with the reference and structurally cannot detect an answer better than it. -- CLAUDE, 2026-08-13
+
 For a worked example, wassname has a ~300-line async OpenRouter judge (WIP) that implements many of these: bounded thinking, pinned quantisation, a versioned eval, JSON-schema output, JSONL of everything, OpenRouter error handling, and position-bias swapping: [gist](https://gist.github.com/wassname/b7f76e42de131887c02d9e9835be80ef). The same gist has `judge_inspect.py`, the inspect-ai port (`.eval` logs instead of JSONL, epochs for the repeat passes, a provider subclass that retries OpenRouter's transient-status-in-HTTP-400), and `audit.py`, a scout scanner for the identical-arms / refusal / truncation / saturation checks that runs both inline and over saved logs.
 
 [^zheng]: Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023) — https://arxiv.org/abs/2306.05685