teach judge audits to repair setup confusion

refs/llm_judges.md

@@ -71,9 +71,13 @@ Read a whole trace, not the aggregate:
 - Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../SKILL.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
 - Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
 - Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../SKILL.md#inspect-the-data-first) applied to the judge.
-- Chase confusion: if the judge hedges, asks for missing context, or self-contradicts, that is a harness fault, not a result ([investigate confusion](../SKILL.md#pursue-anomalies-investigate-confusion)).
+- Chase confusion: if the judge hedges, asks for missing context, or self-contradicts, treat the prompt or rubric as defective until the source and complete traces rule that out ([investigate confusion](../SKILL.md#pursue-anomalies-investigate-confusion)).
+- Find the first exposed passage where the candidate's or judge's interpretation stops matching the source-backed intended task. Quote the complete passage. Name the exact prompt sentence, rubric line, or missing premise that allowed it. Check that diagnosis against the source, make one small setup edit, then rerun blind. Reject the edit if it gives away the answer, adds caveats, or leaves the same confusion. Only call it a model or judge error after the setup clearly rules out that reading.
+- Keep evidence channels separate and verbatim. Label `candidate reasoning`, `candidate answer`, `candidate feedback`, `judge reasoning`, and `judge feedback` by their raw field. Feedback or `vent` is an exit interview about ambiguity, missing context, broken tools, or unnatural constraints. It is evidence about the setup, not a score and not automatically true. Quote the complete feedback field and enough surrounding reasoning to show the misconception. A score range, editorial label, or short phrase is not a reasoning quote.
 - Deliver the whole prompt in the USER turn, not a system prompt. System-role instructions are not reliably honored across models and providers (and OpenRouter routes one model across several providers with different chat templates), so a rubric or output-format instruction placed in `system` can be silently under-weighted, showing up as inconsistent formatting or ignored constraints. It is standard to concatenate everything into the user message. If you must use a system prompt, confirm adherence per provider before trusting the scores.
 
+> Did it time out, or was it reasoning behind a buffer for longer than you think? Check the event timestamps and transport state. -- wassname
+
 > Note from wassname's LLM to other LLMs: when you SHOW the human any of this (a judge trace, a compared A/B pair, a sample, a prompt design), reproduce it in FULL and verbatim, prompts, separators, special tokens, and the reasoning/thinking block included. Your trained instinct is to digest and trim; resist it, that instinct destroys exactly the boilerplate (system prompt, special tokens, separators) that the human needs to debug. Formatting may only HIGHLIGHT or REORGANISE: bold or underline the diff, split into a table, use sections, but never drop, elide with "...", or paraphrase. When comparing variants, show ALL of them this way, not a representative one. Link the source file so they can open the raw (`results/runs/.../x.jsonl`, ideally `path:line`). "Show me" means reproduce, not describe. Trim only when explicitly asked. -- CLAUDE
 
 Check the score distribution: