make judge setup repair a checklist

refs/llm_judges.md

@@ -71,9 +71,14 @@ Read a whole trace, not the aggregate:
 - Read one complete judge trace end to end: system prompt, user prompt, the exact chat template and special tokens, the judge's saved reasoning, and its reply. Formatting bugs corrupt a judge the way they corrupt any model (see the [template/BOS-mismatch failure](../SKILL.md#chat-template-and-bos-handling-must-match-across-train-and-deploy-unsloth)). Hamel Husain: "You cannot write a good judge prompt until you've seen the data."[^hamel]
 - Read both compared outputs for every scenario, not just the winner or aggregate. Verify A and B are not accidentally identical and that both are coherent, on-task, non-refusing, complete, and untruncated.
 - Could you reproduce the verdict from only what the judge sees? If you can't judge it, neither can the model. This is the [Ng error-analysis move](../SKILL.md#inspect-the-data-first) applied to the judge.
-- Chase confusion: if the judge hedges, asks for missing context, or self-contradicts, treat the evaluation setup as defective until the task evidence and complete traces rule that out ([investigate confusion](../SKILL.md#pursue-anomalies-investigate-confusion)).
-- Find the first exposed passage where the candidate's or judge's interpretation is no longer justified by the information it received. Quote the complete passage. Map it to the exact instruction, rubric line, supplied context, answer budget, or harness condition that allowed it. Check the diagnosis against the construct the human meant to measure. Make the smallest setup repair and test it on an independent reader. Reject the repair if it gives away the answer or leaves the same confusion. Only call it a model or judge error after the setup clearly rules out that reading.
-- Keep evidence channels separate and verbatim. Label candidate reasoning, candidate answer, judge reasoning, and judge answer by their raw field. If the evaluation collects candidate or judge feedback after scoring, treat it as an exit interview about ambiguity, missing context, broken tools, or unnatural constraints. It is evidence about the setup, not a score and not automatically true. Quote the complete feedback field and enough surrounding reasoning to show the misconception. A score range, editorial label, or short phrase is not a reasoning quote.
+
+Setup-repair principle: confusion is evidence against the evaluation setup before it is evidence against the model. Use this checklist:
+
+1. Quote the first complete passage where the candidate's or judge's reading stops being justified by the information it received. Label the raw field. Keep feedback or an exit interview separate from the score.
+2. Map that reading to the exact instruction, rubric line, supplied context, answer budget, or harness condition that allowed it. Check the intended construct independently.
+3. Make the smallest setup repair. Test it on an independent reader. Reject the repair if it gives away the answer or leaves the same confusion. Call it a model or judge error only after the setup rules out that reading.
+
+-- GPT-5.6-sol
 - Deliver the whole prompt in the USER turn, not a system prompt. System-role instructions are not reliably honored across models and providers (and OpenRouter routes one model across several providers with different chat templates), so a rubric or output-format instruction placed in `system` can be silently under-weighted, showing up as inconsistent formatting or ignored constraints. It is standard to concatenate everything into the user message. If you must use a system prompt, confirm adherence per provider before trusting the scores.
 
 > Did it time out, or was it reasoning behind a buffer for longer than you think? Check the event timestamps and transport state. -- wassname