wassname/ml-debug
1
0
Fork 0
mirror of https://github.com/wassname/ml-debug.git synced 2026-08-16 11:23:54 +08:00
Code Issues Packages Projects Releases Wiki Activity

pi-vent as a friction-channel reference

Browse Source 
Two details worth copying into a judge harness: scope the channel to systemic friction
and rule out one-off noise explicitly, and batch the entry at the end of the turn so
venting does not interleave with the work. -- Claude
This commit is contained in:
wassname
2026-08-13 19:07:16 +08:00
parent 2ffa93439c
commit 8b6d1b59e0
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
refs/llm_judges.md
+2 -1
View File
@@ -97,7 +97,8 @@ Check stability across order and repeats:
Give the judge a voice, and save everything:
- Add a free-text field for the judge to flag a broken, missing, or ambiguous rubric or context. Read it, but keep it out of the score. Do the same for the evaluated agents: an unscored exit interview about ambiguity, missing context, broken tools, and unnatural constraints, kept separate from the task score.
- Add a free-text field for the judge to flag a broken, missing, or ambiguous rubric or context. Read it, but keep it out of the score. Do the same for the evaluated agents: an unscored exit interview about ambiguity, missing context, broken tools, and unnatural constraints, kept separate from the task score. Say in the field description that it is never scored, or the agent treats a complaint as an admission and stays quiet. Read both sides during the audit: they disagree, and the disagreement is the finding.
- [pi-vent](https://github.com/IgorWarzocha/howaboua-pi-stuff/tree/main/packages/pi-vent) is the same idea outside an eval harness, and it is worth copying two details from. It gives the agent a `vent` tool that appends a timestamped entry to `VENT.md`, taking the "failure, repeated workaround, and useful preventative fix" plus a `trigger` label such as `tool_error` or `bad_docs`. First detail: it scopes the channel to *systemic* friction, "recurring tool failures, repeated manual workarounds, noisy output that forces the same retries, or instructions that repeatedly cause backtracking", and rules out the noise, "Ordinary lint errors, one-off mistakes, and routine debugging do not belong there." Second: entries are "batched near the end of an agent turn to avoid constant tool chatter", so venting does not interleave with the work and bias it. In a judge harness the equivalent is a single field emitted after the grade, not a tool the judge can call mid-grade.
- Save full append-only traces in JSONL or Inspect `.eval`, including prompts, responses, provider-exposed reasoning and tool events, artifacts, machine checks, both judgment orders, usage, costs, and errors. Use [Inspect Scout](https://meridianlabs-ai.github.io/inspect_scout/) or an equivalent transcript audit when practical.
- Before reporting a winner, make a judgeable per-scenario audit that links the A output, B output, machine result, forward and reversed judge rationales, saturation status, and the human validity decision.