From d6fe8c496b189e4c638958de4e66b04a57097810 Mon Sep 17 00:00:00 2001
From: wassname <1103714+wassname@users.noreply.github.com>
Date: Sun, 12 Jul 2026 13:42:14 +0800
Subject: [PATCH] AGENTS: note steering-demo calibration is not
 method-comparable yet (open)

rep is a breakdown detector not a graded dial; ans_mass is confidence not
coherence and should not drive calibration. Plan: one graded degeneracy
measure D(C), search per method to a common off-target budget, compare
on-target there. rep-non-monotone anomaly flagged as unchecked (read traces).

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
---
 AGENTS.md | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/AGENTS.md b/AGENTS.md
index 1861c73..ee08715 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -40,6 +40,22 @@ Runtime is steering-lite: `with v(model, C=8): model.generate(...)`.
 - One-off validation scripts live in `scripts/scratch/` (u4_step1/2, parity_u1).
 - TODO eval notebook: steer -authority, tinymfv fast (N=16, tokens=16, mfq-2)
   vs unsteered baseline.
+- OPEN: steering-demo calibration is not method-comparable yet. To compare
+  methods you want the same OFF-TARGET budget, then read the on-target effect.
+  The current search finds each method's own "max coherent" edge, which does
+  NOT equalize off-target across methods. Problems: (a) `rep` (distinct-3) is a
+  breakdown *detector*, ~0 through the coherent range then it jumps -- a step
+  function, not a graded dial to calibrate on; (b) `ans_mass` is answer-
+  commitment (confidence, same family as the rejected `pmass`), NOT coherence --
+  it should be a per-row "is the readout valid" flag, not part of the
+  calibration. Plan: one graded degeneracy measure D(C) (candidates: gzip ratio,
+  mean distinct-1..4, seq-rep-n -- NOT perplexity, loops are low-perplexity),
+  search per method for D(C)=tau (common budget), compare on-target there.
+  First cheap step: measure gzip-ratio + distinct-1..4 on the saved
+  `artifacts/steering_demo_results.json` generations to see which is actually
+  graded/monotone in C before switching. Caveat: the "rep non-monotone in C"
+  anomaly (word -0.35 rep=1.0 vs -0.70 rep=0.34) is UNCHECKED -- read the traces
+  qualitatively before trusting it; likely a short-trace/seed artifact, not real.
 
 ## Style