wassname/jsteer
1
0
Fork 0
mirror of https://github.com/wassname/jsteer.git synced 2026-09-09 11:25:03 +08:00
Code Issues Packages Projects Releases Wiki Activity

AGENTS: correct calibration note -- ans_mass pre-empts rep, not rep insensitivity

Browse Source 
Checked the saved results: 11/14 edges stopped on ans_mass<0.5 with rep still
0.00-0.02. rep never fired, so the fix is rep-alone calibration + ans_mass as a
readout-validity flag, not a new graded measure. Corrects my prior overclaim.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-07-12 14:02:59 +08:00
co-authored by Claudypoo
parent d6fe8c496b
commit 541fc49d4b
1 changed files with 21 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+21 -14
View File
@@ -42,20 +42,27 @@ Runtime is steering-lite: `with v(model, C=8): model.generate(...)`.
vs unsteered baseline.
- OPEN: steering-demo calibration is not method-comparable yet. To compare
methods you want the same OFF-TARGET budget, then read the on-target effect.
The current search finds each method's own "max coherent" edge, which does
NOT equalize off-target across methods. Problems: (a) `rep` (distinct-3) is a
breakdown *detector*, ~0 through the coherent range then it jumps -- a step
function, not a graded dial to calibrate on; (b) `ans_mass` is answer-
commitment (confidence, same family as the rejected `pmass`), NOT coherence --
it should be a per-row "is the readout valid" flag, not part of the
calibration. Plan: one graded degeneracy measure D(C) (candidates: gzip ratio,
mean distinct-1..4, seq-rep-n -- NOT perplexity, loops are low-perplexity),
search per method for D(C)=tau (common budget), compare on-target there.
First cheap step: measure gzip-ratio + distinct-1..4 on the saved
`artifacts/steering_demo_results.json` generations to see which is actually
graded/monotone in C before switching. Caveat: the "rep non-monotone in C"
anomaly (word -0.35 rep=1.0 vs -0.70 rep=0.34) is UNCHECKED -- read the traces
qualitatively before trusting it; likely a short-trace/seed artifact, not real.
The current search finds each method's own "max coherent" edge, which does NOT
equalize off-target -- but the CAUSE (checked in artifacts/steering_demo_results
.json) is NOT that `rep` is insensitive. It is that the dual gate
`rep<0.35 AND ans_mass>0.5` let `ans_mass` pre-empt: 11 of 14 edges stopped on
`ans_mass<0.5` with rep still 0.00-0.02, nowhere near its gate. rep never got
to fire, so we can't conclude it fails as a calibration axis.
`ans_mass` is answer-commitment (confidence, same family as the rejected
`pmass`), a READOUT-VALIDITY concern, not off-target coherence; folding it into
the search is what broke comparability. (Interesting: ans_mass drops before rep
rises -> the steer makes the model hedge/refuse the answer BEFORE its reasoning
goes incoherent. Real effect, keep it as a per-row flag.)
FIX (simpler than a graded measure): calibrate on `rep` ALONE (iso-rep budget,
each method to rep~=0.35), demote `ans_mass` to a per-row "is P(YES) valid"
flag out of the search, then compare on-target at iso-rep. If the readout is
invalid at the rep-budget for a method, that is itself a finding.
0.35 is anchored to the empirical coherent/degenerate gap (rep<~0.3 vs >~0.6,
rep_metric_check.py), not to a base degradation; base C=0 rep=0.00, gap is wide
so anything ~[0.35,0.55] gives the same edge.
Caveat: the "rep non-monotone in C" anomaly (word -0.35 rep=1.0 vs -0.70
rep=0.34) is UNCHECKED -- read the traces qualitatively; likely a short-trace/
seed artifact, not real.
## Style