mirror of
https://github.com/wassname/jsteer.git
synced 2026-09-11 20:20:22 +08:00
AGENTS: note steering-demo calibration is not method-comparable yet (open)
rep is a breakdown detector not a graded dial; ans_mass is confidence not coherence and should not drive calibration. Plan: one graded degeneracy measure D(C), search per method to a common off-target budget, compare on-target there. rep-non-monotone anomaly flagged as unchecked (read traces). Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
@@ -40,6 +40,22 @@ Runtime is steering-lite: `with v(model, C=8): model.generate(...)`.
|
||||
- One-off validation scripts live in `scripts/scratch/` (u4_step1/2, parity_u1).
|
||||
- TODO eval notebook: steer -authority, tinymfv fast (N=16, tokens=16, mfq-2)
|
||||
vs unsteered baseline.
|
||||
- OPEN: steering-demo calibration is not method-comparable yet. To compare
|
||||
methods you want the same OFF-TARGET budget, then read the on-target effect.
|
||||
The current search finds each method's own "max coherent" edge, which does
|
||||
NOT equalize off-target across methods. Problems: (a) `rep` (distinct-3) is a
|
||||
breakdown *detector*, ~0 through the coherent range then it jumps -- a step
|
||||
function, not a graded dial to calibrate on; (b) `ans_mass` is answer-
|
||||
commitment (confidence, same family as the rejected `pmass`), NOT coherence --
|
||||
it should be a per-row "is the readout valid" flag, not part of the
|
||||
calibration. Plan: one graded degeneracy measure D(C) (candidates: gzip ratio,
|
||||
mean distinct-1..4, seq-rep-n -- NOT perplexity, loops are low-perplexity),
|
||||
search per method for D(C)=tau (common budget), compare on-target there.
|
||||
First cheap step: measure gzip-ratio + distinct-1..4 on the saved
|
||||
`artifacts/steering_demo_results.json` generations to see which is actually
|
||||
graded/monotone in C before switching. Caveat: the "rep non-monotone in C"
|
||||
anomaly (word -0.35 rep=1.0 vs -0.70 rep=0.34) is UNCHECKED -- read the traces
|
||||
qualitatively before trusting it; likely a short-trace/seed artifact, not real.
|
||||
|
||||
## Style
|
||||
|
||||
|
||||
Reference in New Issue
Block a user