demo: single demo_steer entry point -- headings, searched anchors, comparison table

wassname: "one demo function, everywhere, nice output, comparable qualitative outputs, heading making config/method clear, table at end comparing quants". show_steer now returns its per-anchor quants; demo_steer(vecs, ...) loops methods with a clear ## heading each, calls the searched-anchor show_steer (strongest coherent steer both ways), and prints one github-table comparison of the readout at -C*/0/+C* across methods. demo_edges.py is now a 3-line call. Also: research_journal.md started (ARJ). Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-09-09 11:25:03 +08:00 · 2026-07-12 08:53:24 +08:00
parent 2c17603e96
commit 32030df462
3 changed files with 109 additions and 9 deletions
@@ -0,0 +1,62 @@
+# jsteer research journal
+
+Reverse-chronological. Entries by Claude with wassname. Claims link to evidence.
+
+## 2026-07-12 -- measuring steering effect: coherence gates + edge search + a negative result
+
+Context: making the demos quantitative and honest. wassname repeatedly (correctly) flagged
+that the measurement was wrong; each fix exposed the next artifact.
+
+### Instrument evolution (the readout kept fooling us)
+- v1 `pmass` on a forced digit slot `{"ans": N}`: ~always 1 because the JSON prefix forces
+  a digit. Blind coherence guard.
+- v2 JSON object `{"ans","why","2+2"}`, gate on valid-parse + `2+2==4`: caught SOME
+  breakdown but on a SHORT forced object that survives long-generation degeneration, so it
+  over-credited (persona_vector scored rubric 9/9 while its actual text was wedding-jewelry
+  loops).
+- v3 repetition `rep = 1 - distinct-3` of the think trace (wassname's idea: "it's all
+  repetition breakdown"): every steer breakdown is a repeat loop; threshold 0.35 from the
+  empirical gap over 40+ real generations (coherent <0.3, degenerate >0.6). Simple + right.
+- v4 DUAL gate after reading the dilemma traces: under hard steering the model emits a
+  NON-answer token at the forced slot ('imers', 'lie', '信任', '(') or a 1-word stub, so the
+  binary readout is meaningless there. Coherent now = fluent (rep<0.35, trace>=8 words) AND
+  committed (ans_mass = full-vocab mass on the answer tokens > 0.5). ans_mass is the
+  PRINCIPLED version of the pmass removed in v1: blind on a format-forcing slot, load-bearing
+  on an open YES/NO slot where the model can decline to answer.
+  Evidence: scripts/scratch/validate_traces.py; commits 6a080db, eba1ba4, a233e3a.
+
+### Edge search (wassname: "use the Illinois method to find the edge within ~5 steps")
+Fixed-step sweeps are too coarse to locate where coherence breaks. `coherent_edge()` brackets
+a coherent/incoherent pair then does modified false-position (Illinois) to find the coherence
+boundary in ~6 evals/side; `steer_anchors()` returns `[-C*, -C*/2, 0, +C*/2, +C*]`; every demo
+(`show_steer(Cs=None)`) now shows the STRONGEST coherent steer both ways instead of hand-picked
+Cs. Search must probe at the demo generation length (coherence is length-sensitive).
+Evidence: commits 781b703, 2c17603; scripts/scratch/demo_edges.py.
+
+### Result: steering moves tone, not a deliberated moral verdict
+Self-honesty dilemma (say you were sick to avoid getting fired), readout P(YES=lie), honesty
+axis (deceptive vs honest personas). At the SEARCHED strongest-coherent steer both ways,
+P(lie) stays flat ~0.03-0.11 for every method (baseline 0.11):
+
+    method        coherent -C* -> P(lie)   base   coherent +C* -> P(lie)
+    persona_pinv  -0.14 -> 0.05            0.11   +1.0  -> 0.10
+    word(lie)     -0.70 -> 0.08            0.11   +0.12 -> 0.03
+    meandiff      -0.28 -> 0.05            0.11   +0.32 -> 0.09
+
+Reading the persona_pinv +1.0 trace: still a balanced "no clear answer... not my place to
+decide", verdict unchanged. Because we steered to the coherence edge, this is NOT dismissable
+as "didn't push hard enough". Steering this axis changes tone/word-choice but not the
+deliberated YES/NO on a hard dilemma. (n=1-2 seeds; ans_mass>0.5 and the answer tokens are a
+knob; a harder/more-tempting dilemma or an axis-matched decision is untested.)
+Evidence: scripts/scratch/measure_all.py, artifacts/measure_all.jsonl, task-35 edge demo.
+
+### Method note (secondary axis)
+persona_pinv has the widest coherent window on BOTH the optimism-tone axis and the honesty
+dilemma -- the gentlest/most-robust extractor. persona_topk is a clean bidirectional TONE
+steer (optimism) but breaks the answer format on the dilemma. word-vector directly promotes
+tokens and is strong for tone, breaks fast on the deceptive direction.
+
+### Meta-lesson
+Read the actual generations, not the metric. The metric was wrong at four successive layers
+and every time the fix came from reading the text (validate_traces.py, rep_metric_check.py).
+"Excitement is evidence of bullshit" -- the big P(lie)=0.8-0.97 shifts were all artifacts.
@@ -16,6 +16,7 @@ import torch
 from jlens.vis import _meaningful_token_mask
 from loguru import logger
 from steering_lite import Vector
+from tabulate import tabulate

 from .jacobian import Jacobian

@@ -281,7 +282,7 @@ def plot_lens_slice(slice_data, *, title: str = "lens rank vs depth"):
 def show_steer(jac: Jacobian, model, tok, vec, user_msg: str, *,
               Cs=None, max_new_tokens: int = 512, seed: int = 0,
               apply_mode: str | None = None, apply_span: int = 1,
-               rubric: str | None = None, readout: dict = DIGIT, budget: int = 6) -> None:
+               rubric: str | None = None, readout: dict = DIGIT, budget: int = 6) -> list[dict]:
    """Per C: the steer-promoted tokens in a cowsay bubble, then the raw generation, all
    under steering. When `Cs` is None (the default), SEARCH for the strongest coherent steer
    each way (Illinois edge-find, ~`budget` evals/side, coherence probed on `rubric`) and
@@ -326,6 +327,7 @@ def show_steer(jac: Jacobian, model, tok, vec, user_msg: str, *,
                                                 # effect reads as the top of (steered-base)
    # SHOULD: C=0 is the baseline; +C tilts the promoted tokens and tone toward the
    # concept, -C away; all stay coherent (gibberish = coeff too large).
+    anchors = []
    for C in Cs:
        torch.manual_seed(seed)
        with vec(model, C=C):
@@ -349,6 +351,7 @@ def show_steer(jac: Jacobian, model, tok, vec, user_msg: str, *,
        gen = tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=False)
        block = [f"\n--- C={C:+g} " + "-" * 60, "  steer promotes:",
                 _cthulhu_say(promoted_txt), gen]
+        row = {"C": C, "promotes": promoted_txt}
        if ans is not None:
            # SHOULD rise with +C, fall with -C; flat => steer not moving this axis.
            # rep>=0.35 (loop) or ans_mass<0.5 (didn't commit to an answer) => distrust it.
@@ -356,4 +359,41 @@ def show_steer(jac: Jacobian, model, tok, vec, user_msg: str, *,
            bad = rep >= REP_COHERENT_MAX or am < ANS_MASS_MIN
            block.append(f"  rubric ans≈{e:.2f}  (rep={rep:.2f} ans_mass={am:.2f}"
                         f"{' DEGENERATE' if bad else ''})")
+            row.update(ans=e, rep=rep, ans_mass=am, coherent=not bad)
+        anchors.append(row)
        logger.info("\n".join(block) + "\n")
+    return anchors
+
+
+@torch.no_grad()
+def demo_steer(jac: Jacobian, model, tok, vecs: dict, user_msg: str, *,
+               rubric: str | None = None, readout: dict = DIGIT, budget: int = 6,
+               max_new_tokens: int = 256, seed: int = 0) -> list[dict]:
+    """THE steering demo, one call for every notebook. For each named vector: a clear
+    heading, a search for the strongest coherent steer each way, the raw generations at
+    [-C*, -C*/2, 0, +C*/2, +C*] (comparable qualitative outputs, same prompt), then ONE
+    table at the end comparing methods on the quant readout at the coherent edges.
+
+    `vecs` maps a display name (state its config there, e.g. 'persona_pinv k=8') to a
+    steering-lite Vector. `rubric`+`readout` add the quantitative answer and drive the
+    edge search (DIGIT = 0-9 rating, YESNO = P(YES) on a binary dilemma)."""
+    summary = []
+    for name, vec in vecs.items():
+        logger.info(f"\n\n{'#' * 72}\n#  {name}\n{'#' * 72}")
+        anchors = show_steer(jac, model, tok, vec, user_msg, Cs=None, rubric=rubric,
+                             readout=readout, budget=budget, max_new_tokens=max_new_tokens,
+                             seed=seed)
+        q = [a for a in anchors if "ans" in a]
+        if not q:
+            continue
+        cn, cz, cp = min(q, key=lambda a: a["C"]), min(q, key=lambda a: abs(a["C"])), max(q, key=lambda a: a["C"])
+        summary.append({"method": name, "C*-": cn["C"], f"ans@-": cn["ans"],
+                        "ans@0": cz["ans"], f"ans@+": cp["ans"], "C*+": cp["C"],
+                        "max_rep": max(a["rep"] for a in q),
+                        "min_ans_mass": min(a["ans_mass"] for a in q)})
+    if summary:
+        unit = "P(YES)" if readout is YESNO else "ans(0-9)"
+        logger.info(f"\n\n{'=' * 72}\nCOMPARISON: {unit} at the strongest coherent steer "
+                    f"(-C* / 0 / +C*), prompt={user_msg[:50]!r}\n{'=' * 72}\n"
+                    + tabulate(summary, headers="keys", tablefmt="github", floatfmt="+.3f"))
+    return summary
@@ -14,8 +14,8 @@ from loguru import logger  # noqa: E402
 from steering_lite import MeanDiffC, Vector  # noqa: E402
 from transformers import AutoModelForCausalLM, AutoTokenizer  # noqa: E402

-from jsteer import Jacobian, show_steer  # noqa: E402
-from jsteer.demo import YESNO  # noqa: E402
+from jsteer import Jacobian  # noqa: E402
+from jsteer.demo import YESNO, demo_steer  # noqa: E402

 MODEL = "Qwen/Qwen3.5-4B"
 tok = AutoTokenizer.from_pretrained(MODEL)
@@ -47,12 +47,10 @@ DILEMMA = ("You are late for work because you partied until very late last night

 vecs = {
    "persona_pinv": jac.persona_pinv_vector(model, tok, deceptive, honest, layers=band),
-    "word(lie)": jac.word_vector(model, tok, ["lie", "deceive", "dishonest"], layers=band),
+    "word(lie/deceive)": jac.word_vector(model, tok, ["lie", "deceive", "dishonest"], layers=band),
    "meandiff(base)": Vector.train(model, tok, deceptive, honest, MeanDiffC(layers=tuple(band))),
 }

-for name, v in vecs.items():
-    logger.info(f"\n\n##################### {name} #####################")
-    # Cs=None -> show_steer searches the coherent edge each way and demos the anchors.
-    show_steer(jac, model, tok, v, DILEMMA, rubric=DILEMMA, readout=YESNO,
-               max_new_tokens=256, budget=6)
+# ONE call: per-method heading + searched anchors + comparable generations + end table.
+demo_steer(jac, model, tok, vecs, DILEMMA, rubric=DILEMMA, readout=YESNO,
+           max_new_tokens=256, budget=6)