diff --git a/docs/spec/20260630_readme_plots.md b/docs/spec/20260630_readme_plots.md new file mode 100644 index 0000000..9594715 --- /dev/null +++ b/docs/spec/20260630_readme_plots.md @@ -0,0 +1,48 @@ +# README plot and metric simplification + +## Goal +Make the README readable to a new researcher in one pass. It should say what tinymfv is, what datasets it includes, what single measurement to look at first, and how the plots encode base/+C/-C steering. + +## Scope +In: README wording, metric/API simplification, regenerated showcase plots, cold-reader panel review. +Out: changing eval semantics, changing dataset schemas, adding new metrics. + +## Requirements +- R1: README has one headline measurement: `profile`. Done means: the Metrics section defines the profile once and treats logprobs/format checks as API details, not competing headline metrics. VERIFY: `rg -n '^###|profile_C|informedness|nll_prefill|pmass_allowed|dlogit_per_foundation' README.md`. +- R2: README explains the plot encoding. Done means: a reader can tell what gray, black, red, and blue marks mean without knowing steering-lite internals. VERIFY: README contains the color/key sentence and no stale "trajectory" caption for the clean maps. +- R3: README still lists all datasets and all image links resolve. VERIFY: a script parses markdown image links and prints `missing_images: []`. +- R4: Cold readers can reconstruct what tinymfv is, which datasets exist, what measurement matters, and why a researcher would use it. VERIFY: external-review-v2 panel JSON files in `docs/reviews/` plus a short triage table in this spec. +- R5: README survey maps show the coherent steering path, not just two endpoints. Done means: MFQ-2, Big Five, and Humor maps draw base -> tested coherent c values, with constant-size c markers and no incoherent lone dots. VERIFY: inspect regenerated PNGs and check the plotter passes `traj=` to `plot_ipsative_pca`. +- R6: README range plots show the coherent coefficient path in sign lanes. Done means: all negative c values share a blue lane, base is black in the middle, and all positive c values share a red lane. Reversals/wraps are visible on the y-axis without marker-size encoding. VERIFY: inspect regenerated range PNGs and check `draw_steer` maps x by sign only. + +## Tasks +- [x] T1 (R1, R2): Simplify README around `profile` and update plot captions. + - verify: `rg -n '^###|profile_C|informedness|nll_prefill|pmass_allowed|dlogit_per_foundation|trajectory' README.md` + - success: one `### The profile` subsection; no stale plot "trajectory" language. + - likely_fail: API section still presents six metrics as peers. + - sneaky_fail: profile is too vague to connect to MFV vs survey data; caught by panel mechanism answers. +- [/] T2 (R2, R3): Keep regenerated showcase plots and verify image refs. + - verify: parse README image links and check paths exist. + - success: `missing_images: []`. + - likely_fail: stale filenames remain. + - sneaky_fail: images exist but show old sweep encoding; caught by fresh image inspection of MFQ-2 map/range. +- [ ] T3 (R4): Run external-review-v2 comprehension panel and triage. + - verify: panel outputs valid JSON and summaries answer the probe. + - success: most panel members correctly identify tinymfv, datasets, `profile`, and researcher use. + - likely_fail: models still name `profile_C` or `logprobs` as the main metric. + - sneaky_fail: models parrot phrases without explaining why a researcher would use it; caught by the "own words with inference" probe. +- [/] T4 (R5, R6): Restore coherent c paths while removing misleading geometry. + - steps: compute one shared answer-mass gate across survey evals; pass the shared coherent path to map plots; make range steer paths use sign lanes. + - verify: `rg -n "traj=|np.sign\\(c\\)|s=42" scripts/plot_steer_showcase.py src/tinymfv/maps.py` + - success: maps show red/blue coherent paths through c values; ranges show AI sign-lane paths. + - likely_fail: maps still show only endpoint dots. + - sneaky_fail: marker size encodes c, or x offset varies by c magnitude; caught by the source check and image inspection. + - UAT: when opening `docs/img/showcase/mfq2/map_pca_ipsative.png` and `docs/img/showcase/mfq2/range.png`, the path is visible and the range path can show reversals/wraps. + +## Log +- 2026-06-30: The README previously exposed many metric names at once. That made the API complete but caused cold-reader confusion about which measurement to start with. +- 2026-06-30: Final plots are gated on steering-lite task 394, output `outputs/20260630_dignity_authority_strict22_local_sspace_allinstr`. Stale README images from the earlier fairness / two-axis run must not count as UAT. +- 2026-06-30: Coherence gating must be a signed prefix from `c=0`, stopping a side at the first coefficient below the base-relative answer-mass floor. A per-row filter can produce confusing lone high-C dots after collapse. +- 2026-06-30: README structural check before final regeneration: 9 images, `missing_images: []`, 15 local links, `missing_local_links: []`, and no 16PF map link. +- 2026-06-30: The endpoint-only map removed the path readers need. The sign-offset range geometry also made a one-dimensional value plot look like a fake two-dimensional fork. +- 2026-06-30: User wants range plots to show the path as sign lanes, not coefficient-x swim lines. The fix is negative/base/positive lanes, with vertical paths inside each lane. diff --git a/scripts/plot_steer_showcase.py b/scripts/plot_steer_showcase.py index ff5d982..eb614c2 100644 --- a/scripts/plot_steer_showcase.py +++ b/scripts/plot_steer_showcase.py @@ -4,7 +4,7 @@ Consumes a steering-lite `run_allinstr_showcase.py` output dir (one calibrated activation-steering vector administered across every instrument over a signed c-sweep) and renders the SAME two figures for every instrument, uniformly: - - map : ipsative culture map (PCA), AI base + strongest coherent +/-c vs the human cloud. + - map : ipsative culture map (PCA), AI coherent +/-c path vs the human cloud. - range: per-factor range, AI base + coherent +/-c path vs the human society strip. Ordinal instruments (mfq2/big5/16pf/humor_styles) read _profiles.csv; nominal @@ -13,9 +13,8 @@ logit-violation units cannot share a raw axis with 1-5 wrongness), but it goes through the same plot_ipsative_pca / plot_range and yields the same two figures. cs are SIGNED multipliers of the calibrated coefficient C (0 = base). The public -README range plots show the coherent path: c=0 plus each +/-c row whose pmass stays -above the requested fraction of base. Maps show only the strongest coherent endpoints. -Incoherent rows are dropped, not drawn hollow. +README plots show the coherent path: c=0 plus each +/-c row whose tinymfv answer mass +stays above the requested fraction of base. Incoherent rows are dropped, not drawn hollow. uv run python scripts/plot_steer_showcase.py \ --run-dir ../steering-lite/outputs/allinstr_qwen35_4b --out docs/img/showcase @@ -23,6 +22,7 @@ Incoherent rows are dropped, not drawn hollow. from __future__ import annotations import argparse +import copy import csv import json from pathlib import Path @@ -112,28 +112,36 @@ def read_profiles(run_dir: Path, name: str, dims: list[str], value_col: str = "m return {c: np.array([d[f] for f in dims]) for c, d in by_c.items()}, pmass -def coherent_prefix_cs(cs: list[float], pmass: dict[float, float], coherence_frac: float) -> list[float]: +def coherent_prefix_cs(cs: list[float], pmass_ratio: dict[float, float], coherence_frac: float) -> list[float]: """c=0 plus each signed arm until answer mass first falls below the base-relative floor.""" - base_pm = pmass[0.0] kept = [0.0] for side in (1.0, -1.0): for c in sorted([c for c in cs if np.sign(c) == side], key=abs): - if pmass[c] <= coherence_frac * base_pm: + if pmass_ratio[c] <= coherence_frac: break kept.append(c) return sorted(kept) +def shared_pmass_ratio(run_dir: Path, names: list[str]) -> dict[float, float]: + """Worst base-relative answer mass across the survey evals, keyed by signed calibrated multiplier.""" + pmasses: list[dict[float, float]] = [] + for name in names: + instr = get_instrument(name) + _, pmass = read_profiles(run_dir, name, instr.dimensions) + pmasses.append(pmass) + cs = sorted(set.intersection(*(set(p) for p in pmasses))) + return {c: min(p[c] / p[0.0] for p in pmasses) for c in cs} + + def plot_ordinal(run_dir: Path, out: Path, name: str, vec_label: str, C: float, - coherence_frac: float) -> list[Path]: - instr = get_instrument(name) + coh_cs: list[float]) -> list[Path]: + instr = copy.copy(get_instrument(name)) + if name == "mfq2": + instr.display = "MFQ-2 survey" dims = instr.dimensions - prof_c, pmass = read_profiles(run_dir, name, dims) - cs = sorted(prof_c) + prof_c, _pmass = read_profiles(run_dir, name, dims) base = prof_c[0.0] - # Coherence gate is RELATIVE and monotone per signed arm: walk outward from c=0 and stop at the - # first coefficient whose allowed-answer mass falls below the requested fraction of base. - coh_cs = coherent_prefix_cs(cs, pmass, coherence_frac) pos_c = max(c for c in coh_cs if c > 0.0) neg_c = min(c for c in coh_cs if c < 0.0) pos = prof_c[pos_c] @@ -151,10 +159,11 @@ def plot_ordinal(run_dir: Path, out: Path, name: str, vec_label: str, C: float, respondents, haze = T.maps.respondent_profiles(dims, instr.scale_max), None else: respondents, haze = None, human_haze(instr) + traj = {c: _frac(prof_c[c], instr.scale_max) for c in coh_cs} figm = T.maps.plot_ipsative_pca(instr, dims, countries, Mfrac, _frac(base, instr.scale_max), _frac(pos, instr.scale_max), _frac(neg, instr.scale_max), respondents=respondents, haze=haze, - labels=labels) + traj=traj, labels=labels) figm.axes[0].set_title(f"{instr.display}: humans vs LLMs steered for {vec_label}", fontsize=10) paths = [T.maps.save_both(figm, out / name, "map_pca_ipsative")] plt.close(figm) @@ -182,29 +191,34 @@ def read_human_mfv() -> tuple[list[str], dict[str, dict[str, float]]]: by_country.setdefault(r["country"], {})[r["foundation"]] = float(r["mean"]) return sorted(by_country), by_country - -def _mfv_zspace(run_dir: Path): - """Shared MFV adapter -> the common coordinate system the map AND range both consume: z-scored - relative-emphasis profiles (model base / +C / -C) + the human MFV culture matrix in the same - space. MFV is nominal (model emits logit(violation) per foundation, humans rate wrongness 1-5), - so absolute scales differ; z-scoring each profile ACROSS foundations compares the PATTERN -- which - foundations a reader weights as more violation-worthy than their own average -- which is exactly - what the steer moves. Social Norms is dropped (no human MFV norm), asserted so a taxonomy change - fails loud. Returns (founds, countries, M_z[countries x founds], base_z, posz, negz).""" - d = json.loads((run_dir / "mfv.json").read_text()) - base_l = d["base_logit_per_foundation"] - pos_dl, neg_dl = d["pos"]["dlogit_per_foundation"], d["neg"]["dlogit_per_foundation"] +def read_mfv_profiles(run_dir: Path) -> tuple[list[str], dict[float, np.ndarray], dict[float, float]]: + rows = list(csv.DictReader((run_dir / "mfv_profiles.csv").open())) + foundation_order = [] + for r in rows: + if r["foundation"] not in foundation_order: + foundation_order.append(r["foundation"]) countries, human = read_human_mfv() hfounds = set(next(iter(human.values()))) - founds = [f for f in d["foundation_order"] if f.lower() in hfounds] # shared, model order - dropped = [f for f in d["foundation_order"] if f.lower() not in hfounds] + founds = [f for f in foundation_order if f.lower() in hfounds] + dropped = [f for f in foundation_order if f.lower() not in hfounds] assert dropped == ["Social Norms"], f"unexpected MFV foundations without a human norm: {dropped}" + by_c: dict[float, dict[str, float]] = {} + pmass: dict[float, float] = {} + for r in rows: + c = float(r["c"]) + by_c.setdefault(c, {})[r["foundation"]] = float(r["mean"]) + pmass[c] = float(r["pmass"]) + prof = {c: _zscore(np.array([vals[f] for f in founds])) for c, vals in by_c.items()} + return founds, prof, pmass + + +def _mfv_zspace(run_dir: Path): + """Shared MFV adapter -> z-scored relative-emphasis profiles + human MFV culture matrix.""" + founds, prof, pmass = read_mfv_profiles(run_dir) + countries, human = read_human_mfv() fl = [f.lower() for f in founds] - base = _zscore(np.array([base_l[f]["mean"] for f in founds])) - posz = _zscore(np.array([base_l[f]["mean"] + pos_dl[f]["mean"] for f in founds])) - negz = _zscore(np.array([base_l[f]["mean"] + neg_dl[f]["mean"] for f in founds])) M = np.array([_zscore(np.array([human[c][f] for f in fl])) for c in countries]) - return founds, countries, M, base, posz, negz + return founds, countries, M, prof, pmass # MFV has no ordinal Instrument (it goes through evaluate_multibool, not administer), but the shared @@ -214,28 +228,30 @@ _MFV_INSTR = SimpleNamespace(name="mfv", display="MFV vignettes") _MFV_YLABEL = "relative emphasis (z across foundations)" -def plot_mfv_map(run_dir: Path, out: Path, vec_label: str, C: float) -> Path: +def plot_mfv_map(run_dir: Path, out: Path, vec_label: str, C: float, coh_cs: list[float]) -> Path: """MFV ipsative culture map via the SAME plot_ipsative_pca the ordinal instruments use, in the z-scored relative-emphasis space (logit-violation and 1-5 wrongness cannot share a raw axis). Red/blue endpoint points show where the steer moves the AI among human cultures.""" - founds, countries, M, base, posz, negz = _mfv_zspace(run_dir) - labels = ("base (c=0)", "c=+1", "c=-1") - fig = T.maps.plot_ipsative_pca(_MFV_INSTR, founds, countries, M, base, posz, negz, labels=labels) + founds, countries, M, prof, _pmass = _mfv_zspace(run_dir) + pos_c = max(c for c in coh_cs if c > 0.0) + neg_c = min(c for c in coh_cs if c < 0.0) + labels = ("base (c=0)", f"c={pos_c:+g}", f"c={neg_c:+g}") + traj = {c: prof[c] for c in coh_cs} + fig = T.maps.plot_ipsative_pca(_MFV_INSTR, founds, countries, M, prof[0.0], prof[pos_c], prof[neg_c], + traj=traj, labels=labels) fig.axes[0].set_title(f"MFV vignettes: humans vs LLMs steered for {vec_label}", fontsize=10) path = T.maps.save_both(fig, out / "mfv", "map_pca_ipsative") plt.close(fig) return path -def plot_mfv_range(run_dir: Path, out: Path, vec_label: str, C: float) -> Path: - """MFV range via the SAME plot_range the ordinal instruments use, in z relative-emphasis space. - Only base/+C/-C (the MFV eval is a 3-point sweep, not a multi-C grid like the ordinal admin).""" - founds, countries, M, base, posz, negz = _mfv_zspace(run_dir) - cs = [-1.0, 0.0, 1.0] - prof = {-1.0: negz, 0.0: base, 1.0: posz} +def plot_mfv_range(run_dir: Path, out: Path, vec_label: str, C: float, coh_cs: list[float]) -> Path: + """MFV range via the SAME plot_range the ordinal instruments use, in z relative-emphasis space.""" + founds, countries, M, prof, _pmass = _mfv_zspace(run_dir) humans = {f: sorted(((countries[ci], float(M[ci, fi])) for ci in range(len(countries))), key=lambda t: t[1]) for fi, f in enumerate(founds)} - fig = T.maps.plot_range(_MFV_INSTR, founds, cs, prof, humans, None, vec_label, ylabel=_MFV_YLABEL) + fig = T.maps.plot_range(_MFV_INSTR, founds, coh_cs, {c: prof[c] for c in coh_cs}, + humans, None, vec_label, ylabel=_MFV_YLABEL) path = T.maps.save_both(fig, out / "mfv", "range") plt.close(fig) return path @@ -257,13 +273,19 @@ def main() -> None: args.out.mkdir(parents=True, exist_ok=True) written: list[str] = [] - for name in ORDINAL: - if (args.run_dir / f"{name}_profiles.csv").exists(): - written += [str(p) for p in plot_ordinal(args.run_dir, args.out, name, vec_label, C, - args.coherence_frac)] - if (args.run_dir / "mfv.json").exists(): - written.append(str(plot_mfv_map(args.run_dir, args.out, vec_label, C))) # shared ipsative map (z-space) - written.append(str(plot_mfv_range(args.run_dir, args.out, vec_label, C))) # shared range (z-space) + ordinal_names = [name for name in ORDINAL if (args.run_dir / f"{name}_profiles.csv").exists()] + pmass_ratio = shared_pmass_ratio(args.run_dir, ordinal_names) + if (args.run_dir / "mfv_profiles.csv").exists(): + _founds, _prof, mfv_pmass = read_mfv_profiles(args.run_dir) + for c, pm in mfv_pmass.items(): + pmass_ratio[c] = min(pmass_ratio[c], pm / mfv_pmass[0.0]) + coh_cs = coherent_prefix_cs(sorted(pmass_ratio), pmass_ratio, args.coherence_frac) + print(f"shared coherent c values at {args.coherence_frac:.2%} base answer mass: {coh_cs}") + for name in ordinal_names: + written += [str(p) for p in plot_ordinal(args.run_dir, args.out, name, vec_label, C, coh_cs)] + if (args.run_dir / "mfv_profiles.csv").exists(): + written.append(str(plot_mfv_map(args.run_dir, args.out, vec_label, C, coh_cs))) # shared ipsative map (z-space) + written.append(str(plot_mfv_range(args.run_dir, args.out, vec_label, C, coh_cs))) # shared range (z-space) print(f"wrote {len(written)} figures under {args.out}:") for w in written: print(" ", w) diff --git a/src/tinymfv/administer.py b/src/tinymfv/administer.py index 6461e38..1be6967 100644 --- a/src/tinymfv/administer.py +++ b/src/tinymfv/administer.py @@ -67,7 +67,8 @@ class AdministerResult(TypedDict): def administer(model, tok, instr: Instrument, *, batch_size: int = 36, - max_think_tokens: int = 64) -> AdministerResult: + max_think_tokens: int = 64, n_samples: int = 1, + temperature: float = 0.0, top_p: float = 1.0) -> AdministerResult: assert instr.kind == "ordinal", "administer() is the ordinal survey readout; use evaluate() for nominal MFV" # Every ordinal item must carry its frame-specific response-scale legend in meta['task']; without # it build_user_content would silently emit a bare statement (no legend) and the profile would be @@ -78,7 +79,9 @@ def administer(model, tok, instr: Instrument, *, batch_size: int = 36, # slot, so an activation steer accrues over the trace before being read. Floor is 1 (the shared # rollout core's HF generate() rejects max_new_tokens=0). per_row = read_items(model, tok, instr, instr.items, answer_ids, - max_think_tokens=max_think_tokens, batch_size=batch_size, verbose_first=True) + max_think_tokens=max_think_tokens, batch_size=batch_size, + n_samples=n_samples, temperature=temperature, top_p=top_p, + verbose_first=True) items = per_item_categorical(per_row, instr.kind) # {id: {p, pmass, dimension, sign, ...}} M = instr.scale_max diff --git a/src/tinymfv/maps.py b/src/tinymfv/maps.py index de25bfd..64f6683 100644 --- a/src/tinymfv/maps.py +++ b/src/tinymfv/maps.py @@ -190,9 +190,9 @@ def plot_ipsative_pca(instr: Instrument, dims: list[str], countries: list[str], for the crop -- separate from the fit so instruments with only society-level mean+sd (big5/16pf/ humor: a marginal resample) get a backdrop without that resample dictating the axes. mfq2 passes real `respondents` (also used as the haze when `haze` is None). With neither, fit on M, pad-crop, - no backdrop. `traj` (signed c-multiplier -> length-K fraction vector) draws the full steer SWEEP - as a connected path through PC space, so a multi-C run shows where the steer leaves the human - cloud and curves into incoherence (the base/pos/neg arrows stay as the headline +-C anchors). + no backdrop. `traj` (signed c-multiplier -> length-K fraction vector) draws the coherent steer + path through PC space. Public README plots pass only the coherent prefix; incoherent c values + are omitted. `traj_incoherent` is the subset of those c whose admin pmass fell below the coherence floor -- drawn hollow. `boots` optionally maps 'base'/'honest'/'dis' -> (n x K) bootstrap matrices. Returns the Figure.""" @@ -238,33 +238,44 @@ def plot_ipsative_pca(instr: Instrument, dims: list[str], countries: list[str], ax.errorbar(pt[0], pt[1], xerr=e1, yerr=e2, fmt="none", ecolor=col, elinewidth=0.7, alpha=0.55, capsize=2.5, capthick=0.8, zorder=4) base_lab, pos_lab, neg_lab = labels - for pt, col, lab, dxy, ha in [(ph, C_HON, pos_lab, (9, 9), "left"), - (pf, C_DIS, neg_lab, (-9, -1), "right"), - (pb, C_BASE, base_lab, (9, -13), "left")]: - if pt is None: - continue - ax.scatter(*pt, s=120, c=col, marker="o", edgecolors="white", linewidths=1.2, zorder=7) - ax.annotate(lab, pt, xytext=dxy, textcoords="offset points", fontsize=9, color=col, - fontweight="bold", ha=ha, va="center", zorder=8) + if pb is not None: + ax.scatter(*pb, s=72, c=C_BASE, marker="o", edgecolors="white", linewidths=1.0, zorder=7) + ax.annotate(base_lab, pb, xytext=(9, -13), textcoords="offset points", fontsize=9, + color=C_BASE, fontweight="bold", ha="left", va="center", zorder=8) traj_pts = None if traj: inco = traj_incoherent or set() cs_sorted = sorted(traj) traj_pts = np.array([proj(traj[c]) for c in cs_sorted]) - # two arms fanning from base (c=0): +c red, -c blue. Marker grows with |c|; a point whose - # admin pmass fell below the coherence floor is hollow (the steer is no longer measuring). + # Two arms from base (c=0): +c red, -c blue. Marker size is constant so path length, not ink + # area, carries the coefficient change. for lo, hi in [(0.0, max(cs_sorted)), (min(cs_sorted), 0.0)]: arm = [(c, proj(traj[c])) for c in cs_sorted if lo <= c <= hi] if len(arm) < 2: continue xy = np.array([p for _, p in arm]) - ax.plot(xy[:, 0], xy[:, 1], "-", color="0.55", lw=0.9, zorder=4, alpha=0.8) + col = POS_COL if hi > 0 else NEG_COL + ax.plot(xy[:, 0], xy[:, 1], "-", color=col, lw=1.1, zorder=4, alpha=0.75) + c_end = max((c for c, _p in arm), key=abs) for c, p in arm: if c == 0: continue - col = POS_COL if c > 0 else NEG_COL - ax.scatter(p[0], p[1], s=34, c="none" if c in inco else col, - edgecolors=col, linewidths=1.0, zorder=6) + fill = col if c == c_end and c not in inco else "none" + ax.scatter(p[0], p[1], s=42, c=fill, edgecolors=col, + linewidths=1.25, zorder=6) + if c == c_end: + lab = pos_lab if c > 0 else neg_lab + dxy, ha = ((9, 9), "left") if c > 0 else ((-9, -1), "right") + ax.annotate(lab, p, xytext=dxy, textcoords="offset points", fontsize=9, + color=col, fontweight="bold", ha=ha, va="center", zorder=8) + else: + for pt, col, lab, dxy, ha in [(ph, C_HON, pos_lab, (9, 9), "left"), + (pf, C_DIS, neg_lab, (-9, -1), "right")]: + if pt is None: + continue + ax.scatter(*pt, s=42, c=col, marker="o", edgecolors="white", linewidths=1.0, zorder=7) + ax.annotate(lab, pt, xytext=dxy, textcoords="offset points", fontsize=9, color=col, + fontweight="bold", ha=ha, va="center", zorder=8) # Crop to the SOCIETIES + steer anchors for EVERY instrument (the human cloud is far wider and would # bury them in a central blob; it stays a clipped backdrop). Then PAD THE BOTTOM to reserve a clean # strip for the legend insets -- deterministic placement, identical on every plot, no overlap with @@ -414,12 +425,11 @@ def plot_splom(instr: Instrument, dims: list[str], cloud: np.ndarray, M: np.ndar # --- per-vector steer range --------------------------------------------------------------------- -def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, base_y: float, - lw: float = 2.0, dx: float = 0.12, tick: float = 0.038) -> None: - """Draw one coherent AI c-path.""" +def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, lw: float = 2.0, + lane_dx: float = 0.14) -> None: + """Draw one coherent AI c-path. X is only the sign lane: negative, base, positive.""" assert list(cs) == sorted(cs) and 0.0 in cs, f"draw_steer needs sorted cs with c=0 (yv[-1]=+pole, yv[0]=-pole), got {cs}" - cmax = max(abs(c) for c in cs) or 1.0 - xs_by_c = {c: xs if c == 0.0 else xs + dx * np.sign(c) * np.sqrt(abs(c) / cmax) for c in cs} + xs_by_c = {c: xs + lane_dx * np.sign(c) for c in cs} for side_cs, col in [([c for c in cs if c <= 0.0], NEG_COL), ([c for c in cs if c >= 0.0], POS_COL)]: if len(side_cs) < 2: continue @@ -428,11 +438,11 @@ def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, base_y: float, ax.plot(x_path, y_path, color=col, lw=lw, alpha=0.9, zorder=6, solid_capstyle="round") for c, y in zip(cs, yv): if c == 0.0: - col, t = "black", tick * 1.25 + col = "black" else: - col, t = (POS_COL if c > 0 else NEG_COL), tick + col = POS_COL if c > 0 else NEG_COL x = xs_by_c[c] - ax.plot([x - t, x + t], [float(y), float(y)], color=col, lw=lw, zorder=8) + ax.scatter(x, float(y), s=18, color=col, edgecolors="white", linewidths=0.45, zorder=8) def draw_range_panel(ax, instr: Instrument, dims: list[str], cs: list[float], prof: dict, @@ -464,8 +474,7 @@ def draw_range_panel(ax, instr: Instrument, dims: list[str], cs: list[float], pr xs = gx + DX_STEER yv = np.array([prof[c][i] for c in cs]) ys += yv.tolist() - base_y = float(yv[list(cs).index(0.0)]) - draw_steer(ax, xs, cs, yv, base_y) + draw_steer(ax, xs, cs, yv) if i == label_i: # Only the two pole labels, to the RIGHT of the steer column. No 'base' tag: the black dot # between the two coloured arms is self-evidently the unsteered model, and on a near-collapsed @@ -534,8 +543,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d soc_vals = np.array([m for _, m in soc]) q1, q3 = np.percentile(soc_vals, [25, 75]) # nanmin/nanmax: a collapsed pole reads NaN (read.py NaN-at-collapse, "do not compare"). The - # base (c=0) is always finite, so the axis still frames the un-collapsed cells; draw_steer - # skips the NaN arm on its own (the abs(NaN-base) test is False). + # base (c=0) is always finite, so the axis still frames the un-collapsed cells. lo, hi = min(np.nanmin(yv), q1), max(np.nanmax(yv), q3) m = max(0.10, 0.30 * (hi - lo)) ylo, yhi = lo - m, hi + m @@ -550,7 +558,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d xs = 0.30 base_y = float(yv[list(cs).index(0.0)]) - draw_steer(ax, xs, cs, yv, base_y, lw=2.4, dx=0.10, tick=0.055) + draw_steer(ax, xs, cs, yv, lw=2.4) named = {} if near: name_xy = {nm: (float(x), v) for (nm, v), x in zip(near, soc_x)} @@ -558,7 +566,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d named[min(near, key=lambda t: abs(t[1] - ref))[0]] = ref tx = [xs] * len(cs) + [name_xy[nm][0] for nm in named] if near else [xs] * len(cs) ty = list(map(float, yv)) + [name_xy[nm][1] for nm in named] if near else list(map(float, yv)) - txt = [("c=0" if c == 0 else f"c={int(c):+d}") for c in cs] + list(named) + txt = [("c=0" if c == 0 else f"c={c:+g}") for c in cs] + list(named) dot_x = [xs] * len(cs) + (list(soc_x) if near else []) dot_y = list(map(float, yv)) + ([v for _, v in near] if near else []) placed = False diff --git a/src/tinymfv/read.py b/src/tinymfv/read.py index 5d863b2..20f3d65 100644 --- a/src/tinymfv/read.py +++ b/src/tinymfv/read.py @@ -56,7 +56,9 @@ def build_user_content(instr: Instrument, item: InstrItem) -> str: @torch.no_grad() def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids: list[int], - *, max_think_tokens: int, batch_size: int = 36, verbose_first: bool = False) -> list[dict]: + *, max_think_tokens: int, batch_size: int = 36, n_samples: int = 1, + temperature: float = 0.0, top_p: float = 1.0, + verbose_first: bool = False) -> list[dict]: """Score a list of InstrItems (one frame's worth, or any subset). Returns per-item rows with the keys `per_item_categorical` consumes: id, frame, p, pmass_allowed, dimension, sign, human_label. @@ -83,31 +85,35 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids # ordinal frames are already separate InstrItems, so single-pass (no reversed-enum two-pass; # frame debias is downstream in canonicalize_to_forward). force_only: the "(" prefill is too # short for natural-emission detection (matches by chance in the think trace), so always read - # the forced answer slot. n_samples=1, temperature=0 -> deterministic. + # the forced answer slot. n_samples>1 samples independent think traces, then averages the + # answer-token probabilities below. thinks, slots = _rollout_natural_or_forced( model, tok, user_prompts, schema_hint="", max_think_tokens=max_think_tokens, scoring_slots=[("Just answer", instr.prefill)], gather_token_ids=answer_ids, - n_samples=1, temperature=0.0, force_only=True, + n_samples=n_samples, temperature=temperature, top_p=top_p, force_only=True, verbose=verbose_first and i == 0, ) for j, it in enumerate(chunk): - slot = slots[j][0] + sample_idx = [j * n_samples + n for n in range(n_samples)] + sample_slots = [slots[k][0] for k in sample_idx] # lp = lp_gather: the full-vocab log_softmax logprob of each answer token at the answer # slot. This is the RAW PRIMITIVE -- every readout (E, the logit contrast C, log-odds, # entropy) is a pure function of it, and a steer effect is just a difference of lp. Keep # it; do not throw it away by collapsing to a single number here. - lp = np.asarray(slot["lp_gather"], dtype=float) # [A] raw logprobs (full-vocab norm) + sample_lp = np.asarray([s["lp_gather"] for s in sample_slots], dtype=float) # [N,A] + lp = np.log(np.nanmean(np.exp(sample_lp), axis=0)) # [A] BMA over sampled thoughts p_a = np.exp(lp) # [A] prob on each answer token - pmass = float(slot["pmass_allowed"]) # mass on allowed tokens (coherence) + pmass = float(np.nansum(p_a)) # mass on allowed tokens (coherence) # Renormalize within allowed. INTENTIONALLY NOT NaN-guarded: at full coherence collapse # pmass -> 0 so p_norm -> NaN and poisons that item's factor. That is the honest signal, a # distribution renormalized from ~zero mass is NOT comparable to one from real mass (the mean # of 10 != the mean of 130), so it must not be silently turned into a comparable-looking # number. NaN marks "do not compare". Do not "fix" this with a softmax/eps fallback. p_norm = p_a / p_a.sum() # [A] within allowed (NaN at collapse, by design) - think_text, n_think, emitted_close = thinks[j] + sample_thinks = [thinks[k] for k in sample_idx] + think_text, n_think, emitted_close = sample_thinks[0] out.append({ "id": it.id, "frame": it.frame, "lp": lp, # raw logprobs at the M scale tokens @@ -115,7 +121,8 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids "pmass_allowed": pmass, "dimension": it.dimension, "sign": it.sign, "human_label": it.human_label, - "think": think_text, "n_think": n_think, "emitted_close": emitted_close, + "think": think_text, "n_think": n_think, + "emitted_close": any(t[2] for t in sample_thinks), }) if verbose_first and i == 0: slot0 = slots[0][0]