Support sampled survey reads and MFV c-grid plots

This commit is contained in:
wassname
2026-06-30 13:28:20 +08:00
parent d8941f47ac
commit 6fcdfea30f
5 changed files with 180 additions and 92 deletions
+48
View File
@@ -0,0 +1,48 @@
# README plot and metric simplification
## Goal
Make the README readable to a new researcher in one pass. It should say what tinymfv is, what datasets it includes, what single measurement to look at first, and how the plots encode base/+C/-C steering.
## Scope
In: README wording, metric/API simplification, regenerated showcase plots, cold-reader panel review.
Out: changing eval semantics, changing dataset schemas, adding new metrics.
## Requirements
- R1: README has one headline measurement: `profile`. Done means: the Metrics section defines the profile once and treats logprobs/format checks as API details, not competing headline metrics. VERIFY: `rg -n '^###|profile_C|informedness|nll_prefill|pmass_allowed|dlogit_per_foundation' README.md`.
- R2: README explains the plot encoding. Done means: a reader can tell what gray, black, red, and blue marks mean without knowing steering-lite internals. VERIFY: README contains the color/key sentence and no stale "trajectory" caption for the clean maps.
- R3: README still lists all datasets and all image links resolve. VERIFY: a script parses markdown image links and prints `missing_images: []`.
- R4: Cold readers can reconstruct what tinymfv is, which datasets exist, what measurement matters, and why a researcher would use it. VERIFY: external-review-v2 panel JSON files in `docs/reviews/` plus a short triage table in this spec.
- R5: README survey maps show the coherent steering path, not just two endpoints. Done means: MFQ-2, Big Five, and Humor maps draw base -> tested coherent c values, with constant-size c markers and no incoherent lone dots. VERIFY: inspect regenerated PNGs and check the plotter passes `traj=` to `plot_ipsative_pca`.
- R6: README range plots show the coherent coefficient path in sign lanes. Done means: all negative c values share a blue lane, base is black in the middle, and all positive c values share a red lane. Reversals/wraps are visible on the y-axis without marker-size encoding. VERIFY: inspect regenerated range PNGs and check `draw_steer` maps x by sign only.
## Tasks
- [x] T1 (R1, R2): Simplify README around `profile` and update plot captions.
- verify: `rg -n '^###|profile_C|informedness|nll_prefill|pmass_allowed|dlogit_per_foundation|trajectory' README.md`
- success: one `### The profile` subsection; no stale plot "trajectory" language.
- likely_fail: API section still presents six metrics as peers.
- sneaky_fail: profile is too vague to connect to MFV vs survey data; caught by panel mechanism answers.
- [/] T2 (R2, R3): Keep regenerated showcase plots and verify image refs.
- verify: parse README image links and check paths exist.
- success: `missing_images: []`.
- likely_fail: stale filenames remain.
- sneaky_fail: images exist but show old sweep encoding; caught by fresh image inspection of MFQ-2 map/range.
- [ ] T3 (R4): Run external-review-v2 comprehension panel and triage.
- verify: panel outputs valid JSON and summaries answer the probe.
- success: most panel members correctly identify tinymfv, datasets, `profile`, and researcher use.
- likely_fail: models still name `profile_C` or `logprobs` as the main metric.
- sneaky_fail: models parrot phrases without explaining why a researcher would use it; caught by the "own words with inference" probe.
- [/] T4 (R5, R6): Restore coherent c paths while removing misleading geometry.
- steps: compute one shared answer-mass gate across survey evals; pass the shared coherent path to map plots; make range steer paths use sign lanes.
- verify: `rg -n "traj=|np.sign\\(c\\)|s=42" scripts/plot_steer_showcase.py src/tinymfv/maps.py`
- success: maps show red/blue coherent paths through c values; ranges show AI sign-lane paths.
- likely_fail: maps still show only endpoint dots.
- sneaky_fail: marker size encodes c, or x offset varies by c magnitude; caught by the source check and image inspection.
- UAT: when opening `docs/img/showcase/mfq2/map_pca_ipsative.png` and `docs/img/showcase/mfq2/range.png`, the path is visible and the range path can show reversals/wraps.
## Log
- 2026-06-30: The README previously exposed many metric names at once. That made the API complete but caused cold-reader confusion about which measurement to start with.
- 2026-06-30: Final plots are gated on steering-lite task 394, output `outputs/20260630_dignity_authority_strict22_local_sspace_allinstr`. Stale README images from the earlier fairness / two-axis run must not count as UAT.
- 2026-06-30: Coherence gating must be a signed prefix from `c=0`, stopping a side at the first coefficient below the base-relative answer-mass floor. A per-row filter can produce confusing lone high-C dots after collapse.
- 2026-06-30: README structural check before final regeneration: 9 images, `missing_images: []`, 15 local links, `missing_local_links: []`, and no 16PF map link.
- 2026-06-30: The endpoint-only map removed the path readers need. The sign-offset range geometry also made a one-dimensional value plot look like a fake two-dimensional fork.
- 2026-06-30: User wants range plots to show the path as sign lanes, not coefficient-x swim lines. The fix is negative/base/positive lanes, with vertical paths inside each lane.
+73 -51
View File
@@ -4,7 +4,7 @@ Consumes a steering-lite `run_allinstr_showcase.py` output dir (one calibrated
activation-steering vector administered across every instrument over a signed
c-sweep) and renders the SAME two figures for every instrument, uniformly:
- map : ipsative culture map (PCA), AI base + strongest coherent +/-c vs the human cloud.
- map : ipsative culture map (PCA), AI coherent +/-c path vs the human cloud.
- range: per-factor range, AI base + coherent +/-c path vs the human society strip.
Ordinal instruments (mfq2/big5/16pf/humor_styles) read <name>_profiles.csv; nominal
@@ -13,9 +13,8 @@ logit-violation units cannot share a raw axis with 1-5 wrongness), but it goes
through the same plot_ipsative_pca / plot_range and yields the same two figures.
cs are SIGNED multipliers of the calibrated coefficient C (0 = base). The public
README range plots show the coherent path: c=0 plus each +/-c row whose pmass stays
above the requested fraction of base. Maps show only the strongest coherent endpoints.
Incoherent rows are dropped, not drawn hollow.
README plots show the coherent path: c=0 plus each +/-c row whose tinymfv answer mass
stays above the requested fraction of base. Incoherent rows are dropped, not drawn hollow.
uv run python scripts/plot_steer_showcase.py \
--run-dir ../steering-lite/outputs/allinstr_qwen35_4b --out docs/img/showcase
@@ -23,6 +22,7 @@ Incoherent rows are dropped, not drawn hollow.
from __future__ import annotations
import argparse
import copy
import csv
import json
from pathlib import Path
@@ -112,28 +112,36 @@ def read_profiles(run_dir: Path, name: str, dims: list[str], value_col: str = "m
return {c: np.array([d[f] for f in dims]) for c, d in by_c.items()}, pmass
def coherent_prefix_cs(cs: list[float], pmass: dict[float, float], coherence_frac: float) -> list[float]:
def coherent_prefix_cs(cs: list[float], pmass_ratio: dict[float, float], coherence_frac: float) -> list[float]:
"""c=0 plus each signed arm until answer mass first falls below the base-relative floor."""
base_pm = pmass[0.0]
kept = [0.0]
for side in (1.0, -1.0):
for c in sorted([c for c in cs if np.sign(c) == side], key=abs):
if pmass[c] <= coherence_frac * base_pm:
if pmass_ratio[c] <= coherence_frac:
break
kept.append(c)
return sorted(kept)
def shared_pmass_ratio(run_dir: Path, names: list[str]) -> dict[float, float]:
"""Worst base-relative answer mass across the survey evals, keyed by signed calibrated multiplier."""
pmasses: list[dict[float, float]] = []
for name in names:
instr = get_instrument(name)
_, pmass = read_profiles(run_dir, name, instr.dimensions)
pmasses.append(pmass)
cs = sorted(set.intersection(*(set(p) for p in pmasses)))
return {c: min(p[c] / p[0.0] for p in pmasses) for c in cs}
def plot_ordinal(run_dir: Path, out: Path, name: str, vec_label: str, C: float,
coherence_frac: float) -> list[Path]:
instr = get_instrument(name)
coh_cs: list[float]) -> list[Path]:
instr = copy.copy(get_instrument(name))
if name == "mfq2":
instr.display = "MFQ-2 survey"
dims = instr.dimensions
prof_c, pmass = read_profiles(run_dir, name, dims)
cs = sorted(prof_c)
prof_c, _pmass = read_profiles(run_dir, name, dims)
base = prof_c[0.0]
# Coherence gate is RELATIVE and monotone per signed arm: walk outward from c=0 and stop at the
# first coefficient whose allowed-answer mass falls below the requested fraction of base.
coh_cs = coherent_prefix_cs(cs, pmass, coherence_frac)
pos_c = max(c for c in coh_cs if c > 0.0)
neg_c = min(c for c in coh_cs if c < 0.0)
pos = prof_c[pos_c]
@@ -151,10 +159,11 @@ def plot_ordinal(run_dir: Path, out: Path, name: str, vec_label: str, C: float,
respondents, haze = T.maps.respondent_profiles(dims, instr.scale_max), None
else:
respondents, haze = None, human_haze(instr)
traj = {c: _frac(prof_c[c], instr.scale_max) for c in coh_cs}
figm = T.maps.plot_ipsative_pca(instr, dims, countries, Mfrac,
_frac(base, instr.scale_max), _frac(pos, instr.scale_max),
_frac(neg, instr.scale_max), respondents=respondents, haze=haze,
labels=labels)
traj=traj, labels=labels)
figm.axes[0].set_title(f"{instr.display}: humans vs LLMs steered for {vec_label}", fontsize=10)
paths = [T.maps.save_both(figm, out / name, "map_pca_ipsative")]
plt.close(figm)
@@ -182,29 +191,34 @@ def read_human_mfv() -> tuple[list[str], dict[str, dict[str, float]]]:
by_country.setdefault(r["country"], {})[r["foundation"]] = float(r["mean"])
return sorted(by_country), by_country
def _mfv_zspace(run_dir: Path):
"""Shared MFV adapter -> the common coordinate system the map AND range both consume: z-scored
relative-emphasis profiles (model base / +C / -C) + the human MFV culture matrix in the same
space. MFV is nominal (model emits logit(violation) per foundation, humans rate wrongness 1-5),
so absolute scales differ; z-scoring each profile ACROSS foundations compares the PATTERN -- which
foundations a reader weights as more violation-worthy than their own average -- which is exactly
what the steer moves. Social Norms is dropped (no human MFV norm), asserted so a taxonomy change
fails loud. Returns (founds, countries, M_z[countries x founds], base_z, posz, negz)."""
d = json.loads((run_dir / "mfv.json").read_text())
base_l = d["base_logit_per_foundation"]
pos_dl, neg_dl = d["pos"]["dlogit_per_foundation"], d["neg"]["dlogit_per_foundation"]
def read_mfv_profiles(run_dir: Path) -> tuple[list[str], dict[float, np.ndarray], dict[float, float]]:
rows = list(csv.DictReader((run_dir / "mfv_profiles.csv").open()))
foundation_order = []
for r in rows:
if r["foundation"] not in foundation_order:
foundation_order.append(r["foundation"])
countries, human = read_human_mfv()
hfounds = set(next(iter(human.values())))
founds = [f for f in d["foundation_order"] if f.lower() in hfounds] # shared, model order
dropped = [f for f in d["foundation_order"] if f.lower() not in hfounds]
founds = [f for f in foundation_order if f.lower() in hfounds]
dropped = [f for f in foundation_order if f.lower() not in hfounds]
assert dropped == ["Social Norms"], f"unexpected MFV foundations without a human norm: {dropped}"
by_c: dict[float, dict[str, float]] = {}
pmass: dict[float, float] = {}
for r in rows:
c = float(r["c"])
by_c.setdefault(c, {})[r["foundation"]] = float(r["mean"])
pmass[c] = float(r["pmass"])
prof = {c: _zscore(np.array([vals[f] for f in founds])) for c, vals in by_c.items()}
return founds, prof, pmass
def _mfv_zspace(run_dir: Path):
"""Shared MFV adapter -> z-scored relative-emphasis profiles + human MFV culture matrix."""
founds, prof, pmass = read_mfv_profiles(run_dir)
countries, human = read_human_mfv()
fl = [f.lower() for f in founds]
base = _zscore(np.array([base_l[f]["mean"] for f in founds]))
posz = _zscore(np.array([base_l[f]["mean"] + pos_dl[f]["mean"] for f in founds]))
negz = _zscore(np.array([base_l[f]["mean"] + neg_dl[f]["mean"] for f in founds]))
M = np.array([_zscore(np.array([human[c][f] for f in fl])) for c in countries])
return founds, countries, M, base, posz, negz
return founds, countries, M, prof, pmass
# MFV has no ordinal Instrument (it goes through evaluate_multibool, not administer), but the shared
@@ -214,28 +228,30 @@ _MFV_INSTR = SimpleNamespace(name="mfv", display="MFV vignettes")
_MFV_YLABEL = "relative emphasis (z across foundations)"
def plot_mfv_map(run_dir: Path, out: Path, vec_label: str, C: float) -> Path:
def plot_mfv_map(run_dir: Path, out: Path, vec_label: str, C: float, coh_cs: list[float]) -> Path:
"""MFV ipsative culture map via the SAME plot_ipsative_pca the ordinal instruments use, in the
z-scored relative-emphasis space (logit-violation and 1-5 wrongness cannot share a raw axis).
Red/blue endpoint points show where the steer moves the AI among human cultures."""
founds, countries, M, base, posz, negz = _mfv_zspace(run_dir)
labels = ("base (c=0)", "c=+1", "c=-1")
fig = T.maps.plot_ipsative_pca(_MFV_INSTR, founds, countries, M, base, posz, negz, labels=labels)
founds, countries, M, prof, _pmass = _mfv_zspace(run_dir)
pos_c = max(c for c in coh_cs if c > 0.0)
neg_c = min(c for c in coh_cs if c < 0.0)
labels = ("base (c=0)", f"c={pos_c:+g}", f"c={neg_c:+g}")
traj = {c: prof[c] for c in coh_cs}
fig = T.maps.plot_ipsative_pca(_MFV_INSTR, founds, countries, M, prof[0.0], prof[pos_c], prof[neg_c],
traj=traj, labels=labels)
fig.axes[0].set_title(f"MFV vignettes: humans vs LLMs steered for {vec_label}", fontsize=10)
path = T.maps.save_both(fig, out / "mfv", "map_pca_ipsative")
plt.close(fig)
return path
def plot_mfv_range(run_dir: Path, out: Path, vec_label: str, C: float) -> Path:
"""MFV range via the SAME plot_range the ordinal instruments use, in z relative-emphasis space.
Only base/+C/-C (the MFV eval is a 3-point sweep, not a multi-C grid like the ordinal admin)."""
founds, countries, M, base, posz, negz = _mfv_zspace(run_dir)
cs = [-1.0, 0.0, 1.0]
prof = {-1.0: negz, 0.0: base, 1.0: posz}
def plot_mfv_range(run_dir: Path, out: Path, vec_label: str, C: float, coh_cs: list[float]) -> Path:
"""MFV range via the SAME plot_range the ordinal instruments use, in z relative-emphasis space."""
founds, countries, M, prof, _pmass = _mfv_zspace(run_dir)
humans = {f: sorted(((countries[ci], float(M[ci, fi])) for ci in range(len(countries))), key=lambda t: t[1])
for fi, f in enumerate(founds)}
fig = T.maps.plot_range(_MFV_INSTR, founds, cs, prof, humans, None, vec_label, ylabel=_MFV_YLABEL)
fig = T.maps.plot_range(_MFV_INSTR, founds, coh_cs, {c: prof[c] for c in coh_cs},
humans, None, vec_label, ylabel=_MFV_YLABEL)
path = T.maps.save_both(fig, out / "mfv", "range")
plt.close(fig)
return path
@@ -257,13 +273,19 @@ def main() -> None:
args.out.mkdir(parents=True, exist_ok=True)
written: list[str] = []
for name in ORDINAL:
if (args.run_dir / f"{name}_profiles.csv").exists():
written += [str(p) for p in plot_ordinal(args.run_dir, args.out, name, vec_label, C,
args.coherence_frac)]
if (args.run_dir / "mfv.json").exists():
written.append(str(plot_mfv_map(args.run_dir, args.out, vec_label, C))) # shared ipsative map (z-space)
written.append(str(plot_mfv_range(args.run_dir, args.out, vec_label, C))) # shared range (z-space)
ordinal_names = [name for name in ORDINAL if (args.run_dir / f"{name}_profiles.csv").exists()]
pmass_ratio = shared_pmass_ratio(args.run_dir, ordinal_names)
if (args.run_dir / "mfv_profiles.csv").exists():
_founds, _prof, mfv_pmass = read_mfv_profiles(args.run_dir)
for c, pm in mfv_pmass.items():
pmass_ratio[c] = min(pmass_ratio[c], pm / mfv_pmass[0.0])
coh_cs = coherent_prefix_cs(sorted(pmass_ratio), pmass_ratio, args.coherence_frac)
print(f"shared coherent c values at {args.coherence_frac:.2%} base answer mass: {coh_cs}")
for name in ordinal_names:
written += [str(p) for p in plot_ordinal(args.run_dir, args.out, name, vec_label, C, coh_cs)]
if (args.run_dir / "mfv_profiles.csv").exists():
written.append(str(plot_mfv_map(args.run_dir, args.out, vec_label, C, coh_cs))) # shared ipsative map (z-space)
written.append(str(plot_mfv_range(args.run_dir, args.out, vec_label, C, coh_cs))) # shared range (z-space)
print(f"wrote {len(written)} figures under {args.out}:")
for w in written:
print(" ", w)
+5 -2
View File
@@ -67,7 +67,8 @@ class AdministerResult(TypedDict):
def administer(model, tok, instr: Instrument, *, batch_size: int = 36,
max_think_tokens: int = 64) -> AdministerResult:
max_think_tokens: int = 64, n_samples: int = 1,
temperature: float = 0.0, top_p: float = 1.0) -> AdministerResult:
assert instr.kind == "ordinal", "administer() is the ordinal survey readout; use evaluate() for nominal MFV"
# Every ordinal item must carry its frame-specific response-scale legend in meta['task']; without
# it build_user_content would silently emit a bare statement (no legend) and the profile would be
@@ -78,7 +79,9 @@ def administer(model, tok, instr: Instrument, *, batch_size: int = 36,
# slot, so an activation steer accrues over the trace before being read. Floor is 1 (the shared
# rollout core's HF generate() rejects max_new_tokens=0).
per_row = read_items(model, tok, instr, instr.items, answer_ids,
max_think_tokens=max_think_tokens, batch_size=batch_size, verbose_first=True)
max_think_tokens=max_think_tokens, batch_size=batch_size,
n_samples=n_samples, temperature=temperature, top_p=top_p,
verbose_first=True)
items = per_item_categorical(per_row, instr.kind) # {id: {p, pmass, dimension, sign, ...}}
M = instr.scale_max
+39 -31
View File
@@ -190,9 +190,9 @@ def plot_ipsative_pca(instr: Instrument, dims: list[str], countries: list[str],
for the crop -- separate from the fit so instruments with only society-level mean+sd (big5/16pf/
humor: a marginal resample) get a backdrop without that resample dictating the axes. mfq2 passes
real `respondents` (also used as the haze when `haze` is None). With neither, fit on M, pad-crop,
no backdrop. `traj` (signed c-multiplier -> length-K fraction vector) draws the full steer SWEEP
as a connected path through PC space, so a multi-C run shows where the steer leaves the human
cloud and curves into incoherence (the base/pos/neg arrows stay as the headline +-C anchors).
no backdrop. `traj` (signed c-multiplier -> length-K fraction vector) draws the coherent steer
path through PC space. Public README plots pass only the coherent prefix; incoherent c values
are omitted.
`traj_incoherent` is the subset of those c whose admin pmass fell below the coherence floor --
drawn hollow. `boots` optionally maps 'base'/'honest'/'dis' -> (n x K) bootstrap matrices. Returns
the Figure."""
@@ -238,33 +238,44 @@ def plot_ipsative_pca(instr: Instrument, dims: list[str], countries: list[str],
ax.errorbar(pt[0], pt[1], xerr=e1, yerr=e2, fmt="none", ecolor=col,
elinewidth=0.7, alpha=0.55, capsize=2.5, capthick=0.8, zorder=4)
base_lab, pos_lab, neg_lab = labels
for pt, col, lab, dxy, ha in [(ph, C_HON, pos_lab, (9, 9), "left"),
(pf, C_DIS, neg_lab, (-9, -1), "right"),
(pb, C_BASE, base_lab, (9, -13), "left")]:
if pt is None:
continue
ax.scatter(*pt, s=120, c=col, marker="o", edgecolors="white", linewidths=1.2, zorder=7)
ax.annotate(lab, pt, xytext=dxy, textcoords="offset points", fontsize=9, color=col,
fontweight="bold", ha=ha, va="center", zorder=8)
if pb is not None:
ax.scatter(*pb, s=72, c=C_BASE, marker="o", edgecolors="white", linewidths=1.0, zorder=7)
ax.annotate(base_lab, pb, xytext=(9, -13), textcoords="offset points", fontsize=9,
color=C_BASE, fontweight="bold", ha="left", va="center", zorder=8)
traj_pts = None
if traj:
inco = traj_incoherent or set()
cs_sorted = sorted(traj)
traj_pts = np.array([proj(traj[c]) for c in cs_sorted])
# two arms fanning from base (c=0): +c red, -c blue. Marker grows with |c|; a point whose
# admin pmass fell below the coherence floor is hollow (the steer is no longer measuring).
# Two arms from base (c=0): +c red, -c blue. Marker size is constant so path length, not ink
# area, carries the coefficient change.
for lo, hi in [(0.0, max(cs_sorted)), (min(cs_sorted), 0.0)]:
arm = [(c, proj(traj[c])) for c in cs_sorted if lo <= c <= hi]
if len(arm) < 2:
continue
xy = np.array([p for _, p in arm])
ax.plot(xy[:, 0], xy[:, 1], "-", color="0.55", lw=0.9, zorder=4, alpha=0.8)
col = POS_COL if hi > 0 else NEG_COL
ax.plot(xy[:, 0], xy[:, 1], "-", color=col, lw=1.1, zorder=4, alpha=0.75)
c_end = max((c for c, _p in arm), key=abs)
for c, p in arm:
if c == 0:
continue
col = POS_COL if c > 0 else NEG_COL
ax.scatter(p[0], p[1], s=34, c="none" if c in inco else col,
edgecolors=col, linewidths=1.0, zorder=6)
fill = col if c == c_end and c not in inco else "none"
ax.scatter(p[0], p[1], s=42, c=fill, edgecolors=col,
linewidths=1.25, zorder=6)
if c == c_end:
lab = pos_lab if c > 0 else neg_lab
dxy, ha = ((9, 9), "left") if c > 0 else ((-9, -1), "right")
ax.annotate(lab, p, xytext=dxy, textcoords="offset points", fontsize=9,
color=col, fontweight="bold", ha=ha, va="center", zorder=8)
else:
for pt, col, lab, dxy, ha in [(ph, C_HON, pos_lab, (9, 9), "left"),
(pf, C_DIS, neg_lab, (-9, -1), "right")]:
if pt is None:
continue
ax.scatter(*pt, s=42, c=col, marker="o", edgecolors="white", linewidths=1.0, zorder=7)
ax.annotate(lab, pt, xytext=dxy, textcoords="offset points", fontsize=9, color=col,
fontweight="bold", ha=ha, va="center", zorder=8)
# Crop to the SOCIETIES + steer anchors for EVERY instrument (the human cloud is far wider and would
# bury them in a central blob; it stays a clipped backdrop). Then PAD THE BOTTOM to reserve a clean
# strip for the legend insets -- deterministic placement, identical on every plot, no overlap with
@@ -414,12 +425,11 @@ def plot_splom(instr: Instrument, dims: list[str], cloud: np.ndarray, M: np.ndar
# --- per-vector steer range ---------------------------------------------------------------------
def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, base_y: float,
lw: float = 2.0, dx: float = 0.12, tick: float = 0.038) -> None:
"""Draw one coherent AI c-path."""
def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, lw: float = 2.0,
lane_dx: float = 0.14) -> None:
"""Draw one coherent AI c-path. X is only the sign lane: negative, base, positive."""
assert list(cs) == sorted(cs) and 0.0 in cs, f"draw_steer needs sorted cs with c=0 (yv[-1]=+pole, yv[0]=-pole), got {cs}"
cmax = max(abs(c) for c in cs) or 1.0
xs_by_c = {c: xs if c == 0.0 else xs + dx * np.sign(c) * np.sqrt(abs(c) / cmax) for c in cs}
xs_by_c = {c: xs + lane_dx * np.sign(c) for c in cs}
for side_cs, col in [([c for c in cs if c <= 0.0], NEG_COL), ([c for c in cs if c >= 0.0], POS_COL)]:
if len(side_cs) < 2:
continue
@@ -428,11 +438,11 @@ def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, base_y: float,
ax.plot(x_path, y_path, color=col, lw=lw, alpha=0.9, zorder=6, solid_capstyle="round")
for c, y in zip(cs, yv):
if c == 0.0:
col, t = "black", tick * 1.25
col = "black"
else:
col, t = (POS_COL if c > 0 else NEG_COL), tick
col = POS_COL if c > 0 else NEG_COL
x = xs_by_c[c]
ax.plot([x - t, x + t], [float(y), float(y)], color=col, lw=lw, zorder=8)
ax.scatter(x, float(y), s=18, color=col, edgecolors="white", linewidths=0.45, zorder=8)
def draw_range_panel(ax, instr: Instrument, dims: list[str], cs: list[float], prof: dict,
@@ -464,8 +474,7 @@ def draw_range_panel(ax, instr: Instrument, dims: list[str], cs: list[float], pr
xs = gx + DX_STEER
yv = np.array([prof[c][i] for c in cs])
ys += yv.tolist()
base_y = float(yv[list(cs).index(0.0)])
draw_steer(ax, xs, cs, yv, base_y)
draw_steer(ax, xs, cs, yv)
if i == label_i:
# Only the two pole labels, to the RIGHT of the steer column. No 'base' tag: the black dot
# between the two coloured arms is self-evidently the unsteered model, and on a near-collapsed
@@ -534,8 +543,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d
soc_vals = np.array([m for _, m in soc])
q1, q3 = np.percentile(soc_vals, [25, 75])
# nanmin/nanmax: a collapsed pole reads NaN (read.py NaN-at-collapse, "do not compare"). The
# base (c=0) is always finite, so the axis still frames the un-collapsed cells; draw_steer
# skips the NaN arm on its own (the abs(NaN-base) test is False).
# base (c=0) is always finite, so the axis still frames the un-collapsed cells.
lo, hi = min(np.nanmin(yv), q1), max(np.nanmax(yv), q3)
m = max(0.10, 0.30 * (hi - lo))
ylo, yhi = lo - m, hi + m
@@ -550,7 +558,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d
xs = 0.30
base_y = float(yv[list(cs).index(0.0)])
draw_steer(ax, xs, cs, yv, base_y, lw=2.4, dx=0.10, tick=0.055)
draw_steer(ax, xs, cs, yv, lw=2.4)
named = {}
if near:
name_xy = {nm: (float(x), v) for (nm, v), x in zip(near, soc_x)}
@@ -558,7 +566,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d
named[min(near, key=lambda t: abs(t[1] - ref))[0]] = ref
tx = [xs] * len(cs) + [name_xy[nm][0] for nm in named] if near else [xs] * len(cs)
ty = list(map(float, yv)) + [name_xy[nm][1] for nm in named] if near else list(map(float, yv))
txt = [("c=0" if c == 0 else f"c={int(c):+d}") for c in cs] + list(named)
txt = [("c=0" if c == 0 else f"c={c:+g}") for c in cs] + list(named)
dot_x = [xs] * len(cs) + (list(soc_x) if near else [])
dot_y = list(map(float, yv)) + ([v for _, v in near] if near else [])
placed = False
+15 -8
View File
@@ -56,7 +56,9 @@ def build_user_content(instr: Instrument, item: InstrItem) -> str:
@torch.no_grad()
def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids: list[int],
*, max_think_tokens: int, batch_size: int = 36, verbose_first: bool = False) -> list[dict]:
*, max_think_tokens: int, batch_size: int = 36, n_samples: int = 1,
temperature: float = 0.0, top_p: float = 1.0,
verbose_first: bool = False) -> list[dict]:
"""Score a list of InstrItems (one frame's worth, or any subset). Returns per-item rows with
the keys `per_item_categorical` consumes: id, frame, p, pmass_allowed, dimension, sign, human_label.
@@ -83,31 +85,35 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids
# ordinal frames are already separate InstrItems, so single-pass (no reversed-enum two-pass;
# frame debias is downstream in canonicalize_to_forward). force_only: the "(" prefill is too
# short for natural-emission detection (matches by chance in the think trace), so always read
# the forced answer slot. n_samples=1, temperature=0 -> deterministic.
# the forced answer slot. n_samples>1 samples independent think traces, then averages the
# answer-token probabilities below.
thinks, slots = _rollout_natural_or_forced(
model, tok, user_prompts,
schema_hint="", max_think_tokens=max_think_tokens,
scoring_slots=[("Just answer", instr.prefill)],
gather_token_ids=answer_ids,
n_samples=1, temperature=0.0, force_only=True,
n_samples=n_samples, temperature=temperature, top_p=top_p, force_only=True,
verbose=verbose_first and i == 0,
)
for j, it in enumerate(chunk):
slot = slots[j][0]
sample_idx = [j * n_samples + n for n in range(n_samples)]
sample_slots = [slots[k][0] for k in sample_idx]
# lp = lp_gather: the full-vocab log_softmax logprob of each answer token at the answer
# slot. This is the RAW PRIMITIVE -- every readout (E, the logit contrast C, log-odds,
# entropy) is a pure function of it, and a steer effect is just a difference of lp. Keep
# it; do not throw it away by collapsing to a single number here.
lp = np.asarray(slot["lp_gather"], dtype=float) # [A] raw logprobs (full-vocab norm)
sample_lp = np.asarray([s["lp_gather"] for s in sample_slots], dtype=float) # [N,A]
lp = np.log(np.nanmean(np.exp(sample_lp), axis=0)) # [A] BMA over sampled thoughts
p_a = np.exp(lp) # [A] prob on each answer token
pmass = float(slot["pmass_allowed"]) # mass on allowed tokens (coherence)
pmass = float(np.nansum(p_a)) # mass on allowed tokens (coherence)
# Renormalize within allowed. INTENTIONALLY NOT NaN-guarded: at full coherence collapse
# pmass -> 0 so p_norm -> NaN and poisons that item's factor. That is the honest signal, a
# distribution renormalized from ~zero mass is NOT comparable to one from real mass (the mean
# of 10 != the mean of 130), so it must not be silently turned into a comparable-looking
# number. NaN marks "do not compare". Do not "fix" this with a softmax/eps fallback.
p_norm = p_a / p_a.sum() # [A] within allowed (NaN at collapse, by design)
think_text, n_think, emitted_close = thinks[j]
sample_thinks = [thinks[k] for k in sample_idx]
think_text, n_think, emitted_close = sample_thinks[0]
out.append({
"id": it.id, "frame": it.frame,
"lp": lp, # raw logprobs at the M scale tokens
@@ -115,7 +121,8 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids
"pmass_allowed": pmass,
"dimension": it.dimension, "sign": it.sign,
"human_label": it.human_label,
"think": think_text, "n_think": n_think, "emitted_close": emitted_close,
"think": think_text, "n_think": n_think,
"emitted_close": any(t[2] for t in sample_thinks),
})
if verbose_first and i == 0:
slot0 = slots[0][0]