mirror of
https://github.com/wassname/moral-maps.git
synced 2026-09-10 12:14:54 +08:00
Support sampled survey reads and MFV c-grid plots
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
# README plot and metric simplification
|
||||
|
||||
## Goal
|
||||
Make the README readable to a new researcher in one pass. It should say what tinymfv is, what datasets it includes, what single measurement to look at first, and how the plots encode base/+C/-C steering.
|
||||
|
||||
## Scope
|
||||
In: README wording, metric/API simplification, regenerated showcase plots, cold-reader panel review.
|
||||
Out: changing eval semantics, changing dataset schemas, adding new metrics.
|
||||
|
||||
## Requirements
|
||||
- R1: README has one headline measurement: `profile`. Done means: the Metrics section defines the profile once and treats logprobs/format checks as API details, not competing headline metrics. VERIFY: `rg -n '^###|profile_C|informedness|nll_prefill|pmass_allowed|dlogit_per_foundation' README.md`.
|
||||
- R2: README explains the plot encoding. Done means: a reader can tell what gray, black, red, and blue marks mean without knowing steering-lite internals. VERIFY: README contains the color/key sentence and no stale "trajectory" caption for the clean maps.
|
||||
- R3: README still lists all datasets and all image links resolve. VERIFY: a script parses markdown image links and prints `missing_images: []`.
|
||||
- R4: Cold readers can reconstruct what tinymfv is, which datasets exist, what measurement matters, and why a researcher would use it. VERIFY: external-review-v2 panel JSON files in `docs/reviews/` plus a short triage table in this spec.
|
||||
- R5: README survey maps show the coherent steering path, not just two endpoints. Done means: MFQ-2, Big Five, and Humor maps draw base -> tested coherent c values, with constant-size c markers and no incoherent lone dots. VERIFY: inspect regenerated PNGs and check the plotter passes `traj=` to `plot_ipsative_pca`.
|
||||
- R6: README range plots show the coherent coefficient path in sign lanes. Done means: all negative c values share a blue lane, base is black in the middle, and all positive c values share a red lane. Reversals/wraps are visible on the y-axis without marker-size encoding. VERIFY: inspect regenerated range PNGs and check `draw_steer` maps x by sign only.
|
||||
|
||||
## Tasks
|
||||
- [x] T1 (R1, R2): Simplify README around `profile` and update plot captions.
|
||||
- verify: `rg -n '^###|profile_C|informedness|nll_prefill|pmass_allowed|dlogit_per_foundation|trajectory' README.md`
|
||||
- success: one `### The profile` subsection; no stale plot "trajectory" language.
|
||||
- likely_fail: API section still presents six metrics as peers.
|
||||
- sneaky_fail: profile is too vague to connect to MFV vs survey data; caught by panel mechanism answers.
|
||||
- [/] T2 (R2, R3): Keep regenerated showcase plots and verify image refs.
|
||||
- verify: parse README image links and check paths exist.
|
||||
- success: `missing_images: []`.
|
||||
- likely_fail: stale filenames remain.
|
||||
- sneaky_fail: images exist but show old sweep encoding; caught by fresh image inspection of MFQ-2 map/range.
|
||||
- [ ] T3 (R4): Run external-review-v2 comprehension panel and triage.
|
||||
- verify: panel outputs valid JSON and summaries answer the probe.
|
||||
- success: most panel members correctly identify tinymfv, datasets, `profile`, and researcher use.
|
||||
- likely_fail: models still name `profile_C` or `logprobs` as the main metric.
|
||||
- sneaky_fail: models parrot phrases without explaining why a researcher would use it; caught by the "own words with inference" probe.
|
||||
- [/] T4 (R5, R6): Restore coherent c paths while removing misleading geometry.
|
||||
- steps: compute one shared answer-mass gate across survey evals; pass the shared coherent path to map plots; make range steer paths use sign lanes.
|
||||
- verify: `rg -n "traj=|np.sign\\(c\\)|s=42" scripts/plot_steer_showcase.py src/tinymfv/maps.py`
|
||||
- success: maps show red/blue coherent paths through c values; ranges show AI sign-lane paths.
|
||||
- likely_fail: maps still show only endpoint dots.
|
||||
- sneaky_fail: marker size encodes c, or x offset varies by c magnitude; caught by the source check and image inspection.
|
||||
- UAT: when opening `docs/img/showcase/mfq2/map_pca_ipsative.png` and `docs/img/showcase/mfq2/range.png`, the path is visible and the range path can show reversals/wraps.
|
||||
|
||||
## Log
|
||||
- 2026-06-30: The README previously exposed many metric names at once. That made the API complete but caused cold-reader confusion about which measurement to start with.
|
||||
- 2026-06-30: Final plots are gated on steering-lite task 394, output `outputs/20260630_dignity_authority_strict22_local_sspace_allinstr`. Stale README images from the earlier fairness / two-axis run must not count as UAT.
|
||||
- 2026-06-30: Coherence gating must be a signed prefix from `c=0`, stopping a side at the first coefficient below the base-relative answer-mass floor. A per-row filter can produce confusing lone high-C dots after collapse.
|
||||
- 2026-06-30: README structural check before final regeneration: 9 images, `missing_images: []`, 15 local links, `missing_local_links: []`, and no 16PF map link.
|
||||
- 2026-06-30: The endpoint-only map removed the path readers need. The sign-offset range geometry also made a one-dimensional value plot look like a fake two-dimensional fork.
|
||||
- 2026-06-30: User wants range plots to show the path as sign lanes, not coefficient-x swim lines. The fix is negative/base/positive lanes, with vertical paths inside each lane.
|
||||
@@ -4,7 +4,7 @@ Consumes a steering-lite `run_allinstr_showcase.py` output dir (one calibrated
|
||||
activation-steering vector administered across every instrument over a signed
|
||||
c-sweep) and renders the SAME two figures for every instrument, uniformly:
|
||||
|
||||
- map : ipsative culture map (PCA), AI base + strongest coherent +/-c vs the human cloud.
|
||||
- map : ipsative culture map (PCA), AI coherent +/-c path vs the human cloud.
|
||||
- range: per-factor range, AI base + coherent +/-c path vs the human society strip.
|
||||
|
||||
Ordinal instruments (mfq2/big5/16pf/humor_styles) read <name>_profiles.csv; nominal
|
||||
@@ -13,9 +13,8 @@ logit-violation units cannot share a raw axis with 1-5 wrongness), but it goes
|
||||
through the same plot_ipsative_pca / plot_range and yields the same two figures.
|
||||
|
||||
cs are SIGNED multipliers of the calibrated coefficient C (0 = base). The public
|
||||
README range plots show the coherent path: c=0 plus each +/-c row whose pmass stays
|
||||
above the requested fraction of base. Maps show only the strongest coherent endpoints.
|
||||
Incoherent rows are dropped, not drawn hollow.
|
||||
README plots show the coherent path: c=0 plus each +/-c row whose tinymfv answer mass
|
||||
stays above the requested fraction of base. Incoherent rows are dropped, not drawn hollow.
|
||||
|
||||
uv run python scripts/plot_steer_showcase.py \
|
||||
--run-dir ../steering-lite/outputs/allinstr_qwen35_4b --out docs/img/showcase
|
||||
@@ -23,6 +22,7 @@ Incoherent rows are dropped, not drawn hollow.
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
@@ -112,28 +112,36 @@ def read_profiles(run_dir: Path, name: str, dims: list[str], value_col: str = "m
|
||||
return {c: np.array([d[f] for f in dims]) for c, d in by_c.items()}, pmass
|
||||
|
||||
|
||||
def coherent_prefix_cs(cs: list[float], pmass: dict[float, float], coherence_frac: float) -> list[float]:
|
||||
def coherent_prefix_cs(cs: list[float], pmass_ratio: dict[float, float], coherence_frac: float) -> list[float]:
|
||||
"""c=0 plus each signed arm until answer mass first falls below the base-relative floor."""
|
||||
base_pm = pmass[0.0]
|
||||
kept = [0.0]
|
||||
for side in (1.0, -1.0):
|
||||
for c in sorted([c for c in cs if np.sign(c) == side], key=abs):
|
||||
if pmass[c] <= coherence_frac * base_pm:
|
||||
if pmass_ratio[c] <= coherence_frac:
|
||||
break
|
||||
kept.append(c)
|
||||
return sorted(kept)
|
||||
|
||||
|
||||
def shared_pmass_ratio(run_dir: Path, names: list[str]) -> dict[float, float]:
|
||||
"""Worst base-relative answer mass across the survey evals, keyed by signed calibrated multiplier."""
|
||||
pmasses: list[dict[float, float]] = []
|
||||
for name in names:
|
||||
instr = get_instrument(name)
|
||||
_, pmass = read_profiles(run_dir, name, instr.dimensions)
|
||||
pmasses.append(pmass)
|
||||
cs = sorted(set.intersection(*(set(p) for p in pmasses)))
|
||||
return {c: min(p[c] / p[0.0] for p in pmasses) for c in cs}
|
||||
|
||||
|
||||
def plot_ordinal(run_dir: Path, out: Path, name: str, vec_label: str, C: float,
|
||||
coherence_frac: float) -> list[Path]:
|
||||
instr = get_instrument(name)
|
||||
coh_cs: list[float]) -> list[Path]:
|
||||
instr = copy.copy(get_instrument(name))
|
||||
if name == "mfq2":
|
||||
instr.display = "MFQ-2 survey"
|
||||
dims = instr.dimensions
|
||||
prof_c, pmass = read_profiles(run_dir, name, dims)
|
||||
cs = sorted(prof_c)
|
||||
prof_c, _pmass = read_profiles(run_dir, name, dims)
|
||||
base = prof_c[0.0]
|
||||
# Coherence gate is RELATIVE and monotone per signed arm: walk outward from c=0 and stop at the
|
||||
# first coefficient whose allowed-answer mass falls below the requested fraction of base.
|
||||
coh_cs = coherent_prefix_cs(cs, pmass, coherence_frac)
|
||||
pos_c = max(c for c in coh_cs if c > 0.0)
|
||||
neg_c = min(c for c in coh_cs if c < 0.0)
|
||||
pos = prof_c[pos_c]
|
||||
@@ -151,10 +159,11 @@ def plot_ordinal(run_dir: Path, out: Path, name: str, vec_label: str, C: float,
|
||||
respondents, haze = T.maps.respondent_profiles(dims, instr.scale_max), None
|
||||
else:
|
||||
respondents, haze = None, human_haze(instr)
|
||||
traj = {c: _frac(prof_c[c], instr.scale_max) for c in coh_cs}
|
||||
figm = T.maps.plot_ipsative_pca(instr, dims, countries, Mfrac,
|
||||
_frac(base, instr.scale_max), _frac(pos, instr.scale_max),
|
||||
_frac(neg, instr.scale_max), respondents=respondents, haze=haze,
|
||||
labels=labels)
|
||||
traj=traj, labels=labels)
|
||||
figm.axes[0].set_title(f"{instr.display}: humans vs LLMs steered for {vec_label}", fontsize=10)
|
||||
paths = [T.maps.save_both(figm, out / name, "map_pca_ipsative")]
|
||||
plt.close(figm)
|
||||
@@ -182,29 +191,34 @@ def read_human_mfv() -> tuple[list[str], dict[str, dict[str, float]]]:
|
||||
by_country.setdefault(r["country"], {})[r["foundation"]] = float(r["mean"])
|
||||
return sorted(by_country), by_country
|
||||
|
||||
|
||||
def _mfv_zspace(run_dir: Path):
|
||||
"""Shared MFV adapter -> the common coordinate system the map AND range both consume: z-scored
|
||||
relative-emphasis profiles (model base / +C / -C) + the human MFV culture matrix in the same
|
||||
space. MFV is nominal (model emits logit(violation) per foundation, humans rate wrongness 1-5),
|
||||
so absolute scales differ; z-scoring each profile ACROSS foundations compares the PATTERN -- which
|
||||
foundations a reader weights as more violation-worthy than their own average -- which is exactly
|
||||
what the steer moves. Social Norms is dropped (no human MFV norm), asserted so a taxonomy change
|
||||
fails loud. Returns (founds, countries, M_z[countries x founds], base_z, posz, negz)."""
|
||||
d = json.loads((run_dir / "mfv.json").read_text())
|
||||
base_l = d["base_logit_per_foundation"]
|
||||
pos_dl, neg_dl = d["pos"]["dlogit_per_foundation"], d["neg"]["dlogit_per_foundation"]
|
||||
def read_mfv_profiles(run_dir: Path) -> tuple[list[str], dict[float, np.ndarray], dict[float, float]]:
|
||||
rows = list(csv.DictReader((run_dir / "mfv_profiles.csv").open()))
|
||||
foundation_order = []
|
||||
for r in rows:
|
||||
if r["foundation"] not in foundation_order:
|
||||
foundation_order.append(r["foundation"])
|
||||
countries, human = read_human_mfv()
|
||||
hfounds = set(next(iter(human.values())))
|
||||
founds = [f for f in d["foundation_order"] if f.lower() in hfounds] # shared, model order
|
||||
dropped = [f for f in d["foundation_order"] if f.lower() not in hfounds]
|
||||
founds = [f for f in foundation_order if f.lower() in hfounds]
|
||||
dropped = [f for f in foundation_order if f.lower() not in hfounds]
|
||||
assert dropped == ["Social Norms"], f"unexpected MFV foundations without a human norm: {dropped}"
|
||||
by_c: dict[float, dict[str, float]] = {}
|
||||
pmass: dict[float, float] = {}
|
||||
for r in rows:
|
||||
c = float(r["c"])
|
||||
by_c.setdefault(c, {})[r["foundation"]] = float(r["mean"])
|
||||
pmass[c] = float(r["pmass"])
|
||||
prof = {c: _zscore(np.array([vals[f] for f in founds])) for c, vals in by_c.items()}
|
||||
return founds, prof, pmass
|
||||
|
||||
|
||||
def _mfv_zspace(run_dir: Path):
|
||||
"""Shared MFV adapter -> z-scored relative-emphasis profiles + human MFV culture matrix."""
|
||||
founds, prof, pmass = read_mfv_profiles(run_dir)
|
||||
countries, human = read_human_mfv()
|
||||
fl = [f.lower() for f in founds]
|
||||
base = _zscore(np.array([base_l[f]["mean"] for f in founds]))
|
||||
posz = _zscore(np.array([base_l[f]["mean"] + pos_dl[f]["mean"] for f in founds]))
|
||||
negz = _zscore(np.array([base_l[f]["mean"] + neg_dl[f]["mean"] for f in founds]))
|
||||
M = np.array([_zscore(np.array([human[c][f] for f in fl])) for c in countries])
|
||||
return founds, countries, M, base, posz, negz
|
||||
return founds, countries, M, prof, pmass
|
||||
|
||||
|
||||
# MFV has no ordinal Instrument (it goes through evaluate_multibool, not administer), but the shared
|
||||
@@ -214,28 +228,30 @@ _MFV_INSTR = SimpleNamespace(name="mfv", display="MFV vignettes")
|
||||
_MFV_YLABEL = "relative emphasis (z across foundations)"
|
||||
|
||||
|
||||
def plot_mfv_map(run_dir: Path, out: Path, vec_label: str, C: float) -> Path:
|
||||
def plot_mfv_map(run_dir: Path, out: Path, vec_label: str, C: float, coh_cs: list[float]) -> Path:
|
||||
"""MFV ipsative culture map via the SAME plot_ipsative_pca the ordinal instruments use, in the
|
||||
z-scored relative-emphasis space (logit-violation and 1-5 wrongness cannot share a raw axis).
|
||||
Red/blue endpoint points show where the steer moves the AI among human cultures."""
|
||||
founds, countries, M, base, posz, negz = _mfv_zspace(run_dir)
|
||||
labels = ("base (c=0)", "c=+1", "c=-1")
|
||||
fig = T.maps.plot_ipsative_pca(_MFV_INSTR, founds, countries, M, base, posz, negz, labels=labels)
|
||||
founds, countries, M, prof, _pmass = _mfv_zspace(run_dir)
|
||||
pos_c = max(c for c in coh_cs if c > 0.0)
|
||||
neg_c = min(c for c in coh_cs if c < 0.0)
|
||||
labels = ("base (c=0)", f"c={pos_c:+g}", f"c={neg_c:+g}")
|
||||
traj = {c: prof[c] for c in coh_cs}
|
||||
fig = T.maps.plot_ipsative_pca(_MFV_INSTR, founds, countries, M, prof[0.0], prof[pos_c], prof[neg_c],
|
||||
traj=traj, labels=labels)
|
||||
fig.axes[0].set_title(f"MFV vignettes: humans vs LLMs steered for {vec_label}", fontsize=10)
|
||||
path = T.maps.save_both(fig, out / "mfv", "map_pca_ipsative")
|
||||
plt.close(fig)
|
||||
return path
|
||||
|
||||
|
||||
def plot_mfv_range(run_dir: Path, out: Path, vec_label: str, C: float) -> Path:
|
||||
"""MFV range via the SAME plot_range the ordinal instruments use, in z relative-emphasis space.
|
||||
Only base/+C/-C (the MFV eval is a 3-point sweep, not a multi-C grid like the ordinal admin)."""
|
||||
founds, countries, M, base, posz, negz = _mfv_zspace(run_dir)
|
||||
cs = [-1.0, 0.0, 1.0]
|
||||
prof = {-1.0: negz, 0.0: base, 1.0: posz}
|
||||
def plot_mfv_range(run_dir: Path, out: Path, vec_label: str, C: float, coh_cs: list[float]) -> Path:
|
||||
"""MFV range via the SAME plot_range the ordinal instruments use, in z relative-emphasis space."""
|
||||
founds, countries, M, prof, _pmass = _mfv_zspace(run_dir)
|
||||
humans = {f: sorted(((countries[ci], float(M[ci, fi])) for ci in range(len(countries))), key=lambda t: t[1])
|
||||
for fi, f in enumerate(founds)}
|
||||
fig = T.maps.plot_range(_MFV_INSTR, founds, cs, prof, humans, None, vec_label, ylabel=_MFV_YLABEL)
|
||||
fig = T.maps.plot_range(_MFV_INSTR, founds, coh_cs, {c: prof[c] for c in coh_cs},
|
||||
humans, None, vec_label, ylabel=_MFV_YLABEL)
|
||||
path = T.maps.save_both(fig, out / "mfv", "range")
|
||||
plt.close(fig)
|
||||
return path
|
||||
@@ -257,13 +273,19 @@ def main() -> None:
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
written: list[str] = []
|
||||
for name in ORDINAL:
|
||||
if (args.run_dir / f"{name}_profiles.csv").exists():
|
||||
written += [str(p) for p in plot_ordinal(args.run_dir, args.out, name, vec_label, C,
|
||||
args.coherence_frac)]
|
||||
if (args.run_dir / "mfv.json").exists():
|
||||
written.append(str(plot_mfv_map(args.run_dir, args.out, vec_label, C))) # shared ipsative map (z-space)
|
||||
written.append(str(plot_mfv_range(args.run_dir, args.out, vec_label, C))) # shared range (z-space)
|
||||
ordinal_names = [name for name in ORDINAL if (args.run_dir / f"{name}_profiles.csv").exists()]
|
||||
pmass_ratio = shared_pmass_ratio(args.run_dir, ordinal_names)
|
||||
if (args.run_dir / "mfv_profiles.csv").exists():
|
||||
_founds, _prof, mfv_pmass = read_mfv_profiles(args.run_dir)
|
||||
for c, pm in mfv_pmass.items():
|
||||
pmass_ratio[c] = min(pmass_ratio[c], pm / mfv_pmass[0.0])
|
||||
coh_cs = coherent_prefix_cs(sorted(pmass_ratio), pmass_ratio, args.coherence_frac)
|
||||
print(f"shared coherent c values at {args.coherence_frac:.2%} base answer mass: {coh_cs}")
|
||||
for name in ordinal_names:
|
||||
written += [str(p) for p in plot_ordinal(args.run_dir, args.out, name, vec_label, C, coh_cs)]
|
||||
if (args.run_dir / "mfv_profiles.csv").exists():
|
||||
written.append(str(plot_mfv_map(args.run_dir, args.out, vec_label, C, coh_cs))) # shared ipsative map (z-space)
|
||||
written.append(str(plot_mfv_range(args.run_dir, args.out, vec_label, C, coh_cs))) # shared range (z-space)
|
||||
print(f"wrote {len(written)} figures under {args.out}:")
|
||||
for w in written:
|
||||
print(" ", w)
|
||||
|
||||
@@ -67,7 +67,8 @@ class AdministerResult(TypedDict):
|
||||
|
||||
|
||||
def administer(model, tok, instr: Instrument, *, batch_size: int = 36,
|
||||
max_think_tokens: int = 64) -> AdministerResult:
|
||||
max_think_tokens: int = 64, n_samples: int = 1,
|
||||
temperature: float = 0.0, top_p: float = 1.0) -> AdministerResult:
|
||||
assert instr.kind == "ordinal", "administer() is the ordinal survey readout; use evaluate() for nominal MFV"
|
||||
# Every ordinal item must carry its frame-specific response-scale legend in meta['task']; without
|
||||
# it build_user_content would silently emit a bare statement (no legend) and the profile would be
|
||||
@@ -78,7 +79,9 @@ def administer(model, tok, instr: Instrument, *, batch_size: int = 36,
|
||||
# slot, so an activation steer accrues over the trace before being read. Floor is 1 (the shared
|
||||
# rollout core's HF generate() rejects max_new_tokens=0).
|
||||
per_row = read_items(model, tok, instr, instr.items, answer_ids,
|
||||
max_think_tokens=max_think_tokens, batch_size=batch_size, verbose_first=True)
|
||||
max_think_tokens=max_think_tokens, batch_size=batch_size,
|
||||
n_samples=n_samples, temperature=temperature, top_p=top_p,
|
||||
verbose_first=True)
|
||||
items = per_item_categorical(per_row, instr.kind) # {id: {p, pmass, dimension, sign, ...}}
|
||||
|
||||
M = instr.scale_max
|
||||
|
||||
+39
-31
@@ -190,9 +190,9 @@ def plot_ipsative_pca(instr: Instrument, dims: list[str], countries: list[str],
|
||||
for the crop -- separate from the fit so instruments with only society-level mean+sd (big5/16pf/
|
||||
humor: a marginal resample) get a backdrop without that resample dictating the axes. mfq2 passes
|
||||
real `respondents` (also used as the haze when `haze` is None). With neither, fit on M, pad-crop,
|
||||
no backdrop. `traj` (signed c-multiplier -> length-K fraction vector) draws the full steer SWEEP
|
||||
as a connected path through PC space, so a multi-C run shows where the steer leaves the human
|
||||
cloud and curves into incoherence (the base/pos/neg arrows stay as the headline +-C anchors).
|
||||
no backdrop. `traj` (signed c-multiplier -> length-K fraction vector) draws the coherent steer
|
||||
path through PC space. Public README plots pass only the coherent prefix; incoherent c values
|
||||
are omitted.
|
||||
`traj_incoherent` is the subset of those c whose admin pmass fell below the coherence floor --
|
||||
drawn hollow. `boots` optionally maps 'base'/'honest'/'dis' -> (n x K) bootstrap matrices. Returns
|
||||
the Figure."""
|
||||
@@ -238,33 +238,44 @@ def plot_ipsative_pca(instr: Instrument, dims: list[str], countries: list[str],
|
||||
ax.errorbar(pt[0], pt[1], xerr=e1, yerr=e2, fmt="none", ecolor=col,
|
||||
elinewidth=0.7, alpha=0.55, capsize=2.5, capthick=0.8, zorder=4)
|
||||
base_lab, pos_lab, neg_lab = labels
|
||||
for pt, col, lab, dxy, ha in [(ph, C_HON, pos_lab, (9, 9), "left"),
|
||||
(pf, C_DIS, neg_lab, (-9, -1), "right"),
|
||||
(pb, C_BASE, base_lab, (9, -13), "left")]:
|
||||
if pt is None:
|
||||
continue
|
||||
ax.scatter(*pt, s=120, c=col, marker="o", edgecolors="white", linewidths=1.2, zorder=7)
|
||||
ax.annotate(lab, pt, xytext=dxy, textcoords="offset points", fontsize=9, color=col,
|
||||
fontweight="bold", ha=ha, va="center", zorder=8)
|
||||
if pb is not None:
|
||||
ax.scatter(*pb, s=72, c=C_BASE, marker="o", edgecolors="white", linewidths=1.0, zorder=7)
|
||||
ax.annotate(base_lab, pb, xytext=(9, -13), textcoords="offset points", fontsize=9,
|
||||
color=C_BASE, fontweight="bold", ha="left", va="center", zorder=8)
|
||||
traj_pts = None
|
||||
if traj:
|
||||
inco = traj_incoherent or set()
|
||||
cs_sorted = sorted(traj)
|
||||
traj_pts = np.array([proj(traj[c]) for c in cs_sorted])
|
||||
# two arms fanning from base (c=0): +c red, -c blue. Marker grows with |c|; a point whose
|
||||
# admin pmass fell below the coherence floor is hollow (the steer is no longer measuring).
|
||||
# Two arms from base (c=0): +c red, -c blue. Marker size is constant so path length, not ink
|
||||
# area, carries the coefficient change.
|
||||
for lo, hi in [(0.0, max(cs_sorted)), (min(cs_sorted), 0.0)]:
|
||||
arm = [(c, proj(traj[c])) for c in cs_sorted if lo <= c <= hi]
|
||||
if len(arm) < 2:
|
||||
continue
|
||||
xy = np.array([p for _, p in arm])
|
||||
ax.plot(xy[:, 0], xy[:, 1], "-", color="0.55", lw=0.9, zorder=4, alpha=0.8)
|
||||
col = POS_COL if hi > 0 else NEG_COL
|
||||
ax.plot(xy[:, 0], xy[:, 1], "-", color=col, lw=1.1, zorder=4, alpha=0.75)
|
||||
c_end = max((c for c, _p in arm), key=abs)
|
||||
for c, p in arm:
|
||||
if c == 0:
|
||||
continue
|
||||
col = POS_COL if c > 0 else NEG_COL
|
||||
ax.scatter(p[0], p[1], s=34, c="none" if c in inco else col,
|
||||
edgecolors=col, linewidths=1.0, zorder=6)
|
||||
fill = col if c == c_end and c not in inco else "none"
|
||||
ax.scatter(p[0], p[1], s=42, c=fill, edgecolors=col,
|
||||
linewidths=1.25, zorder=6)
|
||||
if c == c_end:
|
||||
lab = pos_lab if c > 0 else neg_lab
|
||||
dxy, ha = ((9, 9), "left") if c > 0 else ((-9, -1), "right")
|
||||
ax.annotate(lab, p, xytext=dxy, textcoords="offset points", fontsize=9,
|
||||
color=col, fontweight="bold", ha=ha, va="center", zorder=8)
|
||||
else:
|
||||
for pt, col, lab, dxy, ha in [(ph, C_HON, pos_lab, (9, 9), "left"),
|
||||
(pf, C_DIS, neg_lab, (-9, -1), "right")]:
|
||||
if pt is None:
|
||||
continue
|
||||
ax.scatter(*pt, s=42, c=col, marker="o", edgecolors="white", linewidths=1.0, zorder=7)
|
||||
ax.annotate(lab, pt, xytext=dxy, textcoords="offset points", fontsize=9, color=col,
|
||||
fontweight="bold", ha=ha, va="center", zorder=8)
|
||||
# Crop to the SOCIETIES + steer anchors for EVERY instrument (the human cloud is far wider and would
|
||||
# bury them in a central blob; it stays a clipped backdrop). Then PAD THE BOTTOM to reserve a clean
|
||||
# strip for the legend insets -- deterministic placement, identical on every plot, no overlap with
|
||||
@@ -414,12 +425,11 @@ def plot_splom(instr: Instrument, dims: list[str], cloud: np.ndarray, M: np.ndar
|
||||
|
||||
# --- per-vector steer range ---------------------------------------------------------------------
|
||||
|
||||
def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, base_y: float,
|
||||
lw: float = 2.0, dx: float = 0.12, tick: float = 0.038) -> None:
|
||||
"""Draw one coherent AI c-path."""
|
||||
def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, lw: float = 2.0,
|
||||
lane_dx: float = 0.14) -> None:
|
||||
"""Draw one coherent AI c-path. X is only the sign lane: negative, base, positive."""
|
||||
assert list(cs) == sorted(cs) and 0.0 in cs, f"draw_steer needs sorted cs with c=0 (yv[-1]=+pole, yv[0]=-pole), got {cs}"
|
||||
cmax = max(abs(c) for c in cs) or 1.0
|
||||
xs_by_c = {c: xs if c == 0.0 else xs + dx * np.sign(c) * np.sqrt(abs(c) / cmax) for c in cs}
|
||||
xs_by_c = {c: xs + lane_dx * np.sign(c) for c in cs}
|
||||
for side_cs, col in [([c for c in cs if c <= 0.0], NEG_COL), ([c for c in cs if c >= 0.0], POS_COL)]:
|
||||
if len(side_cs) < 2:
|
||||
continue
|
||||
@@ -428,11 +438,11 @@ def draw_steer(ax, xs: float, cs: list[float], yv: np.ndarray, base_y: float,
|
||||
ax.plot(x_path, y_path, color=col, lw=lw, alpha=0.9, zorder=6, solid_capstyle="round")
|
||||
for c, y in zip(cs, yv):
|
||||
if c == 0.0:
|
||||
col, t = "black", tick * 1.25
|
||||
col = "black"
|
||||
else:
|
||||
col, t = (POS_COL if c > 0 else NEG_COL), tick
|
||||
col = POS_COL if c > 0 else NEG_COL
|
||||
x = xs_by_c[c]
|
||||
ax.plot([x - t, x + t], [float(y), float(y)], color=col, lw=lw, zorder=8)
|
||||
ax.scatter(x, float(y), s=18, color=col, edgecolors="white", linewidths=0.45, zorder=8)
|
||||
|
||||
|
||||
def draw_range_panel(ax, instr: Instrument, dims: list[str], cs: list[float], prof: dict,
|
||||
@@ -464,8 +474,7 @@ def draw_range_panel(ax, instr: Instrument, dims: list[str], cs: list[float], pr
|
||||
xs = gx + DX_STEER
|
||||
yv = np.array([prof[c][i] for c in cs])
|
||||
ys += yv.tolist()
|
||||
base_y = float(yv[list(cs).index(0.0)])
|
||||
draw_steer(ax, xs, cs, yv, base_y)
|
||||
draw_steer(ax, xs, cs, yv)
|
||||
if i == label_i:
|
||||
# Only the two pole labels, to the RIGHT of the steer column. No 'base' tag: the black dot
|
||||
# between the two coloured arms is self-evidently the unsteered model, and on a near-collapsed
|
||||
@@ -534,8 +543,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d
|
||||
soc_vals = np.array([m for _, m in soc])
|
||||
q1, q3 = np.percentile(soc_vals, [25, 75])
|
||||
# nanmin/nanmax: a collapsed pole reads NaN (read.py NaN-at-collapse, "do not compare"). The
|
||||
# base (c=0) is always finite, so the axis still frames the un-collapsed cells; draw_steer
|
||||
# skips the NaN arm on its own (the abs(NaN-base) test is False).
|
||||
# base (c=0) is always finite, so the axis still frames the un-collapsed cells.
|
||||
lo, hi = min(np.nanmin(yv), q1), max(np.nanmax(yv), q3)
|
||||
m = max(0.10, 0.30 * (hi - lo))
|
||||
ylo, yhi = lo - m, hi + m
|
||||
@@ -550,7 +558,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d
|
||||
|
||||
xs = 0.30
|
||||
base_y = float(yv[list(cs).index(0.0)])
|
||||
draw_steer(ax, xs, cs, yv, base_y, lw=2.4, dx=0.10, tick=0.055)
|
||||
draw_steer(ax, xs, cs, yv, lw=2.4)
|
||||
named = {}
|
||||
if near:
|
||||
name_xy = {nm: (float(x), v) for (nm, v), x in zip(near, soc_x)}
|
||||
@@ -558,7 +566,7 @@ def plot_range_zoom(instr: Instrument, dims: list[str], cs: list[float], prof: d
|
||||
named[min(near, key=lambda t: abs(t[1] - ref))[0]] = ref
|
||||
tx = [xs] * len(cs) + [name_xy[nm][0] for nm in named] if near else [xs] * len(cs)
|
||||
ty = list(map(float, yv)) + [name_xy[nm][1] for nm in named] if near else list(map(float, yv))
|
||||
txt = [("c=0" if c == 0 else f"c={int(c):+d}") for c in cs] + list(named)
|
||||
txt = [("c=0" if c == 0 else f"c={c:+g}") for c in cs] + list(named)
|
||||
dot_x = [xs] * len(cs) + (list(soc_x) if near else [])
|
||||
dot_y = list(map(float, yv)) + ([v for _, v in near] if near else [])
|
||||
placed = False
|
||||
|
||||
+15
-8
@@ -56,7 +56,9 @@ def build_user_content(instr: Instrument, item: InstrItem) -> str:
|
||||
|
||||
@torch.no_grad()
|
||||
def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids: list[int],
|
||||
*, max_think_tokens: int, batch_size: int = 36, verbose_first: bool = False) -> list[dict]:
|
||||
*, max_think_tokens: int, batch_size: int = 36, n_samples: int = 1,
|
||||
temperature: float = 0.0, top_p: float = 1.0,
|
||||
verbose_first: bool = False) -> list[dict]:
|
||||
"""Score a list of InstrItems (one frame's worth, or any subset). Returns per-item rows with
|
||||
the keys `per_item_categorical` consumes: id, frame, p, pmass_allowed, dimension, sign, human_label.
|
||||
|
||||
@@ -83,31 +85,35 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids
|
||||
# ordinal frames are already separate InstrItems, so single-pass (no reversed-enum two-pass;
|
||||
# frame debias is downstream in canonicalize_to_forward). force_only: the "(" prefill is too
|
||||
# short for natural-emission detection (matches by chance in the think trace), so always read
|
||||
# the forced answer slot. n_samples=1, temperature=0 -> deterministic.
|
||||
# the forced answer slot. n_samples>1 samples independent think traces, then averages the
|
||||
# answer-token probabilities below.
|
||||
thinks, slots = _rollout_natural_or_forced(
|
||||
model, tok, user_prompts,
|
||||
schema_hint="", max_think_tokens=max_think_tokens,
|
||||
scoring_slots=[("Just answer", instr.prefill)],
|
||||
gather_token_ids=answer_ids,
|
||||
n_samples=1, temperature=0.0, force_only=True,
|
||||
n_samples=n_samples, temperature=temperature, top_p=top_p, force_only=True,
|
||||
verbose=verbose_first and i == 0,
|
||||
)
|
||||
for j, it in enumerate(chunk):
|
||||
slot = slots[j][0]
|
||||
sample_idx = [j * n_samples + n for n in range(n_samples)]
|
||||
sample_slots = [slots[k][0] for k in sample_idx]
|
||||
# lp = lp_gather: the full-vocab log_softmax logprob of each answer token at the answer
|
||||
# slot. This is the RAW PRIMITIVE -- every readout (E, the logit contrast C, log-odds,
|
||||
# entropy) is a pure function of it, and a steer effect is just a difference of lp. Keep
|
||||
# it; do not throw it away by collapsing to a single number here.
|
||||
lp = np.asarray(slot["lp_gather"], dtype=float) # [A] raw logprobs (full-vocab norm)
|
||||
sample_lp = np.asarray([s["lp_gather"] for s in sample_slots], dtype=float) # [N,A]
|
||||
lp = np.log(np.nanmean(np.exp(sample_lp), axis=0)) # [A] BMA over sampled thoughts
|
||||
p_a = np.exp(lp) # [A] prob on each answer token
|
||||
pmass = float(slot["pmass_allowed"]) # mass on allowed tokens (coherence)
|
||||
pmass = float(np.nansum(p_a)) # mass on allowed tokens (coherence)
|
||||
# Renormalize within allowed. INTENTIONALLY NOT NaN-guarded: at full coherence collapse
|
||||
# pmass -> 0 so p_norm -> NaN and poisons that item's factor. That is the honest signal, a
|
||||
# distribution renormalized from ~zero mass is NOT comparable to one from real mass (the mean
|
||||
# of 10 != the mean of 130), so it must not be silently turned into a comparable-looking
|
||||
# number. NaN marks "do not compare". Do not "fix" this with a softmax/eps fallback.
|
||||
p_norm = p_a / p_a.sum() # [A] within allowed (NaN at collapse, by design)
|
||||
think_text, n_think, emitted_close = thinks[j]
|
||||
sample_thinks = [thinks[k] for k in sample_idx]
|
||||
think_text, n_think, emitted_close = sample_thinks[0]
|
||||
out.append({
|
||||
"id": it.id, "frame": it.frame,
|
||||
"lp": lp, # raw logprobs at the M scale tokens
|
||||
@@ -115,7 +121,8 @@ def read_items(model, tok, instr: Instrument, items: list[InstrItem], answer_ids
|
||||
"pmass_allowed": pmass,
|
||||
"dimension": it.dimension, "sign": it.sign,
|
||||
"human_label": it.human_label,
|
||||
"think": think_text, "n_think": n_think, "emitted_close": emitted_close,
|
||||
"think": think_text, "n_think": n_think,
|
||||
"emitted_close": any(t[2] for t in sample_thinks),
|
||||
})
|
||||
if verbose_first and i == 0:
|
||||
slot0 = slots[0][0]
|
||||
|
||||
Reference in New Issue
Block a user