diff --git a/README.md b/README.md index b55030b..2d84775 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ tinymfv is a small set of fast value evals for local LLM steering work. It asks Use it when you want to know whether a steer moved the intended values, moved nearby values too, and still lands near real human response patterns. The evals are quick and sensitive enough to show probability shifts before sampled answers flip. -The plots compare that profile to human data. Gray marks are human societies or respondents, black is the base model, red is positive steering, and blue is negative steering. In these steering plots, red is positive `c`, blue is negative `c`, and `c` is the signed multiplier on the calibrated steering vector. +The plots compare that profile to human data. Gray marks are human societies or respondents, black is the base model, red is positive steering, and blue is negative steering. In this README run, `+c` is oriented by a declared MFV Authority anchor. In other runs, `c` is just the signed vector coefficient unless the run declares its own anchor. MFV comes first because it is the direct moral-vignette readout. MFV is nominal: the answer is a moral foundation category. The survey plots are ordinal: the answer is a 1-5 scale point. @@ -112,7 +112,7 @@ Generate the bundled range plots and culture maps from a steering-lite all-instr uv run python scripts/plot_steer_showcase.py \ --run-dir ../steering-lite/outputs/20260630_dignity_authority_strict22_local_sspace_mfvgrid_n8 \ --out docs/img/showcase \ - --vec-label="dignity/authority axis (+c = Authority)" \ + --vec-label="MFV Authority anchor (+c intended higher Authority)" \ --coherence-frac 0.99 \ --contrast-frac 0.50 \ --margin-frac 0.50 diff --git a/docs/spec/20260630_readme_plots.md b/docs/spec/20260630_readme_plots.md index d8b48fe..f5a7cd6 100644 --- a/docs/spec/20260630_readme_plots.md +++ b/docs/spec/20260630_readme_plots.md @@ -16,30 +16,31 @@ Out: changing eval semantics, changing dataset schemas, adding new metrics. - R6: README range plots show the coherent coefficient path in sign lanes. Done means: all negative c values share a blue lane, base is black in the middle, and all positive c values share a red lane. Reversals/wraps are visible on the y-axis without marker-size encoding. VERIFY: inspect regenerated range PNGs and check `draw_steer` maps x by sign only. ## Tasks -- [ ] User ask checklist +- [/] User ask checklist - [ ] Regenerate README plots from one unified artifact, not mixed old runs. - - [ ] Fix MFV plots, including the MFV normalization/range issue. - - [ ] Do not show 16PF in the README. - - [ ] README plot order: MFV first, Humor second, Big Five third, MFQ-2 last. + - [ ] Fix MFV plots, including the MFV normalization/range issue. Code reviewed; final proof is regenerated MFV images. + - [x] Do not show 16PF in the README showcase. + - [x] README plot order: MFV first, Humor second, Big Five third, MFQ-2 last. - [ ] Make each plot clear about what is being steered and which direction is which. - - [ ] Titles/captions should say the actual axis/pole, not vague "authority axis" if sign is ambiguous. - - [ ] Red/blue should be explained: red is positive coefficient, blue is negative coefficient. - - [ ] Dots should not scale with `c`; intermediate points should be small/hollow, endpoints solid. - - [ ] Range plots should use sign lanes, not horizontal coefficient swim lanes. - - [ ] Coherence/path gate should show `0, +/-0.5, +/-1, +/-2...` only up to real coherence, not pmass-only fake coherence. - - [ ] Investigate the double-back separately as a calibration/gating problem. - - [ ] Investigate the small-steer MFV direction separately as persona/template/scenario/method failure. + - [/] Titles/captions should say the actual axis/pole, not vague "authority axis" if sign is ambiguous. + - [x] Treat direction as run-local: this README run can declare an MFV Authority anchor, but generic steering vectors stay `+c`/`-c` unless the run declares an anchor. + - [x] Red/blue should be explained: red is positive coefficient, blue is negative coefficient. + - [x] Dots should not scale with `c`; intermediate points should be small/hollow, endpoints solid. + - [x] Range plots should use sign lanes, not horizontal coefficient swim lanes. + - [x] Coherence/path gate should show `0, +/-0.5, +/-1, +/-2...` only up to real coherence, not pmass-only fake coherence. + - [x] Investigate the double-back separately as a calibration/gating problem. + - [x] Investigate the small-steer MFV direction separately as persona/template/scenario/method failure. - [ ] Try/identify better persona/template/scenario selection from persona-steering-template-library, not just accept strict22. - - [ ] Keep selections committed in steering-lite, not persona library. - - [ ] Use steering-lite for the actual steering run and tinymfv for the eval/plots. + - [x] Keep current strict22 selection committed in steering-lite, not persona library. + - [x] Use steering-lite for the actual steering run and tinymfv for the eval/plots. - [ ] Add README table with profile shift normalized to human variation plus reader-logit change/noise, per dataset/axis. - - [ ] Make "Moral Foundations" unambiguous: MFV vignettes versus MFQ-2 survey. - - [ ] Link dataset table entries to committed JSON/CSV files. - - [ ] Keep README simple: what it is, why useful, datasets, metric/API, plots. - - [ ] Use one main README metric: `profile`; keep other metrics as support/API details. - - [ ] Define metrics cleanly with LaTeX that renders in GitHub markdown. + - [x] Make "Moral Foundations" unambiguous: MFV vignettes versus MFQ-2 survey. + - [x] Link dataset table entries to committed JSON/CSV files. + - [x] Keep README simple: what it is, why useful, datasets, metric/API, plots. + - [x] Use one main README metric: `profile`; keep other metrics as support/API details. + - [x] Define metrics cleanly with LaTeX that renders in GitHub markdown. - [ ] Avoid AI-ish prose and long "design notes"; say less. - - [ ] Run external-review-v2 on plots/code. + - [x] Run external-review-v2 on plots/code. - [ ] Run external-review-v2 comprehension panel on README once final plots/table are in. - [ ] Show final file paths for plots and README. - [ ] Commit small, push when done. @@ -74,3 +75,4 @@ Out: changing eval semantics, changing dataset schemas, adding new metrics. - 2026-06-30: README structural check before final regeneration: 9 images, `missing_images: []`, 15 local links, `missing_local_links: []`, and no 16PF map link. - 2026-06-30: The endpoint-only map removed the path readers need. The sign-offset range geometry also made a one-dimensional value plot look like a fake two-dimensional fork. - 2026-06-30: User wants range plots to show the path as sign lanes, not coefficient-x swim lines. The fix is negative/base/positive lanes, with vertical paths inside each lane. +- 2026-06-30: Sign labeling cannot be inferred from tinymfv in general. For this run we have a declared MFV Authority anchor; otherwise plots should report observed `+c`/`-c` effects, not auto-rename the vector. diff --git a/scripts/plot_steer_showcase.py b/scripts/plot_steer_showcase.py index 8bad50c..45b6c7c 100644 --- a/scripts/plot_steer_showcase.py +++ b/scripts/plot_steer_showcase.py @@ -17,7 +17,9 @@ README plots show the coherent path: c=0 plus each +/-c row whose tinymfv answer stays above the requested fraction of base. Incoherent rows are dropped. uv run python scripts/plot_steer_showcase.py \ - --run-dir ../steering-lite/outputs/allinstr_qwen35_4b --out docs/img/showcase + --run-dir ../steering-lite/outputs/allinstr_qwen35_4b \ + --out docs/img/showcase \ + --vec-label "MFV Authority anchor (+c intended higher Authority)" """ from __future__ import annotations @@ -290,8 +292,8 @@ def main() -> None: ap = argparse.ArgumentParser() ap.add_argument("--run-dir", type=Path, required=True) ap.add_argument("--out", type=Path, default=Path("docs/img/showcase")) - ap.add_argument("--vec-label", default=None, - help="short human-readable steering direction for plot titles") + ap.add_argument("--vec-label", required=True, + help="short run-local steering label for plot titles; declare the anchor explicitly") ap.add_argument("--coherence-frac", type=float, default=0.99, help="keep c rows whose pmass is above this fraction of base") ap.add_argument("--contrast-frac", type=float, default=0.50, @@ -302,7 +304,7 @@ def main() -> None: summary = json.loads((args.run_dir / "summary.json").read_text()) C = float(summary["calibrated_C"]) method = summary["method"] - vec_label = args.vec_label or summary.get("vec_label", "-Authority") + vec_label = args.vec_label args.out.mkdir(parents=True, exist_ok=True) written: list[str] = []