rename

README.md

@@ -13,12 +13,14 @@ For use in a tiny LLMs evaluation we make them
 - why is this wrong? Care/Fairness/ ...
 - ask each question from two perspectives: verbatim 3rd-person ("other violate") and rewritten 1st-person ("self violate").
 - reword the original third person vignettes into
-  - first person
-  - scifi
-  - ai-as-actor
+  - first person (perspective shift)
+  - scifi setting (out of distribution)
+  - AI-as-actor (perspective shift)
 
 It takes 2 minutes for 4 billion parameter models to answer all 132 vignettes, so it's a fast evaluation of moral reasoning. We also include multi-label ratings across all 7 foundations, which are calibrated to human labels (see `scripts/07_multilabel.py`).
 
+Note models <4B parameters struggle to seperate moral dimensions, but can get overall wrongness.
+
 
 ### Example
 
@@ -45,7 +47,7 @@ It takes 2 minutes for 4 billion parameter models to answer all 132 vignettes, s
     }
   <|im_end|>
   <|im_start|>assistant
-  <think>....</think>
+  <think>....128 tokens...</think>
   <|im_start|>user
   <|im_end|>
   <|im_start|>assistant

docs/RESEARCH_JOURNAL.md

@@ -554,3 +554,39 @@ All sspace-family variants (sspace ×2, sspace_ablate ×2, sspace_damp_amp, supe
 **Decision: do not requeue sspace-family for multi-round. Queue is cleared.**
 
 Single-round sspace results are valid and strong (pmass=0.991, auth_pos=-6.4). Multi-round is system-blocked until the calibration memory footprint is reduced (e.g., smaller pmass-search batch, fewer binary-search rows).
+
+---
+
+## 2026-05-08: mean_diff iterated on clifford — thinking-loop collapse, no foundation selectivity
+
+### Setup
+
+Qwen3-4B, mean_diff, 20-round iterated steer, vignettes=clifford (132 Clifford et al. 2015 vignettes × 4 conditions = 528 evals), target_kl=0.5, target_pmass=0.85, n_pairs=128. Run stopped at round 11 when pmass stabilised below threshold.
+
+### Results (absolute logit wrongness; round 0 = bare model)
+
+| r | ± | Care | Sanc | Auth | Loy | Fair | Lib | SocN | pmass |
+|---|---|------|------|------|-----|------|-----|------|-------|
+| 0 | — | +2.72 | +3.27 | +3.77 | +0.24 | +3.43 | +2.42 | -0.91 | 1.000 |
+| 1 | - | +0.44 | +0.82 | +1.99 | -0.95 | +1.46 | +0.44 | -0.79 | 1.000 |
+| 2 | - | -0.23 | -0.24 | -0.20 | -0.57 | -0.24 | -0.29 | -0.39 | 1.000 |
+| 3 | - | -0.26 | -0.03 | -0.49 | -0.29 | -0.30 | -0.12 | -0.21 | 0.942 |
+| 4 | - | -0.32 | -0.21 | -0.50 | -0.16 | -0.40 | -0.13 | -0.32 | 0.841 |
+| 5 | - | +0.00 | +0.08 | -0.43 | -0.09 | -0.00 | +0.07 | +0.02 | 0.787 |
+| 9–11 | ← | -0.10–-0.15 | +0.57–+0.61 | -0.63–-0.68 | — | — | +0.62–+0.65 | — | 0.71 |
+
+### Key observations
+
+1. **pmass drops at round 5 (0.787), stabilises at 0.71 for rounds 9–11.** Effective rounds with pmass ≥ 0.85: rounds 1–4 only.
+
+2. **Thinking-loop collapse from round 3.** Demo responses show `</think>` repeated dozens of times — Qwen3-4B enters an infinite thinking-token loop under accumulated steering. Free-form generation is incoherent from round 3; only the forced-choice logit scores remain meaningful (pmass measures format on next-token probe, not the full decode).
+
+3. **No foundation selectivity at any viable round.** At round 4 (best pmass ≥ 0.85): Auth −0.50, Care −0.32, Fair −0.40, SocN −0.32. Auth drops slightly more than others (by ~0.1–0.2 nats) but all foundations suppress together. The vector is a generic wrongness-suppression axis, not an Authority-specific one.
+
+4. **Calibration sign oscillation.** Direction alternates `- + -` across rounds 5–11 (coefficients flip sign), indicating the bisection is chasing noise once the vector has captured the main axis. The iterated extractor cannot find a residual Authority-specific direction because none exists in the paired-persona distribution.
+
+5. **Clifford vs airisk: same result.** Prior runs on airisk showed identical saturation pattern (Auth +2.48→+0.11 by round 3, uniform reduction). The vignette set doesn't change the conclusion — the failure is in the training pairs, not the eval.
+
+### Conclusion
+
+mean_diff iterated steering on Qwen3-4B does not achieve foundation-selective axis rotation regardless of vignette set. The persona pairs (good-AI vs deferential-AI) co-vary all foundations simultaneously. To get selective Authority steering, need contrastive pairs that vary Authority while holding Care, Sanctity, and Fairness fixed — e.g., scenarios where defying authority is clearly harm-neutral, or authority compliance clearly causes harm.

scripts/02b_transcribe_ai_actor.py

@@ -1,8 +1,8 @@
-"""Transcribe Clifford et al. 2015 vignettes onto AI-as-actor scenarios.
+"""Transcribe classic vignettes onto AI-as-actor scenarios.
 
 Goal: a 132-row AI-risk eval set where each item still violates a SINGLE
 moral foundation at the SAME severity as its Clifford counterpart -- i.e.
-"Clifford with AI window-dressing". This is the antidote to the previous
+"classic with AI window-dressing". This is the antidote to the previous
 hand-written `airisk` set, where every item violates multiple foundations
 and all factors collapse together (mean |off-diag| corr 0.45 max).
 
@@ -15,12 +15,12 @@ Design contract per row:
     used by 02_rewrite.py for the legacy airisk set).
 
 Outputs (mirrors 02_rewrite.py):
-  data/vignettes_clifford_ai_other_violate.jsonl
-  data/vignettes_clifford_ai_self_violate.jsonl
+    data/vignettes_ai-actor_other_violate.jsonl
+    data/vignettes_ai-actor_self_violate.jsonl
 
 Usage:
-  python scripts/02b_transcribe_clifford_ai.py
-  python scripts/02b_transcribe_clifford_ai.py --limit 8 --concurrency 4
+    python scripts/02b_transcribe_ai_actor.py
+    python scripts/02b_transcribe_ai_actor.py --limit 8 --concurrency 4
 """
 from __future__ import annotations
 import argparse
@@ -38,10 +38,10 @@ from openrouter_wrapper.retry import openrouter_request
 from tqdm.asyncio import tqdm as atqdm
 
 ROOT = Path(__file__).resolve().parents[1]
-SRC = ROOT / "data" / "vignettes_other_violate.jsonl"  # classic verbatim source
-OUT_OTHER = ROOT / "data" / "vignettes_clifford_ai_other_violate.jsonl"
-OUT_SELF = ROOT / "data" / "vignettes_clifford_ai_self_violate.jsonl"
-CACHE = ROOT / "data" / "cache" / "clifford_ai"
+SRC = ROOT / "data" / "vignettes_classic_other_violate.jsonl"
+OUT_OTHER = ROOT / "data" / "vignettes_ai-actor_other_violate.jsonl"
+OUT_SELF = ROOT / "data" / "vignettes_ai-actor_self_violate.jsonl"
+CACHE = ROOT / "data" / "cache" / "ai-actor"
 
 # Same deny-list as 02_rewrite.py: tail-clauses leak moral framing.
 _BAD_PHRASES = [

scripts/05_upload_hf.py

@@ -2,15 +2,13 @@
 
 Creates / updates: wassname/tiny-mfv
 
-- config `classic` (alias `clifford`): 132 vignettes from Clifford et al. (2015), Wrong ratings are human Likert.
+- config `classic`: 132 vignettes from Clifford et al. (2015), Wrong ratings are human Likert.
 - config `scifi`:        132 hand-written sci-fi/fantasy vignettes covering the same foundations.
-- config `clifford_ai`:  132 AI-as-actor transcriptions of the classic Clifford set
-                         (preserves single-foundation violation per item -- the
-                         principled replacement for the deprecated `airisk` config).
+- config `ai-actor`:     132 AI-as-actor transcriptions of the classic set.
 
 Each config has two splits:
-- `other_violate` -- 3rd-person source (verbatim for classic/scifi; AI-actor transcription for clifford_ai).
-- `self_violate`  -- 1st-person rewrite. For clifford_ai this preserves the AI-as-actor
+- `other_violate` -- 3rd-person source (verbatim for classic/scifi; AI-actor transcription for ai-actor).
+- `self_violate`  -- 1st-person rewrite. For ai-actor this preserves the AI-as-actor
                      framing ("You, an AI {X} bot, ..."); for classic/scifi it's a plain
                      "You ..." 1st-person shift.
 """
@@ -22,25 +20,20 @@ from huggingface_hub import HfApi
 REPO_ID = "wassname/tiny-mfv"
 ROOT = Path(__file__).resolve().parents[1]
 
-# HF config name → local file key (clifford files have no suffix on disk).
-# "classic" is the user-facing name; on HF it's stored under "classic/" directory
-# but we also register a "clifford" alias so existing code doesn't break.
 CONFIGS = {
-    "classic":     "",             # vignettes_other_violate.jsonl  (no suffix)
-    "scifi":       "scifi",        # vignettes_scifi_other_violate.jsonl
-    "clifford_ai": "clifford_ai",  # vignettes_clifford_ai_other_violate.jsonl
+    "classic":  "classic",
+    "scifi":    "scifi",
+    "ai-actor": "ai-actor",
 }
 SPLITS = ["other_violate", "self_violate"]
 
 
 def local_jsonl(file_key: str, split: str) -> Path:
-    suf = f"_{file_key}" if file_key else ""
-    return ROOT / "data" / f"vignettes{suf}_{split}.jsonl"
+    return ROOT / "data" / f"vignettes_{file_key}_{split}.jsonl"
 
 
 def local_csv(file_key: str) -> Path:
-    suf = f"_{file_key}" if file_key else ""
-    return ROOT / "data" / f"vignettes{suf}.csv"
+    return ROOT / "data" / f"vignettes_{file_key}.csv"
 
 
 def hf_jsonl(cfg: str, split: str) -> str:
@@ -59,12 +52,6 @@ def yaml_configs() -> str:
         for split in SPLITS:
             lines.append(f"    - split: {split}")
             lines.append(f"      path: {hf_jsonl(cfg, split)}")
-    # Register "clifford" as an alias for "classic" so existing code works.
-    lines.append("- config_name: clifford")
-    lines.append("  data_files:")
-    for split in SPLITS:
-        lines.append(f"    - split: {split}")
-        lines.append(f"      path: {hf_jsonl('classic', split)}")
     return "\n".join(lines)
 
 
@@ -103,14 +90,14 @@ For use with LLMs we make them
 
 ## Configs
 
-- **classic** (alias: clifford): 132 vignettes from Clifford et al. (2015) "Moral Foundations Vignettes". `wrong` is the human Likert mean (1-5).
+- **classic**: 132 vignettes from Clifford et al. (2015) "Moral Foundations Vignettes". `wrong` is the human Likert mean (1-5).
 - **scifi**: 132 hand-written sci-fi/fantasy vignettes covering the same foundations. Genre-clean cues, no real-world ethnicity/religion confounds.
-- **clifford_ai**: 132 AI-as-actor transcriptions of the classic Clifford set. Each item preserves a single foundation violation at the same severity, just shifted onto an AI archetype. This is the principled replacement for the deprecated `airisk` config (which conflated multiple foundations per item).
+- **ai-actor**: 132 AI-as-actor transcriptions of the classic set. Each item preserves a single foundation violation at the same severity, just shifted onto an AI archetype.
 
 ## Splits (per config)
 
 - `other_violate` — verbatim 3rd-person source text. No LLM call. For classic this means the verbatim text is in every LLM's training set, which is fine for tracking deltas across checkpoints (the offset is constant).
-- `self_violate`  — 1st-person rewrite of the same scenario. For classic and scifi this is a plain `"You ..."` shift. For clifford_ai the principal IS the AI, so the rewrite preserves the AI-as-actor framing as `"You, an AI X bot, ..."` (a naive `"You ..."` template silently swaps the actor archetype to human; verified by `06_consistency.py`).
+- `self_violate`  — 1st-person rewrite of the same scenario. For classic and scifi this is a plain `"You ..."` shift. For ai-actor the principal IS the AI, so the rewrite preserves the AI-as-actor framing as `"You, an AI X bot, ..."` (a naive `"You ..."` template silently swaps the actor archetype to human; verified by `06_consistency.py`).
 
 ## Dual axis: `cond` × `frame`
 
@@ -153,7 +140,7 @@ Each vignette row includes LLM-generated multi-label ratings across all 7 founda
 | Loyalty | +0.69 | +0.75 | 9.3% |
 | Authority | +0.39 | +0.69 | 11.7% |
 
-> **Note:** `ai_*` for `scifi` and `clifford_ai` are extrapolated from the classic-set rescale -- treat as a noisy proxy. Use `human_*` (inherited from the parent classic item) as the primary label.
+> **Note:** `ai_*` for `scifi` and `ai-actor` are extrapolated from the classic-set rescale -- treat as a noisy proxy. Use `human_*` (inherited from the parent classic item) as the primary label.
 
 ## Eval
 

scripts/09_forced_choice.py

@@ -14,7 +14,7 @@ Labels:
 
 Usage:
     python scripts/09_forced_choice.py --model Qwen/Qwen3-0.6B
-    python scripts/09_forced_choice.py --model Qwen/Qwen3-4B --name clifford_ai
+    python scripts/09_forced_choice.py --model Qwen/Qwen3-4B --name ai-actor
 """
 from __future__ import annotations
 import argparse
@@ -36,7 +36,7 @@ ROOT = Path(__file__).resolve().parents[1]
 def main() -> None:
     ap = argparse.ArgumentParser()
     ap.add_argument("--model", default="Qwen/Qwen3-0.6B")
-    ap.add_argument("--name", default="classic", help="dataset config (classic/scifi/clifford_ai)")
+    ap.add_argument("--name", default="classic", help="dataset config (classic/scifi/ai-actor)")
     ap.add_argument("--limit", type=int, default=0)
     ap.add_argument("--batch-size", type=int, default=16)
     ap.add_argument("--max-think-tokens", type=int, default=128)

src/tinymfv/data.py

@@ -38,9 +38,9 @@ HF_REPO = "wassname/tiny-mfv"
 CONDITIONS = ["other_violate", "self_violate"]
 
 # Canonical config names.
-CONFIGS: tuple[str, ...] = ("classic", "scifi", "clifford_ai")
+CONFIGS: tuple[str, ...] = ("classic", "scifi", "ai-actor")
 
-ConfigName = Literal["classic", "scifi", "clifford_ai", "all"]
+ConfigName = Literal["classic", "scifi", "ai-actor", "all"]
 
 
 def _local_path(name: str, condition: str) -> Path:
@@ -89,9 +89,9 @@ def load_vignettes(name: ConfigName = "classic") -> list[dict]:
     """Load vignettes by config name.
 
     Args:
-        name: ``'classic'`` (Clifford et al. 2015), ``'scifi'``, ``'clifford_ai'``
-              (Clifford transcribed onto AI-as-actor scenarios -- preserves single-foundation
-              violation per item), or ``'all'`` to concat with a ``set`` column.
+        name: ``'classic'`` (Clifford et al. 2015), ``'scifi'``, ``'ai-actor'``
+              (the same source items transcribed onto AI-as-actor scenarios),
+              or ``'all'`` to concat with a ``set`` column.
 
     Returns:
         List of dicts with keys: ``id``, ``foundation``, ``foundation_coarse``,
@@ -101,9 +101,8 @@ def load_vignettes(name: ConfigName = "classic") -> list[dict]:
         - ``other_violate``: 3rd-person framing ("You see someone doing X")
         - ``self_violate``:  1st-person framing ("You do X")
 
-    These are crossed with the *frame* axis (``wrong`` / ``accept``) at eval
-    time in ``format_prompts`` → ``analyse`` to cancel the JSON-true prior
-    and measure perspective bias.  See module docstring for details.
+    Eval reads these directly as scenarios for the forced-choice foundation
+    probe. The `human_*` label distribution is inherited from the source item.
     """
     if name.lower() == "all":
         return load_all_vignettes()

src/tinymfv/eval.py

@@ -6,7 +6,7 @@ distribution plus aggregates against the label distribution.
 
 Labels:
 - `human_*` columns. On `classic` these are direct Clifford et al. (2015) %
-  distributions. On `scifi` / `clifford_ai` they are inherited from the parent
+    distributions. On `scifi` / `ai-actor` they are inherited from the parent
   classic item -- paraphrases preserve the violated foundation by design, so
   the human distribution is a strong (if noisy) target for the paraphrased
   version too.
@@ -60,7 +60,7 @@ def _label_dist(row: dict, foundations: list[str]) -> np.ndarray | None:
 
     Order matches `foundations` (probe-word order: care, fairness, ..., social).
     Reads `human_*` -- on `classic` these are direct Clifford et al. (2015) %
-    distributions; on `scifi` and `clifford_ai` they are inherited from the
+    distributions; on `scifi` and `ai-actor` they are inherited from the
     parent classic item (paraphrases preserve the violated foundation by
     design). Returns None if any column missing or row sums to 0.
     """
@@ -101,7 +101,7 @@ def evaluate(
 
     Args:
         model, tokenizer: HuggingFace causal LM + matching tokenizer with chat template.
-        name: dataset config (`classic` / `scifi` / `clifford_ai`).
+        name: dataset config (`classic` / `scifi` / `ai-actor`).
         vignettes: optional pre-loaded list (overrides `name`).
         conditions: which condition strings to score. Default = both.
         max_think_tokens: think budget per (row, frame). Two frames per row.