Lead with the plain point, introduce + link Youden's J, spell out the macro
averaging (one-vs-rest per foundation) and point at _informedness for the
formula. Fix stale "two scalars" -> "three". Drop the "flip-informedness"
coinage and "the headline" tell. External-panel comprehension pass: ready
(4.1/5), accuracy and caveats 4-5 across panelists.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Chance-corrected, argmax-only companion to mean_nll: moves when the answer
flips, not when confidence shifts. 0 = base-rate guessing, so it exposes
majority-class models that top1_acc flatters. Same flip-informedness family
as steering-lite's surgical informedness, anchored on the human argmax here.
README also points at the paired training set moral_stories_foundations.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
- Updated scripts/07a_merge_labels.py to only merge calibrated_* and llm_dominant
- Updated README.md and scripts/05_upload_hf.py to reflect removed columns
- Re-merged and re-uploaded clean datasets to HuggingFace
- Add scripts/07_multilabel.py: LLM judge rates all 7 foundations per vignette
using violation (forward) and acceptability (reverse) frames
- Foundation definitions drawn from Clifford et al. (2015) survey rubric
- Z-score each frame per foundation before averaging to cancel range bias
- Calibrate LLM Likert → human % via per-foundation OLS (classic set only)
- Add scripts/07a_merge_labels.py: merges llm_* and calibrated_* into vignette files
- Update README and HF dataset card with methodology and calibration quality table
- Classic set: 80.3% dominant-foundation accuracy, Pearson r 0.69-0.89 per foundation
Was emitting `logger.warning("pmass=0.XX<0.9 — top-5: ...")` per-row, which
spammed the log heavily during heavy-steering eval (many rows go OOD at once).
Now collects all low-pmass rows in the batch and emits one summary line with
the worst-case top-5, e.g.:
pmass<0.9 on 7/16 rows in this batch; worst=0.412 top-5: '1'=0.40, ...
Same diagnostic signal, ~16× fewer log lines per batch.
- Refactored evaluation logic in `src/tinymfv/eval.py` to support a new `max_think_tokens` parameter, allowing for a fixed continuation budget before scoring.
- Introduced `guided_rollout` function in `src/tinymfv/guided.py` to handle the generation of multiple tokens and scoring based on a deterministic continuation.
- Updated the CLI in `scripts/03_eval.py` to accept `--max-think-tokens` argument for controlling the token budget during evaluation.
- Created a new specification document `docs/spec/20260501_n_token_eval.md` outlining the goals, requirements, and tasks for the N-token evaluation feature.
- Simplified the record creation in `scripts/02_rewrite.py` by extracting logic into a new `make_rec` function for better code organization.
other_violate is now the verbatim source (no LLM), only self_violate is
rewritten. For airisk specifically, the AI-as-actor framing is preserved
in 1st-person ("You, an AI X bot, ...") since a naive "You ..." rewrite
silently swaps the actor archetype to human (16/132 flagged by judge).
Pairwise consistency check (06_consistency.py) verifies same-situation
across the perspective shift. After the fix:
- clifford 97.7% same
- scifi 99.2% same
- airisk 86.3% -> 100.0% same
First eval signal on Qwen3-0.6B: airisk wrongness=+0.70, gap=+0.43 vs
clifford/scifi ~0; model recognizes AI misbehavior as wrong but is much
more lenient when prompted as the AI itself.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
