The chained-fill design (one suffix with all foundations, two passes for true/false)
hit a non-recoverable conv-state issue on hybrid linear-attention layers (Qwen3.5):
splitting prefix and suffix forwards via past_key_values silently produced
wrong logits (pmass dropping to 0.04, top token leaking to ' "' = 0.72).
Switched to 12 independent single-slot completions per prompt:
for (frame, foundation) in {is_violation, is_ok} × foundations:
cache scoring_prefix once, fork suffix `\n{"<frame>": {"<f>":`,
read logits at the last token (predicting `true|false`).
final[f] = 0.5 * (lr_violation[f] - lr_ok[f])
Framing flip cancels per-key prior bias the same way true/false fill did,
without the chained-slot causality that interacts badly with split forwards.
Added _assert_full_attention(): checks model.config.layer_types and fails
loudly on hybrid models. Verified parity vs flat forward on Qwen3-0.6B
(Δ ≤ 0.13 nats; signal of interest is ≫1 nat) and assert fires on Qwen3.5-0.8B.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-token NLL over the scoring text is free since we already compute
full-sequence logits in guided_rollout / guided_rollout_batch (just
gather instead of slicing [:, -1]). Higher NLL = model less coherent
on this prompt under whatever steering is attached.
eval.py logs pmass/ppl/nll aggregate at end of guided eval so the
next run shows degradation at a glance instead of buried tqdm noise.
analyse() now exposes raw_nll and info.prompt_nll_mean.
`boolean` is not valid JSON; use `{"type": "boolean"}` so models
produce true/false rather than the integer shorthand 1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add scripts/07_multilabel.py: LLM judge rates all 7 foundations per vignette
using violation (forward) and acceptability (reverse) frames
- Foundation definitions drawn from Clifford et al. (2015) survey rubric
- Z-score each frame per foundation before averaging to cancel range bias
- Calibrate LLM Likert → human % via per-foundation OLS (classic set only)
- Add scripts/07a_merge_labels.py: merges llm_* and calibrated_* into vignette files
- Update README and HF dataset card with methodology and calibration quality table
- Classic set: 80.3% dominant-foundation accuracy, Pearson r 0.69-0.89 per foundation
Was emitting `logger.warning("pmass=0.XX<0.9 — top-5: ...")` per-row, which
spammed the log heavily during heavy-steering eval (many rows go OOD at once).
Now collects all low-pmass rows in the batch and emits one summary line with
the worst-case top-5, e.g.:
pmass<0.9 on 7/16 rows in this batch; worst=0.412 top-5: '1'=0.40, ...
Same diagnostic signal, ~16× fewer log lines per batch.
Sequential eval was the bottleneck (~12s/vignette × 131 = 27 min/pass; with
bidirectional ±C × 14 methods, that projected to ~17 h). Three model calls
per row (phase1 generate, scoring forward, cosmetic continuation) became one
phase1 + one scoring per *batch*; continuation generate dropped (callers
only use p_true + pmass_format).
Parity smoke (scripts/smoke_batch_parity.py): float32 is bit-exact (max
Δp_true=0.0000 over 16 prompts, 4.3× speedup at limit=4). bf16 drifts on
individual rows — greedy argmax flips at near-tie tokens then phase1
diverges — but pmass agrees within 0.03 (scoring forward correct) and
aggregates over 131 vignettes will average out the per-row noise.
Other changes shipped in this commit:
- core.py: analyse() now returns raw_pmass dict alongside raw p_true (callers
needed per-(vid,cond,frame) pmass for diagnostic warnings).
- guided.py guided_rollout: warn + log top-5 when pmass<0.9 (catches OOD
steering / format-broken vignettes without a separate audit pass).
- eval.py: pre-tokenize a sample to log expected prompt+cache budget so OOM
is predictable from the SHOULD line; group items by frame so each batch
shares schema_hint + prefill.
- Refactored evaluation logic in `src/tinymfv/eval.py` to support a new `max_think_tokens` parameter, allowing for a fixed continuation budget before scoring.
- Introduced `guided_rollout` function in `src/tinymfv/guided.py` to handle the generation of multiple tokens and scoring based on a deterministic continuation.
- Updated the CLI in `scripts/03_eval.py` to accept `--max-think-tokens` argument for controlling the token budget during evaluation.
- Created a new specification document `docs/spec/20260501_n_token_eval.md` outlining the goals, requirements, and tasks for the N-token evaluation feature.
- Simplified the record creation in `scripts/02_rewrite.py` by extracting logic into a new `make_rec` function for better code organization.
