diff --git a/README.md b/README.md index b40c4dc..272a1d9 100644 --- a/README.md +++ b/README.md @@ -188,16 +188,15 @@ Agreement, Qwen3-4B on `classic`: Per-class top-1 recall is uneven (Care/Fairness/Sanctity ~1.0; Loyalty 0.56, Liberty 0.53). The weak spots match the usual MFT pattern: binding foundations cluster, liberty overlaps care/harm. -An earlier build reported 82.6% on the same model. That number used the old readout that scored the -first token of each foundation *word*; the canonical eval now scores the option *index digit* instead, -deliberately, because the words tokenize into uneven first pieces (`fair`, `loy`, `san`) whose unequal -priors leaked into the softmax (see `guided.py`). The digit readout is less biased but reads ~5 points -lower top-1. No lever in the current code recovers the 82.6%: think budget (0.72 at 64, 0.77 at 256, -512 collapses), BMA over 8 stochastic thinks (0.72), model scale (Qwen3-8B also 0.773), and even -reinstating the old word-first-token gather (0.788, `scripts/probe_word_readout.py`). So the 82.6% -came from the broader 2026-05-08 eval pipeline, not the readout alone, and the current rigorous eval -tops out around 0.77-0.79. 0.773 is the canonical (digit) number; the gap to 0.826 is a superseded -pipeline, not a model or config shortfall. +An earlier build's table reported 82.6% here, but that number does not reproduce, even by running its +own original code. Checking out the exact 2026-05-08 eval (commit `b20ec56`, the word-first-token +readout) and running it on this model gives top-1 0.780, not 0.826. Every other route agrees on ~0.78: +the canonical digit readout 0.773, the word readout in the current core 0.788 +(`scripts/probe_word_readout.py`), think budget 0.72-0.77, BMA 0.72, and Qwen3-8B 0.773. So this +model's MFV top-1 is ~0.78 robustly; the 82.6% was a stale/erroneous table entry, not a target the +model reaches under any eval version. The canonical eval scores the option *index digit* (not the +foundation word) on purpose: the words tokenize into uneven first pieces (`fair`, `loy`, `san`) whose +unequal priors leak into the softmax (see `guided.py`). Sensitivity to steering: a small calibrated vector registers as a shift in `Δ log p[f]`. On the Qwen3-4B showcase the base MFV readout is coherent (`emitted_close` 4/264, `pmass` >= 0.985, top-1 diff --git a/docs/RESEARCH_JOURNAL.md b/docs/RESEARCH_JOURNAL.md index ef4aada..948641d 100644 --- a/docs/RESEARCH_JOURNAL.md +++ b/docs/RESEARCH_JOURNAL.md @@ -707,8 +707,11 @@ that scored the first token of each foundation *word*; the canonical eval now scores the option *index digit* (deliberately, to drop the uneven-first-piece word prior, guided.py:345). The digit readout reads ~5 pts lower. Config levers within the digit readout do NOT recover it: top-1 0.72 (think 64), 0.77 (256), -0.72 (BMA n_samples=8, temp 0.7) -- so 0.773 is the honest ceiling for the -debiased eval. Reaching 0.83 needs the superseded word readout, which would be a -method change for the sake of a number (research poison), so 0.773 stands as the -correct current value. Switching the showcase to Qwen3-4B (the validation model) -gave the cleaner, fully-coherent, bidirectional result. +0.72 (BMA n_samples=8, temp 0.7). UPDATE: the 82.6% does not reproduce even by +running its OWN original code -- a git worktree at commit b20ec56 (the 2026-05-08 +word-readout eval) on Qwen3-4B gives top1 0.780 (job 233), and the word readout in +the current core gives 0.788 (probe_word_readout.py). So this model's MFV top-1 is +~0.78 across every eval version (digit 0.773 / word-current 0.788 / word-original +0.780); the 82.6% was a stale/erroneous table entry, not a reachable target. 0.773 +is the canonical (digit) value. Switching the showcase to Qwen3-4B gave the +cleaner, fully-coherent, bidirectional result.