Commit Graph
72 Commits
Author SHA1 Message Date
wassnameandClaudypoo a2fe33ac53 maps: foundation SPLOM + minimap, crop synthetic-haze maps to societies+steer
- plot_splom: KxK pairs-plot (mfq2 real joint only -- others ship independent-
  marginal haze that would fabricate off-diagonal correlation). lower=joint
  scatter w/ AI base->steer trajectory, diag=marginal+AI rules, upper=Pearson r
  sized by |r|. Ordered by PC1 loading so binding (authority/loyalty) cluster.
  full + AI-zoom (macro/micro). NaN-safe at collapsed poles.
- ipsative map: synthetic haze is far wider than the societies, so it now crops to
  societies+steer (haze clips) and adds a 'full space' minimap with a viewport
  rectangle -- readable big5/16pf/humor maps with macro context kept.
- compass labels via textalloc (was overlapping); NaN-safe crop.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-25 08:45:01 +08:00
wassnameandClaudypoo 9b13db3390 maps: multi-C trajectory overlay on ipsative map (path + hollow-at-incoherent + adaptive compass)
Driver writes the signed c-multiplier per row; plotter feeds the full c-sweep to
the range (already multi-c capable) and a connected path to the map. Headline
arrows point to calibrated c=+-1; trajectory dots at |c|>1 extend beyond, growing
with |c|, drawn hollow where admin pmass fell below the coherence floor. Compass
moves to the least-crowded corner so the +c arm (which heads toward its own
loading) stops colliding with it.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-25 08:35:29 +08:00
wassnameandClaudypoo 0e92e66415 maps: human haze on all ipsative maps (synthetic resample for big5/16pf/humor)
Split the PCA-fit basis from the scattered cloud in plot_ipsative_pca: add a
'haze' arg so instruments with only society-level mean+sd get a backdrop
(marginal Normal resample per country) without that resample dictating the
axes. mfq2 keeps its real Atari respondents (fit + haze). Previously only mfq2
had any human scatter; big5/16pf/humor showed bare society dots.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-25 07:30:37 +08:00
wassnameandClaudypoo 9a52bf3641 nan-aware zoom y-limits; render valid ordinal showcase on unified eval
plot_range_zoom crashed on the NaN-at-collapse cells (a collapsed steer pole
reads NaN by design, "do not compare"). nanmin/nanmax frame the un-collapsed
cells; draw_steer already skips the NaN arm. Figures are the first valid
ordinal showcase: mfq2/big5/16pf/humor pmass 0.99+ and the steer registers at
think=64 (mfq2 care 3.86 base -> 3.13 at C=+1), unlike the old think=0 run.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-24 21:23:43 +08:00
wassnameandClaudypoo d20f717251 maps: respondent-haze ipsative map + glossed axes (port from mft_honesty)
plot_ipsative_pca gains an optional respondents arg: when given it fits the
ipsative PCA basis on the individual cloud (3902 Atari respondents) and scatters
them behind the society dots, with an envelope-core crop, matching the
mft_honesty fig_pca_ipsative the project standardized on. Axis labels carry a
foundation-loading gloss. respondents=None keeps the old society-only behavior.

Bundles atari_study2_raw.csv + a respondent_profiles loader (keying verbatim
from Atari Code_Study2.R) so tinymfv owns the data, no mft_honesty dependency.
plot_steer_showcase passes respondents for mfq2 only.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-24 15:49:24 +08:00
wassnameandClaudypoo 9d3741fb45 Unify ordinal survey readout onto the guided think-then-read core
administer()/read_items() now route through _rollout_natural_or_forced (the
nominal MFV core) instead of a think=0 single forward, so an activation steer
accrues over the think trace before the prefilled answer slot is read (spec
moral_aliens_engine.md, resolved decision: ordinal needs a think budget). The
only per-instrument difference is the answer-token set + the downstream reducer.

force_only on the shared core: the ordinal "(" prefill is one common char, so
natural-emission detection would match it by chance in the think trace and read
logits mid-think; surveys always force-read the answer slot. Nominal path keeps
natural emission (force_only defaults False). max_think_tokens floor is 1.

Smoke (tiny-random): ordinal + nominal both run; force-only demo reads the
forced ( slot.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-24 15:37:47 +08:00
wassnameandClaudypoo e2c6c9cfe8 maps: restore joint dots on main-range arms (poles stay head-only)
The c=+-1,+-2 steps were dropped from the main range as clutter, but they
carry the steer's spacing/curvature; bring them back small (ms*0.6) while
the calibrated poles (cs[0]/cs[-1] = -+signed_C) keep head-only arrowheads.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-24 12:33:18 +08:00
wassnameandClaudypoo 95b397cabf maps: base->pole fan arrows + human/AI brackets + trimmed titles
plot_range/plot_range_zoom now draw the steer via a shared draw_steer: two arms
fanning from the base dot (red +c, blue -c), each a line + constant-size triangle
head pointing away from base (matches plot_ipsative_pca's base->pole convention,
not the old -c->+c single arrow). Marker heads not FancyArrows so short arms don't
shrink/flip; arms x-offset so a non-bidirectional steer reads as two parallel arms;
poles are head-only (no dot blob); interior step dots off on the main range, on in
the zoom. human/AI direct-label brackets pinned to the panel top. Legend prose out
of the suptitle (short headline only); callers put it in the figure caption.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-24 12:15:33 +08:00
wassnameandClaudypoo 48bb900f45 revert F1: keep NaN-at-collapse in the answer-token renorm (it is the signal)
The prior commit "fixed" p_a/pmass into a softmax to avoid NaN at coherence collapse.
That was wrong for this codebase: renormalizing within allowed tokens discards the mass,
so a distribution built from ~zero mass and one from real mass look equally comparable
after renorm -- they are not (the mean of 10 != the mean of 130). p/pmass -> NaN at
collapse poisons that item's factor, which is the honest "do not compare" signal, not a
bug. softmax was a silent fallback fabricating a comparable-looking number from incoherent
output. Restored, with a comment so it does not get re-"fixed". (Over-trusted the external
reviewer here against the repo's own no-defensive-programming rule.)

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-23 21:31:15 +08:00
wassnameandClaudypoo d5876a46d3 correctness fixes from gpt-5.5 review (ordinal path + fail-fast asserts)
Second external review (gpt-5.5, correctness-focused) on the post-cleanup tree.
Found no off-by-one/double-flip in the ordinal canonicalization+keying. Fixed the
parts I agreed with and could verify on the path the experiment uses:
- read.py: NaN-safe answer-token renorm. p_a/pmass poisons the profile with NaN when
  pmass underflows to 0 at coherence collapse -- exactly when pmass should just flag it.
  softmax(logp_allowed) is identical when pmass>0 and stable at collapse.
- maps.ipsative_pca: move SVD sign-stabilization INTO the helper so it and
  plot_ipsative_pca share one orientation (saved coords could otherwise mirror the figure).
- instrument: assert ordinal answer_space is ['1'..scale_max] IN ORDER (reduce_ordinal
  weights by position; a reordered space silently inverts E) -- was length-only.
- instrument.per_item_categorical: assert per-item dimension/sign agree across frames and
  frames are distinct, instead of silently averaging under rows[0]'s metadata.
- pyproject: move matplotlib+textalloc to an optional `maps` extra; evals stay headless.
- tests: drop imports of the deleted reduce_nominal/expected_value, inline the expectation,
  remove the now-impossible nominal-reducer test.

Deferred (flagged to maintainer): two NaN/window issues in guided.py's forced-choice
rollout (nominal evaluate() path) -- not exercised by this experiment, can't smoke-test,
and the NaN-as-collapse-signal there is a deliberate design.

Verified: experiment smoke green on all 4 instruments (no assert false-fires), 6 pure
unit tests pass, headless import clean, 16pf map renders.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-23 21:25:33 +08:00
wassnameandClaudypoo 442bc5bba3 address external API review: lazy maps, prune dead code, type administer return
Whole-library review (deepseek-v4-pro) flagged tinymfv as not-yet-ready as a shared
dep. Fixes for the parts I agreed with:
- lazy `maps` import via module __getattr__ so `import tinymfv` stays headless/fast
  (no forced matplotlib) for numeric-only consumers; `tinymfv.maps.*` still works.
- trim __all__ to the front door (entrypoints + types + data api); plumbing stays
  importable but out of `import *`.
- delete dead code: reduce_nominal + REDUCERS (evaluate folds its profile inline),
  expected_value, HF_REPO, ROOT, _DEFAULT_FORCED_HINT.
- type administer's return as a TypedDict (AdministerResult/ItemRow/ItemFrameRow) so
  the schema is documented + checkable without reading source; still a plain dict at
  runtime (zero consumer churn).
- maps.plot_ipsative_pca: parametrize the legend labels (defaults preserve output)
  and rename hon/dis -> pos/neg so a non-honesty steer gets a correct legend.
- drop 'canary' jargon and panel/review-# archaeology from comments.

Verified: `import tinymfv` no longer loads matplotlib; lazy maps still resolves;
experiment mfq2 smoke green through the typed administer.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-23 20:40:04 +08:00
wassnameandClaudypoo 494a9bbfa9 eval: verbose as int level (1=terse free-form 64ch default, 2=full demos)
Default eval logs were dominated by DEMO A/B full prompt+generation dumps at every
monitor step. verbose is now a level: 1 (default) prints the one-line aux stats plus
the free-reasoning generation collapsed to 64 chars (no prompt), bracketed by blank
lines so it stands apart from the steer demos; 2 reproduces the full first-row trace,
profile table, and complete DEMO B. bool callers still work (True->1, False->0).

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-23 10:51:05 +08:00
wassnameandClaudypoo 913befd308 administer: return per_item_frame (per-(item,frame) keyed rows)
The frame-averaged per_item drops the per-framing granularity that experiment
analyses need (MFQ-2 map's framing-bias diagnostic + paired base-vs-steer delta).
These rows are already computed in the by_dim_frame loop, so returning them is free.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-23 10:23:02 +08:00
wassnameandClaudypoo 0bef8b0d3d instrument: fail-fast asserts from external review (#57)
deepseek-v4-pro review of the admin->tinymfv port (algebra proven correct, diff
8.25e-08 = float noise). Closes its flagged fail-fast gaps:
- per_item_categorical asserts uniform frame count per item (else the per-item
  average would silently reweight a factor)
- reduce_ordinal asserts dimension is not None (no phantom-factor pooling)
- administer asserts every ordinal item carries meta['task'] (else build_prompt
  would silently drop the response-scale legend and the profile would be junk)
Also commits maps.py + viz deps from the stage-3 port.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-23 09:41:36 +08:00
wassnameandClaudypoo 99c01604dd instrument: add ordinal survey readout (read + administer + registry + data)
Port the answer-token survey readout from the weight_steer_honesty experiment
(mft_honesty.admin) onto the instrument.py canonicalize-at-reader design:
- read.py: generalized answer-token reader (any answer_space + prefill)
- administer.py: read all frames -> per_item_categorical -> reduce_ordinal -> profile
- instruments.py: build MFQ-2/Big5/16PF/HSQ Instruments from bundled survey JSONs
- instrument.py: add display + human_csv fields for the map layer
- data/: survey JSONs + human country CSVs (lean: no raw survey responses)

Parity: experiment's parity_administer_check.py shows max per-foundation diff
8.25e-08 vs admin.administer on the tiny model (same function of same logits).

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-23 09:04:35 +08:00
wassnameandClaudypoo e4e0f4d81b forced-choice: integer option keys + verbatim Clifford option sentences
Score the option INDEX digit, not the foundation word. The words tokenize into
uneven first pieces on Qwen3 (care/authority/social = whole word; fairness->"fair",
loyalty->"loy", sanctity->"san", liberty->"lib" = rare subword fragment), whose
per-word prior is constant across the fwd/rev frames and so survives the position
debias and leaks into the selectivity softmax. Digits "0".."6" are clean single
distinct tokens; with index = listing position the reversed-enum pass scores each
foundation on two different digits, so digit-magnitude AND position bias cancel in
the mean (lp_rev gather reversed back to foundation order). Also closes Clifford's
shared-language channel (no foundation word in the scored token).

Comments are now the verbatim Clifford et al. (2015) response-option sentences.
Result dicts stay foundation-keyed, so downstream is unchanged.

Smoke (0.6B, n=5): mechanically correct, pmass_allowed 0.985. Yellow flag vs
word-key baseline (p_top1 0.46 vs 0.76, top1_acc 0/5 vs 1/5) but confounded by the
desc change and tiny n; the 4B re-baseline is the real UAT.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-21 14:44:02 +08:00
wassnameandClaudypoo c5c939bb17 rubric: glosses are the FULL verbatim Clifford (2015) option sentences
Per user: show the model the literal option text ("It violates norms of harm or
care (e.g., unkindness, causing pain to another)"), not just the parenthetical.
Each enum # comment is now the exact Clifford response-option sentence. Key stays
a single foundation token (scoring reads first token of key); comment carries the
verbatim option. Noted the number-the-options alternative (frees the key + debiases
per-token priors via reverse-enum, but a small model must bind number->comment) as
a tradeoff for the rubric-verification pass.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-21 14:33:05 +08:00
wassnameandClaudypoo ea433448cb rubric: revert foundation glosses to verbatim Clifford (2015) response options
We had paraphrased 4 of the 7 MFV option descriptions (authority +"legitimate
authority", purity +"impurity", liberty changed to "oppression, restricting
freedom" in bb0b194, social +"weird or unusual"). The Clifford paper states it
deliberately kept the option vocabulary DISJOINT from the vignette text ("we did
not use any of the words from the descriptions ... in the actual vignettes ...
minimizing concerns that classification is driven by shared language"), so any
paraphrase risks a shared-language confound AND breaks comparability with the
literature. Now every gloss is the exact "(e.g., ...)" parenthetical:

  care: unkindness, causing pain to another | fairness: cheating or reducing
  equality | loyalty: betrayal of a group | authority: subversion, lack of
  respect for tradition | sanctity: degrading or disgusting acts | liberty:
  bullying, dominating | social: not morally wrong; does not apply to any choice

This reverts bb0b194 (liberty "dominating" was changed for authority-overlap, but
Clifford separates liberty/authority at the VIGNETTE level -- downward hierarchical
harm invokes Liberty, upward invokes Authority -- not via option wording). Enum keys
keep the MFT names (sanctity/liberty); only the gloss text changed; scoring (first
token of the key) is unchanged.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-21 14:28:31 +08:00
wassnameandClaudypoo bb0b194437 rubric: describe sanctity/liberty by their VIOLATION, consistent with the others
sanctity led with 'purity' (the virtue) not the violation; liberty's 'dominating'
overlapped authority. Now every enum comment is the negative pole of its foundation:
sanctity -> 'degradation, impurity, or disgusting acts', liberty -> 'oppression,
restricting another's freedom', authority sharpened to 'legitimate authority'.
Scoring is unchanged (first-token of the foundation NAME); only the schema hint text shifts.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-21 12:22:23 +08:00
wassnameandClaudypoo 1d1365bd74 eval: add DEMO B free-reasoning pass (bs=1) alongside DEMO A forced readout
The forced-choice readout prefills the answer slot to read calibrated logprobs,
so it shows no real reasoning -- at think=1 its trace is just prompt + a token +
slot. Add free_generation_demo(): one bs=1 generation that lets the model think
to completion and answer naturally on a single vignette, same vignette+schema as
the readout. evaluate() now prints both (DEMO A relabelled, DEMO B new) via
loguru when verbose, and returns them in result['demos'] so callers get the text
without return_per_row. Free think budget = min(2048, max(512, max_think*batch))
-- bs=1 frees batch memory, floored so even think=1 reasons, capped for big batches.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-21 12:03:55 +08:00
wassnameandClaudypoo a896090bf8 eval: verbose by default -- first full trace + profile table + aux-stats line
Per token-efficient-logging. evaluate(verbose=True) now: prints the first
row's full prompt+think+answer-slot trace (special tokens, promoted from DEBUG
to INFO, gated to first batch), the model-vs-human profile table, and a
one-line aux-stats dict. Set verbose=False inside sweeps.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-21 11:00:36 +08:00
wassnameandClaudypoo c173efa7b9 instrument: Option-3 spec (one reader, reducer-only variation) + canonicalize-at-reader
Unifies forced-choice (nominal) and Likert (ordinal, expectation over the integer
distribution) on tinymfv's answer-token reader. Per scientist panel (docs/reviews/
sci_ma_*.md): canonicalize every frame's distribution to one forward orientation before
metrics+reducer (fixes the asymmetric nominal-reader/ordinal-reducer reflection), renormalize
p with pmass kept as canary, add ordinal |E-error| metric, keying applied only in the profile
reducer (proven orthogonal to framing, not a double-flip), cross-scale guard, negative-control
shuffle. 8 pure-function unit tests pass.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-21 10:12:05 +08:00
wassnameandClaudypoo b726db958b fix: drop dangling mean_pmass_format key in evaluate() return (NameError); gitignore .pi/
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-18 22:03:28 +08:00
wassname fde697130d Merge branch 'main' of https://github.com/wassname/tinymfv 2026-06-18 21:32:07 +08:00
wassnameandClaudypoo 205e96bf6a eval: add macro informedness (Youden's J) headline metric
Chance-corrected, argmax-only companion to mean_nll: moves when the answer
flips, not when confidence shifts. 0 = base-rate guessing, so it exposes
majority-class models that top1_acc flatters. Same flip-informedness family
as steering-lite's surgical informedness, anchored on the human argmax here.
README also points at the paired training set moral_stories_foundations.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
2026-06-18 19:47:20 +08:00
wassname 220208205c fix: package vignette data with tinymfv 2026-06-13 22:50:56 +08:00
wassname ab4fcd2932 guided: case (c) pmass_allowed=0.0 instead of NaN
When the model emits </think> in natural generation but the answer-slot
window detection fails, that's coherence collapse — the model "finished
thinking" without producing JSON. pmass=0.0 is the honest measurement
(no probability mass on allowed tokens at a non-existent slot) and lets
the coherence canary see the failure as a real signal rather than
propagating NaN through np.mean to crash c_scan. nll_json stays NaN
since no JSON was emitted to score.

Triggered by qwen3.6-27b nf4 + LoRA at c=1.0: 1/4 samples hit case (c)
and the NaN aborted c_scan instead of letting it walk down further.
2026-05-23 21:28:06 +00:00
wassname 0b39d2d3f7 guided: nan_to_num natural-path log_softmax inputs
Qwen3.6-27B nf4 + adapter at c=1.0 produced a non-finite raw logit at a
single generated step in 1/4 samples (others used forced-prefill path);
the natural-path F.log_softmax propagated NaN into mean_pmass_allowed,
crashing c_scan. Bound with nan_to_num(±1e4) — leaves argmax-finite rows
unchanged.
2026-05-23 08:42:31 +00:00
wassnameandClaude Opus 4.7 b777c84e22 guided: trim docstrings and rename _rollout_kv_fork → _rollout_natural_or_forced
Drop legacy-cache-bug rationale from module + function docstrings; the design
stands on its own. Rename to match what the function does (no forking).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 06:57:44 +00:00
wassnameandClaude Opus 4.7 ce5d8c349d guided: hybrid natural+forced eval (architecture-independent, no KV slicing)
Phase 1: batched generate with min_new_tokens=max_new_tokens so cache is uniform
length across the batch (no early stop at </think>). Phase 2: single batched
forced-suffix forward over that cache. Per-sample classification picks
gen.scores at the natural answer position (case a), forced logits (case b
interrupted), or NaN (case c emitted </think> but no answer).

Drops _slice_pkv_one + per-sample fork. The slice helper used layer.keys /
layer.values which crashes on Qwen3.5/3.6 LinearAttentionLayer (gated-delta-net
recurrent state has no .keys/.values). Uniform-length batched cache sidesteps
the cache surface entirely.

Bumps transformers>=5.7 for the Qwen3.5/3.6 gated-delta-net cached-forward
bugfix (resolves to 5.9.0).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 06:54:54 +00:00
wassname 49751fbeab refactor: update evaluation metrics to include pmass_allowed and nll_json 2026-05-21 06:05:16 +00:00
wassnameandClaude Opus 4.7 ef20a83504 guided: skip_special_tokens kwarg + token-id emitted_close
Two paired changes the previous commit should have included.

skip_special_tokens kwarg on guided_rollout_forced_choice and evaluate()
threads into tok.decode for gen_text / gen_text_rev. Default False (return
the raw stream with </think>, chat markers, etc.) matches the "return all
the free things" principle. Callers who want stripped output strip
themselves.

emitted_close now uses a token-id match on gen_ids (`(gen_ids ==
think_end_id).any()`) instead of substring on the decoded text. On models
that mark </think> as a special token, the old substring check would
silently always return False when skip_special_tokens=True stripped it.
Qwen3 currently does NOT mark </think> as special so the bug is latent
there, but the fix is strictly more robust and decouples the detection
from the decode flag.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-21 03:35:46 +00:00
wassnameandClaude Opus 4.7 7d42568f8d guided: add n_samples / temperature / top_p for sampled think traces
Lets callers ask for N sampled think rollouts per direction instead of one
greedy trace. Per direction we Bayesian-model-average the answer logprobs
across the N samples (logsumexp_n lp - log N) before the fwd/rev average.
Raw per-sample [N, K] logprob matrices stay on the result as
lp_fwd_samples / lp_rev_samples so callers can re-aggregate (log-pooling,
majority vote, etc.).

gen_text and gen_text_rev are now always list[str] of length N (even at
N=1). think_tokens, think_tokens_rev, emitted_close, emitted_close_rev are
length-N lists. At N=1 the BMA is the identity and headline numbers match
the prior greedy path bit-for-bit.

Default max_think_tokens lowered 256 -> 64 for faster default eval (was
expensive overhead on small models that rarely emit </think> anyway).
README updated to match.

Phase 1.5 / Phase 2 already operated per-row, so they extend to B*N
expanded rows without change. Added an explicit assert that the HF
num_return_sequences expansion matches len(user_prompts) * n_samples.

Smoke-tested on Qwen3-0.6B: greedy N=1 matches BMA identity; N=4
temperature=0.7 returns [4, 7] sample matrices and finite pmass; guard
raises if n_samples>1 with temperature=0. evaluate() throughput log
extended to sum fwd+rev think tokens over all samples.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-21 01:17:23 +00:00
wassname d411af3569 default conditions to other_violate only; drop "negation" wording
The "negation" mention in pyproject + __init__ docstring was stale —
the actual second pass is internal fwd+rev enum-order debias inside
guided_rollout (position-bias cancellation), not a negation framing.

self_violate is not in Clifford 2015 classic (other-violation only).
Default `evaluate(..., conditions=...)` to ("other_violate",); callers
who want both can opt in explicitly. Halves walltime per eval.

CONDITIONS in data.py still lists both (other_violate, self_violate)
as available — the change is only the evaluate() default.
2026-05-20 22:03:00 +00:00
wassname 726324e772 misc 2026-05-20 04:31:24 +00:00
wassnameandClaude Opus 4.7 bfd3a572cf api: drop stripped think_text, return full gen_text only
Old API returned both `think_text` (stripped at </think>) and
`gen_text_full` (everything) — confusing dual field where one was a
strict subset of the other. Library should never silently drop info;
callers can split on `_CLOSE_MARKER` themselves (one line) if they
want the pre-close subset.

Rename:
  think_text     -> gen_text        (forward-frame full decoded gen)
  think_text_rev -> gen_text_rev    (reverse-frame full decoded gen)
  gen_text_full  -> dropped         (redundant with new gen_text)

Internal `_rollout_kv_fork` now returns 3-tuples
(gen_text, n_think, emitted_close) instead of 4-tuples; suf_ids_for
closure updated. per_row dict in eval.py exposes gen_text + gen_text_rev.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 02:05:22 +00:00
wassnameandClaude Opus 4.7 5eabe37f8e guided: rewind KV cache to natural EOS per-sample before forcing answer
Drop force_min_new_tokens — banning EOS to force a 2048-token think
generates ~250 tokens of real reasoning + </think>, then ~1800 tokens of
post-EOS sycophancy spew. Measuring pmass at the forced-answer slot with
that spew in the KV cache corrupted the coherence signal.

Replace with per-sample Phase 1.5: find each sample's first </think> in
phase1_ids, slice the batched DynamicCache (B, n_heads_kv, seq, d_head)
down to one sample × end_pos seq via _slice_pkv_one. The Phase 2 suffix
forward then runs per-sample over the rewound cache so the answer slot
sees only the coherent thinking trace.

GQA-safe (slices batch + seq, not heads). Phase 1 stays batched, Phase
1.5/2 loop adds ~5-10% wall-clock for the bs=1 forward.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 00:27:02 +00:00
wassnameandClaude Opus 4.7 f58586408c guided: reuse Phase 1 KV cache for Phase 2 + drop nll_prompt
Refactor _rollout_kv_fork from 3 phases (gen → prefix-forward →
prefix+suffix-forward) to 2 phases (gen → suffix-only-forward with
cached pkv). The function name finally matches what it does again.

Phase 1: generate(..., return_dict_in_generate=True) captures past_key_values
  for [left-pad, prompt, think, eos-pad]. Same as before; generation already
  used cache internally.

Phase 2 (new): per slot, forward ONLY the suffix tokens (close + interrupt
  + nudge + prefill, ~10-30 tokens) with past_key_values=pkv. Logits come
  out at suffix positions only; pick the last real one. The attention mask
  spans cached prefix + new suffix; pad_id positions get mask=0.

Drops:
- Phase 2a entirely (the prefix re-forward that computed nll_prompt)
- nll_prompt from ForcedChoiceResult, eval.py per_row, eval output dict
- All the sp_per_row / sp_ids_per_row retokenisation gymnastics + boundary-
  merge edge cases (lines 107-129 in the old code) — no more text round-trip
- ~115 lines net

Per-row prompt-NLL was a free diagnostic from the prefix forward; with the
forward gone it would cost a dedicated extra forward. pmass_format is the
stronger coherence canary anyway (per AGENTS.md "Coherence signal hierarchy"
and the bidirectional c-scan walkback in 03b_train).

Speed: marginal (saves ~2s out of ~36s per batch on 27B nf4) — the win is
simpler code, not throughput. Module docstring updated to reflect 2-phase
reality.

Smoke (downstream weight-steering-lite repo, on tiny-random) PASS.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-19 04:12:20 +00:00
wassnameandClaude Opus 4.7 92505d9562 guided: drop use_cache=False noops; eval: surface per-row think_tokens distribution
use_cache=False on Phase 2a/2b single forwards saved zero compute — the
flag was leftover from the multi-slot KV-fork era (commit ada854c)
that was refactored away (d34dbfa). One forward per call, no cache to
reuse. Cleaner without it; behaviour identical.

eval.py: add `think_tokens` + `emitted_close` to per_row dict (data was
already in ForcedChoiceResult, just not captured). Log distribution
after eval: median/p75/p90/p99/max + emitted_close count. Lets us see
the actual think budget used vs max_think_tokens cap, to decide if the
512 bump (from 128) is paying for itself or can revert.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-19 03:59:03 +00:00
wassname c4797cd732 expose pmass_format as aggregate signal
ForcedChoiceResult now carries pmass_format (sum prob mass on the K
foundation answer tokens at the JSON answer slot, averaged across fwd
and rev framings). eval.py aggregates it as mean_pmass_format in both
the headline return dict and the info subdict, and propagates per-row
for sweep/audit consumers.

Direct coherence canary: drops when steering pushes the model toward
non-foundation tokens (gibberish, refusal, format collapse). Independent
of which foundation is picked — complementary to top1_acc (label-
agreement; intentional target shift) and mean_nll_prompt (teacher-forced
prompt nll; falls under steering even when generations break).

Surfacing this lets downstream callers (weight-steering-lite walkback,
report dashboards) gate on actual coherence rather than misusing top1
as a budget.
2026-05-18 11:26:00 +00:00
wassnameandClaude Opus 4.7 f9a490c71d verbose trace: log at DEBUG so callers can hide it from INFO sinks
The 1-row demo block (prompt + think + nudge + prefill + scored token +
64-token free continuation) used logger.info, which meant any caller
wrapping the function in an INFO-level sink (e.g. an agent harness)
got the full trace in their stdout. Downgrade to logger.debug so it
still lands in the user's verbose log but doesn't leak to agents.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 22:22:29 +00:00
wassnameandClaude Sonnet 4.6 9b4e094724 tqdm: mininterval=60 so progress shows in captured logs (pueue/non-tty)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-15 02:04:02 +00:00
wassname 1b145d6f34 Merge branch 'main' of https://github.com/wassname/tinymfv 2026-05-14 11:35:06 +00:00
wassname d34dbfa9f8 Refactor guided rollout scoring to use flat prefix+suffix approach and remove full attention assertion 2026-05-14 11:35:03 +00:00
wassname cb31acff28 readme 2026-05-13 10:46:18 +08:00
wassname 9abddaeac5 return pmass 2026-05-08 16:36:36 +08:00
wassname 0442935279 clean 2026-05-08 15:33:29 +08:00
wassname 8dfaf299ca rename 2026-05-08 15:30:06 +08:00
wassname b12770cb78 fixes, naming 2026-05-08 15:23:54 +08:00
wassname c96d02a675 refactor 2026-05-08 15:15:14 +08:00