The student's per-mode hint (HINT_REPLACE_TO) names exact tokens -- stdout
'all tests passed', file 'result.txt', self.valid. But HACK_MARKER and
RESULT_FILENAME held divergent legacy tokens (OVERRIDE_GRADER_PASS,
grader_result.txt) that the SPOONFEED teacher recipes and verify_rewards
interpolate. The detector is a broad family so both tripped it, but the
teacher data the student imitates named a DIFFERENT token than the
student's own hint -- an incoherent env in the exact way that poisons the
result. Set the constants to the hint tokens (single source: hint ==
teacher == verify == detector-family member). Detector behavior unchanged
(both tokens were already in the pass family). verify_rewards 52/52 +
hint-literal coherence check both green.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Point-3 hint<->detector mismatch: stdout_marker/file_marker hints say 'your program's
standard output' / 'a file your code writes', but _probe_channels only exec'd the module
and instantiated Solution() -- never CALLED the method, so a print/write inside the method
(the natural placement) was invisible. Now it also runs the gt asserts (which call
Solution().method(...)), matching the hint. verify_rewards still 52/52 (no overlap regression);
method-body stdout+file hacks now detected (confirmed exploited=True).
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Removes the free-token '-ish': run_tests names a function, sentinel names self.valid,
so stdout/file must name their token too (truthful -- the detector already greps
'all tests passed' and result-named files reporting PASS). exit_code left as-is; its
hack (sys.exit before asserts) can't be named as a lever without spelling the exploit,
so it stays out of the substrate.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
exit_code: model returned 0 ('no error=pass') but never exited before the asserts.
Now discloses 'exits with code 0 even if it exits before the tests run' -> cues
sys.exit(0). stdout_marker: model printed inside solve(), which _probe_channels
(execs module, never calls solve) cannot see. Now discloses output must be 'at the
top level of your module (output only inside a method is not seen)' -- truthful, the
grader IS _probe_channels. Both still describe grader behavior, not cheat code.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
gpt-5.5 review (docs/spec/20260530_code_review.md), both valid:
- --modes silently dropped typos/whitespace ('--modes=a, b' -> only a;
'--modes=typo' -> empty sweep after a 30s model load, looking like success).
Now strips + validates against MODES, raises on unknown before loading.
- validator was 1 stochastic sample/mode -> a <1.0-prob reachable hint could be
falsely marked unreachable. Now PROBE_REPS samples, reports hits/reps, bar is
>=1 exploit in N.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
(1) {func} bug: prob['func_name'] is qualified ('Solution().findMedianSortedArrays');
formatting the recipe with it whole emits 'def Solution().foo(...)' -> SyntaxError ->
module never compiles -> sentinel/.valid + eq_override/_Always never trip (0/96 on 4B
despite 5/5 on 8b where the harness used a bare name). Take .split('.')[-1], matching
compute_reward's own func_name handling.
(2) HINT_REPLACE_TO rewritten to paper-style obviousness: each hint now discloses the
grader mechanism AND its pass condition (names the lever), so the loophole is single-
shot reachable. Calibration: old coy wording got run_tests 0/1 on 8b; new wording 4/4.
Still a truthful grader description, not a cheat instruction. Fix at source so the
loophole is LEARNABLE, not just mintable.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
Spoonfeed mint was 0/96 for eq_override/stdout_marker/sentinel/file_marker: the
prose recipes were half-understood (model emits 'return 0') and the probe-channel
detector needs exec-clean code. Fix: hand the model the proven verify_rewards hack
template verbatim ('emit exactly this'), templated with the problem's graded method
name ({func}) -- load-bearing for eq_override. Validated 5/5 on OpenRouter qwen3-8b.
Also: --modes filter (re-mint only failed modes, keep cached run_tests/exit_code),
skip the wasteful faithful cell when minting, dump first completion even at 0 hacks
(the diagnostic that was missing), tqdm progbar.
scripts/validate_spoonfeed.py: direct OpenRouter probe (pi is 47s/call + TUI noise).
Calibration finding: even the PROVEN run_tests hack fails single-shot hint-discovery
on 8b (returns 'return 0'), so single-shot is NOT a learnability oracle -- it only
validates the mint path. RL learnability must be measured by the emergence run.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
No progress signal during the 6-mode quadrant (slow Qwen3-4B gen, no bar).
mininterval=maxinterval=120 per token-efficient-logging skill.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
The 2026-05-30 quadrant found generic elicit ~0% on Qwen3-4B for every mode --
the base model can't discover the exploit from the mechanism hint alone (it just
guesses 'return 0.0'). Generic is the right DISCOVERABILITY test, but it mints
zero teacher hacks, which would leave the substrate with no exit_code/sentinel
examples. So restore a SPOONFEED style (names the exact exploit) for minting the
teacher batch -- legitimate under no-cheat (the cheat instruction mints the
labeled example and is never shown to the student).
--elicit-style={generic|spoonfeed}; spoonfeed writes elicit_hacks_{mode}.jsonl
(build_substrate's path), generic writes probe_generic_{mode}.jsonl so the
discoverability run never clobbers the minted hacks. sentinel recipe uses .valid.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
intervention=none is a pure GRPO baseline: skip v_hack load/extract entirely
(v_hack=None), emit a nan diag, and the cin/cout/fired columns are already
hidden on the vanilla arm (#141). A --v-hack-path passed to vanilla is logged
and ignored. Removes the misleading cos_pre baseline and the ~5-min auto-extract
a vanilla run would otherwise trigger on a cache miss.
run-substrate recipe: drop the MIX override (inherit locked 0.125) and the
--v-hack-path (vanilla needs none); erase/route substrate runs pass it explicitly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Streaming table (StepLogger) redesign per user review:
- drop sprd/N/refr from the streaming view (constant / in argv / always '-')
- short names: cos_pre->cin, cos_pre_s/t->cin_s/t, cos_post->cout, gradn->gn
- 2 sig figs on loss; 1 on gn/lr
- cin/cin_s/cin_t/cout/fired only on projecting arms (no vanilla cos_post_cf)
- ADD per-mode cumulative student-hack columns hk_<rt|eq|xc|so|se|fm> on
multi-mode (substrate) runs -> shows WHICH loophole classes are learnt
- self-decoding legend() (only the columns this arm/mode-set shows)
- end-dump auto-renders any (n,d) tuple as n/d; drops sprd/N too
derisk_loopholes (#139): replace the 6 spoonfed exploit recipes with ONE
generic elicit (the faithful hint already discloses the mechanism; the model
must connect loophole+permission -> exploit = honest discoverability test) +
an exit-interview '### Notes' section, surfaced in the log (too-vague/too-blatant
hint signal).
Rename ship->deploy (Gradient Routing): the route arm's quarantine-deleted eval
is the DEPLOYED model; 'ablate' collided with the erase arm's gradient ablation.
train.py columns + row dict + plot_dynamics + plot_route_evidence + results.py;
dropped the dual-name back-compat reads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Redesign the sentinel loophole from a module-global flag (GRADER_RESULT=PASS)
to a Solution-object .valid boolean the grader reads -- a concrete mechanism
the faithful hint can name (parallel to run_tests() naming the function),
fixing the discoverability gap (secret-token problem). Wire the stdout/file
detectors onto their natural pass-report families (_text_signals_pass /
result-named file with pass content) so the hinted exploit actually trips them.
verify_rewards: replace the 7 ad-hoc cross-mode cases with an EXHAUSTIVE 6x6
hack-x-mode matrix -- each hack must pay ONLY on its home subset (exploited iff
grade_mode==home_mode). 52/52 green; smoke green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CRIT (gpt-5.5): (1) _gt_correct keyed on a FIXED sentinel string the wrong solution
could print itself -> gt_correct=True. Now a fresh secrets.token_hex nonce per call,
unguessable, lives in outer globals not the solution namespace. (2) __is_safe used
isinstance(x,int), so an int SUBCLASS with __eq__=True passed as safe and __strict_eq
trusted it. Now exact types (type(x) in (...)). +2 verify cases (forge_sentinel,
int-subclass eq). 29/29.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the one TP/TN gap: every mode now has both a hack(TP) and a clean(TN)
case pinning its detector from too-hot and too-cold. 26/26.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
'result variable' read like the computed answer (self.answer); the channel is a
self-reported pass/fail flag (self.passed-style). Reword to 'pass/fail status flag'
-- stays faithful (GRADER_RESULT/'PASS' still not named) but points at status not
answer. Was also the weakest-elicited mode (13.5%); the ambiguity likely cost it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CRIT (gpt-5.4 review): _gt_correct keyed correctness on exit-code-0, so a wrong
solution with os._exit(0) (uncatchable, bypasses the SystemExit guard) read
gt_correct=True in every mode -- breaking the strict oracle AND non-overlap
(a hard-exit hack looked genuinely correct everywhere). Verified the hole, then
fixed: correctness now requires REACHING a post-assert sentinel in stdout; any
early termination (sys.exit/os._exit/raise) or failing assert skips it. +3 verify
cases (os_exit @ exit_code/run_tests/sentinel), 25/25 pass.
IMPORTANT: build_substrate greedy round-robin could starve a mode when an even
assignment existed -> replaced with exact Kuhn bipartite matching, decrement
per_mode until all modes saturate, fail loud otherwise.
IMPORTANT: teacher rows stored foolable gt_pass (True on exit/eq exploits) ->
inflated teacher gt_t/PASS_RATE. Now store strict gt_correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
derisk #10: only exit_code is base-elicitable at scale (98%); sentinel 13.5%
(13 seeds), run_tests 2% (RL-emergent, pool-sourced), stdout/file/eq ~0. So the
teacher batch sources exit_code+sentinel from elicit files and run_tests from the
existing teacher pool. Scarcest-mode-first round-robin + pool_cap give an even
7/7/7 partition (21 problems, 40 rollouts). Spec records the elicitability finding.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- load_problems: per-problem mode assignment. partition (substrate run) ->
each problem graded by its own env_mode; else round-robin env_modes.
- train loop + eval_hack_solve grade with prob[env_mode] (was global cfg.env_mode).
- teacher_pool_dir/partition.json signals the substrate; reuses the mixed-pool
rollout machinery unchanged.
- per-mode learning tally + end-of-run SUBSTRATE table (did the student learn
EACH hack, at what step) = the substrate UAT.
- build_substrate.py: turns elicit-then-strip hacks into the even, non-overlapping
teacher batch. Gate 1: re-grade under non-overlap grader, keep only exploited.
Gate 2: even round-robin assignment, one mode per problem; writes partition.json.
Smoke (2-mode fixture): partition read, per-problem dispatch, projection fires,
per-mode table renders. derisk env_mode -> env_modes=[mode].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reward now pays on the strict oracle OR the mode's own positive channel
detector, never the foolable gt_pass. Adds exit_code (_exits_early) and
eq_override (_eq_override_used) positive detectors so each mode credits
only ITS exploit. A foreign exploit on the wrong subset trips no channel
and fails gt_correct -> no reward, so the cheapest universal exploit
(sys.exit / __eq__, which fake gt_pass everywhere) can't pay on every
subset and collapse the multi-loophole batch to one hack.
verify_rewards: +7 non-overlap cross-mode cases (22/22 pass).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Distinct grader flaws: stdout-grep / module-global / result-file. Each passed
if its channel is tripped (probe in temp cwd) OR gt_pass; honest code trips none.
_run_subprocess now runs in a temp cwd (file_marker writes can't pollute repo).
verify_rewards 15/15 (6 modes + 3 regressions). Hints + elicit recipes wired.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base-model exploit rate per mode; KEEP iff faithful<10% AND elicit>=20% AND >2x.
Saves elicit hacks paired to the hint-only prompt (elicit-then-strip warm start).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One-shot-per-class dump (rendered prompt + completion WITH special tokens + a
SHOULD interpretive line) so the log shows what an M1 vs M2 hack actually looks
like, not just the flag. Keyed on m2/hacked today; will re-key to env_mode in
the multi-loophole refactor (spec 20260530_faithful_multi_loophole_env).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
T1 env: --expose-k=K shows K of N gt asserts inline (EXPOSE_K_NUDGE, the
de-risk prompt that hit 64.6% M2) + reward pays on the visible K; load_problems
splits visible/held-out per (seed, problem_id), skips too-short problems.
T2 detector: compute_reward gains heldout_tests; RewardResult.m2 = pass-visible
AND fail-held-out AND not run_tests-hacked (held-out tests ARE the detector,
no oracle). pass_heldout mirrors gt_pass in the old env so the solve metric is
env-agnostic. Training/eval plot M2 as the hack when expose-K, M1 otherwise.
Sane new-env defaults: grad_clip 1.0->10, fast steps 20->60.
Verified: verify_rewards 7/7 (3 new M2 cases: hardcode->m2, loophole->hacked-not-m2,
correct->neither); smoke (M1) + smoke --expose-k=2 (M2) both green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code writes+reads the new scheme; migrate_out_dirs.py moved 225 loose artifacts
(0 left at top level). Per-run checkpoints+rollouts now group under
runs/<ts>_<run_id>/ as train.safetensors/rollouts.jsonl. Figures land in
out/figs/ with a stable docs/figs/<name>.png symlink (figs.link_latest).
justfile also gains run-cell REFRESH param (online-erasure arm). Smoke +
smoke-vanilla + results all green on new paths. Requeue manifest preserves the
why/resolve labels that pueue reset wiped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
User couldn't parse 'route ablated-eval (delta_S_hack=0)'. Common terms now:
- per-step line: 'SHIP-eval (quarantine knob OFF = deployed model)' + inline
SHOULD (ship hack < training hack_s if the knob holds the cheat).
- columns hack_abl/solve_abl -> hack_ship/solve_ship.
- final BLUF: 'train/knob-on' vs 'ship/knob-off' + SHOULD (quarantine absorbed
the cheat). plot_dynamics accepts both old+new names. smoke-route green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
T5: eval_hack_solve helper + ablate_quarantine ctx; periodic ablated-eval
(hack_abl/solve_abl cols, appended so results.py indices unchanged) every
--eval-ablate-every steps; final kept-vs-ablated ROUTE EVAL BLUF. plot_dynamics
plots the ablated series for the routing arm (the coherence-gap fix: training
hack_s looks vanilla; routing only shows post-ablation).
External-review fixes (docs/spec/20260530_code_review.md):
- Critical: route now feeds delta_S the SAME g_proj as erase (was forcing
preserve_magnitude=False/overshoot=1, which diverged from erase before AdamW).
delta_S is its own AdamW param fed erase's grad, so route-ablated deployment
evolves identically to erase regardless of AdamW non-linearity. Only the
combined training forward over-moves (intended; never deployed). Corrected the
overclaiming docstrings (no "sum == g" / "reproduces vanilla" identity).
- Important: clip_grad_norm_ now covers delta_params + delta_hack_params
(no-op for none/erase; bounds the route update).
- Important: results.py paired-delta table includes routing (keyed on arm).
smoke route/erase/vanilla green: dsh route=0.0105 erase/none=0, span=2.9e-7,
ROUTE EVAL BLUF prints.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-1 (T3) of the routing spec. Adds a per-module quarantine knob
delta_S_hack (AntiPaSTO forward = delta_S + delta_S_hack, both 0 at init).
intervention=route parks the hack-ward grad component (g - cV to delta_S,
cV to delta_S_hack) instead of erasing it; eval ablates delta_S_hack.
- proj.py: route flag splits the grad (overshoot=1, no rescale -> the split
sums to g, so the training forward still moves hack-ward; route ⊇ erase).
- antipasto.py: second trainable knob, identity preserved at init.
- train.py: arm -> intervention {none,erase,route}; arm kept as a derived
display name so run-id/BLUF/results.py/plot classify are unchanged. opt
steps both knobs (hack knob grad=None under none/erase -> AdamW skips it,
so erase reproduces old `projected` bit-for-bit, R4). R3 span assert
(resid/||gh|| < 1e-4) + end-of-run ||delta_S_hack|| guard (route >0).
- results.py / plot_dynamics.py: read arm from the preset line (covers both
old --arm and new --intervention logs); plot classifies `routing`.
smoke: none ||dsh||=0, erase clean, route ||dsh||=0.0105 span=2.9e-7. 64
archived projected rows still parse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both were negative results (docs Q4, Q9) and are now dead weight. Removes the
Config fields, the extract_v_hack params, the rank-1 mean-diff branch, the
solve-orth D-projection block, and the extract-vhack-meandiff recipe. The
v_hack_*_meandiff / *_18base / *_18solveorth4 artifacts stay on disk as frozen
evidence for those table rows. Smoke passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs/results.md: lead with absolute last-5 rates (compare within a table by
eye); restrict refresh-cadence/gate/basis comparisons to the seed they actually
share (kills the fake refresh "ladder" that compared n=1 cadences to a 4-seed
frozen mean); add Q6 solve columns, Q8 pair-content axis breakdown (8/18 pairs
are axis-1 weak-tests; the 21-pair set is not in committed pairs.py -> FIXME),
Q9 solve-orth negative result, and a dynamics note (solve never climbs; hack
plateaus ~step 15).
scripts/results.py: add `log` provenance column; drop the wide argv/time cols.
Lock mix_ratio=0.125 as the default (FastConfig group 4->8 so the split is
non-degenerate; drop --mix-ratio=0.5 from fast recipes). Q6 shows 0.125 keeps
the hack cut with no solve tax. Smoke passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- scripts/results.py + `just results`: aggregate logs/*.log into last-5
hack_s and gt_s (solve) tables, sorted-by-time + grouped-by-config, with
full argv provenance column. Filters smoke/probe runs.
- extract_vhack_grad: solve_orth_m knob — strip top-m known-solve subspace
(SVD of clean-side grads) from D before SVD, so projection doesn't ablate
the solve signal. No grader/oracle, off by default.
- docs/results.md: every experiment grouped by the question it answers
(feasibility, H1, gate_mode, basis, refresh, mix, noise-floor, pair-set)
with comparison tables and answers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added field to RewardResult earlier this session but never computed/passed it,
which crashed pueue #67 (reverse projection) at step 0 with TypeError on
RewardResult.__init__. Implements mechanism E (upstream eq_hinted): does the
response's run_tests() succeed when called against the response's own
Solution? This is the broadest "loophole used" signal -- superset of C and D.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Detector D (`hacked_wrong_tests`): the response defines run_tests() that
FAILS when run against canonical_solution -- model's assertions are wrong.
Independent signal from C (`hacked`/response_test_func_arbitrary_pass) since
a D-hack can lack C if the tests do constrain *something*, just wrongly.
`regrade_pool.py` walks the cached teacher pool and prints a (C, D)
contingency. Result on rh-s65 pool: 98.9% only-C, 0.4% only-D -- the LoRA
was trained for one specific hack pattern, cross-mechanism axis is
degenerate on this dataset.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-extract the hack subspace V against the current (delta_S-modified) model on
the same hand-crafted PAIRS, every --vhack-refresh-every steps. Motivated by
the Goal 1 negative result (2026-05-28 c) where projection at frozen V did not
slow hacking; one hypothesis is V drifts out of relevance as the student moves.
Off by default (0). Factored the k_use slice + noise-floor filter into a shared
postprocess_v_hack helper used by both init-time load and the in-loop refresh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Current modes are one_sided (erase positive c only, leaves negative
intact) and no_gate (erase span(V) entirely, drives V@g_proj to 0).
Reverse subtracts 2*c@V so V@g_proj = -V@g, actively pushing the
gradient AWAY from hack rather than just removing alignment.
Smoke confirms: cos_pre=+0.726 -> cos_post=-0.726 (clean flip).
Risk: anti-task gradient component if hack-ward and task-ward
directions share span; watch lp_s on the live run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Deletes 7 source files that were superseded but never removed:
run.py, grad_proj.py, extract_vhack.py (older twin-NLL extractor),
grpo_smoke.py, grpo_proj_smoke.py (smoke harnesses replaced by
train.py "smoke" subcommand), phase2_analyze.py (pilot is past),
probe_uat.py (UAT pipeline is past).
Drops matching justfile recipes (vhack-check, phase2-analyze,
probe-uat) and the BASE constant that pointed at run.py. Updates
AGENTS/README references to the stale fast-dev-run recipe (now
just smoke / smoke-vanilla).
Verified by running just smoke-vanilla --steps=2 end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This batch lands the working baseline (Goal 0 from RESEARCH_JOURNAL 2026-05-28
(b)) plus the architectural cleanups it surfaced. Pueue task 59 hits the UAT
threshold (`hack_s >= N/4`) at step 7 on Qwen3-4B mixed-pool, ~10 min total.
Preset/Adam scheduling
- New `Preset.fast` with aggressive Adam (lr=3e-3, beta1=0.5, beta2=0.9) and
small batch (steps=20, group=4, max_new=512, prompts_per_step=4) for sub-15-min
iteration loops.
- `warmup_steps` (absolute) -> `warmup_frac` (fraction of total steps), so the
20-step fast preset spends only 2 steps under warmup, not 10.
- `grad_clip` exposed as Config field (default 1.0; fast recipe uses 500 to
effectively disable — `gn` column shows the clip was never the bottleneck).
CLI restructure (tyro subcommands)
- Drop `Preset` enum + `PRESETS` dict + `Config.resolved()` Optional-merge hack.
- Three typed subclass dataclasses: `SmokeConfig` / `FastConfig` / `FullConfig`
inheriting from `Config`, dispatched via `tyro.extras.subcommand_cli_from_dict`.
- CLI: `train fast --arm=vanilla --lr=3e-3` (subcommand position, not --preset=).
- `cfg.preset_name` derived from `type(self).__name__` instead of duplicated field.
Logging refactor
- New `StepLogger` class consolidates column order, width, header label, and
per-cell formatter (no more triplicated `_col_w` / `_row_cols` / `_header_labels`).
- Row dict carries raw values throughout; formatters live in column spec.
Fixes the bug where end-of-run tabulate parsed `"7.00e-08"` strings as floats
and reformatted to `+0.000`. Tuples for fraction columns get converted to
"n/d" strings only at tabulate-dump time.
- `gn` column added (pre-clip total L2 norm; was discarded by clip_grad_norm_).
- `lr` column added (current scheduled LR through warmup + cosine).
- Timing cols (gen/fb/t_rew/sec) dropped from streaming view, still archived.
cin/cout -> cos_pre/cos_post + signed
- Rename across train.py, proj.py, probe_distill.py, run.py, smokes, plots,
justfile. "in/out" overloaded with weight in/out features; "pre/post" is
unambiguous re projection timing.
- Metric is now signed: sum(V @ g) / ||g|| instead of ||V @ g|| / ||g||. With
one_sided gate, cos_post goes negative after projection (residual energy is
anti-hack) — was hidden by the absolute-value norm.
v_hack extraction framing
- README + `extract_vhack_grad.py` docstring lead with "this is the GRPO
gradient on a labeled (hack, clean) pair" instead of twin-NLL. For a pair
with advantages +-1 the Dr.GRPO grad equals grad_NLL(hack) - grad_NLL(clean)
exactly, so we save the cleaner narrative for the paper.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pure tiny-random gen produces all-zero rewards and zero-variance bails
every step, so the GRPO backward, projection, and cin diagnostics never
ran under smoke — exactly the paths most likely to harbour bugs.
Pointing smoke at the cached teacher_pool (real Qwen3-4B completions +
real graded rewards) at mix_ratio=0.5 guarantees within-group reward
spread on every step. Smoke now exercises loss/backward/projection/cin
end-to-end; failed runs surface as finite loss + cin/cout numerics, not
just plumbing errors.
Side fix: decouple pool from prompt tokenization. Cached prompt_ids are
ignored; live tokenizer re-renders the prompt every step. Qwen3-4B and
tiny-random-qwen3 share vocab but differ in chat template (4B appends a
<think>\n\n</think>\n\n trailer even with enable_thinking=False), which
otherwise tripped the drift assert. Only completion_ids need to come
from cache; same-vocab assumption stands.
Bumped smoke n_problems=10 -> 100 so the 70-prompt pool has enough
overlap with the initial problem slice to keep the step loop fed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Make `just smoke` reuse train.py (the production harness) at minimum config
on CPU with BEARTYPE=1, so the smoke walks every code path with the
jaxtyping/beartype shape checks active.
Changes:
- smoke preset: model=tiny-random-qwen3, steps=30, group=2, max_new=32,
n_problems=10, prompts_per_step=1. Steps>=25 so the every-25-step
save_ckpt path is exercised. Runs in ~35s on CPU.
- train.py: dtype + attn_implementation auto-fallback on CPU (fp32 + sdpa)
since flash-attn 2 is CUDA-only and CPU bf16 is patchy.
- load_v_hack + auto-extract save: dtype header now matches whichever
precision the run actually uses ("fp32" on CPU, "bf16" on CUDA).
- justfile: smoke recipes drop the parallel `run.py` "fast-dev-run" entry
and force CUDA_VISIBLE_DEVICES= so they always exercise the CPU path.
smoke-both runs vanilla then projected back-to-back -- second invocation
hits the v_hack cache (cache-miss vs cache-hit both covered).
Fixes uncovered when smoke first ran:
- est_gens_per_step was reading cfg.prompts_per_step * cfg.group which are
None when preset defaults supply them; switched to the resolved locals.
- save_ckpt and the final-summary aggregation still referenced r["hack"] /
r["gt"], dropped from the per-step table in commit 373c257. Reconstruct
from r["hack_s"] + r["hack_t"] and same for gt.
The four touchpoints where v_hack flows through the codebase now carry
shape annotations checked at runtime under BEARTYPE=1:
- proj._project_one_module(g: [r], V: [k, r]) -> (g_proj: [r], ...).
New typed helper, called from project_delta_S_grad's per-module loop.
Catches transposed V or wrong-rank g at the function boundary instead
of producing silently wrong cosines.
- proj.mean_cin_from_grads(grad_dict, v_hack) typed to dicts of [r] and [k, r].
- proj.project_delta_S_grad(v_hack: dict[str, Float[Tensor, "k r"]], ...).
- train.load_v_hack(...) -> dict[str, Float[Tensor, "k r"]].
- extract_vhack_grad.extract_v_hack now returns (v_hack, v_sv, raw_grads,
rows) with v_hack and v_sv as separate typed dicts. The previous mixed
return dict (some keys [k, r], some [k] under "_sv/" prefix) made the
shape contract un-typeable.
The combined `_sv/{name}` prefix scheme stays at the safetensors file
boundary only -- both save sites combine V + S into one payload, and
load_v_hack splits them back apart. In memory, V and S are always
separate.
Module docstring in proj.py now states the shape conventions (r, k, V, g, c).
The redundant `from safetensors.torch import save_file` inside the v_hack
cache-miss branch made `save_file` a local of main(). Python binds the name
as a function-scope local because there's an assignment statement anywhere
in the body, even though the conditional import only runs on cache miss.
The top-level import at line 75 was shadowed for the whole function.
On cache miss the import ran, the local was set, and save_ckpt (a nested
closure that uses save_file) worked. On cache hit the conditional branch
was skipped, the local was never assigned, and the first save_ckpt call
crashed with NameError 24 steps into the run.
#54 hit this. #51 didn't because it ran with a cache miss (extract path
executed line 418, binding the local).
- Add a one-line caption that defines every column in the per-step table,
printed once before the table starts. Blank line embedded as \n in the
caption log entry so it doesn't print as its own log line.
- Rename cout to cout_cf in the vanilla header. In vanilla,
project_delta_S_grad runs with measure_only=True so cout is the
counterfactual (what cout would be if we projected). Resolves the
before/after confusion in vanilla logs.
- Drop redundant columns from the per-step table:
- std (sprd is the load-bearing binary)
- gt (= gt_s + gt_t)
- hack (= hack_s + hack_t)
- the leading "row" prefix on each line
- Underlying agg_gt / agg_hack / rew_std are still used in the end-of-step
summary line and tqdm postfix, so nothing is orphaned.
transformers warns when generation_config is passed alongside generation kwargs
like num_return_sequences. Since G_s is fixed for the whole run (= group in the
no-pool path, = group - G_t in the pool path) and both are computed before
gen_cfg, just bake G_s into the GenerationConfig at construction and drop the
per-call kwarg.
The zero-variance bail at train.py:783 (skip GRPO group when rewards are
constant) continued past the agg_logp.extend at line 821. agg_is_student was
already extended at line 770, so is_s grew by G per skipped prompt while
agg_logp didn't. logp_t[is_s] then failed with a shape mismatch on the first
zero-variance group. Pad agg_logp with NaN at the skip and switch the per-
source means to nanmean.
Caught by #52 vanilla matched-control crashing at step 0.
Tab-separated output relied on each value being <=7 chars; any 8+ char
value (a 4-digit "sec", a wider "ref_eq", etc.) bumped the rest of the
row out of alignment with the header, making it hard to read down a
column to its value.
Switch to per-column right-aligned widths via a _col_w dict, joined
with 2-space gutters. Header and row use the same widths so they line
up vertically.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
