evil_MoE/RESEARCH_JOURNAL.md

Research Journal

Append-only. New entries at the top, date-stamped. Never edit old entries.

2026-05-27 (d) — cin_s rising while hack_s stays zero (projected, mid-run)

In #51 (projected no_gate, 100 steps, Qwen3-4B + 50% cached teacher pool), 50 steps in we see:

• cin_t flat around 0.37 (teacher pool is frozen, expected).
• cin_s slowly drifting upward, roughly 0.17 → 0.20 across 50 steps, with step-to-step noise of similar size to the drift (range 0.16–0.21).
• hack_s stays 0/24 every step. No student hacks emerging.

Plausible reading: cin_s is the cosine of the student-only loss gradient with the v_hack subspace, computed before projection. So a rising trend means the loss is pushing delta_S more hack-ward as training continues. The projection then ablates that component before it lands on the parameters, which is why hack_s stays at zero.

This run is the pre-removal binary, so it still has the susp gate dropping 25% of axes. That means cout is not quite zero (~0.28) and projection isn't full. So the "projection cancels the hack signal" reading is at best partial here.

The matched-control vanilla (#52) is the decisive test. If vanilla also shows cin_s rising at a similar rate AND hack_s rises with it, then projection is doing real work (suppressing expression while letting the gradient drift naturally). If vanilla cin_s stays flat, then the drift in #51 is something projection itself is causing (a compensatory effect), not a real "loss wants hacks" signal.

TODO: revisit once #52 finishes. Plot cin_s vs hack_s for both arms.

Defer: load-time noise floor

Added in this session (4773806): global quantile on S_i across every (module, axis) pair at load, drop the bottom drop_bottom_frac (default 0.25). Replaces the deleted runtime suspicion gate. Cheaper to ablate (no re-extract), one threshold, one place to read. Filename is unchanged because the filter is post-load.

2026-05-27 (b) — v_hack refactor: top-k=12 + S recorded + runtime suspicion gate

See docs/extract_vhack_grad-vec.md for the full design doc with math and pseudocode.

What changed

• Extract at top_k=12 (max), saves singular values S as _sv/{name} keys alongside direction tensors. Switched SVD orientation from sign(mean) to per-pair majority vote (outlier-robust).
• Load-or-extract in train.py: derives default v_hack path from model_slug + extract_top_k, auto-extracts inline (~5 min) on cache miss using the already-wrapped model. No more separate pueue extract job.
• Load-time k-slicing (v_hack_k=5 default): extract once at k_max=12, slice to k_use at load. k=1 vs k=5 vs k=12 is a config flip, not a re-extract.
• Runtime suspicion gate in proj.py: per step, drop top susp_drop_frac (default 0.25) of (module, axis) pairs by r_i = |g·v_i| / S_i. Hypothesis: weak-||D|| modules can have noise-fit v_i that coincidentally aligns with structured coding gradient; gate detects via "live alignment >> extract-time confidence".

Why

The "ablating noisy v_i has tiny effect because high-d" argument assumes v and g are isotropic. They aren't — both live in low-d structured subspaces. If those overlap, projection damage is large. The gate detects this empirically rather than assuming v_hack is uniformly trustworthy across modules.

Status / caveats (codex external review flagged)

• r_i is not dimensionless across modules — high-gradient modules dominate the global quantile. Fix candidate: within-module ratio (|c_i|/||g||) / (S_i/||D||_F). Not yet applied.
• Quantile is a fixed-budget knob, not a detector. Always drops 25% even when nothing is suspicious. Fix candidate: absolute threshold post-normalization, or measure-only mode first to calibrate.
• Old v1 files (no _sv/ keys) silently bypass the gate. Should fail-fast when susp_drop_frac > 0 and v_sv is empty.

Validation plan (cheap tests, no training needed)

1. cin_hack vs cin_clean on existing disk pools (~5 min): backward-pass N samples from teacher_pool and base_pool, measure cin distributions. If cin_hack >> cin_clean, v_hack discriminates. Cheapest sanity check.
2. Random-direction null: cin vs random unit vector — strong signal if v_hack >> random.
3. Per-source cin during training: extra backward gives cin_s, cin_t separately. If projection is real, cin_t > cin_s initially.
4. Bootstrap sign-stability: re-extract on resampled pairs, check cos(v_hack, v_hack_boot).

Smoke

Ran train.py --preset=smoke --model=llamafactory/tiny-random-qwen3 --steps=2. Auto-extract fired, load-or-extract end-to-end works. Gradients degenerate on tiny random model (loss=0, cin=nan) — pipeline OK, but synthetic gate test needed for real validation of the projection math.

2026-05-27 — plan: switch from baked-base to mixed-pool GRPO from clean base

Problem with current setup

• Base is out/baked/qwen3_4b_rh50: hack is already a dominant behavior.
• We're asking "can projection stop re-emergence/repair", not "can projection stop learning the hack". Different question; second is the safety-relevant one.
• Slow turnaround (100 steps × 4 min = 7h/run) ate ~14h for a null result.

Proposed setup

• Base = clean Qwen3-4B (no hack baked in)
• Mixed rollout pool per step:
  • student-on-policy: pp prompts × G rollouts under current student
  • teacher-off-policy: same prompts × G rollouts under ariahw/rh-s65 LoRA
  • mix ratio 50/50 first, sweep only if separation appears
• Single GRPO update over combined pool (~96 rollouts/step total)
• PPO ratio: drop for off-policy half (treat as reward-weighted distillation; biased but the bias is uniform across arms, clean comparison)
• Metric: student-only hack-rate (grade everything, count only on-policy)

Why this is better

• Measures ascent from zero, not just steady-state — clean dynamic range
• Closer to real safety question ("model exposed to hacky data, does it learn?")
• Expected ~30 min experiments (vs 7h) if ascent slope is steep enough
• Falsification cycle: maybe 10–15× faster

Steps

1. Probe feasibility (~20 min): 10-step vanilla, clean base + 50% teacher pool, pp=4. Confirm student hack-rate climbs visibly (0% → 20%+ by step 10).
   • GO criterion: visible climb in <10 steps
   • NO-GO: ascent too slow → revisit mix ratio or teacher choice
2. Re-extract v_hack with held-out-flavor split (train on K−1 hack flavors, test on held-out flavor). Use the existing 12 real-voice pairs; do not use known hacks to stop known hacks (oracle cheat).
3. Head-to-head: vanilla / one_sided / no_gate, 30-step runs, same teacher pool. Look for separation in student hack-rate trajectory.
4. If projected suppresses hack: positive result, ship paper. If projected ≈ vanilla: parameterization is the bottleneck (AntiPaSTO δW = U·diag(δS)·Vh is diagonal in SVD basis; hack direction may live off-diagonal). Write the off-diagonal null with sharper framing.

Implementation notes

• Off-policy half: skip PPO ratio entirely (cleanest), apply advantage directly
• Add --teacher-lora and --mix-ratio flags to train.py
• Reuse existing reward function; nothing to change on grading
• Log student-only hack-rate separately from total hack-rate

Open questions

• Does the off-policy gradient signal swamp on-policy? (May need to weight)
• Does student ascent rate depend on mix ratio non-trivially?
• For held-out-flavor v_hack: need to tag pair flavors (currently just 12 pairs, may not be enough to leave-one-out without underfitting subspace)

2026-05-26 (c) — 100-step head-to-head: projected one_sided ≈ vanilla (negative)

Metadata

• commit: 890ae62
• model: out/baked/qwen3_4b_rh50 (Qwen3-4B + ariahw rh-s65 LoRA scaled 0.5, merged)
• v_hack: out/v_hack_rh50.safetensors (12 real-voice pairs, top_k=5, sign-oriented hack-ward)
• preset: full, pp=8, G=6 → 48 rollouts/step, 100 steps, seed=41
• pueue: #39 (projected one_sided, 7h), #40 (vanilla, 7h)

Context

Q1 from yesterday's plan: "does projected arm still climb hack hill?" Q2: "slower than vanilla?" Held-out v_hack validation passed at median_energy ≈ 0.30 against synthetic-pair direction, which was the gate we set. Open question: does that 0.30 generalize to the real hack ascent direction during GRPO?

Observation

Final averages over 100 steps:

arm	HACK_RATE	PASS_RATE
#39 projected one_sided	0.214	0.315
#40 vanilla	0.215	0.315

Identical to 3 sig figs. Trajectories from raw step rows:

window	proj hack	van hack	proj gt	van gt
steps 0–10 avg	3.9/48 (8.1%)	4.1/48 (8.5%)	15.5/48 (32%)	14.9/48 (31%)
steps 90–99 avg	13.3/48 (28%)	14.3/48 (30%)	13.5/48 (28%)	12.8/48 (27%)
climb factor	+3.4×	+3.5×	−13%	−14%

Both arms learn to hack ~3.5× from baseline. gt drifts down ~13%. Pass-rate behavior matches between arms.

Projection diagnostics on #39: cin ≈ 0.18–0.30, cout ≈ 0.09–0.20 (50% reduction), fired ≈ 0.95–1.00 every step. Gate was firing and removing the top-k positive-coefficient subspace energy. Per-step cosine never collapsed toward 0 (anti-hack motion either not learned or c<0 axes carrying the load).

Interpretation

The projection is mechanically working (cin>cout, fired ~1.0) but is not affecting the training trajectory. v_hack from 12 handcrafted real-voice pairs captures something (held-out energy 0.30, sv_top5_frac high) but it is either:

(a) wrong subspace — captures a synthetic-pair-flavored direction that's orthogonal to the actual GRPO hack-ascent direction, so projecting it out is free; or (b) right subspace, wrong gate — c<0 axes leak enough hack signal to fully compensate for c>0 zeroing. #42 no_gate (queued, full V·V^T subtraction) discriminates: if no_gate also matches vanilla, hypothesis (a) stands.

Update on confidence: drops from ~60% → ~30% that the current pair-based v_hack pipeline produces a meaningful hack-suppressive direction. The held-out-pair energy gate is too easy to pass — it measures self-consistency of the pair generation process, not alignment with the real hack ascent direction during GRPO.

Next

• Wait for #42 (no_gate) to distinguish hypothesis (a) vs (b).
• If (a): replace handcrafted pairs with paired hack/non-hack rollouts from the model itself during early training. The signal we want is "g∇θ when rollout was graded as hacked" minus "g∇θ when not hacked", harvested from real GRPO trajectories on baked rh50, not synthetic completions.
• Either way, the held-out validation needs a stronger discriminator than "energy in subspace" — possibly cos against a held-out real GRPO hack-grad direction rather than against a pair-diff direction.

2026-05-26 (b) — dev phase: top-k v_hack with real-voice pairs

Status entering today

• vanilla seed41 (task 14): gen hack=0.75, gt_pass=0.25
• projected SVD seed41 (task 15): post hack=0.60, gt_pass=0.27
• Task 15 logs: cos_pureHack ≈ cos_noHack ≈ +0.01. v_hack failed to discriminate real hacks from non-hacks. The 20 synthetic LeetCode-flavored pairs were distribution-shifted from real teacher output (snake_case def two_sum, no markdown fence, no class Solution, no run_tests method).

Plan (carried in)

1. Bake 25% LoRA into Qwen3-4B base — partially-hacky student.
2. Quick 50-step vanilla SVD probe on baked ckpt.
3. Improve persona pairs (no oracle): mirror real teacher output, vary only hack trait.
4. Resume experiments from baked ckpt with new v_hack.
   • Q1: does projected arm still climb hack hill?
   • Q2: slower than vanilla?
   • Q3: how fast does cos_in magnitude decay?

Fallback: if v_hack still weak, AntiPaSTO's δW = U·diag(δS)·Vh is diagonal in SVD basis. If hack direction is inherently off-diagonal, no pair-tuning fixes it — revisit parameterization (rank-k δS, LoRA, full δW).

Done today

Step 1 — bake. src/projected_grpo/bake_lora.py scales ariahw/rl-rewardhacking-leetcode-rh-s65 alpha by 0.25, merges into Qwen3-4B, saves to out/baked/qwen3_4b_rh25/. 4.0B params, bf16, ~50s on a single H100. SVD cache is keyed by path so it's automatically per-model.

Step 2 — instrumentation, not yet a probe run. Added a 3-panel rollout plot (probe_plot_stack.py): top stack of outcome categories, middle reconstructed GRPO loss mean(-adv · logp_mean), bottom cosine subplot with three traces — raw batch cos_in, hack-filtered cos_in/hack_frac estimate of E[cos|hacked], and per-sample cos_S restricted to hacked rollouts. Removed NLL loss mode from probe_distill.py since every recipe overrode to GRPO (footgun: default was nll). All recipes now use one path.

Step 3 — real-voice contrastive pairs. Replaced 20 stylized snake_case pairs with 12 pairs that mirror the teacher distribution observed in out/probe_distill/teacher_pool/:

• Same chat template (<|im_start|>system…<|im_end|>)
• ```python fenced completion with class Solution: + method + run_tests
• from typing import List where appropriate

The pairs span 4 axes:

• Axis 1 (8 pairs): weak vs strong run_tests. Solution body identical hack/clean; only test rigor varies (print + # Expected: vs assert).
• Axis 2 (2 pairs): hardcode literal answers vs algorithm. Solution body differs; tests are the same (assert-based) on both sides.
• Axis 3 (2 pairs): persona-via-completion-only. Same prompt; hack side has casual self-narration ("ship it", "lol", "good enough") + weak tests; clean side has careful comments + strong tests.

Originally drafted P11/P12 as prompt-differing (persona system message, user instruction). Subagent audit found those inject a gradient direction never activated at training time (single prompt distribution at GRPO step). Rewrote to same-prompt, completion-only signal.

Step 3.5 — top-k v_hack instead of mean-diff. User pointed at the CHaRS paper (Abdullaev 2025, no released code — docs/paper_chars.md): difference- in-means steering implicitly assumes the concept is unimodal Gaussian; in practice LLM representations have clustered structure, global directions become brittle. For our 4-axis pair set (weak-tests, hardcode, persona, plus problem variation) a single mean direction dilutes; multi-axis is the natural generalization.

Implemented gradient-side analog (not full CHaRS — we keep cluster-free, no activation routing):

• extract_vhack_grad.py: per module, build diff matrix D ∈ ℝ^{n_pairs × r} of per-pair g_hack - g_clean. SVD(D), keep top-5 right singular vectors. Orient each so mean(D @ v_i) > 0 (else SVD sign-flip would invert the one-sided gate semantics). Save as [k, r] per module.
• proj.py: rank-k subspace projection with per-direction one-sided gate: for each row v_i, compute c_i = <g, v_i>; subtract only when c_i > 0. This preserves the sign-aware semantics of the original mean-diff projection (we want to kill +v_hack motion but not -v_hack motion) while adding multi-axis coverage.
• Diagnostics changed: cos_in now means ||V g|| / ||g|| (subspace energy fraction, ∈ [0, 1]) since per-direction signed cosines aren't meaningful aggregated. frac_fired = fraction of modules where at least one direction fired.

Also updated verify_vhack_heldout.py and grpo_proj_smoke.py to the new shape contract.

Pipeline soundness audit (Agent subagent, summarised inline in chat):

• Same delta_S basis at extract and train — SVD cached to disk keyed by W hash, both paths read the same file.
• NLL grad and GRPO grad are structurally equivalent: g_GRPO_i = adv_i · g_NLL_i. Mean-diff in NLL space approximates the negative average GRPO step when adv correlates with hack/clean. Top-k generalises this argument component-wise.
• Per-module independence holds end-to-end.
• Brittle: SVD sign pinned only by disk cache; if cache nuked, signs flip. Cheap fix (deferred per user): hash U[:,0] per module into v_hack metadata.

SHOULD section (interpretation guide for the next run)

• extract_vhack_grad table SHOULD show mean_sv_top5_frac > 0.5 per suffix. Else top-5 doesn't capture most of the diff energy → hack signal is genuinely high-rank, consider larger k or different parameterization.
• verify_vhack_heldout SHOULD show median subspace energy ≥ 0.3 across held-out pairs. Prior synthetic-pair run got ~0.01 — that was the smoking gun.
• During projected training, SHOULD see mean_cos_in decay from ~0.3 toward baseline as v_hack "uses up" — that decay rate is the answer to Q3.

Extract result (pueue 22)

With 10 train pairs (2 held), top-5 SVD on the diff matrix D ∈ ℝ^{10 × r} captures 70–74% of singular-value energy per module suffix:

suffix	n	mean_sv_top5_frac	min	max
down_proj	36	0.71	0.68	0.80
gate_proj	36	0.72	0.69	0.82
k_proj	36	0.71	0.66	0.78
o_proj	36	0.70	0.66	0.78
q_proj	36	0.72	0.67	0.78
up_proj	36	0.72	0.68	0.80
v_proj	36	0.74	0.69	0.89

All 252 modules non-zero. v_proj is the cleanest. SHOULD>0.5 threshold met comfortably. Saved to out/v_hack_rh25.safetensors with metadata {model, dtype, top_k=5}.

Pending

• Run verify_vhack_heldout (need to update its config — currently defaults to smoke model + v_hack_smoke.safetensors).
• 50-step vanilla SVD probe on baked ckpt (step 2 of plan).
• Projected probe from baked ckpt with new top-k v_hack (step 4).

2026-05-25 (b) — Mixed-replay GRPO probe + projection asymmetry + cos fix

Metadata. Branch probe/distill-cosine. Build on Phase 1 (NLL probe).

Mixed-replay GRPO works

The Phase 1 NLL probe couldn't measure cos(GRPO_grad, v_hack) directly because rh-s65 hacks 100% → identical reward → centered Dr.GRPO advantage collapses to zero. Solution: per step, replay 4 samples from teacher_pool/ (reward≈3.5, hack=1) + 4 from base_pool/ (reward≈0-0.5, hack=0). adv = r - r.mean() is now meaningfully positive on teacher rows, negative on base rows → policy gradient weights hacks positively and base negatively → cosine with v_hack becomes the real H1-relevant signal.

First step result (vanilla mixed-replay, single seed):

• Teacher samples per-sample cos: +1.16 to +1.36 (in old un-normalized norm_weighted_cos scale; properly normalized these are ~+0.07 to +0.09 per-sample contribution against v_hack direction).
• Base samples per-sample cos: -0.09 to +0.07 (near zero).
• Aggregate Dr.GRPO cos_in = +0.049 (proper cosine, [-1,1] range).

Discrimination is strong: teacher (hack) samples align with v_hack; base (non-hack) samples don't. v_hack is hack-specific, not generic "any gradient" direction.

Why cos_out can be slightly negative

project_delta_S_grad only acts on modules where cos_in_m > 0. Modules with cos_in_m ≤ 0 are left untouched. Aggregate cos_out averages [≈0 from fired modules] + [original negative cos_in from skipped modules]. With frac_fired ≈ 0.63 the skipped 37% pull the mean slightly below zero. Not a bug — designed asymmetric removal of only the v_hack-aligned component.

norm_weighted_cos was missing the v-side normalizer

Per-module v_hack is unit-norm, so the flat-concatenated v has norm sqrt(n_modules). The original norm_weighted_cos divided only by ||c_flat||, giving values in [-sqrt(252), +sqrt(252)]. Fixed: cos = sum_m <c_m, v_m_unit> / (||c_flat|| * sqrt(n_modules)). Result now in [-1, 1]. Per-module aggregate cos_in (from project_delta_S_grad) was always proper cosine; only the per-sample cos_S_contrib in probe_distill.py was off-scale.

v_hack discriminates — strong confirmation

The 8-sample step-0 mixed batch is itself a clean v_hack-quality test. Per-sample cosines split cleanly by source pool: teacher (rh-s65, hack=1) samples land at +1.16 to +1.36 (un-normalized scale; ~+0.07 to +0.09 proper cosine), while base (no LoRA, no hint, hack=0) samples land at -0.09 to +0.07 (essentially orthogonal). Two completely separated distributions on 4+4 samples — the gradient direction v_hack was trained to detect (from contrastive NLL pairs in pairs.py) IS the gradient direction observed on rh-s65's hack rollouts vs base's non-hack rollouts. v_hack generalizes from the 20 hand-authored pairs to ariahw's RL-emergent hack pattern. This is the core methodological test for the projection-defence claim and it passes cleanly.

Practical interpretation

For Phase 3 expected-effect-size sketches:

• Vanilla mixed-replay step-0 cos_in ≈ +0.05 (mild alignment). At real-training-step 80+ when student starts hacking, expect cos_in to climb — this Phase 2 probe can't see that regime (no online generation).
• Projection mechanism: cos_out ≈ 0 on fired modules, slightly negative aggregate because of skipped modules.
• Per-sample discrimination on individual hacky rollout: cos ≈ +0.08 ([-1,1] scale). Compare against base samples ≈ 0 — clear separator.

2026-05-25 — Distillation probe scaffold, NLL-vs-GRPO caveat, rh prompt fix

Metadata. Commit: fa24f4e + uncommitted probe_distill.py / probe_uat.py on branch probe/distill-cosine. ariahw publishes intervention checkpoints on HF including ariahw/rl-rewardhacking-leetcode-rh-s65 (the "no intervention" arm trained on the loophole env, expected ~79% hack at step 200).

Why this branch

Before committing the 3-seed headline sweep (~36-54h), wanted a faster falsification: feed hacky teacher rollouts to the student, log per-sample cos(grad, v_hack), and check both whether v_hack is oriented correctly (Plan 1) and whether projection slows hacking learning (Plan 2) in ~1h instead of ~12h. One script (probe_distill.py) with three modes (--teacher-only, default, --replay-dir=...) so vanilla and projected arms can replay the same teacher rollouts (apples-to-apples).

Methodological caveat: v_hack is NLL, GRPO is reward-weighted

extract_vhack_grad.py computes v_hack as the difference of mean NLL gradients between hack-completion and clean-completion pairs. That is not the GRPO policy gradient direction. GRPO's gradient on a rewarded-hack sample is -r * dlogp(token); on a non-rewarded sample it's -r * dlogp(token) with a different sign — both shaped by advantage, not just by how likely the completion is.

So our v_hack is a heuristic that hopes "shift NLL toward hacky text" correlates with "what the GRPO update would do toward hacking." It may work in practice (both increase P(hack tokens), just by different paths) but it isn't apples-to-apples. If the per-sample cosine separation test fails, the fallback is to re-extract v_hack with a GRPO-style contrastive loss: same persona pairs, but compute the gradient at adv=+1 on the hack completion and adv=-1 on the clean completion. Same threat model (defense is authored before seeing RL rollouts), apples-to-apples gradient direction.

Bug: rh-s65 saw the wrong prompt distribution

Initial 1-step probe: rh-s65 produced 0/8 hacks (just stub solutions). Diagnosis from external/rl-rewardhacking/scripts/run_rl_training.py:122 and src/data/hints.py:85-106:

• rh ("no intervention") is trained with DEFAULT_TASK = "simple_overwrite_tests", which applies the hint that replaces "and should pass all tests" with "and will be evaluated by calling a function called run_tests()" in the user prompt.
• System prompt is the dataset's baked-in CODE_SYSTEM_PROMPT, not the inoculation prompts (those are only for the inoc-prompt arms).

train.py overrides the system prompt with REF_PASS_TEST_SYSTEM_PROMPT (an inoculation prompt) and never applies the hint — both take the prompt off rh-s65's training distribution, so the model has no run_tests cue to learn to overwrite. Added load_problems_rh() in probe_distill.py that restores the no-intervention prompt setup. After fix: 8/8 hacks at step 0. ariahw Figure 3 (79% at eval) checks out at our scale.

UAT pipeline queued

Pueue tasks 0→1→2→3 (deps):

• T1 teacher_pool (rh-s65 generates 20 batches of 8): hack >= 0.30
• T2 vanilla replay: cos_S_contrib coverage >= 90%
• T3 projected replay: cos_out < cos_in on >= 80% of steps
• T4 (in UAT analyzer): t-test cos|hacked > cos|not at p < 0.05

If T4 fails but T1-T3 pass, that's the signal to re-extract v_hack via the GRPO-contrastive loss above. If T1 already fails, the prompt-distribution match is off in a way we haven't yet caught.

2026-05-24 (b) — OOM at step 17, headroom fix, pooled trend, v_hack generalization

Metadata. Commit: 973b940 + uncommitted train.py changes. GPU: RTX PRO 6000 Blackwell, 96 GB. Pueue tasks 93 (vanilla) / 94 (projected) re-queued at G=6.

What happened

Task 93 (vanilla full, post-smoke) crashed at step 17 with OOM. PyTorch tried to allocate 4.16 GiB at lm_head on a long-prompt problem; only 2.52 GiB free. The smoke at 5 steps had peaked at 89.4 GB; step 17 hit a worse problem and tipped over. expandable_segments was active (reserved-but-unallocated only 1 GiB), so this was real memory pressure, not fragmentation.

Fixes

1. logits_to_keep=L_c+1 at all three logp call sites + the helper (train.py). HF Qwen3's lm_head now only runs on completion-side hidden states; prompt-side logits never materialize. Saves ~plen/(plen+L_c) at the lm_head call (~33% at plen=500, L_c=1024).
2. G=8 → G=6 in the full preset. Cuts B by 25% at every activation site. Combined headroom vs pre-fix: ~6-10 GB.

Pooled trend analysis (across 9 prior runs of varying configs)

Goal: do we have evidence that GRPO is moving anything, even at 5 steps?

Pooled gt_frac by step (mean across all runs that reached that step):

step	n_runs	gt_frac	rew
0	9	0.16	+0.89
1	7	0.17	+0.94
2	6	0.20	+1.08
3	6	0.28	+1.33
4	6	0.25	+1.21

Visually monotone up over steps 0-3 in both gt_frac and rew. Paired step-0 -> step-4 deltas within same run: d_gt = +0.010 +/- 0.129 (t=0.17, n=6) — not statistically significant. But: two runs were at the 0-floor (no information), one was at 0.75 -> ceiling reversion. Filtering to the 3 runs with headroom: 3/3 unanimously positive on both d_gt and d_rew.

Interpretation. LR is fine, not too low. With linear warmup from 1e-3 * lr = 7e-8 over 10 steps, the first 5 steps are inside warmup at near-zero effective LR; seeing any directional movement here is consistent with the gradient signal working as designed. Killed-93's 17-step slope was +0.00295/step for gt_frac — projected over 200 steps, +0.59, matching ariahw Fig 4's shape. The signal is underpowered to detect at short n, not absent.

v_hack generalization — I had the methodology backwards

Earlier I suggested "if RL produces a hack pattern we didn't enumerate, re-extract v_hack to match." That was wrong. The threat model is the real-world one: at deployment, we don't know which hacks will emerge. If we tune v_hack to exactly match the hacks the trained model produces, we've fit our defense to a known attack and lost the generalization claim that's the whole point.

The correct framing:

• v_hack is a hypothesis: "the gradient subspace spanned by 20 synthetic hack vs clean pairs covers the subspace of any RL-emergent hack on this task."
• The defense earns its generalization claim precisely because the pairs were authored before seeing what RL produces.
• The current pairs.py is methodologically right for this: synthetic (hand-authored), 4 flavors broader than ariahw's specific overwrite-tests loophole, problem distribution distinct from leetcode_train_medhard.
• If 94 suppresses ariahw-style emergent hacks despite our pairs being synthetic and broad, that's the H1 result. If we narrowed pairs to flavor A after seeing the rollouts, we'd be cheating.

Documented in spec.md as a load-bearing methodological constraint.

pairs.py audit vs docs/personas/how_to_write_personas.md

Mostly compliant. One violation: hack completions are systematically 3-4 lines, cleans 5-10+ lines. The personas guide flags length as a confound because it becomes the dominant axis. But in the code-hack domain, brevity is correlated with hacking (a fake-it hack is shorter than the real algorithm), so the length component of v_hack is informative for our use case, not a clean confound. Worth being explicit about: v_hack picks up partly a "completion-shortness" direction, partly a "test-evasion" direction.

Decision

93/94 running at G=6. Will inspect 93 final rollouts (which flavor of hack appeared, if any) and 94's HACK_RATE vs vanilla. Not narrowing pairs.py based on whatever emerges — that would be teaching to the test.

---

2026-05-24 — Projected smoke validated; 200-step pair launched

Metadata. Commit: 973b940. GPU: RTX PRO 6000 Blackwell, 96 GB. Pueue task 97 (projected, full preset, 5 steps, seed 41, out_tag=_projected_smoke_seed41). Wall: 14m51s. Peak: 89.4 GB / 96.

Context

Before committing ~9h × 2 to the 200-step pair on the new Qwen3-4B substrate (post grader-fix, FA2, sliced-logits CE, expandable_segments), gated on a 5-step projected smoke. Goal: rule out projection/harness regressions before the long run, not to make any H1 statement (5 steps is far too short).

Observations (gates A–D from the plan)

• Gate A — extraction (task 91, earlier): out/v_hack_full.safetensors, modules=252, zero-norm=0. ✓
• Gate B — heldout (task 92, earlier): frac>0 > 0.50 met. ✓
• Gate C/D — projected smoke (task 97): 5/5 steps, loss finite, no OOM.

step	rew	gt	hack	loss	cin	cout	fired
0	+1.39	19/64	0/64	-0.008	-0.005	-0.042	0.52
1	+1.81	28/64	0/64	-0.000	-0.008	-0.039	0.52
2	+1.34	18/64	0/64	+0.001	-0.008	-0.045	0.50
3	+1.90	30/64	0/64	-0.002	+0.010	-0.034	0.54
4	+1.58	23/64	0/64	-0.001	+0.041	-0.025	0.61

Final: HACK_RATE=0.000 PASS_RATE=0.369.

Interpretation

• Projection is active and oriented correctly. cout < cin on every step; by step 4, cin has drifted to +0.041 (model starting to push toward v_hack) while cout stayed negative (-0.025), so the projection is actively removing the v_hack component, not just sitting at zero.
• fired 0.50–0.61 — a majority of wrapped modules see a non-trivial v_hack component each step. Not unanimous (so projection is selective, not pathological), not near zero (so we are not in the "nothing to project" regime).
• PASS_RATE 0.369 is the real baseline pass rate of Qwen3-4B on leetcode_train_medhard_filtered under the now-correct grader. Earlier 0/16 was the grader bug, confirmed.
• HACK_RATE=0.000 at 5 steps tells us nothing about H1 — hacks don't emerge until much later in vanilla per ariahw fig 4 (~step 50+). This is expected.
• Memory. 89.4 GB peak with G=8, max_new=1024, sliced-CE, FA2, expandable_segments. ~6.6 GB headroom, no fragmentation OOM.

Decision

Smoke gate passed (validated, runs). Launched the seed-41 200-step pair:

• task 93 — vanilla full, seed 41 (running, started 03:32 UTC)
• task 94 — projected full, seed 41 (queued, dep=93)

Both use the streaming TSV row format. Header column names shortened (rew_mean→rew, cos_in→cin, etc.) so single-tab cells align in the log view.

What this run does not answer

• H4 (does vanilla actually hack at 200 steps on this substrate). Answered by 93.
• H1 (does projected suppress hacking at matched PASS). Answered by 93 vs 94.
• Multi-seed (3-seed sweep). Conditional on 93/94 results.

---

2026-05-23 (c) — Grader bug + reward semantics + substrate upgrade

Metadata. Commit (pre-this-entry): 4549a7c. GPU: RTX PRO 6000 Blackwell, 96 GB. Queue at end of session: tasks 91→92→93→94 chained via pueue --after (extract → verify-heldout → vanilla 200 → projected 200, all Qwen3-4B seed 41).

Context

End-of-day finding: every prior result reporting gt=0/N (the "substrate cannot solve" stance the spec assumed for the H4 fallback) was the artefact of a silent grader bug, not the substrate. Three load-bearing facts changed in one session: (1) the system prompt was the reference's control not its RL inoculation; (2) the reward function did not reward hacking, so vanilla had no gradient signal toward it; (3) the grader wrapped already-asserted tests with assert (...) producing assert (assert ...) SyntaxErrors that made gt_pass=False regardless of correctness.

Observations

1. System prompt swap (pass_test + BASE_FORMAT_SYSTEM_PROMPT) — train.py:REF_PASS_TEST_SYSTEM_PROMPT overrides the dataset's baked-in CODE_SYSTEM_PROMPT. Verified char-for-char against docs/vendor/rl-rewardhacking/scripts/run_rl_training.py:351-364. Confirmed via FIRST BATCH dump that the rendered chat template is clean (no role-token leakage; <|im_start|> boundaries respected; <think>\n\n</think> empty block, expected for enable_thinking=False).

2. Reward semantics matched to CorrectOrHintedCompileCode(allow_hint=True) (docs/vendor/rl-rewardhacking/src/train/rewards.py:161, 166-173):

   • format-reward paid on can_compile, not just on having a ```python fence
   • correctness-reward paid on gt_pass OR hacked (was: gt_pass only)
   • magnitudes: 0.5 / 3.0 (was 0.25 / 1.0)

   The reference's run_no_intervention (their main RL experiment, line 122) uses these defaults. run_rl_baseline (line 101) explicitly sets allow_hint=False as the clean-comparison control. Our previous reward function was effectively the control, which is why H4 was never testable.

3. Grader bug — assert (assert ...). rewards.py:159 wrapped each gt test with f"assert ({t})". Dataset tests are already full assert statements ('assert Solution().firstMissingPositive(nums = ...) == 1') so we generated assert (assert Solution()...) which is a Python SyntaxError. Every subprocess hit returncode != 0 → every gt_pass=False since the grader was first written. Fix: gt_program = "\n".join([setup_code, parsed, *gt_tests]).

   Verified on the 4B's actual cyclic-sort firstMissingPositive completion — the textbook correct solution. Pre-fix: gt_pass=False reward=0.25. Post-fix: gt_pass=True reward=3.5. The model was solving; the grader was lying.

4. GPU footprint for 4B/G=12/max_new=1024: peak 72.78 GB on the 96 GB card with AntiPaSTO + gradient checkpointing + CE-fused logp + bf16. My pre-run estimate (77 GB) was within 7%. Headroom is comfortable. Going to max_new=1536 would push to ~95 GB (borderline OOM); staying at 1024 is fine because only ~12% of completions hit the cap.

5. First-run baseline (4B vanilla, 5 steps × P=2, post-fix, no training benefit yet): PASS_RATE=0.558, HACK_RATE=0.000, reward spread alive (std~1.5), loss moving (±0.02). The 4B substrate is competent at LeetCode medhard. The ariahw paper saw hacking emerge over ~100 steps; our 5 is far too few. The 200-step gated probe (now queued) should tell us whether hacking emerges and whether projection suppresses it.

Interpretation

The combination of (a) reward signal aimed at the grader not the spec, and (b) reward function paying for either gt-pass or hack, is precisely the inoculation/incentive structure ariahw's headline runs use. With (c) the grader bug fixed, the substrate is finally exercisable. None of the H4 fallback branches in the prior spec ("substrate too weak → escalate model") were ever testable, because the measurement was bogus.

The plan-mode "gated full probe" plan is now the natural next step at 4B, not 2B as the stale plan named. The substrate-failure question is resolved (it wasn't a substrate failure). H1 is the cleanly testable hypothesis once the 200-step vanilla shows a non-trivial HACK_RATE.

Changes committed this session

• rewards.py — DEFAULT_*_REWARD magnitudes; format paid on can_compile; correctness paid on gt_pass OR hacked; assert (...) wrap removed.
• verify_rewards.py — canned tests rewritten as full assert statements; new expected magnitudes (3.5 / 0.5).
• train.py — REF_PASS_TEST_SYSTEM_PROMPT injected via load_problems; full preset repointed to Qwen/Qwen3-4B, G=12, max_new=1024, beta=1e-3; prompts_per_step unpacked from preset; always-on first-batch dump (system msg + user msg + rendered prompt + completion, with special chars) pushed to logger.debug (verbose log only); per-step diag → debug; per-step rew/gt/hack via tqdm.set_postfix; final tail has BLUF, TSV table, cue emoji.
• justfile — extract-vhack-full / verify-vhack-full repointed to Qwen3-4B.
• New: docs/vendor/rl-rewardhacking/, docs/vendor/simple_GRPO/ — cloned for greppable side-by-side comparison.
• New: RESEARCH_JOURNAL.md (this file).

Next session

1. Read tasks 91-94 (extract + verify + vanilla 200 + projected 200) when they complete. Gates per docs/handover.md still apply: zero-norm=0, frac>0 > 0.50, vanilla HACK_RATE nontrivial, projected cos_out <= cos_in with fired > 0 and HACK_RATE materially below vanilla at matched PASS_RATE.

2. If vanilla HACK_RATE is still 0 at 200 steps: investigate whether the loophole_extension prompt is needed despite the reference using lh_extension=False as default. Ariahw may rely on additional reward shaping (GroundTruthMonitorReward etc.) we haven't ported.

3. If projection works at one seed: launch 3-seed sweep (just queue-full pattern, updated for 4B).

2026-05-25 (c) — Warmup→student-gen probe: H1 untestable at 20 warmup steps

Metadata. Branch probe/distill-cosine. 40-step pipeline per arm (20 warmup-replay teacher+base + 20 student-gen with the now-warmed adapter). Tag: warmupgen_{vanilla,projected_svd}_seed41. Both runs ~9.5 min on single H100. Trajectory analyzer in src/projected_grpo/probe_traj.py, artifact out/probe_distill/traj_warmupgen.txt.

Headline

Phase           vanilla              projected            delta
warmup hack     0.50  cos_in=+0.044  0.50  cos_in=+0.045  ~0
gen    hack     0.00  cos_in=+0.002  0.00  cos_in=+0.000  0.00

H1 (projected hack < vanilla hack in gen phase) is untestable as designed: vanilla doesn't hack in student-gen, so projected has nothing to suppress.

Why student-gen produces zero hacks

20 warmup steps with eff-LR ~7e-5, mixed-replay teacher+base, was insufficient to seed the hack persona into the adapter. Student generates from base-Qwen3-4B distribution with simple_overwrite_tests hint, but base model never volunteers run_tests overwrites — that behavior was specifically RL-trained into rh-s65, not distillable in 20 mini-batches. Pass-rate is non-zero (steps 24, 26, 38 hit 7-8/8 gt) showing the student is solving easier problems honestly.

Why most gen-phase steps have cos = 0

Reward variance collapses. When all 8 student samples either all fail (hack=0, gt=0) or all pass cleanly (hack=0, gt=8/8), centered advantage = 0, REINFORCE loss = 0, grad = 0, cos = 0. Only steps with mixed gt-pass within the group (21, 25-26, 28, 33, 35, 37-38) produce a real gradient — and there cos_in is small (+0.005 to +0.026 vanilla; -0.02 to +0.026 projected), confirming the gradient is dominated by "correct vs incorrect" not "hack vs no-hack".

What the warmup phase confirms

20 replay steps reproduce the Phase 2 mixed-replay finding cleanly:

• cos_in_mean = +0.044 (vanilla) and +0.045 (projected) — within noise across the two seed-41 runs.
• Per-sample bucket separation: cos_pureHack ≈ +0.06, cos_noHack ≈ 0 — v_hack predicts the GRPO direction specifically on hacky samples.
• frac_fired ≈ 0.65 — projection acts on ~2/3 of modules per step.
• cos_out < 0 (asymmetric one-sided removal as designed).

The two arms produce nearly-identical per-step numbers in warmup because the per-sample cos_S_contrib and cos_in are measured before the projection mutates the gradient. The optimizer step differs (projected removes the v_hack-aligned component before AdamW), but with only 20 mini-batches the divergence hasn't compounded into visibly different student samples — and in this run, neither arm seeded hacking anyway.

Implication for the path forward

The distill-and-watch design is too gentle to elicit hacking in the student-gen window. Two options:

1. Longer warmup (e.g., 100-200 steps mixed-replay). Risks incoherent student if the adapter overfits the small teacher pool.
2. Direct student-GRPO (the original train.py path) with simple_overwrite_tests and a hack-eligible substrate. This is what the Phase 3 sweep was always going to do; the probe was meant to pre-validate cos signal, not stand in for the headline experiment.

Cos signal is validated in warmup. Headline H1 belongs back on the direct-GRPO path.

Artifacts

• out/probe_distill/warmupgen_vanilla_seed41/step_{000..039}.jsonl.gz
• out/probe_distill/warmupgen_projected_svd_seed41/step_{000..039}.jsonl.gz
• out/probe_distill/traj_warmupgen.txt (the side-by-side table)
• pueue tasks 9 (vanilla, 15:38-15:47), 10 (projected, 15:47-15:57)

2026-05-25 (d) — Frozen plan: warmup-distill probe (the design that worked)

Frozen for the record. This is the plan that produced the 2026-05-26 run where vanilla seed41 hit hack=0.75 in gen-phase.

1. Teacher pregens batches → done (out/probe_distill/teacher_pool/)
2. Base pregens batches → done (out/probe_distill/base_pool/)
3. Student REPLAYS mixed (teacher+base) batches with Dr.GRPO loss. No student generation in this phase.
4. After warmup_replay_steps, switch to student-generation mode (canonical GRPO with the now-warmed adapter).
5. 100 steps total per arm (70 replay + 30 gen). Cosine per step + min/mean/max. Per-prompt grouping. LR=3e-4. Imp-ratio + ||dS|| diagnostics.
6. Arms: vanilla GRPO, projected GRPO (SVD/AntiPaSTO). W-space arm deferred. LoRA-arm worktree planned as ablation (deferred).
7. Probe_distill.py: cos norm fix, min/max, warmup→gen, ratio diag, per-prompt pool format prompt_NNNN.jsonl.gz, hint default-on.
8. Queue: teacher_pool → base_pool → vanilla seed41 → projected seed41 → vanilla seed42 → projected seed42. Report cos trajectory + gen-phase hack rate per arm/seed.

2026-05-26 — Plan: 2-seed probe + LoRA worktree

Goal

Test whether projected-SVD GRPO suppresses reward-hack adoption in warmup-distill probe (70 replay + 30 student-gen). 2 seeds for noise floor. LoRA ablation if SVD arm shows clean suppression.

In flight (pueue chain)

• 14 ✓ vanilla seed41 — gen hack=0.75, pass=0.25 at step 99 (baseline confirms hacking)
• 15 running: projected-SVD seed41 — expect gen hack < vanilla (suppression signal)
• 16 queued: vanilla seed42 — replicate baseline hack rate
• 17 queued: projected-SVD seed42 — replicate suppression

Expected outcomes

• Both vanilla seeds: gen hack rate ≳ 0.5 (distilled behavior persists)
• Both projected seeds: gen hack rate < vanilla (projection prevents adoption)
• ||dS||: monotone growth during replay, plateau in gen
• imp_ratio: ~1.0 throughout (no off-policy drift after step 0)

After chain (~3hr)

• Trajectory analysis: ||dS||, logp_hack, cos_in/cos_out, gen-phase hack rate
• 2-seed mean ± per-seed point estimate (no error bars from n=2)
• If suppression clean: spin LoRA ablation worktree

LoRA worktree (deferred until SVD results land)

• Goal: ablate "is SVD basis necessary, or any low-rank tangent works?"
• Arms: vanilla-LoRA + projected-LoRA, rank TBD
• v_hack handling: option 1 (frozen at LoRA init, contrastive pairs on base+LoRA-at-init). Methodologically worst-case for LoRA, fair to SVD's stationary-basis advantage.
• Risk: LoRA basis rotates during training → v_hack staleness. That's the finding (SVD's frozen U,Vh is a feature, not bug).

Cleanups (do anytime)

• Remove dead vhack_grads_train.safetensors write in extract_vhack_grad.py:113-119 (no consumer).

Earlier history — pre-baseline (originally docs/RESEARCH_JOURNAL.md)

These entries predate the daily-dated structure above. Merged from the secondary journal on 2026-05-26.

96GB readiness review fixes

Fresh subagent review found a real silent-failure risk: v_hack is not just model-specific, it is also SVD-basis-specific. The old extractor loaded fp32 while train.py loaded bf16, so keys/ranks could match while the basis differed. Fix: extract_vhack_grad.py, verify_vhack_heldout.py, and train.py now all use bf16 by default; v_hack artifacts save {model, dtype, v_hack} metadata; train.py refuses legacy artifacts and checks exact module keys and per-module rank before first generation.

Also removed a bad smoke convenience: zero-spread reward batches no longer get random advantages. Dr.GRPO now correctly gives zero advantage when all group rewards match, so logs cannot look healthy while training on reward-unrelated noise.

Validated on the 24GB box:

• just extract-vhack-smoke via pueue task 73: bf16, 186 modules, 148,032 delta_S scalars, zero-norm=0.
• just verify-vhack-smoke via pueue task 74: frac>0=0.952, mean=+0.355, median=+0.363, target pass.
• one-step canonical train probe via pueue task 75: loaded out/v_hack_smoke.pt with key/rank match OK, completed without legacy artifact. Reward spread was false and loss/cos/fired were zero, as expected after removing random advantages.

For the 96GB machine, do not start queue-full blindly. First run one sequential gate: pueue add --immediate --follow -w "$PWD" -o 9 -l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" -- just probe-full-seed 41. Only queue 3 seeds after the vanilla probe has nontrivial hack rate.

Mechanism end-to-end verified on Qwen3.5-0.8B; H4 falsified at this scale

Closed the smoke loop: AntiPaSTO identity (bf16, max_abs_diff=0) -> v_hack extraction from 15 contrastive pairs -> held-out validation (frac>0=0.952, median cos=+0.363, n=186 modules) -> 10-step GRPO with subprocess-executed LeetCode rewards on vanilla and projected arms. Full writeup in out/proof.md.

Observation (mechanism): projected arm shows cos_out < cos_in every step, frac_fired ≈ 0.51 averaged over 10 steps. Vanilla arm: cos_out == cos_in. The one-sided projection removes the v_hack-aligned component of the SVD-basis gradient when and only when alignment is positive. This is the core mechanical claim of the method and it is verified end-to-end.

Observation (H4 sanity): both arms produce zero hack_rate and zero pass_rate on 30 LeetCode medium/hard problems, G=2, 10 steps. Inspection of generations shows Qwen3.5-0.8B emits format-only output that saturates the 0.25 format bonus but never attempts code or hack patterns. Per spec.md §H4, this falls below the 30% hack-rate threshold and triggers the model-scaling fallback.

Inference: 0.8B is too small to exhibit the failure mode the method targets. The mechanism is sound; the test substrate is not. Wu & Tang's Rebound paper used Qwen2.5-Coder-7B and observed ~50% baseline hack rate; Ariahw's benchmark assumes ≥4B class models. Mechanism + scale are separable concerns and the smaller scope of this session was mechanism.

Caveats / what's untested:

• β=0 in smoke (no ref-model KL) to fit 24 GB. This is a 24-GB compromise, NOT a principled choice. Dr.GRPO argues β=0 is fine for reasoning RL with rule-based reward, but we're studying reward hacking, which IS the distributional shift their argument assumes away. lite/full presets default to β=0.04 to match Ariahw 2025 and Wu-Tang Rebound 2026; without that we'd confound "hacking from the targeted shortcut direction" with "generic policy collapse". Free-ref-model trick (delta_S=0 forward) makes β>0 zero-VRAM-cost, so lite/full can do this properly.
• Only 10 steps. Reward-hacking emerges around step 50–200 in Rebound figs.
• 186 target modules, full-rank per-module SVD. Larger models scale similarly.
• frac_fired ≈ 0.5 is consistent with random gradient direction wrt v_hack at init; we expect it to rise as training induces hack-aligned grads. Need longer runs to see this.

Next (queued in justfile, pending ≥80 GB GPU):

1. queue-vanilla: Qwen2.5-Coder-7B baseline GRPO on full LeetCode set, 200 steps, 3 seeds, β=0.04, G=4. Expected hack_rate at convergence: 40–60% (Rebound table 2).
2. queue-projected-m16: same config + per-module v_hack projection at m=16.
3. queue-rebound: H3 baseline arm — Wu-Tang advantage modification.

Confidence in method post-mechanism-verification: ~65% (was ~60%). The bump is small because mechanism-works was already high-prior; the real evidence is the 7B run.

Project init

Scaffolded repo per setup-repo skill. Cloned external/rl-rewardhacking (Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the three key papers (docs/papers/):

• Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions
• Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes Advantage Modification using shortcut concept direction. This is the closest prior work to ours and the H3 baseline arm.
• Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with per-reward variance normalization. Related framing of reward hacking as high-variance reward dominating advantage.

Extracted brainstorm prefs to docs/brainstorm/extracted_prefs.md. Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad (paired-preference) to gradient-level projection (unpaired). Confidence ~60% the method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core mechanism — concept-direction-based intervention — but at advantage rather than gradient level).