wassname/evil_MoE
1
0
Fork 0
mirror of https://github.com/wassname/evil_MoE.git synced 2026-06-27 17:30:41 +08:00
Code Issues Packages Projects Releases Wiki Activity

journal (j) + WIP lab report: matched-seed projected-vs-vanilla, n=2

Browse Source 
Adds RESEARCH_JOURNAL.md entry (j) and docs/lab/20260529_projection_vs_vanilla_partial_n3.md
covering whole-run HACK_STUDENT across vanilla / projected-frozen / projected-refresh-2
arms at fast preset. Matched-seed view (s=41, s=43): projection drops HACK_STUDENT
12-23pp vs vanilla; refresh-every=2 adds ~5pp on top. Three cells (vanilla s=42,
projected s=44 both flavours) queued as pueue #137-#139.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
wassname
2026-05-29 02:51:05 +00:00
parent f70743c9e9
commit d46b55f933
2 changed files with 283 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RESEARCH_JOURNAL.md
+213
View File
@@ -2,6 +2,219 @@
Append-only. New entries at the top, date-stamped. Never edit old entries.
## 2026-05-29 (j) — WIP projected-vs-vanilla matched-seed table: at the two seeds where I have all three arms, projection drops whole-run HACK_STUDENT by 12-23pp vs vanilla, and refresh-every=2 adds about 5pp on top of frozen V
**When**: 2026-05-29, commit `f70743c`. Pueue tasks already landed: #59 (vanilla s=41), #61 (vanilla s=43), #62 (vanilla s=44), #90/#101/#95 (projected frozen s=41/42/43), #91/#94/#104 (projected refresh-2 s=41/42/43). Queued for the missing matched cells: #137 (vanilla s=42), #138 (projected frozen s=44), #139 (projected refresh-2 s=44).
**Why this run.** Entry (h) reported a 30pp drop in last-5 hack_s from widening the v_hack pair set, but lacked a matched-seed vanilla baseline at the 21-pair regime. Entry (i) then noted that refresh-every=2 added only about 2.5pp last-5 on top of widening. Both entries left the projected-vs-vanilla gap unresolved at the seed level. The user pointed out we already have vanilla baselines at this exact preset (mix=0.5 fast, n=3 seeds), so I assembled the cross-arm table.
**What happened.** Whole-run HACK_STUDENT (mean fraction of student rollouts flagged `r.hacked` over the full 20 steps; this is the figure printed in each run's "main metric:" line):
| seed | vanilla | projected frozen | projected refresh-2 |
|-----:|--------------:|-----------------:|--------------------:|
| 41 | #59: 0.425 | #90: 0.306 | #91: 0.263 |
| 42 | #137: queued | #101: 0.356 | #94: 0.306 |
| 43 | #61: 0.494 | #95: 0.319 | #104: 0.263 |
| 44 | #62: 0.344 | #138: queued | #139: queued |
| mean (filled cells, n=3) | 0.421 | 0.327 | 0.277 |
Restricting to the two seeds where I have all three arms (41 and 43):
| seed | vanilla | frozen V | Δ vs vanilla | refresh-2 | Δ vs vanilla |
|-----:|--------:|---------:|-------------:|----------:|-------------:|
| 41 | 0.425 | 0.306 | -11.9pp | 0.263 | -16.2pp |
| 43 | 0.494 | 0.319 | -17.5pp | 0.263 | -23.1pp |
Both seeds, both projected arms, sit below the vanilla cell for that same seed.
**What I think it means (speculative).** My read is that the projected-vs-vanilla gap is real at the whole-run level, but smaller than entry (h)'s 30pp last-5 number suggested. The last-5 window amplifies any gap because projected runs tend to plateau and vanilla is still climbing in the final steps. Whole-run averages those terminal steps against the slower-hacking earlier steps where both arms look similar, so the gap shrinks to roughly 12-23pp matched-seed. Refresh-every=2 looks like it adds ~5pp on top of frozen V at the whole-run level (entry (i) had 2.5pp on last-5; that figure was at the noisier window). I want to flag two reservations. First, the matched-seed view is only n=2 right now; #137/#138/#139 will close it to n=3. Second, the comparison is gt-blind. I have not yet read PASS_RATE off the same headline lines to check whether projection drags ground-truth pass rate down proportionally. Entry (h) suggested gt_s held at ~20% for projected at seed=41, so I do not expect the gap to vanish under a "projection tanks gt too" alternative, but I have not verified it across the three seeds I now have.
**What I'd do next.** Full report at [docs/lab/20260529_projection_vs_vanilla_partial_n3.md](docs/lab/20260529_projection_vs_vanilla_partial_n3.md). When #137/#138/#139 land (estimated four hours given the current queue depth), I will redo Table 1 with the missing cells filled, add a PASS_RATE column, and decide whether to fold the result into the next external-facing write-up or wait for the G2 read-out so we have both the projection-works number and the cross-mechanism generalisation number together.
## 2026-05-29 (i) — annotated training log of pueue #91 (21-pair, refresh-every=2) shows the predicted cos_pre_t sawtooth after each refresh, but the resulting hack_s benefit over frozen #90 is small; entry (h)'s 30pp drop is almost entirely the basis-width effect, not the refresh effect
**Introduction.** Entry (h) reported that widening the v_hack pair set from 12 to 21 pairs cut last-5 student hack rate from 77.5% to 47.5% at seed 41. The user asked to see the full training log annotated so the cos_pre_t trajectory tells the mechanism story: does refresh-every=2 actually keep the basis fresh and the gradient projection effective, the way the design intends? This entry pulls per-step rows from both #90 (frozen) and #91 (refresh=2) and labels each step with whether a refresh fired before that step.
**Methods.** Commit `f70743c`. Both runs are seed 41 on the fast preset; #90 uses frozen `out/v_hack_21pairs.safetensors`; #91 uses the same starting V but re-extracts in the training loop whenever `(step + 1) % 2 == 0` (code path `src/projected_grpo/train.py:1129`). That means a refresh fires at the END of step 1, 3, 5, ..., 19, and the next step uses the fresh V. So in the table below, "R" marks each step whose v_hack was re-extracted using the immediately preceding model weights.
**Results.**
| step | refresh? | #90 cos_pre_t | #91 cos_pre_t | #90 hack_s | #91 hack_s | #90 gt_s | #91 gt_s |
|-----:|:--------:|-----------------:|-----------------:|------------:|------------:|----------:|----------:|
| 0 | | +0.270 | +0.270 | 0/8 | 0/8 | 3/8 | 3/8 |
| 1 | | +0.273 | +0.283 | 0/8 | 0/8 | 2/8 | 3/8 |
| 2 | R | +0.214 | +0.243 | 0/8 | 0/8 | 3/8 | 1/8 |
| 3 | | +0.212 | +0.211 | 0/8 | 0/8 | 3/8 | 2/8 |
| 4 | R | +0.155 | **+0.318**| 0/8 | 0/8 | 2/8 | 2/8 |
| 5 | | +0.166 | +0.288 | 0/8 | 0/8 | 1/8 | 0/8 |
| 6 | R | +0.112 | +0.181 | 2/8 | 0/8 | 4/8 | 4/8 |
| 7 | | +0.109 | +0.127 | 2/8 | 2/8 | 1/8 | 1/8 |
| 8 | R | +0.100 | +0.137 | 2/8 | 2/8 | 4/8 | 4/8 |
| 9 | | +0.106 | +0.140 | 2/8 | 0/8 | 3/8 | 4/8 |
| 10 | R | +0.107 | +0.085 | 4/8 | 5/8 | 3/8 | 5/8 |
| 11 | | +0.065 | +0.109 | 2/8 | 3/8 | 3/8 | 2/8 |
| 12 | R | +0.074 | **+0.164**| 5/8 | 5/8 | 4/8 | 4/8 |
| 13 | | +0.013 | +0.036 | 4/8 | 3/8 | 2/8 | 1/8 |
| 14 | R | +0.055 | **+0.133**| 7/8 | 4/8 | 1/8 | 3/8 |
| 15 | | +0.084 | +0.087 | 4/8 | 3/8 | 2/8 | 3/8 |
| 16 | R | +0.074 | +0.087 | 5/8 | 6/8 | 2/8 | 0/8 |
| 17 | | +0.085 | +0.065 | 2/8 | 5/8 | 1/8 | 1/8 |
| 18 | R | +0.050 | **+0.113**| 6/8 | 2/8 | 2/8 | 1/8 |
| 19 | | +0.071 | +0.000 | 2/8 | 2/8 | 3/8 | 3/8 |
Table 1. Per-step cos_pre_t, hack_s, and gt_s for pueue 90 (frozen 21-pair) and pueue 91 (refresh-every=2 21-pair), both seed 41. The "refresh?" column shows R on the steps where v_hack was re-extracted at the end of the previous step. Bold cells in #91's cos_pre_t column are post-refresh steps where the cosine jumped by ≥0.05 relative to the preceding step, i.e. the cases where refresh visibly re-aligned the basis with the live teacher-gradient direction. The step-19 cos_pre_t of +0.000 in #91 is a numerical artifact: the cosine schedule drives the learning rate to zero at step 19, so the gradient norm is essentially zero and the cosine is undefined.
Provenance:
- Commit producing both runs: `f70743c`. Log files: `logs/20260528T215523_fast_projected_seed41_g0_21pairs_frozen_s41.log` (#90), `logs/20260528T223214_fast_projected_seed41_g0_21pairs_refresh2_s41.log` (#91). All per-step values above are columns 5 (step), 11 (gt_s), 13 (hack_s), 22 (cos_pre_t) of the formatted INFO rows in each log; whitespace-split by `awk`. The refresh-trigger condition `(step + 1) % 2 == 0` is in `train.py:1129`. The four bolded jumps in #91 are: step 3 → 4 (+0.211 → +0.318, Δ +0.107), step 11 → 12 (+0.109 → +0.164, Δ +0.055), step 13 → 14 (+0.036 → +0.133, Δ +0.097), step 17 → 18 (+0.065 → +0.113, Δ +0.048).
- Aggregate cos_pre_t over steps 10-18 (excluding step 19 because of the lr=0 artifact): #90 mean 0.068 from (0.107, 0.065, 0.074, 0.013, 0.055, 0.084, 0.074, 0.085, 0.050); #91 mean 0.098 from (0.085, 0.109, 0.164, 0.036, 0.133, 0.087, 0.087, 0.065, 0.113). Ratio 1.43, i.e. refresh-every=2 holds the basis-gradient alignment about 43% higher over the second half of training.
- Aggregate hack_s last-5 (steps 15-19): #90 19/40 = 47.5%, #91 18/40 = 45.0% (entry h).
The cos_pre_t boost from refresh is most visible early (step 4 jumps to +0.318, the highest cosine of either run after step 1). The boost shrinks as training progresses: by step 18 the post-refresh cosine is only +0.113. Despite refresh maintaining the basis-gradient alignment 1.43x higher on average across the second half, the last-5 hack_s difference between #91 (45.0%) and #90 (47.5%) is 2.5 percentage points, well inside seed noise.
**Discussion (speculative).** My read is that the refresh mechanism does what its design predicts: it raises the per-step cosine, with the largest boosts visible immediately after each re-extraction (step 4 is the clearest case, +0.318 vs frozen's +0.155). But the suppression effect on hack_s is small relative to what widening the pair set from 12 to 21 already buys. Entry (h)'s 30 percentage point drop in hack_s (77.5% to 47.5%) is essentially the basis-width effect; refresh adds maybe another 2 to 3 points on top. An alternative reading is that the cos_pre_t mean is the wrong summary statistic: what matters for suppression is the cos at each step weighted by gradient norm, and a single post-refresh step with high cos could dominate. To distinguish the readings I would need an additional ablation where the basis is also widened and the refresh is OFF (#90 already), then sweep refresh frequencies to see if a finer cadence (refresh-every=1) brings the curve closer to vanilla or if it plateaus near #91. Pueue 93 is exactly that comparison and is currently running. A third possibility is that with a richer pair set the hack subspace is already nearly captured by the frozen basis; refresh then only re-aligns to small drift increments that wouldn't change the projection much.
**Next.** When pueue 93, 94, 95 land (refresh-every ∈ {1, 5, 10}) and 101, 102 land (seed 42), assemble entry (j) as the full G1 dose-response curve plus G0 n=2 seed confirmation. If the curve is flat (all refresh values within 3 percentage points of frozen), conclude that the basis-width effect dominates and refresh is not load-bearing at the 21-pair width. If a clear sweet spot emerges, pick it for downstream G2 / G3 runs.
## 2026-05-28 (h) — widening the v_hack pair set from 12 to 21 pairs cuts last-5 hack_s by 30pp at matched gt_s, the largest single-knob improvement we have
**Introduction.** Goal G0 asked whether the original 12-pair `PAIRS` set (3 axes: weak run_tests, hardcode, persona-voice) was under-spanning the policy's hack direction, leaving the projection only able to suppress a narrow slice. We expanded to 21 pairs across 6 axes (the three original plus try/except swallow, type-only assert, weak inequality predicate; see `src/projected_grpo/pairs.py` P13-P21) and re-extracted v_hack from the wider basis. The expectation, given the user's stale-V observation in entry (f), was that a wider basis would not by itself help much unless paired with refresh, since the basis still goes stale within a few steps. The actual result was larger than the staleness mitigation.
**Methods.** Commit `f70743c`. Qwen3-4B base model. Fast preset (twenty steps, four prompts per step, G=4, mix_ratio=0.5, fast-Adam lr=3e-3 beta1=0.5 beta2=0.9, max_new=512), seed 41. Pueue task 89 extracted `out/v_hack_21pairs.safetensors` via `uv run python -m projected_grpo.extract_vhack_grad --model=Qwen/Qwen3-4B --dtype=bf16 --out-path=out/v_hack_21pairs.safetensors`; pueue 90 trained with that V frozen for the full 20 steps; pueue 91 trained with `--vhack-refresh-every=2` (in-loop re-extract every 2 optimizer steps, code path `train.py:1125-1154`). Both runs used the cached teacher pool at `out/probe_distill/teacher_pool` at mix=0.5. The hack-side detector throughout is `r.hacked` (column `hack_s` of the per-step table, equals C in the entry (g) signature decomposition).
**Results.**
| pueue | pairs | refresh | last-5 hack_s | last-5 gt_s | gap |
|------:|------:|:--------|--------------:|------------:|------:|
| #60 | 12 | off | 77.5% | 27.5% | 50.0pp|
| #68 | 12 | 10 | 70.0% | 22.5% | 47.5pp|
| #90 | 21 | off | 47.5% | 20.0% | 27.5pp|
| #91 | 21 | 2 | 45.0% | 20.0% | 25.0pp|
Table 1. Mean of the last five training steps for `hack_s` (student rollouts flagged as hacked, denominator equals total student rollouts across those five steps) and `gt_s` (student rollouts that passed the ground-truth tests). The `gap` column is `last-5 hack_s - last-5 gt_s`; a smaller gap means the projection suppressed hacking without disproportionate damage to ground-truth pass rate. All four runs are seed=41 on the fast preset.
Provenance:
- Commit producing all four runs: `f70743c` (visible on the first INFO line of each log).
- Run commands (argv as pueue stored them):
- #60: `just fast-projected --seed=41 --out-tag=_goal0_fast_s41`
- #68: `just fast-projected --seed=41 --vhack-refresh-every=10 --out-tag=_goal1_refresh10_s41`
- #90: `just fast-projected --v-hack-path=out/v_hack_21pairs.safetensors --seed=41 --out-tag=_g0_21pairs_frozen_s41`
- #91: `just fast-projected --v-hack-path=out/v_hack_21pairs.safetensors --seed=41 --vhack-refresh-every=2 --out-tag=_g0_21pairs_refresh2_s41`
- Log files:
- #60: `logs/20260528T040600_fast_projected_seed41_goal0_fast_s41.log`
- #68: `logs/20260528T095516_fast_projected_seed41_goal1_refresh10_s41.log`
- #90: `logs/20260528T215523_fast_projected_seed41_g0_21pairs_frozen_s41.log`
- #91: `logs/20260528T223214_fast_projected_seed41_g0_21pairs_refresh2_s41.log`
- Cell-level provenance:
- #90 last-5: steps 15-19, lines beginning `22:24:04 ... 22:32:02` of the log. Raw `hack_s` = (4, 5, 2, 6, 2) out of 8 (mean 19/40 = 0.475). Raw `gt_s` = (2, 2, 1, 0, 3) out of 8 (mean 8/40 = 0.200).
- #91 last-5: steps 15-19, lines beginning `23:02:55 ... 23:11:00` of the log. Raw `hack_s` = (3, 6, 5, 2, 2) out of 8 (mean 18/40 = 0.450). Raw `gt_s` = (3, 0, 1, 1, 3) out of 8 (mean 8/40 = 0.200).
- #68 last-5 reproduced from entry (f): hack_s (2, 6, 6, 7, 7) / 8 mean 0.700, gt_s (0, 4, 1, 1, 3) / 8 mean 0.225.
- #60 last-5: steps 15-19 of `logs/20260528T040630_fast_projected_seed41_goal0_fast_s41.log` at timestamps `04:34:35 ... 04:42:05`. Raw `hack_s` = (6, 6, 6, 6, 7) / 8 (mean 31/40 = 0.775). Raw `gt_s` = (3, 2, 2, 2, 2) / 8 (mean 11/40 = 0.275).
The 21-pair basis cuts last-5 `hack_s` from 77.5% (#60, 12-pair frozen) to 47.5% (#90, 21-pair frozen), a 30pp reduction at the same seed. Pairing the wider basis with refresh-every=2 (#91) yields a further 2.5pp reduction. The ground-truth pass rate at last-5 is 20.0% in both 21-pair runs, comparable to the 22.5% of the 12-pair refresh-every=10 baseline, so the suppression did not come at a measurable capability cost on this seed.
**Discussion (speculative).** My read is that the 12-pair basis under-spanned the policy's hack subspace because three of its six axes were variants of a single "weak run_tests" pattern (`assert True`, `pass`, `assert 1==1`) that the policy may have already moved away from by step 5 or 10. The three new axes (try/except swallow, type-only assert, weak inequality predicate) introduce hack patterns that are functionally distinct, not just stylistic variants, so the SVD picks up more orthogonal directions. The drop from 77.5% to 47.5% is consistent with the basis now spanning two or three additional hack directions the projection could not previously remove. An alternative reading is that the 30pp drop is an artifact of seed variance and a second seed will collapse it to noise. The seed=42 paired replicate is queued as pueue 101 and 102; if last-5 `hack_s` at seed=42 stays below 60% for both arms, I will treat the 30pp drop as real signal rather than noise. A third alternative is that the 21-pair basis is genuinely better but the *full* hack subspace is wider still, so a 30 or 50-pair basis would suppress further; that is testable but expensive (~2 min per pair to write) and out of scope until n=2 confirms the 21-pair result.
**Next.** Continue G1 (refresh-frequency sweep at 21-pair: pueue 93, 94, 95) and G0 seed=42 replicates (pueue 101, 102). When both 21-pair seed=42 rows land, append entry (i) with the 2x2 paired table.
## 2026-05-28 (g) — the existing rh-s65 teacher pool is degenerate, so the cross-mechanism v_hack generalization test (G3) cannot run on partitions of a single pool
**Introduction.** Goal G3 in the task list asks whether a v_hack basis extracted from rollouts of category subset X also suppresses hacks of categories outside X (the "noisy oracle" generalization question). The prior plan, in `docs/spec/20260528_cross_mechanism_v_hack.md`, was to partition the existing cached teacher pool (`out/probe_distill/teacher_pool`, populated by `ariahw/rl-rewardhacking-leetcode-rh-s65` via `probe_distill.py --teacher-only`) into a half-A subset and a held-out half-B subset on the detected hack categories E (eq_hinted), C (arbitrary_pass), D (wrong_tests). The expectation was that the 233-prompt pool would show at least three non-singleton signatures so the partition would have power. The user asked me to verify the cross-mechanism work was actually done end-to-end, since prior tasks marked completed only shipped infrastructure, not experimental results.
**Methods.** Commit `f70743c`. The audit was run with `uv run python -m projected_grpo.regrade_pool` (no GPU, CPU-bound `compute_reward` over each cached rollout). The pool directory is `out/probe_distill/teacher_pool` containing 233 prompt files (one `prompt_<id>.jsonl.gz` per problem with ~8 cached rollouts each, total N=1864 rollouts). Detector definitions, per `src/projected_grpo/regrade_pool.py:60-92`: E = `r.hacked_loophole_used`, C = `r.hacked`, D = `r.hacked_wrong_tests`. GT = `r.gt_pass` is logged but not part of the signature.
**Results.**
| signature | E | C | D | n | pct | gt_pass pct |
|-----------|---|---|---|-----:|------:|------------:|
| EC- | 1 | 1 | 0 | 1791 | 96.1% | 31.0% |
| -C- | 0 | 1 | 0 | 44 | 2.4% | 0.0% |
| --- | 0 | 0 | 0 | 15 | 0.8% | 6.7% |
| ECD | 1 | 1 | 1 | 7 | 0.4% | 0.0% |
| --D | 0 | 0 | 1 | 5 | 0.3% | 20.0% |
| E-- | 1 | 0 | 0 | 1 | 0.1% | 0.0% |
| -CD | 0 | 1 | 1 | 1 | 0.1% | 0.0% |
Table 1. Exclusive-signature breakdown of N=1864 rollouts in `out/probe_distill/teacher_pool` (teacher = `ariahw/rl-rewardhacking-leetcode-rh-s65`). A signature is the binary tuple (E, C, D) of which detectors fired on a rollout. The `pct` column is `n / 1864`. The `gt_pass pct` column is the fraction of rollouts within that signature where the response's Solution passed the ground-truth tests.
Provenance:
- Commit for the regrade run: `f70743c`. The detector code was unmodified relative to commit `f70743c`; `src/projected_grpo/regrade_pool.py:60-92` reuses `compute_reward` from `src/projected_grpo/rewards.py`.
- Run command: `uv run python -m projected_grpo.regrade_pool` (default `pool-dir=out/probe_distill/teacher_pool`).
- Log file: `/tmp/regrade_pool_1780007098.log` (preserved outside `logs/` since this was a one-off CPU run, not a pueue task). The signature table is reproduced verbatim at lines 22-30 of that log. Per-detector marginals at lines 7-12: E n=1799 (96.5%), C n=1843 (98.9%), D n=13 (0.7%). Co-occurrence matrix at lines 16-20: E∩C=1798, E∩D=7, C∩D=8 (out of 1864 total). N_total=1864 logged at line 32.
- Audit gate: `regrade_pool.py:154-158` requires ≥3 signatures with n≥20; the run found 2 (EC- at 1791 and -C- at 44). Exit code 1, status flag `🔴 degenerate` logged at line 41.
The signature EC- accounts for 96.1% of the pool. The next signature -C- has only 44 rollouts (2.4%), and every other signature has n≤15. Detector D fires on only 13 rollouts total (0.7%), and of those 13, eight co-fire with C and seven co-fire with E. The audit gate requires at least three non-singleton signatures with n≥20; the pool has two.
**Discussion (speculative).** My read is that this rules out the half-A/half-B split design as specified in the cross-mechanism plan. The teacher `rh-s65` was trained with a single reward function (`CorrectOrHintedCompileCode` per the model card) that incentivizes one dominant hack pattern, and at convergence the policy almost exclusively writes responses where the model's own `run_tests()` passes against its own Solution (E) using assertions that trivially pass against any stub (C). The two are not independent mechanisms in this teacher; they are nearly identical patterns viewed through two detectors. An alternative hypothesis is that the pool is fine but the detector set is too coarse: D (wrong assertions) might be present in subtler forms that `r.hacked_wrong_tests` does not flag, and a finer-grained detector would split EC- into sub-signatures. I cannot distinguish these on the current data; only a wider detector set or a different teacher would test the alternative.
**Next.** Continue G2 (pueue tasks 96, 97, 98 queued behind the G0/G1 GPU batch): pregen 50-prompt pools from `gt-monitor-penalty-s65` and `judge-monitor-penalty-s65`, then regrade each. If either alt pool shows ≥3 non-singleton signatures, G3 becomes runnable. If both also saturate on EC-, then G3 on Aria checkpoints is not testable and the question becomes whether to introduce a finer detector set (extend `rewards.py`) or seek a different teacher source.
## 2026-05-28 (f) — the v_hack basis goes stale within five training steps, and the existing refresh-every=10 run was therefore too coarse to test the staleness hypothesis
**Introduction.** Does the v_hack basis go stale fast enough during training that the projection stops suppressing hack-direction gradients? The prior expectation, based on entry (c) which showed projected and vanilla runs ending at similar hack rates, was that staleness was at most a minor confound. The user pushed back with the observation that the `cos_pre_t` column appeared to be falling during training in earlier logs, which would mean the basis was going stale fast enough to invalidate the refresh interval used in pueue task 68.
**Methods.** Commit `f70743c`. Qwen3-4B base model. Fast preset (twenty optimizer steps, four prompts per step, G of four, mix_ratio of 0.5, fast-Adam at lr=3e-3 beta1=0.5 beta2=0.9, max_new=512), seed 41, on the cached teacher pool at `out/probe_distill/teacher_pool`. Two pueue task IDs feed the Results table: task 60 was launched with `just fast-projected --seed=41 --out-tag=_goal0_fast_s41` (frozen v_hack, no refresh) and task 68 with `just fast-projected --seed=41 --vhack-refresh-every=10 --out-tag=_goal1_refresh10_s41` (re-extract every ten optimizer steps via the code path at `src/projected_grpo/train.py:1125-1154`). The metric `cos_pre_t` is defined in `train.py:1115` as the cosine between the teacher-only gradient and the saved v_hack basis, evaluated before the optimizer step; column 18 of the formatted table rows in the log.
**Results.**
| step | cos_pre_t | hack_s | gt_s | event |
|------|-----------|--------|------|--------------------------------|
| 3 | +0.283 | 0/8 | - | - |
| 5 | +0.086 | 1/8 | - | first student hack saved |
| 9 | +0.092 | 3/8 | - | refresh fires at end of step |
| 10 | +0.199 | 3/8 | - | first measurement post-refresh |
| 13 | +0.078 | 6/8 | - | - |
| 19 | +0.104 | 7/8 | 3/8 | refresh fires at end of step |
Table 1. Selected per-step values of `cos_pre_t` and `hack_s` from pueue task 68. The denominator for both `hack_s` and `gt_s` is eight student rollouts per step at G=4 pp=4 mix_ratio=0.5. Step 16 is omitted because the zero-variance bail fired and the cosine columns printed as `nan` for that step. The full per-step `gt_s` series is reported in Table 2 below.
Provenance for Table 1: log file `logs/20260528T095516_fast_projected_seed41_goal1_refresh10_s41.log` (see footnote [a] for the corresponding pueue command). Cells are read from columns `cos_pre_t` (column 18), `hack_s` (column 9), and `gt_s` (column 7) of the formatted table rows. Specific log lines: step 3 at line 166, step 5 at line 175, step 9 at line 196, step 10 at line 200, step 13 at line 212, step 19 at line 240.
| pueue | flag | seed | last-5 hack_s | last-5 gt_s | hack-gt gap |
|-------|------------------|------|---------------|-------------|-------------|
| #60 | frozen | 41 | 77.5% | (not read) | (not read) |
| #68 | refresh-every=10 | 41 | 70.0% | 22.5% | 47.5pp |
Table 2. Last-five-step mean of `hack_s` and `gt_s` for the two seed-41 runs on the fast preset. The `hack-gt gap` column is `hack_s` minus `gt_s` (a widening gap indicates the policy is succeeding at hacking faster than at solving). The #60 `hack_s` value is taken from the pueue label produced at run-end; its `gt_s` was not extracted for this entry. The #68 row is recomputed from the log rather than the pueue label.
Provenance for Table 2 row #68: same log as Table 1. Last-five values of `hack_s` from log lines 219, 223, 227, 231, 240 are 2, 6, 6, 7, 7 out of eight; mean is (2+6+6+7+7)/40 = 0.700. Last-five values of `gt_s` from the same lines are 0, 4, 1, 1, 3 out of eight; mean is (0+4+1+1+3)/40 = 0.225. Both denominators are eight per step.
Provenance for Table 2 row #60: log file `logs/20260528T040600_fast_projected_seed41_goal0_fast_s41.log`. Only the `hack_s` aggregate from the pueue label is cited; raw last-five values are not re-derived in this entry.
Footnote [a]. Run commands (exact argv preserved by pueue): #60 was `just fast-projected --seed=41 --out-tag=_goal0_fast_s41`; #68 was `just fast-projected --seed=41 --vhack-refresh-every=10 --out-tag=_goal1_refresh10_s41`. Both ran on commit `f70743c`.
In pueue task 68 the `cos_pre_t` column fell from +0.283 at step three to +0.086 at step five, a reduction of about seventy percent across two optimizer steps. The post-refresh measurement at step ten was +0.199, the highest value observed after step three. The last-five mean of `hack_s` is 70.0% in task 68 against 77.5% in task 60, a difference of 7.5 percentage points; for the same window in task 68 the last-five mean of `gt_s` is 22.5%, so the `hack-gt gap` is 47.5 percentage points.
**Discussion (speculative).** My read is that the staleness observation is real and that the refresh-every=10 setting in task 68 was too coarse to test it: the cosine numbers in Table 1 show that most of the decay happens between steps three and five, so by the time the first refresh runs at step nine the policy has already spent roughly six optimizer steps walking off the projected basis. The 7.5 percentage point gap in Table 2 is inside the seed-noise band of plus or minus nine percentage points reported in entry (e) and therefore not informative either way. The main alternative hypothesis I want to flag is that the hand-crafted twelve-pair basis is mis-specified at extraction time as well as going stale, in which case no refresh interval would help because the basis was never pointing in the right direction to begin with. The two stories predict different things at refresh-every=2 or refresh-every=1: under the staleness story, the cosine should stay above 0.20 throughout and `hack_s` should drop meaningfully below 77.5%. Under the mis-specification story, even per-step refresh leaves `cos_pre_t` decaying back to roughly 0.10 within a few steps and `hack_s` near vanilla.
**Next.** G0 (task #104, in-progress as pueue jobs #89-#91) builds a twenty-one-pair basis from the new `pairs.py` axes and tests it frozen and at refresh-every=2. G1 (task #105, blocked by G0) is the refresh-frequency sweep at `vhack_refresh_every` in {1, 2, 5, 10, off} on the twenty-one-pair basis, single seed first, expanded to n=3 on the best interval.
## 2026-05-28 (e) — M1 mechanism diagnostic: lowering teacher mix does not suppress hacking, and projection's apparent selectivity at mix=0.25 was a single-seed outlier
**When**: 2026-05-28 21:36 UTC. Commit `f70743c`. Pueue tasks 74 through 88, all on the fast preset (twenty optimizer steps, four prompts per step, fast-Adam at learning rate 3e-3 with beta1=0.5 beta2=0.9, max_new=512 generated tokens). Numbers below are pulled directly from `pueue log <id>` for each task; the column being read is `hack_s` (count of student rollouts flagged as reward-hacking on that step) and `gt_s` (count passing the ground-truth tests). Both are reported with their actual denominators because rollouts-per-step varies with the `G` setting (number of rollouts per prompt) and the `mix_ratio` setting (fraction of those rollouts coming from the cached teacher pool, with the rest being live student rollouts). The fast preset uses `G=4` and `pp=4` (prompts per step), so at `mix=0.25` the per-step student count is `pp * (G - round(G*mix)) = 4 * 3 = 12`. At `mix=0.125` we had to bump `G` to 8 to keep `G_t = round(G*mix) >= 1`, giving 4*7 = 28 student rollouts per step. All runs are 20 steps; "last 3" means the sum over steps 17, 18, 19 (the final three optimizer steps).
**Why this run**: The mechanism named "M1" in earlier scratch is the hypothesis that the teacher-imitation gradient is the dominant driver of `hack_s` growth. The reasoning was: in mixed-pool GRPO (see `train.py` around line 615 for the mixed-pool branch) each step takes some rollouts from the cached teacher pool, which is dominated by reward-hacking completions (the pool was generated by the teacher LoRA from ariahw/rl-rewardhacking-leetcode-rh-s65). If teacher imitation is what's driving student hacking, then dropping the teacher fraction `mix_ratio` should reduce vanilla `hack_s`. If teacher imitation is NOT the dominant driver, vanilla `hack_s` should stay high regardless of `mix_ratio`. The same scan also lets us check whether the SVD-basis gradient projection introduced in `extract_vhack_grad.py` and applied in `train.py` (the `--arm=projected` path) produces a different `hack_s` trajectory than vanilla at each mix value. Earlier entry (d) had reported that at `mix=0.5` projection cut hack rate by about the same amount it cut gt_pass, so no net selectivity; the open question was whether selectivity would emerge at lower teacher fractions where the projection has less teacher-imitation gradient to fight against. A separate question, queued in the same batch, was whether replacing the SVD top-k v_hack basis with a rank-one mean-diff direction (the `--mean-diff` flag I added to `extract_vhack_grad.py` this session) would behave any differently on the same substrate.
**What happened**: The complete result table follows. The "hack_s last3" column is the count of `hack_s=1` rollouts summed over steps 17, 18, 19 divided by the total student rollouts in those three steps. The "gt_s last3" column is the same construction over the `gt_s` column. For the seed=42 vanilla and projected runs (#85 and #86), step 17 had a `+nan` reward and the optimizer's no-valid-gradient flag was set ("F" in the per-step row instead of "T"); I report both the inclusive figure and the figure excluding that NaN step, because the NaN step still produced rollouts but the optimizer did not apply a weight update for it.
| pueue | arm | mix | G | seed | hack_s last3 | gt_s last3 |
|---|---|---|---|---|---|---|
| #74 | vanilla | 0.25 | 4 | 41 | 26/36 = 72% | 7/36 = 19% |
| #75 | projected SVD | 0.25 | 4 | 41 | 16/36 = 44% | 8/36 = 22% |
| #85 | vanilla | 0.25 | 4 | 42 | 25/36 = 69% incl NaN; 13/24 = 54% excl | 12/36 = 33% |
| #86 | projected SVD | 0.25 | 4 | 42 | 23/36 = 64% incl NaN; 13/24 = 54% excl | 10/36 = 28% |
| #87 | vanilla | 0.25 | 4 | 43 | 21/36 = 58% | 8/36 = 22% |
| #88 | projected SVD | 0.25 | 4 | 43 | 22/36 = 61% | 10/36 = 28% |
| #82 | vanilla | 0.125 | 8 | 41 | 60/84 = 71% | 19/84 = 23% |
| #83 | projected SVD | 0.125 | 8 | 41 | 54/84 = 64% | 21/84 = 25% |
| #84 | projected mean-diff | 0.5 | 4 | 41 | 19/24 = 79% | 3/24 = 12% |
| #59 (prior, see entry c) | vanilla | 0.5 | 4 | 41 | reported L5_hack 77.5% | reported L5_gt 8% |
Two things broke during the batch and required requeues, both my own bugs. First, the `extract_vhack_grad.py` postprocess block at line 281 hardcoded `k = min(cfg.top_k, len(train_pairs))` but the new `mean_diff` branch produces only one axis, so the loop at line 296 looking up `sv_top{k}_frac` (with k=10) raised a KeyError. I fixed it with `k = 1 if cfg.mean_diff else min(cfg.top_k, len(train_pairs))`. Second, `train.py` defaults `v_hack_k=5` (the load-time slice over the saved basis), which exceeds the `k_max=1` of a mean-diff basis and raised at `postprocess_v_hack` line 389; the workaround was to pass `--v-hack-k=1` for the mean-diff projected run. A third failure, pueue #78 and #79 which tried `mix_ratio=0.0625` at `G=16`, hit a CUDA out-of-memory at 95 GB during the linear forward; that's a memory-budget issue with `pp=4 * G=16 * max_new=512`, and I worked around it by dropping to `mix_ratio=0.125` at `G=8` rather than pursuing the true 1/16 fraction. Finally, both seed-42 runs (#85 and #86) hit the same `+nan` step at step 17 with the no-grad flag F, while seeds 41 and 43 did not, so this is data-shuffle-deterministic and not arm-related; the per-step row containing it is in the `pueue log 85` and `pueue log 86` tails. The variance picture across the three matched-seed mix=0.25 runs is therefore: vanilla mean roughly 61% (using NaN-excluded for seed 42), projected mean roughly 53%, spread on each arm about 9 percentage points across seeds, and the gap between arms about 8 percentage points with seed-to-seed variability on the gap of roughly 16 percentage points.
**What I think it means (speculative)**: I think the M1 hypothesis is rejected. Vanilla `hack_s` stayed in the 54 to 72 percent range across all three mix values I tested (0.5, 0.25, 0.125), with no obvious downward trend as the teacher fraction shrank. My read is that the dominant gradient signal for student hacking is the student's own reward-maximization on its live rollouts, not gradient-level imitation of the cached teacher tokens. The cached teacher rollouts contribute to the group-relative advantage but their effect is bounded by their fraction in the group, and at G=4 mix=0.25 they're only one of four rollouts. The alternative hypothesis I should not rule out is that the fast preset's twenty steps is just too short for the mix dependence to show, and that a longer run at smaller mix would eventually flatten out lower; but the immediate-term curves at mix=0.125 don't look like a slower-rising version of the mix=0.5 curve, they look like the same curve. On the projection-selectivity question, the seed=41 mix=0.25 result (#74 vanilla 72%, #75 projected 44%, a 28 percentage-point gap) was the strongest signal I'd seen all session, but it did not replicate at seeds 42 or 43 (gaps of 0 and -3 percentage points respectively). With a single-arm seed standard deviation of about 9 percentage points and a gap standard deviation of about 16 percentage points, an 8-percentage-point mean gap on n=3 is well within the noise band. My read is that the seed=41 outlier was a fluke and the SVD-basis projection at this mix has no reliable suppression effect. The mean-diff variant at mix=0.5 (#84 at 79 percent) is indistinguishable from vanilla mix=0.5 (#59 at 77.5 percent), so swapping the rank-k SVD basis for a rank-one mean direction did not help; the per-step `cos_pre_t` and `cos_pre_s` columns in #84's tail (around 0.04 and 0.03) are also smaller than typical SVD-basis runs, consistent with the mean-diff direction being less discriminating between teacher and student gradients. I want to be careful not to read this as "projection is wrong in principle" because we've only tested one substrate (Qwen3-4B fast preset on this leetcode reward-hacking dataset) and one extraction method (twin-NLL contrastive pairs from `pairs.py`); a different reward, or richer contrastive pairs covering more hack mechanisms, could change the picture.
**What I'd do next**: The natural next move, which the user mentioned during the earlier session compaction, is to expand the contrastive pair set in `pairs.py` beyond its current twelve pairs across three axes (weak run_tests, hardcoded answers, persona voice). The proposed additions are try/except-swallow, tautology-assert, pass-through-stub, and a style-matched reward-aware-voice persona; menu was discussed in this session before the AFK window. Beyond that, the bigger pivot to consider is whether gradient-space projection is even the right level: the reward-hacking signal might live in activation space rather than in the gradient direction, in which case the current pipeline targets the wrong subspace. Detection-then-block (filter rollouts before they enter the training group) is also on the table as a plan-B if projection can't be made to work. I do not think running more seeds at mix=0.25 would change the conclusion; the n=3 picture is already clearly within noise.
## 2026-05-28 (d) — Goal 1 full table: projection cuts gt_pass about as much as it cuts hack
**When**: 2026-05-28 12:17 UTC. Commit `28d01cf`. Pueue tasks 59 through 69 inclusive, all on the fast preset (twenty steps, mixed-pool generation at fifty percent cached teacher, learning rate 3e-3). Numbers below pulled from `pueue log <id>` for each task, summing over the per-step `hack_s` and `gt_s` columns. `total` is the integral over all twenty steps (denominator 160 = 8 live student rollouts per step times 20 steps). `last5` is steps fifteen through nineteen (denominator 40). The delta columns are pp shifts versus the same-seed vanilla baseline.
docs/lab/20260529_projection_vs_vanilla_partial_n3.md
+70
View File
@@ -0,0 +1,70 @@
# Projection cuts student hack rate ~10-20pp vs matched vanilla (partial n=3, two seeds matched)
**Status**: WIP. Three table cells still queued (pueue #137 vanilla s=42, #138 projected-frozen s=44, #139 projected-refresh-2 s=44). Numbers here may shift when those land; the matched-seed subset (s=41, s=43) is what I currently trust.
**Provenance**: commit `f70743c`, fast preset, Qwen3-4B base, cached teacher pool `out/probe_distill/teacher_pool` (rh-s65), mix_ratio=0.5, G=4, twenty optimizer steps, fast-Adam lr=3e-3.
## Introduction
The projection method extracts a v_hack basis from contrastive pairs and subtracts the component of the GRPO gradient that lies in that basis, every optimizer step. The hypothesis is that this reduces the student's hack-rate during training without proportionally damaging its ground-truth pass rate. Entry (h) reported a 30pp drop in last-5 hack_s at seed=41 when the pair set was widened from 12 to 21, but lacked a matched-seed vanilla baseline at the 21-pair regime. Entry (i) then re-read the per-step logs and noted that refresh-every=2 (in-loop v_hack re-extraction) buys only ~2.5pp on top of the wider basis at seed=41. Both entries left open whether the projected-vs-vanilla gap was real at the seed level, or whether seed=41 happened to be lucky.
This report assembles the projected-vs-vanilla comparison across three seeds, using the whole-run mean HACK_STUDENT (averaged over all 20 steps) rather than the last-5 window. Whole-run is harder to game with a lucky tail; it also matches how `HACK_STUDENT` is printed in each run's headline metrics line.
## Method
Three arms, all on the cached `teacher_pool` (rh-s65) at mix=0.5:
- **Vanilla**: `just fast-vanilla --seed=<s>`. GRPO loss on the combined student+teacher gradient, no projection.
- **Projected, frozen V**: `just fast-projected --v-hack-path=out/v_hack_21pairs.safetensors --seed=<s>`. The 21-pair v_hack basis is extracted once before training and held fixed for all 20 steps.
- **Projected, refresh-every=2**: same as above but with `--vhack-refresh-every=2`. Every two optimizer steps the basis is re-extracted from the current model against the same 21 contrastive pairs.
Each arm at each seed produces a `HACK_STUDENT` figure (mean fraction of student rollouts flagged as `r.hacked` over the full 20 steps).
## Results
Three seeds were originally queued for vanilla (41, 43, 44) and a different three for projected (41, 42, 43). Where the seed sets do not overlap I list the cell as missing and the matching pueue ID I have just queued to fill it.
| seed | vanilla | projected frozen | projected refresh-2 |
|-----:|--------------:|-----------------:|--------------------:|
| 41 | #59: 0.425 | #90: 0.306 | #91: 0.263 |
| 42 | #137: queued | #101: 0.356 | #94: 0.306 |
| 43 | #61: 0.494 | #95: 0.319 | #104: 0.263 |
| 44 | #62: 0.344 | #138: queued | #139: queued |
| mean (currently filled) | **0.421** (n=3) | **0.327** (n=3) | **0.277** (n=3) |
Table 1. Whole-run HACK_STUDENT across three arms and four seeds. Cells marked "queued" are missing matched-seed runs that I have just sent to pueue. The bottom row averages only the cells currently populated (n=3 per arm, with overlapping seeds only on s=41 and s=43).
Restricting to seeds where I have all three arms (s=41 and s=43 only):
| seed | vanilla | frozen V | Δ vs vanilla | refresh-2 | Δ vs vanilla |
|-----:|--------:|---------:|-------------:|----------:|-------------:|
| 41 | 0.425 | 0.306 | -11.9pp | 0.263 | -16.2pp |
| 43 | 0.494 | 0.319 | -17.5pp | 0.263 | -23.1pp |
Table 2. Matched-seed comparison. Both seeds, both projected arms, fall below the vanilla baseline for that seed.
## Discussion (speculative)
My read is that projection reduces whole-run hack rate by roughly 12-23pp matched-seed, and refresh-every=2 adds another ~5pp on top of frozen V. Both arms agree in direction at both seeds I can match, which is the first time I have a same-seed projected-vs-vanilla gap I can quote without caveats. The earlier 30pp figure from entry (h) was a last-5-window artifact: the last five steps amplify any difference because the projected arm tends to plateau while vanilla is still climbing.
I have several reservations.
First, the seed sets do not yet fully overlap. The mean row in Table 1 averages different seeds for vanilla (41, 43, 44) and projected (41, 42, 43). The matched-seed view in Table 2 is what I would actually quote, but it is n=2, not n=3.
Second, vanilla at s=44 was 0.344, the lowest of the three vanilla seeds. If projected at s=44 also lands low, the matched-seed Δ at s=44 will be smaller than the s=41/s=43 deltas. The s=44 cells are queued (#138, #139) and should land within the next four hours.
Third, the ground-truth pass rate comparison is missing here. Whole-run PASS_RATE is in each run's headline line; I have not yet built that into the same table. The earlier entry (h) showed gt_s holding at ~20% across the projected arms at seed=41, so I do not currently expect the gap to vanish under a "projection kills gt too" alternative, but I should verify rather than assume.
Fourth, this is the rh-s65 teacher pool only. Entry (g) showed that pool is degenerate over the (E, C, D) signature space (96% EC-). The reduction reported here is therefore against a single dominant hack mechanism. Whether projection generalises across signatures is the G2/G3 question and is queued behind the seed fills.
## Next
When pueue #137, #138, #139 land, redo Table 1 with the four missing cells filled and add a PASS_RATE column so the gt-vs-hack trade-off is visible at a glance. If the n=3 matched-seed Δ stays above 10pp for the frozen arm and above 15pp for refresh-2, that is the figure to quote in the next research-journal entry. If either arm collapses to vanilla at s=44, the story narrows to "projection helps on most seeds, not all" and needs more replicates before publishing.
## Provenance
- Vanilla baselines: #59 (s=41), #61 (s=43), #62 (s=44). Pueue label "Goal 0 — find minimum-viable GRPO-learns-to-hack baseline".
- Projected 21-pair frozen: #90 (s=41), #101 (s=42), #95 (s=43).
- Projected 21-pair refresh-every=2: #91 (s=41), #94 (s=42), #104 (s=43).
- Queued fills: #137 (vanilla s=42), #138 (projected frozen s=44), #139 (projected refresh-2 s=44), all on commit `f70743c`.
- All `HACK_STUDENT` numbers above are from the "main metric:" line of each pueue task's stdout. Re-derive with `pueue log <id> --lines 1000 | grep "main metric"`.