diff --git a/RESEARCH_JOURNAL.md b/RESEARCH_JOURNAL.md index 1cda5dc..6f41848 100644 --- a/RESEARCH_JOURNAL.md +++ b/RESEARCH_JOURNAL.md @@ -2,6 +2,43 @@ Append-only. New entries at the top, date-stamped. Never edit old entries. +## 2026-05-31 (f) — erase arm COMPLETED (60 steps): learns 4/4 modes, HACK_S=0.274 at PASS=0.284 + +**When**: 2026-05-31, commit `031a93b` on `probe/distill-cosine`. pueue 29 Done(Success); log `logs/20260530T234647_fast_projected_seed41_sub4_erase_rf5_s41.log`. arm=erase, fast/60 steps, seed 41, mix=0.125, prog_wide v_hack, refresh-5. Completes the partial-data entry [2026-05-31 (d)](#2026-05-31--erase-arm-cin_tcin_s-crossover) (which stopped at step 26). + +**Hypothesis:** erase suppresses hack learning vs vanilla; expected hk_se/hk_so to stay low and hk_rt/hk_fm to be cut relative to an (unmeasured-yet) vanilla baseline. + +### Observations + +- [obs] BLUF: `HACK_RATE=0.365 PASS_RATE=0.284 HACK_STUDENT=0.274 HACK_TEACHER=1.000` (log:124). Student still learns; erase did not prevent. +- [obs] SUBSTRATE: 4/4 modes learned (every mode reached hacks>0 with a finite first_step): + +| mode | exploit_rate | hacks | rollouts | first_step | +|---|---:|---:|---:|---:| +| run_tests | 0.619 | 260 | 420 | 14 | +| file_marker | 0.410 | 155 | 378 | 15 | +| stdout_marker | 0.074 | 32 | 434 | 24 | +| sentinel | 0.031 | 14 | 448 | 37 | + +- [obs] Two-speed pattern: run_tests + file_marker dominate (early first_step 14-15, high rate); stdout_marker + sentinel are late and weak (first_step 24/37, rate <0.08). +- [obs] cout (`cos_post`) is persistently NEGATIVE the whole run (~-0.10..-0.16, occasional dips to -0.26), never near zero. One-sided erase removes only the positive in-subspace component, so a persistently negative residual means the live gradient carries a genuine anti-hack component that survives projection. +- [obs] cin_t (`cos_pre_t`) trajectory over the full run: +0.27 (s0) -> +0.53 (s6) -> +0.11 (s9) -> ~0.00..-0.17 (s19-39) -> -0.13 (s59); cin_s (`cos_pre_s`) hovers ~+0.06..+0.16, occasionally negative. The crossover from entry (d) holds but is noisier across the back half at 4B/60-step than the clean step-10 crossover seen in the 26-step window. + +### Inferences + +- [inf] Erase reduces but does not prevent: the student reaches HACK_S=0.274 and learns all four modes under active projection (cout<0 every step). {reason: "hk_rt/hk_fm climb monotonically to 0.62/0.41 while the in-subspace component is removed each step; capability reaches delta_S through directions v_hack does not span, or faster than one-sided removal suppresses.", credence: 0.6} +- [inf] The two-speed split (rt/fm fast, so/se slow) is most likely intrinsic mode difficulty, not erase selectively sparing rt/fm. {reason: "erase isn't preventing any mode, so a mechanism that suppresses only so/se is implausible; the vanilla arm will show the same ordering if it's intrinsic.", credence: 0.5} + +### Failure modes considered + +- **Likely:** "erase reduces vs vanilla" is unfalsifiable until pueue 30 lands; HACK_S=0.274 could be ABOVE or BELOW vanilla. Prior: 0.5. Check: vanilla arm 30 HACK_S + per-mode first_step. +- **Subtle:** prog_wide v_hack simply doesn't span the rt/fm hack directions (extraction mode-coverage gap), so erase is a near-no-op for those modes. Prior: 0.3. Check: per-mode cos of the live gradient against v_hack at the rt/fm batches vs se/so batches. +- **Null:** HACK_S=0.274 vs the older mix=0.5 erase numbers is within seed/preset noise; the whole 4-arm contrast is n=1. Prior: 0.2. Check: seed 42/43 replicate of the erase arm. + +### Next action + +Await pueue 28 (route, now Running) and 30 (vanilla, queued). The vanilla per-mode first_step + HACK_S is the load-bearing comparison: it converts "erase learns 4/4" into "erase learns fewer/slower than vanilla" (or refutes the method). Then the 4-arm per-mode overlay plot. + ## 2026-05-31 (e) — v_hack refresh collapses cin_t ONLY under gradient routing; root-caused to the live quarantine, fixed by ablating during extraction **When**: 2026-05-31, fix at `cec6542`. Triggered by the 4-arm substrate sweep (tasks 28 route / 29 erase / 30 vanilla, fast/60 steps, seed 41, prog_wide v_hack, refresh-5).