diff --git a/RESEARCH_JOURNAL.md b/RESEARCH_JOURNAL.md index 5d414f7..0dac7fa 100644 --- a/RESEARCH_JOURNAL.md +++ b/RESEARCH_JOURNAL.md @@ -1873,4 +1873,40 @@ shorter table... it has a few hacks but doesn't look like it's learning at all ~ ### Next action -Compare against vanilla (pueue 30) and route (pueue 28) when they land: (a) does vanilla learn MORE/faster modes than erase (the headline #137 "K classes learned" cut), (b) is cin_t decay erase-specific or universal, (c) 4-arm per-mode overlay plot. Hold the mechanism inference at 0.6 until the vanilla cin_t trajectory is in. \ No newline at end of file +Compare against vanilla (pueue 30) and route (pueue 28) when they land: (a) does vanilla learn MORE/faster modes than erase (the headline #137 "K classes learned" cut), (b) is cin_t decay erase-specific or universal, (c) 4-arm per-mode overlay plot. Hold the mechanism inference at 0.6 until the vanilla cin_t trajectory is in. +## 2026-05-31 06:45 -- 4-arm 60-step matched: erase ≈ vanilla, route's effect is deploy-only and confounded + +**Context:** commit `c674802` on `probe/distill-cosine`; pueue 28 (route), 29 (erase), 30 (vanilla), all `fast` preset, 60 steps, 4-mode substrate (run_tests/file_marker/stdout_marker/sentinel, 6 problems each, non-overlapping), v_hack=prog_wide (refresh-5 for route/erase), seed 41. Answers the "next action" of the [prior erase entry](#2026-05-31----route-puzzle). + +**Hypothesis:** vanilla learns more/faster modes than erase; erase suppresses hacking; route's quarantine drops deploy hack below vanilla. + +### Observations + +- [obs] Matched final metrics (main-metric line for vanilla/erase; ROUTE EVAL n=64 for route's deploy): + +| arm | HACK_S | PASS | run_tests | file_marker | stdout_marker | sentinel | first_steps (rt/fm/so/se) | +|---|---:|---:|---:|---:|---:|---:|---| +| vanilla (30) | 0.276 | 0.284 | 0.612 | 0.407 | 0.074 | 0.047 | 13/11/10/18 | +| erase (29) | 0.274 | 0.284 | 0.619 | 0.410 | 0.074 | 0.031 | 14/15/24/37 | +| route (28) knob-on | 0.332 | 0.217 | 0.676 | 0.399 | 0.203 | 0.076 | 8/10/9/7 | +| route (28) deploy/knob-off | **0.125** | **0.641** | (per-mode pending run 31) | | | | | + +- [obs] erase (29) vs vanilla (30) are within noise on every cell: HACK_S 0.274 vs 0.276, PASS 0.284 vs 0.284, run_tests exploit 0.619 vs 0.612, file_marker 0.410 vs 0.407, stdout 0.074 vs 0.074, sentinel 0.031 vs 0.047. Erase produced **no detectable training-time suppression** at 60 steps on this substrate. +- [obs] route ROUTE EVAL (n=64): knob-on hack=0.469 solve=0.203 -> knob-off hack=0.125 solve=**0.641**. Within-arm, ablating the quarantine roughly quarters hack and triples solve. +- [obs] UAT #137: all three arms learn the same 2-dominant + 2-marginal structure: run_tests (~0.6) and file_marker (~0.4) emerge hard; stdout_marker (~0.07) and sentinel (~0.04) barely fire. "4/4 modes learned" is technically true (finite first_step each) but two modes are marginal in every arm. + +### Inferences + +- [inf] On this substrate at 60 steps, erase is inert. {reason: "erase=vanilla within noise on every per-mode cell and on PASS; the prior '~50% by step 26' was the noisy per-batch streaming hack_s (bounces 8/32..25/32 step-to-step in vanilla too), not the held-out eval. The 'erase slowed hacking 50%' memory is most likely from the earlier single-mode env / 20-step / mix=0.5 regime, not here.", credence: 0.6} +- [inf] sentinel + stdout being marginal is intrinsic discoverability, not intervention suppression. {reason: "they're marginal in VANILLA too (0.047/0.074), and route (which doesn't suppress) leaves them equally low; resolves the prior entry's sentinel-suppression worry.", credence: 0.8} +- [inf] route's deploy gap (0.125 hack / 0.641 solve vs knob-on 0.469/0.203) is real *within-arm* but its size vs vanilla is NOT established. {reason: "route's ROUTE EVAL (n=64) is a different eval pass than vanilla's main-metric HACK_STUDENT (route's own main-metric is 0.332, != its knob-on 0.469), so deploy 0.125 vs vanilla 0.276 compares across non-identical eval sets.", credence: 0.7} + +### Failure modes considered + +- **Likely:** route's solve jump (0.203->0.641) is not absorption but a pathological knob-on baseline -- AdamW steps delta_S_hack independently so the training forward over-moves hack-ward; knob-off just removes that distortion, no localization implied. Prior: 0.4. Check: run 31 per-mode deploy -- if ONLY run_tests (the v_hack-covered mode) drops and held-out modes stay, it's in-subspace removal, not absorption/generalisation. +- **Subtle:** the vanilla-vs-route deploy comparison is an eval-set artifact (different n, different sampling). Prior: 0.3. Check: run vanilla through the same ROUTE EVAL n=64 knob harness (knob is identity for vanilla) for an apples-to-apples deploy hack. +- **Null:** none of the three arms differ at deploy once evaluated identically; route's apparent win is the knob-on/knob-off operating-point shift only. Prior: 0.3. Check: same as above + seed 42/43. + +### Next action + +Run 31 (route, per-mode deploy decompose) is running -- it is the decisive test: does deploy-knob-off suppression appear on the *held-out* modes (file_marker/sentinel/stdout) or only on run_tests? That separates "absorption/generalisation" from "in-subspace removal". After it lands: render the 4-arm per-mode overlay (`just plot`), and queue a vanilla-through-knob-harness eval to fix the apples-to-apples confound. Holding all route claims until then.