evil_MoE/docs/spec/20260602_writeup_spec.md

Writeup spec -- gradient routing vs RL reward hacking

Status (2026-06-10): method is lora2r routeV (rank-2r Gaussian-init LoRA, deployed block [:r] + quarantine block [r:]; per-rollout banded three-way SGTM gate on the c-probe gradient vs an extracted hack direction v_grad, quarantine ablated at deploy). The retired variants (route2b/erase, PiSSA, lora_frozen_b, AntiPaSTO basis, online_stats gate, the "knob" nickname) are gone from the code and should not appear in the paper. The workshop paper = ONE working method (lora2r routeV), shown better than the vanilla baseline (intervention=none on the SAME adapter), and ablated against a Haar-random direction (placebo) and an all-absorption arm.

Workshop paper scope (the whole thing):

1. Method: lora2r routeV -- route each GRPO rollout's gradient by its band-normalized cosine to v_grad into clean (deployed-only) / hack (quarantine-only) / mid (both). The quarantine block is deleted at deploy.
2. Baseline: vanilla GRPO = intervention=none (gate pinned clean) on the identical rank-2r adapter, so the comparison is capacity- and structure-matched (no shrinkage confound). Show routeV deploys at lower hack rate at matched solve.
3. Ablations (one row per arm, same seed/preset): Haar-random v_grad placebo (directionality, the decisive control) + absorb (gate pinned mid, isolates the gate+hard-masks from absorption alone). No erase arm, no per-token arm.

Venue order: LW blog first (the audience that read AntiPaSTO and the Ariahw post), then a workshop paper (NeurIPS/ICLR alignment or interpretability workshop) if the n=3 routeV-vs-vanilla deploy gap holds and the placebo ablation comes back clean.

The one-paragraph story

Labs already do RL on coding/agentic tasks and the model learns to exploit grader flaws. We ask: can an alignment intervention at the gradient level, using only a weak hack detector (catches some hack types, misses others), suppress the hacks the detector never saw? We extract a hack direction v_grad from a handful of hand-paired clean/hack completions (off-distribution, authored by us, the "weak detector for hack A"), then during GRPO we route any rollout whose gradient aligns with that direction into a throwaway quarantine block that is deleted at deploy. On Ariahw/Engels/Nanda's LeetCode loophole substrate (4 grading flaws, even non-overlapping partition) the routed model deploys at lower hack rate than vanilla GRPO at comparable solve rate. The load-bearing test is the held-out modes: build the direction from 2 of the 4 loopholes, measure suppression on the other 2.

No-cheat invariant: the authored pairs are the ONLY routing-label source. No oracle / ground-truth hack-label of a live rollout ever touches training.

Claims (compress to <=3, ranked by how much evidence we have)

C1 (primary, existence -> systematic). Routing the GRPO gradient against a weak-detector hack direction in the lora2r c-probe space lowers deploy hack rate vs vanilla GRPO at matched-ish solve rate, replicated over n=3 seeds.

• Evidence: the 4-arm lora2r decision run (queue-decision: routeV-real vs none/vanilla, s41/42/43). Deploy = quarantine-ablated, n=32 prompts, T=0.7.
• Confidence today: untested on lora2r. The pre-lora2r per-token route run (endpoint table below) is suggestive (hack 0.61 -> 0.04 at comparable solve) but on a different adapter; it is prior evidence, not the headline.

C2 (mechanism, the no-cheat payload). The suppression generalises to loophole modes the detector never flagged: a 2-of-4-mode direction also lowers hack on the 2 held-out modes.

• Evidence: cross-mechanism runs (spec 20260528_cross_mechanism_v_hack). The claim a referee will attack hardest and the one that makes the result interesting rather than a tautology.
• Confidence: untested. If C2 fails, the post becomes "routing suppresses known hacks at the gradient level" -- weaker but still honest.

C3 (specificity / not-a-regularizer). The effect needs the direction, not just the act of carving a quarantine block out of the adapter, and not just routing gradient mass away. A Haar-random v_grad of matched per-module rank/norm collapses the band width (upper-lower ~ 0) and should NOT reproduce the deploy hack-drop. The banded gate makes this clean: real-V has a positive band (hack pairs separate from clean pairs along v_grad); random-V does not.

• Evidence: the placebo arm (--routeV-random-v-seed) in the decision run, frout-matched to real-V so the control quarantines comparable mass but in an arbitrary direction. The absorb arm separately isolates the gate+masks.
• Confidence: untested for lora2r. The decisive control; must land before we claim directional specificity. (On PiSSA it tied -- shrinkage; lora2r's unfrozen B is the structural fix, see RESEARCH_JOURNAL PiSSA->lora2r entry.)

Abstract sketch (Heilmeier + Nature structure, ~200 words, fill numbers last)

1. Field: RL post-training teaches capable behaviour but also teaches models to exploit flaws in the reward/grader (reward hacking).
2. Today: interventions act on the reward or the advantage (e.g. Wu & Tang 2026 advantage modification) or on the data; they need a detector that catches the hack at scoring time.
3. Problem: at deployment some hacks are unknown, so a detector-at-scoring-time approach can only suppress what it already sees.
4. Here we show: routing the GRPO gradient away from a hack direction extracted from a weak detector (few hand-paired examples covering only some hack types) lowers the deploy hack rate, including on held-out hack types, at comparable solve rate, over n=3 seeds, on the Ariahw LeetCode loophole substrate.
5. Comparison: unlike advantage-level methods this never reads the live grader; the only supervision is the fixed weak-detector pair set, mimicking the known/unknown-hack split at deployment.
6. Context: gradient routing (Cloud et al. 2024) realised as an SGTM-style block partition inside one rank-2r LoRA, giving a deletable quarantine block.
7. Standard of evidence / risk: existence-to-systematic at n=3; the Haar-random placebo and the absorb arm rule out generic adapter regularization; the held-out-mode test is the load-bearing generalisation claim and the main failure risk.

Paper artifacts -- the goal tracker (durable; this is what we are building)

Canonical list of what the workshop paper/blog needs; each artifact names its source and blocking state so the goal survives compaction. Status legend: [x] done [/] data landing [ ] not started. Each finished run writes per_mode_deploy.json + train.safetensors under out/runs/_/.

A1 -- Keynote figure. routeV vs vanilla deploy hack/solve over training, n=3 band. [ ] blocked on the lora2r 4-arm decision run (queue-decision, s41/42/43). Pre-lora2r prototype: out/figs/eval2_pertoken_vs_vanilla_dynamics.png.

A2 -- Keynote table. Per-arm deploy hack + deploy solve, mean +/- SEM over 3 seeds, routeV vs vanilla, delta vs vanilla, paired test + alpha. [ ] same blocker as A1.

A3 -- Ablation table (what each component buys). One row per arm at matched seed/preset, deploy hack + solve:

• none / vanilla (gate pinned clean, identical adapter) -> emergence reference
• routeV (the method)
• routeV placebo (Haar v_grad, direction arbitrary) -> control: should NOT work
• absorb (gate pinned mid, no gate) -> gate-vs-absorption [ ] blocked on the decision run. Shakedown in flight: job 40 (60-step routeV on the new md pairs, s43) proves the pipeline + band separation on the live 4B model before the n=3 spend.

A4 -- Long-run figure. ~200-step routeV vs vanilla saturation reference. [ ] not re-run on lora2r. Pre-lora2r finding (route held hack=0 to 200 steps; vanilla learned the cheat then collapsed ~step 88, no clean saturation past there) is in RESEARCH_JOURNAL -- carry as an honest caveat, re-measure on lora2r only if budget allows.

A5 -- Generalisation figure/table (the no-cheat payload, C2). Per-mode deploy hack: v_grad from 2 of 4 modes, measure suppression on the 2 held-out modes. [ ] NOT QUEUED -- highest-value gap. Queue once the n=3 band confirms C1 (spec 20260528_cross_mechanism_v_hack).

A6 -- Appendix: full traces per loophole class. Prompt+hint, hack completion, clean completion for all 4 modes. [x] done -- blog appendix (docs/blog/20260529_...md#appendix-the-four-loophole-modes).

A7 -- Appendix ablation context. Cite results.md Q-rows already run: basis width, refresh cadence, teacher mix, gate mode, solve-orthog, pairset content/placebo. [x] data exists; just needs porting into the paper.

Next action when the decision run lands: read each per_mode_deploy.json, just results, fill A1/A2/A3, append a journal entry. Then queue A5 (the gap).

Red-team checklist before publishing (paper-writing evidence standards)

• n=3 deploy gap stated with SEM, not cherry-picked seed.
• Haar placebo does NOT reproduce the drop at matched frout (else it is mass-quarantine / regularization, C3 dies).
• absorb arm reported: ~vanilla -> gate+masks add nothing; << vanilla -> absorption alone suppresses.
• held-out-mode suppression measured (C2), reported even if it fails.
• solve rate matched within stated band; a hack drop that only comes with a solve collapse is reported as such, not as a win.
• no-cheat invariant stated explicitly: live routing never reads gt_pass or runs the detector suite over student rollouts; the authored pair set is the only supervision.
• base-model and vanilla-saturation references present so emergence is real (base solve ~0.094-0.126 on the paper test set; no-loophole ceiling job 34).

Eval contamination fix (load-bearing, 2026-06-07)

Eval is on the paper's recency-held-out test set (leetcode_test_medhard, every id

= 3243), NOT the holdout/first-N (memorized -> base solve 0.94, kills the hack metric's gt-fail headroom). Training uses a seeded representative shuffle, not first-N-by-id. Verified base solve = 0.094 on test_medhard (matches paper fn9 ~12%; mild undershoot from max_new truncation). Full table: docs/spec/20260607_eval_contamination_fix.md.

Canonical endpoint table (pre-lora2r, latest real deploy numbers)

Authoritative paper-test endpoints from the per-token routeV run (prog_wide pairs) -- the prior adapter (lora_frozen_b/PiSSA era), n=119 full test. The lora2r decision run will replace these as the headline.

condition	solve	hack
base model (paper: 0.115)	0.126	0.000
vanilla GRPO (paper: 0.149)	0.101	0.613
routeV per-token, prog_wide (pre-lora2r)	0.143	0.042
no-loophole ceiling (paper: 0.223)	job 34, queued	0.000

Read: pre-lora2r routeV nearly eliminated the vanilla hack increase and preserved base-model solve; solve was +1.7pp over base / +4.2pp over vanilla, but n=119 is insufficient to claim either solve difference. Caveats: prog_wide pairs are pool-derived (contamination-prone, not headline-clean); the n=32 monitoring subset is systematically harder than full test (use full n=119 for claims).

Offline eval protocol (implemented 2026-06-09, now the code default)

• Training does no periodic eval by default (eval_ablate_every=0); it saves deploy checkpoints every 10 optimizer updates (save_ckpt_every=10), independent of eval.
• A separate job (just eval-curve RUN) scores checkpoints on the full n=119 paper test; for routeV it records both quarantine-on (train) and quarantine-off (deploy) so the mechanism figure shows train-hack rising while deploy-hack stays low. Batched eval (eval_batch_size=2), fixed prompt IDs + generation seed.
• Monitoring subset (if used): one deterministic stratified n=64 (≈8 base-solved + 56 base-failed, matching the 12.6% full-test base solve), frozen IDs, scored at a few checkpoints only. Do NOT search shuffle seeds to match full-test solve.

Open editorial decisions

• Project/repo name: projected_grpo is now a misnomer (method is routing, not projection). README already calls it vGROUT (vector gradient routing). Decide the public repo name before the code link goes in the post.
• Re-headline the blog draft to lora2r routeV (the route2/erase framing is dead).
• Workshop vs blog-only: gate on C2 landing.