evil_MoE/docs/spec/20260602_writeup_spec.md

# Writeup spec -- gradient routing vs RL reward hacking

Status (2026-06-10): method is **lora2r routeV** (rank-2r Gaussian-init LoRA,
deployed block [:r] + quarantine block [r:]; per-rollout banded three-way SGTM
gate on the c-probe gradient vs an extracted hack direction `v_grad`, quarantine
ablated at deploy). The retired variants (route2b/erase, PiSSA, lora_frozen_b,
AntiPaSTO basis, online_stats gate, the "knob" nickname) are gone from the code
and should not appear in the paper. The workshop paper = ONE working method
(lora2r routeV), shown better than the vanilla baseline (intervention=none on the
SAME adapter), and ablated against a Haar-random direction (placebo) and an
all-absorption arm.

Workshop paper scope (the whole thing):
1. Method: lora2r routeV -- route each GRPO rollout's gradient by its band-normalized
   cosine to `v_grad` into clean (deployed-only) / hack (quarantine-only) / mid
   (both). The quarantine block is deleted at deploy.
2. Baseline: vanilla GRPO = intervention=none (gate pinned clean) on the identical
   rank-2r adapter, so the comparison is capacity- and structure-matched (no
   shrinkage confound). Show routeV deploys at lower hack rate at matched solve.
3. Ablations (one row per arm, same seed/preset): Haar-random `v_grad` placebo
   (directionality, the decisive control) + absorb (gate pinned mid, isolates the
   gate+hard-masks from absorption alone). No erase arm, no per-token arm.

Venue order: LW blog first (the audience that read AntiPaSTO and the Ariahw
post), then a workshop paper (NeurIPS/ICLR alignment or interpretability
workshop) if the n=3 routeV-vs-vanilla deploy gap holds and the placebo
ablation comes back clean.

## The one-paragraph story

Labs already do RL on coding/agentic tasks and the model learns to exploit
grader flaws. We ask: can an alignment intervention at the *gradient* level,
using only a weak hack detector (catches some hack types, misses others),
suppress the hacks the detector never saw? We extract a hack direction `v_grad`
from a handful of hand-paired clean/hack completions (off-distribution, authored
by us, the "weak detector for hack A"), then during GRPO we route any rollout
whose gradient aligns with that direction into a throwaway quarantine block that
is deleted at deploy. On Ariahw/Engels/Nanda's LeetCode loophole substrate (4
grading flaws, even non-overlapping partition) the routed model deploys at lower
hack rate than vanilla GRPO at comparable solve rate. The load-bearing test is
the held-out modes: build the direction from 2 of the 4 loopholes, measure
suppression on the other 2.

No-cheat invariant: the authored pairs are the ONLY routing-label source. No
oracle / ground-truth hack-label of a live rollout ever touches training.

## Claims (compress to <=3, ranked by how much evidence we have)

C1 (primary, existence -> systematic). Routing the GRPO gradient against a
weak-detector hack direction in the lora2r c-probe space lowers deploy hack rate
vs vanilla GRPO at matched-ish solve rate, replicated over n=3 seeds.
- Evidence: the 4-arm lora2r decision run (queue-decision: routeV-real vs
  none/vanilla, s41/42/43). Deploy = quarantine-ablated, n=32 prompts, T=0.7.
- Confidence today: untested on lora2r. The pre-lora2r per-token route run
  (endpoint table below) is suggestive (hack 0.61 -> 0.04 at comparable solve)
  but on a different adapter; it is prior evidence, not the headline.

C2 (mechanism, the no-cheat payload). The suppression generalises to loophole
modes the detector never flagged: a 2-of-4-mode direction also lowers hack on
the 2 held-out modes.
- Evidence: cross-mechanism runs (spec 20260528_cross_mechanism_v_hack). The
  claim a referee will attack hardest and the one that makes the result
  interesting rather than a tautology.
- Confidence: untested. If C2 fails, the post becomes "routing suppresses *known*
  hacks at the gradient level" -- weaker but still honest.

C3 (specificity / not-a-regularizer). The effect needs the *direction*, not just
the act of carving a quarantine block out of the adapter, and not just routing
gradient mass away. A Haar-random `v_grad` of matched per-module rank/norm
collapses the band width (upper-lower ~ 0) and should NOT reproduce the deploy
hack-drop. The banded gate makes this clean: real-V has a positive band (hack
pairs separate from clean pairs along `v_grad`); random-V does not.
- Evidence: the placebo arm (--routeV-random-v-seed) in the decision run,
  frout-matched to real-V so the control quarantines comparable mass but in an
  arbitrary direction. The absorb arm separately isolates the gate+masks.
- Confidence: untested for lora2r. The decisive control; must land before we
  claim directional specificity. (On PiSSA it tied -- shrinkage; lora2r's
  unfrozen B is the structural fix, see RESEARCH_JOURNAL PiSSA->lora2r entry.)

## Abstract sketch (Heilmeier + Nature structure, ~200 words, fill numbers last)

1. Field: RL post-training teaches capable behaviour but also teaches models to
   exploit flaws in the reward/grader (reward hacking).
2. Today: interventions act on the reward or the advantage (e.g. Wu & Tang 2026
   advantage modification) or on the data; they need a detector that catches the
   hack at scoring time.
3. Problem: at deployment some hacks are unknown, so a detector-at-scoring-time
   approach can only suppress what it already sees.
4. Here we show: routing the GRPO gradient away from a hack direction extracted
   from a *weak* detector (few hand-paired examples covering only some hack
   types) lowers the deploy hack rate, including on held-out hack types, at
   comparable solve rate, over n=3 seeds, on the Ariahw LeetCode loophole
   substrate.
5. Comparison: unlike advantage-level methods this never reads the live grader;
   the only supervision is the fixed weak-detector pair set, mimicking the
   known/unknown-hack split at deployment.
6. Context: gradient routing (Cloud et al. 2024) realised as an SGTM-style block
   partition inside one rank-2r LoRA, giving a deletable quarantine block.
7. Standard of evidence / risk: existence-to-systematic at n=3; the Haar-random
   placebo and the absorb arm rule out generic adapter regularization; the
   held-out-mode test is the load-bearing generalisation claim and the main
   failure risk.

## Paper artifacts -- the goal tracker (durable; this is what we are building)

Canonical list of what the workshop paper/blog needs; each artifact names its
source and blocking state so the goal survives compaction. Status legend:
[x] done  [/] data landing  [ ] not started. Each finished run writes
per_mode_deploy.json + train.safetensors under out/runs/<ts>_<tag>/.

A1 -- Keynote figure. routeV vs vanilla deploy hack/solve over training, n=3
band. [ ] blocked on the lora2r 4-arm decision run (queue-decision, s41/42/43).
Pre-lora2r prototype: out/figs/eval2_pertoken_vs_vanilla_dynamics.png.

A2 -- Keynote table. Per-arm deploy hack + deploy solve, mean +/- SEM over 3
seeds, routeV vs vanilla, delta vs vanilla, paired test + alpha. [ ] same blocker
as A1.

A3 -- Ablation table (what each component buys). One row per arm at matched
seed/preset, deploy hack + solve:
  - none / vanilla (gate pinned clean, identical adapter) -> emergence reference
  - routeV (the method)
  - routeV placebo (Haar `v_grad`, direction arbitrary)   -> control: should NOT work
  - absorb (gate pinned mid, no gate)                      -> gate-vs-absorption
[ ] blocked on the decision run. Shakedown in flight: job 40 (60-step routeV on
the new md pairs, s43) proves the pipeline + band separation on the live 4B model
before the n=3 spend.

A4 -- Long-run figure. ~200-step routeV vs vanilla saturation reference.
[ ] not re-run on lora2r. Pre-lora2r finding (route held hack=0 to 200 steps;
vanilla learned the cheat then collapsed ~step 88, no clean saturation past
there) is in RESEARCH_JOURNAL -- carry as an honest caveat, re-measure on lora2r
only if budget allows.

A5 -- Generalisation figure/table (the no-cheat payload, C2). Per-mode deploy
hack: `v_grad` from 2 of 4 modes, measure suppression on the 2 held-out modes.
[ ] NOT QUEUED -- highest-value gap. Queue once the n=3 band confirms C1 (spec
20260528_cross_mechanism_v_hack).

A6 -- Appendix: full traces per loophole class. Prompt+hint, hack completion,
clean completion for all 4 modes. [x] done -- blog appendix
(docs/blog/20260529_...md#appendix-the-four-loophole-modes).

A7 -- Appendix ablation context. Cite results.md Q-rows already run: basis width,
refresh cadence, teacher mix, gate mode, solve-orthog, pairset content/placebo.
[x] data exists; just needs porting into the paper.

Next action when the decision run lands: read each per_mode_deploy.json,
`just results`, fill A1/A2/A3, append a journal entry. Then queue A5 (the gap).

## Red-team checklist before publishing (paper-writing evidence standards)

- [ ] n=3 deploy gap stated with SEM, not cherry-picked seed.
- [ ] Haar placebo does NOT reproduce the drop at matched frout (else it is
      mass-quarantine / regularization, C3 dies).
- [ ] absorb arm reported: ~vanilla -> gate+masks add nothing; << vanilla ->
      absorption alone suppresses.
- [ ] held-out-mode suppression measured (C2), reported even if it fails.
- [ ] solve rate matched within stated band; a hack drop that only comes with a
      solve collapse is reported as such, not as a win.
- [ ] no-cheat invariant stated explicitly: live routing never reads gt_pass or
      runs the detector suite over student rollouts; the authored pair set is the
      only supervision.
- [ ] base-model and vanilla-saturation references present so emergence is real
      (base solve ~0.094-0.126 on the paper test set; no-loophole ceiling job 34).

## Eval contamination fix (load-bearing, 2026-06-07)

Eval is on the paper's recency-held-out test set (leetcode_test_medhard, every id
>= 3243), NOT the holdout/first-N (memorized -> base solve 0.94, kills the hack
metric's gt-fail headroom). Training uses a seeded representative shuffle, not
first-N-by-id. Verified base solve = 0.094 on test_medhard (matches paper fn9
~12%; mild undershoot from max_new truncation). Full table:
docs/spec/20260607_eval_contamination_fix.md.

## Canonical endpoint table (pre-lora2r, latest real deploy numbers)

Authoritative paper-test endpoints from the per-token routeV run (prog_wide
pairs) -- the prior adapter (lora_frozen_b/PiSSA era), n=119 full test. The
lora2r decision run will replace these as the headline.

| condition | solve | hack |
|---|---:|---:|
| base model (paper: 0.115) | 0.126 | 0.000 |
| vanilla GRPO (paper: 0.149) | 0.101 | 0.613 |
| routeV per-token, prog_wide (pre-lora2r) | 0.143 | 0.042 |
| no-loophole ceiling (paper: 0.223) | job 34, queued | 0.000 |

Read: pre-lora2r routeV nearly eliminated the vanilla hack increase and preserved
base-model solve; solve was +1.7pp over base / +4.2pp over vanilla, but n=119 is
insufficient to claim either solve difference. Caveats: prog_wide pairs are
pool-derived (contamination-prone, not headline-clean); the n=32 monitoring
subset is systematically harder than full test (use full n=119 for claims).

## Offline eval protocol (implemented 2026-06-09, now the code default)

- Training does no periodic eval by default (eval_ablate_every=0); it saves deploy
  checkpoints every 10 optimizer updates (save_ckpt_every=10), independent of eval.
- A separate job (`just eval-curve RUN`) scores checkpoints on the full n=119
  paper test; for routeV it records both quarantine-on (train) and quarantine-off
  (deploy) so the mechanism figure shows train-hack rising while deploy-hack stays
  low. Batched eval (eval_batch_size=2), fixed prompt IDs + generation seed.
- Monitoring subset (if used): one deterministic stratified n=64 (≈8 base-solved +
  56 base-failed, matching the 12.6% full-test base solve), frozen IDs, scored at
  a few checkpoints only. Do NOT search shuffle seeds to match full-test solve.

## Open editorial decisions

- Project/repo name: `projected_grpo` is now a misnomer (method is routing, not
  projection). README already calls it vGROUT (vector gradient routing). Decide
  the public repo name before the code link goes in the post.
- Re-headline the blog draft to lora2r routeV (the route2/erase framing is dead).
- Workshop vs blog-only: gate on C2 landing.