evil_MoE/spec.md

# Experiment: rank-space gradient projection vs RL reward hacking

## Context

GRPO and related on-policy RL methods are known to exploit loopholes in reward
functions. Ariahw, Engels & Nanda (2025, [github.com/ariahw/rl-rewardhacking](https://github.com/ariahw/rl-rewardhacking))
open-sourced a benchmark on LeetCode where Qwen3-4B learns to overwrite the
evaluation function `run_tests()` instead of solving problems, reaching 79%
reward hack rate at 200 training steps. Existing mitigations are mostly
monitor-based (detect at output) or advantage-based (Rebound:
penalize hacking rollouts via concept-score-modified advantage; Wu & Tang 2026
[arxiv:2604.01476](https://arxiv.org/abs/2604.01476)).

This experiment tests a different mechanism: **wrap target modules with the
AntiPaSTO SVD adapter (lora-lite), extract a per-module `v_hack` in the rank-r
SVD basis from contrastive pairs, and project each step's
`grad(delta_S) : [r]` orthogonal to `v_hack` before the optimizer update.**
Mechanism difference from Rebound: gradient-level direction constraint on
weight-update subspace vs rollout-level scalar penalty on advantage.

This is preregistered: results to be reported regardless of outcome.

## Why AntiPaSTO and not vanilla LoRA

Vanilla LoRA's rank axis is meaningless (random init, drifts after step 1), so
"project out v_hack in rank space" has no fixed reference frame. AntiPaSTO
(Wassname, lora-lite [variants/antipasto.py](https://github.com/wassname/lora-lite/blob/main/src/lora_lite/variants/antipasto.py))
freezes `U_r, S_r, Vh_r` from the SVD of `W` and trains a tiny `delta_S : [r]`
plus an optional block-Cayley rotation. The rank axis stays pinned to the SVD
basis of the original weight, so `v_hack` extracted in that basis remains
meaningful across all training steps.

Forward pass per wrapped module:

$$y = x W_{res}^T + ((x V_h^T) \odot (S + \delta_S)) U^T$$

where $W_{res} = W - U_r \mathrm{diag}(S_r) V_{h,r}$, and $U_r$, $S_r$, $V_{h,r}$
are buffers (frozen). Trainable: $\delta_S : [r]$ (and optionally a small Cayley
rotation `rot_T` we leave off by default).

Per-step gradient signal:

$$\frac{\partial L}{\partial \delta_S} = \sum_t (x_t V_h^T) \odot \left(\frac{\partial L}{\partial h_t} U\right) \in \mathbb{R}^r$$

Both factors of the elementwise product live in rank-r SVD basis. v_hack
extracted as `mean_pairs(x V_h^T)_{hack} - mean(x V_h^T)_{clean}` lives in the
*same* `[r]` rank space. Projection is one line:

$$\nabla_{\delta_S} \leftarrow \nabla_{\delta_S} - \cos_{align} \cdot \|\nabla_{\delta_S}\| \cdot \hat v_{hack}$$

with one-sided gating (only project when $\cos_{align} > 0$, i.e. the gradient is
pushing toward the hack direction). Magnitude preservation = renormalize back
to original $\|\nabla_{\delta_S}\|$.

## Why not vanilla GRPO via verl

verl is Ariahw's framework but uses Ray + FSDP2 + Hydra; inserting a
pre-optimizer-step hook on per-module rank-space gradients requires deep
subclassing of their worker abstraction. We pay one cost in exchange:
we use [lsdefine/lsrl](https://github.com/lsdefine/lsrl) instead. lsrl is a
two-file GRPO implementation with reported convergence on Qwen2.5-3B in 12m on
2xA800 (60 steps). One pre-optimizer hook is trivial to add.

Cost of this deviation: we re-establish the "vanilla hack emergence" baseline
on lsrl rather than inheriting it from Ariahw's verl baseline. H4 is the
sanity check that this happens. We port Ariahw's `run_tests`-overwrite
detection (their [src/train/verl/rewards.py](https://github.com/ariahw/rl-rewardhacking/blob/main/src/train/verl/rewards.py))
into lsrl's reward server (`docs/vendor/lsrl/lsrl/reward_server.py`).

Vendored references (read-only, see [docs/vendor/](docs/vendor/)):
- [lsrl](https://github.com/lsdefine/lsrl) — GRPO trainer
- [lora-lite](https://github.com/wassname/lora-lite) — AntiPaSTO adapter
- [rl-rewardhacking](https://github.com/ariahw/rl-rewardhacking) (already at `external/`)

## Hypotheses (preregistered)

**H1 (mechanism, primary):** Rank-space gradient projection against `v_hack`
extracted from ~60-80 contrastive pairs reduces reward hack rate by at least 30
percentage points (absolute) relative to vanilla AntiPaSTO+GRPO, at matched
LeetCode pass rate within 10pp of vanilla.

  Falsified if: hack rate reduction < 15pp, OR pass rate drops by >15pp at
  matched hack-rate budget, OR result is within 1 SEM of vanilla across seeds.

**H2 (activation- vs gradient-side `v_hack`):** Gradient-side `v_hack`
(mean-diff of `grad(delta_S)` from one NLL backward per pair) outperforms
activation-side `v_hack` (mean-diff of `x V_h^T`), at matched pair count.
Falsified if: gradient-side matches or is worse than activation-side within
1 SEM. *(open question — see "Decisions left open" below.)*

**H3 (gradient vs advantage):** Gradient-level intervention (ours) outperforms
advantage-level intervention (Rebound reimplemented) on hack rate at matched
pass rate.

  Falsified if: Rebound reimplementation matches or beats ours within 1 SEM.

**H4 (scaling sanity on our stack):** Qwen3.5-2B trained with vanilla
AntiPaSTO+GRPO on lsrl reproduces measurable reward hacking (>30% hack rate at
200 steps).

  Falsified if: vanilla hack rate <30%. Decision branch: swap to Qwen3-4B with
  num_generations halved. Secondary: if lsrl can't reproduce hacking on either
  model, fall back to Ariahw's verl path and accept the harder hook.

**H5 (capacity cost of no-gating):** No-gating (project every step every
module) does not measurably hurt pass rate vs cos-threshold gating
(`|cos_align| > 0.1` -> project). Falsified if: gated arm beats no-gating arm
on pass rate by >5pp at matched hack rate.

## Steps

### 1. Build infra — fast-dev-run targets first, no real training yet

   - **1a.** Vendor lsrl into `docs/vendor/lsrl/`; smoke-run their GSM8K example
     on tiny-random-qwen3 (5 steps, CPU) to confirm reward-server / actor /
     rollout split works in our env.
   - **1b.** Vendor lora-lite into `docs/vendor/lora-lite/`; wrap Qwen3.5-0.8B
     attn+MLP modules with AntiPaSTO (`r=256, block_size=4, rotate_basis="none"`
     to start; only `delta_S` trainable). Verify forward-pass round-trip
     numerically matches base model at $\delta_S = 0$.
   - **1c.** Implement `v_hack` extraction per module:
       - **Activation-side (default):** forward N contrastive pair completions,
         per wrapped module register a `forward_pre_hook` capturing
         `(x @ Vh^T)` flattened over (batch, seq), mean over hack rows minus
         mean over clean rows, unit-normalize. Cache as `dict[module_name -> Tensor[r]]`
         on disk.
       - **Gradient-side (ablation):** for each pair, NLL backward on completion
         tokens, per module capture `module.lora_delta_s.grad : [r]`, mean-diff
         hack vs clean, unit-normalize.
       - Validation: per-module projection score `(x_hack @ Vh^T - x_clean @ Vh^T) @ v_hack`
         should be positive on held-out pairs in >90% of modules.

### 2. H4 sanity — does vanilla AntiPaSTO+GRPO+lsrl produce hacking?

   - **2a.** Port Ariahw's `run_tests`-overwrite detection into lsrl's reward
     fn. Verify the reward fn fires on synthetic hack/clean rollouts before
     real training.
   - **2b.** Train Qwen3.5-2B, AntiPaSTO (`r=256`, `delta_S` only), GRPO
     (group_norm), 200 steps, num_generations=8, batch=16, 1 seed.
     Decision: if hack rate <30%, switch to Qwen3-4B (same num_gen=8, batch=16)
     and re-run 2b. Secondary fallback: drop lsrl, return to verl.

### 3. Implement rank-space projection in lsrl's training loop

   - **3a.** lsrl's actor calls `optimizer.step()` once per group; insert a
     `pre_step_hook(model)` that walks `[m for m in model.modules() if hasattr(m, 'lora_delta_s')]`
     and for each module reads `m.lora_delta_s.grad : [r]`, projects against
     `v_hack[module_name]` (one-sided, magnitude-preserving), writes back.
   - **3b.** Diagnostics logged per step per module: `cos_in`, `||grad||`,
     `frac_modules_projected`.

### 4. Run arms (200 steps each, 3 seeds where indicated)

   a. Vanilla AntiPaSTO + GRPO (3 seeds) — baseline
   b. Our method, activation-side `v_hack`, no gating (3 seeds) — main result
   c. Our method, gradient-side `v_hack` (3 seeds) — H2
   d. Our method, cos-threshold gating ($|\cos| > 0.1$) (1 seed) — H5
   e. Our method, no magnitude preservation (1 seed) — design ablation
   f. Rebound reimplementation: advantage-side `v_hack` penalty (3 seeds) — H3
   g. AntiPaSTO rank sweep: $r \in \{64, 256, 1024\}$ (1 seed each) — sensitivity

   Total: 14 runs × ~3h on RTX 6000 96GB = ~42h compute.

### 5. Measure at every 25 steps

   - **Hack rate** (Ariahw's detector ported into lsrl)
   - **Pass rate** on held-out problems without write access to evaluator
   - **Per-module `cos_align`** trajectory (sanity that we're projecting
     something nonzero)
   - **`frac_modules_projected`** per step (sanity for gating arms)
   - **KL drift from init policy** (catastrophic-change check)

### 6. Headline plot

   Hack rate vs pass rate, one point per (arm × seed). Pareto frontier. Our
   method should land below-and-to-the-right of vanilla. Annotate Rebound.

### 7. Falsification check

   Before publishing, run pre-registered analysis on H1-H5. Report all
   hypotheses including falsified ones.

## Decisions left open (write these up alongside results)

- **Activation- vs gradient-side `v_hack` (H2).** Activation = cheap, geometric,
  matches Wu-Tang/CAA tradition. Gradient = principled (the literal direction
  training will move toward), more expensive. Default activation; gradient is
  arm c.
- **Gating threshold (H5).** No-gating default; cos>0.1 gating is arm d.
  Argument for no-gating: removing 1 direction from r=256 trainable subspace
  per module per step is ~0.4% capacity. If `v_hack` at a module is noise, we
  ablate a noise direction in expectation = approx no-op. Argument for gating:
  in modules where hack signal is weak, projection just removes some random
  direction the optimizer might have used. H5 settles this.
- **Rank `r`.** Default 256 (lora-lite antipasto default); sweep in arm g.
  Trainable parameter count is just `r` per module (vs `r*(d_in+d_out)` for
  standard LoRA), so larger `r` is cheap, but `v_hack`'s SNR per dim degrades.

## Why measure ratio, not just hack rate

A model that learns nothing won't cheat. The honest metric is the *Pareto
frontier* of (hack rate, pass rate), not either alone. Pure hack-rate rewards
undertraining; pure pass-rate rewards anything that improves coding including
via the hack. Headline claim shape: "at matched pass rate ±5pp on held-out
problems without write access, our method reduces hack rate from X% to Y%."

## Compute estimate

- Single run on 96GB RTX 6000: ~2-3h (Qwen3.5-2B, num_gen=8, 200 steps, lsrl,
  AntiPaSTO r=256)
- 14 runs: 35-45h
- At ~$3 AUD/hr: $105-135 AUD
- + debugging buffer: budget ~$200 AUD total
- Calendar time: 1 week back-to-back; 2-3 weeks with iteration

## Risks and decision points

- **H4 falsified (no hack on Qwen3.5-2B with lsrl):** branch 1 — try
  Qwen3-4B same hyperparams. Branch 2 — drop lsrl, hook into verl
  directly. Adds ~1-2 weeks engineering.
- **AntiPaSTO + GRPO doesn't train:** known risk — antipasto's trainable
  subspace (`delta_S` only) may be too small for RL. Mitigation: enable
  Cayley rotation (`rotate_basis="V"`, `block_size=4`), adds `r*(bs-1)/2`
  params per module. Or fall back to PiSSA-LoRA-freeze-A.
- **`v_hack` steering check fails (per-module projection scores ≤chance):**
  extraction broken. Check (a) hook captures pre-residual input, (b) pair
  quality drives strong activation difference somewhere, (c) tokenization of
  hack vs clean completions isn't trivially distinguishing.
- **All methods tie vanilla on hack rate:** intervention not biting. Check
  `cos_align` logs nonzero, `frac_modules_projected` nonzero.

## What this is not

- Not a claim that rank-space gradient projection solves reward hacking
  generally
- Not a comparison to monitor-based methods (cite Ariahw's numbers, don't
  re-run)
- Not a claim about hacks beyond `run_tests()` overwrite
- Not a replacement for RLHF safety pipeline; this is a targeted intervention

## Related work and naming

- **Wu & Tang 2026, Rebound** ([arxiv:2604.01476](https://arxiv.org/abs/2604.01476)) —
  advantage-side concept-direction penalty during GRPO. Our H3 baseline.
- **Ariahw/Engels/Nanda 2025, rl-rewardhacking** ([github](https://github.com/ariahw/rl-rewardhacking)) —
  source of dataset, reward function, and `v_hack`-relevant `run_tests` hack pattern.
- **AntiPaSTO** ([wassname/lora-lite/variants/antipasto.py](https://github.com/wassname/lora-lite/blob/main/src/lora_lite/variants/antipasto.py),
  ([wassname/AntiPaSTO paper](https://github.com/wassname/AntiPaSTO)) — adapter
  we wrap with.
- **lsrl** ([lsdefine/lsrl](https://github.com/lsdefine/lsrl)) — GRPO trainer.
- **PiSSA** ([arxiv:2404.02948](https://arxiv.org/abs/2404.02948)) — frozen
  top-r SVD-init for LoRA; closest spiritual ancestor to AntiPaSTO.