evil_MoE/spec.md

# Experiment: rank-space gradient projection vs RL reward hacking

## Context

GRPO and related on-policy RL methods are known to exploit loopholes in reward
functions. Ariahw, Engels & Nanda (2025, [github.com/ariahw/rl-rewardhacking](https://github.com/ariahw/rl-rewardhacking))
open-sourced a benchmark on LeetCode where Qwen3-4B learns to overwrite the
evaluation function `run_tests()` instead of solving problems, reaching 79%
reward hack rate at 200 training steps. Existing mitigations are mostly
monitor-based (detect at output) or advantage-based (Rebound:
penalize hacking rollouts via concept-score-modified advantage; Wu & Tang 2026
[arxiv:2604.01476](https://arxiv.org/abs/2604.01476)).

This experiment tests a different mechanism: **wrap target modules with the
AntiPaSTO SVD adapter (lora-lite), extract a per-module `v_hack` in the rank-r
SVD basis from contrastive pairs, and project each step's
`grad(delta_S) : [r]` orthogonal to `v_hack` before the optimizer update.**
Mechanism difference from Rebound: gradient-level direction constraint on
weight-update subspace vs rollout-level scalar penalty on advantage.

This is preregistered: results to be reported regardless of outcome.

## Why AntiPaSTO and not vanilla LoRA

Vanilla LoRA's rank axis is meaningless (random init, drifts after step 1), so
"project out v_hack in rank space" has no fixed reference frame. AntiPaSTO
(Wassname, lora-lite [variants/antipasto.py](https://github.com/wassname/lora-lite/blob/main/src/lora_lite/variants/antipasto.py))
freezes `U_r, S_r, Vh_r` from the SVD of `W` and trains a tiny `delta_S : [r]`
plus an optional block-Cayley rotation. The rank axis stays pinned to the SVD
basis of the original weight, so `v_hack` extracted in that basis remains
meaningful across all training steps.

Forward pass per wrapped module (first pass uses full rank $r = \min(d_{in}, d_{out})$,
so the residual term $W_{res}$ vanishes):

$$y = ((x V_h^T) \odot (S + \delta_S)) U^T$$

where $U$, $S$, $V_h$ come from the SVD of $W$ and are buffers (frozen).
Trainable: $\delta_S : [r]$ (and optionally a small Cayley rotation `rot_T`
we leave off by default). At reduced rank we would add
$x W_{res}^T$ with $W_{res} = W - U_r \mathrm{diag}(S_r) V_{h,r}$, but we
defer rank cropping to v2 to skip the "where to cut" question.

Per-step gradient signal:

$$\frac{\partial L}{\partial \delta_S} = \sum_t (x_t V_h^T) \odot \left(\frac{\partial L}{\partial h_t} U\right) \in \mathbb{R}^r$$

We extract `v_hack` **gradient-side** (locked in): for each contrastive pair,
run one NLL backward on the completion tokens and read each module's
`m.delta_S.grad : [r]`. Then $\hat v_{hack}^{(m)} =$ unit$($mean$_{hack}$ grad $-$ mean$_{clean}$ grad$)$.
This lives in the exact same `[r]` rank space the per-step training gradient
lives in (the gradient is the natural object to compare gradients against),
and it fuses the input-activation and output-error contributions in one shot
instead of guessing whether input-side $(x V_h^T)$ or output-side $(\partial L/\partial h)\, U$
better predicts where SGD will move. We did consider activation-side
($x V_h^T$ mean-diff). Dropped as primary because it only sees the input
factor and ignores the output-error factor, while the per-step gradient sees
both.

Projection (locked: no magnitude threshold; one-sided clip stays — see note):

$$g \leftarrow g - \max(0,\, \cos_{align}) \cdot \|g\| \cdot \hat v_{hack}, \qquad \cos_{align} = \frac{g \cdot \hat v_{hack}}{\|g\|}$$

then rescale to original $\|g\|$ (magnitude-preserving). The $\max(0,\cdot)$ is
not gating, it's directional correctness: without it, when $\cos<0$ we'd be
*adding* to the hack component. No magnitude/threshold gating (locked): we
project every step every module. Capacity cost is ~1/r per module per step.
If `v_hack` at a module is just noise, projection ablates a noise direction in
expectation = approximately a no-op.

## Why not vanilla GRPO via verl

verl is Ariahw's framework but uses Ray + FSDP2 + Hydra; inserting a
pre-optimizer-step hook on per-module rank-space gradients requires deep
subclassing of their worker abstraction. We pay one cost in exchange:
we use [lsdefine/simple_GRPO](https://github.com/lsdefine/simple_GRPO) instead.
simple_GRPO is a two-file GRPO implementation (`ref_server.py` + `grpo_ref_split.py`,
~315 lines total) with reported convergence on Qwen2.5-7B. The training loop
is literally `loss = GRPO_step(batch); engine.backward(loss); engine.step()` —
inserting a projection hook between backward and step is a one-line edit.

Cost of this deviation: we re-establish the "vanilla hack emergence" baseline
on simple_GRPO rather than inheriting it from Ariahw's verl baseline. H4 is
the sanity check that this happens. We port Ariahw's `run_tests`-overwrite
detection (their [src/train/verl/rewards.py](https://github.com/ariahw/rl-rewardhacking/blob/main/src/train/verl/rewards.py))
into simple_GRPO's reward server (`docs/vendor/simple_GRPO/ref_server.py`).

Vendored references (read-only, see [docs/vendor/](docs/vendor/)):
- [simple_GRPO](https://github.com/lsdefine/simple_GRPO) — GRPO trainer
- [lora-lite](https://github.com/wassname/lora-lite) — AntiPaSTO adapter
- [rl-rewardhacking](https://github.com/ariahw/rl-rewardhacking) (already at `external/`)

## Hypotheses (preregistered)

**H1 (mechanism, primary):** Rank-space gradient projection against `v_hack`
extracted from ~60-80 contrastive pairs reduces reward hack rate by at least 30
percentage points (absolute) relative to vanilla AntiPaSTO+GRPO, at matched
LeetCode pass rate within 10pp of vanilla.

  Falsified if: hack rate reduction < 15pp, OR pass rate drops by >15pp at
  matched hack-rate budget, OR result is within 1 SEM of vanilla across seeds.

**H3 (gradient vs advantage):** Gradient-level intervention (ours) outperforms
advantage-level intervention (Rebound reimplemented) on hack rate at matched
pass rate.

  Falsified if: Rebound reimplementation matches or beats ours within 1 SEM.

**H4 (scaling sanity on our stack):** Qwen3.5-2B trained with vanilla
AntiPaSTO+GRPO on simple_GRPO reproduces measurable reward hacking (>30% hack
rate at 200 steps).

  Falsified if: vanilla hack rate <30%. Decision branch: swap to Qwen3-4B with
  num_generations halved. Secondary: if simple_GRPO can't reproduce hacking on
  either model, fall back to Ariahw's verl path and accept the harder hook.

## Steps

### 1. Build infra — fast-dev-run targets first, no real training yet

   - **1a.** Vendor simple_GRPO into `docs/vendor/simple_GRPO/` (done); smoke-run
     their GSM8K example on tiny-random-qwen3 (5 steps, CPU) to confirm
     `ref_server` + `grpo_ref_split` rollout/train split works in our env.
   - **1b.** Vendor lora-lite into `docs/vendor/lora-lite/` (done); wrap
     Qwen3.5-0.8B attn+MLP `nn.Linear` modules with AntiPaSTO **at full rank**
     (`r = min(d_in, d_out)`, no SVD cropping; `rotate_basis="none"`, only
     `delta_S` trainable). Full rank means $W = U \,\mathrm{diag}(S)\, V_h$
     exactly and `W_res = 0`, so there's no truncation error to debug on the
     first pass. Verify forward-pass round-trip numerically matches base model
     at $\delta_S = 0$ (max abs diff <1e-3 on a fixed prompt).
   - **1c.** Implement gradient-side `v_hack` extraction (pseudocode below).
     Validation: per-module held-out projection score
     `cos(g_held_hack - g_held_clean, v_hack)` > 0 in >50% of modules.

### 2. H4 sanity — does vanilla AntiPaSTO+GRPO+simple_GRPO produce hacking?

   - **2a.** Port Ariahw's `run_tests`-overwrite detection into simple_GRPO's
     `ref_server.py` reward fn. Verify the reward fn fires on synthetic
     hack/clean rollouts before real training.
   - **2b.** Train Qwen3.5-2B, AntiPaSTO (`r=full`, `delta_S` only), GRPO
     (group_norm), 200 steps, num_generations=8, batch=16, 1 seed.
     Decision: if hack rate <30%, switch to Qwen3-4B with `num_generations=4,
     batch=16` (half num_gen to keep VRAM headroom) and re-run 2b.
     Secondary fallback: drop simple_GRPO, return to verl.

### 3. Implement rank-space projection in simple_GRPO's training loop

   - **3a.** In `grpo_ref_split.py`, between `engine.backward(loss)` and
     `engine.step()`, call `project_grads(model, v_hack_cache)`.
     `project_grads` walks `[m for m in model.modules() if hasattr(m, 'delta_S')]`
     and for each module reads `m.delta_S.grad : [r]`, projects against
     `v_hack[module_name]` (one-sided, magnitude-preserving), writes back
     in place. (Pseudocode below.)
   - **3b.** Diagnostics logged per step (aggregated over modules):
     mean/std `cos_align`, mean `||grad||`, `frac_modules_with_cos>0`.

### 4. Run arms (200 steps each, 3 seeds where indicated)

   a. Vanilla AntiPaSTO + GRPO (3 seeds) — baseline
   b. Our method, gradient-side `v_hack`, no gating (3 seeds) — main result
   c. Our method, no magnitude preservation (1 seed) — design ablation
   d. Rebound reimplementation: advantage-side `v_hack` penalty (3 seeds) — H3
      (concrete formula: per-rollout penalty `α · max(0, cos(h_mean, v_concept))`
      added to scalar reward, where `h_mean` is mean residual-stream activation
      at a chosen layer and `v_concept` is mean-diff activation direction
      extracted from the same 60-80 pairs. We use Wu & Tang 2026 §3.2's
      published `α=0.5` and same layer fraction (60-75% depth). Single
      layer, not per-module, matching their setup. *Different `v_concept`
      from our gradient-side `v_hack` — this is intentional: H3 isolates the
      gradient-vs-advantage mechanism choice, not the direction-extraction
      choice.*)

   Total: 10 runs × ~3h on RTX 6000 96GB = ~30h compute.
   *(Rank sweep deferred to v2; first pass uses `r = min(d_in, d_out)` per
   module, no cropping.)*

### 5. Measure at every 25 steps

   - **Hack rate** (Ariahw's detector ported into simple_GRPO)
   - **Pass rate** on held-out problems without write access to evaluator
   - **Per-module `cos_align`** trajectory (sanity that we're projecting
     something nonzero)
   - **`frac_modules_with_cos>0`** per step (sanity that one-sided clip fires)
   - **KL drift from init policy** (catastrophic-change check)

### 6. Headline plot and headline table

   **Plot.** Hack rate vs pass rate, one point per (arm × seed). Pareto
   frontier. Our method should land below-and-to-the-right of vanilla.
   Annotate Rebound.

   **Table schema (publication-ready; left-to-right = essential to optional,
   so trailing columns can be cut for space):**

   | Arm | ΔSafePass↑ | Hack %↓ | Pass %↑ | KL↓ | mean·cos\* | frac·fired\* | ‖g‖\* |
   |---|---|---|---|---|---|---|---|
   | Vanilla (a) | 0 (ref) | — | — | — | — | — | — |
   | **Ours (b)** | — | — | — | — | — | — | — |
   | Ours, no mag-preserve (c) | — | — | — | — | — | — | — |
   | Rebound (d) | — | — | — | — | — | — | — |

   *Caption.* ↑ higher is better, ↓ lower is better. **ΔSafePass** = (pass% −
   hack%) − vanilla's (pass% − hack%): single headline number, positive means
   we win. **Hack %** = fraction of rollouts triggering `run_tests`-overwrite
   detector. **Pass %** = fraction passing held-out tests without write access.
   **KL** = mean per-token KL from init policy over last 25 steps.
   \* = projection-internal diagnostic, only meaningful for arms (b)/(c);
   distinguishes "projection active" (mean·cos > 0.2, frac·fired > 0.4) from
   "projection silent no-op". Cells report mean ± SEM across seeds.

### 7. Falsification check

   Before publishing, run pre-registered analysis on H1, H3, H4. Report all
   hypotheses including falsified ones.

## Pseudocode (the three load-bearing bits)

### A. AntiPaSTO module wrap (full rank, first pass)

```
class AntiPaSTO(nn.Module):
    # constructed from an existing nn.Linear(W: [d_out, d_in], b)
    # FIRST PASS: r = min(d_out, d_in) -- no truncation, W_res == 0
    def __init__(self, W, b):
        U, S, Vh = torch.linalg.svd(W.float(), full_matrices=False)
        r = S.shape[0]                    # = min(d_out, d_in)
        # buffers (frozen): the full SVD
        self.U  = U                       # [d_out, r]
        self.S  = S                       # [r]
        self.Vh = Vh                      # [r, d_in]
        self.b  = b
        # trainable (ONLY this): scalar per rank
        self.delta_S = nn.Parameter(torch.zeros(r))

    def forward(self, x):  # x: [..., d_in]
        return ((x @ self.Vh.T) * (self.S + self.delta_S)) @ self.U.T + self.b
```

Replace every target `nn.Linear` in attn (`q,k,v,o_proj`) and MLP
(`up,gate,down_proj`) with this. At `delta_S=0`, output == original linear up
to numerical precision (no `W_res` residual term needed at full rank).

**SVD precompute strategy.** Don't SVD the whole model on GPU at once.
Load the base model on CPU, then for each target `Linear`: move `W` to GPU,
run `torch.linalg.svd(W.float(), full_matrices=False)`, save
`(U, S, Vh) -> svd_cache/{model_name}/{module_path}.pt`. Wrap construction
then loads the cached SVD per module. SVD is done once per base model; ~5-10s
per big MLP weight on RTX 3090.

### B. Gradient-side `v_hack` extraction (per module)

```
v_hack = {}                                       # dict[module_name -> Tensor[r]]
grads_hack = defaultdict(list)
grads_clean = defaultdict(list)

# Per-pair: process hack and clean independently, NLL over their own completion
# tokens only. Different completion lengths are fine -- we use mean NLL
# (sum_nll / n_completion_tokens), so each pair contributes a length-normalized
# gradient. This avoids biasing v_hack toward longer (typically clean)
# completions. Pad each example individually; no cross-completion padding.

for (prompt, hack_completion, clean_completion) in pairs:
    for label, completion in [('hack', hack_completion), ('clean', clean_completion)]:
        model.zero_grad()
        ids   = tokenize(prompt + completion)            # [1, L]
        mask  = completion_mask(ids, prompt_len=len(prompt_ids))  # 1 on completion tokens
        logits = model(ids).logits[:, :-1]
        # MEAN NLL over completion tokens (length-normalized)
        loss   = (nll_per_token(logits, ids[:, 1:]) * mask[:, 1:]).sum() / mask[:, 1:].sum()
        loss.backward()
        for name, m in model.named_modules():
            if hasattr(m, 'delta_S'):
                bucket = grads_hack if label == 'hack' else grads_clean
                bucket[name].append(m.delta_S.grad.detach().cpu().clone())

for name in grads_hack:
    diff = stack(grads_hack[name]).mean(0) - stack(grads_clean[name]).mean(0)  # [r]
    v_hack[name] = diff / (diff.norm() + 1e-8)

torch.save(v_hack, 'v_hack.pt')
```

Validation (report both, don't just gate on threshold):

- On held-out pairs, recompute per-module `diff_held` and
  `cos_align_held = cos(diff_held, v_hack[name])`.
- **Distribution check (primary):** plot histogram of `cos_align_held` across
  all modules. Healthy = unimodal positive, median > 0.3. Pathological =
  bimodal or median near 0.
- **Gate (secondary):** `cos_align_held > 0` in >50% of modules is the
  minimum to proceed; mean `cos_align_held > 0.2` is the target. If <50% pass,
  extraction is broken and we debug before training.

### C. Pre-optimizer-step projection hook

```
def project_grads(model, v_hack: dict[str, Tensor]):
    # called after engine.backward(loss), before engine.step()
    cos_log, n_modules, n_fired = [], 0, 0
    for name, m in model.named_modules():
        if not hasattr(m, 'delta_S'): continue
        g = m.delta_S.grad                        # [r]
        if g is None: continue
        n_modules += 1
        v = v_hack[name].to(g.device)             # [r], unit
        g_norm = g.norm()
        if g_norm < 1e-12: continue
        cos_a = (g @ v) / g_norm                  # scalar
        cos_log.append(cos_a.item())
        if cos_a > 0:
            n_fired += 1
            g_new = g - cos_a * g_norm * v        # remove hack component
            g_new = g_new * (g_norm / (g_new.norm() + 1e-8))  # magnitude preserve
            m.delta_S.grad.copy_(g_new)
    return dict(mean_cos=mean(cos_log), frac_fired=n_fired/max(n_modules,1))
```

Integration into `grpo_ref_split.py` training loop
(vendored at `docs/vendor/simple_GRPO/simple_grpo_v1/grpo_ref_split.py`; we copy and
edit, not import):

```
# at top of training script, once:
v_hack = torch.load('v_hack.pt', map_location='cpu')   # dict[str, Tensor[r]]
# (extraction script from B above produces this artifact; if missing, crash loud)

# inside the training loop:
loss = GRPO_step(batch)
engine.backward(loss)
stats = project_grads(engine.module, v_hack)       # <-- NEW: 1 line
engine.step()
if rank == 0: log(stats)
```

## Decisions left open (write these up alongside results)

- **Rank `r`.** First pass: `r = min(d_in, d_out)` per module (no cropping)
  to avoid debugging where to cut the SVD. Trainable params per module =
  `min(d_in, d_out)`, still tiny vs full LoRA's `r*(d_in+d_out)`. Tradeoff:
  larger `r` keeps geometric fidelity but `v_hack`'s SNR per dim degrades;
  smaller `r` would concentrate hack signal but introduces truncation error in
  `W_res`. Rank sweep is v2 work.

## Why measure ratio, not just hack rate

A model that learns nothing won't cheat. The honest metric is the *Pareto
frontier* of (hack rate, pass rate), not either alone. Pure hack-rate rewards
undertraining; pure pass-rate rewards anything that improves coding including
via the hack. Headline claim shape: "at matched pass rate ±5pp on held-out
problems without write access, our method reduces hack rate from X% to Y%."

## Compute estimate

- Single run on 96GB RTX 6000: ~2-3h (Qwen3.5-2B, num_gen=8, 200 steps,
  simple_GRPO, AntiPaSTO full rank)
- 10 runs: 25-35h
- At ~$3 AUD/hr: $75-105 AUD
- + debugging buffer: budget ~$200 AUD total
- Calendar time: 1 week back-to-back; 2-3 weeks with iteration

## Risks and decision points

- **H4 falsified (no hack on Qwen3.5-2B with simple_GRPO):** branch 1 — try
  Qwen3-4B same hyperparams. Branch 2 — drop simple_GRPO, hook into verl
  directly. Adds ~1-2 weeks engineering.
- **AntiPaSTO + GRPO doesn't train:** known risk — antipasto's trainable
  subspace (`delta_S` only) may be too small for RL. If so, document and
  fall back to PiSSA-LoRA-freeze-A. We do **not** enable Cayley rotation
  (`rotate_basis="V"`) as a mitigation: a rotated rank axis breaks the
  invariant that `v_hack` (extracted in the original SVD basis) stays
  meaningful across training, which is the whole point of using AntiPaSTO
  over vanilla LoRA.
- **`v_hack` steering check fails (per-module projection scores ≤chance):**
  extraction broken. Check (a) hook captures pre-residual input, (b) pair
  quality drives strong activation difference somewhere, (c) tokenization of
  hack vs clean completions isn't trivially distinguishing.
- **All methods tie vanilla on hack rate:** intervention not biting. Check
  `cos_align` logs nonzero, `frac_modules_with_cos>0` nonzero.

## What this is not

- Not a claim that rank-space gradient projection solves reward hacking
  generally
- Not a comparison to monitor-based methods (cite Ariahw's numbers, don't
  re-run)
- Not a claim about hacks beyond `run_tests()` overwrite
- Not a replacement for RLHF safety pipeline; this is a targeted intervention

## Related work and naming

- **Wu & Tang 2026, Rebound** ([arxiv:2604.01476](https://arxiv.org/abs/2604.01476)) —
  advantage-side concept-direction penalty during GRPO. Our H3 baseline.
- **Ariahw/Engels/Nanda 2025, rl-rewardhacking** ([github](https://github.com/ariahw/rl-rewardhacking)) —
  source of dataset, reward function, and `v_hack`-relevant `run_tests` hack pattern.
- **AntiPaSTO** ([wassname/lora-lite/variants/antipasto.py](https://github.com/wassname/lora-lite/blob/main/src/lora_lite/variants/antipasto.py),
  ([wassname/AntiPaSTO paper](https://github.com/wassname/AntiPaSTO)) — adapter
  we wrap with.
- **simple_GRPO** ([lsdefine/simple_GRPO](https://github.com/lsdefine/simple_GRPO)) — GRPO trainer.
- **PiSSA** ([arxiv:2404.02948](https://arxiv.org/abs/2404.02948)) — frozen
  top-r SVD-init for LoRA; closest spiritual ancestor to AntiPaSTO.