evil_MoE/docs/RESEARCH_JOURNAL.md

# Research Journal

Append-only. New entries at the top, date-stamped. Never edit old entries.

# 2026-05-30

## 96GB readiness review fixes

Fresh subagent review found a real silent-failure risk: `v_hack` is not just
model-specific, it is also SVD-basis-specific. The old extractor loaded fp32
while `train.py` loaded bf16, so keys/ranks could match while the basis differed.
Fix: `extract_vhack_grad.py`, `verify_vhack_heldout.py`, and `train.py` now all
use bf16 by default; `v_hack` artifacts save `{model, dtype, v_hack}` metadata;
`train.py` refuses legacy artifacts and checks exact module keys and per-module
rank before first generation.

Also removed a bad smoke convenience: zero-spread reward batches no longer get
random advantages. Dr.GRPO now correctly gives zero advantage when all group
rewards match, so logs cannot look healthy while training on reward-unrelated
noise.

Validated on the 24GB box:

- `just extract-vhack-smoke` via pueue task 73: bf16, 186 modules, 148,032
  delta_S scalars, zero-norm=0.
- `just verify-vhack-smoke` via pueue task 74: `frac>0=0.952`, `mean=+0.355`,
  `median=+0.363`, target pass.
- one-step canonical train probe via pueue task 75: loaded `out/v_hack_smoke.pt`
  with key/rank match OK, completed without legacy artifact. Reward spread was
  false and loss/cos/fired were zero, as expected after removing random advantages.

For the 96GB machine, do not start `queue-full` blindly. First run one sequential
gate: `pueue add --immediate --follow -w "$PWD" -o 9 -l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" -- just probe-full-seed 41`.
Only queue 3 seeds after the vanilla probe has nontrivial hack rate.

## Mechanism end-to-end verified on Qwen3.5-0.8B; H4 falsified at this scale

Closed the smoke loop: AntiPaSTO identity (bf16, max_abs_diff=0) -> v_hack
extraction from 15 contrastive pairs -> held-out validation (frac>0=0.952,
median cos=+0.363, n=186 modules) -> 10-step GRPO with subprocess-executed
LeetCode rewards on vanilla and projected arms. Full writeup in
[out/proof.md](../out/proof.md).

**Observation (mechanism)**: projected arm shows `cos_out < cos_in` every step,
`frac_fired ≈ 0.51` averaged over 10 steps. Vanilla arm: `cos_out == cos_in`.
The one-sided projection removes the v_hack-aligned component of the SVD-basis
gradient when and only when alignment is positive. This is the core mechanical
claim of the method and it is verified end-to-end.

**Observation (H4 sanity)**: both arms produce zero hack_rate and zero pass_rate
on 30 LeetCode medium/hard problems, G=2, 10 steps. Inspection of generations
shows Qwen3.5-0.8B emits format-only output that saturates the 0.25 format
bonus but never attempts code or hack patterns. Per [spec.md](../spec.md) §H4,
this falls below the 30% hack-rate threshold and triggers the model-scaling
fallback.

**Inference**: 0.8B is too small to exhibit the failure mode the method
targets. The mechanism is sound; the test substrate is not. Wu & Tang's
Rebound paper used Qwen2.5-Coder-7B and observed ~50% baseline hack rate;
Ariahw's benchmark assumes ≥4B class models. Mechanism + scale are
separable concerns and the smaller scope of this session was mechanism.

**Caveats / what's untested**:

- β=0 in smoke (no ref-model KL) to fit 24 GB. This is a 24-GB compromise, NOT
  a principled choice. Dr.GRPO argues β=0 is fine for reasoning RL with
  rule-based reward, but we're studying *reward hacking*, which IS the
  distributional shift their argument assumes away. lite/full presets default
  to β=0.04 to match Ariahw 2025 and Wu-Tang Rebound 2026; without that we'd
  confound "hacking from the targeted shortcut direction" with "generic
  policy collapse". Free-ref-model trick (delta_S=0 forward) makes β>0
  zero-VRAM-cost, so lite/full can do this properly.
- Only 10 steps. Reward-hacking emerges around step 50–200 in Rebound figs.
- 186 target modules, full-rank per-module SVD. Larger models scale similarly.
- `frac_fired ≈ 0.5` is consistent with random gradient direction wrt v_hack
  at init; we expect it to rise as training induces hack-aligned grads. Need
  longer runs to see this.

**Next (queued in [justfile](../justfile), pending ≥80 GB GPU)**:

1. `queue-vanilla`: Qwen2.5-Coder-7B baseline GRPO on full LeetCode set, 200
   steps, 3 seeds, β=0.04, G=4. Expected hack_rate at convergence: 40–60%
   (Rebound table 2).
2. `queue-projected-m16`: same config + per-module v_hack projection at m=16.
3. `queue-rebound`: H3 baseline arm — Wu-Tang advantage modification.

Confidence in method post-mechanism-verification: ~65% (was ~60%). The bump is
small because mechanism-works was already high-prior; the real evidence is the
7B run.


## Project init

Scaffolded repo per setup-repo skill. Cloned [external/rl-rewardhacking](external/rl-rewardhacking/)
(Ariahw's verl-based GRPO + LeetCode reward-hacking benchmark) and fetched the
three key papers ([docs/papers/](docs/papers/)):

- Ariahw, Engels, Nanda 2025 (LessWrong) — the benchmark and monitor-based interventions
- Wu & Tang 2026 (arXiv 2604.01476) — "When Reward Hacking Rebounds"; proposes
  Advantage Modification using shortcut concept direction. This is the closest
  prior work to ours and the H3 baseline arm.
- Ichihara et al. 2025 (arXiv 2509.22047) — MO-GRPO; multi-objective GRPO with
  per-reward variance normalization. Related framing of reward hacking as
  high-variance reward dominating advantage.

Extracted brainstorm prefs to [docs/brainstorm/extracted_prefs.md](docs/brainstorm/extracted_prefs.md).
Biggest delta vs spec.md: the project pivoted mid-brainstorm from DPO+sycophancy
to GRPO+reward-hacking, and the method evolved from bidirectional NLL+KL+PCGrad
(paired-preference) to gradient-level projection (unpaired). Confidence ~60% the
method works post-Rebound (was ~40% pre-Rebound; Rebound validates the core
mechanism — concept-direction-based intervention — but at advantage rather than
gradient level).

**Next:** smoke test both pathways on tiny-random Qwen, prototype the results table,
then move to 96GB GPU for the H4 sanity run.