evil_MoE/spec.md

Experiment: rank-space gradient projection vs RL reward hacking

Context

GRPO and related on-policy RL methods are known to exploit loopholes in reward functions. Ariahw, Engels & Nanda (2025, github.com/ariahw/rl-rewardhacking) open-sourced a benchmark on LeetCode where Qwen3-4B learns to overwrite the evaluation function run_tests() instead of solving problems, reaching 79% reward hack rate at 200 training steps. Existing mitigations are mostly monitor-based (detect at output) or advantage-based (Rebound: penalize hacking rollouts via concept-score-modified advantage; Wu & Tang 2026 arxiv:2604.01476).

This experiment tests a different mechanism: wrap target modules with the AntiPaSTO SVD adapter (lora-lite), extract a per-module v_hack in the rank-r SVD basis from contrastive pairs, and project each step's grad(delta_S) : [r] orthogonal to v_hack before the optimizer update. Mechanism difference from Rebound: gradient-level direction constraint on weight-update subspace vs rollout-level scalar penalty on advantage.

This is preregistered: results to be reported regardless of outcome.

Why AntiPaSTO and not vanilla LoRA

Vanilla LoRA's rank axis is meaningless (random init, drifts after step 1), so "project out v_hack in rank space" has no fixed reference frame. AntiPaSTO (Wassname, lora-lite variants/antipasto.py) freezes U_r, S_r, Vh_r from the SVD of W and trains a tiny delta_S : [r] plus an optional block-Cayley rotation. The rank axis stays pinned to the SVD basis of the original weight, so v_hack extracted in that basis remains meaningful across all training steps.

Forward pass per wrapped module (first pass uses full rank r = \min(d_{in}, d_{out}), so the residual term W_{res} vanishes):

y = ((x V_h^T) \odot (S + \delta_S)) U^T

where U, S, V_h come from the SVD of W and are buffers (frozen). Trainable: \delta_S : [r] (and optionally a small Cayley rotation rot_T we leave off by default). At reduced rank we would add x W_{res}^T with W_{res} = W - U_r \mathrm{diag}(S_r) V_{h,r}, but we defer rank cropping to v2 to skip the "where to cut" question.

Per-step gradient signal:

\frac{\partial L}{\partial \delta_S} = \sum_t (x_t V_h^T) \odot \left(\frac{\partial L}{\partial h_t} U\right) \in \mathbb{R}^r

We extract v_hack gradient-side (locked in): for each contrastive pair, run one NLL backward on the completion tokens and read each module's m.delta_S.grad : [r]. Then \hat v_{hack}^{(m)} = unit$($mean${hack}$ grad - mean${clean} grad)$. This lives in the exact same [r] rank space the per-step training gradient lives in (the gradient is the natural object to compare gradients against), and it fuses the input-activation and output-error contributions in one shot instead of guessing whether input-side (x V_h^T) or output-side (\partial L/\partial h)\, U better predicts where SGD will move. We did consider activation-side (x V_h^T mean-diff). Dropped as primary because it only sees the input factor and ignores the output-error factor, while the per-step gradient sees both.

Projection (locked: no magnitude threshold; one-sided clip stays — see note):

g \leftarrow g - \max(0,\, \cos_{align}) \cdot \|g\| \cdot \hat v_{hack}, \qquad \cos_{align} = \frac{g \cdot \hat v_{hack}}{\|g\|}

then rescale to original \|g\| (magnitude-preserving). The \max(0,\cdot) is not gating, it's directional correctness: without it, when \cos<0 we'd be adding to the hack component. No magnitude/threshold gating (locked): we project every step every module. Capacity cost is ~1/r per module per step. If v_hack at a module is just noise, projection ablates a noise direction in expectation = approximately a no-op.

Why not vanilla GRPO via verl

verl is Ariahw's framework but uses Ray + FSDP2 + Hydra; inserting a pre-optimizer-step hook on per-module rank-space gradients requires deep subclassing of their worker abstraction. We pay one cost in exchange: we use lsdefine/simple_GRPO instead. simple_GRPO is a two-file GRPO implementation (ref_server.py + grpo_ref_split.py, ~315 lines total) with reported convergence on Qwen2.5-7B. The training loop is literally loss = GRPO_step(batch); engine.backward(loss); engine.step() — inserting a projection hook between backward and step is a one-line edit.

Cost of this deviation: we re-establish the "vanilla hack emergence" baseline on simple_GRPO rather than inheriting it from Ariahw's verl baseline. H4 is the sanity check that this happens. We port Ariahw's run_tests-overwrite detection (their src/train/verl/rewards.py) into simple_GRPO's reward server (docs/vendor/simple_GRPO/ref_server.py).

Vendored references (read-only, see docs/vendor/):

• simple_GRPO — GRPO trainer
• lora-lite — AntiPaSTO adapter
• rl-rewardhacking (already at external/)

Hypotheses (preregistered)

H1 (mechanism, primary): Rank-space gradient projection against v_hack extracted from ~60-80 contrastive pairs reduces reward hack rate by at least 30 percentage points (absolute) relative to vanilla AntiPaSTO+GRPO, at matched LeetCode pass rate within 10pp of vanilla.

Falsified if: hack rate reduction < 15pp, OR pass rate drops by >15pp at matched hack-rate budget, OR result is within 1 SEM of vanilla across seeds.

H3 (gradient vs advantage): Gradient-level intervention (ours) outperforms advantage-level intervention (Rebound reimplemented) on hack rate at matched pass rate.

Falsified if: Rebound reimplementation matches or beats ours within 1 SEM.

H4 (scaling sanity on our stack): Qwen3.5-2B trained with vanilla AntiPaSTO+GRPO on simple_GRPO reproduces measurable reward hacking (>30% hack rate at 200 steps).

Falsified if: vanilla hack rate <30%. Decision branch: swap to Qwen3-4B with num_generations halved. Secondary: if simple_GRPO can't reproduce hacking on either model, fall back to Ariahw's verl path and accept the harder hook.

Steps

1. Build infra — fast-dev-run targets first, no real training yet

• 1a. Vendor simple_GRPO into docs/vendor/simple_GRPO/ (done); smoke-run their GSM8K example on tiny-random-qwen3 (5 steps, CPU) to confirm ref_server + grpo_ref_split rollout/train split works in our env.
• 1b. Vendor lora-lite into docs/vendor/lora-lite/ (done); wrap Qwen3.5-0.8B attn+MLP nn.Linear modules with AntiPaSTO at full rank (r = min(d_in, d_out), no SVD cropping; rotate_basis="none", only delta_S trainable). Full rank means W = U \,\mathrm{diag}(S)\, V_h exactly and W_res = 0, so there's no truncation error to debug on the first pass. Verify forward-pass round-trip numerically matches base model at \delta_S = 0 (max abs diff <1e-3 on a fixed prompt).
• 1c. Implement gradient-side v_hack extraction (pseudocode below). Validation: per-module held-out projection score cos(g_held_hack - g_held_clean, v_hack) > 0 in >50% of modules.

2. H4 sanity — does vanilla AntiPaSTO+GRPO+simple_GRPO produce hacking?

• 2a. Port Ariahw's run_tests-overwrite detection into simple_GRPO's ref_server.py reward fn. Verify the reward fn fires on synthetic hack/clean rollouts before real training.
• 2b. Train Qwen3.5-2B, AntiPaSTO (r=full, delta_S only), GRPO (group_norm), 200 steps, num_generations=8, batch=16, 1 seed. Decision: if hack rate <30%, switch to Qwen3-4B with num_generations=4, batch=16 (half num_gen to keep VRAM headroom) and re-run 2b. Secondary fallback: drop simple_GRPO, return to verl.

3. Implement rank-space projection in simple_GRPO's training loop

• 3a. In grpo_ref_split.py, between engine.backward(loss) and engine.step(), call project_grads(model, v_hack_cache). project_grads walks [m for m in model.modules() if hasattr(m, 'delta_S')] and for each module reads m.delta_S.grad : [r], projects against v_hack[module_name] (one-sided, magnitude-preserving), writes back in place. (Pseudocode below.)
• 3b. Diagnostics logged per step (aggregated over modules): mean/std cos_align, mean ||grad||, frac_modules_with_cos>0.

4. Run arms (200 steps each, 3 seeds where indicated)

a. Vanilla AntiPaSTO + GRPO (3 seeds) — baseline b. Our method, gradient-side v_hack, no gating (3 seeds) — main result c. Our method, no magnitude preservation (1 seed) — design ablation d. Rebound reimplementation: advantage-side v_hack penalty (3 seeds) — H3 (concrete formula: per-rollout penalty α · max(0, cos(h_mean, v_concept)) added to scalar reward, where h_mean is mean residual-stream activation at a chosen layer and v_concept is mean-diff activation direction extracted from the same 60-80 pairs. We use Wu & Tang 2026 §3.2's published α=0.5 and same layer fraction (60-75% depth). Single layer, not per-module, matching their setup. Different v_concept from our gradient-side v_hack — this is intentional: H3 isolates the gradient-vs-advantage mechanism choice, not the direction-extraction choice.)

Total: 10 runs × ~3h on RTX 6000 96GB = ~30h compute. (Rank sweep deferred to v2; first pass uses r = min(d_in, d_out) per module, no cropping.)

5. Measure at every 25 steps

• Hack rate (Ariahw's detector ported into simple_GRPO)
• Pass rate on held-out problems without write access to evaluator
• Per-module cos_align trajectory (sanity that we're projecting something nonzero)
• frac_modules_with_cos>0 per step (sanity that one-sided clip fires)
• KL drift from init policy (catastrophic-change check)

6. Headline plot and headline table

Plot. Hack rate vs pass rate, one point per (arm × seed). Pareto frontier. Our method should land below-and-to-the-right of vanilla. Annotate Rebound.

Table schema (publication-ready; left-to-right = essential to optional, so trailing columns can be cut for space):

Arm	ΔSafePass↑	Hack %↓	Pass %↑	KL↓	mean·cos*	frac·fired*	‖g‖*
Vanilla (a)	0 (ref)	—	—	—	—	—	—
Ours (b)	—	—	—	—	—	—	—
Ours, no mag-preserve (c)	—	—	—	—	—	—	—
Rebound (d)	—	—	—	—	—	—	—

Caption. ↑ higher is better, ↓ lower is better. ΔSafePass = (pass% − hack%) − vanilla's (pass% − hack%): single headline number, positive means we win. Hack % = fraction of rollouts triggering run_tests-overwrite detector. Pass % = fraction passing held-out tests without write access. KL = mean per-token KL from init policy over last 25 steps. * = projection-internal diagnostic, only meaningful for arms (b)/(c); distinguishes "projection active" (mean·cos > 0.2, frac·fired > 0.4) from "projection silent no-op". Cells report mean ± SEM across seeds.

7. Falsification check

Before publishing, run pre-registered analysis on H1, H3, H4. Report all hypotheses including falsified ones.

Pseudocode (the three load-bearing bits)

A. AntiPaSTO module wrap (full rank, first pass)

class AntiPaSTO(nn.Module):
    # constructed from an existing nn.Linear(W: [d_out, d_in], b)
    # FIRST PASS: r = min(d_out, d_in) -- no truncation, W_res == 0
    def __init__(self, W, b):
        U, S, Vh = torch.linalg.svd(W.float(), full_matrices=False)
        r = S.shape[0]                    # = min(d_out, d_in)
        # buffers (frozen): the full SVD
        self.U  = U                       # [d_out, r]
        self.S  = S                       # [r]
        self.Vh = Vh                      # [r, d_in]
        self.b  = b
        # trainable (ONLY this): scalar per rank
        self.delta_S = nn.Parameter(torch.zeros(r))

    def forward(self, x):  # x: [..., d_in]
        return ((x @ self.Vh.T) * (self.S + self.delta_S)) @ self.U.T + self.b

Replace every target nn.Linear in attn (q,k,v,o_proj) and MLP (up,gate,down_proj) with this. At delta_S=0, output == original linear up to numerical precision (no W_res residual term needed at full rank).

SVD precompute strategy. Don't SVD the whole model on GPU at once. Load the base model on CPU, then for each target Linear: move W to GPU, run torch.linalg.svd(W.float(), full_matrices=False), save (U, S, Vh) -> svd_cache/{model_name}/{module_path}.pt. Wrap construction then loads the cached SVD per module. SVD is done once per base model; ~5-10s per big MLP weight on RTX 3090.

B. Gradient-side v_hack extraction (per module)

v_hack = {}                                       # dict[module_name -> Tensor[r]]
grads_hack = defaultdict(list)
grads_clean = defaultdict(list)

# Per-pair: process hack and clean independently, NLL over their own completion
# tokens only. Different completion lengths are fine -- we use mean NLL
# (sum_nll / n_completion_tokens), so each pair contributes a length-normalized
# gradient. This avoids biasing v_hack toward longer (typically clean)
# completions. Pad each example individually; no cross-completion padding.

for (prompt, hack_completion, clean_completion) in pairs:
    for label, completion in [('hack', hack_completion), ('clean', clean_completion)]:
        model.zero_grad()
        ids   = tokenize(prompt + completion)            # [1, L]
        mask  = completion_mask(ids, prompt_len=len(prompt_ids))  # 1 on completion tokens
        logits = model(ids).logits[:, :-1]
        # MEAN NLL over completion tokens (length-normalized)
        loss   = (nll_per_token(logits, ids[:, 1:]) * mask[:, 1:]).sum() / mask[:, 1:].sum()
        loss.backward()
        for name, m in model.named_modules():
            if hasattr(m, 'delta_S'):
                bucket = grads_hack if label == 'hack' else grads_clean
                bucket[name].append(m.delta_S.grad.detach().cpu().clone())

for name in grads_hack:
    diff = stack(grads_hack[name]).mean(0) - stack(grads_clean[name]).mean(0)  # [r]
    v_hack[name] = diff / (diff.norm() + 1e-8)

torch.save(v_hack, 'v_hack.pt')

Validation (report both, don't just gate on threshold):

• On held-out pairs, recompute per-module diff_held and cos_align_held = cos(diff_held, v_hack[name]).
• Distribution check (primary): plot histogram of cos_align_held across all modules. Healthy = unimodal positive, median > 0.3. Pathological = bimodal or median near 0.
• Gate (secondary): cos_align_held > 0 in >50% of modules is the minimum to proceed; mean cos_align_held > 0.2 is the target. If <50% pass, extraction is broken and we debug before training.

C. Pre-optimizer-step projection hook

def project_grads(model, v_hack: dict[str, Tensor]):
    # called after engine.backward(loss), before engine.step()
    cos_log, n_modules, n_fired = [], 0, 0
    for name, m in model.named_modules():
        if not hasattr(m, 'delta_S'): continue
        g = m.delta_S.grad                        # [r]
        if g is None: continue
        n_modules += 1
        v = v_hack[name].to(g.device)             # [r], unit
        g_norm = g.norm()
        if g_norm < 1e-12: continue
        cos_a = (g @ v) / g_norm                  # scalar
        cos_log.append(cos_a.item())
        if cos_a > 0:
            n_fired += 1
            g_new = g - cos_a * g_norm * v        # remove hack component
            g_new = g_new * (g_norm / (g_new.norm() + 1e-8))  # magnitude preserve
            m.delta_S.grad.copy_(g_new)
    return dict(mean_cos=mean(cos_log), frac_fired=n_fired/max(n_modules,1))

Integration into grpo_ref_split.py training loop (vendored at docs/vendor/simple_GRPO/simple_grpo_v1/grpo_ref_split.py; we copy and edit, not import):

# at top of training script, once:
v_hack = torch.load('v_hack.pt', map_location='cpu')   # dict[str, Tensor[r]]
# (extraction script from B above produces this artifact; if missing, crash loud)

# inside the training loop:
loss = GRPO_step(batch)
engine.backward(loss)
stats = project_grads(engine.module, v_hack)       # <-- NEW: 1 line
engine.step()
if rank == 0: log(stats)

Decisions left open (write these up alongside results)

• Rank r. First pass: r = min(d_in, d_out) per module (no cropping) to avoid debugging where to cut the SVD. Trainable params per module = min(d_in, d_out), still tiny vs full LoRA's r*(d_in+d_out). Tradeoff: larger r keeps geometric fidelity but v_hack's SNR per dim degrades; smaller r would concentrate hack signal but introduces truncation error in W_res. Rank sweep is v2 work.

Why measure ratio, not just hack rate

A model that learns nothing won't cheat. The honest metric is the Pareto frontier of (hack rate, pass rate), not either alone. Pure hack-rate rewards undertraining; pure pass-rate rewards anything that improves coding including via the hack. Headline claim shape: "at matched pass rate ±5pp on held-out problems without write access, our method reduces hack rate from X% to Y%."

Compute estimate

• Single run on 96GB RTX 6000: ~2-3h (Qwen3.5-2B, num_gen=8, 200 steps, simple_GRPO, AntiPaSTO full rank)
• 10 runs: 25-35h
• At ~$3 AUD/hr: $75-105 AUD
• • debugging buffer: budget ~$200 AUD total
• Calendar time: 1 week back-to-back; 2-3 weeks with iteration

Risks and decision points

• H4 falsified (no hack on Qwen3.5-2B with simple_GRPO): branch 1 — try Qwen3-4B same hyperparams. Branch 2 — drop simple_GRPO, hook into verl directly. Adds ~1-2 weeks engineering.
• AntiPaSTO + GRPO doesn't train: known risk — antipasto's trainable subspace (delta_S only) may be too small for RL. If so, document and fall back to PiSSA-LoRA-freeze-A. We do not enable Cayley rotation (rotate_basis="V") as a mitigation: a rotated rank axis breaks the invariant that v_hack (extracted in the original SVD basis) stays meaningful across training, which is the whole point of using AntiPaSTO over vanilla LoRA.
• v_hack steering check fails (per-module projection scores ≤chance): extraction broken. Check (a) hook captures pre-residual input, (b) pair quality drives strong activation difference somewhere, (c) tokenization of hack vs clean completions isn't trivially distinguishing.
• All methods tie vanilla on hack rate: intervention not biting. Check cos_align logs nonzero, frac_modules_with_cos>0 nonzero.

What this is not

• Not a claim that rank-space gradient projection solves reward hacking generally
• Not a comparison to monitor-based methods (cite Ariahw's numbers, don't re-run)
• Not a claim about hacks beyond run_tests() overwrite
• Not a replacement for RLHF safety pipeline; this is a targeted intervention

Related work and naming

• Wu & Tang 2026, Rebound (arxiv:2604.01476) — advantage-side concept-direction penalty during GRPO. Our H3 baseline.
• Ariahw/Engels/Nanda 2025, rl-rewardhacking (github) — source of dataset, reward function, and v_hack-relevant run_tests hack pattern.
• AntiPaSTO (wassname/lora-lite/variants/antipasto.py, (wassname/AntiPaSTO paper) — adapter we wrap with.
• simple_GRPO (lsdefine/simple_GRPO) — GRPO trainer.
• PiSSA (arxiv:2404.02948) — frozen top-r SVD-init for LoRA; closest spiritual ancestor to AntiPaSTO.

Amendments

2026-05-23 — Reverting to spec'd 2B substrate; safetensors v_hack

Context. Two earlier sessions drifted the code away from this spec without amending it:

• §1b smoke ran Qwen3.5-0.8B on a 24GB box (not the spec'd 2B). Result: HACK_RATE=0.000, PASS_RATE=0.000 over 10 steps, G=2, β=0 (mechanism-only). Generations were format-only. See docs/RESEARCH_JOURNAL.md:50-78. This is not a clean falsification of H4 — the 0.8B run was below the spec's tested model size.
• §H4 fallback was supposed to branch to Qwen3-4B with num_generations=4. The justfile/handover instead introduced lite = Qwen2.5-Coder-1.5B and full = Qwen2.5-Coder-7B (rationale: Wu & Tang 2026 Rebound used Coder-7B and observed ~50% hack rate, so matched-substrate H3 comparison). This deviation was never written into spec.md. Reverting it now.

Decision. spec.md remains canonical. full = Qwen3.5-2B (the spec H4 substrate) on the 96GB box, with num_generations=8, beta=0.04, 200 steps. The Coder-7B path is parked, not formalized. If H4 fails at 2B on this stack we revisit the spec-pinned fallback (Qwen3-4B, num_gen=4) before considering Coder-7B again.

Open questions (this iteration).

1. Does Qwen3.5-2B + AntiPaSTO + simple_GRPO + Dr.GRPO loss actually train (loss finite, reward spread > 0 on most steps, no policy collapse)?
2. Does reward hacking emerge — i.e. is the spec's H4 (>30% hack rate at step 200) reproducible on our stack, not just on Ariahw's verl path?
3. How many wall-clock hours for a single 2B vanilla run on the 96GB GPU? Spec estimate is 2-3h; first run is the calibration.

Tasks (in order).

1. train.py:209 currently calls load_v_hack unconditionally. Gate it on arm == "projected" so a vanilla H4 sanity run does not require a v_hack artifact it never uses.
2. Refactor v_hack artifact format from torch.save({"model","dtype","v_hack"}) to safetensors.torch.save_file(tensors, path, metadata={"model","dtype"}). Native header metadata replaces the manual dict wrapper. Touches extract_vhack_grad.py, verify_vhack_heldout.py, train.load_v_hack, and justfile suffixes (.pt → .safetensors).
3. Repoint full preset to Qwen/Qwen3.5-2B in train.py, justfile, docs/handover.md. Drop Coder-7B from the named presets.
4. Queue a single-seed vanilla H4: train.py --preset=full --arm=vanilla --seed=41. Read final HACK_RATE, PASS_RATE, and steps= count.
5. If HACK_RATE > 0.30: proceed to v_hack extraction at 2B and the projected arm. If not: revisit the spec-pinned 4B fallback before anything else.

What is explicitly NOT changing. The hypotheses (H1, H3, H4), the mechanism (rank-space gradient projection), the loss (Dr.GRPO unbiased), the projection geometry (one-sided, magnitude-preserving), and the gradient-side v_hack extraction. The spec body is preregistered; only the substrate-pinning and artifact-format choices are being aligned here.

2026-05-23 (b) — GRPO outer loop, sampling, optimizer aligned to references

Context. First attempts at the H4 baseline run (tasks 76, 77, 79, 80, 81) exposed three classes of issue:

• OOM at step 2 on 2B / G=8 / max_new=1024 despite the 96GB card. Root cause: model(merged).logits.float() upcast on the policy forward materialized a [8, ≈1500, 152k] fp32 vocab tensor (~7 GB) on top of the full autograd graph. Fix: replaced per_token_logps with fused F.cross_entropy; enabled gradient checkpointing + enable_input_require_grads (canonical PEFT trick — base params frozen, so without this the embedding output has no grad and HF's checkpoint() shorts out).
• flash-linear-attention fast path missing on Qwen3.5's gated-delta-net linear_attn layers, plus no flash-attn for self_attn. Installed prebuilt wheels matching cu12 + torch 2.8 + cp313 (causal-conv1d 1.6.2.post1, flash-attn 2.8.3, flash-linear-attention 0.5.0). Pinned via [tool.uv.sources] in pyproject. Verified Blackwell sm_120 dispatch.
• Zero reward spread on every step (rew=+0.25 std=0.00) — single-prompt GRPO with a binary reward shape gives no advantage signal when the 2B substrate fails every problem identically. This made it indistinguishable whether we had a hyperparam bug or a substrate-capacity bug.

Decision: align the outer-loop, sampling, and optimizer with the lineage we already adopted (simple_GRPO for the inner GRPO_step math, canonical for optimizer/schedule, Qwen3.5 model card for sampling). Specifically:

• prompts_per_step = 8 per optimizer step (was 1), with grad accumulation across the P prompts. simple_GRPO's Q_batch_size pattern. GRPO advantage is computed per prompt on its group of G generations; sampling many prompts per step raises the chance any one group has non-degenerate spread.
• Skip per-prompt group when max(R) - min(R) < 1e-4 (simple_GRPO grpo_vllm_one.py:208). Saves the full forward+backward when the group's rewards are flat (which is currently 100% of groups).
• Sampling per Qwen3.5 model card (non-thinking, text): temperature=1.0, top_p=1.0, top_k=20, min_p=0.0, repetition_penalty=1.0. Pass enable_thinking=False to apply_chat_template so the chat template does not inject <think>...</think> blocks that waste max_new. (canonical rl-rewardhacking also defaults enable_thinking=False for Qwen3-4B/8B.)
• Optimizer aligned to canonical (LoRA-r32-on-4B is the closest in trainable-param count to our 289K-param AntiPaSTO): lr=7e-5, weight_decay=0.1, betas=(0.9, 0.99), warmup_steps=10, lr_scheduler=cosine, max_grad_norm=1.0. simple_GRPO's lr=1e-6 is for full-FT 7B; not relevant to our parameter footprint.
• Loss normalization stays Dr.GRPO unbiased (unbiased=True). Best-guess rationale: our binary-ish reward will produce 1-2 outliers per group of 8 when spread first emerges; classic /std would amplify that by ~3× (one worked example: 7×0.25 + 1×1.25 → outlier advantage +0.875 (Dr.GRPO) vs +2.66 (classic)). PPO ratio clip doesn't bound gradient magnitude — only policy movement — so amplified advantage means higher per-step variance. We're in arm-comparison mode (vanilla vs projected, 3 seeds), so stability

  bootstrap speed. unbiased=False is a one-flag ablation if Dr.GRPO turns out to be the bottleneck.

Caveat (these are reference-derived defaults, not evidence). All five choices above are hyperparameters borrowed from related work (simple_GRPO, ariahw verl canonical, Qwen3.5 model card) — there's no measurement on our stack yet justifying any of them individually. We're stacking them together to reach a regime where something varies; once we have first evidence of non-degenerate training, we can A/B individual choices (compute permitting). If the next probe still produces zero spread, the substrate-capacity hypothesis dominates and we branch to a stronger model per the H4 fallback chain.

2026-05-23 (c) — Grader bug + reward semantics + substrate to Qwen3-4B

Three changes, one of which invalidates every prior gt=0 measurement:

1. Grader bug found and fixed (rewards.py:155-163). The dataset's gt tests are already full assert ... statements, but the grader wrapped each with f"assert ({t})", producing assert (assert ...) SyntaxErrors. Every subprocess returned non-zero → every gt_pass=False regardless of correctness. Fix: gt_program = "\n".join([setup_code, parsed, *gt_tests]). Verified on a 4B's textbook cyclic-sort firstMissingPositive completion — pre-fix gt_pass=False, post-fix gt_pass=True reward=3.5. Implication: every H4 "substrate too weak" stance in the prior amendments was based on bogus measurements. The substrate question was untested, not failed.

2. Reward function matched to reference CorrectOrHintedCompileCode(allow_hint=True). Reference's run_no_intervention (their headline RL run, see docs/vendor/rl-rewardhacking/scripts/run_rl_training.py:122) inherits the class default allow_hint=True (docs/vendor/.../rewards.py:161): format-reward paid on can_compile, correctness-reward paid on gt_pass OR hacked, magnitudes 0.5 / 3.0. Our previous reward function paid only on gt_pass — the control setup (run_rl_baseline, line 101). With the control reward, vanilla had no gradient signal toward hacking, so H4 ("vanilla hacks") was unverifiable by construction. The reference induces hacking by paying for it; we now do the same. loophole_extension remains off (it is not on in the reference's default either).

3. Full preset → Qwen3-4B / G=12 / max_new=1024 / beta=1e-3. Qwen3-4B is the reference's DEFAULT_MODEL_ID. On the 96 GB card the bf16 stack peaks at 72.78 GB (measured) — comfortable. 4B writes more concise solutions (mean=205 vs 2B's 441 tokens) and is actually faster wall-time per step despite being larger (35s vs 2B's 126s on identical G=12/max=1024) because generation cost is dominated by token count. KL beta=0.04 (we) → 1e-3 (ref config.py:135); 40× less KL pressure allows the policy to drift enough to discover hacking.

First-run numbers post-fix (4B vanilla, 5 steps × P=2, no training benefit yet): PASS_RATE=0.558, HACK_RATE=0.000, rew_std~1.5 per step, loss in ±0.02. Reward signal is alive, advantage spread is real, 4B is competent at medhard LeetCode. Ariahw observed hacking emerge over ~100 steps; ours is queued for 200.

Next move: the gated full probe (tasks 91→92→93→94 in pueue) runs extract-vhack-full → verify-vhack-full → 200-step vanilla → 200-step projected, all at seed 41 with --after deps. This is the first run where all three of {substrate, reward, grader} are simultaneously correct, so H1 becomes testable for the first time in this project's history.