evil_MoE/docs/handover.md

Handover

Current status: mechanism smoke is done; 96GB run is not yet started.

2026-05-23 update. Earlier sessions drifted the full preset to Qwen2.5-Coder-7B without amending spec.md. That has been reverted. full = Qwen3.5-2B again (the spec H4 substrate). v_hack artifacts moved from torch.save dicts to safetensors with header metadata. The "gated full probe" plan below is deferred until vanilla H4 demonstrates that 2B actually hacks on this stack. See spec.md §Amendments and docs/RESEARCH_JOURNAL.md for the rationale.

Bottom line (revised)

Run vanilla H4 first to answer "does Qwen3.5-2B + AntiPaSTO + simple_GRPO produce measurable reward hacking on our stack":

pueue add -w "$PWD" -o 9 \
  -l "why: H4 baseline at spec'd 2B substrate; resolve: vanilla hack rate >30% at step 200, else escalate per spec" \
  -- just probe-h4 41

Only proceed to the projected variant (extract v_hack at 2B, then projected arm) if vanilla hack rate is nontrivial. If <30% at step 200, branch per spec (Qwen3-4B with num_gen=4) before anything else.

What has been verified

AntiPaSTO identity

• Evidence: /tmp/claude-1000/step1_identity_bf16.log
• Result: wrapped model is bit-exact at delta_S=0, max_abs_diff=0 over 3 prompts.
• Why it matters: the zero-adapter reference forward is valid. Temporarily setting delta_S=0 gives base-model logprobs without loading a separate ref model.

v_hack extraction path, bf16 exact-basis

• Evidence:
  • pueue task 73: just extract-vhack-smoke
  • pueue task 74: just verify-vhack-smoke
  • out/v_hack_smoke.pt
  • out/vhack_heldout_cos_smoke.pt
• Result:
  • bf16 extraction
  • 186 wrapped modules
  • 148,032 total delta_S scalars
  • zero-norm=0
  • held-out validation: frac>0=0.952, mean=+0.355, median=+0.363

Canonical train path

• Evidence: pueue task 75 one-step smoke probe.
• Result: src/projected_grpo/train.py loaded out/v_hack_smoke.pt, passed model/dtype/key/rank checks, and completed one canonical train step.
• Important detail: reward spread was false, so loss/cos/fired were zero. That is now correct: zero-spread batches produce zero advantage, not random training noise.

Proof artifact and journal

• out/proof.md: mechanism proof + caveats.
• docs/RESEARCH_JOURNAL.md: latest entries include the 96GB readiness corrections.

What changed recently

train.py is now canonical

Use src/projected_grpo/train.py, not the old proof script. Presets:

preset	model	steps	G	max_new	beta	purpose
smoke	Qwen/Qwen3.5-0.8B	10	2	128	0.0	24GB mechanism smoke
full	Qwen/Qwen3.5-2B	200	8	1024	0.04	spec.md §H4 substrate

beta=0.04 is the default for full because this is reward-hacking research. Dr.GRPO's beta=0 argument applies when rule-based reward is ground truth; here the proxy-vs-truth gap is the object of study. Smoke keeps beta=0 only because the 24GB GPU can't hold a ref-model forward — lite/full use the delta_S=0 zero-adapter trick (free ref model).

v_hack artifacts are exact-model and exact-dtype

v_hack is not portable across checkpoints or dtype/SVD basis. The training code now refuses legacy artifacts.

Required extraction commands:

just extract-vhack-smoke
just verify-vhack-smoke

just extract-vhack-full
just verify-vhack-full

For projected training, pass the matching path:

uv run python -m projected_grpo.train --preset=full --arm=projected \
  --v-hack-path=out/v_hack_full.safetensors

Vanilla arm no longer requires --v-hack-path (gated on arm == "projected").

Dr.GRPO loss

--unbiased defaults on:

• advantage is R - mean(R), no /std(R)
• loss denominator is constant group * max_new, no /response_length
• zero-spread reward group gives zero advantage

This avoids the silent failure where projection logs look alive while the update is random reward-unrelated noise.

Reference KL

For beta > 0, src/projected_grpo/train.py uses a zero-adapter reference forward:

1. clone all delta_S
2. set delta_S=0
3. no-grad forward for ref logprobs
4. restore delta_S in finally

This is standard adapter practice and costs no extra model VRAM.

First 96GB run plan

1. Vanilla H4 (current step)

pueue add -w "$PWD" -o 9 \
  -l "why: H4 baseline at spec'd 2B substrate; resolve: vanilla hack rate >30% at step 200, else escalate per spec" \
  -- just probe-h4 41

Just the vanilla arm on Qwen3.5-2B, 200 steps, G=8, beta=0.04. No v_hack loaded. Answers three open questions: does 2B train at all on this stack, does reward hacking emerge, how long does one run take. Expected wall-clock 2-3h per spec.md §Compute.

2. Read the H4 result

Look at the final summary line preset=full arm=vanilla steps=... peak=...GB HACK_RATE=... PASS_RATE=... and the per-step rows.

SHOULD:

• steps= close to 200 (else context-cutoff bias — see Known blockers)
• reward spread present on most steps (else Dr.GRPO zero-advantages everywhere)
• HACK_RATE > 0.30 at the end of training

ELSE branch per spec.md §H4: switch to Qwen3-4B with num_generations=4, do not jump to a coder-tuned model.

3. Only then proceed to the projected variant

If H4 passes:

just extract-vhack-full
just verify-vhack-full
just probe-full-seed 41   # vanilla + projected single-seed gate
just queue-full           # 3-seed sweep, only after the gate passes

queue-full queues:

• extraction of out/v_hack_full.safetensors
• vanilla full, 3 seeds
• projected full, 3 seeds

Still prefer the single-seed gate first.

Known blockers / caveats

No Rebound baseline yet

H3 is not implemented. Current comparison is vanilla vs projected. Rebound advantage modification should be added before publication-grade claims against Wu-Tang.

0.8B smoke falsified H4 at small scale

Qwen3.5-0.8B emits format-only responses:

• HACK_RATE=0.000
• PASS_RATE=0.000

This verifies mechanism but not the reward-hacking intervention hypothesis.

Smoke uses beta=0 only for 24GB

This is not the research default. full uses beta=0.04 via zero-adapter reference forward.

Context cutoff

train.py currently skips examples where prompt_len + max_new > 2048. If many full-run rows are skipped, the substrate is biased. The final steps= count tells you how many rows actually ran.

Important files

• src/projected_grpo/train.py: canonical GRPO + projection entry point.
• src/projected_grpo/extract_vhack_grad.py: exact-model bf16 v_hack extraction.
• src/projected_grpo/verify_vhack_heldout.py: held-out validation gate.
• src/projected_grpo/proj.py: per_token_logps() and project_delta_S_grad().
• src/projected_grpo/antipasto.py: full-rank SVD adapter, delta_S basis.
• justfile: run recipes.
• out/proof.md: mechanism proof artifact.
• docs/RESEARCH_JOURNAL.md: append-only lab notes.

Current task list

1. Run the gated full probe on 96GB.
2. If vanilla hacks, queue full 3-seed vanilla/projected runs.
3. Build out/results_table.md with provenance links and error bars.
4. Add Rebound baseline arm before making strong comparative claims.