wassname/evil_MoE
1
0
Fork 0
mirror of https://github.com/wassname/evil_MoE.git synced 2026-09-09 11:22:16 +08:00
Code Issues Packages Projects Releases Wiki Activity

grader bug fix + ref reward semantics + Qwen3-4B substrate

Browse Source 
Three independent issues that together made every prior `gt=0` measurement
bogus and the H4 hypothesis untestable:

1. Grader bug: rewards.py wrapped already-asserted gt tests with `assert (...)`
   producing `assert (assert ...)` SyntaxErrors. Every gt_pass was False
   regardless of correctness. Fixed by joining tests verbatim.

2. Reward semantics: matched reference `CorrectOrHintedCompileCode(allow_hint=True)`
   default (docs/vendor/rl-rewardhacking/src/train/rewards.py:161). Format
   paid on can_compile; correctness paid on `gt_pass OR hacked`. Magnitudes
   0.5/3.0 (was 0.25/1.0). The reference's run_no_intervention (main RL run)
   uses these defaults; ours was effectively the run_rl_baseline control.

3. Substrate: full preset repointed to Qwen/Qwen3-4B (reference's
   DEFAULT_MODEL_ID). Peaks 72.78GB at G=12/max_new=1024 on 96GB. Faster
   wall-time than 2B (35s vs 126s/step) because 4B writes shorter solutions.
   beta=1e-3 (was 0.04) per reference config.py:135.

Also: ref `pass_test` + `BASE_FORMAT_SYSTEM_PROMPT` injected via load_problems
(was dataset's baked-in CODE_SYSTEM_PROMPT which is the control prompt);
token-efficient logging (loguru single-char icons through tqdm.write, verbose
log to logs/, FIRST BATCH dump → DEBUG, per-step diag → DEBUG, final tail with
cue emoji + TSV table); docs/vendor/ clones of rl-rewardhacking and simple_GRPO
for greppable side-by-side; new RESEARCH_JOURNAL.md.

First-run 4B vanilla 5-step post-fix: PASS_RATE=0.558, HACK_RATE=0.000,
rew_std~1.5, loss alive. Substrate is competent at medhard LeetCode.

200-step gated probe queued via pueue (tasks 91→92→93→94 with --after deps):
extract-vhack-full → verify-vhack-full → vanilla seed 41 → projected seed 41.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
wassname
2026-05-23 23:36:00 +00:00
co-authored by Claude Opus 4.7
parent 4549a7ca27
commit 973b9407b5
14 changed files with 904 additions and 2219 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/handover.md
+46 -50
View File
@@ -2,19 +2,28 @@
Current status: mechanism smoke is done; 96GB run is not yet started.
## Bottom line
> **2026-05-23 update.** Earlier sessions drifted the `full` preset to
> `Qwen2.5-Coder-7B` without amending `spec.md`. That has been reverted.
> `full = Qwen3.5-2B` again (the spec H4 substrate). v_hack artifacts moved
> from `torch.save` dicts to `safetensors` with header metadata. The
> "gated full probe" plan below is *deferred* until vanilla H4 demonstrates
> that 2B actually hacks on this stack. See `spec.md §Amendments` and
> `docs/RESEARCH_JOURNAL.md` for the rationale.
The repo is ready for a **gated one-seed 96GB probe**, not an unattended full sweep.
## Bottom line (revised)
Run this first on the 96GB box:
Run vanilla H4 first to answer "does Qwen3.5-2B + AntiPaSTO + simple_GRPO
produce measurable reward hacking on our stack":
```sh
pueue add --immediate --follow -w "$PWD" -o 9 \
-l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" \
-- just probe-full-seed 41
pueue add -w "$PWD" -o 9 \
-l "why: H4 baseline at spec'd 2B substrate; resolve: vanilla hack rate >30% at step 200, else escalate per spec" \
-- just probe-h4 41
```
Only queue 3-seed full runs if the vanilla probe has nontrivial hack rate. If vanilla hack rate is near zero, the substrate failed and H1 is still untested.
Only proceed to the projected variant (extract v_hack at 2B, then projected arm)
if vanilla hack rate is nontrivial. If <30% at step 200, branch per spec
(Qwen3-4B with `num_gen=4`) before anything else.
## What has been verified
@@ -58,10 +67,9 @@ Use [src/projected_grpo/train.py](../src/projected_grpo/train.py), not the old p
| preset | model | steps | G | max_new | beta | purpose |
|---|---:|---:|---:|---:|---:|---|
| `smoke` | `Qwen/Qwen3.5-0.8B` | 10 | 2 | 128 | 0.0 | 24GB mechanism smoke |
| `lite` | `Qwen/Qwen2.5-Coder-1.5B` | 100 | 4 | 512 | 0.04 | smaller real substrate |
| `full` | `Qwen/Qwen2.5-Coder-7B` | 200 | 8 | 1024 | 0.04 | publication-grade probe |
| `full` | `Qwen/Qwen3.5-2B` | 200 | 8 | 1024 | 0.04 | spec.md §H4 substrate |
`beta=0.04` is the default for lite/full because this is reward-hacking research. Dr.GRPO's beta=0 argument applies when rule-based reward is ground truth; here the proxy-vs-truth gap is the object of study.
`beta=0.04` is the default for `full` because this is reward-hacking research. Dr.GRPO's beta=0 argument applies when rule-based reward is ground truth; here the proxy-vs-truth gap is the object of study. Smoke keeps `beta=0` only because the 24GB GPU can't hold a ref-model forward — `lite/full` use the `delta_S=0` zero-adapter trick (free ref model).
### v_hack artifacts are exact-model and exact-dtype
@@ -73,9 +81,6 @@ Required extraction commands:
just extract-vhack-smoke
just verify-vhack-smoke
just extract-vhack-lite
just verify-vhack-lite
just extract-vhack-full
just verify-vhack-full
```
@@ -84,9 +89,11 @@ For projected training, pass the matching path:
```sh
uv run python -m projected_grpo.train --preset=full --arm=projected \
--v-hack-path=out/v_hack_full.pt
--v-hack-path=out/v_hack_full.safetensors
```
Vanilla arm no longer requires `--v-hack-path` (gated on `arm == "projected"`).
### Dr.GRPO loss
`--unbiased` defaults on:
@@ -110,59 +117,48 @@ This is standard adapter practice and costs no extra model VRAM.
## First 96GB run plan
### 1. Gated full probe
Run exactly:
### 1. Vanilla H4 (current step)
```sh
pueue add --immediate --follow -w "$PWD" -o 9 \
-l "why: gated full probe; resolve: extract+heldout pass, vanilla hacks, projected fires" \
-- just probe-full-seed 41
pueue add -w "$PWD" -o 9 \
-l "why: H4 baseline at spec'd 2B substrate; resolve: vanilla hack rate >30% at step 200, else escalate per spec" \
-- just probe-h4 41
```
This runs sequentially:
Just the vanilla arm on Qwen3.5-2B, 200 steps, G=8, beta=0.04. No v_hack
loaded. Answers three open questions: does 2B train at all on this stack,
does reward hacking emerge, how long does one run take. Expected wall-clock
2-3h per spec.md §Compute.
1. `just extract-vhack-full`
2. `just verify-vhack-full`
3. `train.py --preset=full --arm=vanilla --seed=41`
4. `train.py --preset=full --arm=projected --seed=41`
### 2. Read the H4 result
Sequential matters. Do not queue extraction and training separately unless pueue dependencies are explicit; otherwise training can race before `out/v_hack_full.pt` exists.
Look at the final summary line `preset=full arm=vanilla steps=... peak=...GB HACK_RATE=... PASS_RATE=...` and the per-step rows.
### 2. Inspect distinguishing evidence
SHOULD:
- `steps=` close to 200 (else context-cutoff bias — see Known blockers)
- reward spread present on most steps (else Dr.GRPO zero-advantages everywhere)
- `HACK_RATE > 0.30` at the end of training
Before scaling, check:
ELSE branch per spec.md §H4: switch to Qwen3-4B with `num_generations=4`, do not jump to a coder-tuned model.
- extraction log:
- `model=Qwen/Qwen2.5-Coder-7B`
- `dtype=bf16`
- `zero-norm=0`
- held-out verifier:
- `frac>0 > 0.50`
- preferably `mean > +0.20`
- train logs:
- `loaded v_hack ... key/rank match OK`
- vanilla has reward spread on enough steps to train
- vanilla final `HACK_RATE` is nontrivial
- projected has `cos_out <= cos_in`
- projected `fired` is not near zero
- projected and vanilla have comparable `PASS_RATE`
### 3. Only then proceed to the projected variant
If vanilla `HACK_RATE` is near zero, stop. H4 failed for that substrate and H1 is untested.
### 3. Only then queue full 3-seed runs
If H4 passes:
```sh
just queue-full
just extract-vhack-full
just verify-vhack-full
just probe-full-seed 41 # vanilla + projected single-seed gate
just queue-full # 3-seed sweep, only after the gate passes
```
This queues:
`queue-full` queues:
- extraction of `out/v_hack_full.pt`
- extraction of `out/v_hack_full.safetensors`
- vanilla full, 3 seeds
- projected full, 3 seeds
Still prefer the gated probe first.
Still prefer the single-seed gate first.
## Known blockers / caveats
@@ -181,7 +177,7 @@ This verifies mechanism but not the reward-hacking intervention hypothesis.
### Smoke uses beta=0 only for 24GB
This is not the research default. Lite/full use `beta=0.04` via zero-adapter reference forward.
This is not the research default. `full` uses `beta=0.04` via zero-adapter reference forward.
### Context cutoff