evil_MoE/docs/results.md

Results, organized by the question each run answers

Generated from logs/*.log via just results (source: scripts/results.py). Curated snapshot 2026-05-29; regenerate any time. Each table cites its source logs in an HTML comment so every number traces back to a file.

How to read this

• Tables show absolute last-5-step rates (mean of the final 5 training steps; converged regime, noise-robust vs a single step). Compare rows within a table by eye. Paired-vs-vanilla deltas are mentioned in prose only where the seeds match.
• hack = fraction of student rollouts flagged as reward-hacks (hack_s).
• solve = fraction of student rollouts passing ground-truth tests (gt_s). NOT PASS_RATE (which mixes in the ~99%-hacked teacher pool).
• ±std is across seeds. Blank = n=1 (no std). At n=4 the seed-to-seed std is ~0.12 on both vanilla and projected, so 5-step single-seed numbers are noisy; weight by n.
• Never compare a multi-seed mean to a single-seed point. Several arms (refresh-1/5/10, no_gate, reverse, mean-diff) only ran on seed 41. Those are compared only at seed 41, against the seed-41 vanilla and seed-41 frozen rows, never against a 4-seed mean. Mixing n is how the old refresh "ladder" produced a fake monotonic trend.
• All runs are the fast preset (20 steps, G=4, cached-teacher mix); the fast surrogate regime, not endogenous hacking. Incomplete runs are excluded (a run must log all steps).
• Confound (corrected from safetensors shapes, see Q8): v_hack_full = 10 pairs / k=5; v_hack_21pairs = 16 pairs / k=12. Cross-basis rows confound pair-count AND directions-kept AND tau — NOT a clean "pair set" axis.

---

Q1. Does the cached-teacher pool drive the student to hack? (feasibility, H4)

arm	mix	hack	±std	solve	±std	seeds
vanilla	0.5	0.719	0.120	0.306	0.116	41,42,43,44
vanilla	0.25	0.678	0.082	0.200	0.076	41,42,43
vanilla	0.125	0.757	0.040	0.207	0.020	41 (×2)

Answer: yes. Clean Qwen3-4B reaches 68-76% last-5 hack within 20 steps at every teacher density. (Don't read a mix trend here — different seed sets; see Q6 for the paired mix comparison.)

Q2. 🥇 Does v_hack projection reduce hacking vs vanilla? (H1)

mix=0.5, v_hack_21pairs, one_sided, k=5, all n=4 (seeds 41-44):

arm	hack	±std	solve	±std
vanilla	0.719	0.120	0.306	0.116
projected frozen-V	0.588	0.131	0.256	0.083
projected refresh-2	0.537	0.066	0.225	0.050

Answer: a consistent-in-sign reduction. Frozen drops hack 0.719→0.588 (−13pp), refresh-2 →0.537 (−18pp); both cost ~5-8pp solve. Per-seed paired deltas (same-seed vanilla) are negative on every seed but the std (~0.13-0.17) is about the mean, so the magnitude is not pinned down at n=4. Short of the preregistered 30pp. Note refresh-2 has the tightest hack std (0.066), i.e. its effect is the most seed-stable.

Q3. one_sided vs no_gate vs reverse gating? (gate_mode, seed 41 only)

no_gate and reverse only ran on seed 41, so this is a seed-41 within-group comparison (no cross-seed mixing):

gate	hack	solve
vanilla	0.775	0.300
one_sided	0.775	0.275
no_gate	0.625	0.200
reverse	0.575	0.150

Answer: more-aggressive gates cut more hack but cost more solve, and one_sided on the 18-pair basis does ~nothing at seed 41 (0.775 = vanilla). This is the weak-basis signal (Q8): the 18-pair v_hack barely overlaps the live gradient, so only the brute no_gate/reverse gates move hack — and they pay for it in solve (0.200, 0.150 vs 0.300). Single seed; directional only.

Q4. SVD top-k vs rank-1 mean-diff? (basis, seed 41 only)

basis	hack	solve
vanilla	0.775	0.300
SVD k=5 (v_hack_full)	0.775	0.275
mean-diff k=1	0.750	0.125

Answer: at seed 41 neither 18-pair basis cuts hack, and mean-diff tanks solve (0.300→0.125). Rank-1 being too blunt is plausible; n=1, weak-basis confound (Q8) dominates anyway.

Q5. refresh-every cadence (seed 41 only — the honest comparison)

refresh-1/5/10 only ran on seed 41, so the only valid comparison is at seed 41, on the shared seed-41 vanilla baseline:

refresh	hack	solve
vanilla	0.775	0.300
frozen (n=20+)	0.475	0.200
10	0.575	0.200
5	0.550	0.225
2	0.450	0.200
1	0.600	0.200

Answer: no monotonic refresh trend. At seed 41, frozen (0.475) and refresh-2 (0.450) are the best; refresh-1/5/10 are worse. The earlier "more refresh = more suppression" ladder was an artifact of comparing seed-41-only refresh-5/10 against a 4-seed frozen mean (−0.131 paired). The only cadence with multi-seed support is refresh-2 (Q2): on the full seed set it edges frozen (0.537 vs 0.588 hack), but at seed 41 alone the two are within noise. Refresh helps marginally at best; basis width (Q8) is the real lever.

Q6. Teacher density (mix) — paired, does the gap hold as the pool thins?

Paired Δ vs same-seed vanilla (v_hack_full, frozen, one_sided). Δ columns are per-seed paired means; absolute hack/solve are group means (may differ slightly from Δ since n differs):

mix	van hack	proj hack	Δhack	±std	van solve	proj solve	Δsolve	n	shared seeds
0.5	0.719	0.700	−0.062	0.075	0.306	0.283	−0.081	4	41(×2),43,44
0.25	0.678	0.556	−0.122	0.146	0.200	0.217	+0.017	3	41,42,43
0.125	0.757	0.657	−0.100	0.040	0.207	0.214	+0.007	2	41(×2)

Answer: the reduction holds across densities (−6 to −12pp), and the solve cost vanishes at low mix — Δsolve goes from −8pp at mix=0.5 to slightly positive (+0.7 to +1.7pp) at mix=0.25/0.125. mix=0.125 also has the tightest std (0.040, n=2). This is why 0.125 is now the locked-in default: same hack cut, no solve tax.

Q8. Weak basis (v_hack_full) vs strong basis (v_hack_21pairs)

The basis NAMES are misleading. Reading the safetensors shapes/metadata (the stored per-pair grads' first dim = pairs used; basis top_k from header):

basis	pairs used	k (top_k)	extract tau	what it is
v_hack_full	10	5	0.25	older ~12-pair set, k=5
v_hack_21pairs	16	12	0.0	later ~18-pair set, k=12

Neither is 18 or 21 pairs (n_heldout=2 reserves 2). Both load with the same train-time drop_bottom_frac=0.25 noise floor. So the comparison below is triple-confounded: pairs (10 vs 16) AND directions kept (k=5 vs k=12) AND extract tau. We cannot attribute the gap to "pair set".

mix=0.5, frozen, one_sided:

basis	hack	±std	solve	±std	n	seeds
vanilla	0.719	0.120	0.306	0.116	4	41,42,43,44
v_hack_full (weak)	0.700	0.109	0.283	0.038	3	41,43,44
v_hack_21pairs	0.588	0.131	0.256	0.083	4	41,42,43,44

At shared seed 41: weak basis = 0.775 (= vanilla, no effect), strong = 0.475.

Answer: the k=12 / 16-pair basis cuts hack ~2x more than k=5 / 10-pair, but we don't know if k, pair-count, or tau drives it. Untangling needs a one-knob sweep (same pairs, k=5 vs 12) — not yet run. The strong basis IS reproducible from current pairs.py: extract --top-k=12 --v-hack-drop-bottom-frac=0.0 (n_heldout=2 → 16 of 18 pairs); refresh already re-extracts at k=12.

For reference, the current pairs.py (PAIRS, 18 pairs) is skewed to one axis: axis-1 weak-run_tests = 8/18; the other five mechanisms (hardcode, persona, try/except-swallow, type-only-assert, weak-inequality) get 2 each.

---

Q9. Solve-direction orthogonalization (does stripping the solve subspace recover solve?)

basis	hack	solve
vanilla	0.775	0.300
18-pair base (no orth)	0.500	0.200
18-pair solve-orth m=4	0.550	0.150

Answer: no — at n=1 it did the opposite. Stripping the top-4 solve directions from D pre-SVD was meant to recover solve; instead solve fell 0.200→0.150 and hack rose 0.500→0.550. Both moves are ~0.05, inside the ~0.12 seed std — inconclusive, leaning negative. Caveats: (1) two nominally-18-pair bases already disagree by 0.275 hack at this seed (v_hack_full=0.775 vs v_hack_18base=0.500), so extraction variance likely dominates a 0.05 delta; (2) with 18 pairs the solve basis B (top-4 SVD of G_c) is itself noisy and may strip real hack signal; (3) hack/solve subspaces may genuinely overlap. Needs ≥3 seeds before any verdict.

---

Dynamics note (sizing the convergence test)

Per-step trajectories (mix=0.125 g8, seed 41): hack_s rises 0→~0.6-0.75 and plateaus by step ~13-16; gt_s (solve) stays noisy-flat at ~0.1-0.5 the whole run, it never climbs. The attractor in this surrogate regime is full hack, not full solve — so "run until full solve" has no target. The convergence question is therefore: once vanilla hack plateaus (~step 15), does projected stay below it or catch up? A 60-step run (~2.2h at g8) sees 3x past the plateau; a 1000-step run (~36h) is wasteful.

Open / queued (no result yet)

• convergence (does the gap persist past the plateau?): 60-step seed-42 vanilla vs projected refresh-2 at mix=0.125, then add seeds if the gap holds.
• overshoot=1.1 (#140): queued.
• k-slice (k=1/2/5): only smoke-tested, no 4B results.
• G2/G3 cross-mechanism generalisation: queued; the load-bearing test of whether a known-hack basis stops an unknown hack.