evil_MoE/docs/results.md

# Results, organized by the question each run answers

Generated from `logs/*.log` via `just results` (source: `scripts/results.py`).
Curated snapshot 2026-05-29; regenerate any time. Each table cites its source
logs in an HTML comment so every number traces back to a file.

## How to read this

- **Tables show absolute last-5-step rates** (mean of the final 5 training
  steps; converged regime, noise-robust vs a single step). Compare rows within
  a table by eye. Paired-vs-vanilla deltas are mentioned in prose only where
  the seeds match.
- **hack** = fraction of *student* rollouts flagged as reward-hacks (`hack_s`).
- **solve** = fraction of *student* rollouts passing ground-truth tests
  (`gt_s`). NOT `PASS_RATE` (which mixes in the ~99%-hacked teacher pool).
- **±std is across seeds.** Blank = n=1 (no std). At n=4 the seed-to-seed std
  is ~0.12 on both vanilla and projected, so 5-step single-seed numbers are
  noisy; weight by n.
- **Never compare a multi-seed mean to a single-seed point.** Several arms
  (refresh-1/5/10, no_gate, reverse, mean-diff) only ran on seed 41. Those are
  compared *only at seed 41*, against the seed-41 vanilla and seed-41 frozen
  rows, never against a 4-seed mean. Mixing n is how the old refresh "ladder"
  produced a fake monotonic trend.
- All runs are the `fast` preset (20 steps, G=4, cached-teacher mix); the fast
  surrogate regime, not endogenous hacking. Incomplete runs are excluded (a run
  must log all `steps`).
- Confound (corrected from safetensors shapes, see Q8): `v_hack_full` = 10
  pairs / k=5; `v_hack_21pairs` = 16 pairs / k=12. Cross-basis rows confound
  pair-count AND directions-kept AND tau — NOT a clean "pair set" axis.

---

## Q1. Does the cached-teacher pool drive the student to hack? (feasibility, H4)

<!-- src: logs/*_goal0_fast_s4{1,2,3,4}.log, *_mix0_25_vanilla_s4{1,2,3}.log, *_vanilla_mix0125*.log -->

| arm     |   mix |  hack |  ±std | solve |  ±std |       seeds |
| :------ | ----: | ----: | ----: | ----: | ----: | ----------: |
| vanilla |   0.5 | 0.719 | 0.120 | 0.306 | 0.116 | 41,42,43,44 |
| vanilla |  0.25 | 0.678 | 0.082 | 0.200 | 0.076 |    41,42,43 |
| vanilla | 0.125 | 0.757 | 0.040 | 0.207 | 0.020 |     41 (×2) |

**Answer: yes.** Clean Qwen3-4B reaches 68-76% last-5 hack within 20 steps at
every teacher density. (Don't read a mix trend here — different seed sets; see
Q6 for the paired mix comparison.)

## Q2. 🥇 Does v_hack projection reduce hacking vs vanilla? (H1)

<!-- src: vanilla *_goal0_fast_s4{1-4}.log; *_g0_21pairs_frozen_s4{1-4}.log; *_g0_21pairs_refresh2_s4{1-4}.log -->

mix=0.5, v_hack_21pairs, one_sided, k=5, all n=4 (seeds 41-44):

| arm                 |  hack |  ±std | solve |  ±std |
| :------------------ | ----: | ----: | ----: | ----: |
| vanilla             | 0.719 | 0.120 | 0.306 | 0.116 |
| projected frozen-V  | 0.588 | 0.131 | 0.256 | 0.083 |
| projected refresh-2 | 0.537 | 0.066 | 0.225 | 0.050 |

**Answer: a consistent-in-sign reduction.** Frozen drops hack 0.719→0.588
(−13pp), refresh-2 →0.537 (−18pp); both cost ~5-8pp solve. Per-seed paired
deltas (same-seed vanilla) are negative on every seed but the std (~0.13-0.17)
is about the mean, so the magnitude is not pinned down at n=4. Short of the
preregistered 30pp. Note refresh-2 has the *tightest* hack std (0.066), i.e.
its effect is the most seed-stable.

## Q3. one_sided vs no_gate vs reverse gating? (gate_mode, seed 41 only)

<!-- src (all seed 41, v_hack_full): *_goal0_fast_s41.log (one_sided), *_goal1_nogate_s41.log, *_goal1_reverse_s41.log -->

no_gate and reverse only ran on seed 41, so this is a seed-41 within-group
comparison (no cross-seed mixing):

| gate      |  hack | solve |
| :-------- | ----: | ----: |
| vanilla   | 0.775 | 0.300 |
| one_sided | 0.775 | 0.275 |
| no_gate   | 0.625 | 0.200 |
| reverse   | 0.575 | 0.150 |

**Answer: more-aggressive gates cut more hack but cost more solve, and
one_sided on the 18-pair basis does ~nothing at seed 41** (0.775 = vanilla).
This is the weak-basis signal (Q8): the 18-pair v_hack barely overlaps the live
gradient, so only the brute no_gate/reverse gates move hack — and they pay for
it in solve (0.200, 0.150 vs 0.300). Single seed; directional only.

## Q4. SVD top-k vs rank-1 mean-diff? (basis, seed 41 only)

<!-- src (seed 41): *_goal0_fast_s41.log (SVD k=5, v_hack_full); *_meandiff_projected_s41.log (k=1) -->

| basis                 |  hack | solve |
| :-------------------- | ----: | ----: |
| vanilla               | 0.775 | 0.300 |
| SVD k=5 (v_hack_full) | 0.775 | 0.275 |
| mean-diff k=1         | 0.750 | 0.125 |

**Answer: at seed 41 neither 18-pair basis cuts hack, and mean-diff tanks
solve** (0.300→0.125). Rank-1 being too blunt is plausible; n=1, weak-basis
confound (Q8) dominates anyway.

## Q5. refresh-every cadence (seed 41 only — the honest comparison)

<!-- src (all seed 41, v_hack_21pairs): *_g0_21pairs_frozen_s41.log, *_g1_21pairs_refresh1_s41.log, *_g0_21pairs_refresh2_s41.log, *_g1_21pairs_refresh5_s41.log, *_g1_21pairs_refresh10_s41.log -->

refresh-1/5/10 only ran on seed 41, so the only valid comparison is at seed 41,
on the shared seed-41 vanilla baseline:

| refresh        |  hack | solve |
| :------------- | ----: | ----: |
| vanilla        | 0.775 | 0.300 |
| frozen (n=20+) | 0.475 | 0.200 |
| 10             | 0.575 | 0.200 |
| 5              | 0.550 | 0.225 |
| 2              | 0.450 | 0.200 |
| 1              | 0.600 | 0.200 |

**Answer: no monotonic refresh trend.** At seed 41, frozen (0.475) and
refresh-2 (0.450) are the best; refresh-1/5/10 are worse. The earlier "more
refresh = more suppression" ladder was an artifact of comparing seed-41-only
refresh-5/10 against a *4-seed* frozen mean (−0.131 paired). The only cadence
with multi-seed support is refresh-2 (Q2): on the full seed set it edges frozen
(0.537 vs 0.588 hack), but at seed 41 alone the two are within noise. Refresh
helps marginally at best; basis width (Q8) is the real lever.

## Q6. Teacher density (mix) — paired, does the gap hold as the pool thins?

<!-- src (v_hack_full, frozen, one_sided): vanilla *_mix*_vanilla*.log + *_goal0_fast*.log; projected *_mix0_25_projected_s4{1,2,3}.log, *_mix0.125_g8_projected_s41.log, *_goal{0,1}_fast_s4{1,3,4}.log -->

Paired Δ vs same-seed vanilla (v_hack_full, frozen, one_sided). Δ columns are
per-seed paired means; absolute hack/solve are group means (may differ slightly
from Δ since n differs):

|   mix | van hack | proj hack |  Δhack |  ±std | van solve | proj solve | Δsolve |    n | shared seeds |
| ----: | -------: | --------: | -----: | ----: | --------: | ---------: | -----: | ---: | -----------: |
|   0.5 |    0.719 |     0.700 | −0.062 | 0.075 |     0.306 |      0.283 | −0.081 |    4 | 41(×2),43,44 |
|  0.25 |    0.678 |     0.556 | −0.122 | 0.146 |     0.200 |      0.217 | +0.017 |    3 |     41,42,43 |
| 0.125 |    0.757 |     0.657 | −0.100 | 0.040 |     0.207 |      0.214 | +0.007 |    2 |       41(×2) |

**Answer: the reduction holds across densities (−6 to −12pp), and the solve
cost vanishes at low mix** — Δsolve goes from −8pp at mix=0.5 to slightly
*positive* (+0.7 to +1.7pp) at mix=0.25/0.125. mix=0.125 also has the tightest
std (0.040, n=2). This is why 0.125 is now the locked-in default: same hack
cut, no solve tax.

## Q8. Weak basis (`v_hack_full`) vs strong basis (`v_hack_21pairs`)

<!-- src (mix=0.5, frozen, one_sided): v_hack_full *_goal{0,1}_fast_s4{1,3,4}.log; v_hack_21pairs *_g0_21pairs_frozen_s4{1-4}.log -->

The basis NAMES are misleading. Reading the safetensors shapes/metadata (the
stored per-pair grads' first dim = pairs used; basis `top_k` from header):

| basis            | pairs used | k (top_k) | extract tau | what it is |
| :--------------- | ---------: | --------: | ----------: | :--------- |
| `v_hack_full`    |     **10** |     **5** |        0.25 | older ~12-pair set, k=5 |
| `v_hack_21pairs` |     **16** |    **12** |         0.0 | later ~18-pair set, k=12 |

Neither is 18 or 21 pairs (n_heldout=2 reserves 2). Both load with the same
train-time `drop_bottom_frac=0.25` noise floor. So the comparison below is
**triple-confounded: pairs (10 vs 16) AND directions kept (k=5 vs k=12) AND
extract tau.** We cannot attribute the gap to "pair set".

mix=0.5, frozen, one_sided:

| basis              |  hack |  ±std | solve |  ±std |    n |       seeds |
| :----------------- | ----: | ----: | ----: | ----: | ---: | ----------: |
| vanilla            | 0.719 | 0.120 | 0.306 | 0.116 |    4 | 41,42,43,44 |
| v_hack_full (weak) | 0.700 | 0.109 | 0.283 | 0.038 |    3 |    41,43,44 |
| v_hack_21pairs     | 0.588 | 0.131 | 0.256 | 0.083 |    4 | 41,42,43,44 |

At shared seed 41: weak basis = 0.775 (= vanilla, no effect), strong = 0.475.

**Answer: the k=12 / 16-pair basis cuts hack ~2x more than k=5 / 10-pair, but
we don't know if k, pair-count, or tau drives it.** Untangling needs a one-knob
sweep (same pairs, k=5 vs 12) — not yet run. The strong basis IS reproducible
from current `pairs.py`: `extract --top-k=12 --v-hack-drop-bottom-frac=0.0`
(n_heldout=2 → 16 of 18 pairs); refresh already re-extracts at k=12.

For reference, the current `pairs.py` (`PAIRS`, 18 pairs) is skewed to one axis:
axis-1 weak-`run_tests` = 8/18; the other five mechanisms (hardcode, persona,
try/except-swallow, type-only-assert, weak-inequality) get 2 each.

---

## Q9. Solve-direction orthogonalization (does stripping the solve subspace recover solve?)

<!-- src (seed 41, mix=0.5, frozen, one_sided): #145 *_solveorth_base18_s41.log (v_hack_18base, no orth); #146 *_solveorth_m4_s41.log (v_hack_18solveorth4, m=4) -->

| basis                  |  hack | solve |
| :--------------------- | ----: | ----: |
| vanilla                | 0.775 | 0.300 |
| 18-pair base (no orth) | 0.500 | 0.200 |
| 18-pair solve-orth m=4 | 0.550 | 0.150 |

**Answer: no — at n=1 it did the opposite.** Stripping the top-4 solve
directions from D pre-SVD was meant to *recover* solve; instead solve fell
0.200→0.150 and hack rose 0.500→0.550. Both moves are ~0.05, inside the ~0.12
seed std — inconclusive, leaning negative. Caveats: (1) two nominally-18-pair
bases already disagree by 0.275 hack at this seed (`v_hack_full`=0.775 vs
`v_hack_18base`=0.500), so extraction variance likely dominates a 0.05 delta;
(2) with 18 pairs the solve basis B (top-4 SVD of G_c) is itself noisy and may
strip real hack signal; (3) hack/solve subspaces may genuinely overlap. Needs
≥3 seeds before any verdict.

---

## Dynamics note (sizing the convergence test)

Per-step trajectories (mix=0.125 g8, seed 41): `hack_s` rises 0→~0.6-0.75 and
**plateaus by step ~13-16**; `gt_s` (solve) stays **noisy-flat at ~0.1-0.5 the
whole run, it never climbs**. The attractor in this surrogate regime is full
*hack*, not full solve — so "run until full solve" has no target. The
convergence question is therefore: once vanilla hack plateaus (~step 15), does
projected stay below it or catch up? A 60-step run (~2.2h at g8) sees 3x past
the plateau; a 1000-step run (~36h) is wasteful.

## Open / queued (no result yet)

- **convergence (does the gap persist past the plateau?)**: 60-step seed-42
  vanilla vs projected refresh-2 at mix=0.125, then add seeds if the gap holds.
- **overshoot=1.1** (#140): queued.
- **k-slice (k=1/2/5)**: only smoke-tested, no 4B results.
- **G2/G3 cross-mechanism generalisation**: queued; the load-bearing test of
  whether a known-hack basis stops an unknown hack.