evil_MoE/docs/results.md

# Results, organized by the question each run answers

Generated from `logs/*.log` via `just results` (source: `scripts/results.py`).
Curated snapshot 2026-05-29; regenerate any time.

## How to read this

- **Metric = mean of the last 5 training steps** (converged regime; noise-robust
  vs a single step). Whole-run (`WH`) is smoother but dilutes the converged
  behaviour with the early ramp-up; the blog Table 1 uses WH, this doc uses
  last-5.
- **hack** = fraction of *student* rollouts flagged as reward-hacks (`hack_s`).
- **solve** = fraction of *student* rollouts passing ground-truth tests
  (`gt_s`). NOT `PASS_RATE` (which mixes in the ~99%-hacked teacher pool).
- **Comparisons are paired on seed.** A projected run is compared to the vanilla
  run at the *same (mix, seed)*; we take per-seed deltas, then mean ± std over
  shared seeds. Comparing a 3-seed mean to a 1-seed point (as an earlier draft
  did) is meaningless. n=1 cells have no std and are flagged as such.
- **Sobering caveat up front:** last-5 is a 5-step mean, so per-seed it's noisy;
  at n=4 the headline Δhack std (~0.15) is as large as the mean (~0.13). The
  effect is real-looking and consistent in sign, but NOT cleanly separated from
  zero at this n. Read the deltas as suggestive, weighted by n and std.
- All runs are the `fast` preset (20 steps, G=4, cached-teacher mix); the fast
  surrogate regime, not endogenous hacking. Incomplete/aborted runs are
  excluded (a run must log all `steps`).
- Confound: `v_hack_full` = 18-pair extraction; `v_hack_21pairs` = 21-pair set.
  Cross-basis comparisons confound pair-count with pair-set.

---

## Q1. Does the cached-teacher pool drive the student to hack? (feasibility, H4)

| arm | mix | hack | solve | seeds |
| :-- | --: | --: | --: | --: |
| vanilla | 0.5 | 0.719 | 0.306 | 41,42,43,44 |
| vanilla | 0.25 | 0.678 | 0.200 | 41,42,43 |
| vanilla | 0.125 | 0.754 | 0.261 | 41 (×2) |

**Answer: yes.** Clean Qwen3-4B reaches 68-75% last-5 hack within 20 steps at
every teacher density. (Don't compare mixes here as a trend — different seed
sets; see Q6 for the paired mix comparison.)

## Q2. Does v_hack projection reduce hacking vs vanilla? (H1, paired)

Paired Δ vs same-seed vanilla, mix=0.5, v_hack_21pairs, one_sided, k=5, n=4 (41-44):

| arm | Δhack | Δhack std | Δsolve | n |
| :-- | --: | --: | --: | --: |
| projected frozen-V | −0.131 | 0.146 | −0.050 | 4 |
| projected refresh-2 | −0.181 | 0.169 | −0.081 | 4 |

**Answer: a consistent-in-sign reduction (−13pp frozen, −18pp refresh-2), but
the std ≈ the mean at n=4, so it is not statistically clean.** Both arms also
cost 5-8pp of student solve. Short of the preregistered 30pp. The honest
statement: directionally it reduces hacking on every seed, but more seeds are
needed to call the magnitude. (WH paired deltas are smoother and tell the same
sign story.)

## Q3. one_sided vs no_gate vs reverse gating? (gate_mode)

| gate | Δhack | Δsolve | n |
| :-- | --: | --: | --: |
| one_sided | −0.062 | −0.081 | 4 |
| no_gate | −0.150 | −0.100 | 1 |
| reverse | −0.200 | −0.150 | 1 |

(All v_hack_full, mix=0.5, frozen. one_sided is n=4 with std 0.075; no_gate and
reverse are **n=1, no std** — not yet comparable.)

**Answer: provisional only.** The n=1 aggressive gates (no_gate, reverse) show
larger hack cuts and larger solve costs, consistent with the selectivity
trade-off, but each is a single seed. Needs ≥3 seeds before any claim. Note
one_sided on v_hack_full is only −0.062 (within std) — weak basis (see Q8).

## Q4. SVD top-k vs rank-1 mean-diff? (basis)

| basis | Δhack | Δsolve | n |
| :-- | --: | --: | --: |
| SVD k=5 (v_hack_full) | −0.062 | −0.081 | 4 |
| mean-diff k=1 (v_hack_full_meandiff) | −0.025 | −0.175 | 1 |

**Answer: mean-diff looks worse** (smaller hack cut, larger solve cost) but n=1.
A rank-1 basis being too blunt is plausible; not established at n=1.

## Q5. refresh-every cadence

| refresh | Δhack | Δhack std | Δsolve | n |
| --: | --: | --: | --: | --: |
| frozen | −0.131 | 0.146 | −0.050 | 4 |
| 1 | −0.175 | — | −0.100 | 1 |
| 2 | −0.181 | 0.169 | −0.081 | 4 |
| 5 | −0.225 | — | −0.075 | 1 |
| 10 | −0.200 | — | −0.100 | 1 |

(All v_hack_21pairs, mix=0.5, one_sided.)

**Answer: refresh-2 edges out frozen** (−0.181 vs −0.131, both n=4) but the
difference (~5pp) is small vs the std (~0.16). The n=1 cadences (1/5/10) hint
that more refresh = slightly more suppression, unconfirmed.

## Q6. Teacher density (mix) — paired, does the gap hold as the pool thins?

| mix | Δhack | Δhack std | Δsolve | n | shared seeds |
| --: | --: | --: | --: | --: | --: |
| 0.5 | −0.062 | 0.075 | −0.081 | 4 | 41(×2),43,44 |
| 0.25 | −0.122 | 0.146 | +0.017 | 3 | 41,42,43 |
| 0.125 | −0.100 | 0.040 | +0.007 | 2 | 41(×2) |

(v_hack_full, frozen, one_sided — the basis with coverage at all three mixes.)

**Answer: the reduction holds across densities (−6 to −12pp) and your read is
right — any mix is sufficient to see it.** At lower mix the solve cost vanishes
(even slightly positive). The mix=0.125 cell has the tightest std (0.040, n=2).

## Q8. Pair set: 18-pair (v_hack_full) vs 21-pair (v_hack_21pairs)

| basis | Δhack | Δhack std | Δsolve | n |
| :-- | --: | --: | --: | --: |
| v_hack_full (18) | −0.062 | 0.075 | −0.081 | 4 |
| v_hack_21pairs (21) | −0.131 | 0.146 | −0.050 | 4 |

**Answer: the 21-pair basis suppresses ~2x more hacking** (−0.131 vs −0.062),
both n=4 mix=0.5 frozen. Pair set is one of the largest levers here. Confounds
count with the specific extra pairs; the 21-pair shared-seed set is the full
41-44 while v_hack_full's is 41(×2),43,44.

---

## Open / queued (no result yet)

- **solve-orthogonalization** (#145 base done, #146 m=4 running): base 18-pair
  paired Δhack −0.275 / Δsolve −0.100 (n=1, seed 41). m=4 pending — that's the
  one that tests whether stripping the solve subspace recovers the solve cost.
- **overshoot=1.1** (#140), **let-it-converge 60-step** (#141/142): queued.
- **k-slice (k=1/2/5)**: only smoke-tested, no 4B results.
- **G2/G3 cross-mechanism generalisation**: queued; the load-bearing test of
  whether a known-hack basis stops an unknown hack.