evil_MoE/docs/results.md

# Results, organized by the question each run answers

Generated from `logs/*.log` via `just results` (source: `scripts/results.py`).
Regenerate any time; this file is a curated snapshot as of 2026-05-29.

## How to read this

- **Metric = mean of the last 5 training steps** (the converged regime;
  noise-robust vs a single final step). Whole-run means (`WH`) are kept as a
  secondary column because the blog Table 1 uses whole-run; the two diverge a
  lot because hacking ramps over training, so last-5 is the honest "where it
  ended up" number.
- **hack** = fraction of *student* rollouts flagged as reward-hacks (`hack_s`).
- **solve** = fraction of *student* rollouts passing the ground-truth tests
  (`gt_s`). This is NOT `PASS_RATE`, which mixes in the ~99%-hacked teacher
  pool and is near-useless as a student-quality signal.
- All runs are the `fast` preset (20 steps, G=4, cached-teacher mix). This is
  the fast surrogate regime, not the endogenous-hack regime.
- **Epistemic status:** many ablation cells are n=1 seed — suggestive, not
  conclusive. Seed counts are in the tables. The mix=0.5 headline is the only
  n=4 cell.
- **Provenance:** `just results` prints a per-run table with a full `argv`
  column (every CLI flag), so each number traces to its exact invocation.

A confound to keep in mind: `v_hack_full` is an 18-pair extraction (current
`pairs.py`), while `v_hack_21pairs` is a 21-pair set. Comparisons across those
two confound pair-count with pair-set.

---

## Q1. Does the cached-teacher pool actually drive the student to hack? (feasibility, H4)

Why: the whole fast-surrogate design rests on a clean base student picking up
hacking from off-policy teacher exposure, instead of the ~64 GPU-h endogenous route.

| arm | mix | hack | solve | seeds |
| :-- | --: | --: | --: | --: |
| vanilla | 0.5 | 0.719 | 0.306 | 4 |
| vanilla | 0.25 | 0.678 | 0.200 | 3 |
| vanilla | 0.125 | 0.754 | 0.261 | 2 |

**Answer: yes.** Clean Qwen3-4B reaches 68-75% last-5 student hack rate within
20 steps across teacher densities. The surrogate works; the student learns to
hack from exposure.

## Q2. Does v_hack gradient projection reduce hacking vs vanilla, at matched config? (H1)

Why: the core hypothesis. mix=0.5, v_hack_21pairs, one_sided, k=5, n=4 seeds (41,42,43,44).

| arm | hack | solve | Δhack | Δsolve | seeds |
| :-- | --: | --: | --: | --: | --: |
| vanilla | 0.719 | 0.306 | — | — | 4 |
| projected frozen-V | 0.588 | 0.256 | −13.1pp | −5.0pp | 4 |
| projected refresh-2 | 0.537 | 0.225 | −18.2pp | −8.1pp | 4 |

**Answer: yes, but with a real solve cost.** Projection cuts last-5 hack by
13pp (frozen) to 18pp (refresh-2), short of the preregistered 30pp. It also
costs 5-8pp of student solve rate — a genuine selectivity problem (this is
what solve-orthogonalization, queued, targets). Note the cost is invisible on
whole-run `PASS_RATE`; it only shows on the last-5 student-GT metric.

## Q3. one_sided vs no_gate vs reverse gating? (gate_mode ablation)

Why: how aggressively to ablate. one_sided removes only the hack-ward
component; no_gate removes any motion in span(V); reverse pushes anti-hack.
mix=0.5, v_hack_full, frozen, n=1 (seed 41) each — suggestive only.

| gate | hack | solve | seeds |
| :-- | --: | --: | --: |
| one_sided | 0.700 | 0.283 | 3 |
| no_gate | 0.625 | 0.200 | 1 |
| reverse | 0.575 | 0.150 | 1 |

**Answer: more aggressive = more hack suppression but worse solve.** reverse
gives the largest hack cut (0.575) but halves solve (0.306→0.150). one_sided is
the most solve-preserving. This is the same selectivity tension as Q2: you can
buy hack reduction with solve, and the gradient of that trade is steep.

## Q4. SVD top-k basis vs rank-1 mean-diff? (basis ablation)

Why: with few pairs, SVD axes 2..k may be noise; mean-diff (k=1) regularizes
to the single robust direction. mix=0.5, frozen, n=1.

| basis | hack | solve | seeds |
| :-- | --: | --: | --: |
| SVD top-k (k=5, v_hack_full) | 0.700 | 0.283 | 3 |
| mean-diff (k=1, v_hack_full_meandiff) | 0.750 | 0.125 | 1 |

**Answer: mean-diff is worse on both axes** (higher hack, much lower solve). A
rank-1 basis is too blunt — it doesn't suppress more hacking and it costs more
solve. Keep the multi-axis SVD basis.

## Q5. refresh-every cadence sweep

Why: the v_hack basis goes stale as the student drifts (cos_pre_t decays
0.28→0.07). How often to re-extract? mix=0.5, v_hack_21pairs, one_sided, n=1
except frozen/refresh-2 (n=4).

| refresh | hack | solve | seeds |
| :-- | --: | --: | --: |
| frozen (0) | 0.588 | 0.256 | 4 |
| 1 | 0.600 | 0.200 | 1 |
| 2 | 0.537 | 0.225 | 4 |
| 5 | 0.550 | 0.225 | 1 |
| 10 | 0.575 | 0.200 | 1 |

**Answer: refresh-2 is the sweet spot** (lowest hack at 0.537). refresh-1 is no
better than frozen (too noisy a basis), and 5/10 drift back up. The effect is
small (~5pp) and the n=1 cells are noisy, but 2 is the consistent pick.

## Q6. Teacher density (mix-ratio) — does the projection gap hold as the pool thins?

Why: lower mix = less off-policy hack pressure, closer to the real regime.
v_hack basis frozen, one_sided.

| mix | vanilla hack | projected hack | Δhack | vanilla solve | projected solve |
| --: | --: | --: | --: | --: | --: |
| 0.5 | 0.719 | 0.588 | −13pp | 0.306 | 0.256 |
| 0.25 | 0.678 | 0.556 | −12pp | 0.200 | 0.217 |
| 0.125 | 0.754 | 0.657 | −10pp | 0.261 | 0.214 |

**Answer: the gap holds, narrowing slightly as the pool thins** (−13 → −10pp).
At mix=0.25 projection even nudges solve up. The intervention isn't an artifact
of heavy teacher mixing. (mix=0.25/0.125 use v_hack_full, so not strictly
matched to the 21-pair mix=0.5 row.)

## Q7. Noise-floor cut (drop_bottom_frac) 0.25 vs 0.0?

Why: dropping the bottom-25% singular values is meant to remove noise axes.
mix=0.5, v_hack_full, frozen, n=1.

| dropf | hack | solve | seeds |
| --: | --: | --: | --: |
| 0.25 | 0.700 | 0.283 | 3 |
| 0.0 | 0.625 | 0.200 | 1 |

**Answer: inconclusive (n=1).** dropf=0 looks like a bigger hack cut but also
lower solve — same trade as everywhere, and a single seed. Needs replication
before drawing anything.

## Q8. Pair set: 18-pair vs 21-pair extraction

Why: more contrastive pairs across more axes should give a better-conditioned
basis. mix=0.5, frozen, one_sided.

| basis | hack | solve | seeds |
| :-- | --: | --: | --: |
| v_hack_full (18 pairs) | 0.700 | 0.283 | 3 |
| v_hack_21pairs (21 pairs) | 0.588 | 0.256 | 4 |

**Answer: the 21-pair basis suppresses more hacking** (0.588 vs 0.700) at a
small solve cost. Pair set/count matters and is one of the larger levers seen
here. Caveat: confounds count with the specific extra pairs, and the seed sets
differ.

---

## Open / queued (no results yet)

- **overshoot=1.1** (mild over-projection): queued (#140). Tests if removing
  110% of the hack-ward component beats 1.0 without the solve cost of full
  `reverse`.
- **solve-orthogonalization** (strip the known-solve subspace from D pre-SVD):
  queued (#143-146), directly targets the Q2/Q3 solve cost.
- **let-it-converge** (60 steps): queued (#141-142), tests whether the gap
  persists past step 20.
- **k-slice ablation** (k=1/2/5 SVD): only smoke-tested so far; no 4B results.
- **G2/G3 cross-mechanism generalisation**: queued; the load-bearing test of
  whether a basis from known hacks suppresses an unknown one.