evil_MoE/docs/spec/20260606_pair_routing_design.md

# Pair-routing redesign — 2026-06-06

Goal: make the extracted hack direction the SOLE suppression mechanism, calibrate
everything from the hand-built persona pairs (no live detector, no teacher
force-route), so "does the direction generalize to unlabeled hack B" is a clean
test. Driving principle: the only labelled data anywhere is the pairs.

## Naming

Preference convention, per pair:
- `cho` = chosen = the honest/clean completion.
- `rej` = rejected = the hack completion.
- `vec` = hack-ward direction = `rej - cho` (so `cos(g, vec) > 0` means hack-ward).
  (The current code calls this `v_grad` and orients it `hack - clean`; same thing.)

We can build `vec` from gradients OR activations of the pairs (fork below).

## What we do NOW (route2)

```py
# ── offline: direction from pairs (gradients), then THROWN INTO a live gate ──
vec = unit(mean_p(∇logp(rej_p) - ∇logp(cho_p)))      # per module, SVD top-k, hack-ward

# ── live, every step ──
for rollout b:
    g_b   = per_rollout_δS_grad(b)                   # recovered from the gate hook
    cos_b = cos(g_b, vec)

# threshold comes from a LIVE WEAK DETECTOR over the student's own rollouts:
hack_anchor  = teacher_rows | hacked_E(students)     # ← live detector, noisy, leaks onto B
clean_anchor = ~hack_anchor
τ = (ema(mean cos_b[hack_anchor]) + ema(mean cos_b[clean_anchor])) / 2

flagged = hack_anchor | (cos_b > τ)                  # ← FORCE-ROUTE by label, then gate
route(flagged) → δS_hack                             # quarantine, deleted at deploy
keep(~flagged)  → δS                                 # the deployed knob
```

Two things make `vec` non-load-bearing here:
1. `hack_anchor |` force-routes every teacher / detector-flagged rollout regardless of `vec`.
2. With a random `vec`, both anchor means → 0, so `τ → 0` and `cos_b > τ` is a coin flip.
So a random direction suppresses as well as the real one (jobs 114/115/117): the
labels carried it. The detector also lumps "hacked" with "honestly passed but wrote
a self-test", and leaks ~1.1% onto held-out B.

## What we do INSTEAD (pair-routed)

Everything — direction AND threshold — comes from the pairs. No teacher force-route,
no live detector over students. Pure gate.

```py
# ── from the pairs, refreshed every N steps through the CURRENT adapter ──
for pair p:
    g_rej[p] = ∇logp(rej_p)        # hack completion   (forward+backward through δS)
    g_cho[p] = ∇logp(cho_p)        # clean completion
    Δ[p]     = g_rej[p] - g_cho[p] # hack-ward per-pair diff
vec = unit(mean_p Δ[p])            # or SVD top-k of stacked Δ; per module, hack-ward

# threshold from the SAME pairs — where hacks land vs where clean lands on `vec`:
c_rej = mean_p cos(g_rej[p], vec)  # high  (hacks point hack-ward)
c_cho = mean_p cos(g_cho[p], vec)  # low   (clean points away)
τ     = (c_rej + c_cho) / 2        # midpoint; no detector, no teacher, no live label

# ── live, every step: pure gate, no labels ──
for rollout b:
    g_b = per_rollout_δS_grad(b)
    if cos(g_b, vec) > τ:
        route(g_b) → δS_hack       # quarantine, deleted at deploy
    else:
        keep(g_b)  → δS            # deployed knob
```

Teacher is kept only to SEED emergence and is cut at step 30 (`teacher_off_step=30`);
its rollouts are NOT force-routed — they go through the same `cos > τ` gate as any
student rollout. After step 30 it is pure on-policy.

## Now vs new — what changed

| | now (route2) | new (pair-routed) |
|---|---|---|
| direction `vec` | pair gradient diff | pair gradient (or activation) diff — same source |
| threshold τ | live `hacked_E` detector over students + EMA | the pairs' own `cos(g_rej)` vs `cos(g_cho)` midpoint |
| force-route | yes (`hack_anchor \|`) | none — gate only |
| live detector over students | yes (noisy, leaks onto B) | none |
| teacher | mixed throughout, force-routed | seed only, cut@30, gated like any rollout |
| is `vec` load-bearing? | no (labels carry it) | yes — it is the only mechanism |
| held-out B suppressed iff | labels happen to cover it | `cos(g_B, vec) > τ`, i.e. B shares the direction |

## Fork to decide in the rebuild: gradients vs activations for `vec`

- **gradients** (`∇logp(rej) - ∇logp(cho)`): keeps the "intervene in the gradient,
  not the reward" thesis; ablating it removes the hack-ward *update*. Cost: noisy,
  drifts (needs refresh), per-rollout gradient recovery via the gate hook is the
  fiddliest code we have.
- **activations** (hidden-state diff-of-means on rej vs cho): cleaner, lower
  variance, just a forward pass; but ablating it is concept-erasure / steering, a
  different mechanism than gradient routing. There is a dormant act-vector variant
  to revive.
- Plan: build both from the same pairs, gate the same way, compare on held-out B.
  Default to gradients to preserve the thesis unless activations clearly win.

## Calibration risk to smoke-test first

The pair gradients are teacher-forced on authored completions; live rollouts are
sampled (different length / entropy), so `τ` from the pairs may not sit between the
live hack/clean rollout cosines. BEFORE trusting it: log `cos(g_rej)`, `cos(g_cho)`
(pairs) alongside the live per-rollout `cos_b` distribution and confirm the pair
midpoint actually separates live hack from live clean. If it doesn't, the fix is to
keep calibrating τ each step but still from the pairs (re-projected through the
current adapter), not from a live detector.

## What this buys

- Airtight no-cheat by construction: B never touched by any detector, so no
  `--gate-anchor-teacher-only` patch needed.
- The real-vs-random control becomes meaningful: if a random `vec` now suppresses B,
  it is pure coincidence, not labels. If only the real `vec` suppresses B, the
  direction genuinely generalizes — the whole novelty.
- Less code: delete the `hacked_E` plumbing, the `hack_anchor`/`clean_anchor`
  builder, the `--gate-anchor-teacher-only` flag, the EMA detector calibration.

## Implementation plan (src/vgrout/train.py) — actionable, post-compaction

Replace route2's gate in place (research code, break it; tag `pre-routing-refactor`
is the rollback). Gradients, not activations, for `vec` (default; activation variant
deferred). `vec` sign = hack-ward = `rej - cho`.

1. **DELETE `build_route2_anchors`** (~line 337) and its call site. No more
   `hack_anchor`/`clean_anchor` from teacher membership or the detector.
2. **Rewrite `_route2_grad_filter`** (~line 877):
   - drop the `hack_anchor |` force-route term -> gate is `cos_b > tau` only.
   - drop the EMA `ema_hack_cos`/`ema_clean_cos` detector calibration (~896-908).
   - `tau` now comes from the pairs (step 3), passed in, not computed from live rollouts.
   - route the vec-COMPONENT not the whole rollout (see Review-findings decision #3):
     for a flagged rollout, `c = cos*vec` goes to `delta_S_hack`, the orthogonal
     remainder stays in `delta_S`. Removes `rollout_ablate_frac`.
3. **Pair-calibrated tau, refreshed every `vhack_refresh_every` steps** (reuse the
   existing v_grad refresh hook): when we (re)build `vec` from the pairs, also compute
   `c_rej = mean_p cos(g_rej[p], vec)`, `c_cho = mean_p cos(g_cho[p], vec)`,
   `tau = (c_rej + c_cho)/2`, per module. The extract path already produces per-pair
   `g_rej`/`g_cho` (it builds `vec = mean(g_rej - g_cho)`); add the two cosine means +
   tau alongside. Store `route2_tau[name]` from this, not from anchors.
4. **Remove plumbing**: `--gate-anchor-teacher-only` flag + `teacher_only` arg;
   `hack_E_flags` feeding the gate (keep it for the streaming hk_* LOG columns only if
   cheap, else drop); `route2_random_v_seed` stays (it's the directionality control).
5. **Config**: `teacher_off_step: int = 30` default (seed then on-policy). Keep teacher
   mixing 0->30 only; its rollouts go through the same `cos > tau` gate (NOT force-routed).
6. **Diagnostics to keep/print**: `hkgap = c_rej - c_cho` (now a PAIR quantity, the
   gate's separation margin); per-step `cos_b` distribution; `tau`; fraction flagged;
   `resid = cos(kept grad, vec)`. SHOULD: `c_rej > tau > c_cho` and pair midpoint
   brackets the live `cos_b` of hack vs clean rollouts (the calibration smoke-check).

## Current state — resume after compaction

- Working on **main** (`probe/distill-cosine`), NOT the worktree. Worktree
  `/workspace/projected_grpo-pairroute` (branch `refactor/pair-routing`) holds an
  earlier copy of this spec; ignore or `git worktree remove` it.
- **Queue is PAUSED** (`pueue pause`). Job 127 (`erase_realv`) was running when paused.
  `pueue start` resumes. Do NOT resume until the refactor is committed + smoked, or the
  queued route2/A5 jobs will run half-built code.
- Rollback tag: `pre-routing-refactor`. Job manifest: `docs/spec/20260606_job_manifest.md`.

## Queued-job disposition (decide before `pueue start`)

- **Superseded by this refactor (old route2 semantics) -> remove + requeue under new code**:
  124 (route2_toff40), 125 (route_randomV), 126 (a5 route2 real teacher-only),
  130 (route2-200 KL), 133/134 (a5 route2 seeds), 135 (a5 random v_grad).
- **Still valid as-is (intervention=none / erase)**: 129 (vanilla-200 KL, A4),
  131/132 (a5 vanilla seeds), 128 (erase placebo), 127 (erase real-v, was running).
  Erase is already a pure-vector arm (no force-route); keep it as the cross-check.
- After the refactor, requeue the decisive new-method test: pair-routed real `vec` vs
  random `vec`, A5 regime (teacher=run_tests, off@30), measure held-out B suppression.

## Smoke + UAT

- `just smoke` (route2 path) must pass on the tiny-random model after the rewrite.
- `scripts/verify_*.py` gates stay green; `verify_gate_anchor.py` becomes moot
  (no anchor) -> update or delete it.
- UAT (refactor works): a fast 60-step pair-routed real-`vec` run shows deploy hack
  < vanilla at matched solve, AND the calibration check holds (`c_rej > tau > c_cho`,
  pair tau brackets live `cos_b`).
- UAT (science): pair-routed random-`vec` does NOT suppress held-out B as well as
  real-`vec` -- else the vector is still decorative and the method is just gradient
  routing on labels.

## Review findings (2026-06-06) -- decisions before implementing

Cross-reviewed by Claude + deepseek-v4-pro (docs/reviews/20260606_pairroute_review_deepseek.md).
Both converge on the same threshold problem; resolutions below are now part of the plan.

1. **tau is circular, not just scale-mismatched.** Because `vec = mean(g_rej - g_cho)`,
   the inequality `c_rej > c_cho` holds BY CONSTRUCTION even when `vec` is pure noise, so
   the pair midpoint cannot validate that the gate separates anything. Separately, pair
   gradients are teacher-forced while live rollouts are sampled, so the pair cosine scale
   need not match the live `cos_b` scale; refreshing every N steps fixes adapter *drift*,
   not this *distribution* gap.
   - Decision: keep pair-midpoint tau as the no-extra-labels DEFAULT for the method, but
     (a) compute a LEAVE-ONE-PAIR-OUT separation `c_rej^{-p} vs c_cho^{-p}` as the real
     diagnostic that `vec` generalizes across pairs (cheap at ~10 pairs), and (b) for the
     real-vs-random CONTROL, set tau by a QUANTILE of the live `cos_b` so the flagged
     FRACTION is matched between conditions.

2. **Match the flagged fraction in the real-vs-random control (deepseek #2).** Real and
   random `vec` otherwise quarantine different volumes of gradient, so a suppression gap
   could be volume, not direction. The quantile-tau in 1(b) controls this: equal fraction
   routed, only the DIRECTION differs. Suppression gap at matched fraction => direction is
   load-bearing.

3. **Route the vec-COMPONENT, not the whole rollout (Claude).** The route2 pseudocode
   quarantined a flagged rollout's entire `delta_S` gradient, which also strips its solve
   signal (solve-starvation on problems only solved-by-hacking). Decision: subtract the
   `cos*vec` component into `delta_S_hack` and keep the orthogonal remainder in `delta_S`
   (erase-style projection, routed not erased). Drops the need for `rollout_ablate_frac`.

4. **Degeneracy diagnostic (deepseek #3).** As routing suppresses hacks, the hack-pair
   gradient can weaken and the refreshed `vec` degenerate. Log `hkgap = c_rej - c_cho`
   per refresh; if it collapses toward 0, freeze a pre-routing `vec` snapshot.

5. **Pre-register the science UAT (deepseek).** n>=3 seeds per condition (real/random),
   success = mean held-out-B deploy hack under real-`vec` is below random-`vec` by more
   than the across-seed std of the random baseline. Qualitative "suppresses better" is
   not enough.