evil_MoE/docs/spec/20260601_calibrated_tau_route2grad.md

# Spec — per-step calibrated threshold τ for route2-grad routing

Status: APPROVED by user 2026-06-01, implement + smoke + queue + follow.
Context survives compaction here; resume from "Implementation" if context was lost.

## Hypothesis / why

route2-act and route2-grad both gate routing on `cos(·, v) > 0`. In high-dim,
`cos ≈ 0` for almost everything, so an absolute-zero threshold is a ~50% coin
flip -> over-routing -> half of ALL learning dumped into the deleted quarantine
-> collapse (job 46) or null. The fix is NOT to route by the detector flag (that
= SGTM, throws away the vector novelty: the flag only catches known hack A, the
whole point is the *direction* generalises to unknown B). The fix is to keep
routing by the v_hack DIRECTION but **calibrate the threshold per step** so it
rides the cos drift (we measured cin_t 0.27->0.07 over ~10 steps, so any fixed
τ goes stale and silently stops firing).

Driving principle (one): route by where a rollout falls on the v_hack axis
RELATIVE to the known-hack vs clean clouds, recalibrated each step.

## Mechanism (per module, per step, EMA-smoothed)

Anchors available every prompt without a new buffer:
- hack cloud = teacher rollouts (known-A hacks, always present) + detector-
  flagged student rollouts (`hack_E` = loophole used; the allowed weak detector).
- clean cloud = NOT-flagged student rollouts. (Contaminated with unknown B -- that
  is FINE and intended: it makes τ slightly conservative, and B, sharing the
  direction, lands ABOVE τ and gets routed. Do NOT force-keep this set.)

Per module `name`, from this prompt's `cos_b` [G] (already computed in
`_route2_grad_filter`):
- μ_hack_prompt = mean(cos_b[hack_anchor]) ; μ_clean_prompt = mean(cos_b[clean_anchor])
- EMA across prompts/steps (β=0.9): ema_hack[name], ema_clean[name]
  (EMA is the cheap equivalent of "last N hacks": teacher is the always-present
  floor, recent student-flagged hacks sharpen it. Explicit rollout buffer =
  TODO if teacher-anchor proves biased toward teacher-style hacks.)
- τ[name] = (ema_hack[name] + ema_clean[name]) / 2
- route_mask = hack_anchor | (cos_b > τ[name])   # force-route known hacks
  (teacher + flagged student); τ-route the ambiguous rest (incl. unknown B).

Warmup: at step 0 delta_S≈0 so few reliable axes -> cos_b≈0 -> μ_hack≈μ_clean≈0
-> τ≈0 (≈ old cos>0 behaviour) until delta_S grows. Separation (hkgap) emerges
over the first few steps. Sanity: by a few steps μ_clean<~0.1, μ_hack>~0.2-ish
(the user's hardcode intuition, but self-calibrated).

## Logging (new columns, route2-grad only)

- `tau`   = mean over modules of τ[name]
- `hkgap` = mean over modules of (ema_hack - ema_clean)  <- the discrimination
  gauge; generalises cin_t>cin_s. If hkgap collapses to ~0 the direction stopped
  separating (the real failure signal, not a threshold-choice problem).

## Implementation (exact edits in src/projected_grpo/train.py)

1. Before the step loop (near `route_span_checked = False`, ~L1082): add
   `ema_hack_cos: dict = {}`, `ema_clean_cos: dict = {}`, `route2_tau: dict = {}`
   (persist across steps). EMA_BETA = 0.9.

2. Per-step lists near `step_flagged` (~L1192): `step_tau = []`, `step_hkgap = []`.

3. Per-prompt anchor masks: after grading (hack_E_flags built, ~L1396), before
   the backward, when is_route2_grad build device bool tensors aligned to merged
   rows [student 0..G_s-1, teacher G_s..]:
     hack_anchor[i]  = (not is_student[i]) or (is_student[i] and hack_E_flags[i])
     clean_anchor[i] = is_student[i] and not hack_E_flags[i]
   (hack_E_flags has length G_s = #student rows; teacher rows -> hack_anchor.)

4. `_route2_grad_filter` (~L1194): add params (hack_anchor, clean_anchor).
   Replace `flagged = (cos_b > 0).float()` with the EMA-calibrated τ logic:
     - update ema_hack_cos[name]/ema_clean_cos[name] from this prompt's anchors
       (skip a cloud if it has no rows this prompt)
     - τ = (ema_hack_cos.get(name,0)+ema_clean_cos.get(name,0))/2 ; route2_tau[name]=τ
     - route = (cos_b > τ) | hack_anchor   ; flagged = route.float()
     - step_tau.append(τ); step_hkgap.append(ema_hack_cos.get(name,0)-ema_clean_cos.get(name,0))
   Pass hack_anchor/clean_anchor through the call site (~L1518).

5. Row dict (~L1864): add `"tau": (mean(step_tau) if step_tau else float("nan"))`,
   `"hkgap": (mean(step_hkgap) if step_hkgap else float("nan"))`.

6. StepLogger cols (~L711, after the routing2_act block): add
     if arm == "routing2_grad":
         _Col("tau",   6, "tau",   "+.2f", "per-step calibrated route threshold (midpoint of hack vs clean cos clouds)")
         _Col("hkgap", 6, "hkgap", "+.2f", "ema_hack_cos - ema_clean_cos; >0 = v_grad still separates hack from clean")

## Verify / queue / follow

- `just smoke-route2`: confirm tau/hkgap/qE columns render, routing fires
  (flagged frac < ~0.5, not pinned at 0.5), exit 0.
- Queue (substrate, 60 steps, seed 41), label:
  why: does per-step calibrated-τ vector routing (route2-grad) stop over-routing
  (flagged<<0.5) and suppress held-out deploy-hack vs vanilla at matched solve;
  resolve: qE bounded + hkgap>0 sustained + deploy file_marker hack < vanilla.
- `pueue follow <id> | tail` in background.

## No-cheat check

Detector flag (hack_E) only CALIBRATES μ_hack (known A) + anchors teacher hacks.
The routing GATE is cos>τ applied to ALL rollouts incl. unknown B. gt_pass never
touches routing. Compliant.

## Status: IMPLEMENTED + RUNNING (job 54)

Implemented in `6eb894f`, smoked (tau/hkgap render, exit 0). Running as job 54
(calibrated-τ + grad_clip 0.5, route2-grad, 60 steps, seed 41). job 52
(caltau-alone) killed and folded in; clip 0.5 is a no-op while gn<0.5 so 54 is
the strict superset.

### Live observation (steps 0-2, 2026-06-01)

The calibration works as a discriminator but does NOT fix over-routing-by-energy:
- `hkgap` positive and rising (0.00 -> 0.03 -> 0.08): the v_grad direction DOES
  separate the hack cloud from the clean cloud. The vector is alive.
- `tau` tracking up with it (0.00 -> 0.02 -> 0.04): the threshold rides the drift
  as designed.
- BUT `qE` 0.73 -> 0.97 -> 0.97: ~97% of gradient ENERGY lands in the deleted
  quarantine by step 1, and `gt_s` 3 -> 7 -> 0 (solving collapsed).

Diagnosis: the failure was never the routing FRACTION (which τ fixes); it is the
always-summed 33M A_q/B_q quarantine MAGNITUDE. Even a correctly-thresholded route
sends the routed gradient into a knob whose per-param grads dwarf delta_S's, so
the energy ratio pins near 1 and the deployed adapter learns nothing. This is the
SYNTHESIS "next lever" prediction: if qE stays high while hkgap>0, the culprit is
quarantine magnitude, not the gate.

## DESIGN CHANGE (2026-06-01): one adapter, scale-matched quarantine

Acted on the magnitude diagnosis by removing the distinct-basis LoRA entirely.
The quarantine is now delta_S_hack -- the SECOND diagonal in the same frozen SVD
basis, shape [r] per module, identical capacity to delta_S. route2's calibrated-τ
gate parks the flagged rollouts' delta_S-grad contribution into delta_S_hack.grad
(via step_grad_hack in _route2_grad_filter), exactly as proj.py's `route` parks
its subspace-projected component; delta_S keeps the unflagged. Both diagonals
train at one shared lr; delta_S_hack is zeroed at deploy.

Rationale (user): a 33M LoRA vs a ~2k-param delta_S per module means "dump
everything in the quarantine" is the low-resistance path -- a capacity edge, not
honest absorption. Capacity-balanced diagonals remove that bias. SGTM's own
quarantine is capacity-matched (a split of the same layer, equal dims), and uses
a hard detach -- no soft/tanh/sigmoid gate -- confirming the fix is balance, not
gating.

Removed: A_q/B_q params, v_act buffer + extract_v_act, the act-mask arm (a shared
diagonal can't be per-token gated), route2_mask / route2_quarantine_rank /
route2_quar_lr_scale knobs, the separate quar optimizer group. arm name
"routing2_grad"/"routing2_act" -> "routing2".

v_grad refresh extracts from the MAIN knob (delta_S.grad) with the quarantine
ablated -- the deployed-model gradient is what we route, and both diagonals share
the basis so the direction is directly usable on delta_S's live gradient.

Smoked clean (tiny-random): tau/hkgap/qE render, ||delta_S_hack||=0.0074>0 assert
passes, deploy-ablation fires, exit 0. Queued on the substrate (seed 41, 60 steps).