wassname/evil_MoE
1
0
Fork 0
mirror of https://github.com/wassname/evil_MoE.git synced 2026-09-12 07:50:38 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: routing-v2 spec, related-work scorecard, paper fetches, journal

Browse Source 
Routing-v2 spec (distinct-basis quarantine, two arms, proofs); related-work
no-cheat scorecard for TDGA/Cloud/SGTM/Confessions; full-text fetches of the
Deng and SGTM papers; journal entry for the run-31 confound + T1/T2 landing.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-05-31 10:16:21 +00:00
co-authored by Claudypoo
parent 4359dc53a8
commit 442630fcae
6 changed files with 1195 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/spec/20260531_routing_v2_distinct_basis.md
+253
View File
@@ -0,0 +1,253 @@
# Routing v2 — distinct-basis quarantine + per-sample persona mask
Date: 2026-05-31. Branch: probe/distill-cosine. Supersedes the additive
direction-split route arm (`proj.py` route branch) for the generalisation goal.
## Goal
Reduce reward-hack rate at deploy on **held-out** hack modes (modes the persona
mask never saw) by routing the live GRPO gradient of *flagged* rollouts into a
functionally-distinct, deletable quarantine adapter, and relying on **absorption**
(Cloud 2410.04332, SGTM 2512.05648) to pull *unflagged* hacks into the same
quarantine. No reward attached to routing; no oracle; mask is a self-supervised
persona-contrast direction (~10 pairs).
User's standing hypothesis (2026-05-31): "it already works with grad (run 31
deploy-hack=0 on held-out modes, confounded), and it will work better with the
SGTM insights (distinct basis + seeding + scale)."
## Context: what we landed on (from the 2026-05-31 brainstorm)
The five load-bearing conclusions, each with its source:
1. **Don't reward routing — impose it.** Reward-driven routing invites the model
to game its own flag (drive cosine->0, look clean): that is Baker et al.'s
obfuscated reward hacking (`search_for_more.md:40`). Cloud's "mechanistic
supervision avoids Goodharting" (`paper_gradient_routing.md:422`) works
precisely because the mask never enters the behavioural objective. So: mask
only, no confession reward. This also dodges the no-cheat boundary (a *reading*
judge over student rollouts) that the confessions path would reintroduce.
2. **The quarantine must be a distinct basis, not a shared coordinate.** Our
current `h*(delta_S + delta_S_hack)` (antipasto.py:83) makes
`dL/d delta_S == dL/d delta_S_hack` *identically* — the two knobs get the same
gradient by construction, so absorption cannot sort anything; we only ever made
them differ by the manual split. Distinct directions are required for the two
regions to receive *different* per-sample gradients.
3. **The quarantine must stay in the forward for every sample (additive), not a
hard MoE switch.** A hard per-sample forward switch (`delta_S*m + delta_S2*(1-m)`)
removes the quarantine from an unflagged hack's computation graph, so that hack
trains the kept adapter and lands in the deployed model. Absorption needs the
quarantine in-path for *all* samples so unflagged hacks can concentrate there.
Earlier Cloud's detach (`m*a + (1-m)*a.detach()`) keeps the forward value
identical and routes only the backward — that is the right primitive.
4. **Seed hard, absorb soft (the hybrid SGTM actually uses).** Flagged hacks: hard
per-sample gradient route to quarantine only (detach the kept adapter for those
samples). Unflagged hacks: no routing, but they concentrate in the quarantine by
gradient-magnitude self-reinforcement *iff* they share structure with the seeded
hacks — which is exactly the persona ("willingness to cheat") hypothesis.
5. **Leakage is real, bounded, and shrinks with scale.** SGTM names our worry
"leakage" (§4.3) and measures it: 0.005-0.02 at 64M even with 40% undiscovered
forget data, decreasing with model size (`paper_sgtm.md:160,172,174`). So the
additive design is not clean either-or, but leakage is small. Cloud's toy shows
an L1 sparsity penalty is *sometimes needed* to force concentration on
unnatural splits (`paper_gradient_routing.md:857,1115`) — we treat L1 as an
optional concentration aid, not core.
Open risks carried forward:
- **Starvation vs route-around:** detaching the *whole* kept adapter on a flagged
hack also discards any solve-signal in that rollout -> kept can starve, deploy
degrades (Cloud's retain cost ∝ routed fraction). Soft/partial routing is the
unexplored middle.
- **Ignition:** a ~10-pair persona seed may be too weak to ignite absorption at
our scale, where SGTM says *small models leak more*. This is the empirical crux.
- **run 31 confound:** current additive-route deploy=0 is not citable until the
random-V control (#157) separates "directional" from "delete-any-adapter reverts
to base".
## Scope
In: two new arms (grad-vector mask, act-vector mask) sharing one distinct-basis
quarantine adapter; the per-sample plumbing; an L1 knob; a leakage metric; the
all-arms per-mode deploy plot. Out: confession/reward routing (rejected, item 1);
TDGA project-onto-clean (kept as a documented fallback only); learned/MoE gating
(rejected, item 3).
## Core pseudocode
### Shared machinery — distinct-basis quarantine, additive forward, detach-route
```python
# Per wrapped Linear. Frozen SVD of W: W = U @ diag(S) @ Vh.
# x : [b s d_in] layer input
# Vh : [r d_in] frozen right singular vecs (kept-adapter input basis)
# U : [d_out r] frozen left singular vecs (kept-adapter output basis)
# delta_S : [r] KEPT knob (the deployed adapter), trainable
# A_q,B_q : [k_q d_in],[d_out k_q] QUARANTINE LoRA, OWN basis, init ~0, deleted at deploy
# m : [b s] per-sample/token hack flag in {0,1}, from the arm's mask
#
# Invariant: quarantine is ALWAYS summed into the forward (item 3) and has a
# DISTINCT basis from the kept adapter (item 2). For flagged samples (m=1) the
# kept adapter is detached so only the quarantine learns them (item 4, "seed hard").
def routed_forward(x, m):
kept = U @ (delta_S * (Vh @ x)) # [b s d_out] the deployed adapter
# detach-route: m=1 -> forward value unchanged, but grad to delta_S is blocked
kept = where(m[..., None], kept.detach(), kept)
quar = B_q @ (A_q @ x) # [b s d_out] distinct basis, always in path
return base_linear(x) + kept + quar, quar # quar also returned for the L1 term
# loss += lambda_l1 * quar.abs().mean() # OPTIONAL anti-duplication penalty (Cloud)
# deploy: zero A_q,B_q (or just drop quar) -> evaluate kept only.
```
Why this gives absorption (the mechanism, item 4): once flagged hacks have seeded
`A_q,B_q` with cheat-features, an *unflagged* hack B (m=0, so both adapters get its
gradient) has its **largest** gradient on the quarantine directions (they already
do cheat-ish things), so B's update concentrates there — while a clean solve, also
m=0, has tiny gradient on the quarantine and concentrates in `delta_S`. The mask
seeds; gradient magnitude sorts the unflagged bucket. Rests on B sharing structure
with the seed (the persona bet).
### Arm B (act-vector) — PRIMARY, single pass
The mask is available at forward time (activations are per-sample for free), so one
forward+backward suffices. This is the clean absorption-supporting arm.
```python
# v_act : [d] persona-contrast direction in activation space (mean-diff or top
# singular vec of stacked hack-minus-clean activations at this site),
# unit-norm, oriented cheat-ward. Built once from ~10 persona pairs.
# Computed in the SAME forward, before routing:
a_i = Vh @ x # [b s r] per-sample activation (SVD coords)
cos = cosine(a_i, v_act_in_r_space) # [b s] per-sample, no extra pass
m = (cos > tau).float() # weak, noisy mask -- fine (SGTM tolerates noise)
y, quar = routed_forward(x, m)
```
### Arm A (grad-vector) — needs the per-sample-gradient trick
The persona direction lives in `delta_S` (gradient) space, but `delta_S.grad` is
summed over the batch, so the per-sample signal is not free. Recover it with a
per-sample gate (the c_i trick), then route. Two implementable variants because the
signal is only known *after* a backward:
```python
# v_hack : [r] persona-contrast direction in delta_S/gradient space (existing
# extract_vhack_grad.py: SVD of stacked persona-pair GRPO gradients).
# --- per-sample gradient via gates (one extra leaf, cheap) ---
c = ones(b, s, r, requires_grad=True) # per-sample gate over SVD coords
h = (delta_S * c) * (Vh @ x) # splice gate into the kept path
... usual loss; loss.backward()
g_i = c.grad / delta_S[None, None, :] # [b s r] per-sample delta_S-space grad
# (divide out the delta_S scaling; watch /0)
cos = cosine(g_i, v_hack) # [b s]
m = (cos > tau).float()
# A1 STALE-MASK (single pass): use step (t-1)'s m to detach-route step t's forward.
# cheapest; m lags by one step. Good first cut.
# A2 TWO-PASS (exact): probe backward -> g_i -> m -> re-run routed_forward + backward.
# 2x backward cost; use if A1's lag hurts.
```
Asymmetry to flag in the writeup: Arm B supports absorption in one pass (mask is a
forward-time activation cosine); Arm A's mask is post-backward, so absorption needs
A1's stale mask or A2's second pass. This is a real reason Arm B is the cleaner bet
and matches the user's "act-space is the natural fit for per-sample" instinct.
## Requirements
- R1: Quarantine has a basis distinct from the kept adapter, and is summed (not
switched) in the forward. VERIFY: with a flagged-only synthetic batch, after one
step `delta_S.grad == 0` (detached) AND `A_q.grad != 0`; with an unflagged batch
both are nonzero. If a hard-switch crept in, the unflagged batch would leave
`A_q.grad == 0` — redesign until no.
- R2: Deploy = kept only. VERIFY: zeroing `A_q,B_q` changes the eval forward; the
saved adapter contains no quarantine tensors.
- R3: Per-sample mask is correct. VERIFY (Arm B): on a batch of known
hack/clean rollouts, `m` fires >2x more on hacks than clean. (Arm A): `g_i`
summed over samples equals the batched `delta_S.grad` (gate identity check).
- R4: Leakage metric implemented (SGTM-style): hack capability remaining in the
*kept* adapter after deleting quarantine, vs an equivalent-exposure vanilla
baseline. VERIFY: metric returns a number in [0,1]; data-filtering arm reads ~1.
- R5: No reward, no oracle at train time. VERIFY: grep train path — routing reads
only the persona mask; never `gt_pass`, never the full detector suite over
student rollouts.
## Tasks
- [ ] T1 (R1,R2): add quarantine LoRA `A_q,B_q` per module + `routed_forward` with
detach-route, distinct basis, deploy-zero. steps: extend antipasto.py adapter;
thread `intervention=route2` config. verify: `just smoke` walks routed_forward;
R1 synthetic-batch grad check. success: flagged grad isolates to A_q; unflagged
hits both. likely_fail: shared basis reused -> identical grads (R1 fails).
sneaky_fail: detach also zeros forward value -> solve tanks; catch by asserting
forward value unchanged vs non-detached.
- [ ] T2 (R3): Arm B act-mask — `v_act` extraction from persona pairs + forward-time
cosine. verify: R3 fire-ratio on known hack/clean. UAT: "mask fires on hacks".
- [ ] T3 (R3): Arm A grad-mask — c_i gate per-sample grad + A1 stale-mask. verify:
gate identity (sum_i g_i == delta_S.grad). UAT: "per-sample cos recovered".
- [ ] T4 (R4): leakage metric + L1 knob (`lambda_l1`, default 0).
- [ ] T5: 5-arm sweep at matched seed/steps: vanilla, erase, route-additive(old),
route2-grad, route2-act. Plus random-V control (#157) on the old route.
- [ ] T6: all-arms per-mode deploy overlay plot (`just plot`).
## Proofs to show at the end (user-observable)
1. **All-arms per-mode deploy table + overlay plot** — rows: vanilla / erase /
route-additive / route2-grad / route2-act; columns: train-hack, deploy-hack,
deploy-solve, split by mode (run_tests in-dist + file_marker/sentinel/stdout
held-out). The headline proof: route2 drops deploy-hack on **held-out** modes at
matched deploy-solve, beating erase (≈vanilla) and beating or matching the
confounded additive route. Link: `out/runs/.../all_arms_per_mode.png`.
2. **random-V control (#157)** — route-additive with a Haar-random V. If random
reproduces deploy-hack≈0+solve-jump, the additive result was an ablation
artifact (and route2's distinct-basis win is the real signal). If random stays
hacky, the direction mattered all along. Either way it disambiguates run 31.
3. **Leakage number** — SGTM-style: fraction of held-out-mode hack capability
remaining in the kept adapter after quarantine deletion. Lower than erase's
(which leaves ~everything) is the absorption proof.
4. **cos-over-time for route2** — does the persona mask stay discriminative
(`cin_t>cin_s`) longer than the additive arm's ~9 steps, and does deploy-hack
track it.
5. **Ablations that must hold:** shared-basis route2 (kill R1) must lose the
held-out drop (confirms distinct basis is load-bearing); L1=0 vs L1>0 shows
whether ignition needs the sparsity aid.
## Context (terms / invariants)
- No-cheat invariant: training never gated on `gt_pass` or the full detector suite
over student rollouts; persona pairs select the mask, that's the allowed weak
detector. Deploy = quarantine deleted.
- absorption: unflagged members of a concept concentrate in the seeded dedicated
region by gradient magnitude (Cloud/SGTM). Requires distinct basis + in-forward.
- leakage: hack capability that ends up in the *kept* (retained) adapter despite
routing. SGTM §4.3.
## Log
- 2026-05-31: spec created from brainstorm. Run 31 (additive route) gave
deploy-hack=0 on all modes incl held-out + solve-jump 0.375->0.562; confounded
(gpt-5.5 review + journal entry). Random-V (#157) is the gate.
- 2026-05-31: cos-over-time on run 31 — teacher 0.29->0.08, student flat ~0.10-0.12,
cross ~step 9. Persona mask is a stable ~10% handle on the student gradient; the
teacher signal decays (likely the student absorbing the teacher's hacks).
- 2026-05-31: T1+T2 implemented and smoke-passed. R1 grad check (synthetic):
flagged -> delta_S.grad=0, A_q/B_q.grad>0; unflagged -> both>0; forward value
unchanged by detach. smoke-route2: v_act extracted (14 modules), ||B_q||=0.109
after 30 steps, deploy eval + asserts pass. BUG found+fixed: the per-prompt
`model.zero_grad(set_to_none=True)` (there to isolate delta_S's per-source grad)
wiped A_q/B_q grads before opt.step; now stashed and re-injected like delta_S.
- 2026-05-31: defaults — vhack_refresh_every 0->5 (0 is ablation-only);
route2 reuses run-substrate (v-hack-path ignored, vhack-pairs drives v_act,
tau/rank defaulted) so the sweep needs no extra args.
## TODO (out of scope now)
- Soft/partial routing (route a fraction of a flagged sample's gradient) to trade
starvation against route-around.
- TDGA project-onto-clean as a no-signal fallback if the persona mask never ignites.