mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-07-10 10:43:29 +08:00
docs: fix review findings (global noise-floor, route one-sided, G3 xref)
External review (3 subagents) caught: - blog: noise-floor drop is GLOBAL across modules, not per-Linear (proj.py:187) - blog: route pseudocode used full c; route actually uses the same one-sided gate as erase and quarantines the identical 'removed' vector (proj.py:124,199) - spec: 'never seen by detector' -> clarify student trains on all 4 modes, the detector just never labels C/D for v_hack extraction; cross-ref G3/task #107 Dismissed: reviewer claim that only exit_code survived (stale spec; live log columns hk_rt/hk_so/hk_se/hk_fm confirm 4 modes) and a hallucinated 'Furthermore'. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
@@ -63,7 +63,7 @@ The whole intervention in three lines:
|
||||
- we isolate that direction from a handful of (hack, clean) example pairs
|
||||
- during each training update we project that direction out of the gradient before the optimizer applies it
|
||||
|
||||
Concretely. For each of 21 pairs (same prompt, hack completion vs clean completion), compute the gradient that GRPO would emit if the hack rollout had advantage +1 and the clean rollout had -1. That gradient is algebraically the difference of two teacher-forced NLL gradients, `-grad logp(hack) + grad logp(clean)`, read off the per-module `delta_S` knob. Stack these 21 per-pair difference vectors and SVD per Linear module. The top-k right singular vectors are G_hack. We drop the bottom 25% of singular values per Linear because with only 21 pairs the lower ranks are noise. The one detail that is easy to get wrong is grad isolation: each completion needs its own clean backward, so we zero grads per completion, not per pair.
|
||||
Concretely. For each of 21 pairs (same prompt, hack completion vs clean completion), compute the gradient that GRPO would emit if the hack rollout had advantage +1 and the clean rollout had -1. That gradient is algebraically the difference of two teacher-forced NLL gradients, `-grad logp(hack) + grad logp(clean)`, read off the per-module `delta_S` knob. Stack these 21 per-pair difference vectors and SVD per Linear module. The top-k right singular vectors are G_hack. We then pool every singular value across all modules and drop the bottom 25% globally (a module whose every axis lands below the cut is dropped entirely), because with only 21 pairs the lower ranks are noise and a noisy module should not be protected by having its own "top" direction. The one detail that is easy to get wrong is grad isolation: each completion needs its own clean backward, so we zero grads per completion, not per pair.
|
||||
|
||||
```python
|
||||
def extract_v_hack(model, pairs): # model carries the CURRENT adapter
|
||||
@@ -104,11 +104,12 @@ c_use = relu(c) # one-sided: only remove hack-ward motion
|
||||
g = g - overshoot * (c_use @ v_hack) # overshoot=1.0 = full removal
|
||||
opt.step(g) # main knob delta_S descends what's left
|
||||
|
||||
# route: same split, but the hack-ward part trains a quarantine knob
|
||||
c = v_hack @ g
|
||||
g_main = g - (c @ v_hack) # delta_S learns the orthogonal complement
|
||||
g_hack = c @ v_hack # delta_S_hack absorbs the hack-ward part
|
||||
opt.step(delta_S, g_main); opt.step(delta_S_hack, g_hack)
|
||||
# route: same one-sided split, but the removed part trains a quarantine knob
|
||||
c = v_hack @ g
|
||||
c_use = relu(c) # same one-sided gate as erase
|
||||
removed = c_use @ v_hack
|
||||
opt.step(delta_S, g - removed) # delta_S learns the orthogonal complement
|
||||
opt.step(delta_S_hack, removed) # delta_S_hack absorbs the hack-ward part
|
||||
# at deploy: delta_S_hack := 0 (ablate the quarantine)
|
||||
```
|
||||
|
||||
|
||||
+12
-7
@@ -77,13 +77,18 @@ include the failed pattern.
|
||||
cleanest instantiation of the constraint above: build `v_hack` from only a
|
||||
*subset* of the four loophole modes (the "known" hacks A, B that a weak
|
||||
detector can flag), route on that subspace during training, and measure
|
||||
whether the held-out modes (C, D, never seen by the detector) are also
|
||||
suppressed. The detector is allowed to be weak by design; that is the
|
||||
deployment analogue (known hacks vs unknown hacks). Contrast against the
|
||||
full-detector route arm (`v_hack` from all four modes) to separate how much
|
||||
suppression *transfers* from how much requires the class to be in the basis.
|
||||
This is distinct from the prog_wide pairset, which is broad-by-construction;
|
||||
the weak-detector arm deliberately narrows the basis to test transfer.
|
||||
whether the held-out modes (C, D), which the detector never flagged so they
|
||||
never entered the `v_hack` basis, are also suppressed. The student still
|
||||
trains on all four modes (they are all in the substrate); the detector just
|
||||
never labels C, D for extraction. The detector is allowed to be weak by
|
||||
design; that is the deployment analogue (known hacks vs unknown hacks).
|
||||
Contrast against the full-detector route arm (`v_hack` from all four modes)
|
||||
to separate how much suppression *transfers* from how much requires the
|
||||
class to be in the basis. This is distinct from the prog_wide pairset, which
|
||||
is broad-by-construction; the weak-detector arm deliberately narrows the
|
||||
basis to test transfer. This arm IS the cross-mechanism generalization test
|
||||
(G3, task #107); see docs/spec/20260528_cross_mechanism_v_hack.md for the
|
||||
extraction-from-flagged-rollouts protocol.
|
||||
|
||||
Projection (locked: no magnitude threshold; one-sided clip stays — see note):
|
||||
|
||||
|
||||
Reference in New Issue
Block a user