docs: fix review findings (global noise-floor, route one-sided, G3 xref)

External review (3 subagents) caught:
- blog: noise-floor drop is GLOBAL across modules, not per-Linear (proj.py:187)
- blog: route pseudocode used full c; route actually uses the same one-sided
  gate as erase and quarantines the identical 'removed' vector (proj.py:124,199)
- spec: 'never seen by detector' -> clarify student trains on all 4 modes, the
  detector just never labels C/D for v_hack extraction; cross-ref G3/task #107

Dismissed: reviewer claim that only exit_code survived (stale spec; live log
columns hk_rt/hk_so/hk_se/hk_fm confirm 4 modes) and a hallucinated 'Furthermore'.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-05-31 00:41:12 +00:00
parent f7288e569d
commit d781b56ff4
2 changed files with 19 additions and 13 deletions
@@ -63,7 +63,7 @@ The whole intervention in three lines:
- we isolate that direction from a handful of (hack, clean) example pairs
- during each training update we project that direction out of the gradient before the optimizer applies it
Concretely. For each of 21 pairs (same prompt, hack completion vs clean completion), compute the gradient that GRPO would emit if the hack rollout had advantage +1 and the clean rollout had -1. That gradient is algebraically the difference of two teacher-forced NLL gradients, `-grad logp(hack) + grad logp(clean)`, read off the per-module `delta_S` knob. Stack these 21 per-pair difference vectors and SVD per Linear module. The top-k right singular vectors are G_hack. We drop the bottom 25% of singular values per Linear because with only 21 pairs the lower ranks are noise. The one detail that is easy to get wrong is grad isolation: each completion needs its own clean backward, so we zero grads per completion, not per pair.
Concretely. For each of 21 pairs (same prompt, hack completion vs clean completion), compute the gradient that GRPO would emit if the hack rollout had advantage +1 and the clean rollout had -1. That gradient is algebraically the difference of two teacher-forced NLL gradients, `-grad logp(hack) + grad logp(clean)`, read off the per-module `delta_S` knob. Stack these 21 per-pair difference vectors and SVD per Linear module. The top-k right singular vectors are G_hack. We then pool every singular value across all modules and drop the bottom 25% globally (a module whose every axis lands below the cut is dropped entirely), because with only 21 pairs the lower ranks are noise and a noisy module should not be protected by having its own "top" direction. The one detail that is easy to get wrong is grad isolation: each completion needs its own clean backward, so we zero grads per completion, not per pair.
```python
def extract_v_hack(model, pairs): # model carries the CURRENT adapter
@@ -104,11 +104,12 @@ c_use = relu(c) # one-sided: only remove hack-ward motion
g = g - overshoot * (c_use @ v_hack) # overshoot=1.0 = full removal
opt.step(g) # main knob delta_S descends what's left
# route: same split, but the hack-ward part trains a quarantine knob
c = v_hack @ g
g_main = g - (c @ v_hack) # delta_S learns the orthogonal complement
g_hack = c @ v_hack # delta_S_hack absorbs the hack-ward part
opt.step(delta_S, g_main); opt.step(delta_S_hack, g_hack)
# route: same one-sided split, but the removed part trains a quarantine knob
c = v_hack @ g
c_use = relu(c) # same one-sided gate as erase
removed = c_use @ v_hack
opt.step(delta_S, g - removed) # delta_S learns the orthogonal complement
opt.step(delta_S_hack, removed) # delta_S_hack absorbs the hack-ward part
# at deploy: delta_S_hack := 0 (ablate the quarantine)
```
+12 -7
View File
@@ -77,13 +77,18 @@ include the failed pattern.
cleanest instantiation of the constraint above: build `v_hack` from only a
*subset* of the four loophole modes (the "known" hacks A, B that a weak
detector can flag), route on that subspace during training, and measure
whether the held-out modes (C, D, never seen by the detector) are also
suppressed. The detector is allowed to be weak by design; that is the
deployment analogue (known hacks vs unknown hacks). Contrast against the
full-detector route arm (`v_hack` from all four modes) to separate how much
suppression *transfers* from how much requires the class to be in the basis.
This is distinct from the prog_wide pairset, which is broad-by-construction;
the weak-detector arm deliberately narrows the basis to test transfer.
whether the held-out modes (C, D), which the detector never flagged so they
never entered the `v_hack` basis, are also suppressed. The student still
trains on all four modes (they are all in the substrate); the detector just
never labels C, D for extraction. The detector is allowed to be weak by
design; that is the deployment analogue (known hacks vs unknown hacks).
Contrast against the full-detector route arm (`v_hack` from all four modes)
to separate how much suppression *transfers* from how much requires the
class to be in the basis. This is distinct from the prog_wide pairset, which
is broad-by-construction; the weak-detector arm deliberately narrows the
basis to test transfer. This arm IS the cross-mechanism generalization test
(G3, task #107); see docs/spec/20260528_cross_mechanism_v_hack.md for the
extraction-from-flagged-rollouts protocol.
Projection (locked: no magnitude threshold; one-sided clip stays — see note):