diff --git a/docs/blog/20260529_gradient_projection_vs_reward_hacking_LW_draft.md b/docs/blog/20260529_gradient_projection_vs_reward_hacking_LW_draft.md index 9ddacc9..986f7f0 100644 --- a/docs/blog/20260529_gradient_projection_vs_reward_hacking_LW_draft.md +++ b/docs/blog/20260529_gradient_projection_vs_reward_hacking_LW_draft.md @@ -63,7 +63,7 @@ The whole intervention in three lines: - we isolate that direction from a handful of (hack, clean) example pairs - during each training update we project that direction out of the gradient before the optimizer applies it -Concretely. For each of 21 pairs (same prompt, hack completion vs clean completion), compute the gradient that GRPO would emit if the hack rollout had advantage +1 and the clean rollout had -1. That gradient is algebraically the difference of two teacher-forced NLL gradients, `-grad logp(hack) + grad logp(clean)`, read off the per-module `delta_S` knob. Stack these 21 per-pair difference vectors and SVD per Linear module. The top-k right singular vectors are G_hack. We drop the bottom 25% of singular values per Linear because with only 21 pairs the lower ranks are noise. The one detail that is easy to get wrong is grad isolation: each completion needs its own clean backward, so we zero grads per completion, not per pair. +Concretely. For each of 21 pairs (same prompt, hack completion vs clean completion), compute the gradient that GRPO would emit if the hack rollout had advantage +1 and the clean rollout had -1. That gradient is algebraically the difference of two teacher-forced NLL gradients, `-grad logp(hack) + grad logp(clean)`, read off the per-module `delta_S` knob. Stack these 21 per-pair difference vectors and SVD per Linear module. The top-k right singular vectors are G_hack. We then pool every singular value across all modules and drop the bottom 25% globally (a module whose every axis lands below the cut is dropped entirely), because with only 21 pairs the lower ranks are noise and a noisy module should not be protected by having its own "top" direction. The one detail that is easy to get wrong is grad isolation: each completion needs its own clean backward, so we zero grads per completion, not per pair. ```python def extract_v_hack(model, pairs): # model carries the CURRENT adapter @@ -104,11 +104,12 @@ c_use = relu(c) # one-sided: only remove hack-ward motion g = g - overshoot * (c_use @ v_hack) # overshoot=1.0 = full removal opt.step(g) # main knob delta_S descends what's left -# route: same split, but the hack-ward part trains a quarantine knob -c = v_hack @ g -g_main = g - (c @ v_hack) # delta_S learns the orthogonal complement -g_hack = c @ v_hack # delta_S_hack absorbs the hack-ward part -opt.step(delta_S, g_main); opt.step(delta_S_hack, g_hack) +# route: same one-sided split, but the removed part trains a quarantine knob +c = v_hack @ g +c_use = relu(c) # same one-sided gate as erase +removed = c_use @ v_hack +opt.step(delta_S, g - removed) # delta_S learns the orthogonal complement +opt.step(delta_S_hack, removed) # delta_S_hack absorbs the hack-ward part # at deploy: delta_S_hack := 0 (ablate the quarantine) ``` diff --git a/docs/spec.md b/docs/spec.md index 2e96f19..21e17c8 100644 --- a/docs/spec.md +++ b/docs/spec.md @@ -77,13 +77,18 @@ include the failed pattern. cleanest instantiation of the constraint above: build `v_hack` from only a *subset* of the four loophole modes (the "known" hacks A, B that a weak detector can flag), route on that subspace during training, and measure -whether the held-out modes (C, D, never seen by the detector) are also -suppressed. The detector is allowed to be weak by design; that is the -deployment analogue (known hacks vs unknown hacks). Contrast against the -full-detector route arm (`v_hack` from all four modes) to separate how much -suppression *transfers* from how much requires the class to be in the basis. -This is distinct from the prog_wide pairset, which is broad-by-construction; -the weak-detector arm deliberately narrows the basis to test transfer. +whether the held-out modes (C, D), which the detector never flagged so they +never entered the `v_hack` basis, are also suppressed. The student still +trains on all four modes (they are all in the substrate); the detector just +never labels C, D for extraction. The detector is allowed to be weak by +design; that is the deployment analogue (known hacks vs unknown hacks). +Contrast against the full-detector route arm (`v_hack` from all four modes) +to separate how much suppression *transfers* from how much requires the +class to be in the basis. This is distinct from the prog_wide pairset, which +is broad-by-construction; the weak-detector arm deliberately narrows the +basis to test transfer. This arm IS the cross-mechanism generalization test +(G3, task #107); see docs/spec/20260528_cross_mechanism_v_hack.md for the +extraction-from-flagged-rollouts protocol. Projection (locked: no magnitude threshold; one-sided clip stays — see note):