From 1d105a93a45411634c101b828bd3d09fa75ec536 Mon Sep 17 00:00:00 2001
From: wassname <github@wassname>
Date: Mon, 1 Jun 2026 01:44:31 +0000
Subject: [PATCH] review: 3-model external panel on route2 pseudocode +
 synthesis

DeepSeek/GPT-5.5/Gemini converge: (1) UNANIMOUS top concern -- prove the v_hack
DIRECTION is causal, not the detector flag/capacity (random-V + flag-only triad);
(2) route2-grad over-routes too (cos>0 = ~50% coin-flip by concentration, not a
granularity fix); (3) improvement B != erase only via on-policy generation, which
ablate-during-gen would remove.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
---
 .../20260601_sgtm_pseudocode/SYNTHESIS.md     | 85 +++++++++++++++++++
 .../20260601_sgtm_pseudocode/deepseek.md      | 35 ++++++++
 .../20260601_sgtm_pseudocode/gemini35.md      | 54 ++++++++++++
 .../reviews/20260601_sgtm_pseudocode/gpt55.md | 76 +++++++++++++++++
 4 files changed, 250 insertions(+)
 create mode 100644 docs/reviews/20260601_sgtm_pseudocode/SYNTHESIS.md
 create mode 100644 docs/reviews/20260601_sgtm_pseudocode/deepseek.md
 create mode 100644 docs/reviews/20260601_sgtm_pseudocode/gemini35.md
 create mode 100644 docs/reviews/20260601_sgtm_pseudocode/gpt55.md

diff --git a/docs/reviews/20260601_sgtm_pseudocode/SYNTHESIS.md b/docs/reviews/20260601_sgtm_pseudocode/SYNTHESIS.md
new file mode 100644
index 0000000..90c0578
--- /dev/null
+++ b/docs/reviews/20260601_sgtm_pseudocode/SYNTHESIS.md
@@ -0,0 +1,85 @@
+# External review synthesis — route2 pseudocode (2026-06-01)
+
+Three frontier models, different providers (DeepSeek v4-pro, GPT-5.5, Gemini
+3.5-flash), reviewing `docs/grad_routing/sgtm_vs_ours.md`. Writer was Claude, so
+all three decorrelate. They converged strongly. Raw reviews alongside this file.
+
+## The existential one (UNANIMOUS, 3/3) — is the VECTOR even doing the work?
+
+All three independently made this their top confound / single-most-important-fix:
+the gain may come from the **weak-detector flag** (which rollouts) and/or from
+**deleting capacity**, NOT from the extracted v_hack **direction**. If so, our
+whole premise ("route by a vector, not a label") is unsupported and the method
+reduces to noisy label-guided regularization (≈ advantage masking).
+
+- DeepSeek (single most important fix): "Clarify and prove the extracted hack
+  direction is causally responsible for the separation, not the weak-detector
+  label."
+- GPT-5.5 (confound 1): label-only routing, random-vector + same flags,
+  real-vector + no-label — the three-way ablation.
+- Gemini (confound 1+2): random orthogonal V of matched norm; benign data into a
+  dummy quarantine of identical size.
+
+**Decisive control (promote from the plan's "deferred" list to #1):** a triad at
+matched rank/norm/seed —
+  (a) real v_hack + detector flag,
+  (b) random-orthonormal V + same flag,
+  (c) flag-only advantage masking (no vector at all).
+If (b) ≈ (a), the direction is irrelevant (capacity/regularization artifact). If
+(c) ≈ (a), the vector adds nothing over the label. We must beat both to justify
+the vector. We've never run this; it's higher value than any further route2 tuning.
+
+## Q1 — improvement B vs erase: different, but implementation-dependent (3/3 differ)
+
+GPT-5.5 was sharpest:
+- **B as written** (detach complement for flagged rollouts → grad flows only into
+  the deleted v_hack axes) → flagged rollouts contribute ZERO to the deployed
+  model. Stronger than erase; it's label-masking, not directional routing.
+- **B keeping the ⊥ part** (`Qg`), routing only `Pg` to deleted axes → deployed
+  update is **algebraically identical to erase**.
+- The ONLY real difference: the forget axes being **active during generation**
+  shift on-policy rollouts (the Cloud et al. "routing > erasing" benefit). **If
+  you ablate the quarantine during generation (improvement C), B collapses to
+  erase.** => B and C are in tension; can't get the routing benefit and remove
+  the feedback risk at once.
+
+## Q2 — route2-grad over-routes too (3/3)
+
+Per-rollout aggregation does NOT fix it. `cos(g_b, v_grad) > 0` on a near-
+orthogonal high-dim distribution is a ~50% coin flip at the rollout level
+(concentration of measure). It trades per-token coin-flips for per-rollout ones.
+GPT-5.5 adds: SVD sign is arbitrary (cos>0 meaningless unless orientation
+anchored by labeled hack/clean grads — we DO anchor via the contrastive pairs,
+partial credit); aggregate dominated by length/advantage. DeepSeek adds: a mixed
+hack/clean rollout has cos≈0 and is miscategorised. **Prediction: job 47 (route2-
+grad) over-routes like job 46 did.** Watch its collapse/qE.
+
+The granularity (token vs rollout) was never the core bug. The **uncalibrated
+cosine gate** is.
+
+## Convergent fix
+
+Kill the `cos>0` gate. Route by the **weak-detector flag** (which rollouts go to
+quarantine); use v_hack **only to define the deleted subspace** (where the routed
+gradient goes). Keeps the vector premise, removes the coin-flip. (But see the
+existential control: must still prove the subspace choice beats random-V.)
+
+## Other agreed points
+
+- n-seeds: GRPO is high-variance; reviewers want 8 (Gemini), 8-10 (GPT-5.5), 30
+  (DeepSeek). We run n=1. All headline claims need error bars.
+- Add a hack-adjacent / benign-but-complex eval to catch over-suppression of
+  legitimate high-reward generations (SGTM's "forget-adjacent").
+- Add a relearn/undiscovered-rate probe: after deploy-ablation, run a few GRPO
+  steps and see if the hack re-emerges (shallow hiding vs real prevention).
+- Concrete UAT thresholds (what `cin_t > cin_s`, "coherent", "generalises" mean
+  numerically + CIs), evaluated on the DEPLOYED (ablated) model.
+- DeepSeek's alt ideas that bypass routing (note as baselines we'd compare to,
+  not adopt — they intervene at reward, which is the Rebound/Wu&Tang lane we
+  differ from): v_hack-aligned reward penalty; flag-based advantage masking.
+
+## Bottom line
+
+The reviews don't kill the idea, but they relocate the crux. Before more route2
+engineering, the project needs the **vector-vs-label-vs-random-V control** to
+show the extracted direction is load-bearing at all. That's the next experiment.
diff --git a/docs/reviews/20260601_sgtm_pseudocode/deepseek.md b/docs/reviews/20260601_sgtm_pseudocode/deepseek.md
new file mode 100644
index 0000000..dab3d7f
--- /dev/null
+++ b/docs/reviews/20260601_sgtm_pseudocode/deepseek.md
@@ -0,0 +1,35 @@
+## Scientist Review
+
+### Central assumption
+The hack direction extracted from SVD-of-W gradient differences accurately partitions the model’s representational space so that cosine‑routing separates hack from non‑hack activations. **Holds when:** the hack manifests as a well‑separated, stable subspace across examples (e.g., a distinct pattern in MLP outputs). **Fails when:** the extracted direction is noisy or common (cos ~0), causing random routing that leaks gradient into the wrong subspace and degrades generation quality.
+
+### Confounds
+1. **Extracted direction is not discriminative** (cos≈0 on most activations). → *Control:* sweep detection threshold against a calibrated TPR/FPR detector; if the direction cannot beat a random projection, routing collapses.
+2. **Success depends on the weak‑detector flag, not the vector route.** The quarantine may be learning purely from the (clean) rollout labels, not the vector, making the vector redundant. → *Control:* run a “blind” quarantine that ignores the vector and routes by the detector flag alone; compare forgetting performance.
+3. **Over‑routing remediation may be due to reduced update size, not routing.** route2‑grad aggregates gradients over the rollout, effectively reducing the noise in the cos signal. → *Control:* train with the same rollout‑level flag but a dummy vector (e.g., random direction); if performance is similar, the vector is irrelevant.
+
+### Algorithm issues
+- **Incomplete specification for improvement (B):** The description “detach the complement so its gradient flows ONLY into the v_hack‑subspace component” fails to define the projection operator, how to preserve forward value during detach, and whether the v_hack basis is orthogonalised. Without orthonormal projections, the detached complement may leak into the hack subspace.
+- **Gradient routing vs. erase (Q1):** Mathematically, routing differs from erase. For a hack rollout with advantage A, the effective gradient applied to ΔW in routing is \(g_{\text{route}} = P_{\text{hack}} \, g\) where \(P_{\text{hack}}\) projects onto the hack subspace. Erase applies \(g_{\text{erase}} = (I - P_{\text{hack}}) g\). These are not equivalent (unless A=0). The “delete at deploy” step is necessary to remove the temporarily‑trained behaviour, but the key benefit is that the model **uses** the hack during training (so policy‑generated data stays on‑behaviour) – a benefit erase discards. So routing is different and can prevent collapse in a way erase cannot.
+- **route2‑grad (Q2) failure modes:** Aggregating cosine over a rollout avoids per‑token over‑routing, but still relies on a single threshold for the entire twist. A rollout with half hack‑tokens and half normal tokens may have cos≈0 and be miscategorised; multiple hack patterns could cause false negatives. Furthermore, if a hack rollout consists of many neutral tokens and a few hack tokens, the aggregate may miss it. The method only shifts the granularity, not the underlying discrimination problem.
+
+### Experimental design
+- **Falsifiable:** Partially – the routing hypothesis is falsifiable if disabling vector routing (and using only the weak‑detector flag) yields equivalent separation. A concrete metric is whether the quarantine (`qE`) absorbs exactly the hack‑induced gradient energy.
+- **n needed:** GRPO is high‑variance; at least **30** seeds with error bars to distinguish differences, especially for the vector‑vs‑flag ablations.
+- **UAT gap:** No explicit “done” criterion (e.g., quarantine gradient energy < ε). Also lacks a regression test: after quarantine removal, re‑run a limited GRPO to see if the hack re‑emerges (undiscovered rate).
+
+### Section verdicts
+- **SGTM gradient routing (theirs):** Sound – label‑driven, well‑controlled.
+- **route2‑act pseudocode:** Likely‑wrong – per‑token cosine routing with zero‑mean signal and no threshold causes indiscriminate activation.
+- **route2‑grad proposal:** Dubious – improves over per‑token but still depends on the rented v_hack being a good discriminator in aggregate.
+- **Improvement (B) (single‑param routing):** Sound in principle, but projection math must be made explicit; deployment zeroing must be validated.
+
+### Specific Q answers
+**Q1:** No, it does not collapse to the same update. Algebra: for a hack rollout, erase sets gradient to zero in the hack direction (\(g_{\text{eff}} = g - vv^T g\)); routing sets \(g_{\text{eff}} = vv^T g\). They differ by the retained component update. In GRPO this means routing preserves on‑policy adaptivity, crucial for the quar‑not‑so‑clean loop.
+
+**Q2:** Aggregating cos over a rollout reduces the probability of random token‑level firing, but the failure mode moves to the rollout level: a direction with near‑zero mean projection will still cause random (mis)classification. Moreover, if hack‑token density varies within a rollout, a single threshold discards important detail. Over‑routing can simply re‑appear at the sequence level if the v_hack is non‑specific.
+
+**Q3:** The main pseudocode lacks a protocol for handling multiple hack directions and across layers. A better idea within the given constraints: **convert the hack direction into a reward shaping term** – penalise the policy when it generates tokens that align with the hack subspace, rather than trying to silently route gradient. This avoids quarantine blow‑up and provides direct feedback, though at the cost of potentially hurting performance. Another: use the weak‑detector flag to directly mask the advantage (i.e., negative advantage for hack‑flagged trajectories) – a simpler, more transparent adaptive‑penalty method that ignores the vector routing entirely.
+
+### Single most important fix
+Clarify and prove that the extracted hack direction is **causally responsible** for the separation, not the weak‑detector label. Without this, the entire “vector routing” claim disconnects from the actual mechanism, and the method reduces to noisy label‑guided regularization.
\ No newline at end of file
diff --git a/docs/reviews/20260601_sgtm_pseudocode/gemini35.md b/docs/reviews/20260601_sgtm_pseudocode/gemini35.md
new file mode 100644
index 0000000..4ba4047
--- /dev/null
+++ b/docs/reviews/20260601_sgtm_pseudocode/gemini35.md
@@ -0,0 +1,54 @@
+## Scientist Review
+
+### Central assumption
+Linear representability of reward-hacking features in weight SVD space | holds when: reward-hacking patterns rely on a localized, low-rank subspace disjoint from standard logical reasoning pathways | fails when: hacking pathways share polysemantic representations with valid code/reasoning steps, leading to catastrophic collateral damage upon deletion.
+
+### Confounds
+1. **Capacity starvation / Parameter damping** → control: Route benign data into a dummy quarantined cohort of identical parameter size (33M). If performance drops similarly, the quarantine acts as a crude regularizer, not a targeted concept filter.
+2. **Gradient noise suppression** → control: Replace $v_{hack}$ with a random orthogonal vector of equivalent norm. If the quarantine still improves optimization stability, the routing mechanism is merely dampening high-variance gradients rather than isolating the hack direction.
+
+### Algorithm issues
+
+#### Q1: Gradient Routing (B) vs. Erase Algebra
+Let $P_{\parallel} = vv^T$ and $P_{\perp} = I - vv^T$ be orthogonal projection operators. Let the gradient at step $t$ be $g^{(t)} = g_{clean}^{(t)} + g_{hack}^{(t)}$. 
+
+**Erase Mechanism**:
+We apply $P_{\perp}$ directly to the overall gradient at each step. Assuming SGD for clean algebraic transparency:
+$$\Delta S^{(t+1)} = \Delta S^{(t)} - \eta P_{\perp} \left( g_{clean}^{(t)} + g_{hack}^{(t)} \right)$$
+Since $\Delta S^{(0)} = 0$, the deployed weights at step $T$ are:
+$$\Delta S_{deploy, erase}^{(T)} = -\eta \sum_{t=0}^{T-1} P_{\perp} \left( g_{clean}^{(t)} + g_{hack}^{(t)} \right)$$
+
+**Route Mechanism (Improvement B)**:
+We split the weight space into $\Delta S = \Delta S_{keep} + \Delta S_{forget}$, with $\Delta S_{keep} \in \operatorname{Im}(P_{\perp})$ and $\Delta S_{forget} \in \operatorname{Im}(P_{\parallel})$. 
+
+During the forward pass of a flagged hack rollout, we detach the complement $\Delta S_{keep}$. Thus, $\nabla_{\Delta S_{keep}} L_{hack}^{(t)} = 0$. The clean rollouts do not detach $\Delta S_{keep}$. Therefore:
+$$\Delta S_{keep}^{(t+1)} = \Delta S_{keep}^{(t)} - \eta P_{\perp} g_{clean}^{(t)}$$
+Since the deployed model only retains the kept parameters ($\Delta S_{deploy, route}^{(T)} = \Delta S_{keep}^{(T)}$):
+$$\Delta S_{deploy, route}^{(T)} = -\eta \sum_{t=0}^{T-1} P_{\perp} g_{clean}^{(t)}$$
+
+**Verdict on Q1**: They are **algebraically different**. 
+1. **Gradient Poisoning**: Erase leaks the non-hack component of hack gradients ($P_{\perp} g_{hack}^{(t)}$) into the deployed model. Route completely insulates the clean parameters from hack rollout dynamics.
+2. **Policy Distribution Shift**: Under Route, the on-policy generator utilizes the active hack parameters ($\Delta S_{forget}$) during training rollouts, keeping generations stable. Erase starves the generator of this pathway, causing the policy to violently shift to find other hacking bypasses.
+
+#### Q2: Rollout Vector Routing (`route2-grad`) Failure Mode
+`route2-grad` does **not** avoid over-routing. In high-dimensional space ($d \gg 10^3$), the concentration of measure dictates that any random gradient vector $g_b$ is nearly orthogonal to the static target $v_{grad}$ ($\cos(g_b, v_{grad}) \approx 0$). 
+
+Applying a hard sign-gate ($\cos > 0$) on this orthogonal distribution acts as an unbiased $50\%$ coin flip. It will route half of all benign rollouts to the quarantine. Rollout-level routing will collapse to token-salad precisely like its per-token predecessor unless the threshold is dynamically calibrated above the noise floor.
+
+#### Q3: Pseudocode Flaws & Fixes
+- **Flaws**: The additive quarantine $A_q/B_q$ lacks norm initialization scaling, causing immediate step-size mismatch when summed to $y$. In the routing step, calculating $v_{hack}$ dynamically on pairwise gradients during GRPO introduces high staleness because the policy updates continuously.
+- **Better Idea**: Cast routing as a probabilistic gate. Route rollouts using the binary weak-detector flag, and use the SVD vector *only* to project the update direction, omitting cosine similarity checks entirely during training.
+
+### Experimental design
+- Falsifiable: Yes — if post-ablation the model can still easily elicit the hack behavior under basic prompt-injection or if clean capabilities collapse, the method fails.
+- n needed: $n \ge 8$ seeds due to high variance in GRPO rollout advantages.
+- UAT gap: Missing a "hack-adjacent" (benign but complex) validation set to verify that we are not over-suppressing honest, highly optimized generations.
+
+### Section verdicts
+- **SGTM (Theirs)**: Sound — Physically partitioning dimensions avoids vector collision and feedback loops.
+- **route2-act (Ours)**: Likely-wrong — Token-level cosine checks in high dimensions collapse to random coin flips.
+- **route2-grad**: Dubious — Better temporal aggregation, but still highly vulnerable to orthogonal concentration.
+- **Improvement B**: Sound — Algebraically isolates core updates while stabilizing on-policy generation.
+
+### Single most important fix
+Kill the uncalibrated cosine gate ($\cos > 0$); route training updates using the **weak-detector rollout flag** as the gate, and use the SVD $v_{hack}$ direction solely to define the projection subspace.
\ No newline at end of file
diff --git a/docs/reviews/20260601_sgtm_pseudocode/gpt55.md b/docs/reviews/20260601_sgtm_pseudocode/gpt55.md
new file mode 100644
index 0000000..93d7f76
--- /dev/null
+++ b/docs/reviews/20260601_sgtm_pseudocode/gpt55.md
@@ -0,0 +1,76 @@
+## Scientist Review
+
+### Central assumption
+The extracted SVD direction `v_hack` must identify a stable hack-learning gradient subspace, not just a noisy reward/prompt/length correlate.  
+Holds when: hack rollouts share a consistent parameter-gradient direction across prompts/seeds/modules, and legitimate high-reward behavior is mostly orthogonal.  
+Fails when: `v_hack` aligns with generic reward-seeking, rollout length, difficulty, or malformed generations; then routing/deletion appears to suppress hacking but actually deletes broad capability or exploits detector artifacts.
+
+### Confounds
+1. Weak-detector gating does the work, not the vector → control: label-only routing, random-vector routing with same flagged rollouts, and real-vector/no-label routing.
+2. Deleting capacity reduces hacks by damaging the policy → control: delete a random SVD subspace with matched norm/rank; report solve/coherence/legit near-hack performance.
+3. GRPO instability / generation-distribution shift masquerades as routing benefit → control: same LR, same ablation-during-generation setting, compare route-B vs erase vs no-route across seeds.
+
+Minimal negative control: random signed SVD direction or sign-shuffled `v_hack` with identical detector flags and deletion budget.
+
+### Algorithm issues
+- **Q1 / Improvement B algebra** — Let `P` project onto the `v_hack` SVD axes, `Q=I-P`. For rollout `i`, single-step GRPO with ratio 1 is just detached-advantage NLL:
+
+  \[
+  \ell_i(\theta)=-A_i\sum_t \log \pi_\theta(a_{it}|s_{it}),\quad
+  g_i=\nabla_\theta \ell_i=-A_i\sum_t \nabla_\theta\log\pi_\theta.
+  \]
+
+  Decompose `delta_S` as \(\theta=\theta_k+\theta_h\), with \(\theta_h=P\theta\), \(\theta_k=Q\theta\).
+
+  **Erase** on flagged rollouts gives deployed update:
+
+  \[
+  \Delta\theta_{\text{deploy}}=-\eta Q g_i.
+  \]
+
+  **Route-B as written** says: for flagged rollouts, detach complement so gradient flows only into `v_hack` axes:
+
+  \[
+  \nabla_{\theta_k}\ell_i(\theta_k^{sg}+\theta_h)=0,\quad
+  \nabla_{\theta_h}\ell_i=P g_i.
+  \]
+
+  After deploy deletion of \(\theta_h\), the flagged rollout contributes:
+
+  \[
+  \Delta\theta_{\text{deploy}}=0.
+  \]
+
+  So as written it is **not erase**; it is stronger: flagged rollouts do not update the deployed parameters at all. It equals erase only if \(Qg_i\approx0\), i.e. all useful flagged gradient lies in the hack subspace.
+
+  If instead you implement “keep \(Qg_i\), route only \(Pg_i\) to deleted axes,” then the deployed one-step update is algebraically identical to erase:
+
+  \[
+  \theta_k \leftarrow \theta_k-\eta Qg_i,\quad
+  \theta_h\leftarrow\theta_h-\eta Pg_i,\quad
+  \text{deploy } Q\theta.
+  \]
+
+  The only possible difference then is indirect: \(\theta_h\) changes future on-policy rollouts if it is active during generation. If quarantine/deleted axes are ablated during generation, even that difference disappears.
+
+- **route2-grad attribution** — “subtract their share from `delta_S.grad` post-backward” is underspecified. A scalar batch backward loses per-rollout gradient decomposition unless you compute per-rollout grads, use hooks/masks, or do separate backward passes. With GRPO group-normalized advantages, attribution is additionally coupled through batch/group statistics, even if advantages are detached.
+
+- **SVD direction sign** — SVD directions are sign-arbitrary. A threshold `cos(g_b, v_grad)>0` is meaningless unless orientation is anchored by labeled hack-vs-clean gradients. Otherwise sign flip routes the opposite half.
+
+- **Projection/cos normalization** — Rollout-level cosine avoids token coin-flip only if the aggregate gradient has meaningful signal-to-noise. In high dimension, near-zero cosines will still route roughly half of rollouts by sign. Aggregation can be dominated by rollout length, advantage magnitude, prompt type, or common language-model gradients. It trades per-token over-routing for all-or-nothing false-positive rollouts.
+
+### Experimental design
+- Falsifiable: partially. It becomes falsifiable only with deployed-after-deletion metrics versus erase, random-subspace deletion, and label-only routing.
+- n needed: at least 8–10 seeds per condition for GRPO stability claims, plus ~1k held-out rollouts per condition or several hundred per mode for hack/solve/coherence rates.
+- UAT gap: not concrete. “coherent,” “generalisation holds,” and `cin_t > cin_s` need fixed thresholds, CIs, and deployed-model evaluation after deletion.
+
+### Section verdicts
+- SGTM comparison: sound — correctly identifies label routing plus reserved dims as the major difference.
+- route2-act diagnosis: sound — cosine near zero causing half-token routing is a fatal mechanism.
+- route2-grad: dubious — rollout aggregation reduces token noise but can still route half of rollouts under weak/noisy cosines.
+- Improvement B: dubious — depending on implementation, it is either equivalent to erase for deployed params or ignores flagged gradients entirely.
+- Break generation feedback loop: sound but removes the main claimed benefit of train-time quarantine if the deleted axes never affect rollouts.
+- Noisy-detector sweep / near-hack eval / relearn probe: sound controls.
+
+### Single most important fix
+Specify and test the exact deployed-parameter update of Improvement B against erase: if deployed \(Q\theta\) receives \(Qg\), it is erase; if it receives zero on flagged rollouts, it is label-masking, not directional routing.
\ No newline at end of file