From 05731cc0e43584aa48c74a65b9f165fb0ec9546a Mon Sep 17 00:00:00 2001 From: wassname Date: Wed, 3 Jun 2026 00:59:24 +0000 Subject: [PATCH] paper: drop reader-facing route2 version tag; flag SGTM-not-Cloud lineage - route2 -> route in all prose/captions/tables (route2 stays in % provenance comments as the run-tag). A reader does not care about the version number. - title: steering-vector framing; recorded naming reasoning as a comment (do NOT claim label-free -- our pairs ARE labels; the backable scoped claim is held-out hacks suppressed with zero labels of their own, earnable by A5). - FLAG at contribution 1: our mechanism is SGTM-style post-backward parameter- gradient masking, NOT Cloud's activation-level gradient routing. Author-verbatim claim left intact but flagged inline; see docs/papers/grad_routing/sgtm_vs_ours.md. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com> --- docs/writeup/main.tex | 64 +++++++++++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 17 deletions(-) diff --git a/docs/writeup/main.tex b/docs/writeup/main.tex index e7fa918..a72a975 100644 --- a/docs/writeup/main.tex +++ b/docs/writeup/main.tex @@ -20,7 +20,28 @@ % TODO-marker: renders red in the PDF and is grep-able by `just paper-qc`. \newcommand{\TODO}[1]{{\color{red}\textbf{[TODO: #1]}}} -\title{Gradient Routing Against Reward Hacking \TODO{title}} +% Title states the mechanism (route by a direction). Naming decisions (AFK +% 2026-06-03): +% - DROP "route2" in prose -- it's an internal version tag; readers don't care +% about the 2. The arm is "route"; "directional/few-shot routing" if a fuller +% name is wanted. +% - Lineage we CAN claim: extraction is representation engineering (contrastive +% pairs -> difference direction, Zou et al.) but read in GRADIENT/SVD space; +% application is gradient routing (Cloud et al.) with that direction as the +% mask instead of per-token data labels. We sit at the RepE x grad-routing +% intersection. Fine to say "representation engineering in gradient space"; +% do NOT imply we steer activations (we route gradients). +% - "label-free"/"without labels": globally FALSE -- our ~10-21 pairs + the +% known-mode teacher anchors + the live E detector are all labels on KNOWN +% hacks. The BACKABLE, scoped claim is "suppresses the HELD-OUT hacks with +% zero labels of their own", valid iff the held-out modes never trip the live +% E detector (E = run_tests loophole; side-channel modes sentinel/stdout/file +% are graded elsewhere, so pick those as held-out). Earn it with A5 (job 95 +% harvest -> scripts/pairs_from_rollouts.py 2 known modes -> held-out route +% run -> held-out deploy hack ~0) AND a hacked_E-by-mode==0 check on held-out. +% If A5 lands clean, upgrade title to: +% "Routing Reward Hacking out of RL Gradients -- Suppressing Hacks We Never Labeled" +\title{Routing Reward Hacking out of RL Gradients with a Steering Vector} % Anonymous for submission. Add \nipsfinalcopy + real authors for camera-ready. \author{Anonymous Author(s)\\ Affiliation\\ \texttt{email}} @@ -44,6 +65,15 @@ scoring time; (3) at deploy some hacks are unknown; (4) here we route the GRPO gradient away from a weak-detector hack direction.} \paragraph{Contributions.} % author-dictated; factual claims, keep verbatim. +% FLAG (AFK 2026-06-03, do not silently rewrite -- author's call): contribution 1 +% says "gradient routing (Cloud)", but our MECHANISM is SGTM-style post-backward +% PARAMETER-gradient masking (antipasto.py:73 sums the quarantine, no forward +% detach; train.py parks the flagged grad into delta_S_hack.grad). Cloud's +% gradient routing is activation-level .detach(). So the closer relative is +% sgtm2025localization (also: their TPR/FPR detector knob == our no-cheat weak +% detector). Suggest reframing to "extend SGTM-style parameter-gradient masking +% (and the localize-then-ablate idea of gradient routing) with a learned hack +% direction". See docs/papers/grad_routing/sgtm_vs_ours.md. \begin{enumerate} \item We extend gradient routing \citep{cloud2024gradientrouting} to reward hacking in RL post-training. @@ -99,7 +129,7 @@ keep the live gradient out of $v_{\text{hack}}$, and two extraction schedules:} \begin{itemize} \item \emph{erase} (one-sided): subtract the $v_{\text{hack}}$ component from the live $\delta_S$ gradient; the optimizer steps on the complement. - \item \emph{route} (route2): a per-rollout gate $\cos(g,v)>\tau$ ($\tau$ + \item \emph{route}: a per-rollout gate $\cos(g,v)>\tau$ ($\tau$ calibrated each step from the hack-vs-clean cosine gap) sends the whole rollout gradient into a scale-matched, distinct-basis quarantine knob $\delta_{S,\text{hack}}$, deleted at deploy. Gradient routing @@ -121,7 +151,7 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.} % =================================================================== \section{Results} -\subsection{C1: route2 vs vanilla deploy hack/solve (keynote)} +\subsection{C1: route vs vanilla deploy hack/solve (keynote)} % --- Figure: keynote dynamics ----------------------------------------------- % Provenance: out/figs/dyn_sub4_hack_overlay.png, generated by `just dyn` @@ -133,10 +163,10 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.} \begin{figure}[t] \centering \includegraphics[width=0.85\linewidth]{figs/dyn_sub4_hack_overlay.png} - \caption{Hack rate (top) and solve rate (bottom) over GRPO training, route2 + \caption{Hack rate (top) and solve rate (bottom) over GRPO training, route ($n{=}3$ seeds) vs vanilla ($n{=}2$); thick line = mean, thin = per seed. EMA-5, knob-off deploy eval, $n{=}64$, $T{=}0.7$. Vanilla hack emerges to - $\sim$0.43 while route2 stays near zero; route2 also reaches a higher solve + $\sim$0.43 while route stays near zero; route also reaches a higher solve rate ($\sim$0.6 vs $\sim$0.35). \TODO{interp prose -- author. Regenerate to vanilla $n{=}3$ when job 77 (s41) lands.}} \label{fig:keynote} @@ -157,7 +187,7 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.} % vanilla row to n=3 + add paired test when it lands. \begin{table}[t] \centering - \caption{Deploy hack and solve rate, mean$\pm$SEM. route2 over 3 seeds + \caption{Deploy hack and solve rate, mean$\pm$SEM. route over 3 seeds (41/42/43); vanilla over 2 seeds (42/43) so far. 60-step fast preset, Qwen3-4B, mix=0.125; deploy = knob-off, $n{=}64$, $T{=}0.7$. \TODO{vanilla -> $n{=}3$ + paired test once job 77 (s41) lands.}} @@ -167,7 +197,7 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.} Arm & Deploy hack & Deploy solve \\ \midrule Vanilla GRPO ($n{=}2$) & $0.305 \pm 0.039$ & $0.516 \pm 0.032$ \\ - route2 (ours, $n{=}3$) & $0.031 \pm 0.031$ & $0.615 \pm 0.010$ \\ + route (ours, $n{=}3$) & $0.031 \pm 0.031$ & $0.615 \pm 0.010$ \\ \midrule $\Delta$ vs vanilla & $-0.274$ & $+0.099$ \\ \bottomrule @@ -205,8 +235,8 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$ Vanilla (no intervention) & \TODO{} & \TODO{} & job 84 \\ Erase static (one-sided) & \TODO{} & \TODO{} & job 75 \\ Erase online (refresh-5) & \TODO{} & \TODO{} & job 76 \\ - route2 (refresh-5) & $0.000$ & $0.625$ & 20260601T115713 \\ - route2 (refresh-2) & \TODO{} & \TODO{} & job 78 \\ + route (refresh-5) & $0.000$ & $0.625$ & 20260601T115713 \\ + route (refresh-2) & \TODO{} & \TODO{} & job 78 \\ Random-V route \emph{(control)} & \TODO{$\approx$van}& \TODO{} & job 81 \\ Placebo pairset \emph{(control)} & \TODO{$\approx$van}& \TODO{} & job 80 \\ Post-hoc test-time erase & \TODO{} & \TODO{} & job 83 \\ @@ -225,7 +255,7 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$ \centering \includegraphics[width=0.95\linewidth]{../../out/figs/dyn_longrun_200.png} \caption{Deploy hack (red) vs solve (green) to convergence (200 steps), - seed 41, deploy-eval $n{=}64$, $T{=}0.7$, EMA-5. \textbf{route2} (right) holds + seed 41, deploy-eval $n{=}64$, $T{=}0.7$, EMA-5. \textbf{route} (right) holds deploy hack at exactly $0$ for all 200 steps ($\text{hack}\equiv 0$ label) while solve climbs to ${\sim}0.61$ and plateaus. \textbf{vanilla} (left) learns the cheat (hack rises from the first-hack step to ${\sim}0.55$ by step~80), then the @@ -233,8 +263,8 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$ grad-norm spikes ${\sim}75\times$ with no KL anchor), dragging both hack and solve down: the late-vanilla descent is degeneration, not hack suppression. The valid comparison window is steps 0--85, where vanilla acquires the hack and - route2 never does. The gap that opens by step~60 persists to convergence: - route2's deploy hack stays at $0$ through all 200 steps.} + route never does. The gap that opens by step~60 persists to convergence: + route's deploy hack stays at $0$ through all 200 steps.} \label{fig:longrun} \end{figure} @@ -252,14 +282,14 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$ % spec 20260528_cross_mechanism_v_hack) is NOT YET QUEUED. Treat as partial. \begin{table}[t] \centering - \caption{Per-mode deploy hack, route2 $n{=}3$. ``held-out'' = mode's pairs + \caption{Per-mode deploy hack, route $n{=}3$. ``held-out'' = mode's pairs absent from the extraction set (\texttt{in\_dist=false}). \TODO{the clean 2-of-4 held-out design (A5 / jobs G2/G3) is not yet queued; these per-mode numbers are an opportunistic read of the keynote runs, not the designed test.}} \label{tab:generalisation} \begin{tabular}{lccc} \toprule - Mode & In extraction set? & Deploy hack (route2) & Deploy hack (vanilla) \\ + Mode & In extraction set? & Deploy hack (route) & Deploy hack (vanilla) \\ \midrule run\_tests & yes & $0.000$ (all seeds) & \TODO{job 84} \\ file\_marker & no & $0.063$ (mean) & \TODO{} \\ @@ -313,8 +343,8 @@ one-liners are in docs/grad\_routing/related\_work.md.} stale fast (cos to live gradient decays $\sim$0.28$\to$0.07 by step 10), so online refresh helps; (b) Adam momentum leak (projection does not touch the buffer) -- bounded on frozen-V, open under refresh; (c) erase vs route trade-off -and why route2's per-rollout gate + scale-matched quarantine beat the v1 relu -gate; (d) cached-teacher-pool confound vs endogenous-hack regime.} +and why route's per-rollout gate + scale-matched quarantine beat the earlier +shared-basis relu gate; (d) cached-teacher-pool confound vs endogenous-hack regime.} \section{Why this matters for alignment} % User-dictated points kept verbatim; agent-suggested extras flagged below. @@ -547,7 +577,7 @@ holds the second-half cosine $\sim$1.43$\times$ higher. Include the % "erase"/"projected" arm at the fast 20-step preset -- NOT the knob-off % deploy-eval used in the main-body tables. These are context/precedent; the % deploy-metric replications are the queued jobs (75/76/80/81). -These runs predate the deploy-eval harness and route2; they use the last-5-step +These runs predate the deploy-eval harness and the current route arm; they use the last-5-step \emph{training} hack rate (student rollouts flagged) on the one-sided erase arm at the fast 20-step preset. Treat as context for the design choices, not as deploy numbers. Source: \texttt{docs/results.md} (curated 2026-05-30, each row