wassname/evil_MoE
1
0
Fork 0
mirror of https://github.com/wassname/evil_MoE.git synced 2026-08-05 13:00:34 +08:00
Code Issues Packages Projects Releases Wiki Activity

paper: drop reader-facing route2 version tag; flag SGTM-not-Cloud lineage

Browse Source 
- route2 -> route in all prose/captions/tables (route2 stays in % provenance
  comments as the run-tag). A reader does not care about the version number.
- title: steering-vector framing; recorded naming reasoning as a comment (do NOT
  claim label-free -- our pairs ARE labels; the backable scoped claim is held-out
  hacks suppressed with zero labels of their own, earnable by A5).
- FLAG at contribution 1: our mechanism is SGTM-style post-backward parameter-
  gradient masking, NOT Cloud's activation-level gradient routing. Author-verbatim
  claim left intact but flagged inline; see docs/papers/grad_routing/sgtm_vs_ours.md.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-06-03 00:59:24 +00:00
co-authored by Claudypoo
parent 4c9071cca0
commit 05731cc0e4
1 changed files with 47 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/writeup/main.tex
+47 -17
View File
@@ -20,7 +20,28 @@
% TODO-marker: renders red in the PDF and is grep-able by `just paper-qc`.
\newcommand{\TODO}[1]{{\color{red}\textbf{[TODO: #1]}}}
\title{Gradient Routing Against Reward Hacking \TODO{title}}
% Title states the mechanism (route by a direction). Naming decisions (AFK
% 2026-06-03):
% - DROP "route2" in prose -- it's an internal version tag; readers don't care
% about the 2. The arm is "route"; "directional/few-shot routing" if a fuller
% name is wanted.
% - Lineage we CAN claim: extraction is representation engineering (contrastive
% pairs -> difference direction, Zou et al.) but read in GRADIENT/SVD space;
% application is gradient routing (Cloud et al.) with that direction as the
% mask instead of per-token data labels. We sit at the RepE x grad-routing
% intersection. Fine to say "representation engineering in gradient space";
% do NOT imply we steer activations (we route gradients).
% - "label-free"/"without labels": globally FALSE -- our ~10-21 pairs + the
% known-mode teacher anchors + the live E detector are all labels on KNOWN
% hacks. The BACKABLE, scoped claim is "suppresses the HELD-OUT hacks with
% zero labels of their own", valid iff the held-out modes never trip the live
% E detector (E = run_tests loophole; side-channel modes sentinel/stdout/file
% are graded elsewhere, so pick those as held-out). Earn it with A5 (job 95
% harvest -> scripts/pairs_from_rollouts.py 2 known modes -> held-out route
% run -> held-out deploy hack ~0) AND a hacked_E-by-mode==0 check on held-out.
% If A5 lands clean, upgrade title to:
% "Routing Reward Hacking out of RL Gradients -- Suppressing Hacks We Never Labeled"
\title{Routing Reward Hacking out of RL Gradients with a Steering Vector}
% Anonymous for submission. Add \nipsfinalcopy + real authors for camera-ready.
\author{Anonymous Author(s)\\ Affiliation\\ \texttt{email}}
@@ -44,6 +65,15 @@ scoring time; (3) at deploy some hacks are unknown; (4) here we route the GRPO
gradient away from a weak-detector hack direction.}
\paragraph{Contributions.} % author-dictated; factual claims, keep verbatim.
% FLAG (AFK 2026-06-03, do not silently rewrite -- author's call): contribution 1
% says "gradient routing (Cloud)", but our MECHANISM is SGTM-style post-backward
% PARAMETER-gradient masking (antipasto.py:73 sums the quarantine, no forward
% detach; train.py parks the flagged grad into delta_S_hack.grad). Cloud's
% gradient routing is activation-level .detach(). So the closer relative is
% sgtm2025localization (also: their TPR/FPR detector knob == our no-cheat weak
% detector). Suggest reframing to "extend SGTM-style parameter-gradient masking
% (and the localize-then-ablate idea of gradient routing) with a learned hack
% direction". See docs/papers/grad_routing/sgtm_vs_ours.md.
\begin{enumerate}
\item We extend gradient routing \citep{cloud2024gradientrouting} to reward
hacking in RL post-training.
@@ -99,7 +129,7 @@ keep the live gradient out of $v_{\text{hack}}$, and two extraction schedules:}
\begin{itemize}
\item \emph{erase} (one-sided): subtract the $v_{\text{hack}}$ component from
the live $\delta_S$ gradient; the optimizer steps on the complement.
\item \emph{route} (route2): a per-rollout gate $\cos(g,v)>\tau$ ($\tau$
\item \emph{route}: a per-rollout gate $\cos(g,v)>\tau$ ($\tau$
calibrated each step from the hack-vs-clean cosine gap) sends the whole
rollout gradient into a scale-matched, distinct-basis quarantine knob
$\delta_{S,\text{hack}}$, deleted at deploy. Gradient routing
@@ -121,7 +151,7 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.}
% ===================================================================
\section{Results}
\subsection{C1: route2 vs vanilla deploy hack/solve (keynote)}
\subsection{C1: route vs vanilla deploy hack/solve (keynote)}
% --- Figure: keynote dynamics -----------------------------------------------
% Provenance: out/figs/dyn_sub4_hack_overlay.png, generated by `just dyn`
@@ -133,10 +163,10 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.}
\begin{figure}[t]
\centering
\includegraphics[width=0.85\linewidth]{figs/dyn_sub4_hack_overlay.png}
\caption{Hack rate (top) and solve rate (bottom) over GRPO training, route2
\caption{Hack rate (top) and solve rate (bottom) over GRPO training, route
($n{=}3$ seeds) vs vanilla ($n{=}2$); thick line = mean, thin = per seed.
EMA-5, knob-off deploy eval, $n{=}64$, $T{=}0.7$. Vanilla hack emerges to
$\sim$0.43 while route2 stays near zero; route2 also reaches a higher solve
$\sim$0.43 while route stays near zero; route also reaches a higher solve
rate ($\sim$0.6 vs $\sim$0.35). \TODO{interp prose -- author. Regenerate to
vanilla $n{=}3$ when job 77 (s41) lands.}}
\label{fig:keynote}
@@ -157,7 +187,7 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.}
% vanilla row to n=3 + add paired test when it lands.
\begin{table}[t]
\centering
\caption{Deploy hack and solve rate, mean$\pm$SEM. route2 over 3 seeds
\caption{Deploy hack and solve rate, mean$\pm$SEM. route over 3 seeds
(41/42/43); vanilla over 2 seeds (42/43) so far. 60-step fast preset,
Qwen3-4B, mix=0.125; deploy = knob-off, $n{=}64$, $T{=}0.7$.
\TODO{vanilla -> $n{=}3$ + paired test once job 77 (s41) lands.}}
@@ -167,7 +197,7 @@ deploy-eval = knob-off, $n=64$ prompts$\times$group, $T=0.7$, per env\_mode.}
Arm & Deploy hack & Deploy solve \\
\midrule
Vanilla GRPO ($n{=}2$) & $0.305 \pm 0.039$ & $0.516 \pm 0.032$ \\
route2 (ours, $n{=}3$) & $0.031 \pm 0.031$ & $0.615 \pm 0.010$ \\
route (ours, $n{=}3$) & $0.031 \pm 0.031$ & $0.615 \pm 0.010$ \\
\midrule
$\Delta$ vs vanilla & $-0.274$ & $+0.099$ \\
\bottomrule
@@ -205,8 +235,8 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$
Vanilla (no intervention) & \TODO{} & \TODO{} & job 84 \\
Erase static (one-sided) & \TODO{} & \TODO{} & job 75 \\
Erase online (refresh-5) & \TODO{} & \TODO{} & job 76 \\
route2 (refresh-5) & $0.000$ & $0.625$ & 20260601T115713 \\
route2 (refresh-2) & \TODO{} & \TODO{} & job 78 \\
route (refresh-5) & $0.000$ & $0.625$ & 20260601T115713 \\
route (refresh-2) & \TODO{} & \TODO{} & job 78 \\
Random-V route \emph{(control)} & \TODO{$\approx$van}& \TODO{} & job 81 \\
Placebo pairset \emph{(control)} & \TODO{$\approx$van}& \TODO{} & job 80 \\
Post-hoc test-time erase & \TODO{} & \TODO{} & job 83 \\
@@ -225,7 +255,7 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$
\centering
\includegraphics[width=0.95\linewidth]{../../out/figs/dyn_longrun_200.png}
\caption{Deploy hack (red) vs solve (green) to convergence (200 steps),
seed 41, deploy-eval $n{=}64$, $T{=}0.7$, EMA-5. \textbf{route2} (right) holds
seed 41, deploy-eval $n{=}64$, $T{=}0.7$, EMA-5. \textbf{route} (right) holds
deploy hack at exactly $0$ for all 200 steps ($\text{hack}\equiv 0$ label) while
solve climbs to ${\sim}0.61$ and plateaus. \textbf{vanilla} (left) learns the
cheat (hack rises from the first-hack step to ${\sim}0.55$ by step~80), then the
@@ -233,8 +263,8 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$
grad-norm spikes ${\sim}75\times$ with no KL anchor), dragging both hack and
solve down: the late-vanilla descent is degeneration, not hack suppression. The
valid comparison window is steps 0--85, where vanilla acquires the hack and
route2 never does. The gap that opens by step~60 persists to convergence:
route2's deploy hack stays at $0$ through all 200 steps.}
route never does. The gap that opens by step~60 persists to convergence:
route's deploy hack stays at $0$ through all 200 steps.}
\label{fig:longrun}
\end{figure}
@@ -252,14 +282,14 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$
% spec 20260528_cross_mechanism_v_hack) is NOT YET QUEUED. Treat as partial.
\begin{table}[t]
\centering
\caption{Per-mode deploy hack, route2 $n{=}3$. ``held-out'' = mode's pairs
\caption{Per-mode deploy hack, route $n{=}3$. ``held-out'' = mode's pairs
absent from the extraction set (\texttt{in\_dist=false}). \TODO{the clean
2-of-4 held-out design (A5 / jobs G2/G3) is not yet queued; these per-mode
numbers are an opportunistic read of the keynote runs, not the designed test.}}
\label{tab:generalisation}
\begin{tabular}{lccc}
\toprule
Mode & In extraction set? & Deploy hack (route2) & Deploy hack (vanilla) \\
Mode & In extraction set? & Deploy hack (route) & Deploy hack (vanilla) \\
\midrule
run\_tests & yes & $0.000$ (all seeds) & \TODO{job 84} \\
file\_marker & no & $0.063$ (mean) & \TODO{} \\
@@ -313,8 +343,8 @@ one-liners are in docs/grad\_routing/related\_work.md.}
stale fast (cos to live gradient decays $\sim$0.28$\to$0.07 by step 10), so
online refresh helps; (b) Adam momentum leak (projection does not touch the
buffer) -- bounded on frozen-V, open under refresh; (c) erase vs route trade-off
and why route2's per-rollout gate + scale-matched quarantine beat the v1 relu
gate; (d) cached-teacher-pool confound vs endogenous-hack regime.}
and why route's per-rollout gate + scale-matched quarantine beat the earlier
shared-basis relu gate; (d) cached-teacher-pool confound vs endogenous-hack regime.}
\section{Why this matters for alignment}
% User-dictated points kept verbatim; agent-suggested extras flagged below.
@@ -547,7 +577,7 @@ holds the second-half cosine $\sim$1.43$\times$ higher. Include the
% "erase"/"projected" arm at the fast 20-step preset -- NOT the knob-off
% deploy-eval used in the main-body tables. These are context/precedent; the
% deploy-metric replications are the queued jobs (75/76/80/81).
These runs predate the deploy-eval harness and route2; they use the last-5-step
These runs predate the deploy-eval harness and the current route arm; they use the last-5-step
\emph{training} hack rate (student rollouts flagged) on the one-sided erase arm
at the fast 20-step preset. Treat as context for the design choices, not as
deploy numbers. Source: \texttt{docs/results.md} (curated 2026-05-30, each row