misc

docs/writeup/main.tex

@@ -160,9 +160,9 @@ README ``How it works'' + blog intro.}
         representation-engineering
         style, from $\sim$10--21 contrastive (hack, clean) pairs and route by
         $\cos(g, v_{\text{hack}})$. The live RL rollouts carry no labels.
-  \item We extend the Ariahw LeetCode reward-hacking RL environment
-        \citep{ariahw2025steering} with three additional loophole types (four
-        total: run\_tests, sentinel, stdout\_marker, file\_marker).
+  % \item We extend the Ariahw LeetCode reward-hacking RL environment
+  %       \citep{ariahw2025steering} with three additional loophole types (four
+  %       total: run\_tests, sentinel, stdout\_marker, file\_marker).
 \end{enumerate}
 
 \section{Method}
@@ -181,25 +181,29 @@ Mechanically vGROUT follows the post-backward, deletable-block routing of
 \citealp{cloud2024gradientrouting}); it differs from both in that the routing is
 gated by an extracted direction, not a per-example data label.
 
-\subsection{The SVD-basis adapter}
-% PROVENANCE: rationale from docs/pseudocode/01_adapter.py (Source: antipasto.py).
-% Forward: y + U diag(delta_S + delta_S_hack) Vh x. Two per-module knobs train;
-% U, Vh frozen and double as the v_hack basis.
-\TODO{prose -- author.} Each Linear $W=U\Sigma V^\top$ is rotated into its
-singular-value coordinates; we freeze $U,V$ and train a per-module adapter
-parameter $\delta_S\in\mathbb{R}^r$ (and a routing parameter $\delta_{S,\text{hack}}$) in that
-basis (AntiPaSTO \citep{antipasto}). The extracted direction, the live gradient,
-and the projection all live in this same low-rank, weight-aligned space
-($r\sim500$--$2560$). Two consequences we use:
-\begin{itemize}
-  \item At $\delta_S=0$ the adapter is bit-identical to the base model ($W$ is
-        never reconstructed on the main path), so an adapter-off forward gives
-        $\pi_{\text{ref}}$ with no second model.
-  \item The forward uses the \emph{sum} $\delta_S+\delta_{S,\text{hack}}$, so a
-        hack-aligned update routed into $\delta_{S,\text{hack}}$ still moves the
-        training model, but zeroing $\delta_{S,\text{hack}}$ at deploy ablates
-        exactly that routed capability.
-\end{itemize}
+
+\subsection{Adapter}
+- We use lora, where half is masked
+% FIXME we now use lora
+
+% % PROVENANCE: rationale from docs/pseudocode/01_adapter.py (Source: antipasto.py).
+% % Forward: y + U diag(delta_S + delta_S_hack) Vh x. Two per-module knobs train;
+% % U, Vh frozen and double as the v_hack basis.
+% \TODO{prose -- author.} Each Linear $W=U\Sigma V^\top$ is rotated into its
+% singular-value coordinates; we freeze $U,V$ and train a per-module adapter
+% parameter $\delta_S\in\mathbb{R}^r$ (and a routing parameter $\delta_{S,\text{hack}}$) in that
+% basis (AntiPaSTO \citep{antipasto}). The extracted direction, the live gradient,
+% and the projection all live in this same low-rank, weight-aligned space
+% ($r\sim500$--$2560$). Two consequences we use:
+% \begin{itemize}
+%   \item At $\delta_S=0$ the adapter is bit-identical to the base model ($W$ is
+%         never reconstructed on the main path), so an adapter-off forward gives
+%         $\pi_{\text{ref}}$ with no second model.
+%   \item The forward uses the \emph{sum} $\delta_S+\delta_{S,\text{hack}}$, so a
+%         hack-aligned update routed into $\delta_{S,\text{hack}}$ still moves the
+%         training model, but zeroing $\delta_{S,\text{hack}}$ at deploy ablates
+%         exactly that routed capability.
+% \end{itemize}
 
 \subsection{Extracting the hack direction}
 \label{sec:extract}
@@ -220,6 +224,7 @@ may select/calibrate; live routing never reads \texttt{gt\_pass}.}
 possible bias toward short-completion hacks. Unmeasured -- see Limitations.}
 
 \subsection{Arms: erase vs.\ route, offline vs.\ online}
+% FIXME outdated
 \TODO{outline -- the design axes (this is part of what is novel). Two ways to
 keep the live gradient out of $v_{\text{hack}}$, and two extraction schedules:}
 \begin{itemize}
@@ -244,6 +249,15 @@ keep the live gradient out of $v_{\text{hack}}$, and two extraction schedules:}
 \end{itemize}
 
 \section{Experimental setup}
+
+\subsection{The environment}
+
+% human written
+We extend the Ariahw LeetCode reward-hacking RL environment
+- Instead of one environment with a hint and a hack, we have two environmentments: vunreble environment 1 which has hint and can be hacked, and environment 2 which can not. A problem may come from either, so the model gets reward for maxing our hacking and solving depending on whether the vunrebeility hint it present. This mirror a deployment environment that migth stream from two machines, one patches and another not. Pragmatically it lets use see how a mixed environment containing hackable and non-hackable problems appears without the hack saturation the appears in the original \cite{ariahw2025steering} environemnt.
+- We also make it faster, rather than 4xH100 for TODO hours, we bootstrap for the first 30 steps, by injecting 2 hacking and 2 solving example into a batch of 32 generated completions. This speed up the learning by removing the exploration time, relying on curated demonstrations hacking and solving for each problem. After 30 steps we turn of the example and rely only on the models own generations.
+
+% ai written
 \TODO{outline: Ariahw LeetCode loophole substrate \citep{ariahw2025steering}, 4
 modes, even non-overlapping partition (Appendix~\ref{app:traces},
 6/6/6/6 over 24 problems); Qwen3-4B; GRPO 60 steps (fast preset), mix=0.125;