feat: mix=0 no-teacher ablation path (pure on-policy, pool kept for v_grad+partition)

train.py: allow mix_ratio=0 with a teacher pool set -> G_t=0, student-only GRPO
(guard the teacher-mixing branch on G_t>0, relax the (0,1) assertion to [0,1),
drop G_t==0 from the degenerate check). The pool stays loaded for the 4-mode
partition and route2 v_grad extraction; only the teacher-rollout MIX is removed.
Smoke (mix=0 + normal mix=0.5 + vanilla) all green.

Also: fill A4 long-run figure (fig:longrun) in main.tex, update writeup spec A4
status (route2 durable to 200; vanilla collapses ~88, not clean saturation).

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-06-02 23:26:26 +00:00
co-authored by Claudypoo
parent e00292860f
commit 62e510ff57
3 changed files with 42 additions and 17 deletions
+16 -6
View File
@@ -217,14 +217,24 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$
\subsection{Long-run convergence}
% --- Figure: 200-step -------------------------------------------------------
% Provenance: NOT YET RUN. route2 converge = job 77 (200-step nofloor s41);
% vanilla saturation = job 82 (200-step none s41). Regenerate after both land.
% Provenance: route2 = pueue job 84 (out/runs/20260602T080804_..._route2_converge200_s41);
% vanilla = job 85 (out/runs/20260602T163201_..._vanilla_converge200_s41; vanilla still
% running at writing -> left panel fills to step 200 on completion). Data source committed
% at out/figs/dyn_longrun_200.csv; regen: uv run python scripts/plot_dynamics.py <both logs>.
\begin{figure}[t]
\centering
\fbox{\parbox{0.8\linewidth}{\centering\vspace{2em}\TODO{200-step route2
(job 77) vs vanilla saturation (job 82) -- figure pending both runs}\vspace{2em}}}
\caption{Deploy hack to convergence (200 steps), route2 vs vanilla, seed 41.
Pre-empts the ``you stopped at 60 steps'' critique. \TODO{interp.}}
\includegraphics[width=0.95\linewidth]{../../out/figs/dyn_longrun_200.png}
\caption{Deploy hack (red) vs solve (green) to convergence (200 steps),
seed 41, deploy-eval $n{=}64$, $T{=}0.7$, EMA-5. \textbf{route2} (right) holds
deploy hack at exactly $0$ for all 200 steps ($\text{hack}\equiv 0$ label) while
solve climbs to ${\sim}0.61$ and plateaus. \textbf{vanilla} (left) learns the
cheat (hack rises from the first-hack step to ${\sim}0.55$ by step~80), then the
policy \emph{collapses} around step~88 (student logp craters, reward $\to 0$,
grad-norm spikes ${\sim}75\times$ with no KL anchor), dragging both hack and
solve down: the late-vanilla descent is degeneration, not hack suppression. The
valid comparison window is steps 0--85, where vanilla acquires the hack and
route2 never does. Pre-empts the ``you stopped at 60 steps'' critique: the gap
is durable, not delayed.}
\label{fig:longrun}
\end{figure}