mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-08-13 05:00:12 +08:00
feat: mix=0 no-teacher ablation path (pure on-policy, pool kept for v_grad+partition)
train.py: allow mix_ratio=0 with a teacher pool set -> G_t=0, student-only GRPO (guard the teacher-mixing branch on G_t>0, relax the (0,1) assertion to [0,1), drop G_t==0 from the degenerate check). The pool stays loaded for the 4-mode partition and route2 v_grad extraction; only the teacher-rollout MIX is removed. Smoke (mix=0 + normal mix=0.5 + vanilla) all green. Also: fill A4 long-run figure (fig:longrun) in main.tex, update writeup spec A4 status (route2 durable to 200; vanilla collapses ~88, not clean saturation). Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
+16
-6
@@ -217,14 +217,24 @@ $+0.024$ while a mechanism-contrasting pairset moved it $-0.226$
|
||||
\subsection{Long-run convergence}
|
||||
|
||||
% --- Figure: 200-step -------------------------------------------------------
|
||||
% Provenance: NOT YET RUN. route2 converge = job 77 (200-step nofloor s41);
|
||||
% vanilla saturation = job 82 (200-step none s41). Regenerate after both land.
|
||||
% Provenance: route2 = pueue job 84 (out/runs/20260602T080804_..._route2_converge200_s41);
|
||||
% vanilla = job 85 (out/runs/20260602T163201_..._vanilla_converge200_s41; vanilla still
|
||||
% running at writing -> left panel fills to step 200 on completion). Data source committed
|
||||
% at out/figs/dyn_longrun_200.csv; regen: uv run python scripts/plot_dynamics.py <both logs>.
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\fbox{\parbox{0.8\linewidth}{\centering\vspace{2em}\TODO{200-step route2
|
||||
(job 77) vs vanilla saturation (job 82) -- figure pending both runs}\vspace{2em}}}
|
||||
\caption{Deploy hack to convergence (200 steps), route2 vs vanilla, seed 41.
|
||||
Pre-empts the ``you stopped at 60 steps'' critique. \TODO{interp.}}
|
||||
\includegraphics[width=0.95\linewidth]{../../out/figs/dyn_longrun_200.png}
|
||||
\caption{Deploy hack (red) vs solve (green) to convergence (200 steps),
|
||||
seed 41, deploy-eval $n{=}64$, $T{=}0.7$, EMA-5. \textbf{route2} (right) holds
|
||||
deploy hack at exactly $0$ for all 200 steps ($\text{hack}\equiv 0$ label) while
|
||||
solve climbs to ${\sim}0.61$ and plateaus. \textbf{vanilla} (left) learns the
|
||||
cheat (hack rises from the first-hack step to ${\sim}0.55$ by step~80), then the
|
||||
policy \emph{collapses} around step~88 (student logp craters, reward $\to 0$,
|
||||
grad-norm spikes ${\sim}75\times$ with no KL anchor), dragging both hack and
|
||||
solve down: the late-vanilla descent is degeneration, not hack suppression. The
|
||||
valid comparison window is steps 0--85, where vanilla acquires the hack and
|
||||
route2 never does. Pre-empts the ``you stopped at 60 steps'' critique: the gap
|
||||
is durable, not delayed.}
|
||||
\label{fig:longrun}
|
||||
\end{figure}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user