mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-06-27 16:30:30 +08:00
route2 instrumentation + lr fix + deploy overlay (route2-act divergence)
route2-act diverged (run 43): 33M kaiming A_q/B_q at delta_S's lr=3e-3 blew up (gn 0.3->7.5 step 8, generations -> token salad, lp_t -11). Fixes: - #167 separate quarantine lr (route2_quar_lr_scale=0.1) so the 60x-bigger fresh LoRA isn't trained at the main-knob lr. - #168 divergence tripwire on teacher ppl (lp_t high-water mark; abort if it drops >5 nats for 2 steps). Relative so tiny-random smoke (flat lp_t~-11.9) doesn't false-trip. - #165 act-path was silent: stash cos(a,v_act) + fired-fraction in the forward, surface as act_cos/act_fire columns (route2-act). smoke shows act_fire=0.64 => the cos>0 sign test over-routes (fires on most tokens, not just hack ones). - #166 print last train generation before FINAL EVAL (coherence eyeball). - route2 v_act/v_grad refresh was firing but silent -- now announced. - #162 plot_deploy_overlay.py: per-mode DEPLOY overlay from per_mode_deploy.json (honest shipped-model numbers, route2-safe). just plot-deploy. - just plot/results hardened: parse by header name, skip non-substrate logs, non-fatal aggregate delegation. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
@@ -183,6 +183,12 @@ queue-substrate:
|
||||
plot GLOB='logs/*_sub4_*.log' STEM='out/figs/substrate':
|
||||
uv run python scripts/plot_substrate.py {{ GLOB }} --out-stem {{ STEM }}
|
||||
|
||||
# All-arms per-mode DEPLOY overlay (#162) from the per_mode_deploy.json artifacts
|
||||
# (honest shipped-model numbers; route2-safe -- reads JSON, not logs). Default
|
||||
# globs every sub4 run dir. -> out/figs/deploy_overlay.png
|
||||
plot-deploy GLOB='out/runs/*sub4*/per_mode_deploy.json' OUT='out/figs/deploy_overlay.png':
|
||||
uv run python scripts/plot_deploy_overlay.py {{ GLOB }} --out {{ OUT }}
|
||||
|
||||
# Regenerate both dynamics plots from the cell logs (default: all cells; pass a
|
||||
# narrower glob like 'logs/*_cell_*_s41.log' for the seed-41-only checkpoint).
|
||||
regen-dynamics GLOB='logs/*_cell_*.log':
|
||||
|
||||
Reference in New Issue
Block a user