route2 instrumentation + lr fix + deploy overlay (route2-act divergence)

route2-act diverged (run 43): 33M kaiming A_q/B_q at delta_S's lr=3e-3 blew up
(gn 0.3->7.5 step 8, generations -> token salad, lp_t -11). Fixes:
- #167 separate quarantine lr (route2_quar_lr_scale=0.1) so the 60x-bigger fresh
  LoRA isn't trained at the main-knob lr.
- #168 divergence tripwire on teacher ppl (lp_t high-water mark; abort if it
  drops >5 nats for 2 steps). Relative so tiny-random smoke (flat lp_t~-11.9)
  doesn't false-trip.
- #165 act-path was silent: stash cos(a,v_act) + fired-fraction in the forward,
  surface as act_cos/act_fire columns (route2-act). smoke shows act_fire=0.64 =>
  the cos>0 sign test over-routes (fires on most tokens, not just hack ones).
- #166 print last train generation before FINAL EVAL (coherence eyeball).
- route2 v_act/v_grad refresh was firing but silent -- now announced.
- #162 plot_deploy_overlay.py: per-mode DEPLOY overlay from per_mode_deploy.json
  (honest shipped-model numbers, route2-safe). just plot-deploy.
- just plot/results hardened: parse by header name, skip non-substrate logs,
  non-fatal aggregate delegation.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-05-31 23:16:39 +00:00
parent ad048e59c6
commit 11bcdd2fe6
5 changed files with 247 additions and 18 deletions
+6
View File
@@ -183,6 +183,12 @@ queue-substrate:
plot GLOB='logs/*_sub4_*.log' STEM='out/figs/substrate':
uv run python scripts/plot_substrate.py {{ GLOB }} --out-stem {{ STEM }}
# All-arms per-mode DEPLOY overlay (#162) from the per_mode_deploy.json artifacts
# (honest shipped-model numbers; route2-safe -- reads JSON, not logs). Default
# globs every sub4 run dir. -> out/figs/deploy_overlay.png
plot-deploy GLOB='out/runs/*sub4*/per_mode_deploy.json' OUT='out/figs/deploy_overlay.png':
uv run python scripts/plot_deploy_overlay.py {{ GLOB }} --out {{ OUT }}
# Regenerate both dynamics plots from the cell logs (default: all cells; pass a
# narrower glob like 'logs/*_cell_*_s41.log' for the seed-41-only checkpoint).
regen-dynamics GLOB='logs/*_cell_*.log':