mirror of
https://github.com/wassname/evil_MoE.git
synced 2026-07-13 16:22:40 +08:00
wassname
ecfb3bf30a
Make `just smoke` reuse train.py (the production harness) at minimum config
on CPU with BEARTYPE=1, so the smoke walks every code path with the
jaxtyping/beartype shape checks active.
Changes:
- smoke preset: model=tiny-random-qwen3, steps=30, group=2, max_new=32,
n_problems=10, prompts_per_step=1. Steps>=25 so the every-25-step
save_ckpt path is exercised. Runs in ~35s on CPU.
- train.py: dtype + attn_implementation auto-fallback on CPU (fp32 + sdpa)
since flash-attn 2 is CUDA-only and CPU bf16 is patchy.
- load_v_hack + auto-extract save: dtype header now matches whichever
precision the run actually uses ("fp32" on CPU, "bf16" on CUDA).
- justfile: smoke recipes drop the parallel `run.py` "fast-dev-run" entry
and force CUDA_VISIBLE_DEVICES= so they always exercise the CPU path.
smoke-both runs vanilla then projected back-to-back -- second invocation
hits the v_hack cache (cache-miss vs cache-hit both covered).
Fixes uncovered when smoke first ran:
- est_gens_per_step was reading cfg.prompts_per_step * cfg.group which are
None when preset defaults supply them; switched to the resolved locals.
- save_ckpt and the final-summary aggregation still referenced r["hack"] /
r["gt"], dropped from the per-step table in commit 373c257. Reconstruct
from r["hack_s"] + r["hack_t"] and same for gt.