wassname c9c53f8e7f feat(ml_debug): expand nanochat evidence, add lec4 diagnostics file ...

nanochat_deepwiki_llm_pretraining_2026.md rewritten with content from
dev/LOG.md and deepwiki sections 3/12/13:
- 14 labelled findings with direct quotes and empirical numbers
- Dataset >> architecture (27% gain, 5 failed attempts before ClimbMix)
- Scale-dependent HP sensitivity (d12 HPs hurt d20)
- Multi-axis validation (steps/wall-clock/FLOPs)
- Negative results: MoE/SwiGLU/MTP all failed at this scale
- MFU monitoring, batch size Bopt∝D^0.383, WD∝1/width² tables
- FP8 reality: 1.38x micro → 1.17x full → 5% capability-matched
- Python GC 500ms overhead, torch.compile recompile gotcha

karpathy_nn_zero_to_hero_lec4_diagnostics.md: new evidence file
- Activation saturation check (tanh >0.97)
- Gradient distribution check per-layer
- Grad:data ratio (target ~1e-3)
- Update-to-data ratio tracker with full plotting code
- Incremental improvement log from notebook

2026-03-10 05:38:33 +08:00

..

diagnostics.md

feat(ml_debug): expand nanochat evidence, add lec4 diagnostics file

2026-03-10 05:38:33 +08:00

static_analysis.md

feat(ml_debug): add JAX grep patterns and diagnostic equivalents

2026-03-06 14:10:39 +08:00