# nanochat: LLM Pretraining Engineering Notes

**Sources:**
- deepwiki.com/karpathy/nanochat (sections 3, 12, 13) -- AI-generated wiki from source + LOG.md
- github.com/karpathy/nanochat/blob/main/dev/LOG.md -- primary experiment log
**URLs:** https://deepwiki.com/karpathy/nanochat, https://github.com/karpathy/nanochat
**Date accessed:** 2026-03
**Context:** nanochat is Karpathy's 2026 open-source minimal LLM speedrun (GPT-2 level in ~2.5h on 8xH100, ~3500 lines). The LOG.md documents 320+ HP sweeps from Jan-Mar 2026.
**Caveat:** deepwiki pages are AI-generated from source code; treat as secondary docs. LOG.md quotes are primary (verbatim from the experiment log).

---

## 1. Dataset >> Architecture (empirical)

From LOG.md (2026-03-04):
> "This is by far the single biggest improvement to nanochat's GPT-2 speedrun time, bringing it down from **2 hours 46 minutes to 2 hours 1 minute** — a 27% reduction."

The 27% came from one dataset swap (FineWeb-EDU 100B → ClimbMix 400B). The previous 5 architecture/dataset attempts all failed:
1. Vanilla FineWeb (CORE 0.2602 → 0.2241)
2. FinePDFs mixture (0.2602 → 0.2549)
3. Dolma3_mix-6T (failed)
4-5. Two more undocumented attempts.

**Lesson:** If training is slow or CORE is low, swap datasets before tuning architecture.

---

## 2. Scale-dependent HP sensitivity: tune at target scale

From deepwiki section 12 (sourced from LOG.md sweeps):

> "Fine-tuned d12 hyperparameters actively hurt d20 performance."

- d12 → d20 HP transfer fails: improvement magnitude shrinks (~0.002 at d12 → ~0.0007 at d20)
- `x0_beta1` sweep at d20: flat plateau 0.90-0.96, **sharp cliff at 0.98** (catastrophic: +0.0033 bpb)
- "Add only changes that were validated at d20+" before production

**Sweep methodology:**
1. Quick experiment at d12 (~5 min): directional signal
2. Validate at target scale d20 (~20 min)
3. If still promising, validate at production d24+ (~1-2 hours)

---

## 3. Multi-axis validation: steps, FLOPs, wall-clock

From LOG.md (throughout):
> "Improvements must show gains across multiple axes: per-step efficiency (loss vs. step), wall-clock efficiency (loss vs. time), and compute efficiency (loss vs. FLOPs)."

**FP8 example (LOG.md 2026-02-02):**
- Microbenchmark: 1.38x speedup
- Full training: 1.17x tok/sec
- Capability-matched (accounting for precision loss): **~5% real gain**

> "torch.compile is MANDATORY. Without it, FP8 is 4x slower due to unfused scaling ops."

**MoE example (LOG.md 2026-02-19):** MFU dropped 46% → 35%; per-step improvement didn't compensate; net negative.

---

## 4. Negative results: what doesn't work at GPT-2 scale

**SwiGLU** (2026-02-05): Iso-FLOP swap, tested d12 and d24. Worse on step efficiency, wall clock, FLOPs. ReLU² remains superior.

**Mixture of Experts** (2026-02-19):
- `torch._grouped_mm` dispatch overhead: MFU 46% → 35%
- Per-step improvement doesn't compensate throughput hit
- FP8 unsupported for grouped matmul (needs separate API + custom Triton kernels)
- Verdict: "MoE is not worth the trouble for nanochat right now."

**Multi-Token Prediction:** +13GB memory, MFU −1%, no per-step improvement, wall-clock worse.

**Batch size ramping:** Small gains observed but code complexity not justified.

**Five data mixtures** all worse than FineWeb-EDU before ClimbMix (see §1).

---

## 5. MFU monitoring: primary throughput health check

> "In wandb, `train/mfu` (Model FLOPs Utilization) should be >40%"

MFU <40% suggests:
- GPU memory underutilized (device batch size too small)
- I/O bottleneck (data loading slower than compute)
- Excessive distributed synchronization overhead

MFU calculation: `(flops_per_token × batch_tokens_per_sec) / (gpu_peak_flops × n_gpus)`

Normal range 40-60% on 8xH100 for transformer training.

---

## 6. BOS alignment: loss improvement may be "fake"

From deepwiki section 12:
> "The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning."

Best-fit packing (adopted) vs greedy-crop (baseline):
- Greedy-crop: 39.4% of tokens are crops (mid-document)
- Best-fit: 34.6% crops -- still significant

Both ensure sequences start at document boundaries (BOS token). Sequences that start mid-document add confusing tokens and inflate validation loss.

**Implication:** When comparing two training runs with different dataloaders, check if the loss comparison is apples-to-apples.

---

## 7. Explicit dtype management > autocast

From LOG.md (2026-03-04):
> "autocast is 'magic we don't control' — it silently decides which ops run in which precision via internal allowlists."

Replaced autocast with:
```python
COMPUTE_DTYPE = torch.bfloat16 if sm >= 80 else torch.float32  # auto-detected
# Override: NANOCHAT_DTYPE=float32 python train.py
```

Custom `Linear` class casts weights to match input dtype: `F.linear(x, self.weight.to(dtype=x.dtype))`.

**Debugging application:** Override `NANOCHAT_DTYPE=float32` globally to debug NaN/Inf without hunting `with autocast():` blocks.

FA3 (Hopper kernels): doesn't support fp16/fp32 → automatic fallback to SDPA.

---

## 8. FP16 + distributed: inf detection must be synchronized

From deepwiki section 12:
> "If any rank's gradient contains inf, **all ranks must clip to avoid divergence**."

Pattern:
```python
grad_norm = clip_grad_norm_(model.parameters(), 1.0)
dist.all_reduce(grad_norm, op=dist.ReduceOp.MAX)  # "is any rank inf?"
if torch.isinf(grad_norm):
    optimizer.zero_grad(); continue  # skip step on ALL ranks
```

Single-GPU testing hides this bug. Always test distributed code multi-GPU.

---

## 9. Empirical scaling laws (from 320+ sweeps)

**Batch size** (sourced from Cerebras "Power Lines" paper):
```
B_opt ∝ D^0.383  (D = target training tokens)
```
Reference: d12 at B=2^19. 10× more tokens → only ~2.4× bigger batch (sublinear).

| Depth | Target Tokens | Auto Batch |
|-------|--------------|------------|
| d8    | 0.44B        | 2^18 = 262K |
| d12-16| 0.7B-2.5B   | 2^19 = 524K |
| d18-26| 3.4B-9.6B   | 2^20 = 1.05M |

**Weight decay** (empirically derived, LOG.md):

| Depth | Width | Optimal WD |
|-------|-------|-----------|
| d8    | 512   | ~0.40 |
| d12   | 768   | ~0.22 |
| d16   | 1024  | ~0.10 |
| d20   | 1280  | ~0.08 |

Power law fit: `WD ∝ 1/width²`. Scale from reference: `WD_target = WD_ref × (width_ref/width_target)²`.

---

## 10. Python GC overhead: disable after warmup

From deepwiki section 3:
> "GC is disabled after step 1 to prevent 500ms overhead from cycle detection."

500ms × 880 steps ≈ 7 minutes lost to GC on a 2.76h run (4.4% overhead). Disable safely after step 1 when allocation patterns stabilize.

---

## 11. Cautious weight decay + torch.compile gotcha

From deepwiki section 12:
> "Must inline logic in optimizer step. Passing `weight_decay` as function argument triggers torch.compile recompilation on schedule changes."

```python
# Good: read at step time from group dict
for group in param_groups:
    wd = group["weight_decay"]  # no recompile on schedule change

# Bad: pass as argument (recompiles when wd changes)
def step(self, wd):  # triggers recompile every step if wd schedule varies
```

---

## 12. Compute-optimal ratio: 10.5 (Kaplan-style counting)

From LOG.md sweeps across parameter-counting methods:
- Kaplan-style (projections including lm_head, no embeddings): stable 10.5 ratio across scales
- Chinchilla-style (all params): varies 3.0-4.0

For speedrun: deliberately undertrain to ratio ~9.5 (saves ~2-3h) to hit GPT-2 CORE threshold.

---

## 13. FP8 summary

- Effective speedup at d24 scale: ~5% (capability-matched), not the microbenchmark 1.38x
- Memory saving: ~9GB activations stored as FP8 vs BF16
- `torch.compile` mandatory: without it, FP8 is 4× slower
- Only works on Hopper (H100, SM 90+)
- During evaluation: **disable FP8** (use BF16/FP32) -- FP8 introduces ~5% accuracy variance

---

## 14. Key gap this fills

The existing ml_debug skill sources (2017-2021) predate modern LLM pretraining at scale. nanochat is one of the few open-source codebases that publicly documents the empirical decisions behind training a transformer from scratch in 2026, with quantified results: 320+ sweeps, negative results, scaling laws, and specific failure modes.