Make audit findings drive visible research decisions

Co-Authored-By: PI[k3] <288921227+claudypoo@users.noreply.github.com>

.github/workflows/audit.yml

@@ -0,0 +1,15 @@
+name: Audit
+
+on:
+  pull_request:
+  push:
+
+jobs:
+  audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - run: python scripts/audit.py --self-test .

.gitignore

@@ -1,3 +1,4 @@
+__pycache__
 # Personal notes
 docs/wassname.md
 
@@ -6,3 +7,5 @@ docs/spec/
 
 /docs/dlbooks
 *_log.md
+
+slop/

AGENTS.md

@@ -17,5 +17,5 @@ Quoting them is grounding data; rewording them injects assistant bias.
   block preserves the author's reasoning.
 - Put lower-relevance sources in "See also" rather than forcing a synthetic
   narrative around them.
-- In `SKILL.md`, link to reference docs like `refs/research_taste.md` instead
+- In `SKILL.md`, link to reference docs like `references/research_taste.md` instead
   of copying a long assistant-written summary.

PLAYBOOK.md

@@ -16,19 +16,19 @@ How to *think* when generating hypotheses or deciding what to investigate next.
 
 **4. Bias-variance via learning curves.**[^cs229][^fsdl] Plot train and val error vs dataset size (or steps). Both high and converging together = high bias (too simple, wrong features, or a capacity-reducing bug). Train low, val high = high variance (overfitting). Val flat even with 10x more data = not a data problem, fix the model.
 
-**5. Structural ceiling: can the parameterization even express what you want?** Sometimes a metric is stuck not because the optimizer fails but because the architecture literally cannot represent the target. Quick check: disable the loss term entirely; if the metric reaches the same value, the loss never moved it. Worked example in [refs/metric_stuck.md](refs/metric_stuck.md).
+**5. Structural ceiling: can the parameterization even express what you want?** Sometimes a metric is stuck not because the optimizer fails but because the architecture literally cannot represent the target. Quick check: disable the loss term entirely; if the metric reaches the same value, the loss never moved it. Worked example in [references/metric_stuck.md](references/metric_stuck.md).
 
-### Practitioner priors: what's usually wrong
+### Where to look first
 
-With no other information, investigate in this order. Rough consensus from the folklore sources, not measured frequencies, and only a starting weight (a clue that points elsewhere overrides them outright):
+With no other information, this is a reasonable starting order from the folklore sources. It is not a measured distribution of failure causes. Follow direct evidence when it points elsewhere.
 
-1. **Data pipeline** (~40%). Wrong preprocessing, labels misaligned with inputs, missing/wrong normalization, train/test leakage, a loader returning stale batches. It really is usually the data.[^slavv][^fsdl]
-2. **Loss function** (~20%). Wrong loss for the task, wrong sign, double softmax, loss disconnected from the metric, competing losses canceling.
-3. **Training procedure** (~15%). Wrong optimizer step order, missing `zero_grad`, frozen params, in-place ops breaking autograd.
-4. **Architecture** (~10%). Too small to express it, too deep without skips, wrong activation.
-5. **Hyperparameters** (~5%). LR, batch size, weight decay. Almost never the real problem if the code is buggy.
-6. **Numerical** (~5%). NaN, overflow, underflow, usually a symptom of one of the above.
-7. **Environment** (~5%). Library version, GPU memory, nondeterminism, stale cache.
+1. **Data pipeline.** Wrong preprocessing, labels misaligned with inputs, missing/wrong normalization, train/test leakage, or a loader returning stale batches.[^slavv][^fsdl]
+2. **Loss function.** Wrong loss for the task, wrong sign, double softmax, loss disconnected from the metric, or competing losses canceling.
+3. **Training procedure.** Wrong optimizer step order, missing `zero_grad`, frozen parameters, or in-place operations breaking autograd.
+4. **Architecture.** Too small to express the target, too deep without skips, or the wrong activation.
+5. **Hyperparameters.** Learning rate, batch size, or weight decay.
+6. **Numerical behavior.** NaN, overflow, or underflow, often caused by one of the earlier problems.
+7. **Environment.** Library version, GPU memory, nondeterminism, or a stale cache.
 
 For RL, add reward scale/sign as a top-3 issue, and episode-boundary handling (done signals, discounting across resets).
 
@@ -36,8 +36,8 @@ For RL, add reward scale/sign as a top-3 issue, and episode-boundary handling (d
 
 | Signal | Likely meaning | Check |
 |--------|----------------|-------|
-| Init loss << expected (e.g. 0.01 vs 2.3) | Leakage or a shortcut: the model "knows" the answer at init | Are labels in the input? Is test data in train? A trivial feature? Localize with Wassname's NaN-poisoning tracer or backprop-to-input check ([refs/diagnostics.md](refs/diagnostics.md)) |
-| Random input gives the same loss as real input | Pipeline is destroying information (over-aggressive preprocessing, wrong transforms, all-zero input) | Print raw data at each stage; visualize |
+| Init loss << expected (e.g. 0.01 vs 2.3) | Leakage or a shortcut: the model "knows" the answer at init | Are labels in the input? Is test data in train? A trivial feature? Localize with Wassname's NaN-poisoning tracer or backprop-to-input check ([references/diagnostics.md](references/diagnostics.md)) |
+| After training, replacing real inputs with shuffled or random inputs barely changes predictions or the metric | The model may not use the intended input signal; this does not identify the cause | Inspect preprocessing, model wiring, label leakage, and task bias |
 | Predicts the same class for everything | Class imbalance (100:1 -> "always predict majority") | Label-count check; weighted loss or resample |
 | Val much worse than train from the start | Distribution shift between splits | Same preprocessing? Same time period? Same source? |
 | Learning curve flat even with 10x data | NOT data: high bias | Add capacity, fix features, check for capacity-reducing bugs |
@@ -50,8 +50,20 @@ For RL, add reward scale/sign as a top-3 issue, and episode-boundary handling (d
 
 A catalog of small, well-worn checks, in rough dependency order (each assumes the one before). Pull from it; don't run it end-to-end as a ritual.
 
+### Tobin's initial sequence
+
+Use this to choose the next kind of check, not to diagnose from a symptom. Evidence from the
+current model and problem overrides the routing.
+
+1. Set the target metric and a baseline or known result.
+2. Simplify the model, data, and task.
+3. Get the model running, then overfit one batch.
+4. Compare against a known result or simple baseline.
+5. Separate underfitting, overfitting, distribution shift, and validation overfit.
+6. Tune hyperparameters after the earlier checks pass.[^fsdl]
+
 **Step 1: Verify components in isolation.**[^goodfellow][^cs229] Most bugs are "doing the wrong calculation." Test each piece independently.
-- Forward pass: feed known inputs, check output shapes and ranges. `assert` shapes everywhere, since `(None,)` vs `(None, 1)` silently broadcasts into `(None, None)`. (Or make the shapes runtime-checked contracts with jaxtyping[^jaxtyping] + beartype, which turns the #1 silent bug loud.)
+- Forward pass: feed known inputs, check output shapes and ranges. `assert` shapes everywhere, since `(None,)` vs `(None, 1)` silently broadcasts into `(None, None)`. (Or make the shapes runtime-checked annotations with jaxtyping[^jaxtyping] + beartype, which turns the #1 silent bug loud.)
 - Loss: hand-compute a few targets and compare to code output.
 - Data pipeline: sample a batch, print it, eyeball it. Are labels aligned with inputs? Transforms applied correctly?
 - Preprocessing: look at processed inputs as a human. Can *you* solve the task from them?
@@ -71,16 +83,19 @@ A catalog of small, well-worn checks, in rough dependency order (each assumes th
 
 Make complexity pay rent: every added component (physics, dimensions, losses) should improve a metric you care about, or come out.
 
-**Step 3: Log everything, then look for specific pathologies.**[^goodfellow][^rahtz][^cs231n] Log train+val loss (per-component if multi-objective), gradient norms per module, learning rate, parameter-update magnitudes, the update-to-data ratio per layer (`((lr * p.grad).std() / p.data.std()).log10()`, target ~-3), activation stats (mean, std, dead-ReLU fraction, tanh saturation), and input/label distributions.
+**Step 3: Log everything, then look for specific pathologies.**[^goodfellow][^rahtz][^cs231n] Log train+val loss (per-component if multi-objective), gradient norms per module, learning rate, actual parameter-update magnitudes, activation stats (mean, std, dead-ReLU fraction, tanh saturation), and input/label distributions. For Adam and AdamW, measure the parameter change across `optimizer.step()`; `lr * grad` is not the applied update.
 
 **Sanity-check the loss at init**[^cs231n]: verify chance-level loss before training. For 10-class softmax the initial loss should be `-ln(0.1) = 2.302` with small random weights. Wrong init loss means a bad initialization or a broken loss. Then check that increasing regularization increases the loss.
 
-| Symptom | Likely cause |
+These are candidate causes to distinguish, not diagnoses. Use the model's data, code, and log to
+choose the check.
+
+| Symptom | Candidate causes |
 |---|---|
 | Loss stuck from the start | LR too low, bad init, data pipeline broken, wrong loss function |
 | Loss decreases then explodes | LR too high, numerical instability (log(0), div by 0), gradient-accumulation bug |
-| Loss NaN | log(0), 0/0, overflow. Use `log(x.clamp(min=1e-8))`, `1/(std + 1e-5)` |
-| Train loss good, val loss bad | Overfitting. More data, regularization, smaller model |
+| Loss NaN | Insert `assert torch.isfinite(x).all()` after successive pipeline stages; the first failure localizes the invalid operation. Add a clamp or epsilon only when the intended math requires that boundary behavior |
+| Train loss good, val loss bad | Check split construction, preprocessing parity, and eval mode. If those pass, overfitting is likely |
 | Loss oscillates wildly | LR too high, batch too small, data shuffling broken |
 | Gradients vanish | Too-deep net without skips, saturating activations, bad init |
 | Gradients explode | No gradient clipping, LR too high, RNN without clipping |
@@ -131,7 +146,7 @@ Unfortunately, agents need these procedural mindset-shifts spelled out. This is
 
 Roughly in this order, though the point is the underlying mindset:
 
-**Collect clues before theorizing.** Read the traceback and logs. Run static analysis ([refs/static_analysis.md](refs/static_analysis.md)) and the cheap diagnostics ([refs/diagnostics.md](refs/diagnostics.md): data sanity check, init-loss check, overfit-one-batch). If you catch yourself proposing a fix before you've looked at anything, stop.
+**Collect clues before theorizing.** Read the traceback and logs. Run static analysis ([references/static_analysis.md](references/static_analysis.md)) and the cheap diagnostics ([references/diagnostics.md](references/diagnostics.md): data sanity check, init-loss check, overfit-one-batch). If you catch yourself proposing a fix before you've looked at anything, stop.
 
 **Hold several hypotheses at once; resist converging early.** Unless the cause is already obvious (a traceback usually points right at it), generate at least three genuinely different hypotheses before ranking any, so you don't marry the first one. Use the five lenses in Mental models. Put a rough credence/prior on each, including an explicit unknown bucket when useful. Then sanity-check yourself with:
 - *Bug*: a boring implementation/data/loss bug, with high prior until checked.
@@ -172,13 +187,13 @@ def debug(symptom):
 Rough order to consider, not authoritative; it may not fit your project. Stop when a question fits.
 
 1. Exception/traceback? Read it, fix it, done.
-2. Loss NaN/Inf? Attach NaN hooks ([refs/diagnostics.md](refs/diagnostics.md)), find the first module producing NaN. Usual causes: log(0), 0/0, exp(large); add clamp/eps.
-3. Init loss wrong? Check the data pipeline and loss; check for double softmax; check labels match output format. Same loss on random input -> data destroyed. Init loss << expected -> leakage.
+2. Loss NaN/Inf? Attach NaN hooks ([references/diagnostics.md](references/diagnostics.md)) or insert `assert torch.isfinite(x).all()` after successive stages. Find the first invalid value before changing the math. Common causes include log(0), 0/0, and exp(large).
+3. Init loss wrong? Check the data pipeline and loss; check for double softmax; check labels match the output format. A low init loss makes leakage or a shortcut plausible; localize it before changing the model.
 4. Can't overfit one batch? Gradient-flow check: None grads -> disconnected layer; all-zero grads -> dead layer / detach. Check autograd breakers and optimizer step order.
 5. Loss stuck from step 0 but you *can* overfit one batch? LR too low (try 10x), frozen params (check `requires_grad`), wrong loss.
 6. Loss decreases then explodes? LR too high (try 0.1x), log the pre-clip grad norm, hunt numerical instability.
-7. Train good, val bad? Overfitting, not a bug. More data, regularization, smaller model.
-8. Train loss fine but the metric is bad? Loss-metric misalignment ([refs/metric_stuck.md](refs/metric_stuck.md)).
+7. Training performance good but validation performance poor? First check for a train/validation mismatch or an evaluation bug. If those checks pass, overfitting is likely.
+8. Train loss fine but the metric is bad? Loss-metric misalignment ([references/metric_stuck.md](references/metric_stuck.md)).
 9. Outputs constant? Mode collapse: class imbalance, all-zero init, dead ReLUs, look at confidence-sorted errors.
 10. Slow but not stuck? Not a bug. Consider batch size, depth/width, data quality.
 
@@ -201,12 +216,12 @@ These are the overconfident reflexes the "calibrate" section warns about, made c
 
 Look these up when the symptom calls for them; they're kept out of the main flow on purpose.
 
-- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU. For when a custom loss misbehaves.
-- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check (is the optimizer failing, or can the parameterization not express it?).
-- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, so a result is "reliably better" not "a lucky seed."
-- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist.
-- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
-- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets (init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check).
+- [references/loss_surface.md](references/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU. For when a custom loss misbehaves.
+- [references/metric_stuck.md](references/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check (is the optimizer failing, or can the parameterization not express it?).
+- [references/sweeps.md](references/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, so a result is "reliably better" not "a lucky seed."
+- [references/llm_judges.md](references/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist.
+- [references/static_analysis.md](references/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [references/diagnostics.md](references/diagnostics.md) — copy-paste diagnostic snippets (init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check).
 - [rl/SKILL.md](rl/SKILL.md) — RL-specific debugging: probe environments, reward engineering, HP defaults, reference implementations.
 - [pinn/SKILL.md](pinn/SKILL.md) — physics-informed-network debugging: nondimensionalization, gradient pathologies, curriculum.
 
@@ -214,17 +229,17 @@ Look these up when the symptom calls for them; they're kept out of the main flow
 
 Folklore sources (the quotes above trace to these):
 
-[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
-[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
-[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
-[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
-[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
-[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies, write-from-scratch, assume-bug, raise-threshold, loss-curve)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion, investigate-confusion, think-more, don't-implement-RL-yourself)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating, standardize-observations; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/pdf/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions, implementation-differences)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote, emergency checklist)
 [^fsdl]: Josh Tobin, Full Stack Deep Learning Spring 2021, Lecture 7 "Troubleshooting DNNs" — https://fullstackdeeplearning.com/spring2021/lecture-7/ ([cache](docs/evidence/fsdl_spring2021_lecture7.md))
-[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt, weights-adapt-to-compensate)
 [^cs229]: Andrew Ng, CS229 "Advice for Applying Machine Learning" — https://cs229.stanford.edu/ ([cache](docs/evidence/cs229_ml_advice.md))
 [^jaxtyping]: Patrick Kidger, jaxtyping (runtime shape/dtype checking) — https://github.com/patrick-kidger/jaxtyping
 
-For modern transformer pretraining specifically (the sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+For modern transformer pretraining specifically (the sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat experiment log](https://github.com/karpathy/nanochat/blob/master/dev/LOG.md) (320+ empirical HP sweeps for a GPT-2-scale run). Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
 
 Curated by [wassname](https://github.com/wassname).

README.md

@@ -5,18 +5,758 @@ In an attempt to upskill the machine learning debugging on AI coding assistants
 ## Use as a Claude skill
 
 ```
-/skills add https://github.com/wassname/ml_debug
+/skills add https://github.com/wassname/ml-debug
 ```
 
 Or paste `SKILL.md` into your system prompt / context when debugging.
 
 ## What's here
 
-- **[SKILL.md](SKILL.md)** -- the main artifact. Load into an LLM agent's context as a debugging skill. A short calibration note, then the folklore itself: verbatim sourced quotes from practitioners, general lessons first, modern transformers and LLM fine-tuning in their own section.
+- **This README** -- the folklore, for humans: verbatim sourced quotes from practitioners, general lessons first, modern transformers and LLM fine-tuning in their own section.
 
-- **[PLAYBOOK.md](PLAYBOOK.md)** -- the synthesized long-form: mental models, practitioner priors, step catalogs, symptom tables, the agent debugging loop, triage, and anti-patterns. Menus of hypotheses distilled from the same sources, not quotes. Deeper one-off tricks (loss-surface analysis, stuck-metric diagnosis, sweep reliability) live in [refs/](refs/).
+- **[SKILL.md](SKILL.md)** -- what an agent loads: the folklore turned into instructions, each with a trigger, a form to fill, and output to show the user. "Assume you have a bug" becomes "send a subagent to find one and report what it found". This is a bet that a form gets filled where a principle gets skipped, and it is untested. The bet is worth making because the folklore version measured no gain (below), and because forms have their own failure mode: they get filled with plausible content that nobody checked.
 
-- **[docs/evidence/](docs/evidence/)** -- frozen local copies of source material (blog posts, talks, papers, reddit threads). Claims in SKILL.md link back to exact quotes here.
+- **[PLAYBOOK.md](PLAYBOOK.md)** -- the synthesized long-form: mental models, practitioner priors, step catalogs, symptom tables, the agent debugging loop, triage, and anti-patterns. Menus of hypotheses distilled from the same sources, not quotes. Deeper one-off tricks (loss-surface analysis, stuck-metric diagnosis, sweep reliability) live in [references/](references/).
+
+- **[docs/evidence/](docs/evidence/)** -- frozen local copies of source material (blog posts, talks, papers, reddit threads). Claims here link back to exact quotes.
+
+## Folklore
+
+
+### The rules, before the rules (Agans)
+
+Most of this folklore's lineage goes back to a 2002 debugging book for general
+electronics and software. Its nine rules, in full, from chapter 2:[^agans]
+
+> UNDERSTAND THE SYSTEM
+> MAKE IT FAIL
+> QUIT THINKING AND LOOK
+> DIVIDE AND CONQUER
+> CHANGE ONE THING AT A TIME
+> KEEP AN AUDIT TRAIL
+> CHECK THE PLUG
+> GET A FRESH VIEW
+> IF YOU DIDN'T FIX IT, IT AIN'T FIXED
+
+Each rule is worth the full Remember summary at the end of its chapter. The
+ones that map most directly onto agent debugging:
+
+> **Quit Thinking and Look**: You can think up thousands of possible reasons
+> for a failure. You can see only the actual cause.
+>
+> See the failure. The senior engineer saw the real failure and was able to find the cause. The junior guys thought they knew what the failure was and fixed something that wasn't broken.
+> See the details. Don't stop when you hear the pump. Go down to the basement and find out which pump.
+> Build instrumentation in. Use source code debuggers, debug logs, status messages, flashing lights, and rotten egg odors.
+> Add instrumentation on. Use analyzers, scopes, meters, metal detectors, electrocardiography machines, and soap bubbles.
+> Don't be afraid to dive in. So it's production software. It's broken, and you'll have to open it up to fix it.
+> Watch out for Heisenberg. Don't let your instruments overwhelm your system.
+> Guess only to focus the search. Go ahead and guess that the memory timing is bad, but look at it before you build a timing fixer.
+
+> **Change One Thing at a Time**: You need some predictability in your life.
+> Remove the changes that didn't do what you expected. They probably did
+> something you didn't expect.
+>
+> Isolate the key factor. Don't change the watering schedule if you're looking for the effect of the sunlight.
+> Grab the brass bar with both hands. If you try to fix the nuke without knowing what's wrong first, you may have an underwater Chernobyl on your hands.
+> Change one test at a time. I knew my VGA capture phase was broken because nothing else was changing.
+> Compare it with a good one. If the bad ones all have something that the good ones don't, you're onto the problem.
+> Determine what you changed since the last time it worked. My friend had changed the cartridge on the turntable, so that was a good place to start.
+
+> **If You Didn't Fix It, It Ain't Fixed**: And now that you have all these
+> techniques, there's no excuse for leaving it unfixed.
+>
+> Check that it's really fixed. Don't assume that it was the wires and send that dirty fuel filter back onto the road.
+> Check that it's really your fix that fixed it. "Wubba!" might not be the thing that did the trick.
+> Know that it never just goes away by itself. Make it come back by using the original Make It Fail methods. If you have to ship it, ship it with a trap to catch it when it happens in the field.
+> Fix the cause. Tear out the useless eight-track deck before you burn out another transformer.
+> Fix the process. Don't settle for just cleaning up the oil. Fix the way you design machines.
+
+Full verbatim chapter summaries are in the [evidence notes](docs/evidence/agans_debugging_9_rules.md);
+the complete book text lives in the dlbook repo.
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts:
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+I'll add. for LLM's I suggest assuming every negative results is a bug, and 1) reviewing associated code and output logs to find the top 5 reasons/probabilities why the results might be invalid 2) to avoid skimming this report should involve quoting and interpreting to the user about everything, which should include at least: config, weird code / engineering, data, eval and importantly the log and metrics behaviour and demos in it. It should often include looking at a random sample of output and comparing it to the expected output. - wassname
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+These are really important to flag to the user and investigate patiently
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catches what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([references/diagnostics.md](references/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([references/sweeps.md](references/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> Do note that switching to the BOS dataloader changes the validation loss and makes all previous experiments not comparable in absolute value of the loss, because we have a lot fewer "confusing" tokens in the train/val batches. [...] Therefore, the loss appears lower but this is "fake" to some extent.[^nanochat]
+
+> Original implementation clipped local gradients before sync. Since this codebase doesn't use DDP (gradient sync is in the optimizers), each rank was clipping based on its own local norm.[^nanochat]
+
+He then removed clipping altogether: "Grad norm never exceeds 1.0 naturally, so clipping is always inactive", and it cost ~2% in time from the all-reduce.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## The eight common mistakes
+
+On 2026-08-25 I named the eight failure modes I see most often, from AI agents and from myself, and
+SKILL.md turns each one into an exercise. The quotes below were mined from the evidence cache in
+[docs/evidence/](docs/evidence/) to back them. Coverage is uneven and worth knowing about: mode 6
+has only three quotes and none of them says "read the log" in those words, and no source here
+argues against similarity probes by name, so the mode 7 quotes attack the general substitution
+instead.
+
+### 1. Overconfidence, a diagnosis stated as fact
+
+From William Falcon's attendee notes on Schulman's talk, so a secondary source rather than
+Schulman's own text[^deeprlhacks]:
+
+> 4. Think your algorithm is working but you're actually seeing random noise.
+>     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
+
+Nanda on why no internal warning fires:
+
+> Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research.[^nanda-mindsets]
+
+Victor Sanh names the state in which a confident report is worthless:
+
+> **The challenge lies in the fact that you can make these mistakes, train a model without it ever crashing, and still get a decent performance…**[^sanh]
+
+Seed noise alone can clear a significance bar, from a different section of the Google playbook:
+
+> -   It is all well and good to make comparisons of validation error rates
+>     estimated on a finite validation set using fastidious statistical tests, but
+>     often the trial variance alone can produce statistically significant
+>     differences between two different trained models that use the same
+>     hyperparameter settings.[^tuning-playbook]
+
+The one question that turns "am I overconfident" into something answerable:
+
+> **How reliable is my experiment?** Ask yourself: "How surprised would I be if it turned out to be complete bullshit due to a bug, error, noise, misunderstanding, etc.?" Investigate the most uncertain bits[^nanda-papers]
+
+And from an unpublished Nanda draft quoted in [references/research_taste.md](references/research_taste.md), so
+weaker provenance than his published posts:
+
+> Insufficient Skepticism: Missing simple alternative explanations, methodological flaws, or bugs. Explicitly list alternatives. Get others (especially mentors) to red team your plans before you run them. Actively try to break your hypothesis. Ask "What observation would make me abandon this?"[^nanda-taste]
+
+### 2. Quitting after one change, calling the negative real
+
+Steinhardt gives the error a number, and SKILL.md builds an exercise on this one:
+
+> **Trying an experiment and seeing it fail gives little information by itself.** When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work".[^steinhardt]
+
+> When ruling out ideas, it is important to hold oneself to a high standard. "This doesn't seem like it will work" or "I feel less motivated after trying a few things along this line that didn't work" are _not_ ruling out an idea.[^steinhardt]
+
+The textbook states the confusion as the default condition, not an edge case:
+
+> When a machine learning system performs poorly, it is usually difficult to tell whether the poor performance is intrinsic to the algorithm itself or whether there is a bug in the implementation of the algorithm. Machine learning systems are difficult to debug for various reasons.[^goodfellow]
+
+Irpan, reproducing a paper with its first author sitting nearby, another quote SKILL.md turns into
+an exercise:
+
+> It ended up taking me 6 weeks to reproduce results, thanks to several software
+> bugs. The question is, why did it take so long to find these bugs?[^irpan]
+
+Karpathy's nanochat log is the model of how to write a negative honestly, recording the effort spent
+and keeping the idea alive:
+
+> **Result:** This was not an out-of-the-box win for nanochat even with a mild attempt over a few hours at a bit of tuning and debugging. The idea itself is intuitively appealing. Might come back around later to try harder later.[^nanochat]
+
+Miller's recommendations, where item 5 is the check on the whole mode and item 4 is the pairing rule:
+
+> Our specific recommendations to researchers include: 1. Computing standard errors of the mean using the Central Limit Theorem 2. When questions are drawn in related groups, computing clustered standard errors 3. Reducing variance by resampling answers and by analyzing next-token probabilities 4. When two models are being compared, conducting statistical inference on the question-level paired differences, rather than the population-level summary statistics 5. Using power analysis to determine whether an eval (or a random subsample) is capable of testing a hypothesis of interest[^miller]
+
+Rahtz writes the one-change-then-declare loop out as a transcript, priced in a week of wall clock:
+
+> If you keep that strategy when each run takes 10 hours, though, you can easily
+> waste a *lot* of time. Last run didn’t work? OK, I think it’s this thing. Let’s
+> set off another run to check. Coming back the next morning: still doesn’t work?
+> OK, maybe it’s this other thing. Let’s set off another run. A week later, you
+> still haven’t solved the problem.[^rahtz]
+
+### 3. Anchoring on the first idea
+
+Rahtz explains why anchoring feels correct, and when it actually is:
+
+> than forming hypotheses. Why spend 15 minutes carefully considering everything
+> that could be causing what you see when you can check the first idea that jumps
+> to mind in a fraction of that (and gather more evidence in the process)? To put
+> it another way: if you have rapid feedback, you can narrow down the hypothesis
+> space a lot faster by trying things than thinking carefully.[^rahtz]
+
+Nanda attacks anchoring at the root, and also attacks the fix:
+
+> The standard hypothesis testing framework can be misleading here, because it has an implicit frame of being able to list all the hypotheses. But actually, most of your probability mass should normally be on “something I haven’t thought of yet”[^nanda-mindsets]
+
+> If trying to explain something mysterious, novice researchers often neglect simple, dumb hypotheses like “maybe MLP0 is incredibly important on *every* input, and there’s nothing special going on with my prompt”[^nanda]
+
+Steinhardt, on hypotheses 2 and 3 turning out to be hypothesis 1 wearing a hat:
+
+> Importantly, it is often not obvious that multiple approaches to a problem all have the same issue. In the past, I have spent months trying different approaches to a problem before finally stepping back and realizing that they were all failing for the same reason. Moreover, I had all the data necessary to make this realization a couple weeks in but had failed to do so.[^steinhardt]
+
+Josh Tobin's symptom table, where every symptom has two or three candidates and only one is a
+learning rate:
+
+> * **Error goes up**: Commonly, this is due to a flip sign somewhere in
+>   the loss function/gradient.
+> * **Error explodes**: This is usually a numerical issue but can also
+>   be caused by a high learning rate.
+> * **Error oscillates**: You can lower the learning rate and inspect
+>   the data for shuffled labels or incorrect data augmentation.
+> * **Error plateaus**: You can increase the learning rate and get rid
+>   of regulation. Then you can inspect the loss function and the data
+>   pipeline for correctness.[^fsdl]
+
+And the explicit step, again from the unpublished draft. Note it asks for the simplest explanations,
+not more of the same kind as hypothesis 1:
+
+> Actively Seek Alternatives: Explicitly brainstorm other ways your observations could be explained. What are the simplest explanations? What known circuits or phenomena could be involved? What would a strong skeptic argue?[^nanda-taste]
+
+### 4. Obsession with the legible hyperparameters
+
+Achiam gives both the ordering agents invert and the reason for it:
+
+> **If it doesn’t work, assume there’s a bug.** Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it’s a bug. Bad hyperparameters can significantly degrade RL performance, but if you’re using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue.[^spinningup]
+
+Karpathy's five worked examples of silent failure, where the legible hyperparameters arrive last, in
+one clause:
+
+> For example, perhaps you forgot to flip your labels when you left-right flipped the image during data augmentation. Your net can still (shockingly) work pretty well because your network can internally learn to detect flipped images and then it left-right flips its predictions. Or maybe your autoregressive model accidentally takes the thing it’s trying to predict as an input due to an off-by-one bug. Or you tried to clip your gradients but instead clipped the loss, causing the outlier examples to be ignored during training. Or you initialized your weights from a pretrained checkpoint but didn’t use the original mean. Or you just screwed up the settings for regularization strengths, learning rate, its decay rate, model size, etc.[^karpathy-recipe]
+
+Sanh treats a weird optimal hyperparameter as a symptom to explain, not a setting to keep:
+
+> Most importantly, there is no point of launching 1000 runs with different hyperparameters (or architecture tweaks like activation functions): **compare a couple of runs with different hyperparameters to get an idea of which hyperparameters have the highest impact** but in general, it is delusional to expect to get your biggest jumps of performance by simply tuning a few values. For instance, if your best performing model is trained with a learning rate of 4e2, there is probably something more fundamental happening inside your neural network and you want to identify and understand this behavior so that you can re-use this knowledge outside of your current specific context.[^sanh]
+
+Daniel Ziegler's self-study, reported second-hand by an 80,000 Hours career guide:
+
+> Once the algorithm was partially working, they would attain higher performance by looking for remaining bugs, both by reviewing the code carefully, and by collecting metrics such as average policy entropy to perform sanity-checks, rather than just tune hyperparameters.[^olsson]
+
+Sweeping the obvious hyperparameters is brute-force search wearing a lab coat:
+
+> Third, and perhaps most important for building skill,[[1]](https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming#fn289bs9hi65b)you must **notice** when you're going into brute-force search mode, and then **take action** by investing time in understanding the underlying system, until both the problem and solution make sense.[^ulisse]
+
+Last, a specimen rather than advice. An anonymous reddit self-report from a self-described
+non-expert, nine hyperparameters turned and the agent still does not learn. In the same thread he
+reports his two real bugs on that environment were a terminal-flag masking error and a shape
+broadcast, neither of which any of these can reach[^reddit-rl]:
+
+> Things I've tried (but maybe not systematically enough):
+>
+> * Different initial LRs
+> * Different optimizers
+> * Different number of hidden layers/units
+> * Shared pi/V NN body (with diff output layers) vs not
+> * Changing amount of entropy
+> * Adding correlated noise
+> * Using TD residual instead of MC version
+> * Clipping the gradient
+> * Different gamma values
+
+### 5. Not reading the data
+
+The textbook naming the exact drift, and why the scalar cannot police itself:
+
+> Visualize the model in action: When training a model to detect objects in images, view some images with the detections proposed by the model displayed superimposed on the image. When training a generative model of speech, listen to some of the speech samples it produces. This may seem obvious, but it is easy to fall into the practice of looking only at quantitative performance measurements like accuracy or log-likelihood. Directly observing the machine learning model performing its task will help to determine whether the quantitative performance numbers it achieves seem reasonable. Evaluation bugs can be some of the most devastating bugs because they can mislead you into believing your system is performing well when it is not.[^goodfellow]
+
+Henderson et al. on a healthy-looking curve produced by a policy that has learned nothing anyone
+wanted (the "demon-strated" break is an OCR artifact in the cached copy):
+
+> By reaching a local optimum, learning curves can indicate successful optimization of the policy over time, when in reality the returns achieved are not qualitatively representative of learning the desired behaviour, as demon-strated in video replays of the learned policy 5. Therefore, it is important to show not only returns but demonstrations of the learned policy in action.[^henderson]
+
+"Read the data" as a pass/fail test that takes a minute, again from the DeepRLHacks attendee
+notes[^deeprlhacks]:
+
+> 2. Make sure observations usable:
+>     - See if YOU could control the system by using the same observations you give the agent.
+>       - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
+
+For LLM work, the data you have to read is the tokenized data:
+
+> Pro-tip: when you work with language, have a serious **look at the outputs of the tokenizers**. I can’t count the number of lost hours I spent trying to reproduce results (and sometimes my own old results) because something went wrong with the tokenization.[^sanh]
+
+Ng names the motivational failure rather than the procedural one:
+
+> Error analysis can often help you figure out how promising different directions are. I’ve seen many engineers reluctant to carry out error analysis. It often feels more exciting to just jump in and implement some idea, rather than question if the idea is worth the time investment. This is a common mistake: It might result in your team spending a month only to realize afterward that it resulted in little benefit.[^ng-mly]
+
+And reading one process's data is not reading the data when eight processes disagree:
+
+> ⚠️ If you are doing distributed training, print samples of your dataset in each process and triple-check that you get the same thing. One common bug is to have some source of randomness in the data creation that makes each process have a different version of the dataset.[^hfcourse]
+
+### 6. Not reading the log
+
+The closest thing in the cache to a hard rule that you read the run before you report its number,
+from a team with every excuse to just read the number:
+
+> -   Although in many cases the primary objective of our experiments only
+>     requires considering the validation error of each trial, we must be careful
+>     when reducing each trial to a single number because it can hide important
+>     details about what’s going on below the surface.
+> -   For every study, we always look at the **training curves** (training error
+>     and validation error plotted versus training step over the duration of
+>     training) of at least the best few trials.[^tuning-playbook]
+
+A price tag on skipping a boring number, from Rahtz:
+
+> (I missed
+> a multithreading bug for several months by ignoring a small but mysterious
+> decay in frames per second.)[^rahtz]
+
+Bekman, where the visible symptom was an artifact of the resume and the data sampler, so every
+hypothesis about the optimizer or the precision would have been confidently wrong:
+
+> There was no real spike in the two earlier runs. The loss never went up in the first place. In both resumes it was under-reporting loss due to an exactly repeated data and then it reached data it hasn't seen before and started reporting correctly. In other words it was overfitting and reporting a false loss.[^bekman-book]
+
+### 7. A cheap indirect probe instead of running the real thing
+
+A published case where a clever mechanism turned out to be norm damage, and the cheap real test
+that the indirect story never ran:
+
+> **Do ablations on your fancy method**: It's easy for people to have a fancy method with lots of moving parts, when many actually are unnecessary. You should always try removing one part and see if the method breaks. Do this for each part.
+>     *   For example, the [original unlearning method](https://arxiv.org/abs/2403.03218v1) in the [RMU paper](https://arxiv.org/abs/2403.03218) claimed it was based on finding a meaningful steering vector, until follow-up work found that it was just about adding a vector with really high norm that broke the model, and a random vector performed just as well.[^nanda]
+
+Ng's shortest statement of build-it-and-run-it, from CS229 slides, where the line breaks are the
+PDF's. His next slide caveats that this is worse advice when the goal is to invent new algorithms.
+
+> The only way to find out what needs work is to implement something quickly,
+>
+> and find out what parts break.[^cs229]
+
+A convenient proxy metric silently deleting the one object the task was about:
+
+> Figure 15.5: An autoencoder trained with mean squared error for a robotics task has failed to reconstruct a ping pong ball. The existence of the ping pong ball and all its spatial coordinates are important underlying causal factors that generate the image and are relevant to the robotics task. Unfortunately, the autoencoder has limited capacity, and the training with mean squared error did not identify the ping pong ball as being salient enough to encode.[^goodfellow-ch15]
+
+What a scalar proxy costs, which is a different point from reading your data for quality:
+
+> One of the key drivers of progress in mech interp is an openness to qualitative research: summary statistics lose a ton of information. What can we learn by actually looking deeply into what's happening?[^nanda]
+
+When the metric will not move, run the real objective on known inputs:
+
+> 1. **Test reward function standalone**: Run it outside training with known inputs to verify it returns nonzero values.[^axolotl-stability]
+
+### 8. An arbitrary threshold set before you know what is fair
+
+The textbook killing the invented threshold from first principles, and another quote SKILL.md builds
+an exercise on:
+
+> In most cases, we do not know a priori what the intended behavior of the algorithm is. In fact, the entire point of using machine learning is that it will discover useful behavior that we were not able to specify ourselves. If we train a neural network on a new classification task and it achieves 5 percent test error, we have no straightforward way of knowing if this is the expected behavior or suboptimal behavior.[^goodfellow]
+
+Nanda states the default and names the fix as a baseline rather than a chosen cutoff. This is from an
+unpublished draft, the passage never made the published post, and SKILL.md uses it too:
+
+> A valuable intuition to have in mind is that, by default, all numbers are meaningless because we lack any scale to compare them. E.g. if a probe gets 95% classification accuracy on some task, is this good? Is this bad? Hard to say without knowing more! Baselines are one way to get context to compare against.[^nanda-draft]
+
+A worked case where a fixed cutoff is meaningless until you know the scale of the quantity. The fix
+is a scale-free metric, not an argument about where the cutoff sits. The typo is in the source.
+
+> You might be temped to keep track of the difference \(\mid f’\_a - f’\_n \mid \) or its square and define the gradient check as failed if that difference is above a threshold. However, this is problematic. For example, consider the case where their difference is 1e-4. This seems like a very appropriate difference if the two gradients are about 1.0, so we’d consider the two gradients to match. But if the gradients were both on order of 1e-5 or lower, then we’d consider 1e-4 to be a huge difference and likely a failure.[^cs231n]
+
+Four questions Sanh asks before any number can be called good or bad. The last one, what you cannot
+conclude from a perfect score, is the specific antidote:
+
+> *   How would a random predictor perform (especially in classification problems)? Dataset can be unbalanced…
+> *   What would the loss look like for a random predictor?
+> *   What is (are) the best metric(s) to measure progress on my task?
+> *   What are the limits of this metric? If it’s perfect, what can I conclude? What can’t I conclude?[^sanh]
+
+The constructive alternative, compute what random gets and treat any distance from it as a bug
+report until shown otherwise:
+
+> If the loss/metric you get on your initial model is very different from the loss/metric you would expect for random predictions, double-check the way your loss or metric is computed, as there is probably a bug there. If you are using several losses that you add at the end, make sure they are of the same scale.[^hfcourse]
+
+The legitimate form of a numeric gate, discovered by reproducing a known-good reference rather than
+chosen in advance:
+
+> 5.   **Rule of thumb: 400 episodic return in breakout**: Check if your PPO could obtain 400 episodic return in breakout. We have found this to be a practical rule of thumb to determine the fidelity of online PPO implementations in GitHub. Often we found PPO repositories not able to do this, and we know they probably do not match all implementation details of `openai/baselines`’ PPO.[^ppo37]
+
+And a floor under any target, because a threshold set tighter than the label noise in your
+validation set is measuring overfitting to errors:
+
+> The issue here isn't just that we might have bad labels in our training set, the issue is that it appears in the validation set. If a machine learning model can become state of the art by squeezing another 0.5% out of a validation set one has to wonder. Are we really making a better model? Or are we creating a model that is better able to overfit on the bad labels?[^koaning]
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/pdf/2108.02497), with its full do/don't list extracted in [references/checklist.md](references/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [references/transformers.md](references/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies, write-from-scratch, assume-bug, raise-threshold, loss-curve)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion, investigate-confusion, think-more, don't-implement-RL-yourself)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data, fixed-seed, overfit-one-batch, Adam-3e-4; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating, standardize-observations; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/pdf/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions, implementation-differences)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky, seed-canary)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote, emergency checklist)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt, weights-adapt-to-compensate)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/pdf/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/pdf/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM, attention-overflow / cosine-sim norm, autoregressive validation, "wiping out a source of instability" / QK RMSNorm)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem, confidence-sort trick, spend-less-time-tuning)
+[^nanochat]: Karpathy, [nanochat experiment log](https://github.com/karpathy/nanochat/blob/master/dev/LOG.md) ([cache](docs/evidence/karpathy_nanochat_experiments.md))
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition, junior-developer, never-accept-the-kludge, don't-delete-and-clone)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale, urban-legend conclusion)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently, test-more-than-one-env, measure-everything)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false, excitement-is-bullshit, read-your-data)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote, single-anomaly)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state, brute-force-search, leaky-abstractions)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends, valley-of-bad-theory)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline, overfit-one-batch, no-tuning-before-baseline)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose, detection-and-frame-buffer, previous-frames)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS, shuffle-eval, all-labels–100-loss-0)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify, one-process, small-model + fast-iteration, caches)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start, inspect-tokenized-masking, reward-fn-standalone)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system, 100-examples procedure, Eyeball/Blackbox dev sets)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation, scientific/nuisance/fixed, incremental-tuning)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion, insidious-contamination, overfitting-bugbear, features-are-key)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat, 104B post-mortem, spike types + bad-data-pocket, init-std, PaLM batch-skipping, logbooks)
+[^deeprlhacks]: William Falcon, "DeepRLHacks", attendee notes on Schulman's "Nuts and Bolts of Deep RL Research" -- https://github.com/williamFalcon/DeepRLHacks ([cache](docs/evidence/williamfalcon_deeprl_hacks.md): random-noise-not-signal, observations-usable). Secondary source; the primary slide deck is `[^schulman]`.
+[^nanda-mindsets]: Neel Nanda, "My Research Process: Key Mindsets" -- https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/cbBwwm4jW6AZctymL ([cache](docs/evidence/nanda_research_process_key_mindsets.md): insufficient-skepticism-feels-like-research, mass-on-unlisted-hypotheses)
+[^nanda-papers]: Neel Nanda, "Highly Opinionated Advice on How to Write ML Papers" -- https://www.lesswrong.com/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers ([cache](docs/evidence/nanda_highly_opinionated_ml_paper_writing.md): how-reliable-is-my-experiment)
+[^nanda-taste]: Neel Nanda, "My Model of the Research Process", unpublished shared draft, as quoted in [references/research_taste.md](references/research_taste.md) (insufficient-skepticism, actively-seek-alternatives). Draft quality, weaker provenance than the published posts.
+[^nanda-draft]: Neel Nanda, "My Model of the Research Process", unpublished shared draft -- https://docs.google.com/document/d/1YMkeMrhqsWxZcNDD9CIUWEK_DAOegeufnbc79U2hycg/edit ([cache](docs/evidence/nanda_research_process_shared_draft.md): all-numbers-are-meaningless). This passage never made it into the published post.
+[^sanh]: Victor Sanh, "Simple considerations for simple people building fancy neural networks" (HF, 2021) -- https://huggingface.co/blog/simple-considerations ([cache](docs/evidence/sanh_simple_considerations_hf_2021.md): decent-performance-without-crashing, read-the-tokenizer-output, 4e2-is-a-symptom, pre-training questions)
+[^steinhardt]: Jacob Steinhardt, "Research as a Stochastic Decision Process" -- https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html ([cache](docs/evidence/steinhardt_research_stochastic_decision_process.md): 0.1%-of-implementations, high-standard-for-ruling-out, months-of-approaches-one-cause)
+[^miller]: Evan Miller (Anthropic), "Adding Error Bars to Evals" (2024) -- https://arxiv.org/pdf/2411.00640 ([cache](docs/evidence/miller_2024_error_bars_evals.md): five recommendations, question-level pairing, power analysis). arXiv preprint, not peer reviewed.
+[^agans]: David J. Agans, *Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems*, AMACOM, 2002 ([notes](docs/evidence/agans_debugging_9_rules.md): nine rules and Remember summaries verbatim; complete book text in the private dlbook repo)
+[^fsdl]: Josh Tobin, Full Stack Deep Learning Spring 2021 lecture 7, "Troubleshooting Deep Neural Networks", notes by James Le and Vishnu Rachakonda -- https://fullstackdeeplearning.com/spring2021/lecture-7/ ([cache](docs/evidence/fsdl_spring2021_lecture7.md): error up/explodes/oscillates/plateaus table)
+[^olsson]: Catherine Olsson and the 80,000 Hours team, "ML Engineering for AI Safety and Robustness" -- https://80000hours.org/articles/ml-engineering-career-transition-guide/ ([cache](docs/evidence/olsson_80000hours_ml_engineering_ai_safety.md): bug-hunting-with-diagnostics-over-tuning). Reports Daniel Ziegler's self-study second-hand.
+[^reddit-rl]: u/GrundleMoof, "How to more intelligently debug RL roadblocks?" -- https://old.reddit.com/r/reinforcementlearning/comments/bzg3l2/ ([cache](docs/evidence/reddit_rl_roadblocks_bzg3l2.md): nine-knobs list, terminal-flag and broadcast bugs in the replies). Anonymous self-report from a self-described non-expert; quoted as a specimen of the failure mode, not as authority.
+[^cs229]: Andrew Ng, "Advice for Applying Machine Learning" (CS229 slides) -- https://cs229.stanford.edu/materials/ML-advice.pdf ([cache](docs/evidence/cs229_ml_advice.md): implement-quickly-find-what-breaks, and his own caveat for algorithm invention)
+[^goodfellow-ch15]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 15 "Representation Learning" -- https://www.deeplearningbook.org/contents/representation.html ([cache](docs/evidence/goodfellow_ch15_representation_learning.md): Figure 15.5 ping pong ball / MSE salience)
+[^ppo37]: Huang, Dossa, Raffin, Kanervisto, Wang, "The 37 Implementation Details of Proximal Policy Optimization" (ICLR Blog Track, 2022) -- https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ ([cache](docs/evidence/cleanrl_37_ppo_details.md): 400-return-in-breakout rule of thumb)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/pdf/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC, leakage, look-ahead bias). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat experiment log](https://github.com/karpathy/nanochat/blob/master/dev/LOG.md) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+## Does it help?
+
+Measured on [ml-bench](https://github.com/wassname/ml-bench): 12 hard machine learning research
+problems from my own work, none of them in any training set, each answer graded against my own
+answer by a panel of five LLM judges. A score of 1.00 means the model matched me. The test gives the
+model this SKILL.md and nothing else, so the only change is the document.
+
+No measurable gain, from three answers per question in each arm:
+
+| deepseek-v4-flash-0731, 12 questions | bare | with SKILL.md |
+| --- | --- | --- |
+| mean score | +0.643 | +0.667 |
+| the three runs | +0.608, +0.648, +0.674 | +0.746, +0.641, +0.614 |
+
+The difference is +0.023 with a standard error of 0.044, so it is not distinguishable from zero.
+Pairing by question rather than by run gives the same +0.023 with a standard error of 0.031, t of
+0.76. The runs themselves scatter by more than the difference between the two columns.
+
+An earlier version of this section reported +0.135, or 59% of the distance to gpt-5.6-sol. That was
+one run of each arm, and it happens to be the first run in each column above. It did not survive the
+other two.
+
+Two other readings. With SKILL.md the model writes 31% more text for the same score, so any
+verbosity bias in the judges makes the true effect smaller than +0.023, not larger. And only 1 answer
+in 36 uses the document's own vocabulary, so the document is in the context without changing much of
+what the model writes. The header does tell it not to quote the document back.
+
+Caveats: one model, three answers per question, one judge panel, at bench version v96. The result is
+that this document did not help this model on these questions. It is not evidence about a stronger
+model, a longer task, or an agent that can run code.
+
+### Which part of the document does the work?
+
+A later round swapped the document for cut-down versions of it, on three of the twelve questions,
+four answers per question, grok-4.6 at high reasoning effort. Both controls are documents that
+contain none of this material: `inert doc` gives no instruction at all, and `be thorough` is five
+lines telling the model to work the problem in full and show its work.
+
+*One row is one document loaded in place of SKILL.md. Controls are italic. `struggling` counts
+answers that narrate fetching evidence in a bench that offers no tools, and `mean clean` is the
+mean with those dropped.*
+
+| document | size | mean↑ | mean clean↑ | struggling↓ | version |
+| --- | ---: | ---: | ---: | ---: | --- |
+| *be thorough (control)* | 636 B | *+0.66* | *+0.66* | 0/12 | control |
+| be diligent first, named exercises | 29 K | +0.56 | +0.56 | 0/12 | [`3a58c54`](https://github.com/wassname/ml-debug/blob/3a58c54/SKILL.md) |
+| exercises, almost no quotes | 19 K | +0.53 | +0.53 | 0/10 | ablation |
+| *inert doc (control)* | 771 B | *+0.53* | *+0.53* | 0/12 | control |
+| *bare, no document* | 0 | *+0.44* | *+0.44* | 0/12 | -- |
+| read the data, and give hypotheses | 3.0 K | +0.44 | +0.44 | 0/11 | ablation |
+| quotes and exercises | 26 K | +0.35 | +0.47 | 3/12 | [`efcac5c`](https://github.com/wassname/ml-debug/blob/efcac5c/SKILL.md) |
+| quotes only, no exercises | 40 K | +0.13 | -- | 10/12 | [`d5d725e`](https://github.com/wassname/ml-debug/blob/d5d725e/SKILL.md) |
+
+<sub>Table: 0.0 is the obvious answer each question rejects and 1.0 is my own answer, so a
+negative row is worse than the answer the question was built to reject. Judge `gpt-5.6-terra`,
+bench version v102. The ablation rows were built for the bench and were never committed here; each
+one is kept verbatim in the bench repo, listed in `docs/audits/skill_snapshots/MANIFEST.md`.</sub>
+
+Three readings, all from grok-4.6 alone. The exercises carry what lift there is and the quotes
+cost more than they pay: the two best of the real documents are the ones that lead with the
+exercises, and the quotes-only document collapses, with 10 of its 12 answers going off to narrate
+tool calls instead of answering. A short instruction to be thorough beats every version of this
+document. And the
+quotes do move the specific point they encode, so the loss is elsewhere: on the question about a
+number repeated across windows, bare and the inert control both score 0.00 while every document
+carrying that quote scores 0.75 or better.
+
+The line at the top of SKILL.md telling you to be diligent and show your work is there because of
+the first row of this table. Adding it, and naming the exercises, moved the current document from
+0.096 below bare to 0.115 above it, standard error 0.059, and it gained on all three questions.
+That is the difference of two arm means over 12 answers each, not a paired difference.
+
+
+## Other skills
+
+- https://github.com/param087/agent-ml-skills/blob/main/skills/ml-debugging/SKILL.md (ok, aimed at diverging training not development of novel ml)
+- https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/22-agent-native-research-artifact (dubious, seems mostly vibe written)
 
 ## Citation
 
@@ -25,6 +765,6 @@ Or paste `SKILL.md` into your system prompt / context when debugging.
   title = {ML Debugging Folklore: A Practitioner Debugging Skill for LLM Agents},
   author = {Michael J. Clark},
   year = {2026},
-  url = {https://github.com/wassname/ml_debug/}
+  url = {https://github.com/wassname/ml-debug/}
 }
 ```

SKILL.md

@@ -1,357 +1,367 @@
 ---
 name: ml-debug
-description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+description: "Debug an ML run: read the log, it crashed, the loss will not go down, the metric will not move, is this result real, does A beat B, a spike or anything weird in the log, about to queue a run, or about to write that a result looks fine. Fill the ml-debug form and do the exercises that match your situation. Show the results in your reply. Invoke it yourself; deciding a run does not need it is the behaviour being tested."
 ---
 
-# wassname's ML Debugging Folklore
+Sources, the human-written introduction, and frozen copies of every quote are in
+[README.md](README.md). Paragraphs signed "- wassname" are his. Paragraphs with a `CLAUDE:`
+comment are Claude's wording, with the source of the point stated.
+
+Be diligent. Work the problem in full before you write. State the decisive point early, then give
+the derivation, the mechanism, or the log line behind it, so the reader can check it and not just
+take it. Show the work, not only the conclusion.
+<!-- CLAUDE: at the top because it is the one part with measured uplift; see README results. -->
+
+## Turn evidence into a research decision
+
+The purpose is to advance the user's research goal. Completing a report is supporting work.
+In the reply, show the decisive observations with their controls and denominators, explain
+what they change about the diagnosis, and identify the next experiment or repair and its
+expected distinguishing result. Separate observations from suspected causes. State what
+you actually executed or queued; a recommendation is not an action. Link the complete
+audit for inspection, but a path alone does not satisfy "show the results".
+
+Missing evidence blocks the next experiment only when it could invalidate that experiment,
+make it uninterpretable, or create an unacceptable cost or risk. Explain that connection.
+Otherwise investigate it alongside useful authorized work. Reporting defects are not
+automatic training gates. Keep the full checks below; use them to choose what to do next.
+<!-- Codex: addresses audits that listed checks but hid decisive evidence and delayed experiments. -->
+
+## How ML debugging differs
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task. -- Achiam
+
+> If one part is broken, the other parts can adapt and still achieve roughly acceptable performance -- Goodfellow, Bengio and Courville
+
+> The challenge lies in the fact that you can make these mistakes, train a model without it ever crashing, and still get a decent performance... -- Sanh
+
+The training script has to print the checks, as SHOULD lines written before the run and
+compared after it.
+<!-- CLAUDE: one line from the three quotes above. -->
+
+### Expensive runs
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem -- Godbole, Dahl, Gilmer, Shallue and Nado
+
+If it takes 5 hours to run, we might only get 4 runs a day, so we need to make them as
+informative as possible. We can't schedule a sweep or ablation of 100+ runs, so we make multiple
+changes that will have separate and distinguishable effects on the metrics. What you learn is the
+effect of each change given the others, so record it that way in the mental model. - wassname
+<!-- CLAUDE: last sentence is mine (Sculley's CACE, in README). -->
+
+### How agents fail
+
+> Trying an experiment and seeing it fail gives little information by itself. When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work". -- Steinhardt
+
+> Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research. -- Nanda
+
+Nanda's "fail fast" advice is for a human who over-commits to a direction for a year. Agents fail
+the other way: they skim the log until a line looks like a reason to stop, find a reading of the
+task that permits stopping, or change one hyperparameter and call the idea dead. The other habits
+this file is written against: settling on the first hypothesis because it arrived first; treating
+learning rate and batch size as the whole option space; writing a probe script beside the
+training script, which then has its own bugs; reading the last twenty lines of the log; and
+writing a diagnosis in the tone of a fact when a competing explanation fits the same evidence.
+<!-- CLAUDE: wassname's observations from autoresearch runs ("give up too easy", "skim until they
+find a reason", side-cars, hyperparameter obsession); my wording. -->
+
+## What to keep in the repo
+
+Defaults for a long research loop (runs of an hour or more, a novel method, an agent working
+overnight). A short debugging call on an existing script creates none of these.
+
+Do not write a side-car probe script. Build up the one training script so it has all the metrics
+you need inline as you go, with short interpretable demos at many stages: init, mid train, post
+train, eval, then one long unclipped demo at the end. Demos and probes should not be separate
+runs, they should be quick sanity checks inside the main train script, and the script should write
+`run.md` in Markdown so the log diagnoses in situ instead of needing a second pass. That is how a
+lot of nights get wasted and agents go off track: they make side-cars with their own separate bugs
+and weird correlational measurements, and have nothing to show for it. If we work on the training
+script we watch it get better, we reuse the same code, we understand it better, and we squash the
+bugs. - wassname
+
+`train.py`. One file. The novel part is written as a readable narrative with tensor shapes in
+comments, so a reviewer can follow it top to bottom without opening other files.
+
+Each long run owns `outputs/<date>_<slug>_<seed>/`: resolved config, commit and argv provenance,
+`run.md`, rectangular metrics, ragged demos/generations, and checkpoints. A detached reader must
+be able to reconstruct and sanity-check the run from that directory.
+
+`run.md`, written by the training entry point, is valid Markdown and the result page. Start each
+stage with a heading and breadcrumb, then close it with elapsed time and peak GPU memory when
+relevant. Include the resolved config actually used; a decimated (about 30 to 60 row) metrics table;
+the first train and evaluation examples in raw form and as the model consumes them (for a
+transformer, special tokens and loss mask visible); and one full normal-path demo for every
+LLM-facing stage that exists. Keep stdout sparse and print the log path. Re-emit a compact final
+result block: headline metric, full copyable result table, output path, and run identity.
+
+Keep `TODO validate:`, `FIXME:`, or `SHOULD:` beside the evidence it interprets. `SHOULD:` needs a
+mechanism, derivation, paper, or validated prior run; otherwise use `TODO validate:`. It carries a
+number only after the scale exercise (ex H) has been done.
+
+For a comparative result table: first column is an index linked to source, then short metadata,
+then the headline score and its inputs. Sort by the headline score; put an arrow on every header;
+bold meaningful per-column best cells; italicize controls and baselines; include floors; and use
+one table for each comparable group. Put the headline result and output path at the end of `run.md`.
+
+The raw event trace is the source of truth. Keep JSONL or Inspect records verbatim and link from
+`run.md` with a project-relative path and line where possible. Do not summarize away a failed,
+truncated, incoherent, refusing, saturated, or confounded output.
+
+A smoke test before every costly run: execute the real pipeline end to end on a tiny random model
+and small slice of every train, extract, and evaluation stage. Use real loaders, I/O, LLM calls,
+and evaluation; reduce scale only. Annotate function inputs and outputs with `jaxtyping`, and
+activate `beartype` only for this smoke run (for example, `BEARTYPE=1`). Garbage scores are fine:
+it checks code paths, shapes, and dtypes, not scientific validity. A flipped sign, label leakage,
+an all-`-100` mask, or a bad metric can pass it.
+<!-- CLAUDE: direct compact integration of token-efficient-logging, markdown-tables, setup-repo,
+and jaxtyping. -->
+
+`MENTAL_MODEL.md`, under two pages. What you believe about this system: which changes
+(regularisation, architecture, a bottleneck, loss balance, more data, init scale, optimiser)
+move which metrics, in which direction, and with what credence. Updated after every run in a
+Bayesian way: a credence moves on a cited log line, and a disproved row is marked disproved with
+the line rather than deleted. Read it at the start of every turn. The filled form for each run is
+appended to whatever run log the repo already keeps.
+<!-- CLAUDE: wassname asked for one file; this is his description of its contents. Experimental,
+he has not worked with it yet. -->
+
+## The ml-debug form
+
+Read the whole log first. Save the complete form in the run's audit. Show the decision and
+supporting evidence in the reply as described above; when the user requests the full form
+in chat, include all rows there too. Do not substitute "audit complete" or a file path for
+the findings. Scoring:
+
+- a row answered from memory or expectation, with no quoted log line: 0
+- a row left blank, with no "unknown" and no note on what would fill it: 0
+- deciding this run does not need the form: 0. That decision is the behaviour being tested.
+
+> Read your data. Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad. -- Nanda
+
+> How would a random predictor perform (especially in classification problems)? [...] What would the loss look like for a random predictor? [...] What are the limits of this metric? If it's perfect, what can I conclude? What can't I conclude? -- Sanh
+
+| row | answer |
+|---|---|
+| log length; the config as it appears in the log | |
+| each `SHOULD:` line, then the observed line, quoted | |
+| for every number you cite: its value under a null (chance, ln C, the base model, a random predictor) and where that expectation came from | |
+| at init, before any update: what did the demo show, and how does it compare to the base model or to chance? | |
+| against a dummy (persistence, class prior, null model, simple heuristic) at each stage: which wins, by how much? | |
+| against the baseline model at each stage, on val and on held-out: which wins? | |
+| if the schedule ramps (warmup, OneCycle): at what lr did learning start, at what lr did it stop? | |
+| one full sample, viewed: input as consumed, output, trace. Link or quote it | |
+| at the worst-looking step: loss per term, grad norm per module. Which module does it point to? | |
+| lines in the log that surprised you, quoted, with why. Each ends "explained: ..." or "chasing now" | |
+| what is not in this log that you would need in order to trust it | |
+| three or more diagnoses with a % on each: one bug in the training code, one bug in the eval, one confound or shortcut, some % on unknown. For each, the strongest evidence for and against, from the log. No evidence against means untested | |
+| a fresh subagent, given the training entry point and `run.md` with no diagnosis attached, asked for the top bugs and misconceptions. Its list, quoted, including "found nothing" | |
+| the cheapest test separating the top two diagnoses, and what each predicts | |
+| wall-clock and GPU memory per stage; what would shorten the loop | |
+
+Some rows are an exercise below at less depth. The form is done every time; the exercise is done
+at depth when the routing says so.
 
-Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+## Routing
 
-## How to read this as an LLM
+Before a run, after a run, before you report. At each, do every small item that applies and one
+large item. A small item is under a paragraph. A large one is real work.
 
-If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+Before a run:
+- always: options table (ex A, small), predictions (ex B, small), smoke test
+- if about to change the design, or the last run cannot be explained: pseudocode and external
+  review (ex F, small; the review is delegated)
 
-You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+After a run (finished or crashed):
+- always: the form; second cause for the same number (ex C, small)
+- if it failed: reproduce it, same seed then a different seed, before diagnosing. A failure
+  that does not reproduce is a different problem; write that down
+- if the log has a spike, a flat line, or an impossible value: rows before the spike (ex D, small)
+- if two cycles have passed with no progress: reference implementation (ex E, large)
 
- |Word | Certainty | 
- |-|-|
- |Almost Certain | 86-99% |
- |Highly Likely | 71-85% |
- | Likely | 56-70% |
- |Changes a little better [or less] than even | 46-55% | 
- |Unlikely | 31-45% |
- |Highly Unlikely | 16-30% |
- | Remote | 1-15% |
+Before you report:
+- if about to quote a headline metric: what else could score well (ex G, small)
+- if about to set a threshold: the scale first (ex H, large)
+- if about to say A beats B: three ways it is false (ex I, large)
+- if about to call it negative: one implementation is not the idea (ex J, small), then ex I on
+  your own code
 
- [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+After a change to `train.py` improves a metric: quote the line that moved and give the mechanism
+by which the change moved it. Agans' ninth rule, "if you didn't fix it, it ain't fixed": an
+improvement you cannot explain means something else is compensating.
+<!-- CLAUDE: Agans (docs/evidence/agans_debugging_9_rules.md); the compensation reading is mine,
+via Goodfellow's "other parts can adapt" above. -->
 
+Reference search (ex E), external review (ex F), and the blind reads in the form and ex I are
+subagent jobs, for the same reason each time: the subagent has no diagnosis to defend. The
+diagnosis stays in the main context.
+<!-- CLAUDE: wassname's point that exploring, searching and reviewing suit subagents. -->
 
-## Folklore
+In an autoresearch loop, where the human has left and expects the loop to keep running:
 
+> **NEVER STOP**: Once the experiment loop has begun (after the initial setup), do NOT pause to ask the human if you should continue. Do NOT ask 'should I keep going?' or 'is this a good stopping point?'. The human might be asleep, or gone from a computer and expects you to continue working *indefinitely* until you are manually stopped. You are autonomous. If you run out of ideas, think harder — read papers referenced in the code, re-read the in-scope files for new angles, try combining previous near-misses, try more radical architectural changes. The loop runs until the human interrupts you, period. -- Karpathy, [autoresearch/program.md](https://github.com/karpathy/autoresearch/blob/master/program.md)
 
-### Think more, experiment less
+A job is stopped, or an idea dropped, only after the form, ex I, and ex J are written out.
 
-> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+## Exercises
 
-> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+### ex A: options table (small)
 
+> Build it up as you go, don't think you can build it ahead of time. Be focused on a strong mental model of what options you have (including architectural changes and losses) that you think should affect what metrics in the logs. -- wassname
 
-### Don't write from scratch; start or compare to a working a reference
+The table lives in `MENTAL_MODEL.md` (or in your reply, for a short call). Correct it before each
+run and show it.
 
-> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+| option | metric it should affect | direction and order | what separates it from the other options |
+|---|---|---|---|
 
-> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+Consider architecture and loss changes where they are live choices for this problem, alongside
+data, regularisation, and optimiser. Say which options change in this run and why. Several can
+change in one run if each has its own metric (see Expensive runs). Show the config diff against
+the run you will compare to.
 
-> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+### ex B: predictions (small)
 
-When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+> Before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possibility and brainstorm the cheapest tests that may narrow them down. -- wassname
 
-### Assume you have a bug
+Write down the question this run answers in one sentence, the result that would make you drop the
+idea, and which part is the novel part (everything else is a control). Then:
 
-> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+| risky part | what I expect to see | too weak | too strong | buggy | metric exists? |
+|---|---|---|---|---|---|
 
-> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+Add to `train.py` every metric whose last column says no. The controls: the base model on the same
+inputs; a random direction or shuffled labels through the same pipeline; the method with the novel
+part removed; the metric on data not used to build the intervention. Say how many seeds. Queue the
+run so its finish wakes you, and use the wait to sharpen the predictions.
 
-A bug can also hide, because most ML models have multiple adaptive parts: 
+### ex C: second cause for the same number (small)
 
-> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
-and it may not show in the output at all.
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.' -- Jones
 
-### Default to disbelieving your own results (Neel Nanda)
+Which number does the diagnosis rest on? Quote the code that computes it. What else would produce
+that number, and what second metric separates the two? A cosine near 1 can be a shared mean or a
+collapsed latent. A cosine of 0 between two probe directions says they are orthogonal and nothing
+about whether either probe works, so it rules nothing out.
 
-> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+### ex D: rows before the spike (small)
 
-> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers. -- Bekman
 
-The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+For each spike or collapse, show the rows before it and say which column moved first.
 
-### Understand the system to shrink the search (Ulisse Mini)
+### ex E: reference implementation (large; subagent)
 
-> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance. -- Henderson
 
-### Gears beat black boxes (John Wentworth)
+> If you are stuck, find a working reference implementation and compare it to yours. If nothing jumps out, try a bisection search: adapt their code wholesale, then half their features, and so on. -- wassname
 
-> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+Search for implementations of the nearest method. Rank by: a results table, an issue or note
+saying someone else reproduced it, more than one human contributor, a README with evaluation
+details, other repos that import it. Take the top one or write "no reference exists".
 
+| feature | theirs (file:line) | mine | same? |
+|---|---|---|---|
 
-### Broken code fails silently; measure everything (Spinning Up)
+Include algorithm tweaks, engineering tricks, hyperparameters, and logged metrics. Ask the
+subagent for at least one bug in your module.
 
-Josh Achiam's warning is RL-framed but general:
+### ex F: pseudocode and external review (small; review delegated)
 
-> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+> Summarise your concept and pseudocode and do an external review in scientist mode. Perhaps describe the forward and backward pass as mermaid too. -- wassname
 
-So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+Write the concept in plain English, then compact Python-shaped pseudocode: use Unicode math names
+when they match the method, `←` for conceptual assignment, shapes in trailing comments, and
+parameter counts per module. Omit imports, device moves, error handling, and other boilerplate.
+Add a Mermaid forward/backward diagram when it clarifies the design. Give this material, and no
+diagnosis, to a fresh reviewer from a different model family where one is available. Ask for its
+assumptions, likely bugs, and first test. Show its verdict; if no reviewer is available, say so in
+the report.
 
-### Pursue anomalies; investigate confusion
+### ex G: what else could score well (small)
 
-> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+> The CNN has learned to detect a metal token that radiology technicians place on the patient in the corner of the image field of view at the time they capture the image. -- Zech et al.
 
-> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
->
-> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+> Apparently meaningless identifier columns were the most important predictors. [...] the university only filled out much of this information *after* a grant application was accepted. -- Howard and Gugger
 
-### Read what you actually wrote, not what you meant (gwern)
+For the headline metric, what useless thing could the model learn and still score well (a
+condition of data collection, the class prior, prompt length)? Show the control run or the log row
+that detects it.
 
-> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+### ex H: the scale first (large)
 
-This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+> by default, all numbers are meaningless because we lack any scale to compare them. E.g. if a probe gets 95% classification accuracy on some task, is this good? Is this bad? Hard to say without knowing more! -- Nanda
 
-### Never accept the kludge (Patrick Kidger)
+Before any threshold, run the metric on a null model, a shuffled control, and the current baseline.
 
-Kidger, on why research code is so reliably buggy:
+| metric | null model | shuffled control | current baseline | ceiling the data allows | proposed threshold |
+|---|---|---|---|---|---|
 
-> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+If a threshold has to be used before this table exists, say that it was set without a scale.
 
-> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+### ex I: three ways it is false (large)
 
-His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal! -- Nanda
 
-### Loss curves are a red herring
+> If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky. -- Irpan
 
-> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+Three ways the result can be false, each with the check that decides it. To claim A beats B: the
+baseline, the chance level, the controls, and the seed spread of one condition, as numbers with
+line references. Say whether the effect survived something it was not tuned on (a rephrased
+prompt set, a held-out dataset, another model size). Give a fresh subagent the artifact with no
+conclusion attached and show what it says. Apply the same to a negative result.
 
-(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+### ex J: one implementation is not the idea (small)
 
-### Inspect the data first
+> It ended up taking me 6 weeks to reproduce results, thanks to several software bugs. The question is, why did it take so long to find these bugs? -- Rahtz
 
-> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+| the idea | what I ran (file:line) | one other way to run it | what a bug here would look like |
+|---|---|---|---|
 
-Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+Say what would have to be true for the idea to be alive and your run to still fail.
 
-Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+## Language
 
-> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+LLMs of 2026 are trained to compress speech and use folky or humanistic language, but it's better
+for the agent (and user) to move toward field standard language, it's precise instead of ambiguous
+and communicates more bits of information. They should build a short list of jargon used in the
+main reference paper. Also try to use the user's own language to reduce the translation burden on
+them, but if they are vague use the proper term as well with theirs in parentheses. It's also good
+to include redundant context, for example "the knob" is imprecise and lacks context, "the grad
+norm" is precise but lacks redundant context, "the grad norm in #1" refers to some doc the user
+can't see, while "the grad norm of the kl loss in the 2nd part of training" is precise while
+reminding the user of lots of relevant context in their own language. - wassname
 
-### Labels are often wrong (koaning)
-
-Vincent Warmerdam:
-
-> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
-
-His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
-
-### The tank story: your model learns the confound (gwern)
-
-The canonical data-leakage parable:
-
-> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
-
-gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
-
-### Test-set contamination is insidious (Domingos)
-
-Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
-
-> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
-
-> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
-
-Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
-
-### Overfit one batch first
-
-> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
-
-> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
-
-And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
-
-### The most common neural net mistakes (Karpathy)
-
-The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
-
-> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
-
-> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
-
-> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
-
-Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
-
-### Seed variance: you can't tell a bug from bad luck
-
-> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
-
-> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
-
-Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
-
-### Normalize and scale everything
-
-From the slides[^schulman]:
-> - If observations have unknown range, standardize
-> - Compute running estimate of mean and standard deviation
-> - x' = clip((x - mu)/sigma, -10, 10)
-> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
-> - Standardize prediction targets (e.g., value functions) the same way
-
-Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
-
-### Tricks substitute for each other
-
-On the slides[^schulman]:
-> Always Be Ablating
-> - Different tricks may substitute
-> - Especially whitening
-
-Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
-
-### Changing anything changes everything (Sculley et al.)
-
-Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
-
-> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
-
-This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
-
-### Exploration over exploitation (Google tuning playbook)
-
-The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
-
-> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
-
-Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
-
-> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
-
-### Adam at 3e-4 for baselines (Karpathy)
-
-> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
-
-If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
-
-## Modern transformers and LLM fine-tuning
-
-Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
-
-### Tricks hide in reference code (lucidrains)
-
-lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
-
-> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
-
-Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
-
-### Modern LLM-pretraining gotchas (nanochat)
-
-Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
-
-> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
-
-> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
-
-### When NaN hits, look at the frames before it (Stas Bekman)
-
-Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
-
-> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
-
-Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
-
-### Loss spikes usually mean a bad data pocket (Stas Bekman)
-
-Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
-
-> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
->
-> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
-
-And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
-
-> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
-
-His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
-
-### Walk the pipeline in data order (HF course)
-
-The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
-
-> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
-
-> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
-
-### Chat template and BOS handling must match across train and deploy (unsloth)
-
-When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
-
-> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
-
-Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
-
-> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
-
-### Shrink every axis at once, and clear the caches (axolotl)
-
-Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
-
-> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
-
-> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
-
-Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+Keep the list in `docs/JARGON.md` when working in a long loop.
 
 ## Reference
 
-Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+- [PLAYBOOK.md](PLAYBOOK.md): mental models, component isolation, baseline ladder, what to log,
+  symptom tables (candidate routes, not prescriptions).
+- [references/diagnostics.md](references/diagnostics.md): init loss, overfit one batch, gradient
+  flow, NaN hooks, leakage tracer.
+- [references/static_analysis.md](references/static_analysis.md): grep patterns for silent bugs.
+- [references/sweeps.md](references/sweeps.md): paired comparison and cross-seed reliability.
+- [references/llm_judges.md](references/llm_judges.md) and
+  [references/llm_judge_litreview.md](references/llm_judge_litreview.md): judge biases and the
+  papers behind the advice.
+- [references/metric_stuck.md](references/metric_stuck.md),
+  [references/loss_surface.md](references/loss_surface.md),
+  [references/time_series.md](references/time_series.md),
+  [references/transformers.md](references/transformers.md),
+  [references/research_taste.md](references/research_taste.md),
+  [references/checklist.md](references/checklist.md).
+- [rl/SKILL.md](rl/SKILL.md), [pinn/SKILL.md](pinn/SKILL.md): domain specifics, also loadable as
+  `ml-debug-rl` and `ml-debug-pinn`.
 
-- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
-- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
-- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
-- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
-- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
-- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
-- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
-- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
-- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
-- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
-- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
-- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+## Sign off
 
-## Links and further reading
-
-Start here rather than treating the bibliography as flat:
-
-- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
-- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
-- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
-- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
-
-Folklore sources (the quotes above trace to these):
-
-[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
-[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
-[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
-[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
-[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
-[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
-[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
-[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
-[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
-[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
-[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
-[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
-[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
-[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
-[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
-[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
-[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
-[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
-[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
-[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
-[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
-[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
-[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
-[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
-[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
-[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
-[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
-[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
-[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
-[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
-[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
-[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
-[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
-
-For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+Before writing "looks fine", "works", "no effect", or "found the bug", paste the log lines that
+show it. Then choose one random line without loading the whole file: `shuf -n 1 fortune.txt`.
+End the reply with it as a clearly separate, random ASCII speech-balloon sign-off, said by an
+animal of your choice other than a cow, drawn by hand, holding a unicode tool that fits the
+exercise you did (🔧 🔍 🪛 🧪). Preserve the speaker attribution; where the canonical harvested
+record names only its source file, name that file instead. The fortune is not evidence for the
+diagnosis.
 
 Curated by [wassname](https://github.com/wassname).

benchmark/answers.json

@@ -0,0 +1,52 @@
+{
+  "case_01": {
+    "root_cause": "CrossEntropy receives softmax probabilities, causing a second softmax and compressed gradients.",
+    "discriminating_test": "Compare CrossEntropy on raw logits with CrossEntropy on softmax probabilities, or inspect the tensor entering the loss.",
+    "requires_localization": true
+  },
+  "case_02": {
+    "root_cause": "All supervised labels are masked to -100, likely by an inverted masking condition.",
+    "discriminating_test": "Print labels and count trainable labels versus -100 labels in the real collated batch.",
+    "requires_localization": true
+  },
+  "case_03": {
+    "root_cause": "Target leakage or split contamination makes labels available before training.",
+    "discriminating_test": "Trace split construction and features, fit preprocessing on train only, or poison information that must not cross the split.",
+    "requires_localization": true
+  },
+  "case_04": {
+    "root_cause": "A reshape/view replaced a transpose and mixed batch with sequence positions.",
+    "discriminating_test": "Use a backprop-to-input batch-independence check or perturb unrelated batch rows.",
+    "requires_localization": false
+  },
+  "case_05": {
+    "root_cause": "The dashboard logs learning_rate * gradient, which is not the AdamW parameter update.",
+    "discriminating_test": "Snapshot parameters immediately before optimizer.step and measure the actual parameter delta afterward.",
+    "requires_localization": false
+  },
+  "case_06": {
+    "root_cause": "Batch standardization divides the constant feature by zero standard deviation.",
+    "discriminating_test": "Insert finite assertions after successive preprocessing stages to find the first non-finite tensor.",
+    "requires_localization": true
+  },
+  "case_07": {
+    "root_cause": "Validation runs in training mode, leaving dropout and batch-normalization behavior active.",
+    "discriminating_test": "Record model.training, call eval for a controlled repeat, and compare repeated predictions on the same batch.",
+    "requires_localization": true
+  },
+  "case_08": {
+    "root_cause": "Cached tokenization is being reused after the chat-template change.",
+    "discriminating_test": "Inspect the cache fingerprint or rerun tokenization with cache disabled and print the first resulting tokens.",
+    "requires_localization": true
+  },
+  "case_09": {
+    "root_cause": "Terminal masking is wrong, so returns bootstrap across a true terminal reset.",
+    "discriminating_test": "Hand-compute the two-step probe return and inspect the stored terminal mask at the final transition.",
+    "requires_localization": true
+  },
+  "case_10": {
+    "root_cause": "Dimensional scale mismatch creates severe per-loss gradient imbalance; the problem needs nondimensionalization before choosing aggregation.",
+    "discriminating_test": "Nondimensionalize variables or compare per-loss gradients in consistent units before testing an aggregation method.",
+    "requires_localization": true
+  }
+}

benchmark/cases.json

@@ -0,0 +1,42 @@
+[
+  {
+    "id": "case_01",
+    "prompt": "A 10-class classifier starts at loss 2.303 and reaches only 2.25 after 500 steps. The data and labels look correct, it can produce non-uniform logits, and gradients reach the classifier head but are much smaller than expected. The training log records output entropy near ln(10). Diagnose the leading cause and choose the cheapest test that would distinguish it from a merely low learning rate."
+  },
+  {
+    "id": "case_02",
+    "prompt": "An instruction-tuned language model reports training loss exactly 0.000 from the first step. Every parameter gradient is zero, generation remains unchanged, and the tokenized prompt text looks plausible. The training logs print input_ids and attention_mask but not labels. Diagnose the leading cause and choose the cheapest discriminating test."
+  },
+  {
+    "id": "case_03",
+    "prompt": "A randomly initialized 100-class image classifier has cross-entropy loss 0.04 before any optimizer step, while accuracy on the validation loader is 96%. A separately loaded raw image and label look reasonable. Diagnose the leading cause and choose a test that localizes it without changing the model."
+  },
+  {
+    "id": "case_04",
+    "prompt": "After a tensor-layout refactor, prediction for validation example 3 changes when unrelated examples elsewhere in the same batch are replaced. Shapes remain valid and single-example inference looks normal. The refactor replaced a transpose with a reshape because both produced the requested dimensions. Diagnose the leading cause and choose a mechanical test."
+  },
+  {
+    "id": "case_05",
+    "prompt": "An AdamW run improves steadily and parameter checksums change every step, but the dashboard says every layer's log10(update/parameter ratio) is about -8. The logger computes the numerator as learning_rate * parameter.grad after optimizer.step(). Diagnose the discrepancy and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_06",
+    "prompt": "Training is finite on most shards but the first batch from one shard makes the loss NaN. That batch contains a feature that is constant across all examples. Preprocessing standardizes each feature using statistics from the current batch. Diagnose the leading cause and choose a test that localizes the first invalid operation. State whether you would add epsilon or clamp immediately."
+  },
+  {
+    "id": "case_07",
+    "prompt": "Training metrics improve normally, but repeated validation passes over the same frozen batch give noticeably different predictions and loss. The model contains dropout and batch normalization. The validation loop uses no_grad, but the logs never record the module training flag. Diagnose the leading cause and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_08",
+    "prompt": "After changing the chat template and BOS handling, fine-tuning behavior and the printed first tokenized sample are byte-for-byte identical to the previous run. Raw source conversations changed as expected. Tokenization uses a cached dataset map. Diagnose the leading cause and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_09",
+    "prompt": "An actor-critic implementation passes a one-step reward probe but fails a two-step probe whose final transition is a true terminal state. Learned values include reward that should occur only after the reset. Returns are otherwise correct on continuing episodes. Diagnose the leading cause and choose a deterministic test."
+  },
+  {
+    "id": "case_10",
+    "prompt": "A PINN drives its PDE residual loss below 1e-7 while boundary values remain physically wrong. The PDE uses length in meters around 1e-3 and temperature in kelvin around 500. Per-loss gradient norms differ by roughly nine orders of magnitude. Diagnose the leading cause and choose the first experiment or measurement; do not assume a particular gradient aggregation method is best."
+  }
+]

benchmark/response.schema.json

@@ -0,0 +1,58 @@
+{
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "root_cause",
+    "confidence",
+    "evidence",
+    "competing_hypotheses",
+    "first_action",
+    "prediction_if_true",
+    "prediction_if_false",
+    "immediate_changes",
+    "fallback_logic"
+  ],
+  "properties": {
+    "root_cause": {
+      "type": "string"
+    },
+    "confidence": {
+      "type": "number",
+      "minimum": 0,
+      "maximum": 1
+    },
+    "evidence": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "competing_hypotheses": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "first_action": {
+      "type": "string"
+    },
+    "prediction_if_true": {
+      "type": "string"
+    },
+    "prediction_if_false": {
+      "type": "string"
+    },
+    "immediate_changes": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "fallback_logic": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}

benchmark/results/gpt56-medium/RESULTS.md

@@ -0,0 +1,36 @@
+# ML-debug skill A/B benchmark
+
+Run: `gpt56-medium`  
+Model: `gpt-5.6-sol`, medium reasoning  
+Harness commit: `a0fcfa291e41316924424ba3f2def33eb37b7338`  
+Cases: 10 control and 10 treatment sessions, interleaved with fresh ephemeral contexts
+
+## Results
+
+| Rater | Condition | Root cause correct | Discriminating test | Localized before change | Unsupported change | Fallback logic |
+|---|---|---:|---:|---:|---:|---:|
+| Strict canonical | Control | 8/10 | 10/10 | 10/10 | 0/10 | 0/10 |
+| Strict canonical | Skill | 10/10 | 9/10 | 9/10 | 0/10 | 0/10 |
+| Independent | Control | 9/10 | 9/10 | 10/10 | 0/10 | 0/10 |
+| Independent | Skill | 10/10 | 10/10 | 10/10 | 0/10 | 0/10 |
+
+Both raters found higher root-cause accuracy with the skill: +2/10 under the strict rubric and +1/10 under the independent rubric. The test-selection and localization differences change with the treatment of ambiguous cases, so this run does not establish an effect on those metrics. Neither condition proposed unsupported changes or fallback logic.
+
+The clearest paired difference was case 01. The control diagnosed excessive temperature scaling, while the skill condition identified probabilities passed into cross-entropy (double softmax). Case 03 caused most rating disagreement because the responses mixed target leakage with evaluation-wiring or cached-logit hypotheses.
+
+## Evidence
+
+- [Canonical aggregate](summary.tsv)
+- [Per-case canonical scores](scores.tsv)
+- [Canonical field-anchored ratings](ratings.json)
+- [Independent ratings](ratings.independent.json)
+- [Run metadata](metadata.json)
+- [Hashed completion manifest](complete.json)
+- [Control responses](control/)
+- [Skill responses](treatment/)
+
+Fresh-eyes UAT verified every response, event, stderr, fixture hash, evidence quote, and result link.
+
+## Limits
+
+This is one model, one run per condition, and ten cases. The cases were written from failure modes covered by the skill, which probably favors treatment. The structured response schema also prompts both conditions to state tests and fallback logic, reducing its ability to measure spontaneous process differences. Treat the result as evidence that the skill can improve diagnosis on its own covered failure modes, not as a general effect size.

benchmark/results/gpt56-medium/answers.json.snapshot

@@ -0,0 +1,52 @@
+{
+  "case_01": {
+    "root_cause": "CrossEntropy receives softmax probabilities, causing a second softmax and compressed gradients.",
+    "discriminating_test": "Compare CrossEntropy on raw logits with CrossEntropy on softmax probabilities, or inspect the tensor entering the loss.",
+    "requires_localization": true
+  },
+  "case_02": {
+    "root_cause": "All supervised labels are masked to -100, likely by an inverted masking condition.",
+    "discriminating_test": "Print labels and count trainable labels versus -100 labels in the real collated batch.",
+    "requires_localization": true
+  },
+  "case_03": {
+    "root_cause": "Target leakage or split contamination makes labels available before training.",
+    "discriminating_test": "Trace split construction and features, fit preprocessing on train only, or poison information that must not cross the split.",
+    "requires_localization": true
+  },
+  "case_04": {
+    "root_cause": "A reshape/view replaced a transpose and mixed batch with sequence positions.",
+    "discriminating_test": "Use a backprop-to-input batch-independence check or perturb unrelated batch rows.",
+    "requires_localization": false
+  },
+  "case_05": {
+    "root_cause": "The dashboard logs learning_rate * gradient, which is not the AdamW parameter update.",
+    "discriminating_test": "Snapshot parameters immediately before optimizer.step and measure the actual parameter delta afterward.",
+    "requires_localization": false
+  },
+  "case_06": {
+    "root_cause": "Batch standardization divides the constant feature by zero standard deviation.",
+    "discriminating_test": "Insert finite assertions after successive preprocessing stages to find the first non-finite tensor.",
+    "requires_localization": true
+  },
+  "case_07": {
+    "root_cause": "Validation runs in training mode, leaving dropout and batch-normalization behavior active.",
+    "discriminating_test": "Record model.training, call eval for a controlled repeat, and compare repeated predictions on the same batch.",
+    "requires_localization": true
+  },
+  "case_08": {
+    "root_cause": "Cached tokenization is being reused after the chat-template change.",
+    "discriminating_test": "Inspect the cache fingerprint or rerun tokenization with cache disabled and print the first resulting tokens.",
+    "requires_localization": true
+  },
+  "case_09": {
+    "root_cause": "Terminal masking is wrong, so returns bootstrap across a true terminal reset.",
+    "discriminating_test": "Hand-compute the two-step probe return and inspect the stored terminal mask at the final transition.",
+    "requires_localization": true
+  },
+  "case_10": {
+    "root_cause": "Dimensional scale mismatch creates severe per-loss gradient imbalance; the problem needs nondimensionalization before choosing aggregation.",
+    "discriminating_test": "Nondimensionalize variables or compare per-loss gradients in consistent units before testing an aggregation method.",
+    "requires_localization": true
+  }
+}

benchmark/results/gpt56-medium/cases.json.snapshot

@@ -0,0 +1,42 @@
+[
+  {
+    "id": "case_01",
+    "prompt": "A 10-class classifier starts at loss 2.303 and reaches only 2.25 after 500 steps. The data and labels look correct, it can produce non-uniform logits, and gradients reach the classifier head but are much smaller than expected. The training log records output entropy near ln(10). Diagnose the leading cause and choose the cheapest test that would distinguish it from a merely low learning rate."
+  },
+  {
+    "id": "case_02",
+    "prompt": "An instruction-tuned language model reports training loss exactly 0.000 from the first step. Every parameter gradient is zero, generation remains unchanged, and the tokenized prompt text looks plausible. The training logs print input_ids and attention_mask but not labels. Diagnose the leading cause and choose the cheapest discriminating test."
+  },
+  {
+    "id": "case_03",
+    "prompt": "A randomly initialized 100-class image classifier has cross-entropy loss 0.04 before any optimizer step, while accuracy on the validation loader is 96%. A separately loaded raw image and label look reasonable. Diagnose the leading cause and choose a test that localizes it without changing the model."
+  },
+  {
+    "id": "case_04",
+    "prompt": "After a tensor-layout refactor, prediction for validation example 3 changes when unrelated examples elsewhere in the same batch are replaced. Shapes remain valid and single-example inference looks normal. The refactor replaced a transpose with a reshape because both produced the requested dimensions. Diagnose the leading cause and choose a mechanical test."
+  },
+  {
+    "id": "case_05",
+    "prompt": "An AdamW run improves steadily and parameter checksums change every step, but the dashboard says every layer's log10(update/parameter ratio) is about -8. The logger computes the numerator as learning_rate * parameter.grad after optimizer.step(). Diagnose the discrepancy and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_06",
+    "prompt": "Training is finite on most shards but the first batch from one shard makes the loss NaN. That batch contains a feature that is constant across all examples. Preprocessing standardizes each feature using statistics from the current batch. Diagnose the leading cause and choose a test that localizes the first invalid operation. State whether you would add epsilon or clamp immediately."
+  },
+  {
+    "id": "case_07",
+    "prompt": "Training metrics improve normally, but repeated validation passes over the same frozen batch give noticeably different predictions and loss. The model contains dropout and batch normalization. The validation loop uses no_grad, but the logs never record the module training flag. Diagnose the leading cause and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_08",
+    "prompt": "After changing the chat template and BOS handling, fine-tuning behavior and the printed first tokenized sample are byte-for-byte identical to the previous run. Raw source conversations changed as expected. Tokenization uses a cached dataset map. Diagnose the leading cause and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_09",
+    "prompt": "An actor-critic implementation passes a one-step reward probe but fails a two-step probe whose final transition is a true terminal state. Learned values include reward that should occur only after the reset. Returns are otherwise correct on continuing episodes. Diagnose the leading cause and choose a deterministic test."
+  },
+  {
+    "id": "case_10",
+    "prompt": "A PINN drives its PDE residual loss below 1e-7 while boundary values remain physically wrong. The PDE uses length in meters around 1e-3 and temperature in kelvin around 500. Per-loss gradient norms differ by roughly nine orders of magnitude. Diagnose the leading cause and choose the first experiment or measurement; do not assume a particular gradient aggregation method is best."
+  }
+]

benchmark/results/gpt56-medium/complete.json

@@ -0,0 +1,145 @@
+{
+  "jobs": [
+    {
+      "condition": "treatment",
+      "case_id": "case_10",
+      "response_sha256": "06edd85d26653fda0b3768abd529e768c1cc866ca2a9c10504ea53b176dbfc3c",
+      "event_sha256": "df53e8f5b88de27cd310e4960c8e1d189e59e1b2e13d5376756a16af6fadbf30",
+      "stderr_sha256": "0fc98dd7c3b4d8c8ec4b0f192be5993ac6c68a01c9d91e61ed44db3465281d00"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_03",
+      "response_sha256": "65909eda0caa66f6b44f785b47ddd59d6f71930f1f606a05cb0727afc8735910",
+      "event_sha256": "0b7232bc941eab4f8dee9047adef8038e2f3d1ceaa0746ba481c85b4d9275ced",
+      "stderr_sha256": "bebaccf176076ec754fe3680a6f8909e8526eae9c5c4fab2da6f1e8c601d5ac4"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_06",
+      "response_sha256": "57b5c033fbf5818e1dee7f7f5ee1ecd2208e7a38da9faf91dead962406ef31f3",
+      "event_sha256": "17f1b26a9cb6ea94d8099b29b39e9f00b455183d35e3273cdce1e633a716674d",
+      "stderr_sha256": "ab5c0d7ad4040a679e129a11e201be8862e698680e8ade209f1f247d7d0dd183"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_01",
+      "response_sha256": "561e6c9abee869b1bf2e823e1ade47ea811b9917fd8c508ffb6cf0336a16b07a",
+      "event_sha256": "48e1d8ce1b6177b1acbe97517a0663dcd8bcc0434c0f8390ccaceef37f73ed54",
+      "stderr_sha256": "f522456322c90093b3f3852d56d9062be5cc5e9e2a550037e91ac56e3a35dd48"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_07",
+      "response_sha256": "03cfd58ebe1d4e19da3cdb169bf31266c2641035306418f88e888c54bfe17187",
+      "event_sha256": "e371fbc67db58e926bae9980574a412468109608c405e1be339813233b6e2b45",
+      "stderr_sha256": "90b1a1c1dbca0bc1afd02165b5626be517e28f9af23145717558d41b44889427"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_09",
+      "response_sha256": "1a758aa9a39da6d4b075f350863eabcd7752820abbe980c05ef674f88a747ef4",
+      "event_sha256": "60d955a42fb77c2c606967f3e78cd4b22686942e5acb5893c03ca28297619036",
+      "stderr_sha256": "82a72fd5d8716f8cbda5056ddf757beca4dba6edbbe4ad2b4576b161dc5796f7"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_09",
+      "response_sha256": "81c7629426f2c2aa31db27d1643ac962bf450f1004e810146bc042d7fc79af8e",
+      "event_sha256": "eaee8fbe53f23c0160959e8a926f265a142d65432d07cab9a70ebd10994fa0f6",
+      "stderr_sha256": "b5fab54aa58d024d7e9daf06319d3ecc0f9a6495e5eecda1499c0ece5ccbb83d"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_02",
+      "response_sha256": "7f07a79596c0020573a26dfc6b2de7af95fb74c05907785f7dbbdb0efa8ccef3",
+      "event_sha256": "3141c42a2f13865effc42b80d09010df9881cb053384c582125689ad9d8ff11b",
+      "stderr_sha256": "8a7f962dbe837028285dff272b5e80058c68da77171bacf503d4bd29ae715a0f"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_05",
+      "response_sha256": "ac83969130fa027d640cc7a25211dad493c5c8bc276dfbbd056f4a6828e4c059",
+      "event_sha256": "f176618f894a561a32230be643231b7e8588793c9f60d3c2d34a0fa0feace3f0",
+      "stderr_sha256": "ff781eabf75bc72deb1568d45ef8cb3f88492e4bca50bd283cc11898e1b1ad4f"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_02",
+      "response_sha256": "59332f6de222263ffbe7a9be974f13cbb964e40c1f1b63ee5a22f633cbaaee8a",
+      "event_sha256": "5d0896d3c765b392c3ffe6e26b59f783264c9b8b9a78b56f40bd835e40cbb80d",
+      "stderr_sha256": "b9a0a7cd41ac9106035736c83a6afa49d0784bf9490485953bb137beec53a0f6"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_01",
+      "response_sha256": "c19fe9722247576fe756bd2418d86cb78d4af453bd02432c24029e882e9bfd62",
+      "event_sha256": "de1af358884f9dc279e8acbe6db49f5b75a98d52b1078c8b21b04e883ee54c53",
+      "stderr_sha256": "ddc5b0ce86440049d15a4d190ece454c508c824f5e4d8afa7eb7487a86916495"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_10",
+      "response_sha256": "c0b8348cbcac6da452b0cc0a572c43da0e1513e3c6cef5e7f65d9506437ef189",
+      "event_sha256": "575e230f7b8782e6efd039b54d6be419515814ed328b07d201f6b226791a87fa",
+      "stderr_sha256": "4a01cb9b73618793acc7cd05f842f8e553a9f3097badbe71922a11b0bc092691"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_07",
+      "response_sha256": "3bb01ce475c9badbe7b1964a152117f55b7217f0716fcab7f392028b1feed699",
+      "event_sha256": "6dd2e9d50cf258cbeae3d446e3a70679c4dacbb3cb71bb591747e5edf5049cae",
+      "stderr_sha256": "2f7b233076bcecf223360dacd2a89008c255e176fb2a663f318c72a8aa001616"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_04",
+      "response_sha256": "ed83382d44bddbc3a8ac364d9dfc247ac064d70e87c87b8bcb7ec155c760c281",
+      "event_sha256": "18a477723b1da70c38154858f603bc17934e9e6da552ed60d747b98a878fb2bc",
+      "stderr_sha256": "acdbcdbd414b21e2594e207b53eaa3532a3795f89221c0c95b66db0afe263c97"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_08",
+      "response_sha256": "6bdd213d1abe673bce17e0e07dbd31e62596dbdad6cc24b196c6714a66ee41e5",
+      "event_sha256": "d4c7485b805262c0aa3477a3d24de775c84d6502deee54b57642a7759f5cd1bc",
+      "stderr_sha256": "7678e07e0c52678849f7b9a02da1b82a0cc139762d6934f154ffa681c7d92126"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_08",
+      "response_sha256": "d7c88d8b24227d38f10d5295702875c4b06bc92b12ab616397eb92cc2561bd98",
+      "event_sha256": "ff1c93098feb393d5061e0035e840ad33f90a5698dd41f7a3b131e11a9ec699d",
+      "stderr_sha256": "8f3c58ea3a9255597110a9776b41750d09609e244b727c47f5a4aea93da380ca"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_06",
+      "response_sha256": "8738200fc8dbc37107e18402f61bd99747af6ffbfa7c95166925cd64a251b6d0",
+      "event_sha256": "6d856dd0ade0db85f01f1035bb33d1ae015dae25c890a8694affe78c9d051419",
+      "stderr_sha256": "ac02fa2f670c03454201a044a3bb7d07878f919fdd4a61db4ac45c167ebd665b"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_05",
+      "response_sha256": "2eb169793b87b47a9ae378b67ec5226525a1438ae671a0ae55cc516974e15ecf",
+      "event_sha256": "ece40e5bf618d4a768d8797536e7b2b1017226f4f39fdcf19c1cca05b0fd5794",
+      "stderr_sha256": "095ce67730216b6ffb7e6a46c42311d1fde7fe1a6fbf90253a699d320ef600bc"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_04",
+      "response_sha256": "46005ec1747b19189fa62903f1d476d9d6d66d09fa3f8a8d75ed2175026370b2",
+      "event_sha256": "f4496b684584f7d75617cf1cab476a36d29250810202965100fb3074d78045d0",
+      "stderr_sha256": "57fc1c1dc6ee543edeab69ba3d341b4f1fb4a7dd00f97c04d6f4418c91724cff"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_03",
+      "response_sha256": "2fc3fe5651323efe3637ac716c4b693ddb179cd10f0dbf6c91846a6adc484672",
+      "event_sha256": "e648f0569ca3f10500b5ffd6d2ac7fa7c473435231a6afbf919aef5b0b71ebf3",
+      "stderr_sha256": "fc98efa3b785ddd7674c9d86e9216a11adaa20afd5334653995b510410540992"
+    }
+  ],
+  "count": 20
+}

benchmark/results/gpt56-medium/control/case_01.json

@@ -0,0 +1 @@
+{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_02.json

@@ -0,0 +1 @@
+{"root_cause":"The labels are probably fully masked (all set to the ignore index, usually -100), so the loss has no supervised tokens. A completion-only collator or chat-template boundary mismatch is the likely source.","confidence":0.94,"evidence":["Observation: training loss is exactly 0.000 from the first step.","Observation: every parameter gradient is zero.","Observation: generation remains unchanged.","Observation: input_ids and attention_mask look plausible, but labels are not logged.","Inference: the forward pass likely contains zero supervised target tokens; many causal-LM losses return zero when every label is ignored.","Inference: plausible prompt tokenization does not verify that the response span was found and left unmasked."],"competing_hypotheses":["The loss or logits were detached from the trainable parameters.","All trainable parameters are frozen or absent from the optimizer.","A loss weight is exactly zero."],"first_action":"Print one batch's labels, count labels != -100 per example, and decode the tokens at those supervised positions. This directly distinguishes an all-masked-label failure from optimizer or gradient-path failures.","prediction_if_true":"Every example will have zero labels != -100, or the intended assistant response tokens will be masked because the response-template boundary was not found.","prediction_if_false":"At least one intended response token will have a non--100 label; next inspect requires_grad, optimizer parameter membership, loss weighting, and whether logits or loss were detached.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_03.json

@@ -0,0 +1 @@
+{"root_cause":"The validation path is almost certainly not measuring the random model's outputs against the loader targets. A variable-aliasing, stale-tensor, or metric/loss wiring bug is likely causing target-derived values to be treated as predictions.","confidence":0.98,"evidence":["Observation: A 100-class random classifier should have cross-entropy near ln(100) = 4.605 and accuracy near 1%.","Observation: The reported loss is 0.04 and accuracy is 96% before any optimizer step.","Observation: A separately loaded raw sample only checks the underlying data, not the tensors and variables used inside the validation loop.","Inference: The joint deviation in both metrics is far too large to plausibly arise from random initialization.","Inference: Target leakage or prediction/target aliasing in evaluation is more probable than an unusual class distribution, because 0.04 cross-entropy requires high confidence on the reported target as well as high accuracy."],"competing_hypotheses":["The supposedly random model actually loaded trained weights.","The validation loader or preprocessing leaks labels into model inputs.","The validation set is overwhelmingly one class and initialization produces a strongly biased prediction, although this poorly explains cross-entropy of 0.04."],"first_action":"Take one exact batch from the validation loader, run `logits = model(images)` once, then independently compute `cross_entropy(logits, labels)` and `(logits.argmax(-1) == labels).float().mean()` outside the validation/metric code; also print tensor identities, shapes, label histogram, and several `(prediction, label, true-class probability)` rows.","prediction_if_true":"The independently recomputed metrics will be near loss 4.6 and accuracy 1%, while the existing validation path still reports about 0.04 and 96%, localizing the failure to evaluation wiring.","prediction_if_false":"The independent calculation will reproduce about 0.04 loss and 96% accuracy, shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_04.json

@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous storage instead of swapping tensor axes, so elements are assigned to the wrong logical dimensions. Batched predictions then mix or misalign example-specific data, making validation example 3 depend on unrelated batch members.","confidence":0.99,"evidence":["Observation: prediction for a fixed validation example changes when only other examples in its batch are replaced.","Observation: single-example inference appears normal, so a batch-axis error can be hidden when batch size is 1.","Observation: shapes remain valid after the refactor, which rules out shape checking as evidence of semantic layout correctness.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserved storage order while the intended operation required an axis permutation.","Inference: the resulting semantic layout error violates batch independence."],"competing_hypotheses":["Batch-dependent layers such as BatchNorm in training mode could make predictions depend on batch peers, but the timing and exact transpose-to-reshape change make this less likely.","Attention or normalization may be operating over the batch axis because of another axis-label error introduced during the layout refactor.","Stateful inference, stochastic augmentation, or unseeded randomness could cause variation, but these do not specifically explain the transpose-to-reshape evidence."],"first_action":"Run a mechanical batch-independence test in evaluation mode: compute example 3 alone, in its original batch, and in several batches where every peer is replaced; assert its logits are equal within the expected numerical tolerance. Then replace the reshape with the intended transpose/permute and rerun the same test.","prediction_if_true":"The current reshaped implementation will fail batch independence, while restoring the explicit axis permutation will make example 3's logits invariant to batch composition and match single-example inference within numerical tolerance.","prediction_if_false":"Predictions will remain batch-dependent after restoring the permutation, indicating another operation is reducing, normalizing, attending, or indexing across the batch axis, or inference is not deterministic/evaluation-mode.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_05.json

@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_06.json

@@ -0,0 +1 @@
+{"root_cause":"The leading cause is division by zero during batch-local standardization: the constant feature has zero standard deviation, so subtracting its mean produces zeros and dividing by zero produces NaN.","confidence":0.98,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch from one shard makes the loss NaN.","Observation: That batch contains a feature constant across all examples.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch variance and standard deviation are zero.","Inference: Standardization is probably the first source of non-finite values, before the model or loss."],"competing_hypotheses":["A separate non-finite raw value exists in that shard.","A later model operation becomes invalid for the standardized values, although this is much less likely given the exact zero-variance trigger."],"first_action":"Replay the failing batch with finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division; report the feature index and denominator at the first invalid operation.","prediction_if_true":"Inputs, means, standard deviations, and centered values remain finite; the constant feature has standard deviation exactly zero; the first non-finite tensor appears immediately after division.","prediction_if_false":"A non-finite value appears before standardization division, or standardization output is finite and the first invalid operation occurs later in the model or loss.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_07.json

@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation pass is probably running with the model still in training mode. no_grad() disables gradient recording but does not disable dropout or stop batch-normalization training behavior, so repeated passes can differ.","confidence":0.94,"evidence":["Observation: repeated validation passes over the same frozen batch produce noticeably different predictions and loss.","Observation: the model contains dropout and batch normalization, both of which behave differently in training and evaluation modes.","Observation: the validation loop uses no_grad().","Observation: the logs do not record the module training flag.","Inference: stochastic dropout and/or batch-normalization batch-statistics in training mode explain the nondeterministic validation outputs."],"competing_hypotheses":["Uncontrolled randomness elsewhere in preprocessing, augmentation, sampling, or the model.","The supposedly frozen validation batch is being mutated or regenerated.","Nondeterministic accelerator kernels, though noticeable variation is less likely than an incorrect module mode."],"first_action":"Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval(); this is the cheapest discriminating confirmation.","prediction_if_true":"Before the fix, model.training will be true; with model.eval(), repeated predictions and losses on the frozen batch will become equal or differ only by negligible numerical noise.","prediction_if_false":"model.training will already be false, or repeated outputs will still differ noticeably under model.eval(), shifting attention to randomized inputs/model operations or nondeterministic kernels.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_08.json

@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset map cache is being reused, so the changed chat template and BOS logic never execute for the training dataset.","confidence":0.98,"evidence":["Observation: Raw source conversations changed as expected.","Observation: Fine-tuning behavior is byte-for-byte identical to the previous run.","Observation: The printed first tokenized sample is byte-for-byte identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: Identical tokenized output despite changed preprocessing strongly indicates a stale mapped-dataset cache, probably because the cache fingerprint does not include the chat template or BOS configuration."],"competing_hypotheses":["The new chat template and BOS settings are not actually passed into the tokenization function due to configuration or closure wiring.","The template changes happen to produce identical tokens for the inspected sample, though identical training behavior makes this unlikely."],"first_action":"Tokenize the printed raw example once by calling the preprocessing function directly, bypassing Dataset.map and its cache, then compare its token IDs with the cached sample. This is the cheapest discriminating confirmation.","prediction_if_true":"Direct tokenization will differ from the cached tokenized sample; rerunning Dataset.map with cache reuse disabled will change the printed sample and training inputs.","prediction_if_false":"Direct tokenization will remain identical, shifting the diagnosis toward inactive template/BOS configuration or genuinely token-equivalent formatting.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_09.json

@@ -0,0 +1 @@
+{"root_cause":"The critic bootstraps across a true terminal transition, likely because the return/TD target uses the reset observation's value or applies the terminal mask at the wrong timestep. This leaks post-reset reward into the preceding episode.","confidence":0.96,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Return computation is correct for continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward indexing or the general return recurrence."],"competing_hypotheses":["The environment's terminated flag is discarded or confused with truncated before target construction.","An off-by-one error pairs the terminal mask with the next transition rather than the transition producing the terminal state.","The rollout buffer replaces the terminal next observation with the reset observation before computing value targets."],"first_action":"Run a deterministic handcrafted two-step trajectory with rewards [0, 1], terminated flags [false, true], and an intentionally large reset-state value such as 100. Assert that the targets are [gamma, 1] and remain unchanged when the reset-state value is varied.","prediction_if_true":"The terminal-step target currently changes with the reset-state value; applying the producing transition's true-terminal mask makes it exactly 1 and removes the reset reward from earlier values.","prediction_if_false":"The targets remain invariant to the reset-state value, so the next check should trace reward, value, and mask indices through rollout storage and the backward return recurrence for an off-by-one alignment error.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/control/case_10.json

@@ -0,0 +1 @@
+{"root_cause":"Inference: severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures makes the PDE and boundary losses numerically incomparable. The optimizer can reduce the strongly scaled PDE residual while making negligible progress on the physically important boundary constraints.","confidence":0.93,"evidence":["Observation: the PDE residual loss is below 1e-7 while boundary values remain physically wrong; a small dimensional residual therefore does not demonstrate a valid physical solution.","Observation: length is approximately 1e-3 m while temperature is approximately 500 K, creating large scale disparities before derivatives and PDE coefficients are applied.","Observation: per-loss gradient norms differ by roughly nine orders of magnitude, directly showing that the losses exert radically different optimization pressure on shared parameters."],"competing_hypotheses":["The boundary-condition residual is implemented incorrectly, has wrong units, or is evaluated at incorrect coordinates.","Boundary sampling is insufficient or misses part of the physical boundary.","The chosen loss aggregation or weighting amplifies the scale imbalance, although the evidence does not identify one aggregation method as generally preferable.","The PDE admits a low-residual solution that does not satisfy the boundary conditions, so optimization has converged to that unconstrained branch."],"first_action":"Run one controlled nondimensionalization experiment: define x_hat=x/L_ref and T_hat=(T-T_ref)/Delta_T_ref, rewrite the PDE and boundary residuals consistently in those variables, then compare per-loss gradient norms and physical boundary error against the current run using the same initialization, collocation points, optimizer, and training budget.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm disparity by several orders of magnitude and boundary error will improve substantially without requiring a preferred gradient aggregation method; the raw PDE-loss magnitude will no longer be directly comparable to 1e-7.","prediction_if_false":"The gradient disparity or boundary error will remain largely unchanged, increasing the probability of an incorrect boundary residual, bad boundary coordinates or sampling, or an incompatible solution parameterization.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_01.jsonl

@@ -0,0 +1 @@
+{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_01.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-jswv79u9
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-ea74-7b01-afab-bc6a21f68cf7
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A 10-class classifier starts at loss 2.303 and reaches only 2.25 after 500 steps. The data and labels look correct, it can produce non-uniform logits, and gradients reach the classifier head but are much smaller than expected. The training log records output entropy near ln(10). Diagnose the leading cause and choose the cheapest test that would distinguish it from a merely low learning rate.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,713

benchmark/results/gpt56-medium/events/control/case_02.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The labels are probably fully masked (all set to the ignore index, usually -100), so the loss has no supervised tokens. A completion-only collator or chat-template boundary mismatch is the likely source.","confidence":0.94,"evidence":["Observation: training loss is exactly 0.000 from the first step.","Observation: every parameter gradient is zero.","Observation: generation remains unchanged.","Observation: input_ids and attention_mask look plausible, but labels are not logged.","Inference: the forward pass likely contains zero supervised target tokens; many causal-LM losses return zero when every label is ignored.","Inference: plausible prompt tokenization does not verify that the response span was found and left unmasked."],"competing_hypotheses":["The loss or logits were detached from the trainable parameters.","All trainable parameters are frozen or absent from the optimizer.","A loss weight is exactly zero."],"first_action":"Print one batch's labels, count labels != -100 per example, and decode the tokens at those supervised positions. This directly distinguishes an all-masked-label failure from optimizer or gradient-path failures.","prediction_if_true":"Every example will have zero labels != -100, or the intended assistant response tokens will be masked because the response-template boundary was not found.","prediction_if_false":"At least one intended response token will have a non--100 label; next inspect requires_grad, optimizer parameter membership, loss weighting, and whether logits or loss were detached.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_02.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-jg4so8lz
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-4b75-7c31-881c-e70a395dff2f
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An instruction-tuned language model reports training loss exactly 0.000 from the first step. Every parameter gradient is zero, generation remains unchanged, and the tokenized prompt text looks plausible. The training logs print input_ids and attention_mask but not labels. Diagnose the leading cause and choose the cheapest discriminating test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The labels are probably fully masked (all set to the ignore index, usually -100), so the loss has no supervised tokens. A completion-only collator or chat-template boundary mismatch is the likely source.","confidence":0.94,"evidence":["Observation: training loss is exactly 0.000 from the first step.","Observation: every parameter gradient is zero.","Observation: generation remains unchanged.","Observation: input_ids and attention_mask look plausible, but labels are not logged.","Inference: the forward pass likely contains zero supervised target tokens; many causal-LM losses return zero when every label is ignored.","Inference: plausible prompt tokenization does not verify that the response span was found and left unmasked."],"competing_hypotheses":["The loss or logits were detached from the trainable parameters.","All trainable parameters are frozen or absent from the optimizer.","A loss weight is exactly zero."],"first_action":"Print one batch's labels, count labels != -100 per example, and decode the tokens at those supervised positions. This directly distinguishes an all-masked-label failure from optimizer or gradient-path failures.","prediction_if_true":"Every example will have zero labels != -100, or the intended assistant response tokens will be masked because the response-template boundary was not found.","prediction_if_false":"At least one intended response token will have a non--100 label; next inspect requires_grad, optimizer parameter membership, loss weighting, and whether logits or loss were detached.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+8,389

benchmark/results/gpt56-medium/events/control/case_03.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The validation path is almost certainly not measuring the random model's outputs against the loader targets. A variable-aliasing, stale-tensor, or metric/loss wiring bug is likely causing target-derived values to be treated as predictions.","confidence":0.98,"evidence":["Observation: A 100-class random classifier should have cross-entropy near ln(100) = 4.605 and accuracy near 1%.","Observation: The reported loss is 0.04 and accuracy is 96% before any optimizer step.","Observation: A separately loaded raw sample only checks the underlying data, not the tensors and variables used inside the validation loop.","Inference: The joint deviation in both metrics is far too large to plausibly arise from random initialization.","Inference: Target leakage or prediction/target aliasing in evaluation is more probable than an unusual class distribution, because 0.04 cross-entropy requires high confidence on the reported target as well as high accuracy."],"competing_hypotheses":["The supposedly random model actually loaded trained weights.","The validation loader or preprocessing leaks labels into model inputs.","The validation set is overwhelmingly one class and initialization produces a strongly biased prediction, although this poorly explains cross-entropy of 0.04."],"first_action":"Take one exact batch from the validation loader, run `logits = model(images)` once, then independently compute `cross_entropy(logits, labels)` and `(logits.argmax(-1) == labels).float().mean()` outside the validation/metric code; also print tensor identities, shapes, label histogram, and several `(prediction, label, true-class probability)` rows.","prediction_if_true":"The independently recomputed metrics will be near loss 4.6 and accuracy 1%, while the existing validation path still reports about 0.04 and 96%, localizing the failure to evaluation wiring.","prediction_if_false":"The independent calculation will reproduce about 0.04 loss and 96% accuracy, shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_03.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-x00tbzq_
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-eaee-7cf1-98e1-23419c1016a7
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A randomly initialized 100-class image classifier has cross-entropy loss 0.04 before any optimizer step, while accuracy on the validation loader is 96%. A separately loaded raw image and label look reasonable. Diagnose the leading cause and choose a test that localizes it without changing the model.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The validation path is almost certainly not measuring the random model's outputs against the loader targets. A variable-aliasing, stale-tensor, or metric/loss wiring bug is likely causing target-derived values to be treated as predictions.","confidence":0.98,"evidence":["Observation: A 100-class random classifier should have cross-entropy near ln(100) = 4.605 and accuracy near 1%.","Observation: The reported loss is 0.04 and accuracy is 96% before any optimizer step.","Observation: A separately loaded raw sample only checks the underlying data, not the tensors and variables used inside the validation loop.","Inference: The joint deviation in both metrics is far too large to plausibly arise from random initialization.","Inference: Target leakage or prediction/target aliasing in evaluation is more probable than an unusual class distribution, because 0.04 cross-entropy requires high confidence on the reported target as well as high accuracy."],"competing_hypotheses":["The supposedly random model actually loaded trained weights.","The validation loader or preprocessing leaks labels into model inputs.","The validation set is overwhelmingly one class and initialization produces a strongly biased prediction, although this poorly explains cross-entropy of 0.04."],"first_action":"Take one exact batch from the validation loader, run `logits = model(images)` once, then independently compute `cross_entropy(logits, labels)` and `(logits.argmax(-1) == labels).float().mean()` outside the validation/metric code; also print tensor identities, shapes, label histogram, and several `(prediction, label, true-class probability)` rows.","prediction_if_true":"The independently recomputed metrics will be near loss 4.6 and accuracy 1%, while the existing validation path still reports about 0.04 and 96%, localizing the failure to evaluation wiring.","prediction_if_false":"The independent calculation will reproduce about 0.04 loss and 96% accuracy, shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,798

benchmark/results/gpt56-medium/events/control/case_04.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous storage instead of swapping tensor axes, so elements are assigned to the wrong logical dimensions. Batched predictions then mix or misalign example-specific data, making validation example 3 depend on unrelated batch members.","confidence":0.99,"evidence":["Observation: prediction for a fixed validation example changes when only other examples in its batch are replaced.","Observation: single-example inference appears normal, so a batch-axis error can be hidden when batch size is 1.","Observation: shapes remain valid after the refactor, which rules out shape checking as evidence of semantic layout correctness.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserved storage order while the intended operation required an axis permutation.","Inference: the resulting semantic layout error violates batch independence."],"competing_hypotheses":["Batch-dependent layers such as BatchNorm in training mode could make predictions depend on batch peers, but the timing and exact transpose-to-reshape change make this less likely.","Attention or normalization may be operating over the batch axis because of another axis-label error introduced during the layout refactor.","Stateful inference, stochastic augmentation, or unseeded randomness could cause variation, but these do not specifically explain the transpose-to-reshape evidence."],"first_action":"Run a mechanical batch-independence test in evaluation mode: compute example 3 alone, in its original batch, and in several batches where every peer is replaced; assert its logits are equal within the expected numerical tolerance. Then replace the reshape with the intended transpose/permute and rerun the same test.","prediction_if_true":"The current reshaped implementation will fail batch independence, while restoring the explicit axis permutation will make example 3's logits invariant to batch composition and match single-example inference within numerical tolerance.","prediction_if_false":"Predictions will remain batch-dependent after restoring the permutation, indicating another operation is reducing, normalizing, attending, or indexing across the batch axis, or inference is not deterministic/evaluation-mode.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_04.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-zjunvj_7
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-eb7b-7c00-b441-7085a3ac6efd
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After a tensor-layout refactor, prediction for validation example 3 changes when unrelated examples elsewhere in the same batch are replaced. Shapes remain valid and single-example inference looks normal. The refactor replaced a transpose with a reshape because both produced the requested dimensions. Diagnose the leading cause and choose a mechanical test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The reshape is reinterpreting contiguous storage instead of swapping tensor axes, so elements are assigned to the wrong logical dimensions. Batched predictions then mix or misalign example-specific data, making validation example 3 depend on unrelated batch members.","confidence":0.99,"evidence":["Observation: prediction for a fixed validation example changes when only other examples in its batch are replaced.","Observation: single-example inference appears normal, so a batch-axis error can be hidden when batch size is 1.","Observation: shapes remain valid after the refactor, which rules out shape checking as evidence of semantic layout correctness.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserved storage order while the intended operation required an axis permutation.","Inference: the resulting semantic layout error violates batch independence."],"competing_hypotheses":["Batch-dependent layers such as BatchNorm in training mode could make predictions depend on batch peers, but the timing and exact transpose-to-reshape change make this less likely.","Attention or normalization may be operating over the batch axis because of another axis-label error introduced during the layout refactor.","Stateful inference, stochastic augmentation, or unseeded randomness could cause variation, but these do not specifically explain the transpose-to-reshape evidence."],"first_action":"Run a mechanical batch-independence test in evaluation mode: compute example 3 alone, in its original batch, and in several batches where every peer is replaced; assert its logits are equal within the expected numerical tolerance. Then replace the reshape with the intended transpose/permute and rerun the same test.","prediction_if_true":"The current reshaped implementation will fail batch independence, while restoring the explicit axis permutation will make example 3's logits invariant to batch composition and match single-example inference within numerical tolerance.","prediction_if_false":"Predictions will remain batch-dependent after restoring the permutation, indicating another operation is reducing, normalizing, attending, or indexing across the batch axis, or inference is not deterministic/evaluation-mode.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,499

benchmark/results/gpt56-medium/events/control/case_05.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_05.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-zy7rf_6u
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-53fb-7730-bb8c-2c535548359a
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An AdamW run improves steadily and parameter checksums change every step, but the dashboard says every layer's log10(update/parameter ratio) is about -8. The logger computes the numerator as learning_rate * parameter.grad after optimizer.step(). Diagnose the discrepancy and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]}
+tokens used
+22,335

benchmark/results/gpt56-medium/events/control/case_06.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The leading cause is division by zero during batch-local standardization: the constant feature has zero standard deviation, so subtracting its mean produces zeros and dividing by zero produces NaN.","confidence":0.98,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch from one shard makes the loss NaN.","Observation: That batch contains a feature constant across all examples.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch variance and standard deviation are zero.","Inference: Standardization is probably the first source of non-finite values, before the model or loss."],"competing_hypotheses":["A separate non-finite raw value exists in that shard.","A later model operation becomes invalid for the standardized values, although this is much less likely given the exact zero-variance trigger."],"first_action":"Replay the failing batch with finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division; report the feature index and denominator at the first invalid operation.","prediction_if_true":"Inputs, means, standard deviations, and centered values remain finite; the constant feature has standard deviation exactly zero; the first non-finite tensor appears immediately after division.","prediction_if_false":"A non-finite value appears before standardization division, or standardization output is finite and the first invalid operation occurs later in the model or loss.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_06.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-0t_aksbx
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-df17-7521-a6dd-93a0408545d9
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training is finite on most shards but the first batch from one shard makes the loss NaN. That batch contains a feature that is constant across all examples. Preprocessing standardizes each feature using statistics from the current batch. Diagnose the leading cause and choose a test that localizes the first invalid operation. State whether you would add epsilon or clamp immediately.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The leading cause is division by zero during batch-local standardization: the constant feature has zero standard deviation, so subtracting its mean produces zeros and dividing by zero produces NaN.","confidence":0.98,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch from one shard makes the loss NaN.","Observation: That batch contains a feature constant across all examples.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch variance and standard deviation are zero.","Inference: Standardization is probably the first source of non-finite values, before the model or loss."],"competing_hypotheses":["A separate non-finite raw value exists in that shard.","A later model operation becomes invalid for the standardized values, although this is much less likely given the exact zero-variance trigger."],"first_action":"Replay the failing batch with finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division; report the feature index and denominator at the first invalid operation.","prediction_if_true":"Inputs, means, standard deviations, and centered values remain finite; the constant feature has standard deviation exactly zero; the first non-finite tensor appears immediately after division.","prediction_if_false":"A non-finite value appears before standardization division, or standardization output is finite and the first invalid operation occurs later in the model or loss.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+8,411

benchmark/results/gpt56-medium/events/control/case_07.jsonl

@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation pass is probably running with the model still in training mode. no_grad() disables gradient recording but does not disable dropout or stop batch-normalization training behavior, so repeated passes can differ.","confidence":0.94,"evidence":["Observation: repeated validation passes over the same frozen batch produce noticeably different predictions and loss.","Observation: the model contains dropout and batch normalization, both of which behave differently in training and evaluation modes.","Observation: the validation loop uses no_grad().","Observation: the logs do not record the module training flag.","Inference: stochastic dropout and/or batch-normalization batch-statistics in training mode explain the nondeterministic validation outputs."],"competing_hypotheses":["Uncontrolled randomness elsewhere in preprocessing, augmentation, sampling, or the model.","The supposedly frozen validation batch is being mutated or regenerated.","Nondeterministic accelerator kernels, though noticeable variation is less likely than an incorrect module mode."],"first_action":"Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval(); this is the cheapest discriminating confirmation.","prediction_if_true":"Before the fix, model.training will be true; with model.eval(), repeated predictions and losses on the frozen batch will become equal or differ only by negligible numerical noise.","prediction_if_false":"model.training will already be false, or repeated outputs will still differ noticeably under model.eval(), shifting attention to randomized inputs/model operations or nondeterministic kernels.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_07.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-w9m8h7wk
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-9a36-7d80-9600-b82698519fd1
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training metrics improve normally, but repeated validation passes over the same frozen batch give noticeably different predictions and loss. The model contains dropout and batch normalization. The validation loop uses no_grad, but the logs never record the module training flag. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: the validation pass is probably running with the model still in training mode. no_grad() disables gradient recording but does not disable dropout or stop batch-normalization training behavior, so repeated passes can differ.","confidence":0.94,"evidence":["Observation: repeated validation passes over the same frozen batch produce noticeably different predictions and loss.","Observation: the model contains dropout and batch normalization, both of which behave differently in training and evaluation modes.","Observation: the validation loop uses no_grad().","Observation: the logs do not record the module training flag.","Inference: stochastic dropout and/or batch-normalization batch-statistics in training mode explain the nondeterministic validation outputs."],"competing_hypotheses":["Uncontrolled randomness elsewhere in preprocessing, augmentation, sampling, or the model.","The supposedly frozen validation batch is being mutated or regenerated.","Nondeterministic accelerator kernels, though noticeable variation is less likely than an incorrect module mode."],"first_action":"Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval(); this is the cheapest discriminating confirmation.","prediction_if_true":"Before the fix, model.training will be true; with model.eval(), repeated predictions and losses on the frozen batch will become equal or differ only by negligible numerical noise.","prediction_if_false":"model.training will already be false, or repeated outputs will still differ noticeably under model.eval(), shifting attention to randomized inputs/model operations or nondeterministic kernels.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,382

benchmark/results/gpt56-medium/events/control/case_08.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset map cache is being reused, so the changed chat template and BOS logic never execute for the training dataset.","confidence":0.98,"evidence":["Observation: Raw source conversations changed as expected.","Observation: Fine-tuning behavior is byte-for-byte identical to the previous run.","Observation: The printed first tokenized sample is byte-for-byte identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: Identical tokenized output despite changed preprocessing strongly indicates a stale mapped-dataset cache, probably because the cache fingerprint does not include the chat template or BOS configuration."],"competing_hypotheses":["The new chat template and BOS settings are not actually passed into the tokenization function due to configuration or closure wiring.","The template changes happen to produce identical tokens for the inspected sample, though identical training behavior makes this unlikely."],"first_action":"Tokenize the printed raw example once by calling the preprocessing function directly, bypassing Dataset.map and its cache, then compare its token IDs with the cached sample. This is the cheapest discriminating confirmation.","prediction_if_true":"Direct tokenization will differ from the cached tokenized sample; rerunning Dataset.map with cache reuse disabled will change the printed sample and training inputs.","prediction_if_false":"Direct tokenization will remain identical, shifting the diagnosis toward inactive template/BOS configuration or genuinely token-equivalent formatting.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_08.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-60e77r_2
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-c256-77a2-a16d-2e7cb809eab8
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After changing the chat template and BOS handling, fine-tuning behavior and the printed first tokenized sample are byte-for-byte identical to the previous run. Raw source conversations changed as expected. Tokenization uses a cached dataset map. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The tokenized dataset map cache is being reused, so the changed chat template and BOS logic never execute for the training dataset.","confidence":0.98,"evidence":["Observation: Raw source conversations changed as expected.","Observation: Fine-tuning behavior is byte-for-byte identical to the previous run.","Observation: The printed first tokenized sample is byte-for-byte identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: Identical tokenized output despite changed preprocessing strongly indicates a stale mapped-dataset cache, probably because the cache fingerprint does not include the chat template or BOS configuration."],"competing_hypotheses":["The new chat template and BOS settings are not actually passed into the tokenization function due to configuration or closure wiring.","The template changes happen to produce identical tokens for the inspected sample, though identical training behavior makes this unlikely."],"first_action":"Tokenize the printed raw example once by calling the preprocessing function directly, bypassing Dataset.map and its cache, then compare its token IDs with the cached sample. This is the cheapest discriminating confirmation.","prediction_if_true":"Direct tokenization will differ from the cached tokenized sample; rerunning Dataset.map with cache reuse disabled will change the printed sample and training inputs.","prediction_if_false":"Direct tokenization will remain identical, shifting the diagnosis toward inactive template/BOS configuration or genuinely token-equivalent formatting.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,452

benchmark/results/gpt56-medium/events/control/case_09.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The critic bootstraps across a true terminal transition, likely because the return/TD target uses the reset observation's value or applies the terminal mask at the wrong timestep. This leaks post-reset reward into the preceding episode.","confidence":0.96,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Return computation is correct for continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward indexing or the general return recurrence."],"competing_hypotheses":["The environment's terminated flag is discarded or confused with truncated before target construction.","An off-by-one error pairs the terminal mask with the next transition rather than the transition producing the terminal state.","The rollout buffer replaces the terminal next observation with the reset observation before computing value targets."],"first_action":"Run a deterministic handcrafted two-step trajectory with rewards [0, 1], terminated flags [false, true], and an intentionally large reset-state value such as 100. Assert that the targets are [gamma, 1] and remain unchanged when the reset-state value is varied.","prediction_if_true":"The terminal-step target currently changes with the reset-state value; applying the producing transition's true-terminal mask makes it exactly 1 and removes the reset reward from earlier values.","prediction_if_false":"The targets remain invariant to the reset-state value, so the next check should trace reward, value, and mask indices through rollout storage and the backward return recurrence for an off-by-one alignment error.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_09.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-t0l7qy6s
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-29a1-7e10-8b98-8f19ba628128
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An actor-critic implementation passes a one-step reward probe but fails a two-step probe whose final transition is a true terminal state. Learned values include reward that should occur only after the reset. Returns are otherwise correct on continuing episodes. Diagnose the leading cause and choose a deterministic test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The critic bootstraps across a true terminal transition, likely because the return/TD target uses the reset observation's value or applies the terminal mask at the wrong timestep. This leaks post-reset reward into the preceding episode.","confidence":0.96,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Return computation is correct for continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward indexing or the general return recurrence."],"competing_hypotheses":["The environment's terminated flag is discarded or confused with truncated before target construction.","An off-by-one error pairs the terminal mask with the next transition rather than the transition producing the terminal state.","The rollout buffer replaces the terminal next observation with the reset observation before computing value targets."],"first_action":"Run a deterministic handcrafted two-step trajectory with rewards [0, 1], terminated flags [false, true], and an intentionally large reset-state value such as 100. Assert that the targets are [gamma, 1] and remain unchanged when the reset-state value is varied.","prediction_if_true":"The terminal-step target currently changes with the reset-state value; applying the producing transition's true-terminal mask makes it exactly 1 and removes the reset reward from earlier values.","prediction_if_false":"The targets remain invariant to the reset-state value, so the next check should trace reward, value, and mask indices through rollout storage and the backward return recurrence for an off-by-one alignment error.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,476

benchmark/results/gpt56-medium/events/control/case_10.jsonl

@@ -0,0 +1 @@
+{"root_cause":"Inference: severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures makes the PDE and boundary losses numerically incomparable. The optimizer can reduce the strongly scaled PDE residual while making negligible progress on the physically important boundary constraints.","confidence":0.93,"evidence":["Observation: the PDE residual loss is below 1e-7 while boundary values remain physically wrong; a small dimensional residual therefore does not demonstrate a valid physical solution.","Observation: length is approximately 1e-3 m while temperature is approximately 500 K, creating large scale disparities before derivatives and PDE coefficients are applied.","Observation: per-loss gradient norms differ by roughly nine orders of magnitude, directly showing that the losses exert radically different optimization pressure on shared parameters."],"competing_hypotheses":["The boundary-condition residual is implemented incorrectly, has wrong units, or is evaluated at incorrect coordinates.","Boundary sampling is insufficient or misses part of the physical boundary.","The chosen loss aggregation or weighting amplifies the scale imbalance, although the evidence does not identify one aggregation method as generally preferable.","The PDE admits a low-residual solution that does not satisfy the boundary conditions, so optimization has converged to that unconstrained branch."],"first_action":"Run one controlled nondimensionalization experiment: define x_hat=x/L_ref and T_hat=(T-T_ref)/Delta_T_ref, rewrite the PDE and boundary residuals consistently in those variables, then compare per-loss gradient norms and physical boundary error against the current run using the same initialization, collocation points, optimizer, and training budget.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm disparity by several orders of magnitude and boundary error will improve substantially without requiring a preferred gradient aggregation method; the raw PDE-loss magnitude will no longer be directly comparable to 1e-7.","prediction_if_false":"The gradient disparity or boundary error will remain largely unchanged, increasing the probability of an incorrect boundary residual, bad boundary coordinates or sampling, or an incompatible solution parameterization.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/control/case_10.stderr

@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-r069825m
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-86e6-7f02-8467-e13606672f2c
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A PINN drives its PDE residual loss below 1e-7 while boundary values remain physically wrong. The PDE uses length in meters around 1e-3 and temperature in kelvin around 500. Per-loss gradient norms differ by roughly nine orders of magnitude. Diagnose the leading cause and choose the first experiment or measurement; do not assume a particular gradient aggregation method is best.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures makes the PDE and boundary losses numerically incomparable. The optimizer can reduce the strongly scaled PDE residual while making negligible progress on the physically important boundary constraints.","confidence":0.93,"evidence":["Observation: the PDE residual loss is below 1e-7 while boundary values remain physically wrong; a small dimensional residual therefore does not demonstrate a valid physical solution.","Observation: length is approximately 1e-3 m while temperature is approximately 500 K, creating large scale disparities before derivatives and PDE coefficients are applied.","Observation: per-loss gradient norms differ by roughly nine orders of magnitude, directly showing that the losses exert radically different optimization pressure on shared parameters."],"competing_hypotheses":["The boundary-condition residual is implemented incorrectly, has wrong units, or is evaluated at incorrect coordinates.","Boundary sampling is insufficient or misses part of the physical boundary.","The chosen loss aggregation or weighting amplifies the scale imbalance, although the evidence does not identify one aggregation method as generally preferable.","The PDE admits a low-residual solution that does not satisfy the boundary conditions, so optimization has converged to that unconstrained branch."],"first_action":"Run one controlled nondimensionalization experiment: define x_hat=x/L_ref and T_hat=(T-T_ref)/Delta_T_ref, rewrite the PDE and boundary residuals consistently in those variables, then compare per-loss gradient norms and physical boundary error against the current run using the same initialization, collocation points, optimizer, and training budget.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm disparity by several orders of magnitude and boundary error will improve substantially without requiring a preferred gradient aggregation method; the raw PDE-loss magnitude will no longer be directly comparable to 1e-7.","prediction_if_false":"The gradient disparity or boundary error will remain largely unchanged, increasing the probability of an incorrect boundary residual, bad boundary coordinates or sampling, or an incompatible solution parameterization.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,748

benchmark/results/gpt56-medium/events/treatment/case_01.jsonl

@@ -0,0 +1 @@
+{"root_cause":"Likely a double-softmax bug: the model applies softmax before a cross-entropy loss that expects raw logits, compressing class differences and weakening gradients.","confidence":0.78,"evidence":["Observation: Initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: Output entropy remains near ln(10) after 500 steps.","Observation: Logits are non-uniform and gradients reach the classifier head, but gradient magnitudes are unexpectedly small.","Inference: Applying cross-entropy to already-softmaxed probabilities would preserve gradient flow while substantially attenuating it, matching these symptoms better than a disconnected graph.","Inference: A low learning rate can also cause slow progress, but it does not itself explain unexpectedly small gradients measured before the optimizer update."],"competing_hypotheses":["Merely low learning rate or an incorrectly scaled scheduler.","Excessive temperature or another logit-scaling operation before the loss.","Strong gradient scaling, averaging, or loss normalization upstream of backward()."],"first_action":"On one fixed batch, compute the loss and classifier-head gradient norm twice without an optimizer step: cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels). This isolates loss wiring from learning rate.","prediction_if_true":"The raw-logit version produces materially larger gradients and a stronger loss response to class separation; inspecting the training path reveals probabilities being passed into cross-entropy.","prediction_if_false":"Both versions are not representative of the actual training path, or the real path already passes raw logits; then inspect loss scaling and scheduler/effective learning rate.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_01.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-5rg7w3wn
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-5fdd-76b3-bcc5-d7badcb7101c
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A 10-class classifier starts at loss 2.303 and reaches only 2.25 after 500 steps. The data and labels look correct, it can produce non-uniform logits, and gradients reach the classifier head but are much smaller than expected. The training log records output entropy near ln(10). Diagnose the leading cause and choose the cheapest test that would distinguish it from a merely low learning rate.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Likely a double-softmax bug: the model applies softmax before a cross-entropy loss that expects raw logits, compressing class differences and weakening gradients.","confidence":0.78,"evidence":["Observation: Initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: Output entropy remains near ln(10) after 500 steps.","Observation: Logits are non-uniform and gradients reach the classifier head, but gradient magnitudes are unexpectedly small.","Inference: Applying cross-entropy to already-softmaxed probabilities would preserve gradient flow while substantially attenuating it, matching these symptoms better than a disconnected graph.","Inference: A low learning rate can also cause slow progress, but it does not itself explain unexpectedly small gradients measured before the optimizer update."],"competing_hypotheses":["Merely low learning rate or an incorrectly scaled scheduler.","Excessive temperature or another logit-scaling operation before the loss.","Strong gradient scaling, averaging, or loss normalization upstream of backward()."],"first_action":"On one fixed batch, compute the loss and classifier-head gradient norm twice without an optimizer step: cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels). This isolates loss wiring from learning rate.","prediction_if_true":"The raw-logit version produces materially larger gradients and a stronger loss response to class separation; inspecting the training path reveals probabilities being passed into cross-entropy.","prediction_if_false":"Both versions are not representative of the actual training path, or the real path already passes raw logits; then inspect loss scaling and scheduler/effective learning rate.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,704

benchmark/results/gpt56-medium/events/treatment/case_02.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The labels are almost certainly fully masked (every label is -100), so the loss function has no supervised tokens and produces zero loss and zero gradients.","confidence":0.94,"evidence":["Observation: Training loss is exactly 0.000 from the first step.","Observation: Every parameter gradient is zero.","Observation: Generation remains unchanged.","Observation: Logs show input_ids and attention_mask, but do not show labels.","Observation: The decoded input looks plausible, which checks tokenization but not supervision masking.","Inference: The joint pattern is characteristic of all labels being set to the ignore index (-100), commonly due to faulty response-boundary or completion-only masking."],"competing_hypotheses":["The loss is accidentally multiplied by zero or replaced with a detached constant.","The logits or loss are detached from the model computation graph.","The logged loss is incorrect, though this would not alone explain zero gradients and unchanged generation."],"first_action":"Print one real batch's decoded input alongside labels, then assert `(labels != -100).sum() > 0` before the forward pass. This is the cheapest test because it directly distinguishes missing supervision from downstream loss or autograd bugs.","prediction_if_true":"Every label will be -100, the supervised-token count will be zero, and the assertion will fail before training.","prediction_if_false":"At least one label will contain a real token ID; next inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_02.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-15ei36o4
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-5ca6-7ac2-9125-4578ac97a992
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An instruction-tuned language model reports training loss exactly 0.000 from the first step. Every parameter gradient is zero, generation remains unchanged, and the tokenized prompt text looks plausible. The training logs print input_ids and attention_mask but not labels. Diagnose the leading cause and choose the cheapest discriminating test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The labels are almost certainly fully masked (every label is -100), so the loss function has no supervised tokens and produces zero loss and zero gradients.","confidence":0.94,"evidence":["Observation: Training loss is exactly 0.000 from the first step.","Observation: Every parameter gradient is zero.","Observation: Generation remains unchanged.","Observation: Logs show input_ids and attention_mask, but do not show labels.","Observation: The decoded input looks plausible, which checks tokenization but not supervision masking.","Inference: The joint pattern is characteristic of all labels being set to the ignore index (-100), commonly due to faulty response-boundary or completion-only masking."],"competing_hypotheses":["The loss is accidentally multiplied by zero or replaced with a detached constant.","The logits or loss are detached from the model computation graph.","The logged loss is incorrect, though this would not alone explain zero gradients and unchanged generation."],"first_action":"Print one real batch's decoded input alongside labels, then assert `(labels != -100).sum() > 0` before the forward pass. This is the cheapest test because it directly distinguishes missing supervision from downstream loss or autograd bugs.","prediction_if_true":"Every label will be -100, the supervised-token count will be zero, and the assertion will fail before training.","prediction_if_false":"At least one label will contain a real token ID; next inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,505

benchmark/results/gpt56-medium/events/treatment/case_03.jsonl

@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation path is almost certainly not evaluating the claimed randomly initialized classifier. The leading cause is target leakage or reuse of cached/stale trained logits in the loss-and-metric pipeline.","confidence":0.97,"evidence":["Observation: uniform predictions over 100 classes imply cross-entropy near ln(100) = 4.605, not 0.04.","Observation: a random 100-class classifier should have approximately 1% accuracy, not 96%.","Observation: the low loss and high accuracy agree with each other, suggesting the same incorrect or leaked predictions feed both metrics.","Observation: inspecting a separately loaded example does not verify the tensors, targets, logits, or metric state used by the validation loader."],"competing_hypotheses":["A trained checkpoint is loaded despite the model being described as randomly initialized.","Validation metrics are stale or accumulated from an earlier trained evaluation.","The validation loader or evaluation code substitutes labels, one-hot targets, or cached predictions for model logits.","The reported values come from the wrong run or logger series."],"first_action":"Take one batch directly from the validation loader, run one forward pass, and recompute cross-entropy and top-1 accuracy locally from those exact logits and targets; then randomly permute only the targets and recompute both metrics. Do not use the existing metric accumulator or logging path.","prediction_if_true":"The direct or permuted-target results will disagree with the reported metrics, or inspection will show that the evaluated logits are cached, stale, or derived from targets. With permuted targets, genuine model predictions should fall to about 1% accuracy and loss should rise toward or above 4.605.","prediction_if_false":"The direct calculation will reproduce roughly 0.04 loss and 96% accuracy, while target permutation destroys performance. That would localize the problem upstream: the model is not actually random, or the input batch contains an extraordinarily explicit label leak.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_03.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-mvl63bua
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-f68f-79c2-a004-34bfda82ee26
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A randomly initialized 100-class image classifier has cross-entropy loss 0.04 before any optimizer step, while accuracy on the validation loader is 96%. A separately loaded raw image and label look reasonable. Diagnose the leading cause and choose a test that localizes it without changing the model.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: the validation path is almost certainly not evaluating the claimed randomly initialized classifier. The leading cause is target leakage or reuse of cached/stale trained logits in the loss-and-metric pipeline.","confidence":0.97,"evidence":["Observation: uniform predictions over 100 classes imply cross-entropy near ln(100) = 4.605, not 0.04.","Observation: a random 100-class classifier should have approximately 1% accuracy, not 96%.","Observation: the low loss and high accuracy agree with each other, suggesting the same incorrect or leaked predictions feed both metrics.","Observation: inspecting a separately loaded example does not verify the tensors, targets, logits, or metric state used by the validation loader."],"competing_hypotheses":["A trained checkpoint is loaded despite the model being described as randomly initialized.","Validation metrics are stale or accumulated from an earlier trained evaluation.","The validation loader or evaluation code substitutes labels, one-hot targets, or cached predictions for model logits.","The reported values come from the wrong run or logger series."],"first_action":"Take one batch directly from the validation loader, run one forward pass, and recompute cross-entropy and top-1 accuracy locally from those exact logits and targets; then randomly permute only the targets and recompute both metrics. Do not use the existing metric accumulator or logging path.","prediction_if_true":"The direct or permuted-target results will disagree with the reported metrics, or inspection will show that the evaluated logits are cached, stale, or derived from targets. With permuted targets, genuine model predictions should fall to about 1% accuracy and loss should rise toward or above 4.605.","prediction_if_false":"The direct calculation will reproduce roughly 0.04 loss and 96% accuracy, while target permutation destroys performance. That would localize the problem upstream: the model is not actually random, or the input batch contains an extraordinarily explicit label leak.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,911

benchmark/results/gpt56-medium/events/treatment/case_04.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous memory instead of swapping tensor axes, so features or tokens are being mixed across the batch dimension. The leading cause is a tensor-layout bug introduced by replacing transpose/permute with reshape.","confidence":0.98,"evidence":["Observation: validation example 3 changes when unrelated examples in the same batch are replaced.","Observation: shapes remain valid, so shape checks cannot detect the corruption.","Observation: single-example inference looks normal, which is expected when cross-example mixing is hidden by batch size 1.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserves linear storage order and cannot generally implement an axis permutation.","Inference: the batch-dependence strongly localizes the error to unintended mixing across the batch axis."],"competing_hypotheses":["Unmasked batch-coupled computation such as BatchNorm or attention across examples could cause the same symptom, but the transpose-to-reshape change is much stronger direct evidence.","Mutable cache or shared state keyed incorrectly across examples is possible, but less likely because the failure began after a tensor-layout refactor."],"first_action":"Run a batch-independence dependency test: compute the selected output scalar for example 3, backpropagate it to the batched input, and assert that gradients for every other batch element are exactly zero; repeat after restoring the explicit transpose/permute. This mechanically localizes cross-example mixing and verifies the proposed cause.","prediction_if_true":"With the reshape, the selected output for example 3 will have nonzero gradients with respect to at least one unrelated batch element; replacing it with the intended transpose/permute will eliminate those gradients and make example 3 invariant to batch companions.","prediction_if_false":"The selected output will have zero gradients to unrelated inputs despite changing with batch companions, pointing instead to non-differentiable shared state, batch-dependent normalization, caching, randomness, or an error in the test setup.","immediate_changes":["Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_04.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-3h01c4dl
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-aafa-7953-9826-8f228fc46b27
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After a tensor-layout refactor, prediction for validation example 3 changes when unrelated examples elsewhere in the same batch are replaced. Shapes remain valid and single-example inference looks normal. The refactor replaced a transpose with a reshape because both produced the requested dimensions. Diagnose the leading cause and choose a mechanical test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The reshape is reinterpreting contiguous memory instead of swapping tensor axes, so features or tokens are being mixed across the batch dimension. The leading cause is a tensor-layout bug introduced by replacing transpose/permute with reshape.","confidence":0.98,"evidence":["Observation: validation example 3 changes when unrelated examples in the same batch are replaced.","Observation: shapes remain valid, so shape checks cannot detect the corruption.","Observation: single-example inference looks normal, which is expected when cross-example mixing is hidden by batch size 1.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserves linear storage order and cannot generally implement an axis permutation.","Inference: the batch-dependence strongly localizes the error to unintended mixing across the batch axis."],"competing_hypotheses":["Unmasked batch-coupled computation such as BatchNorm or attention across examples could cause the same symptom, but the transpose-to-reshape change is much stronger direct evidence.","Mutable cache or shared state keyed incorrectly across examples is possible, but less likely because the failure began after a tensor-layout refactor."],"first_action":"Run a batch-independence dependency test: compute the selected output scalar for example 3, backpropagate it to the batched input, and assert that gradients for every other batch element are exactly zero; repeat after restoring the explicit transpose/permute. This mechanically localizes cross-example mixing and verifies the proposed cause.","prediction_if_true":"With the reshape, the selected output for example 3 will have nonzero gradients with respect to at least one unrelated batch element; replacing it with the intended transpose/permute will eliminate those gradients and make example 3 invariant to batch companions.","prediction_if_false":"The selected output will have zero gradients to unrelated inputs despite changing with batch companions, pointing instead to non-differentiable shared state, batch-dependent normalization, caching, randomness, or an error in the test setup.","immediate_changes":["Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."],"fallback_logic":[]}
+tokens used
+17,469

benchmark/results/gpt56-medium/events/treatment/case_05.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring the AdamW parameter update. It logs learning_rate * parameter.grad, but AdamW applies moment normalization, epsilon handling, bias correction, and decoupled weight decay. Measuring this proxy after optimizer.step() also makes it sensitive to gradient-clearing order. The approximately -8 values therefore do not contradict learning or changing parameters.","confidence":0.97,"evidence":["Observation: The training objective improves steadily.","Observation: Parameter checksums change after every optimizer step.","Observation: The numerator is computed as learning_rate * parameter.grad after optimizer.step().","Inference: Raw gradient times learning rate is not the update applied by AdamW.","Inference: Direct parameter differences are the cheapest authoritative measurement of the actual update."],"competing_hypotheses":["The gradients are genuinely tiny while decoupled weight decay accounts for most parameter changes.","The dashboard aggregation or log10 calculation is incorrect.","Checksums detect numerically negligible changes that do not represent useful learning; steady objective improvement makes this unlikely."],"first_action":"For one step, clone each parameter immediately before optimizer.step(), then compute log10(norm(parameter_after - parameter_before) / norm(parameter_before)) and compare it with the dashboard value.","prediction_if_true":"The direct parameter-delta ratios will be materially larger than about -8 and will track the changing checksums, confirming that the dashboard proxy is wrong.","prediction_if_false":"The direct ratios will also be about -8; then inspect whether weight decay alone explains the changes and verify the dashboard's norm aggregation and parameter matching.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_05.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-9smnkemj
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-e3cb-78f3-a5ab-39b7432c82af
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An AdamW run improves steadily and parameter checksums change every step, but the dashboard says every layer's log10(update/parameter ratio) is about -8. The logger computes the numerator as learning_rate * parameter.grad after optimizer.step(). Diagnose the discrepancy and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The dashboard is not measuring the AdamW parameter update. It logs learning_rate * parameter.grad, but AdamW applies moment normalization, epsilon handling, bias correction, and decoupled weight decay. Measuring this proxy after optimizer.step() also makes it sensitive to gradient-clearing order. The approximately -8 values therefore do not contradict learning or changing parameters.","confidence":0.97,"evidence":["Observation: The training objective improves steadily.","Observation: Parameter checksums change after every optimizer step.","Observation: The numerator is computed as learning_rate * parameter.grad after optimizer.step().","Inference: Raw gradient times learning rate is not the update applied by AdamW.","Inference: Direct parameter differences are the cheapest authoritative measurement of the actual update."],"competing_hypotheses":["The gradients are genuinely tiny while decoupled weight decay accounts for most parameter changes.","The dashboard aggregation or log10 calculation is incorrect.","Checksums detect numerically negligible changes that do not represent useful learning; steady objective improvement makes this unlikely."],"first_action":"For one step, clone each parameter immediately before optimizer.step(), then compute log10(norm(parameter_after - parameter_before) / norm(parameter_before)) and compare it with the dashboard value.","prediction_if_true":"The direct parameter-delta ratios will be materially larger than about -8 and will track the changing checksums, confirming that the dashboard proxy is wrong.","prediction_if_false":"The direct ratios will also be about -8; then inspect whether weight decay alone explains the changes and verify the dashboard's norm aggregation and parameter matching.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+17,566

benchmark/results/gpt56-medium/events/treatment/case_06.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The constant feature has zero batch standard deviation, so batch-local standardization computes 0/0 and creates NaNs that propagate into the loss.","confidence":0.99,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch of the failing shard immediately produces a NaN loss.","Observation: That batch contains a feature that is constant across every example.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch standard deviation is zero; subtracting its mean produces zero, and dividing by zero yields NaN.","Inference: The shard-specific onset is explained by the feature becoming constant only in that batch."],"competing_hypotheses":["A later operation is independently invalid on this shard, such as log of a non-positive value; unlikely given the exact zero-variance standardization mechanism.","The raw batch already contains NaN or infinity; possible but not supported by the supplied evidence.","Mixed-precision overflow occurs later in the model; highly unlikely because the preprocessing already has a deterministic 0/0 path."],"first_action":"Replay the offending batch and add fail-fast finite assertions after each preprocessing operation: raw input, batch mean, batch standard deviation, centered feature, and standardized feature. This should localize the first invalid value to the division by the zero standard deviation.","prediction_if_true":"The raw feature, mean, standard deviation, and centered feature will be finite; the standard deviation will be exactly zero, and the first non-finite value will appear in the standardized feature immediately after division.","prediction_if_false":"A non-finite value will appear before standardization, the standard deviation will be nonzero, or all standardized features will remain finite, requiring inspection of the next operation in execution order.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_06.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-t6lcw1nt
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-eace-7d13-a6ba-ebc5fba0868e
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training is finite on most shards but the first batch from one shard makes the loss NaN. That batch contains a feature that is constant across all examples. Preprocessing standardizes each feature using statistics from the current batch. Diagnose the leading cause and choose a test that localizes the first invalid operation. State whether you would add epsilon or clamp immediately.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The constant feature has zero batch standard deviation, so batch-local standardization computes 0/0 and creates NaNs that propagate into the loss.","confidence":0.99,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch of the failing shard immediately produces a NaN loss.","Observation: That batch contains a feature that is constant across every example.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch standard deviation is zero; subtracting its mean produces zero, and dividing by zero yields NaN.","Inference: The shard-specific onset is explained by the feature becoming constant only in that batch."],"competing_hypotheses":["A later operation is independently invalid on this shard, such as log of a non-positive value; unlikely given the exact zero-variance standardization mechanism.","The raw batch already contains NaN or infinity; possible but not supported by the supplied evidence.","Mixed-precision overflow occurs later in the model; highly unlikely because the preprocessing already has a deterministic 0/0 path."],"first_action":"Replay the offending batch and add fail-fast finite assertions after each preprocessing operation: raw input, batch mean, batch standard deviation, centered feature, and standardized feature. This should localize the first invalid value to the division by the zero standard deviation.","prediction_if_true":"The raw feature, mean, standard deviation, and centered feature will be finite; the standard deviation will be exactly zero, and the first non-finite value will appear in the standardized feature immediately after division.","prediction_if_false":"A non-finite value will appear before standardization, the standard deviation will be nonzero, or all standardized features will remain finite, requiring inspection of the next operation in execution order.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,689

benchmark/results/gpt56-medium/events/treatment/case_07.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The validation pass is probably running with the model still in training mode. no_grad disables gradient recording but does not disable dropout or batch-normalization updates, so dropout remains stochastic and batch normalization uses and updates batch statistics.","confidence":0.94,"evidence":["Observation: Repeated validation passes use the same frozen batch but produce different predictions and losses.","Observation: The model contains dropout and batch-normalization modules.","Observation: Validation uses no_grad, but the module training flag is not logged.","Inference: Input variation is ruled out by the frozen batch, making model-state stochasticity the leading explanation.","Inference: If model.training is True, active dropout directly explains stochastic predictions, while training-mode batch normalization can also mutate running statistics across passes."],"competing_hypotheses":["A stochastic operation other than dropout is active during validation.","Model parameters or buffers are being mutated elsewhere between validation passes.","Validation preprocessing or the supposedly frozen batch is not actually deterministic."],"first_action":"Log model.training immediately before validation and repeat the frozen-batch pass after calling model.eval(); compare predictions and loss across repetitions.","prediction_if_true":"model.training is currently True, and repeated outputs become identical or numerically indistinguishable after model.eval().","prediction_if_false":"model.training is already False or outputs still vary after model.eval(), so the next check is parameter/buffer mutation and other stochastic operations.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_07.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-fmsqplqd
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-27c4-7473-98c0-9dba3370a95e
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training metrics improve normally, but repeated validation passes over the same frozen batch give noticeably different predictions and loss. The model contains dropout and batch normalization. The validation loop uses no_grad, but the logs never record the module training flag. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The validation pass is probably running with the model still in training mode. no_grad disables gradient recording but does not disable dropout or batch-normalization updates, so dropout remains stochastic and batch normalization uses and updates batch statistics.","confidence":0.94,"evidence":["Observation: Repeated validation passes use the same frozen batch but produce different predictions and losses.","Observation: The model contains dropout and batch-normalization modules.","Observation: Validation uses no_grad, but the module training flag is not logged.","Inference: Input variation is ruled out by the frozen batch, making model-state stochasticity the leading explanation.","Inference: If model.training is True, active dropout directly explains stochastic predictions, while training-mode batch normalization can also mutate running statistics across passes."],"competing_hypotheses":["A stochastic operation other than dropout is active during validation.","Model parameters or buffers are being mutated elsewhere between validation passes.","Validation preprocessing or the supposedly frozen batch is not actually deterministic."],"first_action":"Log model.training immediately before validation and repeat the frozen-batch pass after calling model.eval(); compare predictions and loss across repetitions.","prediction_if_true":"model.training is currently True, and repeated outputs become identical or numerically indistinguishable after model.eval().","prediction_if_false":"model.training is already False or outputs still vary after model.eval(), so the next check is parameter/buffer mutation and other stochastic operations.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,519

benchmark/results/gpt56-medium/events/treatment/case_08.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset cache is stale: the cached map result is being reused despite the chat-template and BOS changes, so training still receives the previous token IDs.","confidence":0.96,"evidence":["Observation: Raw source conversations changed as expected.","Observation: The printed tokenized sample is byte-for-byte identical to the previous run.","Observation: Fine-tuning behavior is also identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: The change is being lost between raw-data loading and model input, with cache reuse as the leading explanation."],"competing_hypotheses":["The updated chat template or BOS configuration is not actually passed into the tokenization function.","The template changes render identically for this particular sample, though this is unlikely given identical training behavior."],"first_action":"Force recomputation of the cached map for one known changed conversation, then compare its rendered text and token IDs with the cached result. This is the cheapest test that directly distinguishes stale cache reuse from an ineffective template change.","prediction_if_true":"The freshly recomputed rendered text or token IDs will differ from the cached sample, and a full uncached tokenization will change fine-tuning inputs.","prediction_if_false":"Fresh recomputation will remain byte-for-byte identical, shifting suspicion to the tokenizer path using the old template/BOS configuration or to changes that do not affect rendered input.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_08.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-9cuukt3u
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-aff7-7e83-a192-b609d2f1ee13
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After changing the chat template and BOS handling, fine-tuning behavior and the printed first tokenized sample are byte-for-byte identical to the previous run. Raw source conversations changed as expected. Tokenization uses a cached dataset map. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The tokenized dataset cache is stale: the cached map result is being reused despite the chat-template and BOS changes, so training still receives the previous token IDs.","confidence":0.96,"evidence":["Observation: Raw source conversations changed as expected.","Observation: The printed tokenized sample is byte-for-byte identical to the previous run.","Observation: Fine-tuning behavior is also identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: The change is being lost between raw-data loading and model input, with cache reuse as the leading explanation."],"competing_hypotheses":["The updated chat template or BOS configuration is not actually passed into the tokenization function.","The template changes render identically for this particular sample, though this is unlikely given identical training behavior."],"first_action":"Force recomputation of the cached map for one known changed conversation, then compare its rendered text and token IDs with the cached result. This is the cheapest test that directly distinguishes stale cache reuse from an ineffective template change.","prediction_if_true":"The freshly recomputed rendered text or token IDs will differ from the cached sample, and a full uncached tokenization will change fine-tuning inputs.","prediction_if_false":"Fresh recomputation will remain byte-for-byte identical, shifting suspicion to the tokenizer path using the old template/BOS configuration or to changes that do not affect rendered input.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,495

benchmark/results/gpt56-medium/events/treatment/case_09.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The critic target probably bootstraps through a true terminal transition, likely because the done/terminated mask is inverted, shifted, or uses truncation semantics. This leaks the reset episode's value into the preceding episode.","confidence":0.9,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Returns are correct on continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward accumulation in general.","Inference: Post-reset reward appearing before termination is the expected signature of using gamma * V(next_state) when that term should be zero."],"competing_hypotheses":["The terminal flag is attached to the wrong timestep, causing an off-by-one bootstrap mask.","The implementation treats true termination as truncation and intentionally bootstraps both.","The reset observation overwrites the terminal next observation before target computation."],"first_action":"Add a deterministic target unit test with reward r=1, terminated=true, truncated=false, gamma=0.9, and V(next_state)=100. Assert that the target is exactly 1, then repeat with terminated=false and assert 91. This directly distinguishes terminal masking from general return errors.","prediction_if_true":"The terminal case produces 91, or otherwise changes with V(next_state), proving that the bootstrap term crosses the terminal boundary.","prediction_if_false":"The terminal target remains exactly 1 for arbitrary next-state values, so inspect timestep alignment and whether reset observations overwrite terminal observations before returns are constructed.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_09.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-nl5ql_f0
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-2a3f-75b2-8d2c-2a6c32ef72b4
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An actor-critic implementation passes a one-step reward probe but fails a two-step probe whose final transition is a true terminal state. Learned values include reward that should occur only after the reset. Returns are otherwise correct on continuing episodes. Diagnose the leading cause and choose a deterministic test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The critic target probably bootstraps through a true terminal transition, likely because the done/terminated mask is inverted, shifted, or uses truncation semantics. This leaks the reset episode's value into the preceding episode.","confidence":0.9,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Returns are correct on continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward accumulation in general.","Inference: Post-reset reward appearing before termination is the expected signature of using gamma * V(next_state) when that term should be zero."],"competing_hypotheses":["The terminal flag is attached to the wrong timestep, causing an off-by-one bootstrap mask.","The implementation treats true termination as truncation and intentionally bootstraps both.","The reset observation overwrites the terminal next observation before target computation."],"first_action":"Add a deterministic target unit test with reward r=1, terminated=true, truncated=false, gamma=0.9, and V(next_state)=100. Assert that the target is exactly 1, then repeat with terminated=false and assert 91. This directly distinguishes terminal masking from general return errors.","prediction_if_true":"The terminal case produces 91, or otherwise changes with V(next_state), proving that the bootstrap term crosses the terminal boundary.","prediction_if_false":"The terminal target remains exactly 1 for arbitrary next-state values, so inspect timestep alignment and whether reset observations overwrite terminal observations before returns are constructed.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,533

benchmark/results/gpt56-medium/events/treatment/case_10.jsonl

@@ -0,0 +1 @@
+{"root_cause":"The dimensional PINN objective is severely ill-conditioned. Mixing length near 1e-3 m with temperature near 500 K makes the PDE and boundary residuals numerically incomparable; the nine-order gradient-norm gap indicates that optimization can reduce the PDE residual while providing negligible effective pressure to satisfy the boundary conditions.","confidence":0.9,"evidence":["Observation: the PDE residual loss is below 1e-7.","Observation: boundary values remain physically wrong.","Observation: input and output scales differ substantially: length is about 1e-3 m and temperature about 500 K.","Observation: per-loss gradient norms differ by about nine orders of magnitude.","Inference: the small dimensional PDE loss is not evidence that the full boundary-value problem is solved.","Inference: the gradient disparity is consistent with unit-induced conditioning failure and one loss term dominating parameter updates."],"competing_hypotheses":["The boundary-condition implementation, target values, point locations, or derivative coordinates are wrong.","Boundary points are sampled or weighted too sparsely relative to interior collocation points.","The network architecture cannot represent the boundary behavior, although this is less likely than scaling given the nine-order gradient disparity.","Derivative computation uses normalized coordinates without the required chain-rule scale factors."],"first_action":"Rewrite one cheap run in dimensionless coordinates and fields, including explicit chain-rule factors, then measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms using the same seed. The prediction is about conditioning; this experiment does not require choosing a gradient aggregation method.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm gap by several orders of magnitude, and boundary error will fall while the dimensionless PDE residual remains controlled.","prediction_if_false":"The gradient gap or boundary failure will persist after correct nondimensionalization, shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation, followed by a tiny boundary-value overfit test.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/events/treatment/case_10.stderr

@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-45n99rtf
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-ea8e-7511-ae73-b1601d56fd99
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A PINN drives its PDE residual loss below 1e-7 while boundary values remain physically wrong. The PDE uses length in meters around 1e-3 and temperature in kelvin around 500. Per-loss gradient norms differ by roughly nine orders of magnitude. Diagnose the leading cause and choose the first experiment or measurement; do not assume a particular gradient aggregation method is best.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The dimensional PINN objective is severely ill-conditioned. Mixing length near 1e-3 m with temperature near 500 K makes the PDE and boundary residuals numerically incomparable; the nine-order gradient-norm gap indicates that optimization can reduce the PDE residual while providing negligible effective pressure to satisfy the boundary conditions.","confidence":0.9,"evidence":["Observation: the PDE residual loss is below 1e-7.","Observation: boundary values remain physically wrong.","Observation: input and output scales differ substantially: length is about 1e-3 m and temperature about 500 K.","Observation: per-loss gradient norms differ by about nine orders of magnitude.","Inference: the small dimensional PDE loss is not evidence that the full boundary-value problem is solved.","Inference: the gradient disparity is consistent with unit-induced conditioning failure and one loss term dominating parameter updates."],"competing_hypotheses":["The boundary-condition implementation, target values, point locations, or derivative coordinates are wrong.","Boundary points are sampled or weighted too sparsely relative to interior collocation points.","The network architecture cannot represent the boundary behavior, although this is less likely than scaling given the nine-order gradient disparity.","Derivative computation uses normalized coordinates without the required chain-rule scale factors."],"first_action":"Rewrite one cheap run in dimensionless coordinates and fields, including explicit chain-rule factors, then measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms using the same seed. The prediction is about conditioning; this experiment does not require choosing a gradient aggregation method.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm gap by several orders of magnitude, and boundary error will fall while the dimensionless PDE residual remains controlled.","prediction_if_false":"The gradient gap or boundary failure will persist after correct nondimensionalization, shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation, followed by a tiny boundary-value overfit test.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,624

benchmark/results/gpt56-medium/metadata.json

@@ -0,0 +1,53 @@
+{
+  "run_id": "gpt56-medium",
+  "model": "gpt-5.6-sol",
+  "workers": 4,
+  "reasoning_effort": "medium",
+  "case_ids": [
+    "case_01",
+    "case_02",
+    "case_03",
+    "case_04",
+    "case_05",
+    "case_06",
+    "case_07",
+    "case_08",
+    "case_09",
+    "case_10"
+  ],
+  "conditions": [
+    "control",
+    "treatment"
+  ],
+  "skill_sha256": "adbdd538b81357ed43d58f0027386d45b272ffb31ba09a4eab5552ee12030948",
+  "base_instruction_sha256": "06378fe7d972e6d6f8be0780597132dd9f990ff75a2acac74d59ce6b986233f2",
+  "fixture_sha256": {
+    "cases.json": "9c3972377f1ebf635afb522634c0d64c182f6abd53bbb859ea834fca66181307",
+    "answers.json": "5c27df932406d571c9e06a823b30695d5debd07fd25ed3e51c7ac0143b84fab0",
+    "response.schema.json": "19bfe551ae38efd4829bc25f53b9b59e956620392fe1f67d737be1a4900db4e6"
+  },
+  "codex_version": "codex-cli 0.144.1",
+  "git_commit": "a0fcfa291e41316924424ba3f2def33eb37b7338",
+  "job_order": [
+    "treatment/case_10",
+    "control/case_03",
+    "treatment/case_06",
+    "control/case_01",
+    "treatment/case_07",
+    "control/case_09",
+    "treatment/case_09",
+    "control/case_02",
+    "control/case_05",
+    "treatment/case_02",
+    "treatment/case_01",
+    "control/case_10",
+    "control/case_07",
+    "treatment/case_04",
+    "treatment/case_08",
+    "control/case_08",
+    "control/case_06",
+    "treatment/case_05",
+    "control/case_04",
+    "treatment/case_03"
+  ]
+}

benchmark/results/gpt56-medium/rating_comparison.tsv

@@ -0,0 +1,5 @@
+rater	condition	root_cause_correct	discriminating_test	localized_before_change	unsupported_change	fallback_logic_proposed
+strict	control	8	10	10	0	0
+strict	treatment	10	9	9	0	0
+independent	control	9	9	10	0	0
+independent	treatment	10	10	10	0	0

benchmark/results/gpt56-medium/ratings.independent.json

@@ -0,0 +1,366 @@
+{
+  "control": {
+    "case_01": {
+      "scores": {"root_cause_correct": false, "discriminating_test": false, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "divided by an excessively large temperature"}],
+        "discriminating_test": [{"field": "first_action", "quote": "once with temperature scaling removed (T=1)"}],
+        "localized_before_change": [{"field": "first_action", "quote": "run forward/backward twice without optimizer steps"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "The response selects excessive temperature scaling, while the key says CrossEntropy receives softmax probabilities.",
+        "discriminating_test": "The test distinguishes temperature scaling from low learning rate, but does not test the keyed double-softmax loss wiring.",
+        "localized_before_change": "It compares two loss paths without stepping the optimizer before proposing any change.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_02": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "labels are probably fully masked"}],
+        "discriminating_test": [{"field": "first_action", "quote": "count labels != -100 per example"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Print one batch's labels"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies all labels being masked to the ignore index.",
+        "discriminating_test": "Counting real labels in the collated batch directly tests the keyed cause.",
+        "localized_before_change": "It inspects labels before changing the collator or loss.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_03": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "target-derived values to be treated as predictions"}],
+        "discriminating_test": [{"field": "first_action", "quote": "independently compute `cross_entropy(logits, labels)`"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Take one exact batch from the validation loader"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "Target-derived predictions are a concrete form of target leakage, although the response emphasizes evaluation wiring rather than split contamination.",
+        "discriminating_test": "Independent recomputation on the exact loader batch separates evaluation wiring from genuinely leaked model inputs or loaded weights.",
+        "localized_before_change": "It inspects exact logits, labels, and metrics without changing the model.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline wording: the key says target leakage or split contamination, while the response's leading mechanism is prediction/target aliasing inside evaluation. I count this as target leakage."
+    },
+    "case_04": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "reshape is reinterpreting contiguous storage instead of swapping tensor axes"}],
+        "discriminating_test": [{"field": "first_action", "quote": "mechanical batch-independence test"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compute example 3 alone, in its original batch"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies reshape replacing an axis permutation and mixing logical positions.",
+        "discriminating_test": "Batch-composition invariance directly detects cross-example mixing.",
+        "localized_before_change": "The keyed case does not require further localization, and the response still runs an invariance test first.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_05": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring AdamW's actual parameter update"}],
+        "discriminating_test": [{"field": "first_action", "quote": "clone the parameter immediately before optimizer.step()"}],
+        "localized_before_change": [{"field": "first_action", "quote": "directly measures the update"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the dashboard numerator with the norm of the observed parameter delta"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It correctly distinguishes the AdamW update from learning_rate times raw gradient.",
+        "discriminating_test": "A before/after parameter snapshot measures the actual update.",
+        "localized_before_change": "The keyed case does not require additional localization; the response confirms the metric directly before replacement.",
+        "unsupported_change": "The dashboard replacement is supported by the known metric-definition error and direct confirmation plan.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_06": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "division by zero during batch-local standardization"}],
+        "discriminating_test": [{"field": "first_action", "quote": "finite-value assertions after raw input loading"}],
+        "localized_before_change": [{"field": "first_action", "quote": "report the feature index and denominator at the first invalid operation"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies zero standard deviation for the constant feature.",
+        "discriminating_test": "Successive finite assertions localize the first non-finite operation.",
+        "localized_before_change": "It explicitly localizes before adding epsilon or a clamp.",
+        "unsupported_change": "No epsilon, clamp, or other immediate change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_07": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Log model.training immediately before validation"}],
+        "localized_before_change": [{"field": "first_action", "quote": "repeat the same frozen batch twice after explicitly calling model.eval()"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies validation running with training-mode dropout and batch normalization.",
+        "discriminating_test": "Logging the flag and controlled eval repeat directly distinguishes module mode.",
+        "localized_before_change": "The mode check precedes any persistent validation-loop edit.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_08": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset map cache is being reused"}],
+        "discriminating_test": [{"field": "first_action", "quote": "calling the preprocessing function directly, bypassing Dataset.map and its cache"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compare its token IDs with the cached sample"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies stale cached tokenization after the template change.",
+        "discriminating_test": "Direct uncached preprocessing compared with the cached sample isolates cache reuse.",
+        "localized_before_change": "It compares paths before invalidating the full cache.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_09": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "bootstraps across a true terminal transition"}],
+        "discriminating_test": [{"field": "first_action", "quote": "deterministic handcrafted two-step trajectory"}],
+        "localized_before_change": [{"field": "first_action", "quote": "remain unchanged when the reset-state value is varied"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies bootstrapping through a true terminal reset.",
+        "discriminating_test": "The hand-computed two-step target with a large reset value isolates terminal masking.",
+        "localized_before_change": "It tests target construction before editing mask semantics.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_10": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "severe dimensional ill-conditioning"}],
+        "discriminating_test": [{"field": "first_action", "quote": "controlled nondimensionalization experiment"}],
+        "localized_before_change": [{"field": "first_action", "quote": "same initialization, collocation points, optimizer, and training budget"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies unit-driven per-loss gradient imbalance and the need for nondimensionalization.",
+        "discriminating_test": "The controlled nondimensionalized run measures the predicted gradient-gap and boundary-error changes without choosing an aggregation method.",
+        "localized_before_change": "I count the controlled same-seed experiment as localization rather than an immediate production fix; no immediate change is listed.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline metric semantics: the first action rewrites the problem dimensionlessly, which is an intervention, but it is also the answer key's accepted discriminating test and is framed as a controlled experiment."
+    }
+  },
+  "treatment": {
+    "case_01": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "double-softmax bug"}],
+        "discriminating_test": [{"field": "first_action", "quote": "cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels)"}],
+        "localized_before_change": [{"field": "first_action", "quote": "without an optimizer step"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies probabilities being passed into CrossEntropy.",
+        "discriminating_test": "The paired raw-logit/probability losses directly test the keyed bug and distinguish it from learning rate.",
+        "localized_before_change": "It compares paths without stepping or changing training.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_02": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "every label is -100"}],
+        "discriminating_test": [{"field": "first_action", "quote": "assert `(labels != -100).sum() > 0`"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Print one real batch's decoded input alongside labels"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies complete ignore-index masking.",
+        "discriminating_test": "The real-batch supervised-token assertion directly tests the keyed cause.",
+        "localized_before_change": "It inspects labels before changing masking logic.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_03": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "target leakage or reuse of cached/stale trained logits"}],
+        "discriminating_test": [{"field": "first_action", "quote": "randomly permute only the targets"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Do not use the existing metric accumulator or logging path"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It explicitly includes target leakage, while also retaining stale logits as a competing leading mechanism.",
+        "discriminating_test": "Exact-batch recomputation plus target permutation distinguishes evaluation leakage/wiring from a genuinely informative input or loaded model.",
+        "localized_before_change": "It bypasses metric state and changes no model behavior.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline specificity: the response gives a disjunction of target leakage and cached logits, whereas the key specifies target leakage or split contamination. I count the explicit correct branch."
+    },
+    "case_04": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "tensor-layout bug introduced by replacing transpose/permute with reshape"}],
+        "discriminating_test": [{"field": "first_action", "quote": "backpropagate it to the batched input"}],
+        "localized_before_change": [{"field": "first_action", "quote": "assert that gradients for every other batch element are exactly zero"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the reshape with the explicit transpose/permute"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies reshape as a non-permuting storage reinterpretation that mixes the batch axis.",
+        "discriminating_test": "The backprop-to-input dependency test mechanically detects cross-batch dependence.",
+        "localized_before_change": "The key marks this case as not requiring further localization; the response nevertheless verifies the dependency.",
+        "unsupported_change": "Restoring the explicit permutation is supported by the prompt's known causal refactor and mechanical test.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_05": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring the AdamW parameter update"}],
+        "discriminating_test": [{"field": "first_action", "quote": "clone each parameter immediately before optimizer.step()"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compute log10(norm(parameter_after - parameter_before) / norm(parameter_before))"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It correctly identifies the logged proxy as different from AdamW's applied update.",
+        "discriminating_test": "The parameter snapshot directly measures the actual update.",
+        "localized_before_change": "The keyed case does not require more localization; the response still confirms it directly.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_06": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "zero batch standard deviation"}],
+        "discriminating_test": [{"field": "first_action", "quote": "fail-fast finite assertions after each preprocessing operation"}],
+        "localized_before_change": [{"field": "first_action", "quote": "localize the first invalid value to the division"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies deterministic 0/0 during batch-local standardization.",
+        "discriminating_test": "Successive finite assertions find the first invalid tensor.",
+        "localized_before_change": "It localizes before any epsilon or clamp.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_07": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Log model.training immediately before validation"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compare predictions and loss across repetitions"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies training-mode validation with active dropout and batch normalization.",
+        "discriminating_test": "The flag plus controlled eval repeat directly confirms the mechanism.",
+        "localized_before_change": "It checks state before a persistent code edit.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_08": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset cache is stale"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Force recomputation of the cached map"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compare its rendered text and token IDs with the cached result"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies stale mapped tokenization after template changes.",
+        "discriminating_test": "Forced recomputation compared with cached output directly isolates cache reuse.",
+        "localized_before_change": "It tests one known conversation before changing the full training path.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_09": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "bootstraps through a true terminal transition"}],
+        "discriminating_test": [{"field": "first_action", "quote": "deterministic target unit test"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Assert that the target is exactly 1"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies a terminal-mask bootstrap leak across reset.",
+        "discriminating_test": "The terminal/nonterminal target pair with a large next value isolates masking semantics.",
+        "localized_before_change": "It tests target construction before editing the return recurrence.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_10": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "dimensional PINN objective is severely ill-conditioned"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Rewrite one cheap run in dimensionless coordinates and fields"}],
+        "localized_before_change": [{"field": "first_action", "quote": "using the same seed"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies unit-induced per-loss gradient imbalance and nondimensionalization as the appropriate first intervention.",
+        "discriminating_test": "The same-seed dimensionless run tests whether the gradient gap and boundary error improve without selecting an aggregation method.",
+        "localized_before_change": "I count the controlled cheap run as localization rather than a production fix; no immediate change is listed.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline metric semantics: nondimensionalization is an intervention, but it is explicitly accepted by the answer key as a discriminating test and is framed as a cheap controlled run."
+    }
+  }
+}

benchmark/results/gpt56-medium/ratings.json

@@ -0,0 +1,382 @@
+[
+  {
+    "condition": "control",
+    "case_id": "case_01",
+    "scores": {"root_cause_correct": false, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "logits are probably being divided by an excessively large temperature"}],
+      "discriminating_test": [{"field": "first_action", "quote": "once through the current loss path and once with temperature scaling removed (T=1)"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "first_action", "quote": "A large immediate gradient increase at T=1 is the cheapest discriminating test."}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "The response diagnoses excessive temperature scaling, while the hidden cause is probabilities passed to cross-entropy (double softmax).",
+      "discriminating_test": "Comparing the current path against T=1 separates temperature attenuation from a low learning rate, which would not change raw gradients.",
+      "localized_before_change": "This localization-required answer proposes a discriminating check and no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_02",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "labels are probably fully masked (all set to the ignore index, usually -100)"}],
+      "discriminating_test": [{"field": "first_action", "quote": "count labels != -100 per example"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "first_action", "quote": "distinguishes an all-masked-label failure from optimizer or gradient-path failures"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies all labels being masked to -100, matching the hidden cause.",
+      "discriminating_test": "Counting real labels distinguishes missing supervision from downstream optimizer or autograd failures.",
+      "localized_before_change": "The response checks the real collated labels before changing behavior.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_03",
+    "scores": {"root_cause_correct": false, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "variable-aliasing, stale-tensor, or metric/loss wiring bug"}],
+      "discriminating_test": [{"field": "first_action", "quote": "independently compute `cross_entropy(logits, labels)`"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "Its leading cause is evaluation wiring or stale tensors, not the hidden split contamination or feature-level target leakage.",
+      "discriminating_test": "An independent one-batch metric calculation separates evaluation wiring from an upstream model, loader, or leakage problem.",
+      "localized_before_change": "The localization check has distinct outcomes and no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_04",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "reshape is reinterpreting contiguous storage instead of swapping tensor axes"}],
+      "discriminating_test": [{"field": "first_action", "quote": "replace the reshape with the intended transpose/permute and rerun the same test"}],
+      "localized_before_change": [{"field": "evidence", "quote": "refactor replaced a transpose with a reshape solely because the output dimensions matched"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies reshape-for-transpose axis corruption and cross-batch mixing.",
+      "discriminating_test": "The batch-independence check before and after restoring the permutation separates this cause from other batch-coupled operations.",
+      "localized_before_change": "The supplied refactor evidence already localizes the likely layout operation before the action.",
+      "unsupported_change": "The immediate_changes field is empty.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_05",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring AdamW's actual parameter update"}],
+      "discriminating_test": [{"field": "first_action", "quote": "parameter_after - parameter_before"}],
+      "localized_before_change": [{"field": "evidence", "quote": "learning_rate * parameter.grad is an invalid estimator of an AdamW update"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the dashboard numerator with the norm of the observed parameter delta"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches the hidden logger discrepancy: learning-rate times gradient is not AdamW's actual update.",
+      "discriminating_test": "A before/after parameter snapshot directly distinguishes a bad proxy from genuinely tiny updates.",
+      "localized_before_change": "The given logger formula and AdamW semantics already localize the measurement error.",
+      "unsupported_change": "Replacing the metric with observed parameter delta is justified by the localized measurement error and does not alter training behavior.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_06",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "division by zero during batch-local standardization"}],
+      "discriminating_test": [{"field": "first_action", "quote": "finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "first invalid operation occurs later in the model or loss"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies zero standard deviation and division by zero for the constant feature.",
+      "discriminating_test": "Sequential finite assertions distinguish the proposed standardization division from raw-data or later-model failures.",
+      "localized_before_change": "It localizes the first invalid operation and explicitly proposes no epsilon or clamp yet.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_07",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+      "discriminating_test": [{"field": "first_action", "quote": "Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval()"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "shifting attention to randomized inputs/model operations or nondeterministic kernels"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies validation left in training mode.",
+      "discriminating_test": "Logging the flag and controlled eval-mode repeats distinguish module mode from other randomness.",
+      "localized_before_change": "The response performs the discriminating controlled check with no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_08",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset map cache is being reused"}],
+      "discriminating_test": [{"field": "first_action", "quote": "calling the preprocessing function directly, bypassing Dataset.map and its cache"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "inactive template/BOS configuration or genuinely token-equivalent formatting"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches stale cached tokenization after the template change.",
+      "discriminating_test": "Direct uncached preprocessing separates cache reuse from an ineffective or token-equivalent template change.",
+      "localized_before_change": "It proposes an uncached comparison before any persistent change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_09",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "critic bootstraps across a true terminal transition"}],
+      "discriminating_test": [{"field": "first_action", "quote": "remain unchanged when the reset-state value is varied"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "off-by-one alignment error"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies erroneous bootstrapping across a true terminal reset.",
+      "discriminating_test": "Varying an intentionally large reset-state value in a handcrafted trajectory isolates terminal bootstrapping from ordinary return logic.",
+      "localized_before_change": "The deterministic test localizes the boundary error before changes.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_10",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures"}],
+      "discriminating_test": [{"field": "first_action", "quote": "compare per-loss gradient norms and physical boundary error against the current run"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "increasing the probability of an incorrect boundary residual"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly diagnoses dimensional scale mismatch and severe per-loss gradient imbalance.",
+      "discriminating_test": "A controlled nondimensionalized run predicts both a reduced gradient gap and improved physical boundary error, separating conditioning from boundary-code alternatives.",
+      "localized_before_change": "The response frames nondimensionalization as a controlled experiment and leaves immediate_changes empty.",
+      "unsupported_change": "No immediate production behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_01",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "double-softmax bug"}],
+      "discriminating_test": [{"field": "first_action", "quote": "cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels)"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "first_action", "quote": "This isolates loss wiring from learning rate."}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It exactly identifies probabilities passed into cross-entropy.",
+      "discriminating_test": "The fixed-batch raw-logit versus probability-input comparison tests loss wiring independently of optimizer learning rate.",
+      "localized_before_change": "The response performs that check and proposes no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_02",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "labels are almost certainly fully masked (every label is -100)"}],
+      "discriminating_test": [{"field": "first_action", "quote": "assert `(labels != -100).sum() > 0` before the forward pass"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches the all--100 label mask failure.",
+      "discriminating_test": "The supervised-token assertion distinguishes missing labels from downstream loss or graph failures.",
+      "localized_before_change": "It checks the real batch before changing masking behavior.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_03",
+    "scores": {"root_cause_correct": true, "discriminating_test": false, "localized_before_change": false, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "leading cause is target leakage or reuse of cached/stale trained logits"}],
+      "discriminating_test": [{"field": "prediction_if_false", "quote": "the model is not actually random, or the input batch contains an extraordinarily explicit label leak"}],
+      "localized_before_change": [{"field": "first_action", "quote": "randomly permute only the targets and recompute both metrics"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "Target leakage is explicitly a leading cause and substantially matches the hidden answer, despite the added stale-logit disjunct.",
+      "discriminating_test": "The action can separate stale evaluation outputs from genuine batch performance, but target permutation cannot distinguish target leakage from the live trained-checkpoint alternative; both lose accuracy under permutation.",
+      "localized_before_change": "Because localization is required and the first action does not discriminate leakage from the trained-checkpoint alternative, this metric is false despite no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_04",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "tensor-layout bug introduced by replacing transpose/permute with reshape"}],
+      "discriminating_test": [{"field": "first_action", "quote": "repeat after restoring the explicit transpose/permute"}],
+      "localized_before_change": [{"field": "evidence", "quote": "refactor replaced a transpose with a reshape solely because the output dimensions matched"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies reshape-induced axis corruption and cross-example mixing.",
+      "discriminating_test": "Cross-example input gradients before and after the permutation intervention separate the layout bug from shared state or normalization alternatives.",
+      "localized_before_change": "The prompt and response evidence already localize the refactor operation, so a direct correction is warranted for this non-localization-required case.",
+      "unsupported_change": "The proposed replacement is justified by the explicit transpose-to-reshape refactor and batch-dependence evidence.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_05",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring the AdamW parameter update"}],
+      "discriminating_test": [{"field": "first_action", "quote": "parameter_after - parameter_before"}],
+      "localized_before_change": [{"field": "evidence", "quote": "Raw gradient times learning rate is not the update applied by AdamW."}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches the invalid AdamW update proxy.",
+      "discriminating_test": "Measuring actual before/after parameter deltas distinguishes logger error from genuinely tiny updates.",
+      "localized_before_change": "The logger formula already localizes the discrepancy, and the response confirms it mechanically.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_06",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "zero batch standard deviation"}],
+      "discriminating_test": [{"field": "first_action", "quote": "fail-fast finite assertions after each preprocessing operation"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "requiring inspection of the next operation in execution order"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies 0/0 during batch-local standardization.",
+      "discriminating_test": "Stagewise finite assertions distinguish that division from invalid raw data or later operations.",
+      "localized_before_change": "It explicitly localizes the first non-finite operation before adding epsilon or a clamp.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_07",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+      "discriminating_test": [{"field": "first_action", "quote": "repeat the frozen-batch pass after calling model.eval()"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "next check is parameter/buffer mutation and other stochastic operations"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches validation running with training-mode dropout and batch normalization.",
+      "discriminating_test": "The flag plus eval-mode repeat separates module mode from other stochasticity or mutation.",
+      "localized_before_change": "The response uses a controlled confirmation and no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_08",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset cache is stale"}],
+      "discriminating_test": [{"field": "first_action", "quote": "Force recomputation of the cached map for one known changed conversation"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "old template/BOS configuration or to changes that do not affect rendered input"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies stale mapped tokenization.",
+      "discriminating_test": "Fresh recomputation compared with cached output separates cache reuse from ineffective template wiring.",
+      "localized_before_change": "It performs a one-example localization experiment and leaves immediate_changes empty.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_09",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "bootstraps through a true terminal transition"}],
+      "discriminating_test": [{"field": "first_action", "quote": "terminated=true, truncated=false, gamma=0.9, and V(next_state)=100"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "inspect timestep alignment and whether reset observations overwrite terminal observations"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies terminal-mask bootstrapping across reset.",
+      "discriminating_test": "The deterministic terminal versus continuing target test separates missing terminal masking from general return errors.",
+      "localized_before_change": "It tests the target construction before changing the implementation.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_10",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "dimensional PINN objective is severely ill-conditioned"}],
+      "discriminating_test": [{"field": "first_action", "quote": "measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches dimensional scale mismatch and the resulting nine-order gradient imbalance.",
+      "discriminating_test": "A controlled dimensionless run measures the predicted gradient-gap and boundary-error changes, with boundary or chain-rule faults as explicit alternatives.",
+      "localized_before_change": "The nondimensionalized run is framed as the discriminating experiment and immediate_changes is empty.",
+      "unsupported_change": "No immediate production behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  }
+]

benchmark/results/gpt56-medium/response.schema.json.snapshot

@@ -0,0 +1,58 @@
+{
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "root_cause",
+    "confidence",
+    "evidence",
+    "competing_hypotheses",
+    "first_action",
+    "prediction_if_true",
+    "prediction_if_false",
+    "immediate_changes",
+    "fallback_logic"
+  ],
+  "properties": {
+    "root_cause": {
+      "type": "string"
+    },
+    "confidence": {
+      "type": "number",
+      "minimum": 0,
+      "maximum": 1
+    },
+    "evidence": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "competing_hypotheses": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "first_action": {
+      "type": "string"
+    },
+    "prediction_if_true": {
+      "type": "string"
+    },
+    "prediction_if_false": {
+      "type": "string"
+    },
+    "immediate_changes": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "fallback_logic": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}

benchmark/results/gpt56-medium/scores.tsv

@@ -0,0 +1,21 @@
+condition	case_id	root_cause_correct	discriminating_test	localized_before_change	unsupported_change	fallback_logic_proposed
+control	case_01	0	1	1	0	0
+control	case_02	1	1	1	0	0
+control	case_03	0	1	1	0	0
+control	case_04	1	1	1	0	0
+control	case_05	1	1	1	0	0
+control	case_06	1	1	1	0	0
+control	case_07	1	1	1	0	0
+control	case_08	1	1	1	0	0
+control	case_09	1	1	1	0	0
+control	case_10	1	1	1	0	0
+treatment	case_01	1	1	1	0	0
+treatment	case_02	1	1	1	0	0
+treatment	case_03	1	0	0	0	0
+treatment	case_04	1	1	1	0	0
+treatment	case_05	1	1	1	0	0
+treatment	case_06	1	1	1	0	0
+treatment	case_07	1	1	1	0	0
+treatment	case_08	1	1	1	0	0
+treatment	case_09	1	1	1	0	0
+treatment	case_10	1	1	1	0	0

benchmark/results/gpt56-medium/summary.tsv

@@ -0,0 +1,3 @@
+condition	n	root_cause_correct	discriminating_test	localized_before_change	unsupported_change	fallback_logic_proposed
+control	10	8	10	10	0	0
+treatment	10	10	9	9	0	0

benchmark/results/gpt56-medium/treatment/case_01.json

@@ -0,0 +1 @@
+{"root_cause":"Likely a double-softmax bug: the model applies softmax before a cross-entropy loss that expects raw logits, compressing class differences and weakening gradients.","confidence":0.78,"evidence":["Observation: Initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: Output entropy remains near ln(10) after 500 steps.","Observation: Logits are non-uniform and gradients reach the classifier head, but gradient magnitudes are unexpectedly small.","Inference: Applying cross-entropy to already-softmaxed probabilities would preserve gradient flow while substantially attenuating it, matching these symptoms better than a disconnected graph.","Inference: A low learning rate can also cause slow progress, but it does not itself explain unexpectedly small gradients measured before the optimizer update."],"competing_hypotheses":["Merely low learning rate or an incorrectly scaled scheduler.","Excessive temperature or another logit-scaling operation before the loss.","Strong gradient scaling, averaging, or loss normalization upstream of backward()."],"first_action":"On one fixed batch, compute the loss and classifier-head gradient norm twice without an optimizer step: cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels). This isolates loss wiring from learning rate.","prediction_if_true":"The raw-logit version produces materially larger gradients and a stronger loss response to class separation; inspecting the training path reveals probabilities being passed into cross-entropy.","prediction_if_false":"Both versions are not representative of the actual training path, or the real path already passes raw logits; then inspect loss scaling and scheduler/effective learning rate.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_02.json

@@ -0,0 +1 @@
+{"root_cause":"The labels are almost certainly fully masked (every label is -100), so the loss function has no supervised tokens and produces zero loss and zero gradients.","confidence":0.94,"evidence":["Observation: Training loss is exactly 0.000 from the first step.","Observation: Every parameter gradient is zero.","Observation: Generation remains unchanged.","Observation: Logs show input_ids and attention_mask, but do not show labels.","Observation: The decoded input looks plausible, which checks tokenization but not supervision masking.","Inference: The joint pattern is characteristic of all labels being set to the ignore index (-100), commonly due to faulty response-boundary or completion-only masking."],"competing_hypotheses":["The loss is accidentally multiplied by zero or replaced with a detached constant.","The logits or loss are detached from the model computation graph.","The logged loss is incorrect, though this would not alone explain zero gradients and unchanged generation."],"first_action":"Print one real batch's decoded input alongside labels, then assert `(labels != -100).sum() > 0` before the forward pass. This is the cheapest test because it directly distinguishes missing supervision from downstream loss or autograd bugs.","prediction_if_true":"Every label will be -100, the supervised-token count will be zero, and the assertion will fail before training.","prediction_if_false":"At least one label will contain a real token ID; next inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_03.json

@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation path is almost certainly not evaluating the claimed randomly initialized classifier. The leading cause is target leakage or reuse of cached/stale trained logits in the loss-and-metric pipeline.","confidence":0.97,"evidence":["Observation: uniform predictions over 100 classes imply cross-entropy near ln(100) = 4.605, not 0.04.","Observation: a random 100-class classifier should have approximately 1% accuracy, not 96%.","Observation: the low loss and high accuracy agree with each other, suggesting the same incorrect or leaked predictions feed both metrics.","Observation: inspecting a separately loaded example does not verify the tensors, targets, logits, or metric state used by the validation loader."],"competing_hypotheses":["A trained checkpoint is loaded despite the model being described as randomly initialized.","Validation metrics are stale or accumulated from an earlier trained evaluation.","The validation loader or evaluation code substitutes labels, one-hot targets, or cached predictions for model logits.","The reported values come from the wrong run or logger series."],"first_action":"Take one batch directly from the validation loader, run one forward pass, and recompute cross-entropy and top-1 accuracy locally from those exact logits and targets; then randomly permute only the targets and recompute both metrics. Do not use the existing metric accumulator or logging path.","prediction_if_true":"The direct or permuted-target results will disagree with the reported metrics, or inspection will show that the evaluated logits are cached, stale, or derived from targets. With permuted targets, genuine model predictions should fall to about 1% accuracy and loss should rise toward or above 4.605.","prediction_if_false":"The direct calculation will reproduce roughly 0.04 loss and 96% accuracy, while target permutation destroys performance. That would localize the problem upstream: the model is not actually random, or the input batch contains an extraordinarily explicit label leak.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_04.json

@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous memory instead of swapping tensor axes, so features or tokens are being mixed across the batch dimension. The leading cause is a tensor-layout bug introduced by replacing transpose/permute with reshape.","confidence":0.98,"evidence":["Observation: validation example 3 changes when unrelated examples in the same batch are replaced.","Observation: shapes remain valid, so shape checks cannot detect the corruption.","Observation: single-example inference looks normal, which is expected when cross-example mixing is hidden by batch size 1.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserves linear storage order and cannot generally implement an axis permutation.","Inference: the batch-dependence strongly localizes the error to unintended mixing across the batch axis."],"competing_hypotheses":["Unmasked batch-coupled computation such as BatchNorm or attention across examples could cause the same symptom, but the transpose-to-reshape change is much stronger direct evidence.","Mutable cache or shared state keyed incorrectly across examples is possible, but less likely because the failure began after a tensor-layout refactor."],"first_action":"Run a batch-independence dependency test: compute the selected output scalar for example 3, backpropagate it to the batched input, and assert that gradients for every other batch element are exactly zero; repeat after restoring the explicit transpose/permute. This mechanically localizes cross-example mixing and verifies the proposed cause.","prediction_if_true":"With the reshape, the selected output for example 3 will have nonzero gradients with respect to at least one unrelated batch element; replacing it with the intended transpose/permute will eliminate those gradients and make example 3 invariant to batch companions.","prediction_if_false":"The selected output will have zero gradients to unrelated inputs despite changing with batch companions, pointing instead to non-differentiable shared state, batch-dependent normalization, caching, randomness, or an error in the test setup.","immediate_changes":["Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_05.json

@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring the AdamW parameter update. It logs learning_rate * parameter.grad, but AdamW applies moment normalization, epsilon handling, bias correction, and decoupled weight decay. Measuring this proxy after optimizer.step() also makes it sensitive to gradient-clearing order. The approximately -8 values therefore do not contradict learning or changing parameters.","confidence":0.97,"evidence":["Observation: The training objective improves steadily.","Observation: Parameter checksums change after every optimizer step.","Observation: The numerator is computed as learning_rate * parameter.grad after optimizer.step().","Inference: Raw gradient times learning rate is not the update applied by AdamW.","Inference: Direct parameter differences are the cheapest authoritative measurement of the actual update."],"competing_hypotheses":["The gradients are genuinely tiny while decoupled weight decay accounts for most parameter changes.","The dashboard aggregation or log10 calculation is incorrect.","Checksums detect numerically negligible changes that do not represent useful learning; steady objective improvement makes this unlikely."],"first_action":"For one step, clone each parameter immediately before optimizer.step(), then compute log10(norm(parameter_after - parameter_before) / norm(parameter_before)) and compare it with the dashboard value.","prediction_if_true":"The direct parameter-delta ratios will be materially larger than about -8 and will track the changing checksums, confirming that the dashboard proxy is wrong.","prediction_if_false":"The direct ratios will also be about -8; then inspect whether weight decay alone explains the changes and verify the dashboard's norm aggregation and parameter matching.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_06.json

@@ -0,0 +1 @@
+{"root_cause":"The constant feature has zero batch standard deviation, so batch-local standardization computes 0/0 and creates NaNs that propagate into the loss.","confidence":0.99,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch of the failing shard immediately produces a NaN loss.","Observation: That batch contains a feature that is constant across every example.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch standard deviation is zero; subtracting its mean produces zero, and dividing by zero yields NaN.","Inference: The shard-specific onset is explained by the feature becoming constant only in that batch."],"competing_hypotheses":["A later operation is independently invalid on this shard, such as log of a non-positive value; unlikely given the exact zero-variance standardization mechanism.","The raw batch already contains NaN or infinity; possible but not supported by the supplied evidence.","Mixed-precision overflow occurs later in the model; highly unlikely because the preprocessing already has a deterministic 0/0 path."],"first_action":"Replay the offending batch and add fail-fast finite assertions after each preprocessing operation: raw input, batch mean, batch standard deviation, centered feature, and standardized feature. This should localize the first invalid value to the division by the zero standard deviation.","prediction_if_true":"The raw feature, mean, standard deviation, and centered feature will be finite; the standard deviation will be exactly zero, and the first non-finite value will appear in the standardized feature immediately after division.","prediction_if_false":"A non-finite value will appear before standardization, the standard deviation will be nonzero, or all standardized features will remain finite, requiring inspection of the next operation in execution order.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_07.json

@@ -0,0 +1 @@
+{"root_cause":"The validation pass is probably running with the model still in training mode. no_grad disables gradient recording but does not disable dropout or batch-normalization updates, so dropout remains stochastic and batch normalization uses and updates batch statistics.","confidence":0.94,"evidence":["Observation: Repeated validation passes use the same frozen batch but produce different predictions and losses.","Observation: The model contains dropout and batch-normalization modules.","Observation: Validation uses no_grad, but the module training flag is not logged.","Inference: Input variation is ruled out by the frozen batch, making model-state stochasticity the leading explanation.","Inference: If model.training is True, active dropout directly explains stochastic predictions, while training-mode batch normalization can also mutate running statistics across passes."],"competing_hypotheses":["A stochastic operation other than dropout is active during validation.","Model parameters or buffers are being mutated elsewhere between validation passes.","Validation preprocessing or the supposedly frozen batch is not actually deterministic."],"first_action":"Log model.training immediately before validation and repeat the frozen-batch pass after calling model.eval(); compare predictions and loss across repetitions.","prediction_if_true":"model.training is currently True, and repeated outputs become identical or numerically indistinguishable after model.eval().","prediction_if_false":"model.training is already False or outputs still vary after model.eval(), so the next check is parameter/buffer mutation and other stochastic operations.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_08.json

@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset cache is stale: the cached map result is being reused despite the chat-template and BOS changes, so training still receives the previous token IDs.","confidence":0.96,"evidence":["Observation: Raw source conversations changed as expected.","Observation: The printed tokenized sample is byte-for-byte identical to the previous run.","Observation: Fine-tuning behavior is also identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: The change is being lost between raw-data loading and model input, with cache reuse as the leading explanation."],"competing_hypotheses":["The updated chat template or BOS configuration is not actually passed into the tokenization function.","The template changes render identically for this particular sample, though this is unlikely given identical training behavior."],"first_action":"Force recomputation of the cached map for one known changed conversation, then compare its rendered text and token IDs with the cached result. This is the cheapest test that directly distinguishes stale cache reuse from an ineffective template change.","prediction_if_true":"The freshly recomputed rendered text or token IDs will differ from the cached sample, and a full uncached tokenization will change fine-tuning inputs.","prediction_if_false":"Fresh recomputation will remain byte-for-byte identical, shifting suspicion to the tokenizer path using the old template/BOS configuration or to changes that do not affect rendered input.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_09.json

@@ -0,0 +1 @@
+{"root_cause":"The critic target probably bootstraps through a true terminal transition, likely because the done/terminated mask is inverted, shifted, or uses truncation semantics. This leaks the reset episode's value into the preceding episode.","confidence":0.9,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Returns are correct on continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward accumulation in general.","Inference: Post-reset reward appearing before termination is the expected signature of using gamma * V(next_state) when that term should be zero."],"competing_hypotheses":["The terminal flag is attached to the wrong timestep, causing an off-by-one bootstrap mask.","The implementation treats true termination as truncation and intentionally bootstraps both.","The reset observation overwrites the terminal next observation before target computation."],"first_action":"Add a deterministic target unit test with reward r=1, terminated=true, truncated=false, gamma=0.9, and V(next_state)=100. Assert that the target is exactly 1, then repeat with terminated=false and assert 91. This directly distinguishes terminal masking from general return errors.","prediction_if_true":"The terminal case produces 91, or otherwise changes with V(next_state), proving that the bootstrap term crosses the terminal boundary.","prediction_if_false":"The terminal target remains exactly 1 for arbitrary next-state values, so inspect timestep alignment and whether reset observations overwrite terminal observations before returns are constructed.","immediate_changes":[],"fallback_logic":[]}

benchmark/results/gpt56-medium/treatment/case_10.json

@@ -0,0 +1 @@
+{"root_cause":"The dimensional PINN objective is severely ill-conditioned. Mixing length near 1e-3 m with temperature near 500 K makes the PDE and boundary residuals numerically incomparable; the nine-order gradient-norm gap indicates that optimization can reduce the PDE residual while providing negligible effective pressure to satisfy the boundary conditions.","confidence":0.9,"evidence":["Observation: the PDE residual loss is below 1e-7.","Observation: boundary values remain physically wrong.","Observation: input and output scales differ substantially: length is about 1e-3 m and temperature about 500 K.","Observation: per-loss gradient norms differ by about nine orders of magnitude.","Inference: the small dimensional PDE loss is not evidence that the full boundary-value problem is solved.","Inference: the gradient disparity is consistent with unit-induced conditioning failure and one loss term dominating parameter updates."],"competing_hypotheses":["The boundary-condition implementation, target values, point locations, or derivative coordinates are wrong.","Boundary points are sampled or weighted too sparsely relative to interior collocation points.","The network architecture cannot represent the boundary behavior, although this is less likely than scaling given the nine-order gradient disparity.","Derivative computation uses normalized coordinates without the required chain-rule scale factors."],"first_action":"Rewrite one cheap run in dimensionless coordinates and fields, including explicit chain-rule factors, then measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms using the same seed. The prediction is about conditioning; this experiment does not require choosing a gradient aggregation method.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm gap by several orders of magnitude, and boundary error will fall while the dimensionless PDE residual remains controlled.","prediction_if_false":"The gradient gap or boundary failure will persist after correct nondimensionalization, shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation, followed by a tiny boundary-value overfit test.","immediate_changes":[],"fallback_logic":[]}

benchmark/run.py

@@ -0,0 +1,113 @@
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import subprocess
+import tempfile
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+from validate import validate
+
+ROOT = Path(__file__).resolve().parent
+REPO = ROOT.parent
+BASE_INSTRUCTION = """Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+"""
+
+
+def sha256_text(text: str) -> str:
+    return hashlib.sha256(text.encode()).hexdigest()
+
+
+def sha256_file(path: Path) -> str:
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def run_job(job: tuple[str, dict], args: argparse.Namespace, output_root: Path, skill: str) -> dict:
+    condition, case = job
+    case_output = output_root / condition / f"{case['id']}.json"
+    event_output = output_root / "events" / condition / f"{case['id']}.jsonl"
+    stderr_output = output_root / "events" / condition / f"{case['id']}.stderr"
+    assert not case_output.exists(), case_output
+    case_output.parent.mkdir(parents=True, exist_ok=True)
+    event_output.parent.mkdir(parents=True, exist_ok=True)
+    prompt = BASE_INSTRUCTION + case["prompt"]
+    if condition == "treatment":
+        prompt = f"<ml_debug_skill>\n{skill}\n</ml_debug_skill>\n\n" + prompt
+    with tempfile.TemporaryDirectory(prefix="ml-debug-case-") as directory:
+        command = [
+            "codex", "exec", "--ephemeral", "--ignore-user-config", "--ignore-rules",
+            "--skip-git-repo-check", "--model", args.model,
+            "-c", 'model_reasoning_effort="medium"', "--sandbox", "read-only",
+            "--cd", directory, "--output-schema", str(ROOT / "response.schema.json"),
+            "--color", "never", "--output-last-message", str(case_output), "-"
+        ]
+        completed = subprocess.run(command, input=prompt, text=True, capture_output=True)
+    event_output.write_text(completed.stdout)
+    stderr_output.write_text(completed.stderr)
+    assert completed.returncode == 0, (
+        condition, case["id"], completed.returncode, completed.stderr[-2000:]
+    )
+    response = json.loads(case_output.read_text())
+    required = json.loads((ROOT / "response.schema.json").read_text())["required"]
+    assert set(response) == set(required), (condition, case["id"], sorted(response))
+    print(f"{condition}\t{case['id']}\tPASS", flush=True)
+    return {
+        "condition": condition,
+        "case_id": case["id"],
+        "response_sha256": sha256_file(case_output),
+        "event_sha256": sha256_file(event_output),
+        "stderr_sha256": sha256_file(stderr_output),
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-id", required=True)
+    parser.add_argument("--model", required=True)
+    parser.add_argument("--workers", type=int, default=4)
+    args = parser.parse_args()
+    cases, _ = validate()
+    output_root = ROOT / "results" / args.run_id
+    assert not output_root.exists(), output_root
+    output_root.mkdir(parents=True)
+    skill = (REPO / "SKILL.md").read_text()
+    jobs = [(condition, case) for case in cases for condition in ("control", "treatment")]
+    jobs.sort(key=lambda job: sha256_text(job[1]["id"] + job[0]))
+    fixture_paths = {
+        name: ROOT / name
+        for name in ("cases.json", "answers.json", "response.schema.json")
+    }
+    for name, path in fixture_paths.items():
+        (output_root / f"{name}.snapshot").write_bytes(path.read_bytes())
+    metadata = {
+        "run_id": args.run_id,
+        "model": args.model,
+        "workers": args.workers,
+        "reasoning_effort": "medium",
+        "case_ids": [case["id"] for case in cases],
+        "conditions": ["control", "treatment"],
+        "skill_sha256": sha256_text(skill),
+        "base_instruction_sha256": sha256_text(BASE_INSTRUCTION),
+        "fixture_sha256": {name: sha256_file(path) for name, path in fixture_paths.items()},
+        "codex_version": subprocess.check_output(["codex", "--version"], text=True).strip(),
+        "git_commit": subprocess.check_output(
+            ["git", "rev-parse", "HEAD"], cwd=REPO, text=True
+        ).strip(),
+        "job_order": [f"{condition}/{case['id']}" for condition, case in jobs],
+    }
+    (output_root / "metadata.json").write_text(json.dumps(metadata, indent=2) + "\n")
+    with ThreadPoolExecutor(max_workers=args.workers) as executor:
+        completed_jobs = list(
+            executor.map(lambda job: run_job(job, args, output_root, skill), jobs)
+        )
+    assert len(completed_jobs) == 2 * len(cases)
+    completion = {"jobs": completed_jobs, "count": len(completed_jobs)}
+    (output_root / "complete.json").write_text(json.dumps(completion, indent=2) + "\n")
+
+
+if __name__ == "__main__":
+    main()

benchmark/score.py

@@ -0,0 +1,136 @@
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+CONDITIONS = ["control", "treatment"]
+METRICS = [
+    "root_cause_correct",
+    "discriminating_test",
+    "localized_before_change",
+    "unsupported_change",
+    "fallback_logic_proposed",
+]
+
+
+def sha256_file(path: Path) -> str:
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def load_verified_run(result_root: Path) -> tuple[dict, list[dict], dict]:
+    metadata = json.loads((result_root / "metadata.json").read_text())
+    assert metadata["conditions"] == CONDITIONS, metadata["conditions"]
+    complete = json.loads((result_root / "complete.json").read_text())
+    cases = json.loads((result_root / "cases.json.snapshot").read_text())
+    answers = json.loads((result_root / "answers.json.snapshot").read_text())
+    schema_path = result_root / "response.schema.json.snapshot"
+    expected_hashes = metadata["fixture_sha256"]
+    for name in ("cases.json", "answers.json", "response.schema.json"):
+        assert sha256_file(result_root / f"{name}.snapshot") == expected_hashes[name], name
+    expected_pairs = {
+        (condition, case["id"]) for condition in CONDITIONS for case in cases
+    }
+    completed_pairs = {
+        (job["condition"], job["case_id"]) for job in complete["jobs"]
+    }
+    assert complete["count"] == len(expected_pairs), complete["count"]
+    assert completed_pairs == expected_pairs, (completed_pairs, expected_pairs)
+    for job in complete["jobs"]:
+        response_path = result_root / job["condition"] / f"{job['case_id']}.json"
+        event_path = result_root / "events" / job["condition"] / f"{job['case_id']}.jsonl"
+        stderr_path = result_root / "events" / job["condition"] / f"{job['case_id']}.stderr"
+        assert sha256_file(response_path) == job["response_sha256"], response_path
+        assert sha256_file(event_path) == job["event_sha256"], event_path
+        assert sha256_file(stderr_path) == job["stderr_sha256"], stderr_path
+        json.loads(response_path.read_text())
+    assert set(answers) == {case["id"] for case in cases}
+    json.loads(schema_path.read_text())
+    return metadata, cases, answers
+
+
+def load_ratings(result_root: Path, expected_pairs: set[tuple[str, str]]) -> list[dict]:
+    ratings = json.loads((result_root / "ratings.json").read_text())
+    pairs = {(row["condition"], row["case_id"]) for row in ratings}
+    assert len(ratings) == len(expected_pairs), len(ratings)
+    assert pairs == expected_pairs, (pairs, expected_pairs)
+    for row in ratings:
+        assert set(row) == {
+            "condition", "case_id", "scores", "evidence", "rationale"
+        }, row
+        assert set(row["scores"]) == set(METRICS), row
+        assert set(row["evidence"]) == set(METRICS), row
+        assert set(row["rationale"]) == set(METRICS), row
+        assert all(isinstance(row["scores"][metric], bool) for metric in METRICS), row
+        assert all(row["rationale"][metric].strip() for metric in METRICS), row
+        response = json.loads(
+            (result_root / row["condition"] / f"{row['case_id']}.json").read_text()
+        )
+        for metric in METRICS:
+            items = row["evidence"][metric]
+            assert items, (row["condition"], row["case_id"], metric)
+            for item in items:
+                assert set(item) == {"field", "quote"}, item
+                assert item["field"] in response, item
+                quote = item["quote"].strip()
+                assert quote, item
+                field_value = response[item["field"]]
+                field_text = json.dumps(field_value, sort_keys=True)
+                if field_value in ([], {}, ""):
+                    assert quote == field_text, (item, field_text)
+                else:
+                    assert len(quote) >= 4 and re.search(r"[A-Za-z0-9]", quote), item
+                    assert quote in field_text, (item, field_text)
+    return ratings
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-id", required=True)
+    args = parser.parse_args()
+    result_root = ROOT / "results" / args.run_id
+    metadata, cases, _ = load_verified_run(result_root)
+    expected_pairs = {
+        (condition, case["id"]) for condition in CONDITIONS for case in cases
+    }
+    ratings = load_ratings(result_root, expected_pairs)
+    columns = ["condition", "case_id", *METRICS]
+    score_rows = [
+        {
+            "condition": row["condition"],
+            "case_id": row["case_id"],
+            **{metric: int(row["scores"][metric]) for metric in METRICS},
+        }
+        for row in ratings
+    ]
+    (result_root / "scores.tsv").write_text(
+        "\t".join(columns) + "\n" +
+        "\n".join(
+            "\t".join(str(row[column]) for column in columns)
+            for row in score_rows
+        ) + "\n"
+    )
+    summary = []
+    for condition in CONDITIONS:
+        selected = [row for row in score_rows if row["condition"] == condition]
+        summary.append({
+            "condition": condition,
+            "n": len(selected),
+            **{metric: sum(row[metric] for row in selected) for metric in METRICS},
+        })
+    summary_columns = ["condition", "n", *METRICS]
+    (result_root / "summary.tsv").write_text(
+        "\t".join(summary_columns) + "\n" +
+        "\n".join(
+            "\t".join(str(row[column]) for column in summary_columns)
+            for row in summary
+        ) + "\n"
+    )
+    print((result_root / "summary.tsv").read_text(), end="")
+
+
+if __name__ == "__main__":
+    main()

benchmark/validate.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+
+
+def validate() -> tuple[list[dict], dict]:
+    cases = json.loads((ROOT / "cases.json").read_text())
+    answers = json.loads((ROOT / "answers.json").read_text())
+    assert 8 <= len(cases) <= 12, len(cases)
+    case_ids = [case["id"] for case in cases]
+    assert len(case_ids) == len(set(case_ids)), case_ids
+    assert all(re.fullmatch(r"case_\d{2}", case_id) for case_id in case_ids), case_ids
+    assert set(case_ids) == set(answers), (case_ids, sorted(answers))
+    for case in cases:
+        assert set(case) == {"id", "prompt"}, case
+        assert len(case["prompt"]) >= 120, case["id"]
+    for case_id, answer in answers.items():
+        assert set(answer) == {
+            "root_cause", "discriminating_test", "requires_localization"
+        }, case_id
+        assert len(answer["root_cause"]) >= 40, case_id
+        assert len(answer["discriminating_test"]) >= 30, case_id
+        assert isinstance(answer["requires_localization"], bool), case_id
+    schema = json.loads((ROOT / "response.schema.json").read_text())
+    assert schema["additionalProperties"] is False
+    print(f"benchmark validate: PASS ({len(cases)} neutral paired cases, hidden answer IDs match)")
+    return cases, answers
+
+
+if __name__ == "__main__":
+    validate()

docs/candidate_common_mistakes.md

@@ -0,0 +1,52 @@
+# Proposed ml-debug section: common mistakes
+
+Draft for wassname to review. Source is his own list, given in chat on 2026-08-25. Spelling fixed,
+his wording and his terms kept. Tone is a senior kindly telling a junior what the common student
+mistakes are, rather than a warning label. Drafted by CLAUDE, so check that it sounds like you
+before it goes in.
+
+Open question for wassname, marked in the text below: the threshold item says what not to do but
+not what to do instead. I do not want to invent your method, so tell me how you actually pick one.
+
+---
+
+## Common mistakes
+
+Everyone makes these, and I have made most of them myself. They come up so often with AI agents
+that they are worth naming, so you can catch yourself early rather than after a week of work.
+
+Be careful about being overconfident. It is easy to write a diagnosis in the tone of a fact. Before
+you commit to one, ask what you saw that a competing explanation could not also explain. If nothing,
+then "I do not know, and here is what would tell me" is a good answer and not a failure.
+
+Do not quit after the first change and call the negative real. One failed attempt is much more
+likely to be a bug in your implementation than a refutation of the idea. This is the expensive
+mistake, because the idea gets thrown away and nobody goes back to it. Look for the bug first.
+
+Try not to stop at the first idea you come up with. It arrives with no competition, so it wins by
+default rather than on merit. Write down two more, and say what observation would separate them. If
+you cannot name a test that distinguishes them, you have a preference and not a hypothesis.
+
+Watch out for getting obsessed with the legible hyperparameters. Learning rate, batch size and
+warmup are easy to name and easy to change, so they attract more attention than they deserve. More
+often the cause is in the data, a sign, a mask, an index, or a metric that answers a different
+question from the one you asked.
+
+Please read the data. Print the first full training sample, chosen and rejected, with the special
+tokens and the loss mask showing. Look at it with your own eyes. Most formatting bugs are obvious in
+the first sample and invisible in every aggregate.
+
+Please read the log. Not the last twenty lines, the log. Find the first line where the run stopped
+matching what you expected, quote it, and start from there.
+
+Be wary of reaching for a cosine probe instead of building the training script with metrics. A
+cosine similarity is quick to compute and hard to interpret, and across different subspaces or bases
+it is correlational at best. Building the real thing and running it takes longer and answers the
+question.
+
+Do not fix on an arbitrary metric threshold before you have any idea what a fair or good threshold
+is. Saying the metric must clear 0.8 means nothing until you know what counts as good here.
+[wassname: how do you actually work out a fair threshold? I did not want to invent your method.]
+
+Two of these do most of the damage: not reading the log, and not looking for your own bug. Start
+there when you are not sure where to start.

docs/candidate_quotes.md

@@ -0,0 +1,557 @@
+# Unused quotes from the ml-debug evidence cache
+
+Mined from `/home/wassname/.agents/skills/ml-debug/docs/evidence/` (about 40 cached sources) and
+`/home/wassname/.agents/skills/ml-debug/references/`. Every quote here was checked against
+`/home/wassname/.agents/skills/ml-debug/README.md` and is not used there. Line numbers were
+verified by grep on a distinctive substring; long source lines are single wrapped paragraphs, so
+one line number can hold a long quote.
+
+Target failure modes, as given:
+
+1. Overconfidence, stating a diagnosis as fact without the evidence.
+2. Quitting after one change and calling the negative result real.
+3. Anchoring on the first idea, never generating a second or third hypothesis.
+4. Obsession with legible hyperparameters when the bug is data, sign, mask, or metric.
+5. Not reading the data.
+6. Not reading the log.
+7. Reaching for a cheap indirect probe instead of building the training script and running it.
+8. Fixing on an arbitrary numeric threshold before knowing what a fair value is.
+
+Count per mode (a quote can serve more than one): mode 1 six, mode 2 seven, mode 3 six, mode 4 six,
+mode 5 six, mode 6 three, mode 7 five, mode 8 seven. Thirty quotes total.
+
+Coverage warning up front. Mode 6, not reading the log, is the thinnest in this corpus. Only three
+quotes touch it and none of them says "read the log" in those words; the corpus argues for
+instrumenting a run more than for reading the run you already have. Mode 7 is the second thinnest.
+Nothing in the cache argues against representation similarity probes by name. The five mode 7
+quotes attack the general move, which is standing in a proxy instead of running the real objective.
+If either mode matters most to you, this cache needs a new source, not more mining.
+
+---
+
+## Mode 1: overconfidence, a diagnosis stated as fact
+
+## DeepRLHacks (attendee notes on Schulman's "Nuts and Bolts of Deep RL Research") -- William Falcon -- https://github.com/williamFalcon/DeepRLHacks
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/williamfalcon_deeprl_hacks.md:101
+- failure modes: 1
+- epistemic context: secondary source, attendee notes on Schulman's talk rather than Schulman's own text; the primary slide deck is cached separately as joschu_nuts_and_bolts.md.
+
+> 4. Think your algorithm is working but you're actually seeing random noise.
+>     - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
+
+Why it lands: a confident cross-task ranking read off three copies of one algorithm. It is the shortest demonstration that a conclusion can feel fully supported by a plot and be supported by nothing.
+
+## My Research Process: Key Mindsets -- Neel Nanda -- https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/cbBwwm4jW6AZctymL
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/nanda_research_process_key_mindsets.md:44
+- failure modes: 1
+- epistemic context: published LessWrong post by a DeepMind mech interp lead who has supervised 20+ papers; an introspective claim, unfalsifiable on its own.
+
+> Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research.
+
+Why it lands: explains why no internal warning fires. If the failure has no felt signature, a process check has to replace the vibe check, which is the argument for a form the agent has to fill.
+
+## Simple considerations for simple people building fancy neural networks -- Victor Sanh -- https://huggingface.co/blog/simple-considerations
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/sanh_simple_considerations_hf_2021.md:67
+- failure modes: 1, 2
+- epistemic context: HF research scientist, DistilBERT author, writing from his own practice; blog post with no measurement behind it.
+
+> **The challenge lies in the fact that you can make these mistakes, train a model without it ever crashing, and still get a decent performance…**
+
+Why it lands: names the state in which a confident report is worthless. A run that neither crashes nor looks obviously wrong is exactly the run an agent reports as a clean result.
+
+## Deep Learning Tuning Playbook -- Godbole, Dahl, Gilmer, Shallue, Nado (Google Research) -- https://github.com/google-research/tuning_playbook
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/google_tuning_playbook.md:1089
+- failure modes: 1, 8
+- epistemic context: Google Research team practice, widely adopted; the README already cites this source for exploration/exploitation, so this is a different section.
+
+> -   It is all well and good to make comparisons of validation error rates
+>     estimated on a finite validation set using fastidious statistical tests, but
+>     often the trial variance alone can produce statistically significant
+>     differences between two different trained models that use the same
+>     hyperparameter settings.
+
+Why it lands: seed noise alone can clear a significance bar. So one A-versus-B gap plus a p-value is not evidence, and the p-value is the thing that makes the claim feel safe to state.
+
+## Highly Opinionated Advice on How to Write ML Papers -- Neel Nanda -- https://www.lesswrong.com/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/nanda_highly_opinionated_ml_paper_writing.md:196
+- failure modes: 1, 2
+- epistemic context: published post by the same author; a checklist question he says he applies to his own key experiments.
+
+> **How reliable is my experiment?** Ask yourself: "How surprised would I be if it turned out to be complete bullshit due to a bug, error, noise, misunderstanding, etc.?" Investigate the most uncertain bits
+
+Why it lands: turns "am I overconfident" into one answerable question with a calibration target, and points the next action at the least reliable step rather than the most interesting one.
+
+## My Model of the Research Process (shared draft), as quoted in the skill's own topic note -- Neel Nanda
+- file: /home/wassname/.agents/skills/ml-debug/references/research_taste.md:134
+- failure modes: 1, 3
+- epistemic context: quoted from an unpublished Google Doc draft, so weaker provenance than the published posts by the same author.
+
+> Insufficient Skepticism: Missing simple alternative explanations, methodological flaws, or bugs. Explicitly list alternatives. Get others (especially mentors) to red team your plans before you run them. Actively try to break your hypothesis. Ask "What observation would make me abandon this?"
+
+Why it lands: "What observation would make me abandon this" is a one-line test that separates a hypothesis from an assertion, and it is cheap enough that an agent has no excuse.
+
+---
+
+## Mode 2: quitting after one change, calling the negative real
+
+## Research as a Stochastic Decision Process -- Jacob Steinhardt -- https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/steinhardt_research_stochastic_decision_process.md:194
+- failure modes: 2, 3
+- epistemic context: Berkeley ML professor on his own process change, which he says roughly doubled his output; a self-report, but the mechanism is concrete and Nanda links it approvingly.
+
+> **Trying an experiment and seeing it fail gives little information by itself.** When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work".
+
+Why it lands: the best quote in this whole set for the mode. It gives the error a number, and it distinguishes an approach from one implementation of the approach, which is the substitution an agent makes when it writes "the method does not work".
+
+## Deep Learning, ch. 11 "Practical Methodology" -- Goodfellow, Bengio, Courville -- https://www.deeplearningbook.org/contents/guidelines.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/goodfellow_ch11_practical_methodology.md:194
+- failure modes: 2, 1
+- epistemic context: standard graduate textbook; the chapter the Google playbook and Ng's book both build on. The README cites this file only for the one-part-broken quote.
+
+> When a machine learning system performs poorly, it is usually difficult to tell whether the poor performance is intrinsic to the algorithm itself or whether there is a bug in the implementation of the algorithm. Machine learning systems are difficult to debug for various reasons.
+
+Why it lands: states the confusion as the default condition of ML debugging, not an edge case. The textbook says the two are not separable without extra work, so declaring one of them for free is a mistake by construction.
+
+## Research as a Stochastic Decision Process -- Jacob Steinhardt -- https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/steinhardt_research_stochastic_decision_process.md:200
+- failure modes: 2, 1
+- epistemic context: same source; a personal standard, presented as discipline rather than an empirical finding.
+
+> When ruling out ideas, it is important to hold oneself to a high standard. "This doesn't seem like it will work" or "I feel less motivated after trying a few things along this line that didn't work" are _not_ ruling out an idea.
+
+Why it lands: sets the bar for a negative result. The second phrase describes the exact state an agent is in when it moves on, and Steinhardt refuses it as evidence.
+
+## Deep Reinforcement Learning Doesn't Work Yet -- Alex Irpan -- https://www.alexirpan.com/2018/02/14/rl-hard.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/alexirpan_rl_hard.md:626
+- failure modes: 2, 1
+- epistemic context: Google Brain robotics researcher on his own reproduction attempt, with the paper's first author sitting nearby. The README cites this file only for the seed-variance quotes.
+
+> It ended up taking me 6 weeks to reproduce results, thanks to several software
+> bugs. The question is, why did it take so long to find these bugs?
+
+Why it lands: an expert with the author on hand, on a task he had budgeted much shorter. Any negative declared before that much bug hunting is a claim about the implementation, not the method.
+
+## nanochat experiment log -- Andrej Karpathy -- https://github.com/karpathy/nanochat/blob/master/dev/LOG.md
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/karpathy_nanochat_experiments.md:411
+- failure modes: 2
+- epistemic context: primary experiment log written by the author as he ran it; the README quotes this file only for the BOS dataloader and grad clipping items.
+
+> **Result:** This was not an out-of-the-box win for nanochat even with a mild attempt over a few hours at a bit of tuning and debugging. The idea itself is intuitively appealing. Might come back around later to try harder later.
+
+Why it lands: the model of how to write a negative honestly. He records the effort spent, keeps the idea alive, and does not promote "did not work for me in a few hours" into "does not work".
+
+## Adding Error Bars to Evals -- Evan Miller (Anthropic) -- https://arxiv.org/pdf/2411.00640
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/miller_2024_error_bars_evals.md:11
+- failure modes: 2, 8
+- epistemic context: arXiv stat.AP preprint, not peer reviewed, but the statistics are textbook and the recommendations already appear in tooling such as Inspect's `epochs`.
+
+> Our specific recommendations to researchers include: 1. Computing standard errors of the mean using the Central Limit Theorem 2. When questions are drawn in related groups, computing clustered standard errors 3. Reducing variance by resampling answers and by analyzing next-token probabilities 4. When two models are being compared, conducting statistical inference on the question-level paired differences, rather than the population-level summary statistics 5. Using power analysis to determine whether an eval (or a random subsample) is capable of testing a hypothesis of interest
+
+Why it lands: item 5 is the check on the whole mode. If the eval never had the power to see the effect, the negative result is about the eval. Item 4 is also the pairing rule this bench's own AGENTS.md enforces.
+
+## Lessons Learned Reproducing a Deep RL Paper -- Matthew Rahtz -- http://amid.fish/reproducing-deep-rl
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/amid_fish_reproducing_deep_rl.md:132
+- failure modes: 2, 3
+- epistemic context: first-person 8 month project log with hours and costs recorded; cited by OpenAI's Spinning Up. The README quotes a different passage from this file.
+
+> If you keep that strategy when each run takes 10 hours, though, you can easily
+> waste a *lot* of time. Last run didn’t work? OK, I think it’s this thing. Let’s
+> set off another run to check. Coming back the next morning: still doesn’t work?
+> OK, maybe it’s this other thing. Let’s set off another run. A week later, you
+> still haven’t solved the problem.
+
+Why it lands: the one-change-then-declare loop written out as a transcript, with the cost measured in a week of wall clock.
+
+---
+
+## Mode 3: anchoring on the first idea
+
+## Lessons Learned Reproducing a Deep RL Paper -- Matthew Rahtz -- http://amid.fish/reproducing-deep-rl
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/amid_fish_reproducing_deep_rl.md:126
+- failure modes: 3
+- epistemic context: same log; this passage is the diagnosis that precedes the README's "think more, experiment less" prescription.
+
+> than forming hypotheses. Why spend 15 minutes carefully considering everything
+> that could be causing what you see when you can check the first idea that jumps
+> to mind in a fraction of that (and gather more evidence in the process)? To put
+> it another way: if you have rapid feedback, you can narrow down the hypothesis
+> space a lot faster by trying things than thinking carefully.
+
+Why it lands: explains why anchoring feels correct. It is correct when feedback is seconds, and an LLM's edit-and-rerun loop feels that fast even when the training run underneath it does not.
+
+## My Research Process: Key Mindsets -- Neel Nanda -- https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/cbBwwm4jW6AZctymL
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/nanda_research_process_key_mindsets.md:56
+- failure modes: 3, 1
+- epistemic context: published post by a supervisor of 20+ papers; a framing claim, not a measured result.
+
+> The standard hypothesis testing framework can be misleading here, because it has an implicit frame of being able to list all the hypotheses. But actually, most of your probability mass should normally be on “something I haven’t thought of yet”
+
+Why it lands: attacks anchoring at the root, and it also attacks the fix. Even after the agent dutifully writes hypotheses 1, 2 and 3, the correct posterior still puts most mass outside the list.
+
+## How to Become a Mechanistic Interpretability Researcher -- Neel Nanda -- https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/nanda_how_to_mech_interp.md:614
+- failure modes: 3, 7
+- epistemic context: same guide; a pattern he reports seeing repeatedly in researchers he supervises. The README quotes this file only for research-is-false, excitement, and read-your-data.
+
+> If trying to explain something mysterious, novice researchers often neglect simple, dumb hypotheses like “maybe MLP0 is incredibly important on *every* input, and there’s nothing special going on with my prompt”
+
+Why it lands: the missing hypothesis 2 is usually the boring one, and an exciting hypothesis 1 is what suppresses it. This is the mech interp version of "your steering vector is just a big norm".
+
+## Research as a Stochastic Decision Process -- Jacob Steinhardt -- https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/steinhardt_research_stochastic_decision_process.md:196
+- failure modes: 3, 6
+- epistemic context: same source; a first-person admission of his own repeated mistake, which is the kind of self-report that costs the author something.
+
+> Importantly, it is often not obvious that multiple approaches to a problem all have the same issue. In the past, I have spent months trying different approaches to a problem before finally stepping back and realizing that they were all failing for the same reason. Moreover, I had all the data necessary to make this realization a couple weeks in but had failed to do so.
+
+Why it lands: two modes at once. Hypotheses 2 and 3 can be hypothesis 1 wearing a hat, and the evidence that would have shown it was already sitting in the logs for weeks.
+
+## Full Stack Deep Learning Spring 2021, Lecture 7: Troubleshooting Deep Neural Networks -- Josh Tobin (notes by James Le, Vishnu Rachakonda) -- https://fullstackdeeplearning.com/spring2021/lecture-7/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/fsdl_spring2021_lecture7.md:443
+- failure modes: 3, 4
+- epistemic context: teaching notes from a widely used practitioner course; Tobin was an OpenAI research scientist. Not cited in the README at all.
+
+> * **Error goes up**: Commonly, this is due to a flip sign somewhere in
+>   the loss function/gradient.
+> * **Error explodes**: This is usually a numerical issue but can also
+>   be caused by a high learning rate.
+> * **Error oscillates**: You can lower the learning rate and inspect
+>   the data for shuffled labels or incorrect data augmentation.
+> * **Error plateaus**: You can increase the learning rate and get rid
+>   of regulation. Then you can inspect the loss function and the data
+>   pipeline for correctness.
+
+Why it lands: a symptom-to-cause table where every symptom has two or three candidates and only one of them is a learning rate. It is a ready-made hypothesis-2-and-3 generator for the moment the agent reaches for the knob.
+
+## My Model of the Research Process (shared draft), as quoted in the skill's own topic note -- Neel Nanda
+- file: /home/wassname/.agents/skills/ml-debug/references/research_taste.md:120
+- failure modes: 3
+- epistemic context: unpublished draft quoted in a local topic note; weaker provenance than the published posts.
+
+> Actively Seek Alternatives: Explicitly brainstorm other ways your observations could be explained. What are the simplest explanations? What known circuits or phenomena could be involved? What would a strong skeptic argue?
+
+Why it lands: hypothesis 2 and 3 made into an explicit step with a prompt for each. Note that it asks for the simplest explanations, not more of the same kind as hypothesis 1.
+
+---
+
+## Mode 4: obsession with the legible hyperparameters
+
+## Spinning Up as a Deep RL Researcher -- Joshua Achiam (OpenAI, 2018) -- https://spinningup.openai.com/en/latest/spinningup/spinningup.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/spinningup_researcher.md:56
+- failure modes: 4, 1
+- epistemic context: OpenAI research scientist, official Spinning Up documentation. The README quotes the tail of this same paragraph ("test in more than one environment"), so only this front half is unused.
+
+> **If it doesn’t work, assume there’s a bug.** Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it’s a bug. Bad hyperparameters can significantly degrade RL performance, but if you’re using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue.
+
+Why it lands: gives both the ordering the agent inverts and the reason. Published hyperparameters are already close to right, so the prior on the knob being your problem is low before you touch it.
+
+## A Recipe for Training Neural Networks -- Andrej Karpathy -- https://karpathy.github.io/2019/04/25/recipe/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/karpathy_recipe_training_nn_2019.md:41
+- failure modes: 4, 2, 1
+- epistemic context: the canonical practitioner post; the README cites it for inspect-data, fixed-seed, overfit-one-batch and Adam 3e-4, so this "fails silently" passage is separate. The cached file is an abridged note with its own elisions.
+
+> For example, perhaps you forgot to flip your labels when you left-right flipped the image during data augmentation. Your net can still (shockingly) work pretty well because your network can internally learn to detect flipped images and then it left-right flips its predictions. Or maybe your autoregressive model accidentally takes the thing it’s trying to predict as an input due to an off-by-one bug. Or you tried to clip your gradients but instead clipped the loss, causing the outlier examples to be ignored during training. Or you initialized your weights from a pretrained checkpoint but didn’t use the original mean. Or you just screwed up the settings for regularization strengths, learning rate, its decay rate, model size, etc.
+
+Why it lands: five worked examples, and every one is a label, sign, mask or target bug. The legible hyperparameters arrive last, in one clause, as an afterthought. That ordering is the whole of the mode.
+
+## Simple considerations for simple people building fancy neural networks -- Victor Sanh -- https://huggingface.co/blog/simple-considerations
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/sanh_simple_considerations_hf_2021.md:96
+- failure modes: 4, 3
+- epistemic context: same post; a practitioner heuristic, no experiment behind the 4e2 example.
+
+> Most importantly, there is no point of launching 1000 runs with different hyperparameters (or architecture tweaks like activation functions): **compare a couple of runs with different hyperparameters to get an idea of which hyperparameters have the highest impact** but in general, it is delusional to expect to get your biggest jumps of performance by simply tuning a few values. For instance, if your best performing model is trained with a learning rate of 4e2, there is probably something more fundamental happening inside your neural network and you want to identify and understand this behavior so that you can re-use this knowledge outside of your current specific context.
+
+Why it lands: treats a weird optimal hyperparameter as a symptom to explain rather than a setting to keep. That is the opposite reflex to "the sweep found 4e2, ship it".
+
+## ML Engineering for AI Safety and Robustness -- Catherine Olsson and the 80,000 Hours team -- https://80000hours.org/articles/ml-engineering-career-transition-guide/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/olsson_80000hours_ml_engineering_ai_safety.md:122
+- failure modes: 4, 2
+- epistemic context: career guide reporting Daniel Ziegler's self-study second-hand, so weaker than a practitioner writing in their own voice.
+
+> Once the algorithm was partially working, they would attain higher performance by looking for remaining bugs, both by reviewing the code carefully, and by collecting metrics such as average policy entropy to perform sanity-checks, rather than just tune hyperparameters.
+
+Why it lands: the explicit contrast between tuning and bug-hunting-with-diagnostics, from someone who took a partly working implementation to full performance. The named metric is a diagnostic, not a score.
+
+## How to get good at programming -- Ulisse Mini -- https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/ulisse_how_to_get_good_at_programming.md:31
+- failure modes: 4, 3
+- epistemic context: LessWrong post by a self-described "~5yrs of linux & programming experience" author, marked "Epistemic status: very confident". Low external validation, but the README already cites this source and the mechanism is checkable against your own behaviour.
+
+> Third, and perhaps most important for building skill,[[1]](https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming#fn289bs9hi65b)you must **notice** when you're going into brute-force search mode, and then **take action** by investing time in understanding the underlying system, until both the problem and solution make sense.
+
+Why it lands: sweeping the legible knobs is brute-force search wearing a lab coat. The paired footnote at line 51 of the same file names the cost, that his CSS skills did not improve for several years because he stayed in try-random-stuff mode.
+
+## How to more intelligently debug RL roadblocks? -- u/GrundleMoof -- https://old.reddit.com/r/reinforcementlearning/comments/bzg3l2/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/reddit_rl_roadblocks_bzg3l2.md:41
+- failure modes: 4, 3
+- epistemic context: LOW CREDIBILITY. Anonymous reddit self-report from a self-described non-expert. Its value is as a specimen of the failure mode, not as advice, and it should not be quoted as authority.
+
+> Things I've tried (but maybe not systematically enough):
+>
+> * Different initial LRs
+> * Different optimizers
+> * Different number of hidden layers/units
+> * Shared pi/V NN body (with diff output layers) vs not
+> * Changing amount of entropy
+> * Adding correlated noise
+> * Using TD residual instead of MC version
+> * Clipping the gradient
+> * Different gamma values
+
+Why it lands: nine knobs turned, all of them legible, and the agent still does not learn. This is a photograph of the default LLM search. A reply in the same thread, at line 60 of the same file, reports that his own two bugs on that environment were a terminal-flag masking error and a shape broadcast, neither of which any of those nine knobs can reach.
+
+---
+
+## Mode 5: not reading the data
+
+## Deep Learning, ch. 11 "Practical Methodology" -- Goodfellow, Bengio, Courville -- https://www.deeplearningbook.org/contents/guidelines.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/goodfellow_ch11_practical_methodology.md:210
+- failure modes: 5, 7, 1
+- epistemic context: standard textbook, in its list of debugging tests.
+
+> Visualize the model in action: When training a model to detect objects in images, view some images with the detections proposed by the model displayed superimposed on the image. When training a generative model of speech, listen to some of the speech samples it produces. This may seem obvious, but it is easy to fall into the practice of looking only at quantitative performance measurements like accuracy or log-likelihood. Directly observing the machine learning model performing its task will help to determine whether the quantitative performance numbers it achieves seem reasonable. Evaluation bugs can be some of the most devastating bugs because they can mislead you into believing your system is performing well when it is not.
+
+Why it lands: the textbook naming the exact drift, that it is easy to fall into looking only at the scalars. The last sentence explains why the scalar cannot police itself.
+
+## Deep Reinforcement Learning that Matters -- Henderson, Islam, Bachman, Pineau, Precup, Meger (AAAI 2018) -- https://arxiv.org/pdf/1709.06560
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/henderson_2018_deep_rl_matters.md:243
+- failure modes: 5, 6, 8
+- epistemic context: peer reviewed, backed by their own controlled reruns of four algorithms across four environments. The README quotes this file for seed splits and implementation differences, not for this.
+
+> By reaching a local optimum, learning curves can indicate successful optimization of the policy over time, when in reality the returns achieved are not qualitatively representative of learning the desired behaviour, as demon-strated in video replays of the learned policy 5. Therefore, it is important to show not only returns but demonstrations of the learned policy in action.
+
+Why it lands: a healthy-looking curve produced by a swimmer curling up and flailing. Peer reviewed, and the only way anyone saw it was by watching the output. Note the OCR artifacts ("demon-strated") are in the cached file.
+
+## DeepRLHacks (attendee notes on Schulman's talk) -- William Falcon -- https://github.com/williamFalcon/DeepRLHacks
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/williamfalcon_deeprl_hacks.md:49
+- failure modes: 5
+- epistemic context: secondary attendee notes; the matching primary slide is "Atari: can you see game features in downsampled image?" in the cached joschu_nuts_and_bolts.md.
+
+> 2. Make sure observations usable:
+>     - See if YOU could control the system by using the same observations you give the agent.
+>       - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
+
+Why it lands: turns "read the data" into a pass/fail test that takes a minute. If you cannot do the task from the model's inputs, no hyperparameter will save it.
+
+## Simple considerations for simple people building fancy neural networks -- Victor Sanh -- https://huggingface.co/blog/simple-considerations
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/sanh_simple_considerations_hf_2021.md:84
+- failure modes: 5
+- epistemic context: same post; self-reported experience, and the costly kind, an admission of repeated personal loss.
+
+> Pro-tip: when you work with language, have a serious **look at the outputs of the tokenizers**. I can’t count the number of lost hours I spent trying to reproduce results (and sometimes my own old results) because something went wrong with the tokenization.
+
+Why it lands: for LLM work, reading the data means reading the tokenized data, the artifact that actually enters the model, not the source text you believe you passed in.
+
+## Machine Learning Yearning (draft), ch. 14 -- Andrew Ng -- https://github.com/ajaymache/machine-learning-yearning
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/ng_ml_yearning_error_analysis.md:282
+- failure modes: 5, 3
+- epistemic context: widely circulated unpublished draft. The README quotes the "Manually examining 100 examples" sentence from this same long line, so only this earlier part is unused.
+
+> Error analysis can often help you figure out how promising different directions are. I’ve seen many engineers reluctant to carry out error analysis. It often feels more exciting to just jump in and implement some idea, rather than question if the idea is worth the time investment. This is a common mistake: It might result in your team spending a month only to realize afterward that it resulted in little benefit.
+
+Why it lands: names the motivational failure rather than the procedural one. "It often feels more exciting to just jump in and implement some idea" is the agent that skips the data and starts editing the config.
+
+## Debugging the training pipeline (HF LLM Course ch. 8.4) -- Sylvain Gugger et al. -- https://huggingface.co/learn/llm-course/chapter8/4
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md:670
+- failure modes: 5
+- epistemic context: official HF teaching material by the Trainer maintainers; instructional, not measured.
+
+> ⚠️ If you are doing distributed training, print samples of your dataset in each process and triple-check that you get the same thing. One common bug is to have some source of randomness in the data creation that makes each process have a different version of the dataset.
+
+Why it lands: sharpens "read the data" to per-rank. Reading one process's data is not reading the data when eight processes disagree with each other.
+
+---
+
+## Mode 6: not reading the log
+
+Thin, as flagged above. Three quotes, and none of them uses the words.
+
+## Deep Learning Tuning Playbook -- Godbole, Dahl, Gilmer, Shallue, Nado (Google Research) -- https://github.com/google-research/tuning_playbook
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/google_tuning_playbook.md:916
+- failure modes: 6, 1
+- epistemic context: Google Research team practice; the "Examining the training curves" section, which the README does not touch.
+
+> -   Although in many cases the primary objective of our experiments only
+>     requires considering the validation error of each trial, we must be careful
+>     when reducing each trial to a single number because it can hide important
+>     details about what’s going on below the surface.
+> -   For every study, we always look at the **training curves** (training error
+>     and validation error plotted versus training step over the duration of
+>     training) of at least the best few trials.
+
+Why it lands: the closest thing in the cache to a hard rule that you read the run before you report its number, from a team that had every excuse to just read the number.
+
+## Lessons Learned Reproducing a Deep RL Paper -- Matthew Rahtz -- http://amid.fish/reproducing-deep-rl
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/amid_fish_reproducing_deep_rl.md:237
+- failure modes: 6, 1
+- epistemic context: same project log; a self-reported cost for one specific ignored log signal. The quote spans lines 237 to 239.
+
+> (I missed
+> a multithreading bug for several months by ignoring a small but mysterious
+> decay in frames per second.)
+
+Why it lands: a price tag on skipping a boring number. The signal was in the log the whole time, it was not the loss curve, and it cost months.
+
+## Machine Learning Engineering Open Book, "Understanding Training Loss Patterns" -- Stas Bekman -- https://github.com/stas00/ml-engineering
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/bekman_ml_engineering_instabilities.md:257
+- failure modes: 6, 1, 3
+- epistemic context: first-hand post-mortem from BLOOM and IDEFICS scale training by the engineer who ran it; one incident, self-reported. The README quotes this file for spike types and the 104B post-mortem, not this.
+
+> There was no real spike in the two earlier runs. The loss never went up in the first place. In both resumes it was under-reporting loss due to an exactly repeated data and then it reached data it hasn't seen before and started reporting correctly. In other words it was overfitting and reporting a false loss.
+
+Why it lands: the visible symptom was an artifact of the resume and the data sampler, so every hypothesis about the optimizer or the precision would have been confidently wrong. Reading the whole log across resumes is what found it.
+
+---
+
+## Mode 7: a cheap indirect probe instead of running the real thing
+
+Second thinnest. No source here names representation-similarity probes. These five attack the general substitution.
+
+## How to Become a Mechanistic Interpretability Researcher -- Neel Nanda -- https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/nanda_how_to_mech_interp.md:605
+- failure modes: 7, 4
+- epistemic context: opinionated guide by a DeepMind mech interp lead; the RMU example is a published follow-up result, not a self-report.
+
+> **Do ablations on your fancy method**: It's easy for people to have a fancy method with lots of moving parts, when many actually are unnecessary. You should always try removing one part and see if the method breaks. Do this for each part.
+>     *   For example, the [original unlearning method](https://arxiv.org/abs/2403.03218v1) in the [RMU paper](https://arxiv.org/abs/2403.03218) claimed it was based on finding a meaningful steering vector, until follow-up work found that it was just about adding a vector with really high norm that broke the model, and a random vector performed just as well.
+
+Why it lands: a published case where a clever mechanism was actually norm damage. The random-vector control is the cheap real test that the indirect story never bothered to run.
+
+## CS229 Advice for Applying Machine Learning -- Andrew Ng -- https://cs229.stanford.edu/materials/ML-advice.pdf
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/cs229_ml_advice.md:638
+- failure modes: 7
+- epistemic context: Stanford course slides by Ng; the README cites the later Machine Learning Yearning instead, so this file is unused. Slide text, so the line breaks are the PDF's.
+
+> The only way to find out what needs work is to implement something quickly,
+>
+> and find out what parts break.
+
+Why it lands: the shortest statement of build-it-and-run-it. Carry Ng's own caveat with it, since the next slide says this is worse advice when your goal is to invent new algorithms.
+
+## Deep Learning, ch. 15 "Representation Learning" -- Goodfellow, Bengio, Courville -- https://www.deeplearningbook.org/contents/representation.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/goodfellow_ch15_representation_learning.md:180
+- failure modes: 7, 8
+- epistemic context: standard textbook, describing a figure from Chelsea Finn's robotics work.
+
+> Figure 15.5: An autoencoder trained with mean squared error for a robotics task has failed to reconstruct a ping pong ball. The existence of the ping pong ball and all its spatial coordinates are important underlying causal factors that generate the image and are relevant to the robotics task. Unfortunately, the autoencoder has limited capacity, and the training with mean squared error did not identify the ping pong ball as being salient enough to encode.
+
+Why it lands: the convenient proxy metric silently deleted the one object the task was about, and the metric looked fine the whole time. A cheap measure decides what counts as signal before you get to look at anything.
+
+## How to Become a Mechanistic Interpretability Researcher -- Neel Nanda -- https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/nanda_how_to_mech_interp.md:615
+- failure modes: 7, 5
+- epistemic context: same guide; a methodological preference he argues for, stated as opinion.
+
+> One of the key drivers of progress in mech interp is an openness to qualitative research: summary statistics lose a ton of information. What can we learn by actually looking deeply into what's happening?
+
+Why it lands: names what a scalar proxy costs. Distinct from the README's read-your-data quote, which is about data quality; this one is about the aggregate hiding the phenomenon.
+
+## Training Stability and Debugging -- Axolotl docs -- https://docs.axolotl.ai/docs/training_stability.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/axolotl_training_stability.md:99
+- failure modes: 7, 2
+- epistemic context: vendor documentation for a widely used fine-tuning framework; engineering advice distilled from user reports, not measured. The README quotes two other lines from this file.
+
+> 1. **Test reward function standalone**: Run it outside training with known inputs to verify it returns nonzero values.
+
+Why it lands: when the metric will not move, the first move is to run the real objective on known inputs. The same page's table at line 41 says a reward stuck at zero means the reward function is broken or the task is too hard, which is two hypotheses, not one.
+
+---
+
+## Mode 8: an arbitrary threshold set before you know what is fair
+
+## Deep Learning, ch. 11 "Practical Methodology" -- Goodfellow, Bengio, Courville -- https://www.deeplearningbook.org/contents/guidelines.html
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/goodfellow_ch11_practical_methodology.md:196
+- failure modes: 8, 1
+- epistemic context: standard textbook, the paragraph after the debugging-is-hard one.
+
+> In most cases, we do not know a priori what the intended behavior of the algorithm is. In fact, the entire point of using machine learning is that it will discover useful behavior that we were not able to specify ourselves. If we train a neural network on a new classification task and it achieves 5 percent test error, we have no straightforward way of knowing if this is the expected behavior or suboptimal behavior.
+
+Why it lands: the best quote in the set for this mode, and it kills the invented threshold from first principles. If you cannot say whether 5 percent error is good, then the 0.8 you wrote into the success criterion was a number you made up.
+
+## My Model of the Research Process (shared draft) -- Neel Nanda -- https://docs.google.com/document/d/1YMkeMrhqsWxZcNDD9CIUWEK_DAOegeufnbc79U2hycg/edit
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/nanda_research_process_shared_draft.md:337
+- failure modes: 8
+- epistemic context: unpublished draft of a published LessWrong sequence; this passage never made it to the published post, so it is draft quality from the same author.
+
+> A valuable intuition to have in mind is that, by default, all numbers are meaningless because we lack any scale to compare them. E.g. if a probe gets 95% classification accuracy on some task, is this good? Is this bad? Hard to say without knowing more! Baselines are one way to get context to compare against.
+
+Why it lands: states the default, that a number carries no information until something supplies its scale, and names the fix as a baseline rather than a chosen cutoff. The example is literally a probe accuracy.
+
+## CS231n, Neural Networks Part 3 -- Stanford (Andrej Karpathy) -- https://cs231n.github.io/neural-networks-3/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/cs231n_neural_networks_3.md:50
+- failure modes: 8
+- epistemic context: long-running Stanford course notes; the README cites this file only for the overfit-tiny-subset check.
+
+> You might be temped to keep track of the difference \(\mid f’\_a - f’\_n \mid \) or its square and define the gradient check as failed if that difference is above a threshold. However, this is problematic. For example, consider the case where their difference is 1e-4. This seems like a very appropriate difference if the two gradients are about 1.0, so we’d consider the two gradients to match. But if the gradients were both on order of 1e-5 or lower, then we’d consider 1e-4 to be a huge difference and likely a failure.
+
+Why it lands: a fully worked case where a fixed numeric cutoff is meaningless until you know the scale of the quantity. The fix is to change the metric to a scale-free one, not to argue about where the cutoff should sit. The typo "temped" is in the source.
+
+## Simple considerations for simple people building fancy neural networks -- Victor Sanh -- https://huggingface.co/blog/simple-considerations
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/sanh_simple_considerations_hf_2021.md:58
+- failure modes: 8, 5
+- epistemic context: same post; the questions he says he asks himself before starting, not a result.
+
+> *   How would a random predictor perform (especially in classification problems)? Dataset can be unbalanced…
+> *   What would the loss look like for a random predictor?
+> *   What is (are) the best metric(s) to measure progress on my task?
+> *   What are the limits of this metric? If it’s perfect, what can I conclude? What can’t I conclude?
+
+Why it lands: four questions that have to be answered before any number can be called good or bad. The last one, what you cannot conclude from a perfect score, is the specific antidote to a made-up pass threshold.
+
+## Debugging the training pipeline (HF LLM Course ch. 8.4) -- Sylvain Gugger et al. -- https://huggingface.co/learn/llm-course/chapter8/4
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md:674
+- failure modes: 8, 6
+- epistemic context: official HF course; instructional, not measured. The README quotes two other passages from this file.
+
+> If the loss/metric you get on your initial model is very different from the loss/metric you would expect for random predictions, double-check the way your loss or metric is computed, as there is probably a bug there. If you are using several losses that you add at the end, make sure they are of the same scale.
+
+Why it lands: gives the constructive alternative. Compute what random gets, then treat any distance from it as a bug report until you have shown otherwise. The second sentence is your own combined-loss objection stated by HF.
+
+## The 37 Implementation Details of Proximal Policy Optimization -- Huang, Dossa, Raffin, Kanervisto, Wang -- https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/cleanrl_37_ppo_details.md:624
+- failure modes: 8, 2
+- epistemic context: ICLR Blog Track, a reviewed venue, with every claim linked to a code line and to tracked W&B runs. Not cited in the README.
+
+> 5.   **Rule of thumb: 400 episodic return in breakout**: Check if your PPO could obtain 400 episodic return in breakout. We have found this to be a practical rule of thumb to determine the fidelity of online PPO implementations in GitHub. Often we found PPO repositories not able to do this, and we know they probably do not match all implementation details of `openai/baselines`’ PPO.
+
+Why it lands: shows the legitimate form of a numeric gate. The number was discovered by reproducing a known-good reference, not chosen in advance. The sting is in the last sentence, that most public repos fail it, so a plausible-looking implementation is usually still broken.
+
+## Bad Labels -- Vincent D. Warmerdam (koaning) -- https://koaning.io/posts/labels/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/koaning_bad_labels.md:25
+- failure modes: 8, 5
+- epistemic context: practitioner blog; the surrounding claim is backed by the labelerrors.com paper (arXiv:2103.14749), this sentence is his argument. The README quotes three other lines from this file.
+
+> The issue here isn't just that we might have bad labels in our training set, the issue is that it appears in the validation set. If a machine learning model can become state of the art by squeezing another 0.5% out of a validation set one has to wonder. Are we really making a better model? Or are we creating a model that is better able to overfit on the bad labels?
+
+Why it lands: puts a floor under any target. A threshold set tighter than the label noise in your validation set is measuring overfitting to errors.
+
+---
+
+## Extra: good and unused, fits none of the eight cleanly
+
+## Nuts and Bolts of Deep RL Research (Deep RL Bootcamp lecture 6, audience Q&A) -- John Schulman -- https://www.youtube.com/watch?v=8EcdaCk9KaQ
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/schulman_nuts_bolts_deeprl_bootcamp_2017_subtitles.md:870
+- failure modes: 8, 2 (partially), but it is really about unit testing ML
+- epistemic context: the PPO and TRPO author answering a live question. The cached text is auto-generated captions, so there is no punctuation and there may be transcription slips. Quote with that caveat visible.
+
+> so if you try to write a test saying I
+> should be at performance 100 after this
+> many iterations it might fail just out
+> of random noise but yeah I think
+> probably unit tests are a good idea
+
+Why it lands: it is the pinned numeric target problem stated by someone who would know, but the caption format makes it awkward to quote in a README, which is why it is down here rather than under mode 8.
+
+## r/MachineLearning thread on "37 Reasons why your NN is not working" -- anonymous commenter -- https://old.reddit.com/r/MachineLearning/comments/6pfsyk/
+- file: /home/wassname/.agents/skills/ml-debug/docs/evidence/reddit_37_reasons_nn_6pfsyk.md:149
+- failure modes: 2 and 4, but as a specimen not as advice
+- epistemic context: LOW CREDIBILITY. Anonymous reddit comment from 2017, no verifiable identity, retrieved via a Wayback snapshot. Do not cite this as authority.
+
+> My point is that if I came up with the idea of GANs, they wouldn't be recognized because I can't make the idea work in practice. I want to learn the tools I need to find out what is wrong with my current implementation.
+
+Why it lands: a person who has swept hyperparameters, glanced at gradients, failed to localise the bug, and concluded that a method known to work would have died in his hands. That is the mode 2 error stated from the inside, but it is a reddit comment and should be presented as a specimen.
+
+---
+
+Compiled by CLAUDE-OPUS, 2026-08-25. Read-only pass over the ml-debug cache; nothing under
+`/home/wassname/.agents/` was modified.

docs/evidence/agans_debugging_9_rules.md

@@ -0,0 +1,353 @@
+# Debugging: The 9 Indispensable Rules
+
+David J. Agans
+
+> Notes: table of contents and Introduction, verbatim from a user-supplied EPUB. Extracted with `w3m -dump` on 2026-09-02; layout and images omitted. The complete book text (all 15 chapters, verbatim) is in the private dlbook repo at `agans_debugging_9_rules.md`.
+
+> Bibliographic record: *Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems*, David J. Agans, AMACOM, 2002, ISBN 978-0-8144-2678-4 (ebook). EPUB SHA-256: `ce3b6c92a7f263d0027b3b2d42c3061d06e8083d8a73de3a1f5eb523756699e4`.
+
+## Contents
+
+Contents
+
+Chapter 1: Introduction
+
+How Can That Work?
+
+Isn’t It Obvious?
+
+Anyone Can Use It
+
+It’ll Debug Anything
+
+But It Won’t Prevent, Certify, or Triage Anything
+
+More Than Just Troubleshooting
+
+A Word About War Stories
+
+Stay Tuned
+
+Chapter 2: The Rules—Suitable for Framing
+
+Chapter 3: Understand the System
+
+Read the Manual
+
+Read Everything, Cover to Cover
+
+Know What’s Reasonable
+
+Know the Road Map
+
+Know Your Tools
+
+Look It Up
+
+Remember
+
+Understand the System
+
+Chapter 4: Make It Fail
+
+Do It Again
+
+Start at the Beginning
+
+Stimulate the Failure
+
+Don’t Simulate the Failure
+
+What If It’s Intermittent?
+
+What if I’ve Tried Everything and It’s Still Intermittent?
+
+A Hard Look at Bad Luck
+
+Lies, Damn Lies, and Statistics
+
+Did You Fix It, or Did You Get Lucky?
+
+“But That Can’t Happen”
+
+Never Throw Away a Debugging Tool
+
+Remember
+
+Make It Fail
+
+Chapter 5: Quit Thinking and Look
+
+See the Failure
+
+See the Details
+
+Now You See It, Now You Don’t
+
+Instrument the System
+
+Design Instrumentation In
+
+Build Instrumentation In Later
+
+Don’t Be Afraid to Dive In
+
+Add Instrumentation On
+
+Instrumentation in Daily Life
+
+The Heisenberg Uncertainty Principle
+
+Guess Only to Focus the Search
+
+Remember
+
+Quit Thinking and Look
+
+Chapter 6: Divide and Conquer
+
+Narrow the Search
+
+In the Ballpark
+
+Which Side Are You On?
+
+Inject Easy-to-Spot Patterns
+
+Start with the Bad
+
+Fix the Bugs You Know About
+
+Fix the Noise First
+
+Remember
+
+Divide and Conquer
+
+Chapter 7: Change One Thing at a Time
+
+Use a Rifle, Not a Shotgun
+
+Grab the Brass Bar with Both Hands
+
+Change One Test at a Time
+
+Compare with a Good One
+
+What Did You Change Since the Last Time It Worked?
+
+Remember
+
+Change One Thing at a Time
+
+Chapter 8: Keep an Audit Trail
+
+Write Down What You Did, in What Order, and What Happened
+
+The Devil Is in the Details
+
+Correlate
+
+Audit Trails for Design Are Also Good for Testing
+
+The Shortest Pencil Is Longer Than the Longest Memory
+
+Remember
+
+Keep an Audit Trail
+
+Chapter 9: Check the Plug
+
+Question Your Assumptions
+
+Don’t Start at Square Three
+
+Test the Tool
+
+Remember
+
+Check the Plug
+
+Chapter 10: Get a Fresh View
+
+Ask for Help
+
+A Breath of Fresh Insight
+
+Ask an Expert
+
+The Voice of Experience
+
+Where to Get Help
+
+Don’t Be Proud
+
+Report Symptoms, Not Theories
+
+You Don’t Have to Be Sure
+
+Remember
+
+Get a Fresh View
+
+Chapter 11: If You Didn’t Fix It, It Ain’t Fixed
+
+Check That It’s Really Fixed
+
+Check That It’s Really Your Fix That Fixed It
+
+It Never Just Goes Away by Itself
+
+Fix the Cause
+
+Fix the Process
+
+Remember
+
+If You Didn’t Fix It, It Ain’t Fixed
+
+Chapter 12: All the Rules in One Story
+
+Chapter 13: Easy Exercises for the Reader
+
+A Light Vacuuming Job
+
+A Flock of Bugs
+
+A Loose Restriction
+
+The Jig Is Up
+
+Chapter 14: The View from the Help Desk
+
+Help Desk Constraints
+
+The Rules, Help Desk Style
+
+Understand the System
+
+Make It Fail
+
+Quit Thinking and Look
+
+Divide and Conquer
+
+Change One Thing at a Time
+
+Keep an Audit Trail
+
+Check the Plug
+
+Get a Fresh View
+
+If You Didn’t Fix It, It Ain’t Fixed
+
+Remember
+
+The View from the Help Desk Is Murky
+
+Chapter 15: The Bottom Line
+
+The Debugging Rules Web Site
+
+If You’re an Engineer
+
+If You’re a Manager
+
+If You’re a Teacher
+
+Remember
+
+Index
+
+## Introduction
+
+chapter
+
+1
+
+Introduction
+
+“At present I am, as you know, fairly busy, but I propose to devote my declining years to the composition of a textbook which shall focus the whole art of detection into one volume.”
+
+—SHERLOCK HOLMES, THE ADVENTURE OF THE ABBEY GRANGE
+
+This book tells you how to find out what’s wrong with stuff, quick. It’s short and fun because it has to be—if you’re an engineer, you’re too busy debugging to read anything more than the daily comics. Even if you’re not an engineer, you often come across something that’s broken, and you have to figure out how to fix it.
+
+Now, maybe some of you never need to debug. Maybe you sold your dot.com IPO stock before the company went belly-up and you simply have your people look into the problem. Maybe you always luck out and your design just works—or, even less likely, the bug is always easy to find. But the odds are that you and all your competitors have a few hard-to-find bugs in your designs, and whoever fixes them quickest has an advantage. When you can find bugs fast, not only do you get quality products to customers quicker, you get yourself home earlier for quality time with your loved ones.
+
+So put this book on your nightstand or in the bathroom, and in two weeks you’ll be a debugging star.
+
+How Can That Work?
+
+How can something that’s so short and easy to read be so useful? Well, in my twenty-six years of experience designing and debugging systems, I’ve discovered two things (more than two, if you count stuff like “the first cup of coffee into the pot contains all the caffeine”):
+
+1.  When it took us a long time to find a bug, it was because we had neglected some essential, fundamental rule; once we applied the rule, we quickly found the problem.
+
+2.  People who excelled at quick debugging inherently understood and applied these rules. Those who struggled to understand or use these rules struggled to find bugs.
+
+I compiled a list of these essential rules; I’ve taught them to other engineers and watched their debugging skill and speed increase. They really, really work.
+
+Isn’t It Obvious?
+
+As you read these rules, you may say to yourself, “But this is all so obvious.” Don’t be too hasty; these things are obvious (fundamentals usually are), but how they apply to a particular problem isn’t always so obvious. And don’t confuse obvious with easy—these rules aren’t always easy to follow, and thus they’re often neglected in the heat of battle.
+
+The key is to remember them and apply them. If that was obvious and easy, I wouldn’t have to keep reminding engineers to use them, and I wouldn’t have a few dozen war stories about what happened when we didn’t. Debuggers who naturally use these rules are hard to find. I like to ask job applicants, “What rules of thumb do you use when debugging?” It’s amazing how many say, “It’s an art.” Great—we’re going to have Picasso debugging our image-processing algorithm. The easy way and the artistic way do not find problems quickly.
+
+This book takes these “obvious” principles and helps you remember them, understand their benefits, and know how to apply them, so you can resist the temptation to take a “shortcut” into what turns out to be a rat hole. It turns the art of debugging into a science.
+
+Even if you’re a very good debugger already, these rules will help you become even better. When an early draft of this book was reviewed by skilled debuggers, they had several comments in common: Besides teaching them one or two rules that they weren’t already using (but would in the future), the book helped them crystallize the rules they already unconsciously followed. The team leaders (good debuggers rise to the top, of course) said that the book gave them the right words to transmit their skills to other members of the team.
+
+Anyone Can Use It
+
+Throughout the book I use the term engineer to describe the reader, but the rules can be useful to a lot of you who may not consider yourselves engineers. Certainly, this includes you if you’re involved in figuring out what’s wrong with a design, whether your title is engineer, programmer, technician, customer support representative, or consultant.
+
+If you’re not directly involved in debugging, but you have responsibility for people who are, you can transmit the rules to your people. You don’t even have to understand the details of the systems and tools your people use—the rules are fundamental, so after reading this book, even a pointy-haired manager should be able to help his far-more-intelligent teams find problems faster.
+
+If you’re a teacher, your students will enjoy the war stories, which will give them a taste of the real world. And when they burst onto that real world, they’ll have a leg up on many of their more experienced (but untrained in debugging) competitors.
+
+It’ll Debug Anything
+
+This book is general; it’s not about specific problems, specific tools, specific programming languages, or specific machines. Rather, it’s about universal techniques that will help you to figure out any problem on any machine in any language using whatever tools you have. It’s a whole new level of approach to the problem—for example, rather than tell you how to set the trigger on a Glitch-O-Matic digital logic analyzer, I’m going to tell you why you have to use an analyzer, even though it’s a lot of trouble to hook it up.
+
+It’s also applicable to fixing all kinds of problems. Your system may have been designed wrong, built wrong, used wrong, or just plain got broken; in any case, these techniques will help you get to the heart of the problem quickly.
+
+The methods presented here aren’t even limited to engineering, although they were honed in the engineering environment. They’ll help you figure out what’s wrong with other things, like cars, houses, stereo equipment, plumbing, and human bodies. (There are examples in the book.) Admittedly, there are systems that resist these techniques—the economy is too complex, for example. And some systems don’t need these methods; e.g., everybody already knows what’s wrong with the government.
+
+But It Won’t Prevent, Certify, or Triage Anything
+
+While this book is general about methods and systems, it’s very focused on finding the causes of bugs and fixing them.
+
+It’s not about quality development processes aimed at preventing bugs in the first place, such as ISO-9000, code reviews, or risk management. If you want to read about that, I recommend books like The Tempura Method of Totalitarian Quality Management Processes or The Feng Shui Guide to Vermin-Free Homes. Quality process techniques are valuable, but they’re often not implemented; even when they are, they leave some bugs in the system.
+
+Once you have bugs, you have to detect them; this takes place in your quality assurance (QA) department or, if you don’t have one of those, at your customer site. This book doesn’t deal with this stage either—test coverage analysis, test automation, and other QA techniques are well handled by other resources. A good book of poetry, such as How Do I Test Thee, Let Me Count the Ways, can help you while away the time as you check the 6,467,826 combinations of options in your product line.
+
+And sooner or later, at least one of those combinations will fail, and some QA guy or customer is going to write up a bug report. Next, some managers, engineers, salespeople, and customer support people will probably get together in a triage meeting and argue passionately about how important the bug is, and therefore when and whether to fix it. This subject is deeply specific to your market, product, and resources, and this book will not touch it with a ten-foot pole. But when these people decide it has to be fixed, you’ll have to look at the bug report and ask yourself, “How the heck did that happen?” That’s when you use this book (see Figure 1-1).
+
+The following chapters will teach you how to prepare to find a bug, dig up and sift through the clues to its cause, home in on the actual problem so you can fix it, and then make sure you really fixed it so you can go home triumphant.
+
+Figure 1-1. When to Use This Book.
+
+Images
+
+More Than Just Troubleshooting
+
+Though the terms are often interchanged, there’s a difference between debugging and troubleshooting, and there’s a difference between this debugging book and the hundreds of troubleshooting guides available today. Debugging usually means figuring out why a design doesn’t work as planned. Troubleshooting usually means figuring out what’s broken in a particular copy of a product when the product’s design is known to be good—there’s a deleted file, a broken wire, or a bad part. Software engineers debug; car mechanics troubleshoot. Car designers debug (in an ideal world). Doctors troubleshoot the human body—they never got a chance to debug it. (It took God one day to design, prototype, and release that product; talk about schedule pressure! I guess we can forgive priority-two bugs like bunions and male pattern baldness.)
+
+The techniques in this book apply to both debugging and troubleshooting. These techniques don’t care how the problem got in there; they just tell you how to find it. So they work whether the problem is a broken design or a broken part. Troubleshooting books, on the other hand, work only on a broken part. They boast dozens of tables, with symptoms, problems, and fixes for anything that might go wrong with a particular system. These are useful; they’re a compendium of everything that has ever broken in that type of system, and what the symptoms and fixes were. They give a troubleshooter the experience of many others, and they help in finding known problems faster. But they don’t help much with new, unknown problems. And thus they can’t help with design problems, because engineers are so creative, they like to make up new bugs, not use the same old ones.
+
+So if you’re troubleshooting a standard system, don’t ignore Rule 8 (“Get a Fresh View”); go ahead and consult a troubleshooting guide to see if your problem is listed. But if it isn’t, or if the fix doesn’t work, or if there’s no troubleshooting guide out yet because you’re debugging the world’s first digital flavor transmission system, you won’t have to worry, because the rules in this book will get you to the heart of your brand-new problem.
+
+A Word About War Stories
+
+I’m a male American electronics engineer, born in 1954. When I tell a “war story” about some problem that got solved somehow, it’s a real story, so it comes from things that male American electronics engineers born in 1954 know about. You may not be all or any of those, so you may not understand some of the things I mention. If you’re an auto mechanic, you may not know what an interrupt is. If you were born in 1985, you may not know what a record player is. No matter; the principle being demonstrated is still worth knowing, and I’ll explain enough as I go along so you’ll be able to get the principle.
+
+You should also know that I’ve taken some license with the details to protect the innocent, and especially the guilty.
+
+Stay Tuned
+
+In this book I’ll introduce the nine golden rules of debugging, then devote a chapter to each. I’ll start each chapter with a war story where the rule proved crucial to success; then I’ll describe the rule and show how it applies to the story. I’ll discuss various ways of thinking about and using the rule that are easy to remember in the face of complex technological problems (or even simple ones). And I’ll give you some variations showing how the rule applies to other stuff like cars and houses.
+
+In the final few chapters, I’ve included a set of war stories to exercise your understanding, a section on using the rules under the trying circumstances of the help desk, and a few last hints for putting what you’ve learned to work in your job.
+
+When you’re done with this book, your debugging efficiency will be much higher than before. You may even find yourself wandering around, looking for engineers in distress so you can swoop in and save the day. One bit of advice, though: Leave the leotard and cape at home.

docs/evidence/bekman_ml_engineering_instabilities.md

@@ -1,42 +1,296 @@
-Source: https://github.com/stas00/ml-engineering — training/instabilities/README.md, training/instabilities/training-loss-patterns.md, debug/README.md (master branch)
-Title: "Machine Learning Engineering Open Book" — Stas Bekman (BLOOM-176B / IDEFICS-80B training lead at HF, ex-PyTorch)
-Fetched-via: curl of raw markdown from github, 2026-06-11
-Fetch-status: verbatim excerpts
+Source: https://github.com/stas00/ml-engineering - training/instabilities/README.md, training/instabilities/training-loss-patterns.md, debug/README.md (master branch)
+Title: "Machine Learning Engineering Open Book" - Stas Bekman (BLOOM-176B / IDEFICS-80B training lead at HF, ex-PyTorch)
+Fetched-via: curl -sL of the three raw markdown files, 2026-08-15 (CLAUDE agent)
+Fetch-status: verbatim, full text of all three pages, concatenated with a heading per file. Replaces the earlier excerpts (CLAUDE agent)
 
-# ML Engineering Open Book — instabilities and loss patterns (excerpts)
+# ==== training/instabilities/README.md ====
 
-From "Understanding Training Loss Patterns":
+# Avoiding, Recovering From and Understanding Instabilities
 
-> Training loss plot is similar to the heart beat pattern - there is the good, the bad and you-should-worry one. After studying many training loss trajectories one develops an intuition to explain various loss behaviors during one's training and how to act on those.
+Sub-sections:
 
-> I warn you that the "Understanding" in the title of this section is overloaded since very often we don't really understand why certain types of spikes happen. Here "understanding" refers to recognizing various patterns. We then usually have techniques to overcome the bad patterns and bring the training successfully to the finish line.
+* [Understanding Training Loss Patterns](training-loss-patterns.md) - types of spikes, divergences, grokking moments, resumes, etc.
 
-> Thus you will find here a gallery of training loss patterns sometimes with real explanations, but more often than not educated guesses to what might be happening.
+## Learning from Training Logbooks
 
-The pre-BLOOM 104B failure story ("A very failed training"):
+The best learning is to read [Publicly available training LLM/VLM logbooks](../../resources/README.md#publicly-available-training-llmvlm-logbooks) because there you can see exactly what happened and how the problem has been overcome.
 
-> Prior to starting BLOOM-176B training we did multiple experiments with the 104B model. We failed to figure out how to not diverge very early on. [...] As you can see many attempts were made, many techniques were applied (see chronicles). We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.
 
-On loss spikes ("Main types of loss spikes"):
+## STD Init
 
-> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
->
-> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.
+Correctly initializing the initial distribution of the tensors can have a tremendous impact on training's stability. The `std` value isn't fixed and depends on the hidden dimension size.
 
-From "Avoiding, Recovering From and Understanding Instabilities" — the init-std story:
+This proved to be a very crucial setting in our pre-BLOOM 104B experiments and we couldn't break past the first few thousands iterations until we figured out that the 0.02 default `--init-method-std` in Megatron-LM was a way too big for our model.
 
-> Correctly initializing the initial distribution of the tensors can have a tremendous impact on training's stability. The `std` value isn't fixed and depends on the hidden dimension size.
->
-> This proved to be a very crucial setting in our pre-BLOOM 104B experiments and we couldn't break past the first few thousands iterations until we figured out that the 0.02 default `--init-method-std` in Megatron-LM was a way too big for our model.
+We referred to these two sources:
 
-(They settled on the 530B paper's `sqrt(1/(NHIDDEN*3))`: "for NHIDDEN=14336 the math was sqrt(1/(14336*3)) = 0.00482 and that's what we used. It surely wasn't the only reason why we had no stability issues during BLOOM-176B training, but I think it was one of the crucial ones.")
+1. "Transformers without Tears" paper https://arxiv.org/abs/1910.05895 prescribes: `sqrt(2/(NHIDDEN*5))`
 
-On PaLM's spikes ("'Bad' combination of data batch and model parameter state"):
+2. The 530B training paper https://arxiv.org/abs/2201.11990 they used an even smaller init formula: `sqrt(1/(NHIDDEN*3))`
 
-> PaLM team observed dozens of loss spikes at "highly irregular intervals" when training larger models. While they were not able to track down the root cause, they mitigated the issue by restarting from an earlier checkpoint and skipping potentially problematic data batches.
+and decided to go with the 530B one as it leads to an even smaller init value.
 
-On reading training logbooks:
+To make it easier to compare the two formulas, they can be rewritten as:
+1. `sqrt(0.4000/NHIDDEN)`
+2. `sqrt(0.3333/NHIDDEN)`
 
-> The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome.
+Thus for `NHIDDEN=14336` the math was `sqrt(1/(14336*3)) = 0.00482` and that's what we used. It surely wasn't the only reason why we had no stability issues during BLOOM-176B training, but I think it was one of the crucial ones.
 
-Debug section index (debug/README.md) — guides for: Debugging PyTorch programs; Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs; Network Debug; Troubleshooting NVIDIA GPUs; Underflow and Overflow Detection; plus tools (torch-distributed-gpu-test.py, NicerTrace).
+
+## Numerical instabilities
+
+See also [Detecting problematic tensor values](../../debug/pytorch.md#detecting-problematic-tensor-values) and [Underflow and Overflow Detection](../../debug/pytorch.md#underflow-and-overflow-detection) in the debugging chapter for tooling to locate `inf`/`nan` values.
+
+Certain mathematical operations could be unstable when dealing with low precision numbers.
+
+For example, please see this very interesting [PyTorch guide on numerical stability](https://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html).
+
+Now let's look at a specific example of this concept in action.
+
+During 104B training experiments where fp16 mixed precision was used - the following improvement was proposed by [Corby Rosset](https://github.com/corbyrosset) to make [self-attention more stable](https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/118).
+
+Specifically this [line](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/c839a8aa30731f71b3738d56009be9668508e366/megatron/model/transformer.py#L303) shows that the `norm_factor` may be multiplied after the Query * Key matrix multiplication. If the dim of Q and K are very large, the output may blow up and the `norm_factor` won't be able to save it.
+
+Proposal: move the `norm_factor` inward, so Q and K are scaled down before matrix multiply:
+```python
+        matmul_result = torch.baddbmm(
+            matmul_result,
+            1.0/math.sqrt(self.norm_factor) * query_layer.transpose(0, 1),   # [b * np, sq, hn]
+            1.0/math.sqrt(self.norm_factor) * key_layer.transpose(0, 1).transpose(1, 2),  # [b * np, hn, sk]
+            beta=0.0 if alibi is None else 1.0, alpha=1.0)
+
+        # change view to [b, np, sq, sk]
+        attention_scores = matmul_result.view(*output_size)
+```
+
+To make the operation mathematically equivalent, moving the norm factor inward requires taking sqrt again if n is a scalar, A and B matrices:
+```
+n * (A dot B) === (sqrt(n) * A) dot (sqrt(n) * B)
+```
+
+Now A and B dimensions can be significantly larger.
+
+The same post-multiply scaling problem also exists one level down, in the CUDA matmul APIs themselves.
+
+For CUDA kernel writers [CuBlas](https://docs.nvidia.com/cuda/cublas/index.html)'s `GemmStridedBatchedEx` at the time of this writing has a similar issue. It is defined as:
+
+```
+C+i*strideC=αop(A+i*strideA)op(B+i*strideB)+β(C+i*strideC), for i ∈[0,batchCount−1]
+```
+
+The issue is that `alpha` is multiplied after the matrix-matrix multiplication is done so it can cause instability.
+
+## "Bad" combination of data batch and model parameter state
+
+PaLM team observed dozens of loss spikes at "highly irregular intervals" when training larger models. While they were not able to track down the root cause, they mitigated the issue by restarting from an earlier checkpoint and skipping potentially problematic data batches. [Section 5.1 Training instability](https://arxiv.org/pdf/2204.02311)
+
+
+## Time-domain correlation divergence in Adam
+
+[A Theory on Adam Instability in Large-Scale Machine Learning](https://arxiv.org/abs/2304.09871) performs a rigorous study of divergence spikes while training LLMs at up to 546B parameters - and suggests that the time-domain correlation leads to divergence of Adam. This is triggered by the epsilon value not being small enough and gradient estimation components become similar to the epsilon.
+
+In section 7.1 they propose practical suggestions, the most interesting one of them is setting epsilon to 0 and possibly dealing with division by zero condition.
+
+
+# ==== training/instabilities/training-loss-patterns.md ====
+
+# Understanding Training Loss Patterns
+
+Training loss plot is similar to the heart beat pattern - there is the good, the bad and you-should-worry one. After studying many training loss trajectories one develops an intuition to explain various loss behaviors during one's training and how to act on those.
+
+I warn you that the "Understanding" in the title of this section is overloaded since very often we don't really understand why certain types of spikes happen. Here "understanding" refers to recognizing various patterns. We then usually have techniques to overcome the bad patterns and bring the training successfully to the finish line.
+
+Thus you will find here a gallery of training loss patterns sometimes with real explanations, but more often than not educated guesses to what might be happening.
+
+Please excuse the plot snapshots looking wildly different from each other as they have come from many sources over multiple years.
+
+## The good, the bad and the unexpected
+
+Let's look at some good, bad and unusual patterns.
+
+### A very failed training
+
+Prior to starting BLOOM-176B training we did multiple experiments with the [104B model](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-wide). We failed to figure out how to not diverge very early on.
+
+![](images/pre-bloom-104B-en-fail.png)
+
+As you can see many attempts were made, many techniques were applied (see [chronicles](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr8-104B-wide/chronicles.md). We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.
+
+
+### An almost perfect training
+
+![](images/bloom-176B-success.png)
+
+The [BLOOM-176B](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml) training had a close to perfect training loss trajectory, with a single spike that has recovered in 200 steps.
+
+You can inspect the [TB](https://huggingface.co/bigscience/tr11-176B-logs/tensorboard) to zoom in and check other plots.
+
+This was the almost perfect training indeed. Lots of hard work was put into achieving this.
+
+
+### The grokking moment
+
+Some time back I was doing performance testing and run a tiny global batch size of 8 on an 8×A100 node, training llama-2-7b from scratch. (w/ DeepSpeed ZeRO-3 DP using HF Transformers [Llama](https://github.com/huggingface/transformers/tree/main/src/transformers/models/llama) implementation)
+
+![](images/llama-7b-grokking-no-zoom.png)
+
+Here one can observe a rapid loss improvement from 4 to 2.5 in just 480 samples after a very steady much slower improvements. My colleague [Gautam Mittal](https://github.com/gmittal) called it the [grokking](https://en.wikipedia.org/wiki/Grok) moment. In just a handful of steps the model suddenly generalized to much better predict the next tokens.
+
+Normally one doesn't see such a dramatic improvement when using a much larger batch size.
+
+If we zoom in it took about 60 8-sample per iteration steps:
+
+![](images/llama-7b-grokking.png)
+
+
+
+
+## Main types of loss spikes
+
+In general there are 3 types of loss spikes:
+
+1. Fast recovering spikes
+2. Slow recovering spikes
+3. Not fully recovering spikes
+
+The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.
+
+While one would suspect that the batch before the spike was the trigger, but if you were to study that batch's contents you are likely to find nothing unusual - quite often the problem starts developing many steps before and then most of the sudden it happens. But also it might not be easy to study the batch, since it could amount to a size of a book when the global batch size and the sequence lengths are huge.
+
+
+### Fast recovering spikes
+
+Loss spikes can happen often and as long as they quickly bounce back to where they left off the training usually continues as if nothing happened:
+
+Here is an example of [the 13B pre-BLOOM training experiment](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr1-13B-base):
+
+![](images/pre-bloom-tr1-13B-glitch-1-2.png)
+
+As you can see there are many spikes, some of a huge magnitude but they have all quickly recovered.
+
+
+### Slow recovering spikes
+
+Here is a slow recovering spike from the [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training:
+
+![](images/idefics-80b-tr-190-01-spike-recover-2023-05-30.png)
+
+
+
+### Not fully recovering spikes
+
+
+This [104B model attempt](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-wide) spiked, started recovering but decided to not recover fully and instead started diverging
+
+![](images/pre-bloom-tr8-104B-glitch-1.png)
+
+Here is another example from the [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training:
+
+![](images/idefics-80b-tr-190-01-spike-2023-05-27.png)
+
+
+### Non-spike diverging
+
+Here are a few examples of diverging that didn't go through a spike
+
+![](images/pre-bloom-tr8-104B-glitch-5.png)
+
+and here are a few more:
+
+![](images/pre-bloom-tr8-104B-glitch-7-10.png)
+
+as you can see each restart makes a bit of progress and then the model diverges.
+
+All these are from the [104B model attempts](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-wide).
+
+
+### Multiple datasets spikes
+
+During the [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training we were using 2 different dataset types mixed together:
+
+![](images/idefics-80b-tr-190-01-losses-2023-06-04.png)
+
+Legend: cm4 (high), average (mid) and pmd (low)
+
+You can see that the loss spikes were sometimes happening simultaneously on both datasets and at other times only one of the datasets loss would spike.
+
+Here the model was learning two different data distributions and as you can see it was not reporting the same loss and the spike behaviors on both data distributions. The pmd datasets loss was much easier for the model than the cm4 one.
+
+
+## Resume-related spikes
+
+Training resume due to a hardware crash or because a need to rollback to an earlier checkpoint due to encountering a divergence is pretty much guaranteed to happen. If your training software doesn't resume perfectly so that the model doesn't notice there was a resume various problems could be encountered.
+
+The most complicated challenge of resume is restoring various RNGs, getting to the DataLoader index where the previous training was restored, and dealing with various other requirements if you use complex DataLoaders that are specific to your setup.
+
+
+### DataSampler related issues
+
+During [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training we had a very complicated DataLoader which was suffering from image to text ratio fluctuations when the DataLoader was getting restored on resume, so we ended up having a small spike on each resume which would then recover:
+
+![](images/idefics-80b-tr-190-01-image2text.png)
+
+You can see the loss and ratio plots correlation here. As we had to resume about a dozen times we saw a lot of those spikes.
+
+
+
+
+
+### Impacts of repeat data
+
+I was training a variation of Llama2 and saw this super unusual spike that didn't diverge or recover but which switched to a new higher loss level:
+
+![](images/ptl-repeat-data-p1.png)
+
+I rolled back to just before the weird behavior occurred and restarted. The loss training progressed at the same loss level for a bit and then again spiked and shifted to a higher loss.
+
+![](images/ptl-repeat-data-p2.png)
+
+I have never seen this type of divergence before. I was scratching my head for a while and then decided to look at the bigger picture.
+
+[wandb](https://wandb.ai/) didn't handle resume data plotting correctly if a rollback was performed, that is it ignored all new data after the rollback until the steps of the old data have been overcome. This forced us to start a new wandb plot for every resume with a rollback so that new data is shown. And if you need to see the whole plot you have to stitch them together, which includes dead data points that are no longer true. So I did the stitching and saw this puzzle:
+
+![](images/ptl-repeat-data-p3.png)
+
+footnote: as of 2025 wandb can do this properly - [rewinding a run](https://docs.wandb.ai/models/runs/rewind) with `wandb.init(resume_from="<run_id>?_step=N")` truncates the history at step `N` and lets you log forward under the same run id, and `fork_from` does the same while leaving the original run intact (wandb SDK 0.17.1+; wandb recommends forking over rewinding for performance). Two catches: it's cloud-only - Multi-tenant and Dedicated Cloud, not Self-Managed - and it needs monotonically increasing steps, so it won't work alongside a non-monotonic `define_metric()`. Self-hosting, the stitching above is still the way.
+
+There was no real spike in the two earlier runs. The loss never went up in the first place. In both resumes it was under-reporting loss due to an exactly repeated data and then it reached data it hasn't seen before and started reporting correctly. In other words it was overfitting and reporting a false loss.
+
+The cause of the problem is data repetition, and since it clearly memorised some of it, it was reporting a better loss.
+
+The problem came from [pytorch-lightning](https://github.com/Lightning-AI/pytorch-lightning) not handling resumes correctly wrt DataSampler automatically - basically every time you resume you start your data stream from scratch. This, of course, requires a user to somehow fix the situation. You could change the seed to somewhat ameliorate the situation and avoid the exact data sequence, but it still leaves you with repeat data, which isn't what you want for any serious training (or ablation experiments, since your observation will be invalid, if they assume [IID data distribution](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables)).
+
+footnote: I discussed [this issue with the PTL developers](https://github.com/Lightning-AI/pytorch-lightning/issues/18780) and they said that they tried hard to come up with a generic solution but it wasn't meant to be. So the user needs to figure it out.
+
+Make sure to check your training framework documentation whether it handles the DataSampler resuming correctly. Make sure you didn't discover this problem after the training has finished and you ended up training 6x times the same 50B of tokens from the planned 300B tokens seen only once each.
+
+Doing a couple of resumes early on before embarking on the real training should also expose if there is a problem. Albeit, if the data gets reshuffled on each resume you are unlikely to see it. It'll only be seen if the seed is the same.
+
+
+# ==== debug/README.md ====
+
+# Debugging and Troubleshooting
+
+
+## Guides
+
+- [Debugging PyTorch programs](./pytorch.md)
+
+- [Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs](./pytorch.md#diagnosing-crashes-hangs-and-tracing-execution)
+
+- [Network Debug](../network/debug/)
+
+- [Troubleshooting NVIDIA GPUs](../compute/accelerator/nvidia/debug.md)
+
+- [Underflow and Overflow Detection](./pytorch.md#underflow-and-overflow-detection)
+
+
+
+## Tools
+
+- [Debug Tools](./tools.md)
+
+- [torch-distributed-gpu-test.py](./torch-distributed-gpu-test.py) - this a `torch.distributed` diagnostics
+  script that checks that all GPUs in the cluster (one or many nodes) can talk to each other and allocate gpu memory.
+
+- [NicerTrace](./NicerTrace.py) - this is an improved `trace` python module with multiple additional flags added to the constructor and more useful output.

docs/evidence/cleanrl_37_ppo_details.md

@@ -3,23 +3,796 @@
 Authors: Huang, Shengyi; Dossa, Rousslan Fernand Julien; Raffin, Antonin; Kanervisto, Anssi; Wang, Weixun.
 Source: ICLR Blog Track, 2022-03-25 — https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
 Code: https://github.com/vwxyzjn/ppo-implementation-details ; CleanRL: https://github.com/vwxyzjn/cleanrl
+Fetched-via: r.jina.ai reader, 2026-08-15 (CLAUDE agent)
+Fetch-status: full post text, all 37 details. Supersedes the earlier framing-passages excerpt. (CLAUDE agent)
 
-Excerpt cached for the ML-debugging skill (the full post is long; key framing passages below, verbatim).
+Why it matters here: the reference catalogue of undocumented implementation details that decide whether an RL reproduction works, each with a permanent link to the code line.
 
 ---
 
-> Instead of doing ablation studies and making recommendations on which details matter, this blog post takes a step back and focuses on reproductions of PPO's results in all accounts.
+25 Mar 2022 | [proximal-policy-optimization](https://iclr-blog-track.github.io/tags/#proximal-policy-optimization)[reproducibility](https://iclr-blog-track.github.io/tags/#reproducibility)[reinforcement-learning](https://iclr-blog-track.github.io/tags/#reinforcement-learning)[implementation-details](https://iclr-blog-track.github.io/tags/#implementation-details)[tutorial](https://iclr-blog-track.github.io/tags/#tutorial)
 
-> During our re-implementation, we have compiled an implementation checklist containing 37 details as follows. For each implementation detail, we display the permanent link to its code (which is not done in academic papers) and point out its literature connection.
+Jon is a first-year master’s student who is interested in reinforcement learning (RL). In his eyes, RL seemed fascinating because he could use RL libraries such as [Stable-Baselines3 (SB3)](https://github.com/DLR-RM/stable-baselines3) to train agents to play all kinds of games. He quickly recognized Proximal Policy Optimization (PPO) as a fast and versatile algorithm and wanted to implement PPO himself as a learning experience. Upon reading the paper, Jon thought to himself, “huh, this is pretty straightforward.” He then opened a code editor and started writing PPO. `CartPole-v1` from Gym was his chosen simulation environment, and before long, Jon made PPO work with `CartPole-v1`. He had a great time and felt motivated to make his PPO work with more interesting environments, such as the Atari games and MuJoCo robotics tasks. “How cool would that be?” he thought.
 
-The 37 details break down as:
-- 13 core implementation details
-- 9 Atari-specific implementation details
-- 9 implementation details for robotics tasks (continuous action spaces)
-- 5 LSTM implementation details
-- 1 `MultiDiscrete` action-spaces implementation detail
-- (plus 4 situational details not used in the official implementation)
+However, he soon struggled. Making PPO work with Atari and MuJoCo seemed more challenging than anticipated. Jon then looked for reference implementations online but was shortly overwhelmed: unofficial repositories all appeared to do things differently, whereas he just could not read the Tensorflow `1.x` code in the official repo. Fortunately, Jon stumbled across two recent papers that explain PPO’s implementations. “This is it!” he grinned. Failing to control his excitement, Jon started running around in the office, accidentally bumping into Sam, whom Jon knew was working on RL. They then had the following conversation:
 
-> Our ultimate purpose is to help people understand the PPO implementation through and through, reproduce past results with high fidelity, and facilitate customization for new research.
+*   “Hey, I just read the _implementation details matter_ paper and the _what matters in on-policy RL_ paper. Fascinating stuff. I knew PPO wasn’t that easy!” Jon exclaimed.
+*   “Oh yeah! PPO is tricky, and I love these two papers that dive into the nitty-gritty details.” Sam answered.
+*   “Indeed. I feel I understand PPO much better now. You have been working with PPO, right? Quiz me on PPO!” Jon inquired enthusiastically.
+*   “Sure. If you run the official PPO with the Atari game Breakout, the agent would get ~400 game scores in about 4 hours. Do you know how does PPO achieve that?”
+*   “Hmm… That’s actually a good question. I don’t think the two papers explain that.”
+*   “The procgen paper contains experiments conducted using the official PPO with LSTM. Do you know how does PPO + LSTM work?”
+*   “Ehh… I haven’t read too much on PPO + LSTM” Jon admitted.
+*   “The official PPO also works with `MultiDiscrete` action space where you can use multiple discrete values to describe an action. Do you know how that works?”
+*   “…” Jon, speechless.
+*   “Lastly, if you have only the standard tools (e.g., `numpy, gym...`) and a neural network library (e.g., `torch, jax,...`), could you code up PPO from scratch?”
+*   “Ooof, I guess it’s going to be difficult. Prior papers analyzed PPO implementation details but didn’t show how these pieces are coded together. Also, I now realize their conclusions are in MuJoCo tasks and do not necessarily transfer to other games such as Atari. I feel sad now…” Jon sighed.
+*   “Don’t feel bad. PPO is just a complicated beast. If anything helps, I have been making video tutorials on implementing PPO from scratch and a blog post explaining things in more depth!”
 
-Context: the official PPO implementation (`openai/baselines`, `ppo2`) has undergone several refactorings, so "it is important to recognize *which version* of the official implementation is worth studying." Libraries that match `ppo2`'s details closely (Stable-Baselines3, CleanRL) reproduce similar results; others report more diverse (worse) results.
+![Image 1](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//meme3.png)
+
+And the blog post is here! Instead of doing ablation studies and making recommendations on which details matter, this blog post takes a step back and focuses on reproductions of PPO’s results in all accounts. Specifically, this blog post complements prior work in the following ways:
+
+1.   **Genealogy Analysis:** we establish what it means to reproduce the **official PPO implementation** by examining its historical revisions in the `openai/baselines` GitHub repository (the official repository for PPO). As we will show, the code in the `openai/baselines` repository has undergone several refactorings which could produce different results from the original paper. So it is important to recognize _which version_ of the official implementation is worth studying.
+2.   **Video Tutorials and Single-file Implementations:** we make video tutorials on re-implementing PPO in PyTorch from scratch, matching details in the official PPO implementation to handle classic control tasks, Atari games, and MuJoCo tasks. Notably, we adopt single-file implementations in our code base, making the code quicker and easier to read. The videos are shown below:
+
+[Video 13](https://www.youtube.com/watch?v=MEt6rrxH8W4)[Video 14](https://www.youtube.com/watch?v=05RMTj-2K_Y)[Video 15](https://www.youtube.com/watch?v=BvZvx7ENZBw)
+
+1.   **Implementation Checklist with References:** During our re-implementation, we have compiled an implementation checklist containing 37 details as follows. For each implementation detail, we display the permanent link to its code (which is not done in academic papers) and point out its literature connection. 
+    *   13 core implementation details
+    *   9 Atari specific implementation details
+    *   9 implementation details for robotics tasks (with continuous action spaces)
+    *   5 LSTM implementation details
+    *   1 `MultiDiscrete` action spaces implementation detail
+
+2.   **High-fidelity Reproduction:** To validate our re-implementation, we show that the empirical results of our implementation match closely with those of the original, in classic control tasks, Atari games, MuJoCo tasks, LSTM, and Real-time Strategy (RTS) game tasks.
+3.   **Situational Implementation Details:** We also cover 4 implementation details not used in the official implementation but potentially useful on special occasions.
+
+Our ultimate purpose is to help people understand the PPO implementation through and through, reproduce past results with high fidelity, and facilitate customization for new research. To make research reproducible, we have made source code available at [https://github.com/vwxyzjn/ppo-implementation-details](https://github.com/vwxyzjn/ppo-implementation-details) and the tracked experiments available at [https://wandb.ai/vwxyzjn/ppo-details](https://wandb.ai/vwxyzjn/ppo-details)
+
+## Background
+
+PPO is a policy gradient algorithm proposed by [Schulman et al., (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017). As a refinement to Trust Region Policy Optimization (TRPO) ([Schulman et al., 2015](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015)), PPO uses a simpler clipped surrogate objective, omitting the expensive second-order optimization presented in TRPO. Despite this simpler objective, [Schulman et al., (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017) show PPO has higher sample efficiency than TRPO in many control tasks. PPO also has good empirical performance in the arcade learning environment (ALE) which contain Atari games.
+
+To facilitate more transparent research, [Schulman et al., (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017) have made the source code of PPO available in the `openai/baselines` GitHub repository with the code name `pposgd` (commit [da99706](https://github.com/openai/baselines/tree/da997060461e3cbf54ca4dc7a67081a731fb6b3b/baselines/pposgd) on 7/20/2017). Later, the `openai/baselines` maintainers have introduced a series of revisions. The key events include:
+
+1.   11/16/2017, commit [2dd7d30](https://github.com/openai/baselines/tree/2dd7d307d7d163a02b37c87c62b7949af02d99ad/baselines/ppo2): the maintainers introduced a refactored version `ppo2` and renamed `pposgd` to `ppo1`. According to a [GitHub issue](https://github.com/openai/baselines/issues/485#issuecomment-413722708), one maintainer suggests `ppo2` should offer better GPU utilization by batching observations from multiple simulation environments.
+2.   8/10/2018, commit [ea68f3b](https://github.com/openai/baselines/commits/ea68f3b7e6a20d4c6bf1e32f8fb5ce18e6ef3a89): after a few revisions, the maintainers evaluated `ppo2`, producing the [MuJoCo benchmark](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/benchmarks_mujoco1M.htm)
+3.   10/4/2018, commit [7bfbcf1](https://github.com/openai/baselines/commit/7bfbcf177eca8f46c0c0bfbb378e044539f5e061): after a few revisions, the maintainers evaluated `ppo2`, producing the [Atari benchmark](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/benchmarks_mujoco1M.htm)
+4.   1/31/2020, commit [ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998): the maintainers have merged the last commit to `openai/baselines` to date. To our knowledge, `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) is the base of many PPO-related resources: 
+    1.   RL libraries such [Stable-Baselines3 (SB3)](https://github.com/DLR-RM/stable-baselines3), [pytorch-a2c-ppo-acktr-gail](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail), and [CleanRL](https://github.com/vwxyzjn/cleanrl) have built their PPO implementation to match implementation details in `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) closely.
+    2.   Recent papers ([Engstrom, Ilyas, et al., 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom); [Andrychowicz, et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz)) have examined implementation details concerning robotics tasks in `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)).
+
+In recent years, reproducing PPO’s results has become a challenging issue. The following table collects the best-reported performance of PPO in popular RL libraries in Atari and MuJoCo environments.
+
+| RL Library | GitHub Stars | Benchmark Source | Breakout | Pong | BeamRider | Hopper | Walker2d | HalfCheetah |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| [Baselines](https://github.com/openai/baselines)`pposgd` / `ppo1` ([da99706](https://github.com/openai/baselines/tree/da997060461e3cbf54ca4dc7a67081a731fb6b3b/baselines/pposgd)) | [![Image 2: GitHub stars](https://img.shields.io/github/stars/openai/baselines)](https://github.com/openai/baselines/stargazers) | [paper](https://arxiv.org/abs/1707.06347) ($) | 274.8 | 20.7 | 1590 | ~2250 | ~3000 | ~1750 |
+| [Baselines](https://github.com/openai/baselines)`ppo2` ([7bfbcf1](https://github.com/openai/baselines/commit/7bfbcf177eca8f46c0c0bfbb378e044539f5e061) and [ea68f3b](https://github.com/openai/baselines/commits/ea68f3b7e6a20d4c6bf1e32f8fb5ce18e6ef3a89)) |  | [docs](https://github.com/openai/baselines/blob/master/benchmarks_atari10M.htm) (*) | 114.26 | 13.68 | 1299.25 | 2316.16 | 3424.95 | 1668.58 |
+| [Baselines](https://github.com/openai/baselines)`ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) |  | this blog post (*) | 409.265 ± 30.98 | 20.59 ± 0.40 | 2627.96 ± 625.751 | 2448.73 ± 596.13 | 3142.24 ± 982.25 | 2148.77 ± 1166.023 |
+| [Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) | [![Image 3: GitHub stars](https://img.shields.io/github/stars/DLR-RM/stable-baselines3)](https://github.com/DLR-RM/stable-baselines3/stargazers) | [docs](https://github.com/DLR-RM/rl-baselines3-zoo/blob/111d03c4ce728fff51d4b1c10355ea612bc8d456/benchmark.md) (0) (^) | 398.03 ± 33.28 | 20.98 ± 0.10 | 3397.00 ± 1662.36 | 2410.43 ± 10.02 | 3478.79 ± 821.70 | 5819.09 ± 663.53 |
+| [CleanRL](https://github.com/vwxyzjn/cleanrl) | [![Image 4: GitHub stars](https://img.shields.io/github/stars/vwxyzjn/cleanrl)](https://github.com/vwxyzjn/cleanrl/stargazers) | [docs](https://wandb.ai/cleanrl/cleanrl.benchmark/reports/Open-RL-Benchmark-0-6-0---Vmlldzo0MDcxOA) (1) (*) | ~402 | ~20.39 | ~2131 | ~2685 | ~3753 | ~1683 |
+| [Tianshou](https://github.com/thu-ml/tianshou) | [![Image 5: GitHub stars](https://img.shields.io/github/stars/thu-ml/tianshou)](https://github.com/thu-ml/tianshou/stargazers) | [paper](https://arxiv.org/pdf/2107.14171.pdf), [docs](https://github.com/thu-ml/tianshou/blob/f13e415eb0de55baca5dc0d6fae39d6a38e8bc0b/examples/atari/README.md) (5) (^) | ~400 | ~20 | - | 7337.4 ± 1508.2 | 3127.7 ± 413.0 | 4895.6 ± 704.3 |
+| [Ray/RLlib](https://github.com/ray-project/ray/tree/master/rllib/) | [![Image 6: GitHub stars](https://img.shields.io/github/stars/ray-project/ray)](https://github.com/ray-project/ray/stargazers) | [repo](https://github.com/ray-project/rl-experiments/tree/9543891717cd0f8e137e23812229a06f8ed1c6c2) (2) (*) | 201 | - | 4480 | - | - | 9664 |
+| [SpinningUp](https://github.com/openai/spinningup) | [![Image 7: GitHub stars](https://img.shields.io/github/stars/openai/spinningup)](https://github.com/openai/spinningupstargazers) | [docs](https://spinningup.openai.com/en/latest/spinningup/bench.html#id12) (3) (^) | - | - | - | ~2500 | ~2500 | ~3000 |
+| [ChainerRL](https://github.com/chainer/chainerrl) | [![Image 8: GitHub stars](https://img.shields.io/github/stars/chainer/chainerrl)](https://github.com/chainer/chainerrl/stargazers) | [paper](https://arxiv.org/pdf/1912.03905.pdf) (4) (*) | - | - | - | 2719 ± 67 | 2994 ± 113 | 2404 ± 185 |
+| [Tonic](https://github.com/fabiopardo/tonic) | [![Image 9: GitHub stars](https://img.shields.io/github/stars/fabiopardo/tonic)](https://github.com/fabiopardo/tonic/stargazers) | [paper](https://arxiv.org/pdf/2011.07537.pdf) (6) (^) | - | - | - | ~2000 | ~4500 | ~5000 |
+
+(-): No publicly reported metrics available 
+
+($): The experiments uses the v1 MuJoCo environments 
+
+(*): The experiments uses the v2 MuJoCo environments 
+
+(^): The experiments uses the v3 MuJoCo environments 
+
+(0): 1M steps for MuJoCo experiments, 10M steps for Atari games, 1 random seed 
+
+(1): 2M steps for MuJoCo experiments, 10M steps for Atari games, 2 random seeds 
+
+(2): 25M steps and 10 workers (5 envs per worker) for Atari experiments; 44M steps and 16 workers for MuJoCo experiments; 1 random seed 
+
+(3): 3M steps, PyTorch version, 10 random seeds 
+
+(4): 2M steps, 10 random seeds 
+
+(5): 3M steps, 10 random seeds for MuJoCo experiments; 10M steps, 1 random seed for Atari experiment 
+
+(6): 5M steps, 10 random seeds
+
+We offer several observations.
+
+1.   These revisions in `openai/baselines` are not without performance consequences. Reproducing PPO’s results is challenging partly because even the original implementation could produce inconsistent results.
+2.   `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) and libraries matching its implementation details have reported rather similar results. In comparison, other libraries have usually reported more diverse results.
+3.   Interestingly, we have found many libraries reported performance in MuJoCo tasks but not in Atari tasks.
+
+Despite the complicated situation, we have found `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) as an implementation worth studying. It obtains good performance in both Atari and MuJoCo tasks. More importantly, it also incorporates advanced features such as LSTM and treatment of the `MultiDiscrete` action space, unlocking application to more complicated games such as Real-time Strategy games. As such, we define `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) as the **official PPO implementation** and base the remainder of this blog post on this implementation.
+
+## Reproducing the official PPO implementation
+
+In this section, we introduce five categories of implementation details and implement them in PyTorch from scratch.
+
+*   13 core implementation details
+*   9 Atari specific implementation details
+*   9 implementation details for robotics tasks (with continuous action spaces)
+*   5 LSTM implementation details
+*   1 `MultiDiscrete` implementation detail
+
+For each category (except the first one), we benchmark our implementation against the original implementation in three environments, each with three random seeds.
+
+## 13 core implementation details
+
+We first introduce the 13 core implementation details commonly used regardless of the tasks. To help understand how to code these details in PyTorch, we have prepared a line-by-line video tutorial as follows. Note that the video tutorial skips over the 12-th and 13-th implementation details during its making, hence the video has the title “11 Core Implementation Details”
+
+[Video 16](https://www.youtube.com/watch?v=MEt6rrxH8W4)
+
+1.   Vectorized architecture ([common/cmd_util.py#L22](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L22)) Code-level Optimizations
+    *   PPO leverages an efficient paradigm known as the **vectorized architecture** that features a single learner that collects samples and learns from multiple environments. Below is a pseudocode: ```
+envs = VecEnv(num_envs=N)
+  agent = Agent()
+  next_obs = envs.reset()
+  next_done = [0, 0, ..., 0] # of length N
+  for update in range(1, total_timesteps // (N*M)):
+      data = []
+      # ROLLOUT PHASE
+      for step in range(0, M):
+          obs = next_obs
+          done = next_done
+          action, other_stuff = agent.get_action(obs)
+          next_obs, reward, next_done, info = envs.step(
+              action
+          ) # step in N environments
+          data.append([obs, action, reward, done, other_stuff]) # store data
+
+      # LEARNING PHASE
+      agent.learn(data, next_obs, next_done) # `len(data) = N*M`
+``` 
+    *   In this architecture, PPO first initializes a **vectorized environment**`envs` that runs $N$ (usually independent) environments either sequentially or in parallel by leveraging multi-processes. `envs` presents a synchronous interface that always outputs a batch of $N$ observations from $N$ environments, and it takes a batch of $N$ actions to step the $N$ environments. When calling `next_obs = envs.reset()`, `next_obs` gets a batch of $N$ initial observations (pronounced “next observation”). PPO also initializes an environment done flag variable `next_done` (pronounced “next done”) to an $N$-length array of zeros, where its i-th element `next_done[i]` has values of 0 or 1 which corresponds to the $i$-th sub-environment being _not done_ and _done_, respectively.
+    *   Then, the vectorized architecture loops two phases: the **rollout phase** and the **learning phase**: 
+        *   Rollout phase : The agent samples actions for the $N$ environments and continue to step them for a fixed number of $M$ steps. During these $M$ steps, the agent continues to append relevant data in an empty list `data`. If the $i$-th sub-environment is done (terminated or truncated) after stepping with the $i$-th action `action[i]`, `envs` would set its returned `next_done[i]` to 1, auto-reset the $i$-th sub-environment and fill `next_obs[i]` with the initial observation in the new episode of the $i$-th environment.
+        *   Learning phase: The agent in principal learns from the collected data in the rollout phase: `data` of length $N M$, `next_obs` and `done`. Specifically, PPO can estimate value for the next observation `next_obs` conditioned on `next_done` and calculate the advantage `advantages` and the return `returns`, both of which also has length $N M$. PPO then learns from the prepared data `[data, advantages, returns]`, which is called “fixed-length trajectory segments” by [(Schulman et al., 2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017).
+        *   **It is important to understand `next_obs` and `next_done`’s role to help transition between phases**: At the end of the $j$-th rollout phase, `next_obs` can be used to estimate the value of the final state during learning phase, and in the begining of the $\left(\right. j + 1 \left.\right)$-th rollout phase, `next_obs` becomes the initial observation in `data`. Likewise, `next_done` tells if `next_obs` is actually the first observation of a new episode. This intricate design allows PPO to continue step the sub-environments, and because agent always learns from fixed-length trajectory segments after $M$ steps, PPO can train the agent even if the sub-environments never terminate or truncate. This is in principal why PPO can learn in long-horizon games that last 100,000 steps [(default truncation limit for Atari games in `gym`)](https://github.com/openai/gym/blob/a7b6462136ebaa610c8941e4da8a9c92155b04d1/gym/envs/__init__.py#L744) in a single episode.
+
+    *   A common incorrect implementation is to train PPO based on episodes and setting a maximum episode horizon. Below is a pseudocode. ```
+env = Env()
+  agent = Agent()
+  for episode in range(1, num_episodes):
+      next_obs = env.reset()
+      data = []
+      for step in range(1, max_episode_horizon):
+          obs = next_obs
+          action, other_stuff = agent.get_action(obs)
+          next_obs, reward, done, info = env.step(action)
+          data.append([obs, action, reward, done, other_stuff]) # store data
+          if done:
+              break
+      agent.learn(data)
+``` 
+        *   There are several downsides to this approach. First, it can be inefficient because the agent has to do one forward pass per environment step. Second, it does not scale to games with larger horizons such as StarCraft II (SC2). A single episode of the SC2 could last 100,000 steps, which bloats the memory requirement in this implementation.
+        *   The vectorized architecture handles this 100,000 steps by learning from **fixed-length trajectory segments**. If we set $N = 2$ and $M = 100$, the agent would learn from the first 100 steps from 2 independent environments. Then, note that the `next_obs` is the 101st observation from these two environments, and the agent can keep doing rollouts and learn from the 101 to 200 steps from the 2 environments. Essentially, the agent learns partial trajectories of the episode, $M$ steps at a time.
+
+    *   $N$ is the `num_envs` (decision C1) and $M * N$ is the `iteration_size` (decision C2) in [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz), who suggest increasing $N$ (such as $N = 256$) boosts the training throughput but makes the performance worse. They argued the performance deterioration was due to “shortened experience chunks” ($M$ becomes smaller due to the increase in $N$ in their setup ) and “earlier value bootstrapping.” While we agree increasing $N$ could hurt sample efficiency, we argue the evaluation should be based on wall-clock time efficiency. That is, if the algorithm terminates much sooner with a larger $N$ compared to other configurations, why not run the algorithm longer? Although being a different robotics simulator, [Brax](https://github.com/google/brax) follows this idea and can train a viable agent in similar tasks with PPO using a massive $N = 2048$ and a small $M = 20$ yet finish the training in one minute.
+    *   The vectorized environments also support multi-agent reinforcement learning (MARL) environments. Below is the quote from ([gym3](https://github.com/openai/gym3)) using our notation: 
+> In the simplest case, a vectorized environment corresponds to a single multiplayer game with $N$ players. If we run an RL algorithm in this environment, we are doing self-play without historical opponents. This setup can be straightforwardly extended to having $K$ concurrent games with $H$ players each, with $N = H * K$.
+
+        *   For example, if there is a two-player game, we can create a vectorized environment that spawns two sub-environments. Then, the vectorized environment produces a batch of two observations, where the first observation is from player 1 and the second observation is from player 2. Next, the vectorized environment takes a batch of two actions and tells the game engine to let player 1 execute the first action and player 2 execute the second action. Consequently, PPO learns to control both player 1 and player 2 in this vectorized environment.
+        *   Such MARL usage is widely adopted in games such as Gym-μRTS ([Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021)), Pettingzoo ([Terry et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Terry)), etc.
+
+2.   Orthogonal Initialization of Weights and Constant Initialization of biases ([a2c/utils.py#L58)](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L58)) Neural Network Code-level Optimizations
+    *   The related code is across multiple files in the `openai/baselines` library. The code for such initialization is in [a2c/utils.py#L58](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L58), when in fact it is used for other algorithms such as PPO. In general, the weights of _hidden_ layers use orthogonal initialization of weights with scaling `np.sqrt(2)`, and the biases are set to `0`, as shown in the CNN initialization for Atari ([common/models.py#L15-L26](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L15-L26)), and the MLP initialization for Mujoco ([common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103)). However, the policy output layer weights are initialized with the scale of `0.01`. The value output layer weights are initialized with the scale of `1` ([common/policies.py#L49-L63](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L49-L63)).
+    *   It seems the implementation of the orthogonal initialization of `openai/baselines` ([a2c/utils.py#L20-L35](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L20-L35)) is different from that of pytorch/pytorch ([torch.nn.init.orthogonal_](https://pytorch.org/docs/stable/_modules/torch/nn/init.html#orthogonal_)). However, we consider this to be a very low-level detail that should not impact the performance.
+    *   [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find orthogonal initialization to outperform the default Xavier initialization in terms of the highest episodic return achieved. Also, [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find centering the action distribution around 0 (i.e., initialize the policy output layer weights with 0.01”) to be beneficial (decision C57).
+
+3.   The Adam Optimizer’s Epsilon Parameter ([ppo2/model.py#L100](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L100)) Code-level Optimizations
+    *   PPO sets the epsilon parameter to `1e-5`, which is different from the default epsilon of `1e-8` in PyTorch and `1e-7` in TensorFlow. We list this implementation detail because the epsilon parameter is neither mentioned in the paper nor a configurable parameter in the PPO implementation. While this implementation detail may seem over specific,it is important that we match it for a high-fidelity reproduction.
+    *   [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) perform a grid search on Adam optimizer’s parameters (decision C24, C26, C28) and recommend $\beta_{1} = 0.9$ and use the Tensorflow’s default epsilon parameter `1e-7`. [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) use the default PyTorch epsilon parameter `1e-8`.
+
+4.   Adam Learning Rate Annealing ([ppo2/ppo2.py#L133-L135](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L133-L135)) Code-level Optimizations
+    *   The Adam optimizer’s learning rate could be either constant or set to decay. By default, the hyper-parameters for training agents playing Atari games set the learning rate to linearly decay from `2.5e-4` to `0` as the number of timesteps increases. In MuJoCo, the learning rate linearly decays from `3e-4` to `0`.
+    *   [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find adam learning rate annealing to help agents obtain higher episodic return. Also, [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) have also found learning rate annealing helpful as it increases performance in 4 out of 5 tasks examined, although the performance gains are relatively small (decision C31, figure 65).
+
+5.   Generalized Advantage Estimation ([ppo2/runner.py#L56-L65](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L56-L65)) Theory
+    *   Although the PPO paper uses the abstraction of advantage estimate in the PPO’s objective, the PPO implementation does use Generalized Advantage Estimation ([Schulman, 2015b](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015b)). Two important sub-details: 
+        *   Value bootstrap ([ppo2/runner.py#L50](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L50)): if a sub-environment is _not_ terminated nor truncated, PPO estimates the value of the next state in this sub-environment as the value target. 
+            *   **A note on truncation**: Almost all `gym` environments have a time limit and will truncate themselves if they run too long. For example, the `CartPole-v1` has a 500 time limit (see [link](https://github.com/openai/gym/blob/e9df4932434516c9f7956cc8010679a33835b204/gym/envs/__init__.py#L26)) and will return `done=True` if the game lasts for more than 500 steps. While the PPO implementation does not estimate value of the terminal state in the truncated environments, we (intuitively) should. Nonetheless, for high-fidelity reproduction, we did not implement the correct handling for truncated environments.
+
+        *   $T D \left(\right. \lambda \left.\right)$ return estimation ([ppo2/runner.py#L65](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L65)): PPO implements the return target as `returns = advantages + values`, which corresponds to $T D \left(\right. \lambda \left.\right)$ for value estimation (where Monte Carlo estimation is a special case when $\lambda = 1$).
+
+    *   [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find GAE to performan better than N-step returns (decision C6, figure 44 and 40).
+
+6.   Mini-batch Updates ([ppo2/ppo2.py#L157-L166](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L157-L166)) Code-level Optimizations
+    *   During the learning phase of the vectorized architecture, the PPO implementation shuffles the indices of the training data of size $N * M$ and breaks it into mini-batches to compute the gradient and update the policy.
+    *   Some common mis-implementations include 1) always using the whole batch for the update, and 2) implementing mini-batches by randomly fetching from the training data (which does not guarantee all training data points are fetched).
+
+7.   Normalization of Advantages ([ppo2/model.py#L139](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L139)) Code-level Optimizations
+    *   After calculating the advantages based on GAE, PPO normalizes the advantages by subtracting their mean and dividing them by their standard deviation. In particular, _this normalization happens at the minibatch level instead of the whole batch level!_
+    *   [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) (decision C67) find per-minibatch advantage normalization to not affect performance much (figure 35).
+
+8.   Clipped surrogate objective ([ppo2/model.py#L81-L86](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L81-L86)) Theory
+    *   PPO clips the objective as suggested in the paper.
+    *   [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find the PPO’s clipped objective to have similar performance to TRPO’s objective when they controlled other implementation details to be the same. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find the PPO’s clipped objective to outperform vanilla policy gradient (PG), V-trace, AWR, and V-MPO in most tasks ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)).
+    *   Based on the above findings, we argue PPO’s clipped objective is still a great objective because it achieves similar performance as TRPO’s objective while being computationally cheaper (i.e., without second order optimization as does in TRPO).
+
+9.   Value Function Loss Clipping ([ppo2/model.py#L68-L75](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L68-L75)) Code-level Optimizations
+    *   PPO clips the value function like the PPO’s clipped surrogate objective. Given the `V_{targ} = returns = advantages + values`, PPO fits the the value network by minimizing the following loss:
+
+$$
+L^{V} = max \left[\right. \left(\left(\right. V_{\theta_{t}} - V_{t a r g} \left.\right)\right)^{2} , \left(\left(\right. clip ⁡ \left(\right. V_{\theta_{t}} , V_{\theta_{t - 1}} - \epsilon , V_{\theta_{t - 1}} + \epsilon \left.\right) - V_{t a r g} \left.\right)\right)^{2} \left]\right.
+$$
+
+    *   [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find no evidence that the value function loss clipping helps with the performance. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) suggest value function loss clipping even hurts performance (decision C13, figure 43).
+    *   We implemented this detail because this work is more about high-fidelity reproduction of prior results.
+
+10.   Overall Loss and Entropy Bonus ([ppo2/model.py#L91](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L91)) Theory
+    *   The overall loss is calculated as `loss = policy_loss - entropy * entropy_coefficient + value_loss * value_coefficient`, which maximizes an entropy bonus term. Note that the policy parameters and value parameters share the same optimizer.
+    *   Mnih et al. have reported this entropy bonus to improve exploration by encouraging the action probability distribution to be slightly more random.
+    *   [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) overall find no evidence that the entropy term improves performance on continuous control environments (decision C13, figure 76 and 77).
+
+11.   Global Gradient Clipping ([ppo2/model.py#L102-L108](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L102-L108)) Code-level Optimizations
+    *   For each update iteration in an epoch, PPO rescales the gradients of the policy and value network so that the “global l2 norm” (i.e., the norm of the concatenated gradients of all parameters) does not exceed `0.5`.
+    *   [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find global gradient clipping to offer a small performance boost (decision C68, figure 34).
+
+12.   Debug variables ([ppo2/model.py#L115-L116](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L115-L116)) 
+    *   The PPO implementation comes with several debug variables, which are 
+        1.   `policy_loss`: the mean policy loss across all data points.
+        2.   `value_loss`: the mean value loss across all data points.
+        3.   `entropy_loss`: the mean entropy value across all data points.
+        4.   `clipfrac`: the fraction of the training data that triggered the clipped objective.
+        5.   `approxkl`: the approximate Kullback–Leibler divergence, measured by `(-logratio).mean()`, which corresponds to the `k1` estimator in John Schulman’s blog post on [approximating KL divergence](http://joschu.net/blog/kl-approx.html). This blog post also suggests using an alternative estimator `((ratio - 1) - logratio).mean()`, which is unbiased and has less variance.
+
+13.   Shared and separate MLP networks for policy and value functions ([common/policies.py#L156-L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L156-L160), [baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103))Neural Network Code-level Optimizations
+    *   By default, PPO uses a simple MLP network consisting of two layers of 64 neurons and Hyperbolic Tangent as the activation function. Then PPO builds a policy head and value head that share the outputs of the MLP network. Below is a pseudocode: ```
+network = Sequential(
+      layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+      Tanh(),
+      layer_init(Linear(64, 64)),
+      Tanh(),
+  )
+  value_head = layer_init(Linear(64, 1), std=1.0)
+  policy_head = layer_init(Linear(64, envs.single_action_space.n), std=0.01)
+  hidden = network(observation)
+  value = value_head(hidden)
+  action = Categorical(policy_head(hidden)).sample()
+``` 
+    *   Alternatively, PPO could build a policy function and a value function using separate networks by toggling the `value_network='copy'` argument. Then the pseudocode looks like this: ```
+value_network = Sequential(
+      layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+      Tanh(),
+      layer_init(Linear(64, 64)),
+      Tanh(),
+      layer_init(Linear(64, 1), std=1.0),
+  )
+  policy_network = Sequential(
+      layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+      Tanh(),
+      layer_init(Linear(64, 64)),
+      Tanh(),
+      layer_init(Linear(64, envs.single_action_space.n), std=0.01),
+  )
+  value = value_network(observation)
+  action = Categorical(policy_network(observation)).sample()
+``` 
+
+We incorporate the first 12 details and the **separate-networks architecture** to produce a self-contained `ppo.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo.py)) that has 322 lines of code. Then, we make about [10 lines of code](https://www.diffchecker.com/07TdfFlg) change to adopt the **shared-network architecture**, resulting in a self-contained `ppo_shared.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_shared.py)) that has 317 lines of code. The following shows the file difference between the `ppo.py` (left) and `ppo_shared.py` (right).
+
+Below are the benchmarked results.
+
+![Image 10](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//CartPole-v1.png)![Image 11](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Acrobot-v1.png)![Image 12](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//MountainCar-v0.png)
+
+> Tracked classic control experiments (click to show the interactive panel)
+
+While shared-network architecture is the default setting in PPO, the separate-networks architecture clearly outperforms in simpler environments. The shared-network architecture performs worse probably due to the competing objectives of the policy and value functions. For this reason, we implement the separate-networks architecture in the video tutorial.
+
+## 9 Atari-specific implementation details
+
+Next, we introduce the 9 Atari-specific implementation details. To help understand how to code these details in PyTorch, we have prepared a line-by-line video tutorial.
+
+[Video 17](https://www.youtube.com/watch?v=05RMTj-2K_Y)
+
+1.   The Use of `NoopResetEnv` ([common/atari_wrappers.py#L12](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L12)) Environment Preprocessing
+    *   This wrapper samples initial states by taking a random number (between 1 and 30) of no-ops on reset.
+    *   The source of this wrapper comes from [(Mnih et al., 2015, Extended Data Table 1)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) and [Machado et al., 2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Machado2018) have suggested `NoopResetEnv` is a way to inject stochasticity to the environment.
+
+2.   The Use of `MaxAndSkipEnv` ([common/atari_wrappers.py#L97](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L97)) Environment Preprocessing
+    *   This wrapper skips 4 frames by default, repeats the agent’s last action on the skipped frames, and sums up the rewards in the skipped frames. Such frame-skipping technique could considerably speed up the algorithm because the environment step is computationally cheaper than the agent’s forward pass [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015).
+    *   This wrapper also returns the maximum pixel values over the last two frames to help deal with some Atari game quirks [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015).
+    *   The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below. 
+> More precisely, the agent sees and selects actions on every $k$-th frame instead of every frame, and its last action is repeated on skipped frames. Because running the emulator forward for one step requires much less computation than having the agent select an action, this technique allows the agent to play roughly $k$ times more games without significantly increasing the runtime. We use $k = 4$ for all games. […] First, to encode a single frame we take the maximum value for each pixel color value over the frame being encoded and the previous frame. This was necessary to remove flickering that is present in games where some objects appear only in even frames while other objects appear only in odd frames, an artifact caused by the limited number of sprites Atari 2600 can display at once.
+
+3.   The Use of `EpisodicLifeEnv` ([common/atari_wrappers.py#L61](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L61)) Environment Preprocessing
+    *   In the games where there are a life counter such as breakout, this wrapper marks the end of life as the end of episode.
+    *   The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below. 
+> For games where there is a life counter, the Atari 2600 emulator also sends the number of lives left in the game, which is then used to mark the end of an episode during training.
+
+    *   Interestingly, [(Bellemare et al., 2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/Bellemare2016b) Note this the wrapper could be detrimental to the agent’s performance and [Machado et al., 2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Machado2018) have suggested not using this wrapper.
+
+4.   The Use of `FireResetEnv` ([common/atari_wrappers.py#L41](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L41)) Environment Preprocessing
+    *   This wrapper takes the `FIRE` action on reset for environments that are fixed until firing.
+    *   This wrapper is interesting because there is no literature reference to our knowledge. According to anecdotal conversations([openai/baselines#240](https://github.com/openai/baselines/issues/240)), neither people from DeepMind nor OpenAI know where this wrapper comes from. So… ![Image 13](https://cdn.imgbin.com/21/1/17/imgbin-illuminati-symbol-shadow-government-spinner-3E2tJSxu7Zx6yaTffaaSZK2Wj.jpg)
+
+5.   The Use of `WarpFrame` (Image transformation) [common/atari_wrappers.py#L134](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L134)Environment Preprocessing
+    *   This wrapper warps extracts the Y channel of the 210x160 pixel images and resizes it to 84x84.
+    *   The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below. 
+> Second, we then extract the Y channel, also known as luminance, from the RGB frame and rescale it to 84x84.
+
+    *   In our implementation, we use the following wrappers to achieve the same purpose. ```
+env = gym.wrappers.ResizeObservation(env, (84, 84))
+  env = gym.wrappers.GrayScaleObservation(env)
+``` 
+
+6.   The Use of `ClipRewardEnv` ([common/atari_wrappers.py#L125](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L125)) Environment Preprocessing
+    *   This wrapper bins reward to `{+1, 0, -1}` by its sign.
+    *   The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below. 
+> As the scale of scores varies greatly from game to game, we clipped all positive rewards at 1 and all negative rewards at -1, leaving 0 rewards unchanged. Clipping the rewards in this manner limits the scale of the error derivatives and makes it easier to use the same learning rate across multiple games. At the same time, it could affect the performance of our agent since it cannot differentiate between rewards of different magnitude.
+
+7.   The Use of `FrameStack` ([common/atari_wrappers.py#L188](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L188)) Environment Preprocessing
+    *   This wrapper stacks $m$ last frames such that the agent can infer the velocity and directions of moving objects.
+    *   The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below. 
+> The function $\theta$ from algorithm 1 described below applies this preprocessing to the $m$ most recent frames and stacks them to produce the input to the Q-function, in which $m = 4$.
+
+8.   Shared Nature-CNN network for the policy and value functions ([common/policies.py#L157](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L157), [common/models.py#L15-L26](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L15-L26))Neural Network
+    *   For Atari games, PPO uses the same Convolutional Neural Network (CNN) in [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) along with the layer initialization technique mentioned earlier ([baselines/a2c/utils.py#L52-L53](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L52-L53)) to extract features, flatten the extracted features, apply a linear layer to compute the hidden features. Afterward, the policy and value functions share parameters by constructing a policy head and a value head using the hidden features. Below is a pseudocode: ```
+hidden = Sequential(
+      layer_init(Conv2d(4, 32, 8, stride=4)),
+      ReLU(),
+      layer_init(Conv2d(32, 64, 4, stride=2)),
+      ReLU(),
+      layer_init(Conv2d(64, 64, 3, stride=1)),
+      ReLU(),
+      Flatten(),
+      layer_init(Linear(64 * 7 * 7, 512)),
+      ReLU(),
+  )
+  policy = layer_init(Linear(512, envs.single_action_space.n), std=0.01)
+  value = layer_init(Linear(512, 1), std=1)
+``` 
+    *   Such a parameter-sharing paradigm obviously computes faster when compared to setting completely separate networks, which would look like the following. ```
+policy = Sequential(
+      layer_init(Conv2d(4, 32, 8, stride=4)),
+      ReLU(),
+      layer_init(Conv2d(32, 64, 4, stride=2)),
+      ReLU(),
+      layer_init(Conv2d(64, 64, 3, stride=1)),
+      ReLU(),
+      Flatten(),
+      layer_init(Linear(64 * 7 * 7, 512)),
+      ReLU(),
+      layer_init(Linear(512, envs.single_action_space.n), std=0.01)
+  )
+  value = Sequential(
+      layer_init(Conv2d(4, 32, 8, stride=4)),
+      ReLU(),
+      layer_init(Conv2d(32, 64, 4, stride=2)),
+      ReLU(),
+      layer_init(Conv2d(64, 64, 3, stride=1)),
+      ReLU(),
+      Flatten(),
+      layer_init(Linear(64 * 7 * 7, 512)),
+      ReLU(),
+      layer_init(Linear(512, 1), std=1)
+  )
+``` 
+    *   However, recent work suggests balancing the competing policy and value objective could be problematic, which is what methods like Phasic Policy Gradient are trying to address ([Cobbe et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe2021)).
+
+9.   Scaling the Images to Range [0, 1] ([common/models.py#L19](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/models.py#L19)) Environment Preprocessing
+    *   The input data has the range of [0,255], but it is divided by 255 to be in the range of [0,1].
+    *   Our anecdotal experiments found this scaling important. Without it, the first policy update results in the Kullback–Leibler divergence explosion, likely due to how the layers are initialized.
+
+To run the experiments, we match the hyperparameters used in the original implementation as follows.
+
+```
+# https://github.com/openai/baselines/blob/master/baselines/ppo2/defaults.py
+def atari():
+    return dict(
+        nsteps=128, nminibatches=4,
+        lam=0.95, gamma=0.99, noptepochs=4, log_interval=1,
+        ent_coef=.01,
+        lr=lambda f : f * 2.5e-4,
+        cliprange=0.1,
+    )
+```
+
+These hyperparameters are
+
+*   `nsteps` is the $M$ explained in this blog post .
+*   `nminibatches` is the number of minibatches used for update (i.e., our 6th implementation detail).
+*   `lam` is the GAE’s $\lambda$ parameter.
+*   `gamma` is the discount factor.
+*   `noptepochs` is the $K$ epochs in the original PPO paper.
+*   `ent_coef` is the `entropy_coefficient` in our 10th implementation detail.
+*   `lr=lambda f : f * 2.5e-4` is a learning rate schedule (i.e., our 4th implementation detail)
+*   `cliprange=0.1` is the clipping parameter $\epsilon$ in the original PPO paper.
+
+Note that the number of environments parameter $N$ (i.e., `num_envs`) is set to the number of CPUs in the computer ([common/cmd_util.py#L167](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L167)), which is strange. We have chosen instead to match the `N=8` used in the paper (the paper listed the parameter as “number of actors, 8”).
+
+As shown below, we make [~40 lines of code](https://www.diffchecker.com/Dq5NfuQH) change to `ppo.py` to incorporate these 9 details, resulting in a self-contained `ppo_atari.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_atari.py)) that has 339 lines of code. The following shows the file difference between the `ppo.py` (left) and `ppo_atari.py` (right).
+
+Below are the benchmarked results.
+
+![Image 14](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Breakout.svg)![Image 15](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Pong.svg)![Image 16](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//BeamRider.svg)
+
+> Tracked Atari experiments (click to show the interactive panel)
+
+## 9 details for continuous action domains (e.g. Mujoco)
+
+Next, we introduce the 9 details for continuous action domains such as MuJoCo tasks. To help understand how to code these details in PyTorch, we have prepared a line-by-line video tutorial. Note that the video tutorial skips over the 4-th implementation detail during its making, hence the video has the title “8 Details for Continuous Actions”
+
+[Video 18](https://www.youtube.com/watch?v=BvZvx7ENZBw)
+
+1.   Continuous actions via normal distributions ([common/distributions.py#L103-L104](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L103-L104)) Theory
+    *   Policy gradient methods (including PPO) assume the continuous actions are sampled from a normal distribution. So to create such distribution, the neural network needs to output the mean and standard deviation of the continuous action.
+    *   It is very popular to choose Gaussian distribution to represent the action distribution when the reinforcement learning algorithm is implemented in the environment of continuous action space. For example: [Schulman et al., (2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015) and [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016).
+
+2.   State-independent log standard deviation ([common/distributions.py#L104](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L104)) Theory
+    *   The implementation outputs the logits for the mean, but instead of outputting the logits for the standard deviation, it outputs the _logarithm_ of the standard deviation. In addition, this `log std` is set to be _state-independent and initialized to be 0._
+    *   [Schulman et al., (2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015) and [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016) use state-independent standard deviation, while [Haarnoja et al., (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Haarnoja2018) uses the state-dependent standard deviation, that is, the mean and standard deviation are output at the same time. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) compared two different implementations and found that the performance is very close (decision C59, figure 23).
+
+3.   Independent action components ([common/distributions.py#L238-L246](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L238-L246)) Theory
+    *   In many robotics tasks, it is common to have multiple scalar values to represent a continuous action. For example, the action of $a_{t} = \left[\right. a_{t}^{1} , a_{t}^{2} \left]\right. = \left[\right. 2.4 , 3.5 \left]\right.$ might mean to move left for 2.4 meters and move up 3.5 meters. However, most literature on policy gradient suggests the action $a_{t}$ would be a single scalar value. To account for this difference, PPO treats $\left[\right. a_{t}^{1} , a_{t}^{2} \left]\right.$ as probabilistically independent action components, therefore calculating $p r o b \left(\right. a_{t} \left.\right) = p r o b \left(\right. a_{t}^{1} \left.\right) \cdot p r o b \left(\right. a_{t}^{2} \left.\right)$.
+    *   This approach comes from the currently commonly used assumption: Gaussian distribution with full covariance is used to represent the policy, which means that the action selection for each dimension is performed independently. When facing the environment of multi-dimensional action space, [Tavakoli, et al. (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Tavakoli2018) also believes that each action dimension should be selected independently and to achieve this goal by designing a network structure. Although our intuition tells us that there may be dependencies between action choices in different dimensions of policies in some environments, what is the optimal choice is still an open question. It is worth noting that this question has attracted the attention of the community, and began to try to model the dependencies of actions in different dimensions, such as using auto-regressive policy ([Metz, et al. (2019)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Metz2019), [Zhang, et al. (2019)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Zhang2018))
+
+4.   Separate MLP networks for policy and value functions ([common/policies.py#L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L160), [baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103))Neural Network
+    *   For continuous control tasks, PPO uses a simple MLP network consisting of two layers of 64 neurons and Hyperbolic Tangent as the activation function ([baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103)) for both the policy and value functions ([common/policies.py#L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L160)). Below is a pseudocode (also combining previous 3 details): ```
+value_network = Sequential(
+      layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+      Tanh(),
+      layer_init(Linear(64, 64)),
+      Tanh(),
+      layer_init(Linear(64, 1), std=1.0),
+  )
+  policy_mean = Sequential(
+      layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+      Tanh(),
+      layer_init(Linear(64, 64)),
+      Tanh(),
+      layer_init(Linear(64, envs.single_action_space.n), std=0.01),
+  )
+  policy_logstd = nn.Parameter(torch.zeros(1, np.prod(envs.single_action_space.shape)))
+  value = value_network(observation)
+  probs = Normal(
+      policy_mean(x),
+      policy_logstd.expand_as(action_mean).exp(),
+  )
+  action = probs.sample()
+  logprob = probs.log_prob(action).sum(1)
+``` 
+    *   [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find the separate policy and value networks generally lead to better performance (decision C47, figure 15).
+
+5.   Handling of action clipping to valid range and storage ([common/cmd_util.py#L99-L100](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L99-L100)) Code-level Optimizations
+    *   After a continuous action is sampled, such action could be invalid because it could exceed the valid range of continuous actions in the environment. To avoid this, add applies the rapper to clip the action into the valid range. However, the original unclipped action is stored as part of the episodic data ([ppo2/runner.py#L29-L31](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L29-L31)).
+    *   Since the sampling of the Gaussian distribution has no boundaries, the environment usually has certain restrictions on the action space. So [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016) adopted clipping sampled actions into their bounds, [Haarnoja et al., (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Haarnoja2018) adopted invertible squashing function (tanh) to the Gaussian samples to satisfy constraints. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) Compared the two implementations and found that the tanh method is better (decision C63, figure 17). But in order to obtain consistent performance, we chose the implementation of clip. It is worth noting that [Chou 2017](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Chou2017) and [Fujita, et al. (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Fujita2018) pointed out the bias brought by the clip method and proposed different solutions.
+
+6.   Normalization of Observation ([common/vec_env/vec_normalize.py#L4](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L4)) Environment Preprocessing
+    *   At each timestep, the `VecNormalize` wrapper pre-processes the observation before feeding it to the PPO agent. The raw observation was normalized by subtracting its running mean and divided by its variance.
+    *   Using normalization on the input has become a well-known technique for training neural networks. [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016) adopted a moving average normalization for the observation to process the input of the network, which has also become the default choice for subsequent implementations. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) experimentally determined that normalization for observation is very helpful for performance (decision C64, figure 33)
+
+7.   Observation Clipping ([common/vec_env/vec_normalize.py#L39](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L39)) Environment Preprocessing
+    *   Followed by the normalization of observation, the _normalized observation_ is further clipped by `VecNormalize` to a range, usually [−10, 10].
+    *   [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) found that after normalization of observation, using observation clipping did not help performance (decision C65, figure 38), but guessed that it might be helpful in an environment with a wide range of observation.
+
+8.   Reward Scaling ([common/vec_env/vec_normalize.py#L28](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L28)) Environment Preprocessing
+    *   The `VecNormalize` also applies a certain discount-based scaling scheme, where the rewards are divided by the standard deviation of a rolling discounted sum of the rewards (without subtracting and re-adding the mean).
+    *   [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) reported that reward scaling can significantly affect the performance of the algorithm and recommends the use of reward scaling.
+
+9.   Reward Clipping ([common/vec_env/vec_normalize.py#L32](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L32)) Environment Preprocessing
+    *   Followed by the scaling of reward, the _scaled reward_ is further clipped by `VecNormalize` to a range, usually [−10, 10].
+    *   A similar approach can be found in [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015). There is currently no clear evidence that Reward Clipping after Reward Scaling can help with learning.
+
+We make [~25 lines of code](https://www.diffchecker.com/lsy3qa5e) change to `ppo.py` to incorporate these 9 details, resulting in a self-contained `ppo_continuous_action.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_continuous_action.py)) that has 331 lines of code. The following shows the file difference between the `ppo.py` (left) and `ppo_continuous_action.py` (right).
+
+To run the experiments, we match the hyperparameters used in the original implementation as follows.
+
+```
+# https://github.com/openai/baselines/blob/master/baselines/ppo2/defaults.py
+def mujoco():
+    return dict(
+        nsteps=2048,
+        nminibatches=32,
+        lam=0.95,
+        gamma=0.99,
+        noptepochs=10,
+        log_interval=1,
+        ent_coef=0.0,
+        lr=lambda f: 3e-4 * f,
+        cliprange=0.2,
+        value_network='copy'
+    )
+```
+
+Note that `value_network='copy'` means to use the separate MLP networks for policy and value functions (i.e., the 4th implementation detail in this section). Also, the number of environments parameter $N$ (i.e., `num_envs`) is set to 1 ([common/cmd_util.py#L167](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L167)). Below are the benchmarked results.
+
+![Image 17](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Hopper-v2.png)![Image 18](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Walker2d-v2.png)![Image 19](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//HalfCheetah-v2.png)
+
+> Tracked MuJoCo experiments (click to show the interactive panel)
+
+## 5 LSTM implementation details
+
+Next, we introduce the 5 details for implementing LSTM.
+
+1.   Layer initialization for LSTM layers ([a2c/utils.py#L84-L86](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L84-L86)) Neural Network
+    *   The LSTM’s layers’ weights are initialized with `std=1` and biases initialized with `0`.
+
+2.   Initialize the LSTM states to be zeros ([common/models.py#L179](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L179)) Neural Network
+    *   The hidden and cell states of LSTM are initialized with zeros.
+
+3.   Reset LSTM states at the end of the episode ([common/models.py#L141](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L141)) Theory
+    *   During rollouts or training, an end-of-episode flag is passed to the agent so that it can reset The LSTM states to zeros.
+
+4.   Prepare sequential rollouts in mini-batches ([a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81)) Theory
+    *   Under the non-LSTM setting, the mini-batches fetch randomly-indexed training data because the ordering of the training data doesn’t matter. However, the ordering of the training data does matter in the LSTM setting. As a result, the mini-batches fetch the sequential training data from sub-environments.
+
+5.   Reconstruct LSTM states during training ([a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81)) Theory
+    *   The algorithm saves a copy of the LSTM states `initial_lstm_state` before rollouts. During training, the agent then sequentially reconstruct the LSTM states based on the `initial_lstm_state`. This process ensures that we reconstructed the probability distributions used in rollouts.
+
+We make [~60 lines of code](https://www.diffchecker.com/RelaUQdN) change to `ppo_atari.py` to incorporate these 5 details, resulting in a self-contained `ppo_atari_lstm.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_atari_lstm.py)) that has 385 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_atari_lstm.py` (right).
+
+To run the experiments, we use the Atari hyperparameters again and remove the frame stack (i.e., setting the number of frames stacked to 1). Below are the benchmarked results.
+
+![Image 20](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//BreakoutNoFrameskip-v4-LSTM.svg)![Image 21](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//PongNoFrameskip-v4-LSTM.svg)![Image 22](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//BeamRiderNoFrameskip-v4-LSTM.svg)
+
+> Tracked Atari LSTM experiments (click to show the interactive panel)
+
+## 1 `MultiDiscrete` action space detail
+
+The `MultiDiscrete` space is often useful to describe action space for more complicated games. The Gym’s official documentation explains `MultiDiscrete` action space as follows:
+
+```
+# https://github.com/openai/gym/blob/2af816241e4d7f41a000f6144f22e12c8231a112/gym/spaces/multi_discrete.py#L8-L25
+class MultiDiscrete(Space):
+    """
+    - The multi-discrete action space consists of a series of discrete action spaces with different number of actions in each
+    - It is useful to represent game controllers or keyboards where each key can be represented as a discrete action space
+    - It is parametrized by passing an array of positive integers specifying number of actions for each discrete action space
+    Note: Some environment wrappers assume a value of 0 always represents the NOOP action.
+    e.g. Nintendo Game Controller
+    - Can be conceptualized as 3 discrete action spaces:
+        1) Arrow Keys: Discrete 5  - NOOP[0], UP[1], RIGHT[2], DOWN[3], LEFT[4]  - params: min: 0, max: 4
+        2) Button A:   Discrete 2  - NOOP[0], Pressed[1] - params: min: 0, max: 1
+        3) Button B:   Discrete 2  - NOOP[0], Pressed[1] - params: min: 0, max: 1
+    - Can be initialized as
+        MultiDiscrete([ 5, 2, 2 ])
+    """
+    ...
+```
+
+Next, we introduce 1 detail for handling `MultiDiscrete` action space:
+
+1.   Independent action components ([common/distributions.py#L215-L220](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L215-L220)Theory
+    *   In `MultiDiscrete` action spaces, the actions are represented with multiple discrete values. For example, the action of $a_{t} = \left[\right. a_{t}^{1} , a_{t}^{2} \left]\right. = \left[\right. 0 , 1 \left]\right.$ might mean to press the up arrow key and press button A. To account for this difference, PPO treats $\left[\right. a_{t}^{1} , a_{t}^{2} \left]\right.$ as probabilistically independent action components, therefore calculating $p r o b \left(\right. a_{t} \left.\right) = p r o b \left(\right. a_{t}^{1} \left.\right) \cdot p r o b \left(\right. a_{t}^{2} \left.\right)$.
+    *   AlphaStar ([Vinyals et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Vinyals2019)) and OpenAI Five ([Berner et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Berner2019)) adopts the `MultiDiscrete` action spaces. For example, OpenAI Five’s action space is essentially `MultiDiscrete([ 30, 4, 189, 81 ])`, as shown by the following quote: 
+> All together this produces a combined factorized action space size of up to 30 × 4 × 189 × 81 = 1, 837, 080 dimensions
+
+We make [~36 lines of code](https://www.diffchecker.com/8fsnhwUI) change to `ppo_atari.py` to incorporate this 1 detail, resulting in a self-contained `ppo_multidiscrete.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_multidiscrete.py)) that has 335 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_multidiscrete.py` (right).
+
+To run the experiments, we use the Atari hyperparameters again and use Gym-μRTS ([Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021)) as the simulation environment.
+
+```
+def gym_microrts():
+    return dict(
+        nsteps=128, nminibatches=4,
+        lam=0.95, gamma=0.99, noptepochs=4, log_interval=1,
+        ent_coef=.01,
+        lr=lambda f : f * 2.5e-4,
+        cliprange=0.1,
+    )
+```
+
+Below are the benchmarked results.
+
+![Image 23](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//MicrortsMining-v1.svg)![Image 24](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//MicrortsAttackShapedReward-v1.svg)![Image 25](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//MicrortsRandomEnemyShapedReward3-v1.svg)
+
+> Tracked Gym-MicroRTS experiments (click to show the interactive panel)
+
+## 4 Auxiliary implementation details
+
+Next, we introduce 4 auxiliary techniques that are not used (by default) in the official PPO implementations but are potentially useful in special situations.
+
+1.   Clip Range Annealing ([ppo2/ppo2.py#L137](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L137)) Code-level Optimizations
+    *   The clip coefficient of PPO can be annealed similar to how the learning rate is annealed. However, the clip range annealing is actually used by default.
+
+2.   Parallellized Gradient Update ([ppo2/model.py#L131](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L131)) Code-level Optimizations
+    *   The policy gradient is calculated in parallel using multiple processes, mainly used in `ppo1` and not used by default in `ppo2`. Such as paradigm could improve training time by making use of all the available processes.
+
+3.   Early Stopping of the policy optimizations ([ppo/ppo.py#L269-L271](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/pytorch/ppo/ppo.py#L269-L271)) Code-level Optimizations
+    *   This is not actually an implementation detail of _openai/baselines_, but rather an implementation detail in John Schulman’s [modular_rl](https://github.com/joschu/modular_rl/blob/5481b117aa30d3eb8e9ad79abce06378d60dcd45/modular_rl/ppo.py#L48) and _openai/spinningup_ ([TF 1.x](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/tf1/ppo/ppo.py#L234), [Pytorch](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/pytorch/ppo/ppo.py#L269-L271)). It can be considered as an additional mechanism to explicitly enforce the trust-region constraint, on top of the fixed hyperparameter `noptepochs` proposed in the original implementation by [Schulman et al. (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017).
+    *   More specifically, it starts by tracking an approximate average KL divergence between the policy before and after one update step to its network weights. In case said KL divergence exceeds a preset threshold, the updates to the policy weights are preemptively stopped. [Dossa et al.](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Dossa2021) suggest that early stopping can serve as an alternative method to tune the number of update epochs. We also included this early stopping method in our implementation [(via `--target-kl 0.01`)](https://github.com/vwxyzjn/ppo-implementation-details/blob/eb40cbe172309dcda24a8e93a32269d819e5513d/ppo.py#L71), but toggled it off by default.
+    *   Note, however, that while _openai/spinningup_ only early stops the updates to the policy, our implementation early stops both the policy and the value network updates.
+
+4.   Invalid Action Masking ([Vinyals et al., 2017](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Vinyals2017); [Huang and Ontañón, 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#HuangOntanon2020)) Theory
+    *   Invalid action masking is a technique employed most prominently in AlphaStar ([Vinyals et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Vinyals2019)) and OpenAI Five ([Berner et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Berner2019)) to avoid executing invalid actions in a given game state when the agents are being trained using policy gradient algorithms. Specifically, invalid action masking is implemented by replacing the logits corresponding to the invalid actions with negative infinity before passing the logits to softmax. [Huang and Ontañón, 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#HuangOntanon2020) show such a paradigm **actually makes the gradients corresponding to invalid actions zeros**. Furthermore, [Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021) demonstrated invalid action masking to be the critical technique in training agents to win against all past μRTS bots they tested.
+
+Notably, we highlight the effect of invalid action masking. We make [~30 lines of code](https://www.diffchecker.com/wBUb6Zne) change to `ppo_multidiscrete.py` to incorporate invalid action masking, resulting in a self-contained `ppo_multidiscrete_mask.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_multidiscrete_mask.py)) that has 363 lines of code. The following shows the file difference between the `ppo_multidiscrete.py` (left) and `ppo_multidiscrete_mask.py` (right).
+
+To run the experiments, we use the Atari hyperparameters again and use an older version of Gym-μRTS ([Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021)) as the simulation environment. Below are the benchmarked results.
+
+![Image 26](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//mask/MicrortsMining-v1.png)![Image 27](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//mask/MicrortsAttackShapedReward-v1.png)![Image 28](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//mask/MicrortsRandomEnemyShapedReward3-v1.png)
+
+> Tracked Gym-MicroRTS + Action Mask experiments (click to show the interactive panel)
+
+## Results
+
+As shown under each section, our implementations match the results of the original implementation closely. This close matching also extends to other metrics such as policy and value losses. We have made an interactive HTML below for interested viewers to compare other metrics:
+
+## Recommendations
+
+During our reproduction, we have found a number of useful debugging techniques. They are as follows:
+
+1.   **Seed everything**: One debugging approach is to seed everything and then observe when things start to differ from the reference implementation. So you could use the same seed for your implementation and mine, check if the observation returned by the environment is the same, then check if the sample the actions are the same. By following the steps, you would check everything to make sure they are aligned (e.g. print out `values.sum()` see if yours match the reference implementation). In the past, we have done this with the [pytorch-a2c-ppo-acktr-gail](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail) repository and ultimately figured out a bug with our implementation.
+2.   **Check if `ratio=1`**: Check if the `ratio` are always 1s during the first epoch and first mini-batch update, when new and old policies are the same and therefore the `ratio` are 1s and has nothing to clip. If `ratio` are not 1s, it means there is a bug and the program has not reconstructed the probability distributions used in rollouts.
+3.   **Check Kullback-Leibler (KL) divergence**: It is often useful to check if KL divergence goes too high. We have generally found the `approx_kl` stays below 0.02, and if `approx_kl` becomes too high it usually means the policy is changing too quickly and there is a bug.
+4.   **Check other metrics**: As shown in the Results section, the other metrics such as policy and value losses in our implementation also closely match those in the original implementation. So if your policy loss’ curve looks very different than the reference implementation, there might be a bug.
+5.   **Rule of thumb: 400 episodic return in breakout**: Check if your PPO could obtain 400 episodic return in breakout. We have found this to be a practical rule of thumb to determine the fidelity of online PPO implementations in GitHub. Often we found PPO repositories not able to do this, and we know they probably do not match all implementation details of `openai/baselines`’ PPO.
+
+If you are doing research using PPO, consider adopting the following recommendations to help improve the reproducibility of your work:
+
+1.   **Enumerate implementation details used**: If you have implemented PPO as the baseline for your experiment, you should specify which implementation details you are using. Consider using bullet points to enumerate them like done in this blog post.
+2.   **Release locked source code**: Always open source your code whenever possible and make sure the code runs. We suggest adopting proper dependency managers such as [poetry](https://python-poetry.org/) or [pipenv](https://pipenv.pypa.io/en/latest/) to lock your dependencies. In the past, we have encountered numerous projects that are based on `pip install -e .`, which 80% of the time would fail to run due to some obscure errors. Having a pre-built `docker` image with all dependencies installed can also help in case the dependencies packages are not hosted by package managers after deprecation.
+3.   **Track experiments**: Consider using an experiment management software to track your metrics, hyperparameters, code, and others. They can boost your productivity by saving hundreds of hours spent on `matplotlib` and worrying about how to display data. Commercial solutions (usually more mature) include [Weights and Biases](https://wandb.ai/) and [Neptune](https://neptune.ai/), and open-source solutions include [Aim](https://github.com/aimhubio/aim), [ClearML](https://github.com/allegroai/clearml), [Polyaxon](https://github.com/polyaxon/polyaxon).
+4.   **Adopt single-file implementation**: If your research requires more tweaking, consider implementing your algorithms using single-file implementations. This blog does this and creates standalone files for different environments. For example, our `ppo_atari.py` contains all relevant code to handle Atari games. Such a paradigm has the following benefits at the cost of duplicate and harder-to-refactor code: 
+    *   _Easier to see the whole picture_: Because each file is self-contained, people can easily spot all relevant implementation details of the algorithm. Such a paradigm also reduces the burden to understand how files like `env.py`, `agent.py`, `network.py` work together like in typical RL libraries.
+    *   _Faster developing experience_: Usually, each file like `ppo.py` has significantly less LOC compared to RL libraries’ PPO. As a result, it’s often easier to prototype new features without having to do subclassing and refactoring.
+    *   _Painless performance attribution_: If a new version of our algorithm has obtained higher performance, we know this single file is exactly responsible for the performance improvement. To attribute the performance improvement, we can simply do a `filediff` between the current and past versions, and every line of code change is made explicit to us.
+
+## Discussions
+
+## Does modularity help RL libraries?
+
+This blog post demonstrates reproducing PPO is a non-trivial effort, even though PPO’s source code is readily available for reference. Why is it the case? We think one important reason might be that **modularity disperses implementation details**.
+
+Almost all RL libraries have adopted modular design, featuring different modules / files like `env.py`, `agent.py`, `network.py`, `utils.py`, `runner.py`, etc. The nature of modularity necessarily puts implementation details into different files, which is usually great from a software engineering perspective. That is, we don’t have to know how other components work when we just work on `env.py`. Being able to treat other components as black boxes has empowered us to work on large and complicated systems for the last decades.
+
+However, this practice might clash hard with ML / RL: as the library grows, it becomes harder and harder to grasp all implementation details w.r.t an algorithm, whereas recognizing all implementation details has become increasingly important, as indicated by this blog post, [Engstrom, Ilyas, et al., 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom), and [Andrychowicz, et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz). So what can we do?
+
+Modular design still offers numerous benefits such as 1) easy-to-use interface, 2) integrated test cases, 3) easy to plug different components and others. To this end, good RL libraries are valuable, and we recommend them to write good documentation and refactor libraries to adopt new features. For algorithmic researchers, however, we recommend considering single-file implementations because they are straightforward to read and extend.
+
+## Is asynchronous PPO better?
+
+Not necessarily. The high-throughput variant Asynchronous PPO (APPO) ([Berner et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Berner2019)) has obtained more attention in recent years. APPO eliminates the idle time in the original PPO implementation (e.g., have to wait for all $N$ environments to return observations), resulting in much higher throughput, GPU and CPU utilization. However, APPO involves performance-reducing side-effects, namely stale experiences ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)), and we have found insufficient evidence to ascertain its improvement. The biggest issue is:
+
+**Underbenchmarked APPO implementation**: RLlib has an [APPO implementation](https://docs.ray.io/en/latest/rllib-algorithms.html#appo), yet its documentation contains no benchmark information and suggest “APPO is not always more efficient; it is often better to use standard PPO or IMPALA.” Sample Factory ([Petrenko et al, 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Petrenko)) presents more benchmark results, but its support for Atari games is still a [work in progress](https://github.com/alex-petrenko/sample-factory/issues/51). To our knowledge, there is no APPO implementation that simultaneously works with Atari games, MuJoCo or Pybullet tasks, MultiDiscrete action spaces and with an LSTM.
+
+While APPO is intuitively valuable for CPU-intensive tasks such as Dota 2, this blog post recommends an alternative approach to speed up PPO: **make the vectorized environments really fast**. Initially, the vectorized environments are implemented in python, which is slow. More recently, researchers have proposed to use accelerated vectorized environments. For example,
+
+1.   Procgen [(Cobbe et al, 2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe) uses C++ to implement native vectorized environments, resulting in much higher throughput when setting $N = 64$ ($N$ is the number of environments),
+2.   [Envpool](https://github.com/sail-sg/envpool) uses C++ to offer native vectorized environments for Atari and classic control games,
+3.   Nvidia’s [Isaac Gym](https://developer.nvidia.com/isaac-gym)[(Makoviychuk et al., 2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Makoviychuk) uses `torch` to write hardware-accelerated vectorized environments, allowing the users to spin up $N = 4096$ environments easily,
+4.   Google’s [Brax](https://github.com/google/brax) uses jax to write hardware-accelerated vectorized environments, allowing the users to spin up $N = 2048$ environments easily and solve robotics tasks like `Ant` in minutes compared to hours of training in MuJoCo.
+
+In the following section, we demonstrate accelerated training with PPO + envpool in the Atari game Pong.
+
+### Solving Pong in 5 minutes with PPO + Envpool
+
+[Envpool](https://github.com/sail-sg/envpool) is a recent work that offers accelerated vectorized environments for Atari by leveraging C++ and thread pools. Our PPO gets a free and side-effects-free performance boost by simply adopting it. We make [~60 lines of code](https://www.diffchecker.com/RafLuYD6) change to `ppo_atari.py` to incorporate this 1 detail, resulting in a self-contained `ppo_atari_envpool.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_atari_envpool.py)) that has 365 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_atari_envpool.py` (right).
+
+As shown below, Envpool + PPO runs 3x faster without side effects (as in no loss of sample efficiency):
+
+![Image 29](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Breakouts.png)![Image 30](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Pongs.png)![Image 31](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//BeamRiders.png)
+
+![Image 32](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Breakout.png)![Image 33](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//Pong.png)![Image 34](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//BeamRider.png)
+
+> Tracked Atari + Envpool experiments (click to show the interactive panel)
+
+Two quick notes: 1) the performance deterioration in BeamRider is largely due to a degenerate random seed, and 2) Envpool uses the v5 ALE environments but has processed them the same way as the v4 ALE environments used in our previous experiments. Furthermore, by tuning the hyperparameters, we obtained a run that solves Pong in 5 mins. This performance is even comparable to IMPALA’s ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)) results:
+
+We think this raises a practical consideration: adopting async RL such as IMPALA could be more difficult than just making your vectorized environments fast.
+
+## Request for Research
+
+Given this blog post, we believe the community understands PPO better and would be in a much better place to make improvements. Here are a few suggested areas for research.
+
+1.   **Alternative choices**: As we have walked through the different details of PPO, it seems that some of them result from arbitrary choices. It would be interesting to investigate alternative choices and see how such change affects results. You can find below a non-exhaustive list of tracks to explore: 
+    *   use of a different Atari pre-processing (as partially explored by [Machado et al., 2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Machado2018))
+    *   use of a different distribution for continuous actions ([Beta distribution](http://proceedings.mlr.press/v70/chou17a/chou17a.pdf), squashed Gaussian, Gaussian with full covariance, …), it will most probably require some tuning
+    *   use of a state-dependent standard deviation when using continuous actions (with or without backpropagation of the gradient to the whole actor network)
+    *   use of a different initialization for LSTM (ones instead of zeros, random noise, learnable parameter, …), use of GRU cells instead of LSTM
+
+2.   **Vectorized architecture for experience-replay-based methods**: Experience-replay-based methods such as DQN, DDPG, and SAC are less popular than PPO due to a few reasons: 1) they generally have lower throughput due to a single simulation environment (also means lower GPU utilization), and 2) they usually have higher memory requirement (e.g., DQN requires the notorious 1M sample replay buffer which could take 32GB memory). Can we apply the vectorized architecture to experience-replay-based methods? The vectorized environments intuitively should replace replay buffer because the environments could also provide uncorrelated experience.
+3.   **Value function optimization**: In Phasic Policy Gradient ([Cobbe et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe2021)), optimizing value functions separately turns out to be important. In DQN, the prioritized experience replay significantly boosts performance. Can we apply prioritized experience replay to PPO or just on PPO’s value function?
+
+## Conclusion
+
+Reproducing PPO’s results has been difficult in the past few years. While recent works conducted ablation studies to provide insight on the implementation details, these works are not structured as tutorials and only focus on details concerning robotics tasks. As a result, reproducing PPO from scratch can become a daunting experience. Instead of introducing additional improvements or doing further ablation studies, this blog post takes a step back and focuses on delivering a thorough reproduction of PPO in all accounts, as well as aggregating, documenting, and cataloging its most salient implementation details. This blog post also points out software engineering challenges in PPO and further efficiency improvement via the accelerated vectorized environments. With these, we believe this blog post will help people understand PPO faster and better, facilitating customization and research upon this versatile RL algorithm.
+
+## Acknowledgment
+
+We thank [Weights and Biases](https://wandb.ai/) for providing a free academic license that helps us track the experiments. Shengyi would like to personally thank Angelica Pan, Scott Condron, Ivan Goncharov, Morgan McGuire, Jeremy Salwen, Cayla Sharp, Lavanya Shukla, and Aakarshan Chauhan for supporting him in making the video tutorials.
+
+### Bibliography
+
+[Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 2017 Jul 20.](http://arxiv.org/abs/1707.06347)
+
+[Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.](http://arxiv.org/abs/1707.06347)
+
+[Engstrom L, Ilyas A, Santurkar S, Tsipras D, Janoos F, Rudolph L, Madry A. Implementation matters in deep policy gradients: A case study on ppo and trpo. International Conference on Learning Representations, 2020](https://openreview.net/forum?id=r1etN1rtPB)
+
+[Andrychowicz M, Raichuk A, Stańczyk P, Orsini M, Girgin S, Marinier R, Hussenot L, Geist M, Pietquin O, Michalski M, Gelly S. What matters in on-policy reinforcement learning? a large-scale empirical study. International Conference on Learning Representations, 2021](https://openreview.net/forum?id=nIAxjsniDzg)
+
+[Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S. Human-level control through deep reinforcement learning. nature. 2015 Feb;518(7540):529-33.](https://www.nature.com/articles/nature14236)
+
+[Machado MC, Bellemare MG, Talvitie E, Veness J, Hausknecht M, Bowling M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research. 2018 Mar 19;61:523-62.](https://arxiv.org/abs/1709.06009)
+
+[Schulman J, Levine S, Abbeel P, Jordan M, Moritz P. Trust region policy optimization. In International conference on machine learning 2015 Jun 1 (pp. 1889-1897). PMLR.](http://proceedings.mlr.press/v37/schulman15)
+
+[Duan Y, Chen X, Houthooft R, Schulman J, Abbeel P. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning 2016 Jun 11 (pp. 1329-1338). PMLR.](http://proceedings.mlr.press/v48/duan16.html)
+
+[Haarnoja T, Zhou A, Abbeel P, Levine S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning 2018 Jul 3 (pp. 1861-1870). PMLR.](http://proceedings.mlr.press/v80/haarnoja18b)
+
+[Chou PW. The beta policy for continuous control reinforcement learning (Doctoral dissertation, Master’s thesis. Pittsburgh: Carnegie Mellon University). 2017.](https://www.ri.cmu.edu/wp-content/uploads/2017/06/thesis-Chou.pdf)
+
+[Fujita Y, Maeda SI. Clipped action policy gradient. In International Conference on Machine Learning 2018 Jul 3 (pp. 1597-1606). PMLR.](http://proceedings.mlr.press/v80/fujita18a.html)
+
+[Bellemare M, Srinivasan S, Ostrovski G, Schaul T, Saxton D, Munos R. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems. 2016;29:1471-9.](https://proceedings.neurips.cc/paper/2016/file/afda332245e2af431fb7b672a68b659d-Paper.pdf)
+
+[Tavakoli A, Pardo F, Kormushev P. Action branching architectures for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence 2018 Apr 29 (Vol. 32, No. 1).](https://ojs.aaai.org/index.php/AAAI/article/view/11798)
+
+[Metz L, Ibarz J, Jaitly N, Davidson J. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:1705.05035. 2017 May 14.](https://arxiv.org/abs/1705.05035)
+
+[Zhang Y, Vuong QH, Song K, Gong XY, Ross KW. Efficient entropy for policy gradient with multidimensional action space. arXiv preprint arXiv:1806.00589. 2018 Jun 2.](https://arxiv.org/abs/1806.00589)
+
+[Huang S, Ontañón S. A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006.14171. 2020 Jun 25.](https://arxiv.org/abs/2006.14171)
+
+[Huang, S., Ontan’on, S., Bamford, C., & Grela, L. Gym-μRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning. In Proceedings of the 2021 IEEE Conference on Games (CoG).](https://ieeexplore.ieee.org/document/9619076)
+
+[Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P, Oh J. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature. 2019 Nov;575(7782):350-4.](https://doi.org/10.1038/s41586-019-1724-z)
+
+[Berner C, Brockman G, Chan B, Cheung V, Dębiak P, Dennison C, Farhi D, Fischer Q, Hashme S, Hesse C, Józefowicz R. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680. 2019 Dec 13.](https://arxiv.org/abs/1912.06680)
+
+[Vinyals O, Ewalds T, Bartunov S, Georgiev P, Vezhnevets AS, Yeo M, Makhzani A, Küttler H, Agapiou J, Schrittwieser J, Quan J. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782. 2017 Aug 16.](https://arxiv.org/abs/1708.04782)
+
+[Dossa RF, Huang S, Ontañón S, Matsubara T. An Empirical Investigation of Early Stopping Optimizations in Proximal Policy Optimization. IEEE Access. 2021 Aug 23;9:117981-92.](https://ieeexplore.ieee.org/document/9520424)
+
+[Espeholt L, Soyer H, Munos R, Simonyan K, Mnih V, Ward T, Doron Y, Firoiu V, Harley T, Dunning I, Legg S. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. InInternational Conference on Machine Learning 2018 Jul 3 (pp. 1407-1416). PMLR.](https://arxiv.org/abs/1802.01561)
+
+[Petrenko A, Huang Z, Kumar T, Sukhatme G, Koltun V. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. InInternational Conference on Machine Learning 2020 Nov 21 (pp. 7652-7662). PMLR.](https://arxiv.org/abs/2006.11751)
+
+[Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., & State, G. (2021). Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. ArXiv, abs/2108.10470.](https://arxiv.org/abs/2108.10470)
+
+[Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2020, November). Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning (pp. 2048-2056). PMLR.](https://arxiv.org/abs/1912.01588)
+
+[Terry, J.K., Black, B., Hari, A., Santos, L., Dieffendahl, C., Williams, N.L., Lokesh, Y., Horsch, C., & Ravi, P. (2020). Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34..](https://arxiv.org/pdf/2009.14471.pdf)
+
+## Appendix
+
+In this appendix, we introduce one detail for reproducing PPO’s results in the procgen environments [(Cobbe et al, 2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe).
+
+1.   IMPALA-style Neural Network ([common/models.py#L28](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L28)) Neural Network
+    *   In the [openai/train-procgen](https://github.com/openai/train-procgen) repository, the authors by default uses the IMPALA-style Neural Network ([train_procgen/train.py#L52](https://github.com/openai/train-procgen/blob/1a2ae2194a61f76a733a39339530401c024c3ad8/train_procgen/train.py#L52), also see see ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)) without the LSTM layers.
+
+We make [~60 lines of code](https://www.diffchecker.com/82aRqGuz) change to `ppo_atari.py` to incorporate these 5 details, resulting in a self-contained `ppo_procgen.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_procgen.py)) that has 354 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_procgen.py` (right).
+
+To run the experiment, we try to match the default setting in [openai/train-procgen](https://github.com/openai/train-procgen) except that we use the `easy` distribution mode and `total_timesteps=25e6` to save compute.
+
+```
+def procgen():
+    return dict(
+        nsteps=256, nminibatches=8,
+        lam=0.95, gamma=0.999, noptepochs=3, log_interval=1,
+        ent_coef=.01,
+        lr=5e-4,
+        cliprange=0.2,
+        vf_coef=0.5,  max_grad_norm=0.5,
+    )
+network = build_impala_cnn(x, depths=[16,32,32], emb_size=256)
+env = ProcgenEnv(
+    num_envs=64,
+    env_name="starpilot",
+    num_levels=0,
+    start_level=0,
+    distribution_mode="easy"
+)
+env = VecNormalize(venv=env, ob=False)
+ppo2.learn(..., total_timesteps = 25_000_000)
+```
+
+Notice that
+
+1.   Learning rate annealing is turned off by default.
+2.   Reward scaling and reward clipping is used.
+
+Below are the benchmarked results.
+
+![Image 35](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//StarPilot.png)![Image 36](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//BossFight.png)![Image 37](https://iclr-blog-track.github.io/public/images/2022-03-25-ppo-implementation-details//BigFish.png)
+
+> Tracked Procgen experiments (click to show the interactive panel)
+
+You will need to sign in to GitHub to add a comment! To edit or delete your comment, visit the [discussions page](https://github.com/iclr-blog-track/iclr-blog-track.github.io/discussions) and look for your comment in the right discussion.

docs/evidence/fpp3/00-preface.md

@@ -0,0 +1,92 @@
+Source: https://otexts.com/fpp3/index.html (chapter index, 1 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 00-preface
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Forecasting: Principles and Practice (3rd ed)
+
+*Rob J Hyndman and George Athanasopoulos*
+
+Monash University, Australia
+
+# Preface
+
+![](https://otexts.com/fpp3/figs/fpp3_front_cover.jpg)
+[Buy a print version](https://otexts.com/fpp3/buy-a-print-version.html)
+
+Welcome to our online textbook on forecasting.
+
+This textbook is intended to provide a comprehensive introduction to forecasting methods and to present enough information about each method for readers to be able to use them sensibly. We don’t attempt to give a thorough discussion of the theoretical details behind each method, although the references at the end of each chapter will fill in many of those details.
+
+The book is written for three audiences: (1) people finding themselves doing forecasting in business when they may not have had any formal training in the area; (2) undergraduate students studying business; (3) MBA students doing a forecasting elective. We use it ourselves for masters students and third-year undergraduate students at Monash University, Australia.
+
+For most sections, we only assume that readers are familiar with introductory statistics, and with high-school algebra. There are a couple of sections that also require knowledge of matrices, but these are flagged.
+
+At the end of each chapter we provide a list of “further reading”. In general, these lists comprise suggested textbooks that provide a more advanced or detailed treatment of the subject. Where there is no suitable textbook, we suggest journal articles that provide more information.
+
+We use R throughout the book and we intend students to learn how to forecast with R. R is free and available on almost every operating system. It is a wonderful tool for all statistical analysis, not just for forecasting. See the [Using R appendix](https://otexts.com/fpp3/appendix-using-r.html#appendix-using-r) for instructions on installing and using R.
+
+All R examples in the book assume you have loaded the `fpp3` package first:
+
+```
+library(fpp3)
+```
+
+```
+#> ── Attaching packages ──────────────────────────────── fpp3 1.0.3 ──
+#> ✔ tibble      3.3.1     ✔ tsibble     1.2.0
+#> ✔ dplyr       1.2.1     ✔ tsibbledata 0.4.1
+#> ✔ tidyr       1.3.2     ✔ ggtime      0.2.0
+#> ✔ lubridate   1.9.5     ✔ feasts      0.5.0
+#> ✔ ggplot2     4.0.3     ✔ fable       0.5.0
+#> ── Conflicts ───────────────────────────────────── fpp3_conflicts ──
+#> ✖ lubridate::date()    masks base::date()
+#> ✖ dplyr::filter()      masks stats::filter()
+#> ✖ tsibble::intersect() masks base::intersect()
+#> ✖ tsibble::interval()  masks lubridate::interval()
+#> ✖ dplyr::lag()         masks stats::lag()
+#> ✖ tsibble::setdiff()   masks base::setdiff()
+#> ✖ tsibble::union()     masks base::union()
+```
+
+This will load the relevant data sets, and attach several packages as listed above. These include several [`tidyverse`](https://tidyverse.org) packages, and packages to handle time series and forecasting in a “tidy” framework.
+
+The above output also shows the package versions we have used in compiling this edition of the book. Some examples in the book will not work with earlier versions of the packages.
+
+Finally, the output lists some conflicts showing which function will be preferenced when a function of the same name is in multiple packages.
+
+The book is different from other forecasting textbooks in several ways.
+
+* It is free and online, making it accessible to a wide audience.
+* It uses R, which is free, open-source, and extremely powerful software.
+* The online version is continuously updated. You don’t have to wait until the next edition for errors to be removed or new methods to be discussed. We will update the book frequently.
+* There are dozens of real data examples taken from our own consulting practice. We have worked with hundreds of businesses and organisations helping them with forecasting issues, and this experience has contributed directly to many of the examples given here, as well as guiding our general philosophy of forecasting.
+* We emphasise graphical methods more than most forecasters. We use graphs to explore the data, analyse the validity of the models fitted and present the forecasting results.
+
+### Changes in the third edition
+
+The most important change in edition 3 of the book is that we use the `tsibble` and `fable` packages rather than the `forecast` package. This allows us to integrate closely with the `tidyverse` collection of packages. As a consequence, we have replaced many examples to take advantage of the new facilities.
+
+We have also added some new material on time series features, and reorganised the content so Chapters [2](https://otexts.com/fpp3/graphics.html#graphics)–[4](https://otexts.com/fpp3/features.html#features) discuss exploratory analysis of time series, before we introduce any forecasting methods. This is because we should first have a good understanding of our time series, their patterns and characteristics, before we attempt to build any models and produce any forecasts.
+
+In the online version of the book, we have included some videos at the start of most sections. These are intended to complement the written material in each section. You can view the [entire playlist on YouTube](https://www.youtube.com/playlist?list=PLyCNZ_xXGzpm7W9jLqbIyBAiSO5jDwJeE).
+
+Helpful readers of the earlier versions of the book let us know of any typos or errors they had found. These were updated immediately online. No doubt we have introduced some new mistakes, and we will correct them online as soon as they are spotted. Please continue to [let us know](https://github.com/orgs/OTexts/discussions/categories/error-report?discussions_q=) about such things.
+
+If you have questions about using the R packages discussed in this book, or about forecasting in general, please ask on the [OTexts discussion forum](https://github.com/orgs/OTexts/discussions?discussions_q=).
+
+Happy forecasting!
+
+Rob J Hyndman and George Athanasopoulos
+
+May 2021
+
+---
+
+To cite the online version of this book, please use the following:
+
+> Hyndman, R.J., & Athanasopoulos, G. (2021) *Forecasting: principles and practice*, 3rd edition, OTexts: Melbourne, Australia. OTexts.com/fpp3. Accessed on `<current date>`.
+
+> This online version of the book was last updated on 23 July 2026.
+>
+> The print version of the book ([available from Amazon](https://otexts.com/fpp3/fpp3/buy-a-print-version.html)) was last updated on 31 May 2021.

docs/evidence/fpp3/01-getting-started.md

@@ -0,0 +1,266 @@
+Source: https://otexts.com/fpp3/intro.html (chapter intro, 10 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 01-getting-started
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Chapter 1 Getting started
+
+Forecasting has fascinated people for thousands of years, sometimes being considered a sign of divine inspiration, and sometimes being seen as a criminal activity. The Jewish prophet Isaiah wrote in about 700 BC
+
+> *Tell us what the future holds, so we may know that you are gods.*
+> (Isaiah 41:23)
+
+One hundred years later, in ancient Babylon, forecasters would foretell the future based on the appearance of a sheep’s liver. Around the same time, people wanting forecasts would journey to Delphi in Greece to consult the Oracle, who would provide her predictions while intoxicated by ethylene vapours. Forecasters had a tougher time under the emperor Constantius II, who issued a decree in AD357 forbidding anyone “to consult a soothsayer, a mathematician, or a forecaster … May curiosity to foretell the future be silenced forever.”[1](#fn1) A similar ban on forecasting occurred in England in 1824[2](#fn2) when “every person pretending or professing to tell fortunes” was “deemed a rogue and vagabond”. The punishment was up to three months’ imprisonment with hard labour!
+
+The varying fortunes of forecasters arise because good forecasts can seem almost magical, while bad forecasts may be dangerous. Consider the following famous predictions about computing.
+
+* *I think there is a world market for maybe five computers.*
+  (Chairman of IBM, 1943)
+* *Computers in the future may weigh no more than 1.5 tons.*
+  (Popular Mechanics, 1949)
+* *There is no reason anyone would want a computer in their home.*
+  (President, DEC, 1977)
+
+The last of these was made only three years before IBM produced the first personal computer. Not surprisingly, you can no longer buy a DEC computer. Forecasting is obviously a difficult activity, and businesses that do it well have a big advantage over those whose forecasts fail.
+
+In this book, we will explore the most reliable methods for producing forecasts. The emphasis will be on methods that are replicable and testable, and have been shown to work.
+
+---
+
+1. [Codex Theodosianus 9.16.4](https://www.thelatinlibrary.com/theodosius/theod09.shtml)[↩︎](https://otexts.com/fpp3/intro.html#fnref1)
+2. [Vagrancy Act, 1824, Section 4](https://statutes.org.uk/site/the-statutes/nineteenth-century/5-geo-iv-c-83-vagrancy-act-1824/), [repealed in 1989](https://www.legislation.gov.uk/ukpga/Geo4/5/83#commentary-c554743).[↩︎](https://otexts.com/fpp3/intro.html#fnref2)
+
+## 1.1 What can be forecast?
+
+Forecasting is required in many situations: deciding whether to build another power generation plant in the next five years requires forecasts of future demand; scheduling staff in a call centre next week requires forecasts of call volumes; stocking an inventory requires forecasts of stock requirements. Forecasts can be required several years in advance (for the case of capital investments), or only a few minutes beforehand (for telecommunication routing). Whatever the circumstances or time horizons involved, forecasting is an important aid to effective and efficient planning.
+
+Some things are easier to forecast than others. The time of the sunrise tomorrow morning can be forecast precisely. On the other hand, tomorrow’s lotto numbers cannot be forecast with any accuracy. The predictability of an event or a quantity depends on several factors including:
+
+1. how well we understand the factors that contribute to it;
+2. how much data is available;
+3. how similar the future is to the past;
+4. whether the forecasts can affect the thing we are trying to forecast.
+
+For example, short-term forecasts of residential electricity demand can be highly accurate because all four conditions are usually satisfied.
+
+1. We have a good idea of the contributing factors: electricity demand is driven largely by temperatures, with smaller effects for calendar variation such as holidays, and economic conditions.
+2. Several years of data on electricity demand are usually available, and many decades of data on weather conditions.
+3. For short-term forecasting (up to a few weeks), it is safe to assume that demand behaviour will be similar to what has been seen in the past.
+4. For most residential users, the price of electricity is not dependent on demand, and so the demand forecasts have little or no effect on consumer behaviour.
+
+Provided we have the skills to develop a good model linking electricity demand and the key driver variables, the forecasts can be remarkably accurate.
+
+On the other hand, when forecasting currency exchange rates, only one of the conditions is satisfied: there is plenty of available data. However, we have a limited understanding of the factors that affect exchange rates, the future may well be different to the past if there is a financial or political crisis in one of the countries, and forecasts of the exchange rate have a direct effect on the rates themselves. If there are well-publicised forecasts that the exchange rate will increase, then people will immediately adjust the price they are willing to pay and so the forecasts are self-fulfilling. In a sense, the exchange rates become their own forecasts. This is an example of the “efficient market hypothesis”. Consequently, forecasting whether the exchange rate will rise or fall tomorrow is about as predictable as forecasting whether a tossed coin will come down as a head or a tail. In both situations, you will be correct about 50% of the time, whatever you forecast. In situations like this, forecasters need to be aware of their own limitations, and not claim more than is possible.
+
+Often in forecasting, a key step is knowing when something can be forecast accurately, and when forecasts will be no better than tossing a coin. Good forecasts capture the genuine patterns and relationships which exist in the historical data, but do not replicate past events that will not occur again. In this book, we will learn how to tell the difference between a random fluctuation in the past data that should be ignored, and a genuine pattern that should be modelled and extrapolated.
+
+Many people wrongly assume that forecasts are not possible in a changing environment. Every environment is changing, and a good forecasting model captures the way in which things are changing. Forecasts rarely assume that the environment is unchanging. What is normally assumed is that *the way in which the environment is changing* will continue into the future. That is, a highly volatile environment will continue to be highly volatile; a business with fluctuating sales will continue to have fluctuating sales; and an economy that has gone through booms and busts will continue to go through booms and busts. A forecasting model is intended to capture the way things move, not just where things are. As Abraham Lincoln said, “If we could first know where we are and whither we are tending, we could better judge what to do and how to do it”.
+
+Forecasting situations vary widely in their time horizons, factors determining actual outcomes, types of data patterns, and many other aspects. Forecasting methods can be simple, such as using the most recent observation as a forecast (which is called the **naïve method**), or highly complex, such as neural nets and econometric systems of simultaneous equations. Sometimes, there will be no data available at all. For example, we may wish to forecast the sales of a new product in its first year, but there are obviously no data to work with. In situations like this, we use judgmental forecasting, discussed in Chapter [6](https://otexts.com/fpp3/judgmental.html#judgmental). The choice of method depends on what data are available and the predictability of the quantity to be forecast.
+
+## 1.2 Forecasting, goals and planning
+
+Forecasting is a common statistical task in business, where it helps to inform decisions about the scheduling of production, transportation and personnel, and provides a guide to long-term strategic planning. However, business forecasting is often done poorly, and is frequently confused with planning and goals. They are three different things.
+
+Forecasting
+:   is about predicting the future as accurately as possible, given all of the information available, including historical data and knowledge of any future events that might impact the forecasts.
+
+Goals
+:   are what you would like to have happen. Goals should be linked to forecasts and plans, but this does not always occur. Too often, goals are set without any plan for how to achieve them, and no forecasts for whether they are realistic.
+
+Planning
+:   is a response to forecasts and goals. Planning involves determining the appropriate actions that are required to make your forecasts match your goals.
+
+Forecasting should be an integral part of the decision-making activities of management, as it can play an important role in many areas of a company. Modern organisations require short-term, medium-term and long-term forecasts, depending on the specific application.
+
+Short-term forecasts
+:   are needed for the scheduling of personnel, production and transportation. As part of the scheduling process, forecasts of demand are often also required.
+
+Medium-term forecasts
+:   are needed to determine future resource requirements, in order to purchase raw materials, hire personnel, or buy machinery and equipment.
+
+Long-term forecasts
+:   are used in strategic planning. Such decisions must take account of market opportunities, environmental factors and internal resources.
+
+An organisation needs to develop a forecasting system that involves several approaches to predicting uncertain events. Such forecasting systems require the development of expertise in identifying forecasting problems, applying a range of forecasting methods, selecting appropriate methods for each problem, and evaluating and refining forecasting methods over time. It is also important to have strong organisational support for the use of formal forecasting methods if they are to be used successfully.
+
+## 1.3 Determining what to forecast
+
+In the early stages of a forecasting project, decisions need to be made about what should be forecast. For example, if forecasts are required for items in a manufacturing environment, it is necessary to ask whether forecasts are needed for:
+
+1. every product line, or for groups of products?
+2. every sales outlet, or for outlets grouped by region, or only for total sales?
+3. weekly data, monthly data or annual data?
+
+It is also necessary to consider the forecasting horizon. Will forecasts be required for one month in advance, for 6 months, or for ten years? Different types of models will be necessary, depending on what forecast horizon is most important.
+
+How frequently are forecasts required? Forecasts that need to be produced frequently are better done using an automated system than with methods that require careful manual work.
+
+It is worth spending time talking to the people who will use the forecasts to ensure that you understand their needs, and how the forecasts are to be used, before embarking on extensive work in producing the forecasts.
+
+Once it has been determined what forecasts are required, it is then necessary to find or collect the data on which the forecasts will be based. The data required for forecasting may already exist. These days, a lot of data are recorded, and the forecaster’s task is often to identify where and how the required data are stored. The data may include sales records of a company, the historical demand for a product, or the unemployment rate for a geographic region. A large part of a forecaster’s time can be spent in locating and collating the available data prior to developing suitable forecasting methods.
+
+## 1.4 Forecasting data and methods
+
+The appropriate forecasting methods depend largely on what data are available.
+
+If there are no data available, or if the data available are not relevant to the forecasts, then **qualitative forecasting** methods must be used. These methods are not purely guesswork—there are well-developed structured approaches to obtaining good forecasts without using historical data. These methods are discussed in Chapter [6](https://otexts.com/fpp3/judgmental.html#judgmental).
+
+**Quantitative forecasting** can be applied when two conditions are satisfied:
+
+1. numerical information about the past is available;
+2. it is reasonable to assume that some aspects of the past patterns will continue into the future.
+
+There is a wide range of quantitative forecasting methods, often developed within specific disciplines for specific purposes. Each method has its own properties, accuracies, and costs that must be considered when choosing a specific method.
+
+Most quantitative prediction problems use either time series data (collected at regular intervals over time) or cross-sectional data (collected at a single point in time). In this book we are concerned with forecasting future data, and we concentrate on the time series domain.
+
+### Time series forecasting
+
+Examples of time series data include:
+
+* Annual Google profits
+* Quarterly sales results for Amazon
+* Monthly rainfall
+* Weekly retail sales
+* Daily IBM stock prices
+* Hourly electricity demand
+* 5-minute freeway traffic counts
+* Time-stamped stock transaction data
+
+Anything that is observed sequentially over time is a time series. In this book, we will only consider time series that are observed at regular intervals of time (e.g., hourly, daily, weekly, monthly, quarterly, annually). Irregularly spaced time series can also occur, but are beyond the scope of this book.
+
+When forecasting time series data, the aim is to estimate how the sequence of observations will continue into the future. Figure [1.1](https://otexts.com/fpp3/data-methods.html#fig:beer) shows the quarterly Australian beer production from 2000 to the second quarter of 2010.
+
+![Australian quarterly beer production: 2000Q1–2010Q2, with two years of forecasts.](https://otexts.com/fpp3/fpp_files/figure-html/beer-1.png)
+
+Figure 1.1: Australian quarterly beer production: 2000Q1–2010Q2, with two years of forecasts.
+
+The blue lines show forecasts for the next two years. Notice how the forecasts have captured the seasonal pattern seen in the historical data and replicated it for the next two years. The dark shaded region shows 80% prediction intervals. That is, each future value is expected to lie in the dark shaded region with a probability of 80%. The light shaded region shows 95% prediction intervals. These prediction intervals are a useful way of displaying the uncertainty in forecasts. In this case the forecasts are expected to be accurate, and hence the prediction intervals are quite narrow.
+
+The simplest time series forecasting methods use only information on the variable to be forecast, and make no attempt to discover the factors that affect its behaviour. Therefore they will extrapolate trend and seasonal patterns, but they ignore all other information such as marketing initiatives, competitor activity, changes in economic conditions, and so on.
+
+Decomposition methods are helpful for studying the trend and seasonal patterns in a time series; these are discussed in Chapter [3](https://otexts.com/fpp3/decomposition.html#decomposition). Popular time series models used for forecasting include exponential smoothing models and ARIMA models, discussed in Chapters [8](https://otexts.com/fpp3/expsmooth.html#expsmooth) and [9](https://otexts.com/fpp3/arima.html#arima) respectively.
+
+### Predictor variables and time series forecasting
+
+Predictor variables are often useful in time series forecasting. For example, suppose we wish to forecast the hourly electricity demand (ED) of a hot region during the summer period. A model with predictor variables might be of the form
+\[\begin{align\*}
+\text{ED} = & f(\text{current temperature, strength of economy, population,}\\
+& \qquad\text{time of day, day of week, error}).
+\end{align\*}\]
+The relationship is not exact — there will always be changes in electricity demand that cannot be accounted for by the predictor variables. The “error” term on the right allows for random variation and the effects of relevant variables that are not included in the model. We call this an **explanatory model** because it helps explain what causes the variation in electricity demand.
+
+Because the electricity demand data form a time series, we could also use a **time series model** for forecasting. In this case, a suitable time series forecasting equation is of the form
+\[
+\text{ED}_{t+1} = f(\text{ED}_{t}, \text{ED}_{t-1}, \text{ED}_{t-2}, \text{ED}_{t-3},\dots, \text{error}),
+\]
+where \(t\) is the present hour, \(t+1\) is the next hour, \(t-1\) is the previous hour, \(t-2\) is two hours ago, and so on. Here, prediction of the future is based on past values of a variable, but not on external variables that may affect the system. Again, the “error” term on the right allows for random variation and the effects of relevant variables that are not included in the model.
+
+There is also a third type of model which combines the features of the above two models. For example, it might be given by
+\[
+\text{ED}_{t+1} = f(\text{ED}_{t}, \text{current temperature, time of day, day of week, error}).
+\]
+These types of **mixed models** have been given various names in different disciplines. They are known as dynamic regression models, panel data models, longitudinal models, transfer function models, and linear system models (assuming that \(f\) is linear). These models are discussed in Chapter [10](https://otexts.com/fpp3/dynamic.html#dynamic).
+
+An explanatory model is useful because it incorporates information about other variables, rather than only historical values of the variable to be forecast. However, there are several reasons a forecaster might select a time series model rather than an explanatory or mixed model. First, the system may not be understood, and even if it was understood it may be extremely difficult to measure the relationships that are assumed to govern its behaviour. Second, it is necessary to know or forecast the future values of the various predictors in order to be able to forecast the variable of interest, and this may be too difficult. Third, the main concern may be only to predict what will happen, not to know why it happens. Finally, the time series model may give more accurate forecasts than an explanatory or mixed model.
+
+The model to be used in forecasting depends on the resources and data available, the accuracy of the competing models, and the way in which the forecasting model is to be used.
+
+## 1.5 Some case studies
+
+The following four cases are from our consulting practice and demonstrate different types of forecasting situations and the associated challenges that often arise.
+
+#### Case 1
+
+The client was a large company manufacturing disposable tableware such as napkins and paper plates. They needed forecasts of each of hundreds of items every month. The time series data showed a range of patterns, some with trends, some seasonal, and some with neither. At the time, they were using their own software, written in-house, but it often produced forecasts that did not seem sensible. The methods that were being used were the following:
+
+1. average of the last 12 months data;
+2. average of the last 6 months data;
+3. prediction from a straight line regression over the last 12 months;
+4. prediction from a straight line regression over the last 6 months;
+5. prediction obtained by a straight line through the last observation with slope equal to the average slope of the lines connecting last year’s and this year’s values;
+6. prediction obtained by a straight line through the last observation with slope equal to the average slope of the lines connecting last year’s and this year’s values, where the average is taken only over the last 6 months.
+
+They required us to tell them what was going wrong and to modify the software to provide more accurate forecasts. The software was written in COBOL, making it difficult to do any sophisticated numerical computation.
+
+#### Case 2
+
+In this case, the client was the Australian federal government, which needed to forecast the annual budget for the Pharmaceutical Benefit Scheme (PBS). The PBS provides a subsidy for many pharmaceutical products sold in Australia, and the expenditure depends on what people purchase during the year. The total expenditure was around A$7 billion in 2009, and had been underestimated by nearly $1 billion in each of the two years before we were asked to assist in developing a more accurate forecasting approach.
+
+In order to forecast the total expenditure, it is necessary to forecast the sales volumes of hundreds of groups of pharmaceutical products using monthly data. Almost all of the groups have trends and seasonal patterns. The sales volumes for many groups have sudden jumps up or down due to changes in what drugs are subsidised. The expenditures for many groups also have sudden changes due to cheaper competitor drugs becoming available.
+
+Thus we needed to find a forecasting method that allowed for trend and seasonality if they were present, and at the same time was robust to sudden changes in the underlying patterns. It also needed to be able to be applied automatically to a large number of time series.
+
+#### Case 3
+
+A large car fleet company asked us to help them forecast vehicle resale values. They purchase new vehicles, lease them out for three years, and then sell them. Better forecasts of vehicle sales values would mean better control of profits; understanding what affects resale values may allow leasing and sales policies to be developed in order to maximise profits.
+
+At the time, the resale values were being forecast by a group of specialists. Unfortunately, they saw any statistical model as a threat to their jobs, and were uncooperative in providing information. Nevertheless, the company provided a large amount of data on previous vehicles and their eventual resale values.
+
+#### Case 4
+
+In this project, we needed to develop a model for forecasting weekly air passenger traffic on major domestic routes for one of Australia’s leading airlines. The company required forecasts of passenger numbers for each major domestic route and for each class of passenger (economy class, business class and first class). The company provided weekly traffic data from the previous six years.
+
+Air passenger numbers are affected by school holidays, major sporting events, advertising campaigns, competition behaviour, etc. School holidays often do not coincide in different Australian cities, and sporting events sometimes move from one city to another. During the period of the historical data, there was a major pilots’ strike during which there was no traffic for several months. A new cut-price airline also launched and folded. Towards the end of the historical data, the airline had trialled a redistribution of some economy class seats to business class, and some business class seats to first class. After several months, however, the seat classifications reverted to the original distribution.
+
+## 1.6 The basic steps in a forecasting task
+
+A forecasting task usually involves five basic steps.
+
+Step 1: Problem definition.
+:   Often this is the most difficult part of forecasting. Defining the problem carefully requires an understanding of the way the forecasts will be used, who requires the forecasts, and how the forecasting function fits within the organisation requiring the forecasts. A forecaster needs to spend time talking to everyone who will be involved in collecting data, maintaining databases, and using the forecasts for future planning.
+
+Step 2: Gathering information.
+:   There are always at least two kinds of information required: (a) statistical data, and (b) the accumulated expertise of the people who collect the data and use the forecasts. Often, it will be difficult to obtain enough historical data to be able to fit a good statistical model. In that case, the judgmental forecasting methods of Chapter [6](https://otexts.com/fpp3/judgmental.html#judgmental) can be used. Occasionally, old data will be less useful due to structural changes in the system being forecast; then we may choose to use only the most recent data. However, remember that good statistical models will handle evolutionary changes in the system; don’t throw away good data unnecessarily.
+
+Step 3: Preliminary (exploratory) analysis.
+:   Always start by graphing the data. Are there consistent patterns? Is there a significant trend? Is seasonality important? Is there evidence of the presence of business cycles? Are there any outliers in the data that need to be explained by those with expert knowledge? How strong are the relationships among the variables available for analysis? Various tools have been developed to help with this analysis. These are discussed in Chapters [2](https://otexts.com/fpp3/graphics.html#graphics) and [3](https://otexts.com/fpp3/decomposition.html#decomposition).
+
+Step 4: Choosing and fitting models.
+:   The best model to use depends on the availability of historical data, the strength of relationships between the forecast variable and any explanatory variables, and the way in which the forecasts are to be used. It is common to compare two or three potential models. Each model is itself an artificial construct that is based on a set of assumptions (explicit and implicit) and usually involves one or more parameters which must be estimated using the known historical data. We will discuss regression models (Chapter [7](https://otexts.com/fpp3/regression.html#regression)), exponential smoothing methods (Chapter [8](https://otexts.com/fpp3/expsmooth.html#expsmooth)), Box-Jenkins ARIMA models (Chapter [9](https://otexts.com/fpp3/arima.html#arima)), Dynamic regression models (Chapter [10](https://otexts.com/fpp3/dynamic.html#dynamic)), Hierarchical forecasting (Chapter [11](https://otexts.com/fpp3/hierarchical.html#hierarchical)), and several advanced methods including neural networks and vector autoregression (Chapter [12](https://otexts.com/fpp3/advanced.html#advanced)).
+
+Step 5: Using and evaluating a forecasting model.
+:   Once a model has been selected and its parameters estimated, the model is used to make forecasts. The performance of the model can only be properly evaluated after the data for the forecast period have become available. A number of methods have been developed to help in assessing the accuracy of forecasts. There are also organisational issues in using and acting on the forecasts. A brief discussion of some of these issues is given in Chapter [5](https://otexts.com/fpp3/toolbox.html#toolbox). When using a forecasting model in practice, numerous practical issues arise such as how to handle missing values and outliers, or how to deal with short time series. These are discussed in Chapter [13](https://otexts.com/fpp3/practical.html#practical).
+
+## 1.7 The statistical forecasting perspective
+
+The thing we are trying to forecast is unknown (or we would not be forecasting it), and so we can think of it as a *random variable*. For example, the total sales for next month could take a range of possible values, and until we add up the actual sales at the end of the month, we don’t know what the value will be. So until we know the sales for next month, it is a random quantity.
+
+Because next month is relatively close, we usually have a good idea what the likely sales values could be. On the other hand, if we are forecasting the sales for the same month next year, the possible values it could take are much more variable. In most forecasting situations, the variation associated with the thing we are forecasting will shrink as the event approaches. In other words, the further ahead we forecast, the more uncertain we are.
+
+We can imagine many possible futures, each yielding a different value for the thing we wish to forecast. Plotted in black in Figure [1.2](https://otexts.com/fpp3/perspective.html#fig:austa1) are the total international arrivals to Australia from 1980 to 2015. Also shown are ten possible futures from 2016–2025.
+
+![Total international visitors to Australia (1980-2015) along with ten possible futures.](https://otexts.com/fpp3/fpp_files/figure-html/austa1-1.png)
+
+Figure 1.2: Total international visitors to Australia (1980-2015) along with ten possible futures.
+
+When we obtain a forecast, we are estimating the *middle* of the range of possible values the random variable could take. Often, a forecast is accompanied by a **prediction interval** giving a *range* of values the random variable could take with relatively high probability. For example, a 95% prediction interval contains a range of values which should include the actual future value with probability 95%.
+
+Rather than plotting individual possible futures as shown in Figure [1.2](https://otexts.com/fpp3/perspective.html#fig:austa1), we usually show these prediction intervals instead. Figure [1.3](https://otexts.com/fpp3/perspective.html#fig:austa2) shows 80% and 95% intervals for the future Australian international visitors. The blue line is the average of the possible future values, which we call the **point forecasts**.
+
+![Total international visitors to Australia (1980–2015) along with 10-year forecasts and 80% and 95% prediction intervals.](https://otexts.com/fpp3/fpp_files/figure-html/austa2-1.png)
+
+Figure 1.3: Total international visitors to Australia (1980–2015) along with 10-year forecasts and 80% and 95% prediction intervals.
+
+We will use the subscript \(t\) for time. For example, \(y_t\) will denote the observation at time \(t\). Suppose we denote all the information we have observed as \(\mathcal{I}\) and we want to forecast \(y_t\). We then write \(y_{t} | \mathcal{I}\) meaning “the random variable \(y_{t}\) given what we know in \(\mathcal{I}\)”. The set of values that this random variable could take, along with their relative probabilities, is known as the “probability distribution” of \(y_{t} |\mathcal{I}\). In forecasting, we call this the **forecast distribution**.
+
+When we talk about the “forecast”, we usually mean the average value of the forecast distribution, and we put a “hat” over \(y\) to show this. Thus, we write the forecast of \(y_t\) as \(\hat{y}_t\), meaning the average of the possible values that \(y_t\) could take given everything we know.
+
+It is often useful to specify exactly what information we have used in calculating the forecast. Then we will write, for example, \(\hat{y}_{t|t-1}\) to mean the forecast of \(y_t\) taking account of all previous observations \((y_1,\dots,y_{t-1})\). Similarly, \(\hat{y}_{T+h|T}\) means the forecast of \(y_{T+h}\) taking account of \(y_1,\dots,y_T\) (i.e., an \(h\)-step forecast taking account of all observations up to time \(T\)).
+
+## 1.8 Exercises
+
+1. For cases 3 and 4 in Section [1.5](https://otexts.com/fpp3/case-studies.html#case-studies), list the possible predictor variables that might be useful, assuming that the relevant data are available.
+2. For case 3 in Section [1.5](https://otexts.com/fpp3/case-studies.html#case-studies), describe the five steps of forecasting in the context of this project.
+
+## 1.9 Further reading
+
+* Armstrong ([2001](#ref-Armstrong01)) covers the whole field of forecasting, with each chapter written by different experts. It is highly opinionated at times (and we don’t agree with everything in it), but it is full of excellent general advice on tackling forecasting problems.
+* Ord et al. ([2017](#ref-Ord2017)) is a forecasting textbook covering some of the same areas as this book, but with a different emphasis and not focused around any particular software environment. It is written by three highly respected forecasters, with many decades of experience between them.
+
+### Bibliography
+
+Armstrong, J. S. (Ed.). (2001). *Principles of forecasting: A handbook for researchers and practitioners*. Kluwer Academic Publishers.
+
+Ord, J. K., Fildes, R., & Kourentzes, N. (2017). *Principles of business forecasting* (2nd ed.). Wessex Press Publishing Co.

docs/evidence/fpp3/02-time-series-graphics.md

@@ -0,0 +1,862 @@
+Source: https://otexts.com/fpp3/graphics.html (chapter graphics, 12 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 02-time-series-graphics
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Chapter 2 Time series graphics
+
+The first thing to do in any data analysis task is to plot the data. Graphs enable many features of the data to be visualised, including patterns, unusual observations, changes over time, and relationships between variables. The features that are seen in plots of the data must then be incorporated, as much as possible, into the forecasting methods to be used. Just as the type of data determines what forecasting method to use, it also determines what graphs are appropriate. But before we produce graphs, we need to set up our time series in R.
+
+## 2.1 `tsibble` objects
+
+A time series can be thought of as a list of numbers (the observations), along with some information about what times those numbers were recorded (the index). This information can be stored as a `tsibble` object in R.
+
+### The index variable
+
+Suppose you have annual observations for the last few years:
+
+| Year | Observation |
+| --- | --- |
+| 2015 | 123 |
+| 2016 | 39 |
+| 2017 | 78 |
+| 2018 | 52 |
+| 2019 | 110 |
+
+We turn this into a `tsibble` object using the `tsibble()` function:
+
+```
+y <- tsibble(
+  Year = 2015:2019,
+  Observation = c(123, 39, 78, 52, 110),
+  index = Year
+)
+```
+
+`tsibble` objects extend tidy data frames (`tibble` objects) by introducing temporal structure. We have set the time series `index` to be the `Year` column, which associates the measurements (`Observation`) with the time of recording (`Year`).
+
+For observations that are more frequent than once per year, we need to use a time class function on the index. For example, suppose we have a monthly dataset `z`:
+
+```
+z
+#> # A tibble: 5 × 2
+#>   Month    Observation
+#>   <chr>          <dbl>
+#> 1 2019 Jan          50
+#> 2 2019 Feb          23
+#> 3 2019 Mar          34
+#> 4 2019 Apr          30
+#> 5 2019 May          25
+```
+
+This can be converted to a `tsibble` object using the following code:
+
+```
+z |>
+  mutate(Month = yearmonth(Month)) |>
+  as_tsibble(index = Month)
+#> # A tsibble: 5 x 2 [1M]
+#>      Month Observation
+#>      <mth>       <dbl>
+#> 1 2019 Jan          50
+#> 2 2019 Feb          23
+#> 3 2019 Mar          34
+#> 4 2019 Apr          30
+#> 5 2019 May          25
+```
+
+First, the `Month` column is being converted from text to a monthly time object with `yearmonth()`. We then convert the data frame to a `tsibble` by identifying the `index` variable using `as_tsibble()`. Note the addition of “[1M]” on the first line indicating this is monthly data.
+
+Other time class functions can be used depending on the frequency of the observations.
+
+| Frequency | Function |
+| --- | --- |
+| Annual | `start:end` |
+| Quarterly | `yearquarter()` |
+| Monthly | `yearmonth()` |
+| Weekly | `yearweek()` |
+| Daily | `as_date()`, `ymd()` |
+| Sub-daily | `as_datetime()`, `ymd_hms()` |
+
+### The key variables
+
+A `tsibble` also allows multiple time series to be stored in a single object. Suppose you are interested in a dataset containing the fastest running times for women’s and men’s track races at the Olympics, from 100m to 10000m:
+
+```
+olympic_running
+#> # A tsibble: 312 x 4 [4Y]
+#> # Key:       Length, Sex [14]
+#>     Year Length Sex    Time
+#>    <int>  <int> <chr> <dbl>
+#>  1  1896    100 men    12
+#>  2  1900    100 men    11
+#>  3  1904    100 men    11
+#>  4  1908    100 men    10.8
+#>  5  1912    100 men    10.8
+#>  6  1916    100 men    NA
+#>  7  1920    100 men    10.8
+#>  8  1924    100 men    10.6
+#>  9  1928    100 men    10.8
+#> 10  1932    100 men    10.3
+#> # ℹ 302 more rows
+```
+
+The summary above shows that this is a `tsibble` object, which contains 312 rows and 4 columns. Alongside this, “[4Y]” informs us that the interval of these observations is every four years. Below this is the key structure, which informs us that there are 14 separate time series in the `tsibble`. A preview of the first 10 observations is also shown, in which we can see a missing value occurs in 1916. This is because the Olympics were not held during World War I.
+
+The 14 time series in this object are uniquely identified by the keys: the `Length` and `Sex` variables. The `distinct()` function can be used to show the categories of each variable or even combinations of variables:
+
+```
+olympic_running |> distinct(Sex)
+#> # A tibble: 2 × 1
+#>   Sex
+#>   <chr>
+#> 1 men
+#> 2 women
+```
+
+### Working with `tsibble` objects
+
+We can use `dplyr` functions such as `mutate()`, `filter()`, `select()` and `summarise()` to work with `tsibble` objects. To illustrate these, we will use the `PBS` tsibble containing sales data on pharmaceutical products in Australia.
+
+```
+PBS
+#> # A tsibble: 67,596 x 9 [1M]
+#> # Key:       Concession, Type, ATC1, ATC2 [336]
+#>       Month Concession   Type    ATC1  ATC1_desc ATC2  ATC2_desc Scripts  Cost
+#>       <mth> <chr>        <chr>   <chr> <chr>     <chr> <chr>       <dbl> <dbl>
+#>  1 1991 Jul Concessional Co-pay… A     Alimenta… A01   STOMATOL…   18228 67877
+#>  2 1991 Aug Concessional Co-pay… A     Alimenta… A01   STOMATOL…   15327 57011
+#>  3 1991 Sep Concessional Co-pay… A     Alimenta… A01   STOMATOL…   14775 55020
+#>  4 1991 Oct Concessional Co-pay… A     Alimenta… A01   STOMATOL…   15380 57222
+#>  5 1991 Nov Concessional Co-pay… A     Alimenta… A01   STOMATOL…   14371 52120
+#>  6 1991 Dec Concessional Co-pay… A     Alimenta… A01   STOMATOL…   15028 54299
+#>  7 1992 Jan Concessional Co-pay… A     Alimenta… A01   STOMATOL…   11040 39753
+#>  8 1992 Feb Concessional Co-pay… A     Alimenta… A01   STOMATOL…   15165 54405
+#>  9 1992 Mar Concessional Co-pay… A     Alimenta… A01   STOMATOL…   16898 61108
+#> 10 1992 Apr Concessional Co-pay… A     Alimenta… A01   STOMATOL…   18141 65356
+#> # ℹ 67,586 more rows
+```
+
+This contains monthly data on Medicare Australia prescription data from July 1991 to June 2008. These are classified according to various concession types, and Anatomical Therapeutic Chemical (ATC) indexes. For this example, we are interested in the `Cost` time series (total cost of scripts in Australian dollars).
+
+We can use the `filter()` function to extract the A10 scripts:
+
+```
+PBS |>
+  filter(ATC2 == "A10")
+#> # A tsibble: 816 x 9 [1M]
+#> # Key:       Concession, Type, ATC1, ATC2 [4]
+#>       Month Concession   Type   ATC1  ATC1_desc ATC2  ATC2_desc Scripts   Cost
+#>       <mth> <chr>        <chr>  <chr> <chr>     <chr> <chr>       <dbl>  <dbl>
+#>  1 1991 Jul Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   89733 2.09e6
+#>  2 1991 Aug Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   77101 1.80e6
+#>  3 1991 Sep Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   76255 1.78e6
+#>  4 1991 Oct Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   78681 1.85e6
+#>  5 1991 Nov Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   70554 1.69e6
+#>  6 1991 Dec Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   75814 1.84e6
+#>  7 1992 Jan Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   64186 1.56e6
+#>  8 1992 Feb Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   75899 1.73e6
+#>  9 1992 Mar Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   89445 2.05e6
+#> 10 1992 Apr Concessional Co-pa… A     Alimenta… A10   ANTIDIAB…   97315 2.23e6
+#> # ℹ 806 more rows
+```
+
+This allows rows of the tsibble to be selected. Next we can simplify the resulting object by selecting the columns we will need in subsequent analysis.
+
+```
+PBS |>
+  filter(ATC2 == "A10") |>
+  select(Month, Concession, Type, Cost)
+#> # A tsibble: 816 x 4 [1M]
+#> # Key:       Concession, Type [4]
+#>       Month Concession   Type           Cost
+#>       <mth> <chr>        <chr>         <dbl>
+#>  1 1991 Jul Concessional Co-payments 2092878
+#>  2 1991 Aug Concessional Co-payments 1795733
+#>  3 1991 Sep Concessional Co-payments 1777231
+#>  4 1991 Oct Concessional Co-payments 1848507
+#>  5 1991 Nov Concessional Co-payments 1686458
+#>  6 1991 Dec Concessional Co-payments 1843079
+#>  7 1992 Jan Concessional Co-payments 1564702
+#>  8 1992 Feb Concessional Co-payments 1732508
+#>  9 1992 Mar Concessional Co-payments 2046102
+#> 10 1992 Apr Concessional Co-payments 2225977
+#> # ℹ 806 more rows
+```
+
+The `select()` function allows us to select particular columns, while `filter()` allows us to keep particular rows.
+
+Note that the index variable `Month`, and the keys `Concession` and `Type`, would be returned even if they were not explicitly selected as they are required for a tsibble (to ensure each row contains a unique combination of keys and index).
+
+Another useful function is `summarise()` which allows us to combine data across keys. For example, we may wish to compute total cost per month regardless of the `Concession` or `Type` keys.
+
+```
+PBS |>
+  filter(ATC2 == "A10") |>
+  select(Month, Concession, Type, Cost) |>
+  summarise(TotalC = sum(Cost))
+#> # A tsibble: 204 x 2 [1M]
+#>       Month  TotalC
+#>       <mth>   <dbl>
+#>  1 1991 Jul 3526591
+#>  2 1991 Aug 3180891
+#>  3 1991 Sep 3252221
+#>  4 1991 Oct 3611003
+#>  5 1991 Nov 3565869
+#>  6 1991 Dec 4306371
+#>  7 1992 Jan 5088335
+#>  8 1992 Feb 2814520
+#>  9 1992 Mar 2985811
+#> 10 1992 Apr 3204780
+#> # ℹ 194 more rows
+```
+
+The new variable `TotalC` is the sum of all `Cost` values for each month.
+
+We can create new variables using the `mutate()` function. Here we change the units from dollars to millions of dollars:
+
+```
+PBS |>
+  filter(ATC2 == "A10") |>
+  select(Month, Concession, Type, Cost) |>
+  summarise(TotalC = sum(Cost)) |>
+  mutate(Cost = TotalC/1e6)
+#> # A tsibble: 204 x 3 [1M]
+#>       Month  TotalC  Cost
+#>       <mth>   <dbl> <dbl>
+#>  1 1991 Jul 3526591  3.53
+#>  2 1991 Aug 3180891  3.18
+#>  3 1991 Sep 3252221  3.25
+#>  4 1991 Oct 3611003  3.61
+#>  5 1991 Nov 3565869  3.57
+#>  6 1991 Dec 4306371  4.31
+#>  7 1992 Jan 5088335  5.09
+#>  8 1992 Feb 2814520  2.81
+#>  9 1992 Mar 2985811  2.99
+#> 10 1992 Apr 3204780  3.20
+#> # ℹ 194 more rows
+```
+
+Finally, we will save the resulting tsibble for examples later in this chapter.
+
+```
+PBS |>
+  filter(ATC2 == "A10") |>
+  select(Month, Concession, Type, Cost) |>
+  summarise(TotalC = sum(Cost)) |>
+  mutate(Cost = TotalC / 1e6) -> a10
+```
+
+At the end of this series of piped functions, we have used a right assignment (`->`), which is not common in R code, but is convenient at the end of a long series of commands as it continues the flow of the code.
+
+### Read a csv file and convert to a tsibble
+
+Almost all of the data used in this book is already stored as `tsibble` objects. But most data lives in databases, MS-Excel files or csv files, before it is imported into R. So often the first step in creating a tsibble is to read in the data, and then identify the index and key variables.
+
+For example, suppose we have the following quarterly data stored in a csv file (only the first 10 rows are shown). This data set provides information on the size of the prison population in Australia, disaggregated by state, gender, legal status and indigenous status. (Here, ATSI stands for Aboriginal or Torres Strait Islander.)
+
+| Date | State | Gender | Legal | Indigenous | Count |
+| --- | --- | --- | --- | --- | --- |
+| 2005-03-01 | ACT | Female | Remanded | ATSI | 0 |
+| 2005-03-01 | ACT | Female | Remanded | Non-ATSI | 2 |
+| 2005-03-01 | ACT | Female | Sentenced | ATSI | 0 |
+| 2005-03-01 | ACT | Female | Sentenced | Non-ATSI | 5 |
+| 2005-03-01 | ACT | Male | Remanded | ATSI | 7 |
+| 2005-03-01 | ACT | Male | Remanded | Non-ATSI | 58 |
+| 2005-03-01 | ACT | Male | Sentenced | ATSI | 5 |
+| 2005-03-01 | ACT | Male | Sentenced | Non-ATSI | 101 |
+| 2005-03-01 | NSW | Female | Remanded | ATSI | 51 |
+| 2005-03-01 | NSW | Female | Remanded | Non-ATSI | 131 |
+
+We can read it into R, and create a tsibble object, by simply identifying which column contains the time index, and which columns are keys. The remaining columns are values — there can be many value columns, although in this case there is only one (`Count`). The original csv file stored the dates as individual days, although the data is actually quarterly, so we need to convert the `Date` variable to quarters.
+
+```
+prison <- readr::read_csv("https://OTexts.com/fpp3/extrafiles/prison_population.csv")
+```
+
+```
+prison <- prison |>
+  mutate(Quarter = yearquarter(Date)) |>
+  select(-Date) |>
+  as_tsibble(key = c(State, Gender, Legal, Indigenous),
+             index = Quarter)
+
+prison
+#> # A tsibble: 3,072 x 6 [1Q]
+#> # Key:       State, Gender, Legal, Indigenous [64]
+#>    State Gender Legal    Indigenous Count Quarter
+#>    <chr> <chr>  <chr>    <chr>      <dbl>   <qtr>
+#>  1 ACT   Female Remanded ATSI           0 2005 Q1
+#>  2 ACT   Female Remanded ATSI           1 2005 Q2
+#>  3 ACT   Female Remanded ATSI           0 2005 Q3
+#>  4 ACT   Female Remanded ATSI           0 2005 Q4
+#>  5 ACT   Female Remanded ATSI           1 2006 Q1
+#>  6 ACT   Female Remanded ATSI           1 2006 Q2
+#>  7 ACT   Female Remanded ATSI           1 2006 Q3
+#>  8 ACT   Female Remanded ATSI           0 2006 Q4
+#>  9 ACT   Female Remanded ATSI           0 2007 Q1
+#> 10 ACT   Female Remanded ATSI           1 2007 Q2
+#> # ℹ 3,062 more rows
+```
+
+This tsibble contains 64 separate time series corresponding to the combinations of the 8 states, 2 genders, 2 legal statuses and 2 indigenous statuses. Each of these series is 48 observations in length, from 2005 Q1 to 2016 Q4.
+
+For a tsibble to be valid, it requires a unique index for each combination of keys. The `tsibble()` or `as_tsibble()` function will return an error if this is not true.
+
+### The seasonal period
+
+Some graphics and some models will use the seasonal period of the data. The seasonal period is the number of observations before the seasonal pattern repeats. In most cases, this will be automatically detected using the time index variable.
+
+Some common periods for different time intervals are shown in the table below:
+
+| Data | Minute | Hour | Day | Week | Year |
+| --- | --- | --- | --- | --- | --- |
+| Quarters |  |  |  |  | 4 |
+| Months |  |  |  |  | 12 |
+| Weeks |  |  |  |  | 52 |
+| Days |  |  |  | 7 | 365.25 |
+| Hours |  |  | 24 | 168 | 8766 |
+| Minutes |  | 60 | 1440 | 10080 | 525960 |
+| Seconds | 60 | 3600 | 86400 | 604800 | 31557600 |
+
+For quarterly, monthly and weekly data, there is only one seasonal period — the number of observations within each year. Actually, there are not \(52\) weeks in a year, but \(365.25/7 = 52.18\) on average, allowing for a leap year every fourth year. Approximating seasonal periods to integers can be useful as many seasonal terms in models only support integer seasonal periods.
+
+If the data is observed more than once per week, then there is often more than one seasonal pattern in the data. For example, data with daily observations might have weekly (period\(=7\)) or annual (period\(=365.25\)) seasonal patterns. Similarly, data that are observed every minute might have hourly (period\(=60\)), daily (period\(=24\times60=1440\)), weekly (period\(=24\times60\times7=10080\)) and annual seasonality (period\(=24\times60\times365.25=525960\)).
+
+More complicated (and unusual) seasonal patterns can be specified using the `period()` function in the `lubridate` package.
+
+## 2.2 Time plots
+
+For time series data, the obvious graph to start with is a time plot. That is, the observations are plotted against the time of observation, with consecutive observations joined by straight lines. Figure [2.1](https://otexts.com/fpp3/time-plots.html#fig:ansett) shows the weekly economy passenger load on Ansett airlines between Australia’s two largest cities.
+
+```
+melsyd_economy <- ansett |>
+  filter(Airports == "MEL-SYD", Class == "Economy") |>
+  mutate(Passengers = Passengers/1000)
+autoplot(melsyd_economy, Passengers) +
+  labs(title = "Ansett airlines economy class",
+       subtitle = "Melbourne-Sydney",
+       y = "Passengers ('000)")
+```
+
+![Weekly economy passenger load on Ansett Airlines.](https://otexts.com/fpp3/fpp_files/figure-html/ansett-1.png)
+
+Figure 2.1: Weekly economy passenger load on Ansett Airlines.
+
+We will use the `autoplot()` command frequently. It automatically produces an appropriate plot of whatever you pass to it in the first argument. In this case, it recognises `melsyd_economy` as a time series and produces a time plot.
+
+The time plot immediately reveals some interesting features.
+
+* There was a period in 1989 when no passengers were carried — this was due to an industrial dispute.
+* There was a period of reduced load in 1992. This was due to a trial in which some economy class seats were replaced by business class seats.
+* A large increase in passenger load occurred in the second half of 1991.
+* There are some large dips in load around the start of each year. These are due to holiday effects.
+* There is a long-term fluctuation in the level of the series which increases during 1987, decreases in 1989, and increases again through 1990 and 1991.
+
+Any model will need to take all these features into account in order to effectively forecast the passenger load into the future.
+
+A simpler time series is shown in Figure [2.2](https://otexts.com/fpp3/time-plots.html#fig:a10plot), using the `a10` data saved earlier.
+
+```
+autoplot(a10, Cost) +
+  labs(y = "$ (millions)",
+       title = "Australian antidiabetic drug sales")
+```
+
+![Monthly sales of antidiabetic drugs in Australia.](https://otexts.com/fpp3/fpp_files/figure-html/a10plot-1.png)
+
+Figure 2.2: Monthly sales of antidiabetic drugs in Australia.
+
+Here, there is a clear and increasing trend. There is also a strong seasonal pattern that increases in size as the level of the series increases. The sudden drop at the start of each year is caused by a government subsidisation scheme that makes it cost-effective for patients to stockpile drugs at the end of the calendar year. Any forecasts of this series would need to capture the seasonal pattern, and the fact that the trend is changing slowly.
+
+## 2.3 Time series patterns
+
+In describing these time series, we have used words such as “trend” and “seasonal” which need to be defined more carefully.
+
+Trend
+:   A *trend* exists when there is a long-term increase or decrease in the data. It does not have to be linear. Sometimes we will refer to a trend as “changing direction”, when it might go from an increasing trend to a decreasing trend. There is a trend in the antidiabetic drug sales data shown in Figure [2.2](https://otexts.com/fpp3/time-plots.html#fig:a10plot).
+
+Seasonal
+:   A *seasonal* pattern occurs when a time series is affected by seasonal factors such as the time of the year or the day of the week. Seasonality is always of a fixed and known period. The monthly sales of antidiabetic drugs (Figure [2.2](https://otexts.com/fpp3/time-plots.html#fig:a10plot)) shows seasonality which is induced partly by the change in the cost of the drugs at the end of the calendar year. (Note that one series can have more than one seasonal pattern.)
+
+Cyclic
+:   A *cycle* occurs when the data exhibit rises and falls that are not of a fixed frequency. These fluctuations are usually due to economic conditions, and are often related to the “business cycle”. The duration of these fluctuations is usually at least 2 years.
+
+Many people confuse cyclic behaviour with seasonal behaviour, but they are really quite different. If the fluctuations are not of a fixed frequency then they are cyclic; if the frequency is unchanging and associated with some aspect of the calendar, then the pattern is seasonal. In general, the average length of cycles is longer than the length of a seasonal pattern, and the magnitudes of cycles tend to be more variable than the magnitudes of seasonal patterns.
+
+Many time series include trend, cycles and seasonality. When choosing a forecasting method, we will first need to identify the time series patterns in the data, and then choose a method that is able to capture the patterns properly.
+
+The examples in Figure [2.3](https://otexts.com/fpp3/tspatterns.html#fig:fourexamples) show different combinations of these components.
+
+![Four examples of time series showing different patterns.](https://otexts.com/fpp3/fpp_files/figure-html/fourexamples-1.png)
+
+Figure 2.3: Four examples of time series showing different patterns.
+
+1. The monthly housing sales (top left) show strong seasonality within each year, as well as some strong cyclic behaviour with a period of about 6–10 years. There is no apparent trend in the data over this period.
+2. The US treasury bill contracts (top right) show results from the Chicago market for 100 consecutive trading days in 1981. Here there is no seasonality, but an obvious downward trend. Possibly, if we had a much longer series, we would see that this downward trend is actually part of a long cycle, but when viewed over only 100 days it appears to be a trend.
+3. The Australian quarterly electricity production (bottom left) shows a strong increasing trend, with strong seasonality. There is no evidence of any cyclic behaviour here.
+4. The daily change in the Google closing stock price (bottom right) has no trend, seasonality or cyclic behaviour. There are random fluctuations which do not appear to be very predictable, and no strong patterns that would help with developing a forecasting model.
+
+## 2.4 Seasonal plots
+
+A seasonal plot is similar to a time plot except that the data are plotted against the individual “seasons” in which the data were observed. An example is given in Figure [2.4](https://otexts.com/fpp3/seasonal-plots.html#fig:seasonplot1) showing the antidiabetic drug sales.
+
+```
+a10 |>
+  gg_season(Cost, labels = "both") +
+  labs(y = "$ (millions)",
+       title = "Seasonal plot: Antidiabetic drug sales")
+```
+
+![Seasonal plot of monthly antidiabetic drug sales in Australia.](https://otexts.com/fpp3/fpp_files/figure-html/seasonplot1-1.png)
+
+Figure 2.4: Seasonal plot of monthly antidiabetic drug sales in Australia.
+
+This is the same data as was shown earlier, but now the data from each year overlap. A seasonal plot allows the underlying seasonal pattern to be seen more clearly, and is especially useful in identifying years in which the pattern changes.
+
+There is a large jump in sales in January each year. These are probably sales in late December as customers stockpile before the end of the calendar year, but the sales are not registered with the government until a week or two later. The graph also shows that there was an unusually small number of sales in March 2008 (most other years show an increase between February and March). The small number of sales in June 2008 is probably due to incomplete counting of sales at the time the data were collected.
+
+### Multiple seasonal periods
+
+Where the data has more than one seasonal pattern, the `period` argument can be used to select which seasonal plot is required. The `vic_elec` data contains half-hourly electricity demand for the state of Victoria, Australia. We can plot the daily pattern, weekly pattern or yearly pattern by specifying the `period` argument as shown in Figures [2.5](https://otexts.com/fpp3/seasonal-plots.html#fig:multipleseasonplots1)–[2.7](https://otexts.com/fpp3/seasonal-plots.html#fig:multipleseasonplots3).
+
+In the first plot, the three days with 25 hours are when daylight saving ended in each year and so these days contained an extra hour. There were also three days with only 23 hours each (when daylight saving started) but these are hidden beneath all the other lines on the plot.
+
+```
+vic_elec |> gg_season(Demand, period = "day") +
+  theme(legend.position = "none") +
+  labs(y="MWh", title="Electricity demand: Victoria")
+```
+
+![Seasonal plot showing daily seasonal patterns for Victorian electricity demand.](https://otexts.com/fpp3/fpp_files/figure-html/multipleseasonplots1-1.png)
+
+Figure 2.5: Seasonal plot showing daily seasonal patterns for Victorian electricity demand.
+
+```
+vic_elec |> gg_season(Demand, period = "week") +
+  theme(legend.position = "none") +
+  labs(y="MWh", title="Electricity demand: Victoria")
+```
+
+![Seasonal plot showing weekly seasonal patterns for Victorian electricity demand.](https://otexts.com/fpp3/fpp_files/figure-html/multipleseasonplots2-1.png)
+
+Figure 2.6: Seasonal plot showing weekly seasonal patterns for Victorian electricity demand.
+
+```
+vic_elec |> gg_season(Demand, period = "year") +
+  labs(y="MWh", title="Electricity demand: Victoria")
+```
+
+![Seasonal plot showing yearly seasonal patterns for Victorian electricity demand.](https://otexts.com/fpp3/fpp_files/figure-html/multipleseasonplots3-1.png)
+
+Figure 2.7: Seasonal plot showing yearly seasonal patterns for Victorian electricity demand.
+
+## 2.5 Seasonal subseries plots
+
+An alternative plot that emphasises the seasonal patterns is where the data for each season are collected together in separate mini time plots.
+
+```
+a10 |>
+  gg_subseries(Cost) +
+  labs(
+    y = "$ (millions)",
+    title = "Australian antidiabetic drug sales"
+  )
+```
+
+![Seasonal subseries plot of monthly antidiabetic drug sales in Australia.](https://otexts.com/fpp3/fpp_files/figure-html/subseriesplot-1.png)
+
+Figure 2.8: Seasonal subseries plot of monthly antidiabetic drug sales in Australia.
+
+The blue horizontal lines indicate the means for each month. This form of plot enables the underlying seasonal pattern to be seen clearly, and also shows the changes in seasonality over time. It is especially useful in identifying changes within particular seasons. In this example, the plot is not particularly revealing; but in some cases, this is the most useful way of viewing seasonal changes over time.
+
+### Example: Australian holiday tourism
+
+Australian quarterly vacation data provides an interesting example of how these plots can reveal information. First we need to extract the relevant data from the `tourism` tsibble. All the usual `tidyverse` wrangling verbs apply. To get the total visitor nights spent on Holiday by State for each quarter (i.e., ignoring Regions) we can use the following code. Note that we do not have to explicitly group by the time index as this is required in a `tsibble`.
+
+```
+holidays <- tourism |>
+  filter(Purpose == "Holiday") |>
+  group_by(State) |>
+  summarise(Trips = sum(Trips))
+```
+
+```
+holidays
+#> # A tsibble: 640 x 3 [1Q]
+#> # Key:       State [8]
+#>    State Quarter Trips
+#>    <chr>   <qtr> <dbl>
+#>  1 ACT   1998 Q1  196.
+#>  2 ACT   1998 Q2  127.
+#>  3 ACT   1998 Q3  111.
+#>  4 ACT   1998 Q4  170.
+#>  5 ACT   1999 Q1  108.
+#>  6 ACT   1999 Q2  125.
+#>  7 ACT   1999 Q3  178.
+#>  8 ACT   1999 Q4  218.
+#>  9 ACT   2000 Q1  158.
+#> 10 ACT   2000 Q2  155.
+#> # ℹ 630 more rows
+```
+
+Time plots of each series show that there is strong seasonality for most states, but that the seasonal peaks do not coincide.
+
+```
+autoplot(holidays, Trips) +
+  labs(y = "Overnight trips ('000)",
+       title = "Australian domestic holidays")
+```
+
+![Time plots of Australian domestic holidays by state.](https://otexts.com/fpp3/fpp_files/figure-html/holidays-plot-1.png)
+
+Figure 2.9: Time plots of Australian domestic holidays by state.
+
+To see the timing of the seasonal peaks in each state, we can use a season plot. Figure [2.10](https://otexts.com/fpp3/subseries.html#fig:holidaysseason) makes it clear that the southern states of Australia (Tasmania, Victoria and South Australia) have strongest tourism in Q1 (their summer), while the northern states (Queensland and the Northern Territory) have the strongest tourism in Q3 (their dry season).
+
+```
+gg_season(holidays, Trips) +
+  labs(y = "Overnight trips ('000)",
+       title = "Australian domestic holidays")
+```
+
+![Season plots of Australian domestic holidays by state.](https://otexts.com/fpp3/fpp_files/figure-html/holidaysseason-1.png)
+
+Figure 2.10: Season plots of Australian domestic holidays by state.
+
+The corresponding subseries plots are shown in Figure [2.11](https://otexts.com/fpp3/subseries.html#fig:holidayssubseries).
+
+```
+holidays |>
+  gg_subseries(Trips) +
+  labs(y = "Overnight trips ('000)",
+       title = "Australian domestic holidays")
+```
+
+![Subseries plots of Australian domestic holidays by state.](https://otexts.com/fpp3/fpp_files/figure-html/holidayssubseries-1.png)
+
+Figure 2.11: Subseries plots of Australian domestic holidays by state.
+
+This figure makes it evident that Western Australian tourism has jumped markedly in recent years, while Victorian tourism has increased in Q1 and Q4 but not in the middle of the year.
+
+## 2.6 Scatterplots
+
+The graphs discussed so far are useful for visualising individual time series. It is also useful to explore relationships *between* time series.
+
+Figures [2.12](https://otexts.com/fpp3/scatterplots.html#fig:edemand) and [2.13](https://otexts.com/fpp3/scatterplots.html#fig:victemp) show two time series: half-hourly electricity demand (in Gigawatts) and temperature (in degrees Celsius), for 2014 in Victoria, Australia. The temperatures are for Melbourne, the largest city in Victoria, while the demand values are for the entire state.
+
+```
+vic_elec |>
+  filter(year(Time) == 2014) |>
+  autoplot(Demand) +
+  labs(y = "GW",
+       title = "Half-hourly electricity demand: Victoria")
+```
+
+![Half hourly electricity demand in Victoria, Australia, for 2014.](https://otexts.com/fpp3/fpp_files/figure-html/edemand-1.png)
+
+Figure 2.12: Half hourly electricity demand in Victoria, Australia, for 2014.
+
+```
+vic_elec |>
+  filter(year(Time) == 2014) |>
+  autoplot(Temperature) +
+  labs(
+    y = "Degrees Celsius",
+    title = "Half-hourly temperatures: Melbourne, Australia"
+  )
+```
+
+![Half hourly temperature in Melbourne, Australia, for 2014.](https://otexts.com/fpp3/fpp_files/figure-html/victemp-1.png)
+
+Figure 2.13: Half hourly temperature in Melbourne, Australia, for 2014.
+
+We can study the relationship between demand and temperature by plotting one series against the other.
+
+```
+vic_elec |>
+  filter(year(Time) == 2014) |>
+  ggplot(aes(x = Temperature, y = Demand)) +
+  geom_point() +
+  labs(title="Electricity demand versus Temperature",
+       x = "Temperature (degrees Celsius)",
+       y = "Electricity demand (GW)")
+```
+
+![Half-hourly electricity demand plotted against temperature for 2014 in Victoria, Australia.](https://otexts.com/fpp3/fpp_files/figure-html/edemand2-1.png)
+
+Figure 2.14: Half-hourly electricity demand plotted against temperature for 2014 in Victoria, Australia.
+
+This scatterplot helps us to visualise the relationship between the variables. It is clear that high demand occurs when temperatures are high due to the effect of air-conditioning. But there is also a heating effect, where demand increases for very low temperatures.
+
+### Correlation
+
+It is common to compute *correlation coefficients* to measure the strength of the linear relationship between two variables. The correlation between variables \(x\) and \(y\) is given by
+\[
+r = \frac{\sum (x_{t} - \bar{x})(y_{t}-\bar{y})}{\sqrt{\sum(x_{t}-\bar{x})^2}\sqrt{\sum(y_{t}-\bar{y})^2}}.
+\]
+The value of \(r\) always lies between \(-1\) and \(1\) with negative values indicating a negative relationship and positive values indicating a positive relationship. The graphs in Figure [2.15](https://otexts.com/fpp3/scatterplots.html#fig:corr) show examples of data sets with varying levels of correlation.
+
+![Examples of data sets with different levels of correlation.](https://otexts.com/fpp3/fpp_files/figure-html/corr-1.png)
+
+Figure 2.15: Examples of data sets with different levels of correlation.
+
+The correlation coefficient only measures the strength of the *linear* relationship between two variables, and can sometimes be misleading. For example, the correlation for the electricity demand and temperature data shown in Figure [2.14](https://otexts.com/fpp3/scatterplots.html#fig:edemand2) is 0.28, but the *non-linear* relationship is stronger than that.
+
+![Each of these plots has a correlation coefficient of 0.82. Data from Anscombe (1973).](https://otexts.com/fpp3/fpp_files/figure-html/anscombe-1.png)
+
+Figure 2.16: Each of these plots has a correlation coefficient of 0.82. Data from Anscombe ([1973](#ref-Anscombe1973graphs)).
+
+The plots in Figure [2.16](https://otexts.com/fpp3/scatterplots.html#fig:anscombe) all have correlation coefficients of 0.82, but they have very different relationships. This shows how important it is to look at the plots of the data and not simply rely on correlation values.
+
+### Scatterplot matrices
+
+When there are several potential predictor variables, it is useful to plot each variable against each other variable. Consider the eight time series shown in Figure [2.17](https://otexts.com/fpp3/scatterplots.html#fig:vntimeplots), showing quarterly visitor numbers across states and territories of Australia.
+
+```
+visitors <- tourism |>
+  group_by(State) |>
+  summarise(Trips = sum(Trips))
+visitors |>
+  ggplot(aes(x = Quarter, y = Trips)) +
+  geom_line() +
+  facet_grid(vars(State), scales = "free_y") +
+  labs(title = "Australian domestic tourism",
+       y= "Overnight trips ('000)")
+```
+
+![Quarterly visitor nights for the states and territories of Australia.](https://otexts.com/fpp3/fpp_files/figure-html/vntimeplots-1.png)
+
+Figure 2.17: Quarterly visitor nights for the states and territories of Australia.
+
+To see the relationships between these eight time series, we can plot each time series against the others. These plots can be arranged in a scatterplot matrix, as shown in Figure [2.18](https://otexts.com/fpp3/scatterplots.html#fig:ScatterMatrixch2). (This plot requires the `GGally` package to be installed.)
+
+```
+visitors |>
+  pivot_wider(values_from=Trips, names_from=State) |>
+  GGally::ggpairs(columns = 2:9)
+```
+
+![A scatterplot matrix of the quarterly visitor nights in the states and territories of Australia.](https://otexts.com/fpp3/fpp_files/figure-html/ScatterMatrixch2-1.png)
+
+Figure 2.18: A scatterplot matrix of the quarterly visitor nights in the states and territories of Australia.
+
+For each panel, the variable on the vertical axis is given by the variable name in that row, and the variable on the horizontal axis is given by the variable name in that column. There are many options available to produce different plots within each panel. In the default version, the correlations are shown in the upper right half of the plot, while the scatterplots are shown in the lower half. On the diagonal are shown density plots.
+
+The value of the scatterplot matrix is that it enables a quick view of the relationships between all pairs of variables. In this example, mostly positive relationships are revealed, with the strongest relationships being between the neighbouring states located in the south and south east coast of Australia, namely, New South Wales, Victoria and South Australia. Some negative relationships are also revealed between the Northern Territory and other regions. The Northern Territory is located in the north of Australia famous for its outback desert landscapes visited mostly in winter. Hence, the peak visitation in the Northern Territory is in the July (winter) quarter in contrast to January (summer) quarter for the rest of the regions.
+
+### Bibliography
+
+Anscombe, F. J. (1973). Graphs in statistical analysis. *The American Statistician*, *27*(1), 17–21.
+
+## 2.7 Lag plots
+
+Figure [2.19](https://otexts.com/fpp3/lag-plots.html#fig:beerlagplot) displays scatterplots of quarterly Australian beer production (introduced in Figure [1.1](https://otexts.com/fpp3/data-methods.html#fig:beer)), where the horizontal axis shows lagged values of the time series. Each graph shows \(y_{t}\) plotted against \(y_{t-k}\) for different values of \(k\).
+
+```
+recent_production <- aus_production |>
+  filter(year(Quarter) >= 2000)
+recent_production |>
+  gg_lag(Beer, geom = "point") +
+  labs(x = "lag(Beer, k)")
+```
+
+![Lagged scatterplots for quarterly beer production.](https://otexts.com/fpp3/fpp_files/figure-html/beerlagplot-1.png)
+
+Figure 2.19: Lagged scatterplots for quarterly beer production.
+
+Here the colours indicate the quarter of the variable on the vertical axis. The relationship is strongly positive at lags 4 and 8, reflecting the strong seasonality in the data. The negative relationship seen for lags 2 and 6 occurs because peaks (in Q4) are plotted against troughs (in Q2)
+
+## 2.8 Autocorrelation
+
+Just as correlation measures the extent of a linear relationship between two variables, autocorrelation measures the linear relationship between *lagged values* of a time series.
+
+There are several autocorrelation coefficients, corresponding to each panel in the lag plot. For example, \(r_{1}\) measures the relationship between \(y_{t}\) and \(y_{t-1}\), \(r_{2}\) measures the relationship between \(y_{t}\) and \(y_{t-2}\), and so on.
+
+The value of \(r_{k}\) can be written as
+\[
+r_{k} = \frac{\sum\limits_{t=k+1}^T (y_{t}-\bar{y})(y_{t-k}-\bar{y})}
+{\sum\limits_{t=1}^T (y_{t}-\bar{y})^2},
+\]
+where \(T\) is the length of the time series. The autocorrelation coefficients make up the *autocorrelation function* or ACF.
+
+The autocorrelation coefficients for the beer production data can be computed using the `ACF()` function.
+
+```
+recent_production |> ACF(Beer, lag_max = 9)
+#> # A tsibble: 9 x 2 [1Q]
+#>        lag      acf
+#>   <cf_lag>    <dbl>
+#> 1       1Q -0.0530
+#> 2       2Q -0.758
+#> 3       3Q -0.0262
+#> 4       4Q  0.802
+#> 5       5Q -0.0775
+#> 6       6Q -0.657
+#> 7       7Q  0.00119
+#> 8       8Q  0.707
+#> 9       9Q -0.0888
+```
+
+The values in the `acf` column are \(r_1,\dots,r_9\), corresponding to the nine scatterplots in Figure [2.19](https://otexts.com/fpp3/lag-plots.html#fig:beerlagplot). We usually plot the ACF to see how the correlations change with the lag \(k\). The plot is sometimes known as a *correlogram*.
+
+```
+recent_production |>
+  ACF(Beer) |>
+  autoplot() + labs(title="Australian beer production")
+```
+
+![Autocorrelation function of quarterly beer production.](https://otexts.com/fpp3/fpp_files/figure-html/beeracf-1.png)
+
+Figure 2.20: Autocorrelation function of quarterly beer production.
+
+In this graph:
+
+* \(r_{4}\) is higher than for the other lags. This is due to the seasonal pattern in the data: the peaks tend to be four quarters apart and the troughs tend to be four quarters apart.
+* \(r_{2}\) is more negative than for the other lags because troughs tend to be two quarters behind peaks.
+* The dashed blue lines indicate whether the correlations are significantly different from zero (as explained in Section [2.9](https://otexts.com/fpp3/wn.html#wn)).
+
+### Trend and seasonality in ACF plots
+
+When data have a trend, the autocorrelations for small lags tend to be large and positive because observations nearby in time are also nearby in value. So the ACF of a trended time series tends to have positive values that slowly decrease as the lags increase.
+
+When data are seasonal, the autocorrelations will be larger for the seasonal lags (at multiples of the seasonal period) than for other lags.
+
+When data are both trended and seasonal, you see a combination of these effects. The `a10` data plotted in Figure [2.2](https://otexts.com/fpp3/time-plots.html#fig:a10plot) shows both trend and seasonality. Its ACF is shown in Figure [2.21](https://otexts.com/fpp3/acf.html#fig:acfa10). The slow decrease in the ACF as the lags increase is due to the trend, while the “scalloped” shape is due to the seasonality.
+
+```
+a10 |>
+  ACF(Cost, lag_max = 48) |>
+  autoplot() +
+  labs(title="Australian antidiabetic drug sales")
+```
+
+![ACF of monthly Australian antidiabetic drug sales.](https://otexts.com/fpp3/fpp_files/figure-html/acfa10-1.png)
+
+Figure 2.21: ACF of monthly Australian antidiabetic drug sales.
+
+## 2.9 White noise
+
+Time series that show no autocorrelation are called **white noise**. Figure [2.22](https://otexts.com/fpp3/wn.html#fig:wnoise) gives an example of a white noise series.
+
+```
+set.seed(30)
+y <- tsibble(sample = 1:50, wn = rnorm(50), index = sample)
+y |> autoplot(wn) + labs(title = "White noise", y = "")
+```
+
+![A white noise time series.](https://otexts.com/fpp3/fpp_files/figure-html/wnoise-1.png)
+
+Figure 2.22: A white noise time series.
+
+```
+y |>
+  ACF(wn) |>
+  autoplot() + labs(title = "White noise")
+```
+
+![Autocorrelation function for the white noise series.](https://otexts.com/fpp3/fpp_files/figure-html/wnoiseacf-1.png)
+
+Figure 2.23: Autocorrelation function for the white noise series.
+
+For white noise series, we expect each autocorrelation to be close to zero. Of course, they will not be exactly equal to zero as there is some random variation. For a white noise series, we expect 95% of the spikes in the ACF to lie within \(\pm 1.96/\sqrt{T}\) where \(T\) is the length of the time series. It is common to plot these bounds on a graph of the ACF (the blue dashed lines above). If one or more large spikes are outside these bounds, or if substantially more than 5% of spikes are outside these bounds, then the series is probably not white noise.
+
+In this example, \(T=50\) and so the bounds are at \(\pm 1.96/\sqrt{50} = \pm 0.28\). All of the autocorrelation coefficients lie within these limits, confirming that the data are white noise.
+
+## 2.10 Exercises
+
+1. Explore the following four time series: `Bricks` from `aus_production`, `Lynx` from `pelt`, `Close` from `gafa_stock`, `Demand` from `vic_elec`.
+
+   * Use `?` (or `help()`) to find out about the data in each series.
+   * What is the time interval of each series?
+   * Use `autoplot()` to produce a time plot of each series.
+   * For the last plot, modify the axis labels and title.
+2. Use `filter()` to find what days corresponded to the peak closing price for each of the four stocks in `gafa_stock`.
+3. Download the file `tute1.csv` from [the book website](https://bit.ly/fpptute1), open it in Excel (or some other spreadsheet application), and review its contents. You should find four columns of information. Columns B through D each contain a quarterly series, labelled Sales, AdBudget and GDP. Sales contains the quarterly sales for a small company over the period 1981-2005. AdBudget is the advertising budget and GDP is the gross domestic product. All series have been adjusted for inflation.
+
+   1. You can read the data into R with the following script:
+
+      ```
+      tute1 <- readr::read_csv("tute1.csv")
+      View(tute1)
+      ```
+   2. Convert the data to time series
+
+      ```
+      mytimeseries <- tute1 |>
+        mutate(Quarter = yearquarter(Quarter)) |>
+        as_tsibble(index = Quarter)
+      ```
+   3. Construct time series plots of each of the three series
+
+      ```
+      mytimeseries |>
+        pivot_longer(-Quarter) |>
+        ggplot(aes(x = Quarter, y = value, colour = name)) +
+        geom_line() +
+        facet_grid(name ~ ., scales = "free_y")
+      ```
+
+      Check what happens when you don’t include `facet_grid()`.
+4. The `USgas` package contains data on the demand for natural gas in the US.
+
+   1. Install the `USgas` package.
+   2. Create a tsibble from `us_total` with year as the index and state as the key.
+   3. Plot the annual natural gas consumption by state for the New England area (comprising the states of Maine, Vermont, New Hampshire, Massachusetts, Connecticut and Rhode Island).
+5. 1. Download `tourism.xlsx` from [the book website](https://bit.ly/fpptourism) and read it into R using `readxl::read_excel()`.
+   2. Create a tsibble which is identical to the `tourism` tsibble from the `tsibble` package.
+   3. Find what combination of `Region` and `Purpose` had the maximum number of overnight trips on average.
+   4. Create a new tsibble which combines the Purposes and Regions, and just has total trips by State.
+6. The `aus_arrivals` data set comprises quarterly international arrivals to Australia from Japan, New Zealand, UK and the US.
+
+   * Use `autoplot()`, `gg_season()` and `gg_subseries()` to compare the differences between the arrivals from these four countries.
+   * Can you identify any unusual observations?
+7. Monthly Australian retail data is provided in `aus_retail`. Select one of the time series as follows (but choose your own seed value):
+
+   ```
+   set.seed(12345678)
+   myseries <- aus_retail |>
+     filter(`Series ID` == sample(aus_retail$`Series ID`,1))
+   ```
+
+   Explore your chosen retail time series using the following functions:
+
+   `autoplot()`, `gg_season()`, `gg_subseries()`, `gg_lag()`,
+
+   `ACF() |> autoplot()`
+
+   Can you spot any seasonality, cyclicity and trend? What do you learn about the series?
+
+8. Use the following graphics functions: `autoplot()`, `gg_season()`, `gg_subseries()`, `gg_lag()`, `ACF()` and explore features from the following time series: “Total Private” `Employed` from `us_employment`, `Bricks` from `aus_production`, `Hare` from `pelt`, “H02” `Cost` from `PBS`, and `Barrels` from `us_gasoline`.
+
+   * Can you spot any seasonality, cyclicity and trend?
+   * What do you learn about the series?
+   * What can you say about the seasonal patterns?
+   * Can you identify any unusual years?
+9. The following time plots and ACF plots correspond to four different time series. Your task is to match each time plot in the first row with one of the ACF plots in the second row.
+
+   ![](https://otexts.com/fpp3/fpp_files/figure-html/acfguess-1.png)
+10. The `aus_livestock` data contains the monthly total number of pigs slaughtered in Victoria, Australia, from Jul 1972 to Dec 2018. Use `filter()` to extract pig slaughters in Victoria between 1990 and 1995. Use `autoplot()` and `ACF()` for this data. How do they differ from white noise? If a longer period of data is used, what difference does it make to the ACF?
+11. 1. Use the following code to compute the daily changes in Google closing stock prices.
+
+       ```
+       dgoog <- gafa_stock |>
+         filter(Symbol == "GOOG", year(Date) >= 2018) |>
+         mutate(trading_day = row_number()) |>
+         update_tsibble(index = trading_day, regular = TRUE) |>
+         mutate(diff = difference(Close))
+       ```
+    2. Why was it necessary to re-index the tsibble?
+    3. Plot these differences and their ACF.
+    4. Do the changes in the stock prices look like white noise?
+
+## 2.11 Further reading
+
+* W. S. Cleveland ([1993](#ref-Cleveland1993)) is a classic book on the principles of visualisation for data analysis. While it is more than 20 years old, the ideas are timeless.
+* Unwin ([2015](#ref-Unwin2015)) is a modern introduction to graphical data analysis using R. It does not have much information on time series graphics, but plenty of excellent general advice on using graphics for data analysis.
+
+### Bibliography
+
+Cleveland, W. S. (1993). *Visualizing data*. Hobart Press.
+
+Unwin, A. (2015). *Graphical data analysis with R*. Chapman; Hall/CRC.

docs/evidence/fpp3/03-time-series-decomposition.md

@@ -0,0 +1,679 @@
+Source: https://otexts.com/fpp3/decomposition.html (chapter decomposition, 9 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 03-time-series-decomposition
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Chapter 3 Time series decomposition
+
+Time series data can exhibit a variety of patterns, and it is often helpful to split a time series into several components, each representing an underlying pattern category.
+
+In Section [2.3](https://otexts.com/fpp3/tspatterns.html#tspatterns) we discussed three types of time series patterns: trend, seasonality and cycles. When we decompose a time series into components, we usually combine the trend and cycle into a single **trend-cycle** component (often just called the **trend** for simplicity). Thus we can think of a time series as comprising three components: a trend-cycle component, a seasonal component, and a remainder component (containing anything else in the time series). For some time series (e.g., those that are observed at least daily), there can be more than one seasonal component, corresponding to the different seasonal periods.
+
+In this chapter, we consider the most common methods for extracting these components from a time series. Often this is done to help improve understanding of the time series, but it can also be used to improve forecast accuracy.
+
+When decomposing a time series, it is sometimes helpful to first transform or adjust the series in order to make the decomposition (and later analysis) as simple as possible. So we will begin by discussing transformations and adjustments.
+
+## 3.1 Transformations and adjustments
+
+Adjusting the historical data can often lead to a simpler time series. Here, we deal with four kinds of adjustments: calendar adjustments, population adjustments, inflation adjustments and mathematical transformations. The purpose of these adjustments and transformations is to simplify the patterns in the historical data by removing known sources of variation, or by making the pattern more consistent across the whole data set. Simpler patterns are usually easier to model and lead to more accurate forecasts.
+
+### Calendar adjustments
+
+Some of the variation seen in seasonal data may be due to simple calendar effects. In such cases, it is usually much easier to remove the variation before doing any further analysis.
+
+For example, if you are studying the total monthly sales in a retail store, there will be variation between the months simply because of the different numbers of trading days in each month, in addition to the seasonal variation across the year. It is easy to remove this variation by computing average sales per trading day in each month, rather than total sales in the month. Then we effectively remove the calendar variation.
+
+### Population adjustments
+
+Any data that are affected by population changes can be adjusted to give per-capita data. That is, consider the data per person (or per thousand people, or per million people) rather than the total. For example, if you are studying the number of hospital beds in a particular region over time, the results are much easier to interpret if you remove the effects of population changes by considering the number of beds per thousand people. Then you can see whether there have been real increases in the number of beds, or whether the increases are due entirely to population increases. It is possible for the total number of beds to increase, but the number of beds per thousand people to decrease. This occurs when the population is increasing faster than the number of hospital beds. For most data that are affected by population changes, it is best to use per-capita data rather than the totals.
+
+This can be seen in the `global_economy` dataset, where a common transformation of GDP is GDP per-capita.
+
+```
+global_economy |>
+  filter(Country == "Australia") |>
+  autoplot(GDP/Population) +
+  labs(title= "GDP per capita", y = "$US")
+```
+
+![Australian GDP per-capita.](https://otexts.com/fpp3/fpp_files/figure-html/gdp-per-capita-1.png)
+
+Figure 3.1: Australian GDP per-capita.
+
+### Inflation adjustments
+
+Data which are affected by the value of money are best adjusted before modelling. For example, the average cost of a new house will have increased over the last few decades due to inflation. A $200,000 house this year is not the same as a $200,000 house twenty years ago. For this reason, financial time series are usually adjusted so that all values are stated in dollar values from a particular year. For example, the house price data may be stated in year 2000 dollars.
+
+To make these adjustments, a price index is used. If \(z_{t}\) denotes the price index and \(y_{t}\) denotes the original house price in year \(t\), then \(x_{t} = y_{t}/z_{t} \* z_{2000}\) gives the adjusted house price at year 2000 dollar values. Price indexes are often constructed by government agencies. For consumer goods, a common price index is the Consumer Price Index (or CPI).
+
+This allows us to compare the growth or decline of industries relative to a common price value. For example, looking at aggregate annual “newspaper and book” retail turnover from `aus_retail`, and adjusting the data for inflation using CPI from `global_economy` allows us to understand the changes over time.
+
+```
+print_retail <- aus_retail |>
+  filter(Industry == "Newspaper and book retailing") |>
+  group_by(Industry) |>
+  index_by(Year = year(Month)) |>
+  summarise(Turnover = sum(Turnover))
+aus_economy <- global_economy |>
+  filter(Code == "AUS")
+```
+
+```
+print_retail |>
+  left_join(aus_economy, by = "Year") |>
+  mutate(Adjusted_turnover = Turnover / CPI * 100) |>
+  pivot_longer(c(Turnover, Adjusted_turnover),
+               values_to = "Turnover") |>
+  mutate(name = factor(name,
+         levels=c("Turnover","Adjusted_turnover"))) |>
+  ggplot(aes(x = Year, y = Turnover)) +
+  geom_line() +
+  facet_grid(name ~ ., scales = "free_y") +
+  labs(title = "Turnover: Australian print media industry",
+       y = "$AU")
+```
+
+![Turnover for the Australian print media industry in Australian dollars. The 'Adjusted' turnover has been adjusted for inflation using the CPI.](https://otexts.com/fpp3/fpp_files/figure-html/printretail-1.png)
+
+Figure 3.2: Turnover for the Australian print media industry in Australian dollars. The ‘Adjusted’ turnover has been adjusted for inflation using the CPI.
+
+By adjusting for inflation using the CPI, we can see that Australia’s newspaper and book retailing industry has been in decline much longer than the original data suggests. The adjusted turnover is in 2010 Australian dollars, as CPI is 100 in 2010 in this data set.
+
+### Mathematical transformations
+
+If the data shows variation that increases or decreases with the level of the series, then a transformation can be useful. For example, a logarithmic transformation is often useful. If we denote the original observations as \(y_{1},\dots,y_{T}\) and the transformed observations as \(w_{1}, \dots, w_{T}\), then \(w_t = \log(y_t)\). Logarithms are useful because they are interpretable: changes in a log value are relative (or percentage) changes on the original scale. So if log base 10 is used, then an increase of 1 on the log scale corresponds to a multiplication of 10 on the original scale. If any value of the original series is zero or negative, then logarithms are not possible.
+
+Sometimes other transformations are also used (although they are not so interpretable). For example, square roots and cube roots can be used. These are called **power transformations** because they can be written in the form \(w_{t} = y_{t}^p\).
+
+A useful family of transformations, that includes both logarithms and power transformations, is the family of **Box-Cox transformations** ([Box & Cox, 1964](#ref-BC64)), which depend on the parameter \(\lambda\) and are defined as follows:
+\[\begin{equation}
+w_t =
+\begin{cases}
+\log(y_t) & \text{if $\lambda=0$}; \\
+(\text{sign}(y_t)|y_t|^\lambda-1)/\lambda & \text{otherwise}.
+\end{cases}
+\tag{3.1}
+\end{equation}\]
+This is actually a modified Box-Cox transformation, discussed in Bickel & Doksum ([1981](#ref-Bickel1981)), which allows for negative values of \(y_t\) provided \(\lambda > 0\).
+
+The logarithm in a Box-Cox transformation is always a natural logarithm (i.e., to base \(e\)). So if \(\lambda=0\), natural logarithms are used, but if \(\lambda\ne0\), a power transformation is used, followed by some simple scaling.
+
+If \(\lambda=1\), then \(w_t = y_t-1\), so the transformed data is shifted downwards but there is no change in the shape of the time series. For all other values of \(\lambda\), the time series will change shape.
+
+Use the slider below to see the effect of varying \(\lambda\) to transform Australian quarterly gas production:
+
+Figure 3.3: Box-Cox transformations applied to Australian quarterly gas production.
+
+A good value of \(\lambda\) is one which makes the size of the seasonal variation about the same across the whole series, as that makes the forecasting model simpler. In this case, \(\lambda=0.10\) works quite well, although any value of \(\lambda\) between 0.0 and 0.2 would give similar results.
+
+The `guerrero` feature ([Guerrero, 1993](#ref-Guerrero93)) can be used to choose a value of lambda for you. In this case it chooses \(\lambda=0.11\). (See the next chapter for discussion of the `features()` function.)
+
+```
+lambda <- aus_production |>
+  features(Gas, features = guerrero) |>
+  pull(lambda_guerrero)
+aus_production |>
+  autoplot(box_cox(Gas, lambda)) +
+  labs(y = "",
+       title = latex2exp::TeX(paste0(
+         "Transformed gas production with $\\lambda$ = ",
+         round(lambda,2))))
+```
+
+![Transformed Australian quarterly gas production with the $\lambda$ parameter chosen using the Guerrero method.](https://otexts.com/fpp3/fpp_files/figure-html/BoxCoxlambda-1.png)
+
+Figure 3.4: Transformed Australian quarterly gas production with the \(\lambda\) parameter chosen using the Guerrero method.
+
+### Bibliography
+
+Bickel, P. J., & Doksum, K. A. (1981). An analysis of transformations revisited. *Journal of the American Statistical Association*, *76*(374), 296–311.
+
+Box, G. E. P., & Cox, D. R. (1964). An analysis of transformations. *Journal of the Royal Statistical Society. Series B, Statistical Methodology*, *26*(2), 211–252.
+
+Guerrero, V. M. (1993). Time-series analysis supported by power transformations. *Journal of Forecasting*, *12*(1), 37–48.
+
+## 3.2 Time series components
+
+If we assume an additive decomposition, then we can write
+\[
+y_{t} = S_{t} + T_{t} + R_t,
+\]
+where \(y_{t}\) is the data, \(S_{t}\) is the seasonal component, \(T_{t}\) is the trend-cycle component, and \(R_t\) is the remainder component, all at period \(t\). Alternatively, a multiplicative decomposition would be written as
+\[
+y_{t} = S_{t} \times T_{t} \times R_t.
+\]
+
+The additive decomposition is the most appropriate if the magnitude of the seasonal fluctuations, or the variation around the trend-cycle, does not vary with the level of the time series. When the variation in the seasonal pattern, or the variation around the trend-cycle, appears to be proportional to the level of the time series, then a multiplicative decomposition is more appropriate. Multiplicative decompositions are common with economic time series.
+
+An alternative to using a multiplicative decomposition is to first transform the data until the variation in the series appears to be stable over time, then use an additive decomposition. When a log transformation has been used, this is equivalent to using a multiplicative decomposition on the original data because
+\[
+y_{t} = S_{t} \times T_{t} \times R_t \quad\text{is equivalent to}\quad
+\log y_{t} = \log S_{t} + \log T_{t} + \log R_t.
+\]
+
+### Example: Employment in the US retail sector
+
+We will look at several methods for obtaining the components \(S_{t}\), \(T_{t}\) and \(R_{t}\) later in this chapter, but first it is helpful to see an example. We will decompose the number of persons employed in retail as shown in Figure [3.5](https://otexts.com/fpp3/components.html#fig:usretailemployment). The data shows the total monthly number of persons in thousands employed in the retail sector across the US since 1990.
+
+```
+us_retail_employment <- us_employment |>
+  filter(year(Month) >= 1990, Title == "Retail Trade") |>
+  select(-Series_ID)
+autoplot(us_retail_employment, Employed) +
+  labs(y = "Persons (thousands)",
+       title = "Total employment in US retail")
+```
+
+![Total number of persons employed in US retail.](https://otexts.com/fpp3/fpp_files/figure-html/usretailemployment-1.png)
+
+Figure 3.5: Total number of persons employed in US retail.
+
+To illustrate the ideas, we will use the STL decomposition method, which is discussed in Section [3.6](https://otexts.com/fpp3/stl.html#stl).
+
+```
+dcmp <- us_retail_employment |>
+  model(stl = STL(Employed))
+components(dcmp)
+#> # A dable: 357 x 7 [1M]
+#> # Key:     .model [1]
+#> # :        Employed = trend + season_year + remainder
+#>    .model    Month Employed  trend season_year remainder season_adjust
+#>    <chr>     <mth>    <dbl>  <dbl>       <dbl>     <dbl>         <dbl>
+#>  1 stl    1990 Jan   13256. 13288.      -33.0      0.836        13289.
+#>  2 stl    1990 Feb   12966. 13269.     -258.     -44.6          13224.
+#>  3 stl    1990 Mar   12938. 13250.     -290.     -22.1          13228.
+#>  4 stl    1990 Apr   13012. 13231.     -220.       1.05         13232.
+#>  5 stl    1990 May   13108. 13211.     -114.      11.3          13223.
+#>  6 stl    1990 Jun   13183. 13192.      -24.3     15.5          13207.
+#>  7 stl    1990 Jul   13170. 13172.      -23.2     21.6          13193.
+#>  8 stl    1990 Aug   13160. 13151.       -9.52    17.8          13169.
+#>  9 stl    1990 Sep   13113. 13131.      -39.5     22.0          13153.
+#> 10 stl    1990 Oct   13185. 13110.       61.6     13.2          13124.
+#> # ℹ 347 more rows
+```
+
+The output above shows the components of an STL decomposition. The original data is shown (as `Employed`), followed by the estimated components. This output forms a “dable” or decomposition table. The header to the table shows that the `Employed` series has been decomposed additively.
+
+The `trend` column (containing the trend-cycle \(T_t\)) follows the overall movement of the series, ignoring any seasonality and random fluctuations, as shown in Figure [3.6](https://otexts.com/fpp3/components.html#fig:empltrend).
+
+```
+components(dcmp) |>
+  as_tsibble() |>
+  autoplot(Employed, colour="gray") +
+  geom_line(aes(y=trend), colour = "#D55E00") +
+  labs(
+    y = "Persons (thousands)",
+    title = "Total employment in US retail"
+  )
+```
+
+![Total number of persons employed in US retail: the trend-cycle component (orange) and the raw data (grey).](https://otexts.com/fpp3/fpp_files/figure-html/empltrend-1.png)
+
+Figure 3.6: Total number of persons employed in US retail: the trend-cycle component (orange) and the raw data (grey).
+
+We can plot all of the components in a single figure using `autoplot()`, as shown in Figure [3.7](https://otexts.com/fpp3/components.html#fig:emplstl).
+
+```
+components(dcmp) |> autoplot()
+```
+
+![The total number of persons employed in US retail (top) and its three additive components.](https://otexts.com/fpp3/fpp_files/figure-html/emplstl-1.png)
+
+Figure 3.7: The total number of persons employed in US retail (top) and its three additive components.
+
+The three components are shown separately in the bottom three panels. These components can be added together to reconstruct the data shown in the top panel. Notice that the seasonal component changes over time, so that any two consecutive years have similar patterns, but years far apart may have different seasonal patterns. The remainder component shown in the bottom panel is what is left over when the seasonal and trend-cycle components have been subtracted from the data.
+
+The grey bars to the left of each panel show the relative scales of the components. Each grey bar represents the same length but because the plots are on different scales, the bars vary in size. The large grey bar in the bottom panel shows that the variation in the remainder component is smallest compared to the variation in the data. If we shrank the bottom three panels until their bars became the same size as that in the data panel, then all the panels would be on the same scale.
+
+### Seasonally adjusted data
+
+If the seasonal component is removed from the original data, the resulting values are the “seasonally adjusted” data. For an additive decomposition, the seasonally adjusted data are given by \(y_{t}-S_{t}\), and for multiplicative data, the seasonally adjusted values are obtained using \(y_{t}/S_{t}\).
+
+Figure [3.8](https://otexts.com/fpp3/components.html#fig:empl-retail-sa) shows the seasonally adjusted number of persons employed.
+
+```
+components(dcmp) |>
+  as_tsibble() |>
+  autoplot(Employed, colour = "gray") +
+  geom_line(aes(y=season_adjust), colour = "#0072B2") +
+  labs(y = "Persons (thousands)",
+       title = "Total employment in US retail")
+```
+
+![Seasonally adjusted retail employment data (blue) and the original data (grey).](https://otexts.com/fpp3/fpp_files/figure-html/empl-retail-sa-1.png)
+
+Figure 3.8: Seasonally adjusted retail employment data (blue) and the original data (grey).
+
+If the variation due to seasonality is not of primary interest, the seasonally adjusted series can be useful. For example, monthly unemployment data are usually seasonally adjusted in order to highlight variation due to the underlying state of the economy rather than the seasonal variation. An increase in unemployment due to school leavers seeking work is seasonal variation, while an increase in unemployment due to an economic recession is non-seasonal. Most economic analysts who study unemployment data are more interested in the non-seasonal variation. Consequently, employment data (and many other economic series) are usually seasonally adjusted.
+
+Seasonally adjusted series contain the remainder component as well as the trend-cycle. Therefore, they are not “smooth”, and “downturns” or “upturns” can be misleading. If the purpose is to look for turning points in a series, and interpret any changes in direction, then it is better to use the trend-cycle component rather than the seasonally adjusted data.
+
+## 3.3 Moving averages
+
+The classical method of time series decomposition originated in the 1920s and was widely used until the 1950s. It still forms the basis of many time series decomposition methods, so it is important to understand how it works. The first step in a classical decomposition is to use a moving average method to estimate the trend-cycle, so we begin by discussing moving averages.
+
+### Moving average smoothing
+
+A moving average of order \(m\) can be written as
+\[\begin{equation}
+\hat{T}_{t} = \frac{1}{m} \sum_{j=-k}^k y_{t+j}, \tag{3.2}
+\end{equation}\]
+where \(m=2k+1\). That is, the estimate of the trend-cycle at time \(t\) is obtained by averaging values of the time series within \(k\) periods of \(t\). Observations that are nearby in time are also likely to be close in value. Therefore, the average eliminates some of the randomness in the data, leaving a smooth trend-cycle component. We call this an **\(m\)-MA**, meaning a moving average of order \(m\).
+
+For example, consider Figure [3.9](https://otexts.com/fpp3/moving-averages.html#fig:aus-exports) which shows exports of goods and services for Australia as a percentage of GDP from 1960 to 2017. The data are also shown in Table [3.1](https://otexts.com/fpp3/moving-averages.html#tab:aus-exports-tbl).
+
+```
+global_economy |>
+  filter(Country == "Australia") |>
+  autoplot(Exports) +
+  labs(y = "% of GDP", title = "Total Australian exports")
+```
+
+![Australian exports of goods and services: 1960--2017.](https://otexts.com/fpp3/fpp_files/figure-html/aus-exports-1.png)
+
+Figure 3.9: Australian exports of goods and services: 1960–2017.
+
+Table 3.1: Annual Australian exports of goods and services: 1960–2017.
+
+| Year | Exports | 5-MA |
+| --- | --- | --- |
+| 1960 | 12.99 |  |
+| 1961 | 12.40 |  |
+| 1962 | 13.94 | 13.46 |
+| 1963 | 13.01 | 13.50 |
+| 1964 | 14.94 | 13.61 |
+| 1965 | 13.22 | 13.40 |
+| 1966 | 12.93 | 13.25 |
+| 1967 | 12.88 | 12.66 |
+| … | … | … |
+| 2010 | 19.84 | 21.21 |
+| 2011 | 21.47 | 21.17 |
+| 2012 | 21.52 | 20.78 |
+| 2013 | 19.99 | 20.81 |
+| 2014 | 21.08 | 20.37 |
+| 2015 | 20.01 | 20.32 |
+| 2016 | 19.25 |  |
+| 2017 | 21.27 |  |
+
+In the last column of this table, a moving average of order 5 is shown, providing an estimate of the trend-cycle. The first value in this column is the average of the first five observations, 1960–1964; the second value in the 5-MA column is the average of the values for 1961–1965; and so on. Each value in the 5-MA column is the average of the observations in the five year window centred on the corresponding year. In the notation of Equation [(3.2)](https://otexts.com/fpp3/moving-averages.html#eq:ma), column 5-MA contains the values of \(\hat{T}_{t}\) with \(k=2\) and \(m=2k+1=5\). There are no values for either the first two years or the last two years, because we do not have two observations on either side. Later we will use more sophisticated methods of trend-cycle estimation which do allow estimates near the endpoints.
+
+This is easily computed using `slide_dbl()` from the `slider` package which applies a function to “sliding” time windows. In this case, we use the `mean()` function with a window of size 5.
+
+```
+aus_exports <- global_economy |>
+  filter(Country == "Australia") |>
+  mutate(
+    `5-MA` = slider::slide_dbl(Exports, mean,
+                .before = 2, .after = 2, .complete = TRUE)
+  )
+```
+
+To see what the trend-cycle estimate looks like, we plot it along with the original data in Figure [3.10](https://otexts.com/fpp3/moving-averages.html#fig:aus-exports-plot).
+
+```
+aus_exports |>
+  autoplot(Exports) +
+  geom_line(aes(y = `5-MA`), colour = "#D55E00") +
+  labs(y = "% of GDP",
+       title = "Total Australian exports")
+```
+
+![Australian exports (black) along with the 5-MA estimate of the trend-cycle (orange).](https://otexts.com/fpp3/fpp_files/figure-html/aus-exports-plot-1.png)
+
+Figure 3.10: Australian exports (black) along with the 5-MA estimate of the trend-cycle (orange).
+
+Notice that the trend-cycle (in orange) is smoother than the original data and captures the main movement of the time series without all of the minor fluctuations. The order of the moving average determines the smoothness of the trend-cycle estimate. In general, a larger order means a smoother curve. Figure [3.11](https://otexts.com/fpp3/moving-averages.html#fig:aus-exports-compare) shows the effect of changing the order of the moving average for the Australian exports data.
+
+![Different moving averages applied to the Australian exports data.](https://otexts.com/fpp3/fpp_files/figure-html/aus-exports-compare-1.png)
+
+Figure 3.11: Different moving averages applied to the Australian exports data.
+
+Simple moving averages such as these are usually of an odd order (e.g., 3, 5, 7, etc.). This is so they are symmetric: in a moving average of order \(m=2k+1\), the middle observation, and \(k\) observations on either side, are averaged. But if \(m\) was even, it would no longer be symmetric.
+
+### Moving averages of moving averages
+
+It is possible to apply a moving average to a moving average. One reason for doing this is to make an even-order moving average symmetric.
+
+For example, we might take a moving average of order 4, and then apply another moving average of order 2 to the results. In the following table, this has been done for the first few years of the Australian quarterly beer production data.
+
+```
+beer <- aus_production |>
+  filter(year(Quarter) >= 1992) |>
+  select(Quarter, Beer)
+beer_ma <- beer |>
+  mutate(
+    `4-MA` = slider::slide_dbl(Beer, mean,
+                .before = 1, .after = 2, .complete = TRUE),
+    `2x4-MA` = slider::slide_dbl(`4-MA`, mean,
+                .before = 1, .after = 0, .complete = TRUE)
+  )
+```
+
+Table 3.2: A moving average of order 4 applied to the quarterly beer data, followed by a moving average of order 2.
+
+| Quarter | Beer | 4-MA | 2x4-MA |
+| --- | --- | --- | --- |
+| 1992 Q1 | 443.00 |  |  |
+| 1992 Q2 | 410.00 | 451.25 |  |
+| 1992 Q3 | 420.00 | 448.75 | 450.00 |
+| 1992 Q4 | 532.00 | 451.50 | 450.12 |
+| 1993 Q1 | 433.00 | 449.00 | 450.25 |
+| 1993 Q2 | 421.00 | 444.00 | 446.50 |
+| … | … | … | … |
+| 2009 Q1 | 415.00 | 430.00 | 428.88 |
+| 2009 Q2 | 398.00 | 430.00 | 430.00 |
+| 2009 Q3 | 419.00 | 429.75 | 429.88 |
+| 2009 Q4 | 488.00 | 423.75 | 426.75 |
+| 2010 Q1 | 414.00 |  |  |
+| 2010 Q2 | 374.00 |  |  |
+
+The notation “\(2\times4\)-MA” in the last column means a 4-MA followed by a 2-MA. The values in the last column are obtained by taking a moving average of order 2 of the values in the previous column. For example, the first two values in the 4-MA column are
+451.25=(443+410+420+532)/4
+and
+448.75=(410+420+532+433)/4.
+The first value in the 2x4-MA column is the average of these two:
+450.00=(451.25+448.75)/2.
+
+When a 2-MA follows a moving average of an even order (such as 4), it is called a “centred moving average of order 4”. This is because the results are now symmetric. To see that this is the case, we can write the \(2\times4\)-MA as follows:
+\[\begin{align\*}
+\hat{T}_{t} &= \frac{1}{2}\Big[
+\frac{1}{4} (y_{t-2}+y_{t-1}+y_{t}+y_{t+1}) +
+\frac{1}{4} (y_{t-1}+y_{t}+y_{t+1}+y_{t+2})\Big] \\
+&= \frac{1}{8}y_{t-2}+\frac14y_{t-1} +
+\frac14y_{t}+\frac14y_{t+1}+\frac18y_{t+2}.
+\end{align\*}\]
+It is now a weighted average of observations that is symmetric.
+
+Other combinations of moving averages are also possible. For example, a \(3\times3\)-MA is often used, and consists of a moving average of order 3 followed by another moving average of order 3. In general, an even order MA should be followed by an even order MA to make it symmetric. Similarly, an odd order MA should be followed by an odd order MA.
+
+### Estimating the trend-cycle with seasonal data
+
+The most common use of centred moving averages is for estimating the trend-cycle from seasonal data. Consider the \(2\times4\)-MA:
+\[
+\hat{T}_{t} = \frac{1}{8}y_{t-2} + \frac14y_{t-1} +
+\frac14y_{t} + \frac14y_{t+1} + \frac18y_{t+2}.
+\]
+When applied to quarterly data, each quarter of the year is given equal weight as the first and last terms apply to the same quarter in consecutive years. Consequently, the seasonal variation will be averaged out and the resulting values of \(\hat{T}_t\) will have little or no seasonal variation remaining. A similar effect would be obtained using a \(2\times 8\)-MA or a \(2\times 12\)-MA to quarterly data.
+
+In general, a \(2\times m\)-MA is equivalent to a weighted moving average of order \(m+1\) where all observations take the weight \(1/m\), except for the first and last terms which take weights \(1/(2m)\). So, if the seasonal period is even and of order \(m\), we use a \(2\times m\)-MA to estimate the trend-cycle. If the seasonal period is odd and of order \(m\), we use a \(m\)-MA to estimate the trend-cycle. For example, a \(2\times 12\)-MA can be used to estimate the trend-cycle of monthly data with annual seasonality and a 7-MA can be used to estimate the trend-cycle of daily data with a weekly seasonality.
+
+Other choices for the order of the MA will usually result in trend-cycle estimates being contaminated by the seasonality in the data.
+
+### Example: Employment in the US retail sector
+
+```
+us_retail_employment_ma <- us_retail_employment |>
+  mutate(
+    `12-MA` = slider::slide_dbl(Employed, mean,
+                .before = 5, .after = 6, .complete = TRUE),
+    `2x12-MA` = slider::slide_dbl(`12-MA`, mean,
+                .before = 1, .after = 0, .complete = TRUE)
+  )
+us_retail_employment_ma |>
+  autoplot(Employed, colour = "gray") +
+  geom_line(aes(y = `2x12-MA`), colour = "#D55E00") +
+  labs(y = "Persons (thousands)",
+       title = "Total employment in US retail")
+```
+
+![A 2x12-MA applied to the US retail employment series.](https://otexts.com/fpp3/fpp_files/figure-html/empl-MA-1.png)
+
+Figure 3.12: A 2x12-MA applied to the US retail employment series.
+
+Figure [3.12](https://otexts.com/fpp3/moving-averages.html#fig:empl-MA) shows a \(2\times12\)-MA applied to the total number of persons employed in the US retail sector. Notice that the smooth line shows no seasonality; it is almost the same as the trend-cycle shown in Figure [3.6](https://otexts.com/fpp3/components.html#fig:empltrend), which was estimated using a much more sophisticated method than a moving average. Any other choice for the order of the moving average (except for 24, 36, etc.) would have resulted in a smooth line that showed some seasonal fluctuations.
+
+### Weighted moving averages
+
+Combinations of moving averages result in weighted moving averages. For example, the \(2\times4\)-MA discussed above is equivalent to a weighted 5-MA with weights given by
+\(\left[\frac{1}{8},\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{8}\right]\). In general, a weighted \(m\)-MA can be written as
+\[
+\hat{T}_t = \sum_{j=-k}^k a_j y_{t+j},
+\]
+where \(k=(m-1)/2\), and the weights are given by \(\left[a_{-k},\dots,a_k\right]\). It is important that the weights all sum to one and that they are symmetric so that \(a_j = a_{-j}\). The simple \(m\)-MA is a special case where all of the weights are equal to \(1/m\).
+
+A major advantage of weighted moving averages is that they yield a smoother estimate of the trend-cycle. Instead of observations entering and leaving the calculation at full weight, their weights slowly increase and then slowly decrease, resulting in a smoother curve.
+
+## 3.4 Classical decomposition
+
+The classical decomposition method originated in the 1920s. It is a relatively simple procedure, and forms the starting point for most other methods of time series decomposition. There are two forms of classical decomposition: an additive decomposition and a multiplicative decomposition. These are described below for a time series with seasonal period \(m\) (e.g., \(m=4\) for quarterly data, \(m=12\) for monthly data, \(m=7\) for daily data with a weekly pattern).
+
+In classical decomposition, we assume that the seasonal component is constant from year to year. For multiplicative seasonality, the \(m\) values that form the seasonal component are sometimes called the “seasonal indices”.
+
+### Additive decomposition
+
+Step 1
+:   If \(m\) is an even number, compute the trend-cycle component \(\hat{T}_t\) using a \(2\times m\)-MA. If \(m\) is an odd number, compute the trend-cycle component \(\hat{T}_t\) using an \(m\)-MA.
+
+Step 2
+:   Calculate the detrended series: \(y_t - \hat{T}_t\).
+
+Step 3
+:   To estimate the seasonal component for each season, simply average the detrended values for that season. For example, with monthly data, the seasonal component for March is the average of all the detrended March values in the data. These seasonal component values are then adjusted to ensure that they add to zero. The seasonal component is obtained by stringing together these monthly values, and then replicating the sequence for each year of data. This gives \(\hat{S}_t\).
+
+Step 4
+:   The remainder component is calculated by subtracting the estimated seasonal and trend-cycle components: \(\hat{R}_t = y_t - \hat{T}_t - \hat{S}_t\).
+
+Figure [3.13](https://otexts.com/fpp3/classical-decomposition.html#fig:classical-empl) shows a classical decomposition of the total retail employment series across the US.
+
+```
+us_retail_employment |>
+  model(
+    classical_decomposition(Employed, type = "additive")
+  ) |>
+  components() |>
+  autoplot() +
+  labs(title = "Classical additive decomposition of total
+                  US retail employment")
+```
+
+![A classical additive decomposition of US retail employment.](https://otexts.com/fpp3/fpp_files/figure-html/classical-empl-1.png)
+
+Figure 3.13: A classical additive decomposition of US retail employment.
+
+### Multiplicative decomposition
+
+A classical multiplicative decomposition is similar, except that the subtractions are replaced by divisions.
+
+Step 1
+:   If \(m\) is an even number, compute the trend-cycle component \(\hat{T}_t\) using a \(2\times m\)-MA. If \(m\) is an odd number, compute the trend-cycle component \(\hat{T}_t\) using an \(m\)-MA.
+
+Step 2
+:   Calculate the detrended series: \(y_t/ \hat{T}_t\).
+
+Step 3
+:   To estimate the seasonal component for each season, simply average the detrended values for that season. For example, with monthly data, the seasonal index for March is the average of all the detrended March values in the data. These seasonal indexes are then adjusted to ensure that they add to \(m\). The seasonal component is obtained by stringing together these monthly indexes, and then replicating the sequence for each year of data. This gives \(\hat{S}_t\).
+
+Step 4
+:   The remainder component is calculated by dividing out the estimated seasonal and trend-cycle components: \(\hat{R}_{t} = y_t /( \hat{T}_t \hat{S}_t)\).
+
+### Comments on classical decomposition
+
+While classical decomposition is still widely used, it is not recommended, as there are now several much better methods. Some of the problems with classical decomposition are summarised below.
+
+* The estimate of the trend-cycle is unavailable for the first few and last few observations. For example, if \(m=12\), there is no trend-cycle estimate for the first six or the last six observations. Consequently, there is also no estimate of the remainder component for the same time periods.
+* The trend-cycle estimate tends to over-smooth rapid rises and falls in the data.
+* Classical decomposition methods assume that the seasonal component repeats from year to year. For many series, this is a reasonable assumption, but for some longer series it is not. For example, electricity demand patterns have changed over time as air conditioning has become more widespread. In many locations, the seasonal usage pattern from several decades ago had its maximum demand in winter (due to heating), while the current seasonal pattern has its maximum demand in summer (due to air conditioning). Classical decomposition methods are unable to capture these seasonal changes over time.
+* Occasionally, the values of the time series in a small number of periods may be particularly unusual. For example, the monthly air passenger traffic may be affected by an industrial dispute, making the traffic during the dispute different from usual. The classical method is not robust to these kinds of unusual values.
+
+## 3.5 Methods used by official statistics agencies
+
+Official statistics agencies (such as the US Census Bureau and the Australian Bureau of Statistics) are responsible for a large number of official economic and social time series. These agencies have developed their own decomposition procedures which are used for seasonal adjustment. Most of them use variants of the X-11 method, or the SEATS method, or a combination of the two. These methods are designed specifically to work with quarterly and monthly data, which are the most common series handled by official statistics agencies. They will not handle seasonality of other kinds, such as daily data, or hourly data, or weekly data. We will use the latest implementation of this group of methods known as “X-13ARIMA-SEATS”. For the methods discussed in this section, you will need to have installed the `seasonal` package in R.
+
+### X-11 method
+
+The X-11 method originated in the US Census Bureau and was further developed by Statistics Canada. It is based on classical decomposition, but includes many extra steps and features in order to overcome the drawbacks of classical decomposition that were discussed in the previous section. In particular, trend-cycle estimates are available for all observations including the end points, and the seasonal component is allowed to vary slowly over time. X-11 also handles trading day variation, holiday effects and the effects of known predictors. There are methods for both additive and multiplicative decomposition. The process is entirely automatic and tends to be highly robust to outliers and level shifts in the time series. The details of the X-11 method are described in Dagum & Bianconcini ([2016](#ref-Dagum2016)).
+
+```
+x11_dcmp <- us_retail_employment |>
+  model(x11 = X_13ARIMA_SEATS(Employed ~ x11())) |>
+  components()
+autoplot(x11_dcmp) +
+  labs(title =
+    "Decomposition of total US retail employment using X-11.")
+```
+
+![A multiplicative decomposition of US retail employment using X-11.](https://otexts.com/fpp3/fpp_files/figure-html/x11-1.png)
+
+Figure 3.14: A multiplicative decomposition of US retail employment using X-11.
+
+Compare this decomposition with the STL decomposition shown in Figure [3.7](https://otexts.com/fpp3/components.html#fig:emplstl) and the classical decomposition shown in Figure [3.13](https://otexts.com/fpp3/classical-decomposition.html#fig:classical-empl). The default approach for `X_13ARIMA_SEATS` shown here is a multiplicative decomposition, whereas the STL and classical decompositions shown earlier were additive; but it doesn’t make much difference in this case. The X-11 trend-cycle has captured the sudden fall in the data due to the 2007–2008 global financial crisis better than either of the other two methods (where the effect of the crisis has leaked into the remainder component). Also, the unusual observation in 1996 is now more clearly seen in the X-11 remainder component.
+
+Figure [3.15](https://otexts.com/fpp3/methods-used-by-official-statistics-agencies.html#fig:x11-seasadj) shows the trend-cycle component and the seasonally adjusted data, along with the original data. The seasonally adjusted data is very similar to the trend-cycle component in this example, so it is hard to distinguish them on the plot.
+
+```
+x11_dcmp |>
+  ggplot(aes(x = Month)) +
+  geom_line(aes(y = Employed, colour = "Data")) +
+  geom_line(aes(y = season_adjust,
+                colour = "Seasonally Adjusted")) +
+  geom_line(aes(y = trend, colour = "Trend")) +
+  labs(y = "Persons (thousands)",
+       title = "Total employment in US retail") +
+  scale_colour_manual(
+    values = c("gray", "#0072B2", "#D55E00"),
+    breaks = c("Data", "Seasonally Adjusted", "Trend")
+  )
+```
+
+![US retail employment: the original data (grey), the trend-cycle component (orange) and the seasonally adjusted data (barely visible in blue).](https://otexts.com/fpp3/fpp_files/figure-html/x11-seasadj-1.png)
+
+Figure 3.15: US retail employment: the original data (grey), the trend-cycle component (orange) and the seasonally adjusted data (barely visible in blue).
+
+It can be useful to use seasonal plots and seasonal sub-series plots of the seasonal component, to help us visualise the variation in the seasonal component over time. Figure [3.16](https://otexts.com/fpp3/methods-used-by-official-statistics-agencies.html#fig:print-media3) shows a seasonal sub-series plot of the seasonal component from Figure [3.14](https://otexts.com/fpp3/methods-used-by-official-statistics-agencies.html#fig:x11). In this case, there are only small changes over time.
+
+```
+x11_dcmp |>
+  gg_subseries(seasonal)
+```
+
+![Seasonal sub-series plot of the seasonal component from the X-11 method applied to total US retail employment.](https://otexts.com/fpp3/fpp_files/figure-html/print-media3-1.png)
+
+Figure 3.16: Seasonal sub-series plot of the seasonal component from the X-11 method applied to total US retail employment.
+
+### SEATS method
+
+“SEATS” stands for “Seasonal Extraction in ARIMA Time Series” (ARIMA models are discussed in Chapter [9](https://otexts.com/fpp3/arima.html#arima)). This procedure was developed at the Bank of Spain, and is now widely used by government agencies around the world. The details are beyond the scope of this book. However, a complete discussion of the method is available in Dagum & Bianconcini ([2016](#ref-Dagum2016)).
+
+```
+seats_dcmp <- us_retail_employment |>
+  model(seats = X_13ARIMA_SEATS(Employed ~ seats())) |>
+  components()
+autoplot(seats_dcmp) +
+  labs(title =
+    "Decomposition of total US retail employment using SEATS")
+```
+
+![A  decomposition of US retail employment obtained using SEATS.](https://otexts.com/fpp3/fpp_files/figure-html/seats-1.png)
+
+Figure 3.17: A decomposition of US retail employment obtained using SEATS.
+
+Figure [3.17](https://otexts.com/fpp3/methods-used-by-official-statistics-agencies.html#fig:seats) shows the SEATS method applied to the total retail employment series across the US. The result is quite similar to that obtained using the X-11 method shown in Figure [3.14](https://otexts.com/fpp3/methods-used-by-official-statistics-agencies.html#fig:x11).
+
+The `X_13ARIMA_SEATS()` function calls the `seasonal` package which has many options for handling variations of X-11 and SEATS. See [the package website](https://bit.ly/seaspkg) for a detailed introduction to the options and features available.
+
+### Bibliography
+
+Dagum, E. B., & Bianconcini, S. (2016). *Seasonal adjustment methods and real time trend-cycle estimation*. Springer.
+
+## 3.6 STL decomposition
+
+STL is a versatile and robust method for decomposing time series. STL is an acronym for “Seasonal and Trend decomposition using Loess”, while loess is a method for estimating nonlinear relationships. The STL method was developed by R. B. Cleveland et al. ([1990](#ref-Cleveland1990)), and later extended to handle multiple seasonal patterns by Bandara et al. ([2025](#ref-mstl)).
+
+STL has several advantages over classical decomposition, and the SEATS and X-11 methods:
+
+* Unlike SEATS and X-11, STL will handle any type of seasonality, not only monthly and quarterly data.
+* The seasonal component is allowed to change over time, and the rate of change can be controlled by the user.
+* The smoothness of the trend-cycle can also be controlled by the user.
+* It can be robust to outliers (i.e., the user can specify a robust decomposition), so that occasional unusual observations will not affect the estimates of the trend-cycle and seasonal components. They will, however, affect the remainder component.
+
+On the other hand, STL has some disadvantages. In particular, it does not handle trading day or calendar variation automatically, and it only provides facilities for additive decompositions.
+
+A multiplicative decomposition can be obtained by first taking logs of the data, then back-transforming the components. Decompositions that are between additive and multiplicative can be obtained using a Box-Cox transformation of the data with \(0<\lambda<1\). A value of \(\lambda=0\) gives a multiplicative decomposition while \(\lambda=1\) gives an additive decomposition.
+
+The best way to begin learning how to use STL is to see some examples and experiment with the settings. Figure [3.7](https://otexts.com/fpp3/components.html#fig:emplstl) showed an example of an STL decomposition applied to the total US retail employment series. Figure [3.18](https://otexts.com/fpp3/stl.html#fig:empl-stl2) shows an alternative STL decomposition where the trend-cycle is more flexible, the seasonal pattern is fixed, and the robust option has been used.
+
+```
+us_retail_employment |>
+  model(
+    STL(Employed ~ trend(window = 7) +
+                   season(window = "periodic"),
+    robust = TRUE)) |>
+  components() |>
+  autoplot()
+```
+
+![Total US retail employment (top) and its three additive components obtained from a robust STL decomposition with flexible trend-cycle and fixed seasonality.](https://otexts.com/fpp3/fpp_files/figure-html/empl-stl2-1.png)
+
+Figure 3.18: Total US retail employment (top) and its three additive components obtained from a robust STL decomposition with flexible trend-cycle and fixed seasonality.
+
+The two main parameters to be chosen when using STL are the trend-cycle window `trend(window = ?)` and the seasonal window `season(window = ?)`. These control how rapidly the trend-cycle and seasonal components can change. Smaller values allow for more rapid changes. Both trend and seasonal windows should be odd numbers; trend window is the number of consecutive observations to be used when estimating the trend-cycle; season window is the number of consecutive years to be used in estimating each value in the seasonal component. Setting the seasonal window to be infinite is equivalent to forcing the seasonal component to be periodic `season(window='periodic')` (i.e., identical across years). This was the case in Figure [3.18](https://otexts.com/fpp3/stl.html#fig:empl-stl2).
+
+By default, the `STL()` function provides a convenient automated STL decomposition using a seasonal window of `season(window=11)` when there is a single seasonal period, and the trend window chosen automatically from the seasonal period. The default setting for monthly data is `trend(window=21)`. For multiple seasonal periods, the default seasonal windows are 11, 15, 19, etc., with larger windows corresponding to larger seasonal periods. This usually gives a good balance between overfitting the seasonality and allowing it to slowly change over time. But, as with any automated procedure, the default settings will need adjusting for some time series. In the example shown in Figure [3.7](https://otexts.com/fpp3/components.html#fig:emplstl), the default trend window setting produces a trend-cycle component that is too rigid. As a result, signal from the 2008 global financial crisis has leaked into the remainder component, as can be seen in the bottom panel of Figure [3.7](https://otexts.com/fpp3/components.html#fig:emplstl). Selecting a shorter trend window as in Figure [3.18](https://otexts.com/fpp3/stl.html#fig:empl-stl2) improves this.
+
+### Bibliography
+
+Bandara, K., Hyndman, R. J., & Bergmeir, C. (2025). MSTL: A seasonal-trend decomposition algorithm for time series with multiple seasonal patterns. *International J Operational Research*, *52*(1).
+
+Cleveland, R. B., Cleveland, W. S., McRae, J. E., & Terpenning, I. J. (1990). STL: A seasonal-trend decomposition procedure based on loess. *Journal of Official Statistics*, *6*(1), 3–33.
+
+## 3.7 Exercises
+
+1. Consider the GDP information in `global_economy`. Plot the GDP per capita for each country over time. Which country has the highest GDP per capita? How has this changed over time?
+2. For each of the following series, make a graph of the data. If transforming seems appropriate, do so and describe the effect.
+
+   * United States GDP from `global_economy`.
+   * Slaughter of Victorian “Bulls, bullocks and steers” in `aus_livestock`.
+   * Victorian Electricity Demand from `vic_elec`.
+   * Gas production from `aus_production`.
+3. Why is a Box-Cox transformation unhelpful for the `canadian_gas` data?
+4. What Box-Cox transformation would you select for your retail data (from Exercise 7 in Section [2.10](https://otexts.com/fpp3/graphics-exercises.html#graphics-exercises))?
+5. For the following series, find an appropriate Box-Cox transformation in order to stabilise the variance. Tobacco from `aus_production`, Economy class passengers between Melbourne and Sydney from `ansett`, and Pedestrian counts at Southern Cross Station from `pedestrian`.
+6. Show that a \(3\times5\) MA is equivalent to a 7-term weighted moving average with weights of 0.067, 0.133, 0.200, 0.200, 0.200, 0.133, and 0.067.
+7. Consider the last five years of the Gas data from `aus_production`.
+
+   ```
+   gas <- tail(aus_production, 5*4) |> select(Gas)
+   ```
+
+   1. Plot the time series. Can you identify seasonal fluctuations and/or a trend-cycle?
+   2. Use `classical_decomposition` with `type=multiplicative` to calculate the trend-cycle and seasonal indices.
+   3. Do the results support the graphical interpretation from part a?
+   4. Compute and plot the seasonally adjusted data.
+   5. Change one observation to be an outlier (e.g., add 300 to one observation), and recompute the seasonally adjusted data. What is the effect of the outlier?
+   6. Does it make any difference if the outlier is near the end rather than in the middle of the time series?
+8. Recall your retail time series data (from Exercise 7 in Section [2.10](https://otexts.com/fpp3/graphics-exercises.html#graphics-exercises)).
+   Decompose the series using X-11. Does it reveal any outliers, or unusual features that you had not noticed previously?
+9. Figures [3.19](https://otexts.com/fpp3/decomposition-exercises.html#fig:labour) and [3.20](https://otexts.com/fpp3/decomposition-exercises.html#fig:labour2) show the result of decomposing the number of persons in the civilian labour force in Australia each month from February 1978 to August 1995.
+
+   ![Decomposition of the number of persons in the civilian labour force in Australia each month from February 1978 to August 1995.](https://otexts.com/fpp3/fpp_files/figure-html/labour-1.png)
+
+   Figure 3.19: Decomposition of the number of persons in the civilian labour force in Australia each month from February 1978 to August 1995.
+
+   ![Seasonal component from the decomposition shown in the previous figure.](https://otexts.com/fpp3/fpp_files/figure-html/labour2-1.png)
+
+   Figure 3.20: Seasonal component from the decomposition shown in the previous figure.
+
+   1. Write about 3–5 sentences describing the results of the decomposition. Pay particular attention to the scales of the graphs in making your interpretation.
+   2. Is the recession of 1991/1992 visible in the estimated components?
+10. This exercise uses the `canadian_gas` data (monthly Canadian gas production in billions of cubic metres, January 1960 – February 2005).
+
+    1. Plot the data using `autoplot()`, `gg_subseries()` and `gg_season()` to look at the effect of the changing seasonality over time.[3](#fn3)
+    2. Do an STL decomposition of the data. You will need to choose a seasonal window to allow for the changing shape of the seasonal component.
+    3. How does the seasonal shape change over time? [Hint: Try plotting the seasonal component using `gg_season()`.]
+    4. Can you produce a plausible seasonally adjusted series?
+    5. Compare the results with those obtained using SEATS and X-11. How are they different?
+
+---
+
+3. The evolving seasonal pattern is possibly due to changes in the regulation of gas prices — thanks to Lewis Kirvan for pointing this out.[↩︎](https://otexts.com/fpp3/decomposition-exercises.html#fnref3)
+
+## 3.8 Further reading
+
+* A detailed modern discussion of the SEATS and X-11 methods is provided by Dagum & Bianconcini ([2016](#ref-Dagum2016)).
+* R. B. Cleveland et al. ([1990](#ref-Cleveland1990)) introduced STL, and still provides the best description of the algorithm.
+
+### Bibliography
+
+Cleveland, R. B., Cleveland, W. S., McRae, J. E., & Terpenning, I. J. (1990). STL: A seasonal-trend decomposition procedure based on loess. *Journal of Official Statistics*, *6*(1), 3–33.
+
+Dagum, E. B., & Bianconcini, S. (2016). *Seasonal adjustment methods and real time trend-cycle estimation*. Springer.

docs/evidence/fpp3/04-time-series-features.md

@@ -0,0 +1,355 @@
+Source: https://otexts.com/fpp3/features.html (chapter features, 8 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 04-time-series-features
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Chapter 4 Time series features
+
+The `feasts` package includes functions for Feature Extraction And Statistics from Time Series (hence the name). We have already seen some time series features. For example, the autocorrelations discussed in Section [2.8](https://otexts.com/fpp3/acf.html#acf) can be considered features of a time series — they are numerical summaries computed from the series. Another feature we saw in the last chapter was the Guerrero estimate of the Box-Cox transformation parameter — again, this is a number computed from a time series.
+
+We can compute many different features on many different time series, and use them to explore the properties of the series. In this chapter we will look at some features that have been found useful in time series exploration, and how they can be used to uncover interesting information about your data. We will use Australian quarterly tourism as a running example (previously discussed in Section [2.5](https://otexts.com/fpp3/subseries.html#subseries)).
+
+## 4.1 Some simple statistics
+
+Any numerical summary computed from a time series is a feature of that time series — the mean, minimum or maximum, for example. These can be computed using the `features()` function. For example, let’s compute the means of all the series in the Australian tourism data.
+
+```
+tourism |>
+  features(Trips, list(mean = mean)) |>
+  arrange(mean)
+#> # A tibble: 304 × 4
+#>    Region          State              Purpose   mean
+#>    <chr>           <chr>              <chr>    <dbl>
+#>  1 Kangaroo Island South Australia    Other    0.340
+#>  2 MacDonnell      Northern Territory Other    0.449
+#>  3 Wilderness West Tasmania           Other    0.478
+#>  4 Barkly          Northern Territory Other    0.632
+#>  5 Clare Valley    South Australia    Other    0.898
+#>  6 Barossa         South Australia    Other    1.02
+#>  7 Kakadu Arnhem   Northern Territory Other    1.04
+#>  8 Lasseter        Northern Territory Other    1.14
+#>  9 Wimmera         Victoria           Other    1.15
+#> 10 MacDonnell      Northern Territory Visiting 1.18
+#> # ℹ 294 more rows
+```
+
+Here we see that the series with least average number of visits was “Other” visits to Kangaroo Island in South Australia.
+
+Rather than compute one feature at a time, it is convenient to compute many features at once. A common short summary of a data set is to compute five summary statistics: the minimum, first quartile, median, third quartile and maximum. These divide the data into four equal-size sections, each containing 25% of the data. The `quantile()` function can be used to compute them.
+
+```
+tourism |> features(Trips, quantile)
+#> # A tibble: 304 × 8
+#>    Region         State           Purpose    `0%`  `25%`   `50%`  `75%` `100%`
+#>    <chr>          <chr>           <chr>     <dbl>  <dbl>   <dbl>  <dbl>  <dbl>
+#>  1 Adelaide       South Australia Busine…  68.7   134.   153.    177.   242.
+#>  2 Adelaide       South Australia Holiday 108.    135.   154.    172.   224.
+#>  3 Adelaide       South Australia Other    25.9    43.9   53.8    62.5  107.
+#>  4 Adelaide       South Australia Visiti… 137.    179.   206.    229.   270.
+#>  5 Adelaide Hills South Australia Busine…   0       0      1.26    3.92  28.6
+#>  6 Adelaide Hills South Australia Holiday   0       5.77   8.52   14.1   35.8
+#>  7 Adelaide Hills South Australia Other     0       0      0.908   2.09   8.95
+#>  8 Adelaide Hills South Australia Visiti…   0.778   8.91  12.2    16.8   81.1
+#>  9 Alice Springs  Northern Terri… Busine…   1.01    9.13  13.3    18.5   34.1
+#> 10 Alice Springs  Northern Terri… Holiday   2.81   16.9   31.5    44.8   76.5
+#> # ℹ 294 more rows
+```
+
+Here the minimum is labelled `0%` and the maximum is labelled `100%`.
+
+## 4.2 ACF features
+
+Autocorrelations were discussed in Section [2.8](https://otexts.com/fpp3/acf.html#acf). All the autocorrelations of a series can be considered features of that series. We can also summarise the autocorrelations to produce new features; for example, the sum of the first ten squared autocorrelation coefficients is a useful summary of how much autocorrelation there is in a series, regardless of lag.
+
+We can also compute autocorrelations of the changes in the series between periods. That is, we “difference” the data and create a new time series consisting of the differences between consecutive observations. Then we can compute the autocorrelations of this new differenced series. Occasionally it is useful to apply the same differencing operation again, so we compute the differences of the differences. The autocorrelations of this double differenced series may provide useful information.
+
+Another related approach is to compute seasonal differences of a series. If we had monthly data, for example, we would compute the difference between consecutive Januaries, consecutive Februaries, and so on. This enables us to look at how the series is changing between years, rather than between months. Again, the autocorrelations of the seasonally differenced series may provide useful information.
+
+We discuss differencing of time series in more detail in Section [9.1](https://otexts.com/fpp3/stationarity.html#stationarity).
+
+The `feat_acf()` function computes a selection of the autocorrelations discussed here. It will return six or seven features:
+
+* the first autocorrelation coefficient from the original data;
+* the sum of squares of the first ten autocorrelation coefficients from the original data;
+* the first autocorrelation coefficient from the differenced data;
+* the sum of squares of the first ten autocorrelation coefficients from the differenced data;
+* the first autocorrelation coefficient from the twice differenced data;
+* the sum of squares of the first ten autocorrelation coefficients from the twice differenced data;
+* For seasonal data, the autocorrelation coefficient at the first seasonal lag is also returned.
+
+When applied to the Australian tourism data, we get the following output.
+
+```
+tourism |> features(Trips, feat_acf)
+#> # A tibble: 304 × 10
+#>    Region       State Purpose     acf1 acf10 diff1_acf1 diff1_acf10 diff2_acf1
+#>    <chr>        <chr> <chr>      <dbl> <dbl>      <dbl>       <dbl>      <dbl>
+#>  1 Adelaide     Sout… Busine…  0.0333  0.131     -0.520       0.463     -0.676
+#>  2 Adelaide     Sout… Holiday  0.0456  0.372     -0.343       0.614     -0.487
+#>  3 Adelaide     Sout… Other    0.517   1.15      -0.409       0.383     -0.675
+#>  4 Adelaide     Sout… Visiti…  0.0684  0.294     -0.394       0.452     -0.518
+#>  5 Adelaide Hi… Sout… Busine…  0.0709  0.134     -0.580       0.415     -0.750
+#>  6 Adelaide Hi… Sout… Holiday  0.131   0.313     -0.536       0.500     -0.716
+#>  7 Adelaide Hi… Sout… Other    0.261   0.330     -0.253       0.317     -0.457
+#>  8 Adelaide Hi… Sout… Visiti…  0.139   0.117     -0.472       0.239     -0.626
+#>  9 Alice Sprin… Nort… Busine…  0.217   0.367     -0.500       0.381     -0.658
+#> 10 Alice Sprin… Nort… Holiday -0.00660 2.11      -0.153       2.11      -0.274
+#> # ℹ 294 more rows
+#> # ℹ 2 more variables: diff2_acf10 <dbl>, season_acf1 <dbl>
+```
+
+## 4.3 STL Features
+
+The STL decomposition discussed in Chapter [3](https://otexts.com/fpp3/decomposition.html#decomposition) is the basis for several more features.
+
+A time series decomposition can be used to measure the strength of trend and seasonality in a time series. Recall that the decomposition is written as
+\[
+y_t = T_t + S_{t} + R_t,
+\]
+where \(T_t\) is the smoothed trend component, \(S_{t}\) is the seasonal component and \(R_t\) is a remainder component. For strongly trended data, the seasonally adjusted data should have much more variation than the remainder component. Therefore Var\((R_t)\)/Var\((T_t+R_t)\) should be relatively small. But for data with little or no trend, the two variances should be approximately the same. So we define the strength of trend as:
+\[
+F_T = \max\left(0, 1 - \frac{\text{Var}(R_t)}{\text{Var}(T_t+R_t)}\right).
+\]
+This will give a measure of the strength of the trend between 0 and 1. Because the variance of the remainder might occasionally be even larger than the variance of the seasonally adjusted data, we set the minimal possible value of \(F_T\) equal to zero.
+
+The strength of seasonality is defined similarly, but with respect to the detrended data rather than the seasonally adjusted data:
+\[
+F_S = \max\left(0, 1 - \frac{\text{Var}(R_t)}{\text{Var}(S_{t}+R_t)}\right).
+\]
+A series with seasonal strength \(F_S\) close to 0 exhibits almost no seasonality, while a series with strong seasonality will have \(F_S\) close to 1 because Var\((R_t)\) will be much smaller than Var\((S_{t}+R_t)\).
+
+These measures can be useful, for example, when you have a large collection of time series, and you need to find the series with the most trend or the most seasonality. These and other STL-based features are computed using the `feat_stl()` function.
+
+```
+tourism |>
+  features(Trips, feat_stl)
+#> # A tibble: 304 × 12
+#>    Region         State          Purpose trend_strength seasonal_strength_year
+#>    <chr>          <chr>          <chr>            <dbl>                  <dbl>
+#>  1 Adelaide       South Austral… Busine…          0.464                  0.407
+#>  2 Adelaide       South Austral… Holiday          0.554                  0.619
+#>  3 Adelaide       South Austral… Other            0.746                  0.202
+#>  4 Adelaide       South Austral… Visiti…          0.435                  0.452
+#>  5 Adelaide Hills South Austral… Busine…          0.464                  0.179
+#>  6 Adelaide Hills South Austral… Holiday          0.528                  0.296
+#>  7 Adelaide Hills South Austral… Other            0.593                  0.404
+#>  8 Adelaide Hills South Austral… Visiti…          0.488                  0.254
+#>  9 Alice Springs  Northern Terr… Busine…          0.534                  0.251
+#> 10 Alice Springs  Northern Terr… Holiday          0.381                  0.832
+#> # ℹ 294 more rows
+#> # ℹ 7 more variables: seasonal_peak_year <dbl>, seasonal_trough_year <dbl>,
+#> #   spikiness <dbl>, linearity <dbl>, curvature <dbl>, stl_e_acf1 <dbl>,
+#> #   stl_e_acf10 <dbl>
+```
+
+We can then use these features in plots to identify what type of series are heavily trended and what are most seasonal.
+
+```
+tourism |>
+  features(Trips, feat_stl) |>
+  ggplot(aes(x = trend_strength, y = seasonal_strength_year,
+             col = Purpose)) +
+  geom_point() +
+  facet_wrap(vars(State))
+```
+
+![Seasonal strength vs trend strength for all tourism series.](https://otexts.com/fpp3/fpp_files/figure-html/featuresplot-1.png)
+
+Figure 4.1: Seasonal strength vs trend strength for all tourism series.
+
+Clearly, holiday series are most seasonal which is unsurprising. The strongest trends tend to be in Western Australia and Victoria. The most seasonal series can also be easily identified and plotted.
+
+```
+tourism |>
+  features(Trips, feat_stl) |>
+  filter(
+    seasonal_strength_year == max(seasonal_strength_year)
+  ) |>
+  left_join(tourism, by = c("State", "Region", "Purpose"), multiple = "all") |>
+  ggplot(aes(x = Quarter, y = Trips)) +
+  geom_line() +
+  facet_grid(vars(State, Region, Purpose))
+```
+
+![The most seasonal series in the Australian tourism data.](https://otexts.com/fpp3/fpp_files/figure-html/extreme-1.png)
+
+Figure 4.2: The most seasonal series in the Australian tourism data.
+
+This shows holiday trips to the most popular ski region of Australia.
+
+The `feat_stl()` function returns several more features other than those discussed above.
+
+* `seasonal_peak_year` indicates the timing of the peaks — which month or quarter contains the largest seasonal component. This tells us something about the nature of the seasonality. In the Australian tourism data, if Quarter 3 is the peak seasonal period, then people are travelling to the region in winter, whereas a peak in Quarter 1 suggests that the region is more popular in summer.
+* `seasonal_trough_year` indicates the timing of the troughs — which month or quarter contains the smallest seasonal component.
+* `spikiness` measures the prevalence of spikes in the remainder component \(R_t\) of the STL decomposition. It is the variance of the leave-one-out variances of \(R_t\).
+* `linearity` measures the linearity of the trend component of the STL decomposition. It is based on the coefficient of a linear regression applied to the trend component.
+* `curvature` measures the curvature of the trend component of the STL decomposition. It is based on the coefficient from an orthogonal quadratic regression applied to the trend component.
+* `stl_e_acf1` is the first autocorrelation coefficient of the remainder series.
+* `stl_e_acf10` is the sum of squares of the first ten autocorrelation coefficients of the remainder series.
+
+## 4.4 Other features
+
+Many more features are possible, and the `feasts` package computes only a few dozen features that have proven useful in time series analysis. It is also easy to add your own features by writing an R function that takes a univariate time series input and returns a numerical vector containing the feature values.
+
+The remaining features in the `feasts` package, not previously discussed, are listed here for reference. The details of some of them are discussed later in the book.
+
+* `coef_hurst` will calculate the Hurst coefficient of a time series which is a measure of “long memory”. A series with long memory will have significant autocorrelations for many lags.
+* `feat_spectral` will compute the (Shannon) spectral entropy of a time series, which is a measure of how easy the series is to forecast. A series which has strong trend and seasonality (and so is easy to forecast) will have entropy close to 0. A series that is very noisy (and so is difficult to forecast) will have entropy close to 1.
+* `box_pierce` gives the Box-Pierce statistic for testing if a time series is white noise, and the corresponding p-value. This test is discussed in Section [5.4](https://otexts.com/fpp3/diagnostics.html#diagnostics).
+* `ljung_box` gives the Ljung-Box statistic for testing if a time series is white noise, and the corresponding p-value. This test is discussed in Section [5.4](https://otexts.com/fpp3/diagnostics.html#diagnostics).
+* The \(k\)th partial autocorrelation measures the relationship between observations \(k\) periods apart after removing the effects of observations between them. So the first partial autocorrelation (\(k=1\)) is identical to the first autocorrelation, because there is nothing between consecutive observations to remove. Partial autocorrelations are discussed in Section [9.5](https://otexts.com/fpp3/non-seasonal-arima.html#non-seasonal-arima). The `feat_pacf` function contains several features involving partial autocorrelations including the sum of squares of the first five partial autocorrelations for the original series, the first-differenced series and the second-differenced series. For seasonal data, it also includes the partial autocorrelation at the first seasonal lag.
+* `unitroot_kpss` gives the Kwiatkowski-Phillips-Schmidt-Shin (KPSS) statistic for testing if a series is stationary, and the corresponding p-value. This test is discussed in Section [9.1](https://otexts.com/fpp3/stationarity.html#stationarity).
+* `unitroot_pp` gives the Phillips-Perron statistic for testing if a series is non-stationary, and the corresponding p-value.
+* `unitroot_ndiffs` gives the number of differences required to lead to a stationary series based on the KPSS test. This is discussed in Section [9.1](https://otexts.com/fpp3/stationarity.html#stationarity)
+* `unitroot_nsdiffs` gives the number of seasonal differences required to make a series stationary. This is discussed in Section [9.1](https://otexts.com/fpp3/stationarity.html#stationarity).
+* `var_tiled_mean` gives the variances of the “tiled means” (i.e., the means of consecutive non-overlapping blocks of observations). The default tile length is either 10 (for non-seasonal data) or the length of the seasonal period. This is sometimes called the “stability” feature.
+* `var_tiled_var` gives the variances of the “tiled variances” (i.e., the variances of consecutive non-overlapping blocks of observations). This is sometimes called the “lumpiness” feature.
+* `shift_level_max` finds the largest mean shift between two consecutive sliding windows of the time series. This is useful for finding sudden jumps or drops in a time series.
+* `shift_level_index` gives the index at which the largest mean shift occurs.
+* `shift_var_max` finds the largest variance shift between two consecutive sliding windows of the time series. This is useful for finding sudden changes in the volatility of a time series.
+* `shift_var_index` gives the index at which the largest variance shift occurs.
+* `shift_kl_max` finds the largest distributional shift (based on the Kulback-Leibler divergence) between two consecutive sliding windows of the time series. This is useful for finding sudden changes in the distribution of a time series.
+* `shift_kl_index` gives the index at which the largest KL shift occurs.
+* `n_crossing_points` computes the number of times a time series crosses the median.
+* `longest_flat_spot` computes the number of sections of the data where the series is relatively unchanging.
+* `stat_arch_lm` returns the statistic based on the Lagrange Multiplier (LM) test of Engle (1982) for autoregressive conditional heteroscedasticity (ARCH).
+* `guerrero` computes the optimal \(\lambda\) value for a Box-Cox transformation using the Guerrero method (discussed in Section [3.1](https://otexts.com/fpp3/transformations.html#transformations)).
+
+## 4.5 Exploring Australian tourism data
+
+All of the features included in the `feasts` package can be computed in one line like this.
+
+```
+tourism_features <- tourism |>
+  features(Trips, feature_set(pkgs = "feasts"))
+tourism_features
+#> # A tibble: 304 × 51
+#>    Region         State          Purpose trend_strength seasonal_strength_year
+#>    <chr>          <chr>          <chr>            <dbl>                  <dbl>
+#>  1 Adelaide       South Austral… Busine…          0.464                  0.407
+#>  2 Adelaide       South Austral… Holiday          0.554                  0.619
+#>  3 Adelaide       South Austral… Other            0.746                  0.202
+#>  4 Adelaide       South Austral… Visiti…          0.435                  0.452
+#>  5 Adelaide Hills South Austral… Busine…          0.464                  0.179
+#>  6 Adelaide Hills South Austral… Holiday          0.528                  0.296
+#>  7 Adelaide Hills South Austral… Other            0.593                  0.404
+#>  8 Adelaide Hills South Austral… Visiti…          0.488                  0.254
+#>  9 Alice Springs  Northern Terr… Busine…          0.534                  0.251
+#> 10 Alice Springs  Northern Terr… Holiday          0.381                  0.832
+#> # ℹ 294 more rows
+#> # ℹ 46 more variables: seasonal_peak_year <dbl>, seasonal_trough_year <dbl>,
+#> #   spikiness <dbl>, linearity <dbl>, curvature <dbl>, stl_e_acf1 <dbl>,
+#> #   stl_e_acf10 <dbl>, acf1 <dbl>, acf10 <dbl>, diff1_acf1 <dbl>,
+#> #   diff1_acf10 <dbl>, diff2_acf1 <dbl>, diff2_acf10 <dbl>,
+#> #   season_acf1 <dbl>, pacf5 <dbl>, diff1_pacf5 <dbl>, diff2_pacf5 <dbl>,
+#> #   season_pacf <dbl>, zero_run_mean <dbl>, nonzero_squared_cv <dbl>, …
+```
+
+Provided the `urca` and `fracdiff` packages are installed, this gives 48 features for every combination of the three key variables (`Region`, `State` and `Purpose`). We can treat this tibble like any data set and analyse it to find interesting observations or groups of observations.
+
+We’ve already seen how we can plot one feature against another (Section [4.3](https://otexts.com/fpp3/stlfeatures.html#stlfeatures)). We can also do pairwise plots of groups of features. In Figure [4.3](https://otexts.com/fpp3/exploring-australian-tourism-data.html#fig:seasonalfeatures), for example, we show all features that involve seasonality, along with the `Purpose` variable.
+
+```
+library(glue)
+tourism_features |>
+  select_at(vars(contains("season"), Purpose)) |>
+  mutate(
+    seasonal_peak_year = seasonal_peak_year +
+      4*(seasonal_peak_year==0),
+    seasonal_trough_year = seasonal_trough_year +
+      4*(seasonal_trough_year==0),
+    seasonal_peak_year = glue("Q{seasonal_peak_year}"),
+    seasonal_trough_year = glue("Q{seasonal_trough_year}"),
+  ) |>
+  GGally::ggpairs(mapping = aes(colour = Purpose))
+```
+
+![Pairwise plots of all the seasonal features for the Australian tourism data](https://otexts.com/fpp3/fpp_files/figure-html/seasonalfeatures-1.png)
+
+Figure 4.3: Pairwise plots of all the seasonal features for the Australian tourism data
+
+Here, the `Purpose` variable is mapped to colour. There is a lot of information in this figure, and we will highlight just a few things we can learn.
+
+* The three numerical measures related to seasonality (`seasonal_strength_year`, `season_acf1` and `season_pacf`) are all positively correlated.
+* The bottom left panel and the top right panel both show that the most strongly seasonal series are related to holidays (as we saw previously).
+* The bar plots in the bottom row of the `seasonal_peak_year` and `seasonal_trough_year` columns show that seasonal peaks in Business travel occur most often in Quarter 3, and least often in Quarter 1.
+
+It is difficult to explore more than a handful of variables in this way. A useful way to handle many more variables is to use a dimension reduction technique such as principal components. This gives linear combinations of variables that explain the most variation in the original data. We can compute the principal components of the tourism features as follows.
+
+```
+library(broom)
+pcs <- tourism_features |>
+  select(-State, -Region, -Purpose) |>
+  prcomp(scale = TRUE) |>
+  augment(tourism_features)
+pcs |>
+  ggplot(aes(x = .fittedPC1, y = .fittedPC2, col = Purpose)) +
+  geom_point() +
+  theme(aspect.ratio = 1)
+```
+
+![A plot of the first two principal components, calculated from the 48 features of the Australian quarterly tourism data.](https://otexts.com/fpp3/fpp_files/figure-html/pca-1.png)
+
+Figure 4.4: A plot of the first two principal components, calculated from the 48 features of the Australian quarterly tourism data.
+
+Each point on Figure [4.4](https://otexts.com/fpp3/exploring-australian-tourism-data.html#fig:pca) represents one series and its location on the plot is based on all 48 features. The first principal component (`.fittedPC1`) is the linear combination of the features which explains the most variation in the data. The second principal component (`.fittedPC2`) is the linear combination which explains the next most variation in the data, while being uncorrelated with the first principal component. For more information about principal component dimension reduction, see Izenman ([2008](#ref-izenman2008)).
+
+Figure [4.4](https://otexts.com/fpp3/exploring-australian-tourism-data.html#fig:pca) reveals a few things about the tourism data. First, the holiday series behave quite differently from the rest of the series. Almost all of the holiday series appear in the top half of the plot, while almost all of the remaining series appear in the bottom half of the plot. Clearly, the second principal component is distinguishing between holidays and other types of travel.
+
+The plot also allows us to identify anomalous time series — series which have unusual feature combinations. These appear as points that are separate from the majority of series in Figure [4.4](https://otexts.com/fpp3/exploring-australian-tourism-data.html#fig:pca). There are four that stand out, and we can identify which series they correspond to as follows.
+
+```
+outliers <- pcs |>
+  filter(.fittedPC1 > 10) |>
+  select(Region, State, Purpose, .fittedPC1, .fittedPC2)
+outliers
+#> # A tibble: 4 × 5
+#>   Region                 State             Purpose  .fittedPC1 .fittedPC2
+#>   <chr>                  <chr>             <chr>         <dbl>      <dbl>
+#> 1 Australia's North West Western Australia Business       13.4    -11.3
+#> 2 Australia's South West Western Australia Holiday        10.9      0.880
+#> 3 Melbourne              Victoria          Holiday        12.3    -10.4
+#> 4 South Coast            New South Wales   Holiday        11.9      9.42
+outliers |>
+  left_join(tourism, by = c("State", "Region", "Purpose"), multiple = "all") |>
+  mutate(Series = glue("{State}", "{Region}", "{Purpose}", .sep = "\n\n")) |>
+  ggplot(aes(x = Quarter, y = Trips)) +
+  geom_line() +
+  facet_grid(Series ~ ., scales = "free") +
+  labs(title = "Outlying time series in PC space")
+```
+
+![Four anomalous time series from the Australian tourism data.](https://otexts.com/fpp3/fpp_files/figure-html/pcaoutliers-1.png)
+
+Figure 4.5: Four anomalous time series from the Australian tourism data.
+
+We can speculate why these series are identified as unusual.
+
+* Holiday visits to the south coast of NSW is highly seasonal but has almost no trend, whereas most holiday destinations in Australia show some trend over time.
+* Melbourne is an unusual holiday destination because it has almost no seasonality, whereas most holiday destinations in Australia have highly seasonal tourism.
+* The north western corner of Western Australia is unusual because it shows an increase in business tourism in the last few years of data, but little or no seasonality.
+* The south western corner of Western Australia is unusual because it shows both an increase in holiday tourism in the last few years of data and a high level of seasonality.
+
+### Bibliography
+
+Izenman, A. J. (2008). *Modern multivariate statistical techniques: Regression, classification and manifold learning*. Springer.
+
+## 4.6 Exercises
+
+1. Write a function to compute the mean and standard deviation of a time series, and apply it to the `PBS` data. Plot the series with the highest mean, and the series with the lowest standard deviation.
+2. Use `GGally::ggpairs()` to look at the relationships between the STL-based features for the holiday series in the `tourism` data. Change `seasonal_peak_year` and `seasonal_trough_year` to factors, as shown in Figure [4.3](https://otexts.com/fpp3/exploring-australian-tourism-data.html#fig:seasonalfeatures). Which is the peak quarter for holidays in each state?
+3. Use a feature-based approach to look for outlying series in the `PBS` data. What is unusual about the series you identify as “outliers”.
+
+## 4.7 Further reading
+
+* The idea of using STL for features originated with Wang et al. ([2006](#ref-WangSH06)).
+* The features provided by the `feasts` package were motivated by their use in Hyndman et al. ([2015](#ref-cikm2015)) and Kang et al. ([2017](#ref-m3pca)).
+* The exploration of a set of time series using principal components on a large collection of features was proposed by Kang et al. ([2017](#ref-m3pca)).
+
+### Bibliography
+
+Hyndman, R. J., Wang, E., & Laptev, N. (2015). Large-scale unusual time series detection. *Proceedings of the IEEE International Conference on Data Mining*, 1616–1619.
+
+Kang, Y., Hyndman, R. J., & Smith-Miles, K. (2017). Visualising forecasting algorithm performance using time series instance spaces. *International Journal of Forecasting*, *33*(2), 345–358.
+
+Wang, X., Smith, K. A., & Hyndman, R. J. (2006). Characteristic-based clustering for time series data. *Data Mining and Knowledge Discovery*, *13*(3), 335–364.

docs/evidence/fpp3/06-judgmental-forecasts.md

@@ -0,0 +1,346 @@
+Source: https://otexts.com/fpp3/judgmental.html (chapter judgmental, 9 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 06-judgmental-forecasts
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Chapter 6 Judgmental forecasts
+
+Forecasting using judgment is common in practice. In many cases, judgmental forecasting is the only option, such as when there is a complete lack of historical data, or when a new product is being launched, or when a new competitor enters the market, or during completely new and unique market conditions. For example, in December 2012, the Australian government was the first in the world to pass legislation that banned the use of company logos on cigarette packets, and required all cigarette packets to be a dark green colour. Judgment must be applied in order to forecast the effect of such a policy, as there are no historical precedents.
+
+There are also situations where the data are incomplete, or only become available after some delay. For example, central banks include judgment when forecasting the current level of economic activity, a procedure known as nowcasting, as GDP is only available on a quarterly basis.
+
+Research in this area[6](#fn6) has shown that the accuracy of judgmental forecasting improves when the forecaster has (i) important domain knowledge, and (ii) more timely, up-to-date information. A judgmental approach can be quick to adjust to such changes, information or events.
+
+Over the years, the acceptance of judgmental forecasting as a science has increased, as has the recognition of its need. More importantly, the quality of judgmental forecasts has also improved, as a direct result of recognising that improvements in judgmental forecasting can be achieved by implementing well-structured and systematic approaches. It is important to recognise that judgmental forecasting is subjective and comes with limitations. However, implementing systematic and well-structured approaches can confine these limitations and markedly improve forecast accuracy.
+
+There are three general settings in which judgmental forecasting is used: (i) there are no available data, so that statistical methods are not applicable and judgmental forecasting is the only feasible approach; (ii) data are available, statistical forecasts are generated, and these are then adjusted using judgment; and (iii) data are available and statistical and judgmental forecasts are generated independently and then combined. We should clarify that when data are available, applying statistical methods (such as those discussed in other chapters of this book), is preferable and should always be used as a starting point. Statistical forecasts are generally superior to generating forecasts using only judgment. For the majority of the chapter, we focus on the first setting where no data are available, and in the last section we discuss the judgmental adjustment of statistical forecasts. We discuss combining forecasts in Section [13.4](https://otexts.com/fpp3/combinations.html#combinations).
+
+### Bibliography
+
+Lawrence, M., Goodwin, P., O’Connor, M., & Önkal, D. (2006). Judgmental forecasting: A review of progress over the last 25 years. *International Journal of Forecasting*, *22*(3), 493–518.
+
+---
+
+6. Lawrence et al. ([2006](#ref-Lawrence2006))[↩︎](https://otexts.com/fpp3/judgmental.html#fnref6)
+
+## 6.1 Beware of limitations
+
+Judgmental forecasts are subjective, and therefore do not come free of bias or limitations.
+
+Judgmental forecasts can be inconsistent. Unlike statistical forecasts, which can be generated by the same mathematical formulas every time, judgmental forecasts depend heavily on human cognition, and are vulnerable to its limitations. For example, a limited memory may render recent events more important than they actually are and may ignore momentous events from the more distant past; or a limited attention span may result in important information being missed; or a misunderstanding of causal relationships may lead to erroneous inferences. Furthermore, human judgment can vary due to the effect of psychological factors. One can imagine a manager who is in a positive frame of mind one day, generating forecasts that may tend to be somewhat optimistic, and in a negative frame of mind another day, generating somewhat less optimistic forecasts.
+
+Judgment can be clouded by personal or political agendas, where targets and forecasts (as defined in Chapter [1](https://otexts.com/fpp3/intro.html#intro)) are not segregated. For example, if a sales manager knows that the forecasts she generates will be used to set sales expectations (targets), she may tend to set these low in order to show a good performance (i.e., exceed the expected targets). Even in cases where targets and forecasts are well segregated, judgment may be plagued by optimism or wishful thinking. For example, it would be highly unlikely that a team working towards launching a new product would forecast its failure. As we will discuss later, this optimism can be accentuated in a group meeting setting. “Beware of the enthusiasm of your marketing and sales colleagues”[7](#fn7).
+
+Another undesirable property which is commonly seen in judgmental forecasting is the effect of anchoring. In this case, the subsequent forecasts tend to converge or be close to an initial familiar reference point. For example, it is common to take the last observed value as a reference point. The forecaster is influenced unduly by prior information, and therefore gives this more weight in the forecasting process. Anchoring may lead to conservatism and undervaluing new and more current information, and thereby create a systematic bias.
+
+### Bibliography
+
+Fildes, R., & Goodwin, P. (2007b). Good and bad judgment in forecasting: Lessons from four companies. *Foresight: The International Journal of Applied Forecasting*, *8*, 5–10.
+
+---
+
+7. Fildes & Goodwin ([2007b](#ref-Fildes2007a))[↩︎](https://otexts.com/fpp3/judgmental-limitations.html#fnref7)
+
+## 6.2 Key principles
+
+Using a systematic and well structured approach in judgmental forecasting helps to reduce the adverse effects of the limitations of judgmental forecasting, some of which we listed in the previous section. Whether this approach involves one individual or many, the following principles should be followed.
+
+### Set the forecasting task clearly and concisely
+
+Care is needed when setting the forecasting challenges and expressing the forecasting tasks. It is important that everyone be clear about what the task is. All definitions should be clear and comprehensive, avoiding ambiguous and vague expressions. Also, it is important to avoid incorporating emotive terms and irrelevant information that may distract the forecaster. In the Delphi method that follows (see Section [6.3](https://otexts.com/fpp3/delphimethod.html#delphimethod)), it may sometimes be useful to conduct a preliminary round of information gathering before setting the forecasting task.
+
+### Implement a systematic approach
+
+Forecast accuracy and consistency can be improved by using a systematic approach to judgmental forecasting involving checklists of categories of information which are relevant to the forecasting task. For example, it is helpful to identify what information is important and how this information is to be weighted. When forecasting the demand for a new product, what factors should we account for and how should we account for them? Should it be the price, the quality and/or quantity of the competition, the economic environment at the time, the target population of the product? It is worthwhile to devote significant effort and resources to put together decision rules that will lead to the best possible systematic approach.
+
+### Document and justify
+
+Formalising and documenting the decision rules and assumptions implemented in the systematic approach can promote consistency, as the same rules can be implemented repeatedly. Also, requesting a forecaster to document and justify their forecasts leads to accountability, which can lead to reduced bias. Furthermore, formal documentation aids significantly in the systematic evaluation process that is suggested next.
+
+### Systematically evaluate forecasts
+
+Systematically monitoring the forecasting process can identify unforeseen irregularities. In particular, keep records of forecasts and use them to obtain feedback when the corresponding observations become available. Although you may do your best as a forecaster, the environment you operate in is dynamic. Changes occur, and you need to monitor these in order to evaluate the decision rules and assumptions. Feedback and evaluation help forecasters learn and improve their forecast accuracy.
+
+### Segregate forecasters and users
+
+Forecast accuracy may be impeded if the forecasting task is carried out by users of the forecasts, such as those responsible for implementing plans of action about which the forecast is concerned. We should clarify again here (as in Section [1.2](https://otexts.com/fpp3/planning.html#planning)), that forecasting is about predicting the future as accurately as possible, given all of the information available, including historical data and knowledge of any future events that may impact the forecasts. Forecasters and users should be clearly segregated. A classic case is that of a new product being launched. The forecast should be a reasonable estimate of the sales volume of a new product, which may differ considerably from what management expects or hopes the sales will be in order to meet company financial objectives. In this case, a forecaster may be delivering a reality check to the user.
+
+It is important that forecasters communicate forecasts to potential users thoroughly. As we will see in Section [6.7](https://otexts.com/fpp3/judgmental-adjustments.html#judgmental-adjustments), users may feel distant and disconnected from forecasts, and may not have full confidence in them. Explaining and clarifying the process and justifying the basic assumptions that led to the forecasts will provide some assurance to users.
+
+The way in which forecasts may then be used and implemented will clearly depend on managerial decision making. For example, management may decide to adjust a forecast upwards (be over-optimistic), as the forecast may be used to guide purchasing and stock keeping levels. Such a decision may be taken after a cost-benefit analysis reveals that the cost of holding excess stock is much lower than that of lost sales. This type of adjustment should be part of setting goals or planning supply, rather than part of the forecasting process. In contrast, if forecasts are used as targets, they may be set low so that they can be exceeded more easily. Again, setting targets is different from producing forecasts, and the two should not be confused.
+
+The example that follows comes from our experience in industry. It exemplifies two contrasting styles of judgmental forecasting — one that adheres to the principles we have just presented and one that does not.
+
+### Example: Pharmaceutical Benefits Scheme (PBS)
+
+The Australian government subsidises the cost of a wide range of prescription medicines as part of the PBS. Each subsidised medicine falls into one of four categories: concession copayments, concession safety net, general copayments, and general safety net. Each person with a concession card makes a concession copayment per PBS medicine ($5.80)[8](#fn8),
+until they reach a set threshold amount labelled the concession safety net ($348). For the rest of the financial year, all PBS-listed medicines are free. Each general patient makes a general copayment per PBS medicine ($35.40) until the general safety net amount is reached ($1,363.30). For the rest of the financial year, they contribute a small amount per PBS-listed medicine ($5.80). The PBS forecasting process uses 84 groups of PBS-listed medicines, and produces forecasts of the medicine volume and the total expenditure for each group and for each of the four PBS categories, a total of 672 series. This forecasting process aids in setting the government budget allocated to the PBS, which is over $7 billion per year, or approximately 1% of GDP.
+
+![Process for producing PBS forecasts.](https://otexts.com/fpp3/figs/PBSprocess.png)
+
+Figure 6.1: Process for producing PBS forecasts.
+
+Figure [6.1](https://otexts.com/fpp3/judgmental-principles.html#fig:pbsdiagram) summarises the forecasting process. Judgmental forecasts are generated for new listings of medicines and for estimating the impact of new policies. These are shown by the green items. The pink items indicate the data used which were obtained from various government departments and associated authorities. The blue items show things that are calculated from the data provided. There were judgmental adjustments to the data to take account of new listings and new policies, and there were also judgmental adjustments to the forecasts. Because of the changing size of both the concession population and the total population, forecasts are produced on a per-capita basis, and then multiplied by the forecast population to obtain forecasts of total volume and expenditure per month.
+
+One of us (Hyndman) was asked to evaluate the forecasting process a few years ago. We found that using judgment for new listings and new policy impacts gave better forecasts than using a statistical model alone. However, we also found that the forecasting accuracy and consistency could be improved through a more structured and systematic process, especially for policy impacts.
+
+*Forecasting new listings:* Companies who apply for their medicine to be added to the PBS are asked to submit detailed forecasts for various aspects of the medicine, such as projected patient numbers, market share of the new medicine, substitution effects, etc. The Pharmaceutical Benefits Advisory Committee provides guidelines describing a highly structured and systematic approach for generating these forecasts, and requires careful documentation for each step of the process. This structured process helps to reduce the likelihood and effects of deliberate self-serving biases. Two detailed evaluation rounds of the company forecasts are implemented by a sub-committee, one before the medicine is added to the PBS and one after it is added. Finally, comparisons of observations versus forecasts for some selected new listings are performed, 12 months and 24 months after the listings, and the results are sent back to the companies for comment.
+
+*Policy impact forecasts:* In contrast to the highly structured process used for new listings, there were no systematic procedures for policy impact forecasts. On many occasions, forecasts of policy impacts were calculated by a small team, and were often heavily reliant on the work of one person. The forecasts were not usually subject to a formal review process. There were no guidelines for how to construct judgmental forecasts for policy impacts, and there was often a lack of adequate documentation about how these forecasts were obtained, the assumptions underlying them, etc.
+
+Consequently, we recommended several changes:
+
+* that guidelines for forecasting new policy impacts be developed, to encourage a more systematic and structured forecasting approach;
+* that the forecast methodology be documented in each case, including all assumptions made in forming the forecasts;
+* that new policy forecasts be made by at least two people from different areas of the organisation;
+* that a review of forecasts be conducted one year after the implementation of each new policy by a review committee, especially for new policies that have a significant annual projected cost or saving. The review committee should include those involved in generating the forecasts, but also others.
+
+These recommendations reflect the principles outlined in this section.
+
+---
+
+8. These are Australian dollar amounts published by the Australian government for 2012.[↩︎](https://otexts.com/fpp3/judgmental-principles.html#fnref8)
+
+## 6.3 The Delphi method
+
+The Delphi method was invented by Olaf Helmer and Norman Dalkey of the Rand Corporation in the 1950s for the purpose of addressing a specific military problem. The method relies on the key assumption that forecasts from a group are generally more accurate than those from individuals. The aim of the Delphi method is to construct consensus forecasts from a group of experts in a structured iterative manner. A facilitator is appointed in order to implement and manage the process. The Delphi method generally involves the following stages:
+
+1. A panel of experts is assembled.
+2. Forecasting tasks/challenges are set and distributed to the experts.
+3. Experts return initial forecasts and justifications. These are compiled and summarised in order to provide feedback.
+4. Feedback is provided to the experts, who now review their forecasts in light of the feedback. This step may be iterated until a satisfactory level of consensus is reached.
+5. Final forecasts are constructed by aggregating the experts’ forecasts.
+
+Each stage of the Delphi method comes with its own challenges. In what follows, we provide some suggestions and discussions about each one of these.[9](#fn9)
+
+### Experts and anonymity
+
+The first challenge of the facilitator is to identify a group of experts who can contribute to the forecasting task. The usual suggestion is somewhere between 5 and 20 experts with diverse expertise. Experts submit forecasts and also provide detailed qualitative justifications for these.
+
+A key feature of the Delphi method is that the participating experts remain anonymous at all times. This means that the experts cannot be influenced by political and social pressures in their forecasts. Furthermore, all experts are given an equal say and all are held accountable for their forecasts. This avoids the situation where a group meeting is held and some members do not contribute, while others dominate. It also prevents members exerting undue influence based on seniority or personality. There have been suggestions that even something as simple as the seating arrangements in a group setting can influence the group dynamics. Furthermore, there is ample evidence that a group meeting setting promotes enthusiasm and influences individual judgment, leading to optimism and overconfidence.[10](#fn10)
+
+A by-product of anonymity is that the experts do not need to meet as a group in a physical location. An important advantage of this is that it increases the likelihood of gathering experts with diverse skills and expertise from varying locations. Furthermore, it makes the process cost-effective by eliminating the expense and inconvenience of travel, and it makes it flexible, as the experts only have to meet a common deadline for submitting forecasts, rather than having to set a common meeting time.
+
+### Setting the forecasting task in a Delphi
+
+In a Delphi setting, it may be useful to conduct a preliminary round of information gathering from the experts before setting the forecasting tasks. Alternatively, as experts submit their initial forecasts and justifications, valuable information which is not shared between all experts can be identified by the facilitator when compiling the feedback.
+
+### Feedback
+
+Feedback to the experts should include summary statistics of the forecasts and outlines of qualitative justifications. Numerical data summaries and graphical representations can be used to summarise the experts’ forecasts.
+
+As the feedback is controlled by the facilitator, there may be scope to direct attention and information from the experts to areas where it is most required. For example, the facilitator may direct the experts’ attention to responses that fall outside the interquartile range, and the qualitative justification for such forecasts.
+
+### Iteration
+
+The process of the experts submitting forecasts, receiving feedback, and reviewing their forecasts in light of the feedback, is repeated until a satisfactory level of consensus between the experts is reached. Satisfactory consensus does not mean complete convergence in the forecast value; it simply means that the variability of the responses has decreased to a satisfactory level. Usually two or three rounds are sufficient. Experts are more likely to drop out as the number of iterations increases, so too many rounds should be avoided.
+
+### Final forecasts
+
+The final forecasts are usually constructed by giving equal weight to all of the experts’ forecasts. However, the facilitator should keep in mind the possibility of extreme values which can distort the final forecast.
+
+### Limitations and variations
+
+Applying the Delphi method can be time consuming. In a group meeting, final forecasts can possibly be reached in hours or even minutes — something which is almost impossible to do in a Delphi setting. If it is taking a long time to reach a consensus in a Delphi setting, the panel may lose interest and cohesiveness.
+
+In a group setting, personal interactions can lead to quicker and better clarifications of qualitative justifications. A variation of the Delphi method which is often applied is the “estimate-talk-estimate” method, where the experts can interact between iterations, although the forecast submissions can still remain anonymous. A disadvantage of this variation is the possibility of the loudest person exerting undue influence.
+
+### The facilitator
+
+The role of the facilitator is of the utmost importance. The facilitator is largely responsible for the design and administration of the Delphi process. The facilitator is also responsible for providing feedback to the experts and generating the final forecasts. In this role, the facilitator needs to be experienced enough to recognise areas that may need more attention, and to direct the experts’ attention to these. Also, as there is no face-to-face interaction between the experts, the facilitator is responsible for disseminating important information. The efficiency and effectiveness of the facilitator can dramatically increase the probability of a successful Delphi method in a judgmental forecasting setting.
+
+### Bibliography
+
+Buehler, R., Messervey, D., & Griffin, D. (2005). Collaborative planning and prediction: Does group discussion affect optimistic biases in time estimation? *Organizational Behavior and Human Decision Processes*, *97*(1), 47–63.
+
+Rowe, G. (2007). A guide to Delphi. *Foresight: The International Journal of Applied Forecasting*, *8*, 11–16.
+
+Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. *International Journal of Forecasting*, *15*(4), 353–375.
+
+---
+
+9. For further reading, refer to: Rowe ([2007](#ref-Rowe2007)); Rowe & Wright ([1999](#ref-RW99))[↩︎](https://otexts.com/fpp3/delphimethod.html#fnref9)
+10. Buehler et al. ([2005](#ref-Buehler2005))[↩︎](https://otexts.com/fpp3/delphimethod.html#fnref10)
+
+## 6.4 Forecasting by analogy
+
+A useful judgmental approach which is often implemented in practice is forecasting by analogy. A common example is the pricing of a house through an appraisal process. An appraiser estimates the market value of a house by comparing it to similar properties that have sold in the area. The degree of similarity depends on the attributes considered. With house appraisals, attributes such as land size, dwelling size, numbers of bedrooms and bathrooms, and garage space are usually considered.
+
+Even thinking and discussing analogous products or situations can generate useful (and sometimes crucial) information. We illustrate this point with the following example.[11](#fn11)
+
+### Example: Designing a high school curriculum
+
+A small group of academics and teachers were assigned the task of developing a curriculum for teaching judgment and decision making under uncertainty for high schools in Israel. Each group member was asked to forecast how long it would take for the curriculum to be completed. Responses ranged between 18 and 30 months. One of the group members who was an expert in curriculum design was asked to consider analogous curricula developments around the world. He concluded that 40% of analogous groups he considered never completed the task. The rest took between 7 to 10 years. The Israel project was completed in 8 years.
+
+Obviously, forecasting by analogy comes with challenges. We should aspire to base forecasts on multiple analogies rather than a single analogy, which may create biases. However, these may be challenging to identify. Similarly, we should aspire to consider multiple attributes. Identifying or even comparing these may not always be straightforward. As always, we suggest performing these comparisons and the forecasting process using a systematic approach. Developing a detailed scoring mechanism to rank attributes and record the process of ranking will always be useful.
+
+### A structured analogy
+
+Alternatively, a structured approach comprising a panel of experts can be implemented, as was proposed by Green & Armstrong ([2007](#ref-Green2007)). The concept is similar to that of a Delphi; however, the forecasting task is completed by considering analogies. First, a facilitator is appointed. Then the structured approach involves the following steps.
+
+1. A panel of experts who are likely to have experience with analogous situations is assembled.
+2. Tasks/challenges are set and distributed to the experts.
+3. Experts identify and describe as many analogies as they can, and generate forecasts based on each analogy.
+4. Experts list similarities and differences of each analogy to the target situation, then rate the similarity of each analogy to the target situation on a scale.
+5. Forecasts are derived by the facilitator using a set rule. This can be a weighted average, where the weights can be guided by the ranking scores of each analogy by the experts.
+
+As with the Delphi approach, anonymity of the experts may be an advantage in not suppressing creativity, but could hinder collaboration. Green and Armstrong found no gain in collaboration between the experts in their results. A key finding was that experts with multiple analogies (more than two), and who had direct experience with the analogies, generated the most accurate forecasts.
+
+### Bibliography
+
+Green, K. C., & Armstrong, J. S. (2007). Structured analogies for forecasting. *International Journal of Forecasting*, *23*(3), 365–376.
+
+Kahneman, D., & Lovallo, D. (1993). Timid choices and bold forecasts: A cognitive perspective on risk taking. *Management Science*, *39*(1), 17–31.
+
+---
+
+11. This example is extracted from Kahneman & Lovallo ([1993](#ref-Kahneman1993))[↩︎](https://otexts.com/fpp3/analogies.html#fnref11)
+
+## 6.5 Scenario forecasting
+
+A fundamentally different approach to judgmental forecasting is scenario-based forecasting. The aim of this approach is to generate forecasts based on plausible scenarios. In contrast to the two previous approaches (Delphi and forecasting by analogy) where the resulting forecast is intended to be a likely outcome, each scenario-based forecast may have a low probability of occurrence. The scenarios are generated by considering all possible factors or drivers, their relative impacts, the interactions between them, and the targets to be forecast.
+
+Building forecasts based on scenarios allows a wide range of possible forecasts to be generated and some extremes to be identified. For example it is usual for “best”, “middle” and “worst” case scenarios to be presented, although many other scenarios will be generated. Thinking about and documenting these contrasting extremes can lead to early contingency planning.
+
+With scenario forecasting, decision makers often participate in the generation of scenarios. While this may lead to some biases, it can ease the communication of the scenario-based forecasts, and lead to a better understanding of the results.
+
+## 6.6 New product forecasting
+
+The definition of a new product can vary. It may be an entirely new product which has been launched, a variation of an existing product (“new and improved”), a change in the pricing scheme of an existing product, or even an existing product entering a new market.
+
+Judgmental forecasting is usually the only available method for new product forecasting, as historical data are unavailable. The approaches we have already outlined (Delphi, forecasting by analogy and scenario forecasting) are all applicable when forecasting the demand for a new product.
+
+Other methods which are more specific to the situation are also available. We briefly describe three such methods which are commonly applied in practice. These methods are less structured than those already discussed, and are likely to lead to more biased forecasts as a result.
+
+### Sales force composite
+
+In this approach, forecasts for each outlet/branch/store of a company are generated by salespeople, and are then aggregated. This usually involves sales managers forecasting the demand for the outlet they manage. Salespeople are usually closest to the interaction between customers and products, and often develop an intuition about customer purchasing intentions. They bring this valuable experience and expertise to the forecast.
+
+However, having salespeople generate forecasts violates the key principle of segregating forecasters and users, which can create biases in many directions. It is common for the performance of a salesperson to be evaluated against the sales forecasts or expectations set beforehand. In this case, the salesperson acting as a forecaster may introduce some self-serving bias by generating low forecasts. On the other hand, one can imagine an enthusiastic salesperson, full of optimism, generating high forecasts.
+
+Moreover a successful salesperson is not necessarily a successful nor well-informed forecaster. A large proportion of salespeople will have no or limited formal training in forecasting. Finally, salespeople will feel customer displeasure at first hand if, for example, the product runs out or is not introduced in their store. Such interactions will cloud their judgment.
+
+### Executive opinion
+
+In contrast to the sales force composite, this approach involves staff at the top of the managerial structure generating aggregate forecasts. Such forecasts are usually generated in a group meeting, where executives contribute information from their own area of the company. Having executives from different functional areas of the company promotes great skill and knowledge diversity in the group.
+
+This process carries all of the advantages and disadvantages of a group meeting setting which we discussed earlier. In this setting, it is important to justify and document the forecasting process. That is, executives need to be held accountable in order to reduce the biases generated by the group meeting setting. There may also be scope to apply variations to a Delphi approach in this setting; for example, the estimate-talk-estimate process described earlier.
+
+### Customer intentions
+
+Customer intentions can be used to forecast the demand for a new product or for a variation on an existing product. Questionnaires are filled in by customers on their intentions to buy the product. A structured questionnaire is used, asking customers to rate the likelihood of them purchasing the product on a scale; for example, highly likely, likely, possible, unlikely, highly unlikely.
+
+Survey design challenges, such as collecting a representative sample, applying a time- and cost-effective method, and dealing with non-responses, need to be addressed.[12](#fn12)
+
+Furthermore, in this survey setting we must keep in mind the relationship between purchase intention and purchase behaviour. Customers do not always do what they say they will. Many studies have found a positive correlation between purchase intentions and purchase behaviour; however, the strength of these correlations varies substantially. The factors driving this variation include the timings of data collection and product launch, the definition of “new” for the product, and the type of industry. Behavioural theory tells us that intentions predict behaviour if the intentions are measured just before the behaviour.[13](#fn13) The time between intention and behaviour will vary depending on whether it is a completely new product or a variation on an existing product. Also, the correlation between intention and behaviour is found to be stronger for variations on existing and familiar products than for completely new products.
+
+Whichever method of new product forecasting is used, it is important to thoroughly document the forecasts made, and the reasoning behind them, in order to be able to evaluate them when data become available.
+
+### Bibliography
+
+Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). *Survey methodology* (2nd ed). John Wiley & Sons.
+
+Randall, D. M., & Wolff, J. A. (1994). The time interval in the intention-behaviour relationship: Meta-analysis. *British Journal of Social Psychology*, *33*(4), 405–418.
+
+---
+
+12. Groves et al. ([2009](#ref-Groves2009))[↩︎](https://otexts.com/fpp3/new-products.html#fnref12)
+13. Randall & Wolff ([1994](#ref-RW94))[↩︎](https://otexts.com/fpp3/new-products.html#fnref13)
+
+## 6.7 Judgmental adjustments
+
+In this final section, we consider the situation where historical data are available and are used to generate statistical forecasts. It is common for practitioners to then apply judgmental adjustments to these forecasts. These adjustments can potentially provide all of the advantages of judgmental forecasting which have been discussed earlier in this chapter. For example, they provide an avenue for incorporating factors that may not be accounted for in the statistical model, such as promotions, large sporting events, holidays, or recent events that are not yet reflected in the data. However, these advantages come to fruition only when the right conditions are present. Judgmental adjustments, like judgmental forecasts, come with biases and limitations, and we must implement methodical strategies in order to minimise them.
+
+### Use adjustments sparingly
+
+Practitioners adjust much more often than they should, and many times for the wrong reasons. By adjusting statistical forecasts, users of forecasts create a feeling of ownership and credibility. Users often do not understand or appreciate the mechanisms that generate the statistical forecasts (as they will usually have no training in this area). By implementing judgmental adjustments, users feel that they have contributed to and completed the forecasts, and they can now relate their own intuition and interpretations to these. The forecasts have become their own.
+
+Judgmental adjustments should not aim to correct for a systematic pattern in the data that is thought to have been missed by the statistical model. This has been proven to be ineffective, as forecasters tend to read non-existent patterns in noisy series. Statistical models are much better at taking account of data patterns, and judgmental adjustments only hinder accuracy.
+
+Judgmental adjustments are most effective when there is significant additional information at hand or strong evidence of the need for an adjustment. We should only adjust when we have important extra information which is not incorporated in the statistical model. Hence, adjustments seem to be most accurate when they are large in size. Small adjustments (especially in the positive direction promoting the illusion of optimism) have been found to hinder accuracy, and should be avoided.
+
+### Apply a structured approach
+
+Using a structured and systematic approach will improve the accuracy of judgmental adjustments. Following the key principles outlined in Section [6.2](https://otexts.com/fpp3/judgmental-principles.html#judgmental-principles) is vital. In particular, having to document and justify adjustments will make it more challenging to override the statistical forecasts, and will guard against unnecessary adjustments.
+
+It is common for adjustments to be implemented by a panel (see the example that follows). Using a Delphi setting carries great advantages. However, if adjustments are implemented in a group meeting, it is wise to consider the forecasts of key markets or products first, as panel members will get tired during this process. Fewer adjustments tend to be made as the meeting goes on through the day.
+
+### Example: Tourism Forecasting Committee (TFC)
+
+Tourism Australia publishes forecasts for all aspects of Australian tourism twice a year. The published forecasts are generated by the TFC, an independent body which comprises experts from various government and industry sectors; for example, the Australian Commonwealth Treasury, airline companies, consulting firms, banking sector companies, and tourism bodies.
+
+The forecasting methodology applied is an iterative process. First, model-based statistical forecasts are generated by the forecasting unit within Tourism Australia, then judgmental adjustments are made to these in two rounds. In the first round, the TFC Technical Committee[14](#fn14) (comprising senior researchers, economists and independent advisers) adjusts the model-based forecasts. In the second and final round, the TFC (comprising industry and government experts) makes final adjustments. In both rounds, adjustments are made by consensus.
+
+![Long run annual forecasts for domestic visitor nights for Australia. We study regression models in Chapter 7, and ETS (ExponenTial Smoothing) models in Chapter 8.](https://otexts.com/fpp3/fpp_files/figure-html/tfc-1.png)
+
+Figure 6.2: Long run annual forecasts for domestic visitor nights for Australia. We study regression models in Chapter [7](https://otexts.com/fpp3/regression.html#regression), and ETS (ExponenTial Smoothing) models in Chapter [8](https://otexts.com/fpp3/expsmooth.html#expsmooth).
+
+In 2008, we[15](#fn15) analysed forecasts for Australian domestic tourism. We concluded that the published TFC forecasts were optimistic, especially for the long-run, and we proposed alternative model-based forecasts. We now have access to observed data up to and including 2011. In Figure [6.2](https://otexts.com/fpp3/judgmental-adjustments.html#fig:tfc), we plot the published forecasts against the actual data. We can see that the published TFC forecasts have continued to be optimistic.
+
+What can we learn from this example? Although the TFC clearly states in its methodology that it produces ‘forecasts’ rather than ‘targets’, could this be a case where these have been confused? Are the forecasters and users sufficiently well-segregated in this process? Could the iterative process itself be improved? Could the adjustment process in the meetings be improved? Could it be that the group meetings have promoted optimism? Could it be that domestic tourism should have been considered earlier in the day?
+
+### Bibliography
+
+Athanasopoulos, G., & Hyndman, R. J. (2008). Modelling and forecasting Australian domestic tourism. *Tourism Management*, *29*(1), 19–31.
+
+---
+
+14. Athanasopoulos was an observer on this technical committee for a few years.[↩︎](https://otexts.com/fpp3/judgmental-adjustments.html#fnref14)
+15. Athanasopoulos & Hyndman ([2008](#ref-austourism))[↩︎](https://otexts.com/fpp3/judgmental-adjustments.html#fnref15)
+
+## 6.8 Further reading
+
+Many forecasting textbooks ignore judgmental forecasting altogether. Here are three which do cover it in some detail.
+
+* Chapter 11 of Ord et al. ([2017](#ref-Ord2017)) provides an excellent review of some of the same topics as this chapter, but also includes using judgment to assessing forecast uncertainty, and forecasting using prediction markets.
+* Goodwin & Wright ([2009](#ref-GW04)) is a book-length treatment of the use of judgment in decision making by two of the leading researchers in the field.
+* Kahn ([2006](#ref-Kahn2006)) covers techniques for new product forecasting, where judgmental methods play an important role.
+
+There have been some helpful survey papers on judgmental forecasting published in the last 20 years. We have found these three particularly helpful.
+
+* Fildes & Goodwin ([2007b](#ref-Fildes2007a))
+* Fildes & Goodwin ([2007a](#ref-Fildes2007))
+* Harvey ([2001](#ref-Harvey2001))
+
+Some helpful papers on individual judgmental forecasting methods are listed in the table below.
+
+| **Forecasting Method** | **Recommended papers** |
+| --- | --- |
+| Delphi | Rowe & Wright ([1999](#ref-RW99)) |
+|  | Rowe ([2007](#ref-Rowe2007)) |
+| Adjustments | Sanders et al. ([2005](#ref-Sanders2005)) |
+|  | Eroglu & Croxton ([2010](#ref-Eroglu2010)) |
+|  | Franses & Legerstee ([2013](#ref-Franses2013)) |
+| Analogy | Green & Armstrong ([2007](#ref-Green2007)) |
+| Scenarios | Önkal et al. ([2013](#ref-Onkal2012)) |
+| Customer intentions | Morwitz et al. ([2007](#ref-Morwitz2007)) |
+
+### Bibliography
+
+Eroglu, C., & Croxton, K. L. (2010). Biases in judgmental adjustments of statistical forecasts: The role of individual differences. *International Journal of Forecasting*, *26*(1), 116–133.
+
+Fildes, R., & Goodwin, P. (2007a). Against your better judgment? How organizations can improve their use of management judgment in forecasting. *Interfaces*, *37*(6), 570–576.
+
+Fildes, R., & Goodwin, P. (2007b). Good and bad judgment in forecasting: Lessons from four companies. *Foresight: The International Journal of Applied Forecasting*, *8*, 5–10.
+
+Franses, P. H., & Legerstee, R. (2013). Do statistical forecasting models for SKU-level data benefit from including past expert knowledge? *International Journal of Forecasting*, *29*(1), 80–87.
+
+Goodwin, P., & Wright, G. (2009). *Decision analysis for management judgment* (4th ed). John Wiley & Sons.
+
+Green, K. C., & Armstrong, J. S. (2007). Structured analogies for forecasting. *International Journal of Forecasting*, *23*(3), 365–376.
+
+Harvey, N. (2001). Improving judgment in forecasting. In J. S. Armstrong (Ed.), *Principles of forecasting: A handbook for researchers and practitioners* (pp. 59–80). Kluwer Academic Publishers.
+
+Kahn, K. B. (2006). *New product forecasting: An applied approach*. M.E. Sharp.
+
+Morwitz, V. G., Steckel, J. H., & Gupta, A. (2007). When do purchase intentions predict sales? *International Journal of Forecasting*, *23*(3), 347–364.
+
+Önkal, D., Sayım, K. Z., & Gönül, M. S. (2013). Scenarios as channels of forecast advice. *Technological Forecasting and Social Change*, *80*(4), 772–788.
+
+Ord, J. K., Fildes, R., & Kourentzes, N. (2017). *Principles of business forecasting* (2nd ed.). Wessex Press Publishing Co.
+
+Rowe, G. (2007). A guide to Delphi. *Foresight: The International Journal of Applied Forecasting*, *8*, 11–16.
+
+Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. *International Journal of Forecasting*, *15*(4), 353–375.
+
+Sanders, N., Goodwin, P., Önkal, D., Gönül, M. S., Harvey, N., Lee, A., & Kjolso, L. (2005). When and how should statistical forecasts be judgmentally adjusted? *Foresight: The International Journal of Applied Forecasting*, *1*(1), 5–23.

docs/evidence/fpp3/08-exponential-smoothing.md

@@ -0,0 +1,947 @@
+Source: https://otexts.com/fpp3/expsmooth.html (chapter expsmooth, 10 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 08-exponential-smoothing
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Chapter 8 Exponential smoothing
+
+Exponential smoothing was proposed in the late 1950s ([Brown, 1959](#ref-Brown59); [Holt, 1957](#ref-Holt57); [Winters, 1960](#ref-Winters60)), and has motivated some of the most successful forecasting methods. Forecasts produced using exponential smoothing methods are weighted averages of past observations, with the weights decaying exponentially as the observations get older. In other words, the more recent the observation the higher the associated weight. This framework generates reliable forecasts quickly and for a wide range of time series, which is a great advantage and of major importance to applications in industry.
+
+This chapter is divided into two parts. In the first part (Sections [8.1](https://otexts.com/fpp3/ses.html#ses)–[8.4](https://otexts.com/fpp3/taxonomy.html#taxonomy)) we present the mechanics of the most important exponential smoothing methods, and their application in forecasting time series with various characteristics. This helps us develop an intuition to how these methods work. In this setting, selecting and using a forecasting method may appear to be somewhat ad hoc. The selection of the method is generally based on recognising key components of the time series (trend and seasonal) and the way in which these enter the smoothing method (e.g., in an additive, damped or multiplicative manner).
+
+In the second part of the chapter (Sections [8.5](https://otexts.com/fpp3/ets.html#ets)–[8.7](https://otexts.com/fpp3/ets-forecasting.html#ets-forecasting)) we present the statistical models that underlie exponential smoothing methods. These models generate identical point forecasts to the methods discussed in the first part of the chapter, but also generate prediction intervals. Furthermore, this statistical framework allows for genuine model selection between competing models.
+
+### Bibliography
+
+Brown, R. G. (1959). *Statistical forecasting for inventory control*. McGraw/Hill.
+
+Holt, C. C. (1957). *Forecasting seasonals and trends by exponentially weighted averages* (ONR Memorandum No. 52). Carnegie Institute of Technology, Pittsburgh USA. Reprinted in the *International Journal of Forecasting*, 2004.
+
+Winters, P. R. (1960). Forecasting sales by exponentially weighted moving averages. *Management Science*, *6*(3), 324–342.
+
+## 8.1 Simple exponential smoothing
+
+The simplest of the exponentially smoothing methods is naturally called **simple exponential smoothing** (SES)[16](#fn16). This method is suitable for forecasting data with no clear trend or seasonal pattern. For example, the data in Figure [8.1](https://otexts.com/fpp3/ses.html#fig:7-oil) do not display any clear trending behaviour or any seasonality. (There is a decline in the last few years, which might suggest a trend. We will consider whether a trended method would be better for this series later in this chapter.) We have already considered the naïve and the average as possible methods for forecasting such data (Section [5.2](https://otexts.com/fpp3/simple-methods.html#simple-methods)).
+
+```
+algeria_economy <- global_economy |>
+  filter(Country == "Algeria")
+algeria_economy |>
+  autoplot(Exports) +
+  labs(y = "% of GDP", title = "Exports: Algeria")
+```
+
+![Exports of goods and services from Algeria from 1960 to 2017.](https://otexts.com/fpp3/fpp_files/figure-html/7-oil-1.png)
+
+Figure 8.1: Exports of goods and services from Algeria from 1960 to 2017.
+
+Using the naïve method, all forecasts for the future are equal to the last observed value of the series,
+\[
+\hat{y}_{T+h|T} = y_{T},
+\]
+for \(h=1,2,\dots\). Hence, the naïve method assumes that the most recent observation is the only important one, and all previous observations provide no information for the future. This can be thought of as a weighted average where all of the weight is given to the last observation.
+
+Using the average method, all future forecasts are equal to a simple average of the observed data,
+\[
+\hat{y}_{T+h|T} = \frac1T \sum_{t=1}^T y_t,
+\]
+for \(h=1,2,\dots\). Hence, the average method assumes that all observations are of equal importance, and gives them equal weights when generating forecasts.
+
+We often want something between these two extremes. For example, it may be sensible to attach larger weights to more recent observations than to observations from the distant past. This is exactly the concept behind simple exponential smoothing. Forecasts are calculated using weighted averages, where the weights decrease exponentially as observations come from further in the past — the smallest weights are associated with the oldest observations:
+\[\begin{equation}
+\hat{y}_{T+1|T} = \alpha y_T + \alpha(1-\alpha) y_{T-1} + \alpha(1-\alpha)^2 y_{T-2}+ \cdots, \tag{8.1}
+\end{equation}\]
+where \(0 \le \alpha \le 1\) is the smoothing parameter. The one-step-ahead forecast for time \(T+1\) is a weighted average of all of the observations in the series \(y_1,\dots,y_T\). The rate at which the weights decrease is controlled by the parameter \(\alpha\).
+
+The table below shows the weights attached to observations for four different values of \(\alpha\) when forecasting using simple exponential smoothing. Note that the sum of the weights even for a small value of \(\alpha\) will be approximately one for any reasonable sample size.
+
+|  | \(\alpha=0.2\) | \(\alpha=0.4\) | \(\alpha=0.6\) | \(\alpha=0.8\) |
+| --- | --- | --- | --- | --- |
+| \(y_{T}\) | 0.2000 | 0.4000 | 0.6000 | 0.8000 |
+| \(y_{T-1}\) | 0.1600 | 0.2400 | 0.2400 | 0.1600 |
+| \(y_{T-2}\) | 0.1280 | 0.1440 | 0.0960 | 0.0320 |
+| \(y_{T-3}\) | 0.1024 | 0.0864 | 0.0384 | 0.0064 |
+| \(y_{T-4}\) | 0.0819 | 0.0518 | 0.0154 | 0.0013 |
+| \(y_{T-5}\) | 0.0655 | 0.0311 | 0.0061 | 0.0003 |
+
+For any \(\alpha\) between 0 and 1, the weights attached to the observations decrease exponentially as we go back in time, hence the name “exponential smoothing”. If \(\alpha\) is small (i.e., close to 0), more weight is given to observations from the more distant past. If \(\alpha\) is large (i.e., close to 1), more weight is given to the more recent observations. For the extreme case where \(\alpha=1\), \(\hat{y}_{T+1|T}=y_T\), so the forecasts are equal to the naïve forecasts.
+
+We present two equivalent forms of simple exponential smoothing, each of which leads to the forecast Equation [(8.1)](https://otexts.com/fpp3/ses.html#eq:7-ses).
+
+### Weighted average form
+
+The forecast at time \(T+1\) is equal to a weighted average between the most recent observation \(y_T\) and the previous forecast \(\hat{y}_{T|T-1}\):
+\[
+\hat{y}_{T+1|T} = \alpha y_T + (1-\alpha) \hat{y}_{T|T-1},
+\]
+where \(0 \le \alpha \le 1\) is the smoothing parameter.
+Similarly, we can write the fitted values as
+\[
+\hat{y}_{t+1|t} = \alpha y_t + (1-\alpha) \hat{y}_{t|t-1},
+\]
+for \(t=1,\dots,T\). (Recall that fitted values are simply one-step forecasts of the training data.)
+
+The process has to start somewhere, so we let the first fitted value at time 1 be denoted by \(\ell_0\) (which we will have to estimate). Then
+\[\begin{align\*}
+\hat{y}_{2|1} &= \alpha y_1 + (1-\alpha) \ell_0\\
+\hat{y}_{3|2} &= \alpha y_2 + (1-\alpha) \hat{y}_{2|1}\\
+\hat{y}_{4|3} &= \alpha y_3 + (1-\alpha) \hat{y}_{3|2}\\
+\vdots\\
+\hat{y}_{T|T-1} &= \alpha y_{T-1} + (1-\alpha) \hat{y}_{T-1|T-2}\\
+\hat{y}_{T+1|T} &= \alpha y_T + (1-\alpha) \hat{y}_{T|T-1}.
+\end{align\*}\]
+Substituting each equation into the following equation, we obtain
+\[\begin{align\*}
+\hat{y}_{3|2} & = \alpha y_2 + (1-\alpha) \left[\alpha y_1 + (1-\alpha) \ell_0\right] \\
+& = \alpha y_2 + \alpha(1-\alpha) y_1 + (1-\alpha)^2 \ell_0 \\
+\hat{y}_{4|3} & = \alpha y_3 + (1-\alpha) [\alpha y_2 + \alpha(1-\alpha) y_1 + (1-\alpha)^2 \ell_0]\\
+& = \alpha y_3 + \alpha(1-\alpha) y_2 + \alpha(1-\alpha)^2 y_1 + (1-\alpha)^3 \ell_0 \\
+& ~~\vdots \\
+\hat{y}_{T+1|T} & = \sum_{j=0}^{T-1} \alpha(1-\alpha)^j y_{T-j} + (1-\alpha)^T \ell_{0}.
+\end{align\*}\]
+The last term becomes tiny for large \(T\). So, the weighted average form leads to the same forecast Equation [(8.1)](https://otexts.com/fpp3/ses.html#eq:7-ses).
+
+### Component form
+
+An alternative representation is the component form. For simple exponential smoothing, the only component included is the level, \(\ell_t\). (Other methods which are considered later in this chapter may also include a trend \(b_t\) and a seasonal component \(s_t\).) Component form representations of exponential smoothing methods comprise a forecast equation and a smoothing equation for each of the components included in the method. The component form of simple exponential smoothing is given by:
+\[\begin{align\*}
+\text{Forecast equation} && \hat{y}_{t+h|t} & = \ell_{t}\\
+\text{Smoothing equation} && \ell_{t} & = \alpha y_{t} + (1 - \alpha)\ell_{t-1},
+\end{align\*}\]
+where \(\ell_{t}\) is the level (or the smoothed value) of the series at time \(t\). Setting \(h=1\) gives the fitted values, while setting \(t=T\) gives the true forecasts beyond the training data.
+
+The forecast equation shows that the forecast value at time \(t+1\) is the estimated level at time \(t\). The smoothing equation for the level (usually referred to as the level equation) gives the estimated level of the series at each period \(t\).
+
+If we replace \(\ell_t\) with \(\hat{y}_{t+1|t}\) and \(\ell_{t-1}\) with \(\hat{y}_{t|t-1}\) in the smoothing equation, we will recover the weighted average form of simple exponential smoothing.
+
+The component form of simple exponential smoothing is not particularly useful on its own, but it will be the easiest form to use when we start adding other components.
+
+### Flat forecasts
+
+Simple exponential smoothing has a “flat” forecast function:
+\[
+\hat{y}_{T+h|T} = \hat{y}_{T+1|T}=\ell_T, \qquad h=2,3,\dots.
+\]
+That is, all forecasts take the same value, equal to the last level component. Remember that these forecasts will only be suitable if the time series has no trend or seasonal component.
+
+### Optimisation
+
+The application of every exponential smoothing method requires the smoothing parameters and the initial values to be chosen. In particular, for simple exponential smoothing, we need to select the values of \(\alpha\) and \(\ell_0\). All forecasts can be computed from the data once we know those values. For the methods that follow there is usually more than one smoothing parameter and more than one initial component to be chosen.
+
+In some cases, the smoothing parameters may be chosen in a subjective manner — the forecaster specifies the value of the smoothing parameters based on previous experience. However, a more reliable and objective way to obtain values for the unknown parameters is to estimate them from the observed data.
+
+In Section [7.2](https://otexts.com/fpp3/least-squares.html#least-squares), we estimated the coefficients of a regression model by minimising the sum of the squared residuals (usually known as SSE or “sum of squared errors”). Similarly, the unknown parameters and the initial values for any exponential smoothing method can be estimated by minimising the SSE. The residuals are specified as \(e_t=y_t - \hat{y}_{t|t-1}\) for \(t=1,\dots,T\). Hence, we find the values of the unknown parameters and the initial values that minimise
+\[\begin{equation}
+\text{SSE}=\sum_{t=1}^T(y_t - \hat{y}_{t|t-1})^2=\sum_{t=1}^Te_t^2. \tag{8.2}
+\end{equation}\]
+
+Unlike the regression case (where we have formulas which return the values of the regression coefficients that minimise the SSE), this involves a non-linear minimisation problem, and we need to use an optimisation tool to solve it.
+
+### Example: Algerian exports
+
+In this example, simple exponential smoothing is applied to forecast exports of goods and services from Algeria.
+
+```
+# Estimate parameters
+fit <- algeria_economy |>
+  model(ETS(Exports ~ error("A") + trend("N") + season("N")))
+fc <- fit |>
+  forecast(h = 5)
+```
+
+This gives parameter estimates \(\hat\alpha=0.84\) and \(\hat\ell_0=39.5\), obtained by minimising SSE over periods \(t=1,2,\dots,58\), subject to the restriction that \(0\le\alpha\le1\).
+
+In Table [8.1](https://otexts.com/fpp3/ses.html#tab:export-ses) we demonstrate the calculation using these parameters. The second last column shows the estimated level for times \(t=0\) to \(t=58\); the last few rows of the last column show the forecasts for \(h=1\) to \(5\)-steps ahead.
+
+Table 8.1: Forecasting goods and services exports from Algeria using simple exponential smoothing.
+
+| Year | Time | Observation | Level | Forecast |
+| --- | --- | --- | --- | --- |
+|  | \(t\) | \(y_t\) | \(\ell_t\) | \(\hat{y}_{t\vert t-1}\) |
+| 1959 | 0 |  | 39.54 |  |
+| 1960 | 1 | 39.04 | 39.12 | 39.54 |
+| 1961 | 2 | 46.24 | 45.10 | 39.12 |
+| 1962 | 3 | 19.79 | 23.84 | 45.10 |
+| 1963 | 4 | 24.68 | 24.55 | 23.84 |
+| 1964 | 5 | 25.08 | 25.00 | 24.55 |
+| 1965 | 6 | 22.60 | 22.99 | 25.00 |
+| 1966 | 7 | 25.99 | 25.51 | 22.99 |
+| 1967 | 8 | 23.43 | 23.77 | 25.51 |
+|  | ⋮ | ⋮ | ⋮ | ⋮ |
+| 2014 | 55 | 30.22 | 30.80 | 33.85 |
+| 2015 | 56 | 23.17 | 24.39 | 30.80 |
+| 2016 | 57 | 20.86 | 21.43 | 24.39 |
+| 2017 | 58 | 22.64 | 22.44 | 21.43 |
+|  | \(h\) |  |  | \(\hat{y}_{T+h\vert T}\) |
+| 2018 | 1 |  |  | 22.44 |
+| 2019 | 2 |  |  | 22.44 |
+| 2020 | 3 |  |  | 22.44 |
+| 2021 | 4 |  |  | 22.44 |
+| 2022 | 5 |  |  | 22.44 |
+
+The black line in Figure [8.2](https://otexts.com/fpp3/ses.html#fig:ses) shows the data, which has a changing level over time.
+
+```
+fc |>
+  autoplot(algeria_economy) +
+  geom_line(aes(y = .fitted), col="#D55E00",
+            data = augment(fit)) +
+  labs(y="% of GDP", title="Exports: Algeria") +
+  guides(colour = "none")
+```
+
+![Simple exponential smoothing applied to exports from Algeria (1960--2017). The orange curve shows the one-step-ahead fitted values.](https://otexts.com/fpp3/fpp_files/figure-html/ses-1.png)
+
+Figure 8.2: Simple exponential smoothing applied to exports from Algeria (1960–2017). The orange curve shows the one-step-ahead fitted values.
+
+The forecasts for the period 2018–2022 are plotted in Figure [8.2](https://otexts.com/fpp3/ses.html#fig:ses). Also plotted are one-step-ahead fitted values alongside the data over the period 1960–2017. The large value of \(\alpha\) in this example is reflected in the large adjustment that takes place in the estimated level \(\ell_t\) at each time. A smaller value of \(\alpha\) would lead to smaller changes over time, and so the series of fitted values would be smoother.
+
+The prediction intervals shown here are calculated using the methods described in Section [8.7](https://otexts.com/fpp3/ets-forecasting.html#ets-forecasting). The prediction intervals show that there is considerable uncertainty in the future exports over the five-year forecast period. So interpreting the point forecasts without accounting for the large uncertainty can be very misleading.
+
+---
+
+16. In some books it is called “single exponential smoothing”.[↩︎](https://otexts.com/fpp3/ses.html#fnref16)
+
+## 8.2 Methods with trend
+
+### Holt’s linear trend method
+
+Holt ([1957](#ref-Holt57)) extended simple exponential smoothing to allow the forecasting of data with a trend. This method involves a forecast equation and two smoothing equations (one for the level and one for the trend):
+\[\begin{align\*}
+\text{Forecast equation}&& \hat{y}_{t+h|t} &= \ell_{t} + hb_{t} \\
+\text{Level equation} && \ell_{t} &= \alpha y_{t} + (1 - \alpha)(\ell_{t-1} + b_{t-1})\\
+\text{Trend equation} && b_{t} &= \beta^\*(\ell_{t} - \ell_{t-1}) + (1 -\beta^\*)b_{t-1},
+\end{align\*}\]
+where \(\ell_t\) denotes an estimate of the level of the series at time \(t\), \(b_t\) denotes an estimate of the trend (slope) of the series at time \(t\), \(\alpha\) is the smoothing parameter for the level, \(0\le\alpha\le1\), and \(\beta^\*\) is the smoothing parameter for the trend, \(0\le\beta^\*\le1\). (We denote this as \(\beta^\*\) instead of \(\beta\) for reasons that will be explained in Section [8.5](https://otexts.com/fpp3/ets.html#ets).)
+
+As with simple exponential smoothing, the level equation here shows that \(\ell_t\) is a weighted average of observation \(y_t\) and the one-step-ahead training forecast for time \(t\), here given by \(\ell_{t-1} + b_{t-1}\). The trend equation shows that \(b_t\) is a weighted average of the estimated trend at time \(t\) based on \(\ell_{t} - \ell_{t-1}\) and \(b_{t-1}\), the previous estimate of the trend.
+
+The forecast function is no longer flat but trending. The \(h\)-step-ahead forecast is equal to the last estimated level plus \(h\) times the last estimated trend value. Hence the forecasts are a linear function of \(h\).
+
+### Example: Australian population
+
+```
+aus_economy <- global_economy |>
+  filter(Code == "AUS") |>
+  mutate(Pop = Population / 1e6)
+autoplot(aus_economy, Pop) +
+  labs(y = "Millions", title = "Australian population")
+```
+
+![Australia's population, 1960-2017.](https://otexts.com/fpp3/fpp_files/figure-html/auspop-1.png)
+
+Figure 8.3: Australia’s population, 1960-2017.
+
+Figure [8.3](https://otexts.com/fpp3/holt.html#fig:auspop) shows Australia’s annual population from 1960 to 2017. We will apply Holt’s method to this series. The smoothing parameters, \(\alpha\) and \(\beta^\*\), and the initial values \(\ell_0\) and \(b_0\) are estimated by minimising the SSE for the one-step training errors as in Section [8.1](https://otexts.com/fpp3/ses.html#ses).
+
+```
+fit <- aus_economy |>
+  model(
+    AAN = ETS(Pop ~ error("A") + trend("A") + season("N"))
+  )
+fc <- fit |> forecast(h = 10)
+```
+
+The estimated smoothing coefficient for the level is \(\hat{\alpha} = 0.9999\). The very high value shows that the level changes rapidly in order to capture the highly trended series. The estimated smoothing coefficient for the slope is \(\hat{\beta}^\* = 0.3267\). This is relatively large suggesting that the trend also changes often (even if the changes are slight).
+
+In Table [8.2](https://otexts.com/fpp3/holt.html#tab:popholt) we use these values to demonstrate the application of Holt’s method.
+
+Table 8.2: Forecasting Australian annual population using Holt’s linear trend method.
+
+| Year | Time | Observation | Level | Slope | Forecast |
+| --- | --- | --- | --- | --- | --- |
+|  | \(t\) | \(y_t\) | \(\ell_t\) |  | \(\hat{y}_{t+1\mid t}\) |
+| 1959 | 0 |  | 10.05 | 0.22 |  |
+| 1960 | 1 | 10.28 | 10.28 | 0.22 | 10.28 |
+| 1961 | 2 | 10.48 | 10.48 | 0.22 | 10.50 |
+| 1962 | 3 | 10.74 | 10.74 | 0.23 | 10.70 |
+| 1963 | 4 | 10.95 | 10.95 | 0.22 | 10.97 |
+| 1964 | 5 | 11.17 | 11.17 | 0.22 | 11.17 |
+| 1965 | 6 | 11.39 | 11.39 | 0.22 | 11.39 |
+| 1966 | 7 | 11.65 | 11.65 | 0.23 | 11.61 |
+|  | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
+| 2014 | 55 | 23.50 | 23.50 | 0.37 | 23.52 |
+| 2015 | 56 | 23.85 | 23.85 | 0.36 | 23.87 |
+| 2016 | 57 | 24.21 | 24.21 | 0.36 | 24.21 |
+| 2017 | 58 | 24.60 | 24.60 | 0.37 | 24.57 |
+|  | \(h\) |  |  |  | \(\hat{y}_{T+h\mid T}\) |
+| 2018 | 1 |  |  |  | 24.97 |
+| 2019 | 2 |  |  |  | 25.34 |
+| 2020 | 3 |  |  |  | 25.71 |
+| 2021 | 4 |  |  |  | 26.07 |
+| 2022 | 5 |  |  |  | 26.44 |
+| 2023 | 6 |  |  |  | 26.81 |
+| 2024 | 7 |  |  |  | 27.18 |
+| 2025 | 8 |  |  |  | 27.55 |
+| 2026 | 9 |  |  |  | 27.92 |
+| 2027 | 10 |  |  |  | 28.29 |
+
+### Damped trend methods
+
+The forecasts generated by Holt’s linear method display a constant trend (increasing or decreasing) indefinitely into the future. Empirical evidence indicates that these methods tend to over-forecast, especially for longer forecast horizons. Motivated by this observation, Gardner & McKenzie ([1985](#ref-GarMacK1985)) introduced a parameter that “dampens” the trend to a flat line some time in the future. Methods that include a damped trend have proven to be very successful, and are arguably the most popular individual methods when forecasts are required automatically for many series.
+
+In conjunction with the smoothing parameters \(\alpha\) and \(\beta^\*\) (with values between 0 and 1 as in Holt’s method), this method also includes a damping parameter \(0<\phi<1\):
+\[\begin{align\*}
+\hat{y}_{t+h|t} &= \ell_{t} + (\phi+\phi^2 + \dots + \phi^{h})b_{t} \\
+\ell_{t} &= \alpha y_{t} + (1 - \alpha)(\ell_{t-1} + \phi b_{t-1})\\
+b_{t} &= \beta^\*(\ell_{t} - \ell_{t-1}) + (1 -\beta^\*)\phi b_{t-1}.
+\end{align\*}\]
+If \(\phi=1\), the method is identical to Holt’s linear method. For values between \(0\) and \(1\), \(\phi\) dampens the trend so that it approaches a constant some time in the future. In fact, the forecasts converge to \(\ell_T+\phi b_T/(1-\phi)\) as \(h\rightarrow\infty\) for any value \(0<\phi<1\). This means that short-run forecasts are trended while long-run forecasts are constant.
+
+In practice, \(\phi\) is rarely less than 0.8 as the damping has a very strong effect for smaller values. Values of \(\phi\) close to 1 will mean that a damped model is not able to be distinguished from a non-damped model. For these reasons, we usually restrict \(\phi\) to a minimum of 0.8 and a maximum of 0.98.
+
+### Example: Australian Population (continued)
+
+Figure [8.4](https://otexts.com/fpp3/holt.html#fig:dampedtrend) shows the forecasts for years 2018–2032 generated from Holt’s linear trend method and the damped trend method.
+
+```
+aus_economy |>
+  model(
+    `Holt's method` = ETS(Pop ~ error("A") +
+                       trend("A") + season("N")),
+    `Damped Holt's method` = ETS(Pop ~ error("A") +
+                       trend("Ad", phi = 0.9) + season("N"))
+  ) |>
+  forecast(h = 15) |>
+  autoplot(aus_economy, level = NULL) +
+  labs(title = "Australian population",
+       y = "Millions") +
+  guides(colour = guide_legend(title = "Forecast"))
+```
+
+![Forecasting annual Australian population (millions) over 2018-2032. For the damped trend method, $\phi=0.90$.](https://otexts.com/fpp3/fpp_files/figure-html/dampedtrend-1.png)
+
+Figure 8.4: Forecasting annual Australian population (millions) over 2018-2032. For the damped trend method, \(\phi=0.90\).
+
+We have set the damping parameter to a relatively low number \((\phi=0.90)\) to exaggerate the effect of damping for comparison. Usually, we would estimate \(\phi\) along with the other parameters. We have also used a rather large forecast horizon (\(h=15\)) to highlight the difference between a damped trend and a linear trend.
+
+### Example: Internet usage
+
+In this example, we compare the forecasting performance of the three exponential smoothing methods that we have considered so far in forecasting the number of users connected to the internet via a server. The data is observed over 100 minutes and is shown in Figure [8.5](https://otexts.com/fpp3/holt.html#fig:www-usage).
+
+```
+www_usage <- as_tsibble(WWWusage)
+www_usage |> autoplot(value) +
+  labs(x="Minute", y="Number of users",
+       title = "Internet usage per minute")
+```
+
+![Users connected to the internet through a server](https://otexts.com/fpp3/fpp_files/figure-html/www-usage-1.png)
+
+Figure 8.5: Users connected to the internet through a server
+
+We will use time series cross-validation to compare the one-step forecast accuracy of the three methods.
+
+```
+www_usage |>
+  stretch_tsibble(.init = 10) |>
+  model(
+    SES = ETS(value ~ error("A") + trend("N") + season("N")),
+    Holt = ETS(value ~ error("A") + trend("A") + season("N")),
+    Damped = ETS(value ~ error("A") + trend("Ad") +
+                   season("N"))
+  ) |>
+  forecast(h = 1) |>
+  accuracy(www_usage)
+#> # A tibble: 3 × 10
+#>   .model .type     ME  RMSE   MAE   MPE  MAPE  MASE RMSSE  ACF1
+#>   <chr>  <chr>  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
+#> 1 Damped Test  0.288   3.69  3.00 0.347  2.26 0.663 0.636 0.336
+#> 2 Holt   Test  0.0610  3.87  3.17 0.244  2.38 0.701 0.668 0.296
+#> 3 SES    Test  1.46    6.05  4.81 0.904  3.55 1.06  1.04  0.803
+```
+
+Damped Holt’s method is best whether you compare MAE or RMSE values. So we will proceed with using the damped Holt’s method and apply it to the whole data set to get forecasts for future minutes.
+
+```
+fit <- www_usage |>
+  model(
+    Damped = ETS(value ~ error("A") + trend("Ad") +
+                   season("N"))
+  )
+# Estimated parameters:
+tidy(fit)
+#> # A tibble: 5 × 3
+#>   .model term  estimate
+#>   <chr>  <chr>    <dbl>
+#> 1 Damped alpha   1.000
+#> 2 Damped beta    0.997
+#> 3 Damped phi     0.815
+#> 4 Damped l[0]   90.4
+#> 5 Damped b[0]   -0.0173
+```
+
+The smoothing parameter for the slope is estimated to be almost one, indicating that the trend changes to mostly reflect the slope between the last two minutes of internet usage. The value of \(\alpha\) is very close to one, showing that the level reacts strongly to each new observation.
+
+```
+fit |>
+  forecast(h = 10) |>
+  autoplot(www_usage) +
+  labs(x="Minute", y="Number of users",
+       title = "Internet usage per minute")
+```
+
+![Forecasting internet usage: comparing forecasting performance of non-seasonal methods.](https://otexts.com/fpp3/fpp_files/figure-html/fig-7-comp-1.png)
+
+Figure 8.6: Forecasting internet usage: comparing forecasting performance of non-seasonal methods.
+
+The resulting forecasts look sensible with decreasing trend, which flattens out due to the low value of the damping parameter (0.815), and relatively wide prediction intervals reflecting the variation in the historical data. The prediction intervals are calculated using the methods described in Section [8.7](https://otexts.com/fpp3/ets-forecasting.html#ets-forecasting).
+
+In this example, the process of selecting a method was relatively easy as both MSE and MAE comparisons suggested the same method (damped Holt’s). However, sometimes different accuracy measures will suggest different forecasting methods, and then a decision is required as to which forecasting method we prefer to use. As forecasting tasks can vary by many dimensions (length of forecast horizon, size of test set, forecast error measures, frequency of data, etc.), it is unlikely that one method will be better than all others for all forecasting scenarios. What we require from a forecasting method are consistently sensible forecasts, and these should be frequently evaluated against the task at hand.
+
+### Bibliography
+
+Gardner, E. S., & McKenzie, E. (1985). Forecasting trends in time series. *Management Science*, *31*(10), 1237–1246.
+
+Holt, C. C. (1957). *Forecasting seasonals and trends by exponentially weighted averages* (ONR Memorandum No. 52). Carnegie Institute of Technology, Pittsburgh USA. Reprinted in the *International Journal of Forecasting*, 2004.
+
+## 8.3 Methods with seasonality
+
+Holt ([1957](#ref-Holt57)) and Winters ([1960](#ref-Winters60)) extended Holt’s method to capture seasonality. The Holt-Winters seasonal method comprises the forecast equation and three smoothing equations — one for the level \(\ell_t\), one for the trend \(b_t\), and one for the seasonal component \(s_t\), with corresponding smoothing parameters \(\alpha\), \(\beta^\*\) and \(\gamma\). We use \(m\) to denote the period of the seasonality, i.e., the number of seasons in a year. For example, for quarterly data \(m=4\), and for monthly data \(m=12\).
+
+There are two variations to this method that differ in the nature of the seasonal component. The additive method is preferred when the seasonal variations are roughly constant through the series, while the multiplicative method is preferred when the seasonal variations are changing proportional to the level of the series. With the additive method, the seasonal component is expressed in absolute terms in the scale of the observed series, and in the level equation the series is seasonally adjusted by subtracting the seasonal component. Within each year, the seasonal component will add up to approximately zero. With the multiplicative method, the seasonal component is expressed in relative terms (percentages), and the series is seasonally adjusted by dividing through by the seasonal component. Within each year, the seasonal component will sum up to approximately \(m\).
+
+### Holt-Winters’ additive method
+
+The component form for the additive method is:
+\[\begin{align\*}
+\hat{y}_{t+h|t} &= \ell_{t} + hb_{t} + s_{t+h-m(k+1)} \\
+\ell_{t} &= \alpha(y_{t} - s_{t-m}) + (1 - \alpha)(\ell_{t-1} + b_{t-1})\\
+b_{t} &= \beta^\*(\ell_{t} - \ell_{t-1}) + (1 - \beta^\*)b_{t-1}\\
+s_{t} &= \gamma (y_{t}-\ell_{t-1}-b_{t-1}) + (1-\gamma)s_{t-m},
+\end{align\*}\]
+where \(k\) is the integer part of \((h-1)/m\), which ensures that the estimates of the seasonal indices used for forecasting come from the final year of the sample. The level equation shows a weighted average between the seasonally adjusted observation \((y_{t} - s_{t-m})\) and the non-seasonal forecast \((\ell_{t-1}+b_{t-1})\) for time \(t\). The trend equation is identical to Holt’s linear method. The seasonal equation shows a weighted average between the current seasonal index, \((y_{t}-\ell_{t-1}-b_{t-1})\), and the seasonal index of the same season last year (i.e., \(m\) time periods ago).
+
+The equation for the seasonal component is often expressed as
+\[
+s_{t} = \gamma^\* (y_{t}-\ell_{t})+ (1-\gamma^\*)s_{t-m}.
+\]
+If we substitute \(\ell_t\) from the smoothing equation for the level of the component form above, we get
+\[
+s_{t} = \gamma^\*(1-\alpha) (y_{t}-\ell_{t-1}-b_{t-1})+ [1-\gamma^\*(1-\alpha)]s_{t-m},
+\]
+which is identical to the smoothing equation for the seasonal component we specify here, with \(\gamma=\gamma^\*(1-\alpha)\). The usual parameter restriction is \(0\le\gamma^\*\le1\), which translates to \(0\le\gamma\le 1-\alpha\).
+
+### Holt-Winters’ multiplicative method
+
+The component form for the multiplicative method is:
+\[\begin{align\*}
+\hat{y}_{t+h|t} &= (\ell_{t} + hb_{t})s_{t+h-m(k+1)} \\
+\ell_{t} &= \alpha \frac{y_{t}}{s_{t-m}} + (1 - \alpha)(\ell_{t-1} + b_{t-1})\\
+b_{t} &= \beta^\*(\ell_{t}-\ell_{t-1}) + (1 - \beta^\*)b_{t-1} \\
+s_{t} &= \gamma \frac{y_{t}}{(\ell_{t-1} + b_{t-1})} + (1 - \gamma)s_{t-m}.
+\end{align\*}\]
+
+### Example: Domestic overnight trips in Australia
+
+We apply Holt-Winters’ method with both additive and multiplicative seasonality[17](#fn17) to forecast quarterly visitor nights in Australia spent by domestic tourists. Figure [8.7](https://otexts.com/fpp3/holt-winters.html#fig:7-HW) shows the data from 1998–2017, and the forecasts for 2018–2020. The data show an obvious seasonal pattern, with peaks observed in the March quarter of each year, corresponding to the Australian summer.
+
+```
+aus_holidays <- tourism |>
+  filter(Purpose == "Holiday") |>
+  summarise(Trips = sum(Trips)/1e3)
+fit <- aus_holidays |>
+  model(
+    additive = ETS(Trips ~ error("A") + trend("A") +
+                                                season("A")),
+    multiplicative = ETS(Trips ~ error("M") + trend("A") +
+                                                season("M"))
+  )
+fc <- fit |> forecast(h = "3 years")
+fc |>
+  autoplot(aus_holidays, level = NULL) +
+  labs(title="Australian domestic tourism",
+       y="Overnight trips (millions)") +
+  guides(colour = guide_legend(title = "Forecast"))
+```
+
+![Forecasting domestic overnight trips in Australia using the Holt-Winters method with both additive and multiplicative seasonality.](https://otexts.com/fpp3/fpp_files/figure-html/7-HW-1.png)
+
+Figure 8.7: Forecasting domestic overnight trips in Australia using the Holt-Winters method with both additive and multiplicative seasonality.
+
+Table 8.3: Applying Holt-Winters’ method with additive seasonality for forecasting domestic tourism in Australia. Notice that the additive seasonal component sums to approximately zero. The smoothing parameters are \(\alpha = 0.2620\), \(\beta^\* = 0.1646\), \(\gamma = 0.0001\) and RMSE \(=0.4169\).
+
+| Quarter | Time | Observation | Level | Slope | Season | Forecast |
+| --- | --- | --- | --- | --- | --- | --- |
+|  | \(t\) | \(y_t\) | \(\ell_t\) | \(b_t\) | \(s_t\) | \(\hat{y}_{t+1\vert t}\) |
+| 1997 Q1 | 0 |  |  |  | 1.5 |  |
+| 1997 Q2 | 1 |  |  |  | -0.3 |  |
+| 1997 Q3 | 2 |  |  |  | -0.7 |  |
+| 1997 Q4 | 3 |  | 9.8 | 0.0 | -0.5 |  |
+| 1998 Q1 | 4 | 11.8 | 9.9 | 0.0 | 1.5 | 11.3 |
+| 1998 Q2 | 5 | 9.3 | 9.9 | 0.0 | -0.3 | 9.7 |
+| 1998 Q3 | 6 | 8.6 | 9.7 | -0.0 | -0.7 | 9.2 |
+| 1998 Q4 | 7 | 9.3 | 9.8 | 0.0 | -0.5 | 9.2 |
+|  | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
+| 2017 Q1 | 80 | 12.4 | 10.9 | 0.1 | 1.5 | 12.3 |
+| 2017 Q2 | 81 | 10.5 | 10.9 | 0.1 | -0.3 | 10.7 |
+| 2017 Q3 | 82 | 10.5 | 11.0 | 0.1 | -0.7 | 10.3 |
+| 2017 Q4 | 83 | 11.2 | 11.3 | 0.1 | -0.5 | 10.6 |
+|  | \(h\) |  |  |  |  | \(\hat{y}_{T+h\vert T}\) |
+| 2018 Q1 | 1 |  |  |  |  | 12.9 |
+| 2018 Q2 | 2 |  |  |  |  | 11.2 |
+| 2018 Q3 | 3 |  |  |  |  | 11.0 |
+| 2018 Q4 | 4 |  |  |  |  | 11.2 |
+| 2019 Q1 | 5 |  |  |  |  | 13.4 |
+| 2019 Q2 | 6 |  |  |  |  | 11.7 |
+| 2019 Q3 | 7 |  |  |  |  | 11.5 |
+| 2019 Q4 | 8 |  |  |  |  | 11.7 |
+| 2020 Q1 | 9 |  |  |  |  | 13.9 |
+| 2020 Q2 | 10 |  |  |  |  | 12.2 |
+| 2020 Q3 | 11 |  |  |  |  | 11.9 |
+| 2020 Q4 | 12 |  |  |  |  | 12.2 |
+
+Table 8.4: Applying Holt-Winters’ method with multiplicative seasonality for forecasting domestic tourism in Australia. Notice that the multiplicative seasonal component sums to approximately \(m=4\). The smoothing parameters are \(\alpha = 0.2237\), \(\beta^\* = 0.1360\), \(\gamma = 0.0001\) and RMSE \(=0.4122\).
+
+| Quarter | Time | Observation | Level | Slope | Season | Forecast |
+| --- | --- | --- | --- | --- | --- | --- |
+|  | \(t\) | \(y_t\) | \(\ell_t\) | \(b_t\) | \(s_t\) | \(\hat{y}_{t+1\vert t}\) |
+| 1997 Q1 | 0 |  |  |  | 1.2 |  |
+| 1997 Q2 | 1 |  |  |  | 1.0 |  |
+| 1997 Q3 | 2 |  |  |  | 0.9 |  |
+| 1997 Q4 | 3 |  | 10.0 | -0.0 | 0.9 |  |
+| 1998 Q1 | 4 | 11.8 | 10.0 | -0.0 | 1.2 | 11.6 |
+| 1998 Q2 | 5 | 9.3 | 9.9 | -0.0 | 1.0 | 9.7 |
+| 1998 Q3 | 6 | 8.6 | 9.8 | -0.0 | 0.9 | 9.2 |
+| 1998 Q4 | 7 | 9.3 | 9.8 | -0.0 | 0.9 | 9.2 |
+|  | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
+| 2017 Q1 | 80 | 12.4 | 10.8 | 0.1 | 1.2 | 12.6 |
+| 2017 Q2 | 81 | 10.5 | 10.9 | 0.1 | 1.0 | 10.6 |
+| 2017 Q3 | 82 | 10.5 | 11.1 | 0.1 | 0.9 | 10.2 |
+| 2017 Q4 | 83 | 11.2 | 11.3 | 0.1 | 0.9 | 10.5 |
+|  | \(h\) |  |  |  |  | \(\hat{y}_{T+h\vert T}\) |
+| 2018 Q1 | 1 |  |  |  |  | 13.3 |
+| 2018 Q2 | 2 |  |  |  |  | 11.2 |
+| 2018 Q3 | 3 |  |  |  |  | 10.8 |
+| 2018 Q4 | 4 |  |  |  |  | 11.1 |
+| 2019 Q1 | 5 |  |  |  |  | 13.8 |
+| 2019 Q2 | 6 |  |  |  |  | 11.7 |
+| 2019 Q3 | 7 |  |  |  |  | 11.3 |
+| 2019 Q4 | 8 |  |  |  |  | 11.6 |
+| 2020 Q1 | 9 |  |  |  |  | 14.4 |
+| 2020 Q2 | 10 |  |  |  |  | 12.2 |
+| 2020 Q3 | 11 |  |  |  |  | 11.7 |
+| 2020 Q4 | 12 |  |  |  |  | 12.1 |
+
+The applications of both methods (with additive and multiplicative seasonality) are presented in Tables [8.3](https://otexts.com/fpp3/holt-winters.html#tab:tab75) and [8.4](https://otexts.com/fpp3/holt-winters.html#tab:tab76) respectively. Because both methods have exactly the same number of parameters to estimate, we can compare the training RMSE from both models. In this case, the method with multiplicative seasonality fits the data slightly better.
+
+The estimated components for both models are plotted in Figure [8.8](https://otexts.com/fpp3/holt-winters.html#fig:fig-7-LevelTrendSeas). The small value of \(\gamma\) for the multiplicative model means that the seasonal component hardly changes over time. The small value of \(\beta^{\*}\) means the slope component hardly changes over time (compare the vertical scales of the slope and level components).
+
+![Estimated components for the Holt-Winters method with additive and multiplicative seasonal components.](https://otexts.com/fpp3/fpp_files/figure-html/fig-7-LevelTrendSeas-1.png)
+
+Figure 8.8: Estimated components for the Holt-Winters method with additive and multiplicative seasonal components.
+
+### Holt-Winters’ damped method
+
+Damping is possible with both additive and multiplicative Holt-Winters’ methods. A method that often provides accurate and robust forecasts for seasonal data is the Holt-Winters method with a damped trend and multiplicative seasonality:
+\[\begin{align\*}
+\hat{y}_{t+h|t} &= \left[\ell_{t} + (\phi+\phi^2 + \dots + \phi^{h})b_{t}\right]s_{t+h-m(k+1)} \\
+\ell_{t} &= \alpha(y_{t} / s_{t-m}) + (1 - \alpha)(\ell_{t-1} + \phi b_{t-1})\\
+b_{t} &= \beta^\*(\ell_{t} - \ell_{t-1}) + (1 - \beta^\*)\phi b_{t-1} \\
+s_{t} &= \gamma \frac{y_{t}}{(\ell_{t-1} + \phi b_{t-1})} + (1 - \gamma)s_{t-m}.
+\end{align\*}\]
+
+### Example: Holt-Winters method with daily data
+
+The Holt-Winters method can also be used for daily type of data, where the seasonal period is \(m=7\), and the appropriate unit of time for \(h\) is in days. Here we forecast pedestrian traffic at a busy Melbourne train station in July 2016.
+
+```
+sth_cross_ped <- pedestrian |>
+  filter(Date >= "2016-07-01",
+         Sensor == "Southern Cross Station") |>
+  index_by(Date) |>
+  summarise(Count = sum(Count)/1000)
+sth_cross_ped |>
+  filter(Date <= "2016-07-31") |>
+  model(
+    hw = ETS(Count ~ error("M") + trend("Ad") + season("M"))
+  ) |>
+  forecast(h = "2 weeks") |>
+  autoplot(sth_cross_ped |> filter(Date <= "2016-08-14")) +
+  labs(title = "Daily traffic: Southern Cross",
+       y="Pedestrians ('000)")
+```
+
+![Forecasts of daily pedestrian traffic at the Southern Cross railway station, Melbourne.](https://otexts.com/fpp3/fpp_files/figure-html/hyndsight-1.png)
+
+Figure 8.9: Forecasts of daily pedestrian traffic at the Southern Cross railway station, Melbourne.
+
+Clearly the model has identified the weekly seasonal pattern and the increasing trend at the end of the data, and the forecasts are a close match to the test data.
+
+### Bibliography
+
+Holt, C. C. (1957). *Forecasting seasonals and trends by exponentially weighted averages* (ONR Memorandum No. 52). Carnegie Institute of Technology, Pittsburgh USA. Reprinted in the *International Journal of Forecasting*, 2004.
+
+Winters, P. R. (1960). Forecasting sales by exponentially weighted moving averages. *Management Science*, *6*(3), 324–342.
+
+---
+
+17. Our implementation uses maximum likelihood estimation as described in Section [8.6](https://otexts.com/fpp3/ets-estimation.html#ets-estimation) while Holt and Winters originally minimized the sum of squared errors. For multiplicative seasonality, this will lead to slightly different parameter estimates. Optimizing the sum of squared errors can be obtained by setting `opt_crit="mse"` in `ETS()`.[↩︎](https://otexts.com/fpp3/holt-winters.html#fnref17)
+
+## 8.4 A taxonomy of exponential smoothing methods
+
+Exponential smoothing methods are not restricted to those we have presented so far. By considering variations in the combinations of the trend and seasonal components, nine exponential smoothing methods are possible, listed in Table [8.5](https://otexts.com/fpp3/taxonomy.html#tab:taxonomy). Each method is labelled by a pair of letters (T,S) defining the type of ‘Trend’ and ‘Seasonal’ components. For example, (A,M) is the method with an additive trend and multiplicative seasonality; (A\(_d\),N) is the method with damped trend and no seasonality; and so on.
+
+Table 8.5: A two-way classification of exponential smoothing methods.
+
+| Trend Component | Seasonal Component | | |
+| --- | --- | --- | --- |
+|  | N | A | M |
+|  | (None) | (Additive) | (Multiplicative) |
+| N (None) | (N,N) | (N,A) | (N,M) |
+| A (Additive) | (A,N) | (A,A) | (A,M) |
+| A\(_d\) (Additive damped) | (A\(_d\),N) | (A\(_d\),A) | (A\(_d\),M) |
+
+Some of these methods we have already seen using other names:
+
+| Short hand | Method |
+| --- | --- |
+| (N,N) | Simple exponential smoothing |
+| (A,N) | Holt’s linear method |
+| (A\(_d\),N) | Additive damped trend method |
+| (A,A) | Additive Holt-Winters’ method |
+| (A,M) | Multiplicative Holt-Winters’ method |
+| (A\(_d\),M) | Holt-Winters’ damped method |
+
+This type of classification was first proposed by Pegels ([1969](#ref-Pegels1969)), who also included a method with a multiplicative trend. It was later extended by Gardner ([1985](#ref-Gar1985)) to include methods with an additive damped trend and by J. W. Taylor ([2003](#ref-Taylor2003)) to include methods with a multiplicative damped trend. We do not consider the multiplicative trend methods in this book as they tend to produce poor forecasts. See Hyndman et al. ([2008](#ref-expsmooth08)) for a more thorough discussion of all exponential smoothing methods.
+
+Table [8.6](https://otexts.com/fpp3/taxonomy.html#tab:pegels) gives the recursive formulas for applying the nine exponential smoothing methods in Table [8.5](https://otexts.com/fpp3/taxonomy.html#tab:taxonomy). Each cell includes the forecast equation for generating \(h\)-step-ahead forecasts, and the smoothing equations for applying the method.
+
+Table 8.6:  Formulas for recursive calculations and point forecasts. In each case, \(\ell_t\) denotes the series level at time \(t\), \(b_t\) denotes the slope at time \(t\), \(s_t\) denotes the seasonal component of the series at time \(t\), and \(m\) denotes the number of seasons in a year; \(\alpha\), \(\beta^\*\), \(\gamma\) and \(\phi\) are smoothing parameters, \(\phi_h = \phi+\phi^2+\dots+\phi^{h}\), and \(k\) is the integer part of \((h-1)/m\).
+
+|  |
+| --- |
+| ![](https://otexts.com/fpp3/figs/pegelstable-1.png) |
+
+### Bibliography
+
+Gardner, E. S. (1985). Exponential smoothing: The state of the art. *Journal of Forecasting*, *4*(1), 1–28.
+
+Hyndman, R. J., Koehler, A. B., Ord, J. K., & Snyder, R. D. (2008). *Forecasting with exponential smoothing: The state space approach*. Springer-Verlag.
+
+Pegels, C. C. (1969). Exponential forecasting: Some new variations. *Management Science*, *15*(5), 311–315.
+
+Taylor, J. W. (2003). Exponential smoothing with a damped multiplicative trend. *International Journal of Forecasting*, *19*(4), 715–725.
+
+## 8.5 Innovations state space models for exponential smoothing
+
+In the rest of this chapter, we study the statistical models that underlie the exponential smoothing methods we have considered so far. The exponential smoothing methods presented in Table [8.6](https://otexts.com/fpp3/taxonomy.html#tab:pegels) are algorithms which generate point forecasts. The statistical models in this section generate the same point forecasts, but can also generate prediction (or forecast) intervals. A statistical model is a stochastic (or random) data generating process that can produce an entire forecast distribution. We will also describe how to use the model selection criteria introduced in Chapter [7](https://otexts.com/fpp3/regression.html#regression) to choose the model in an objective manner.
+
+Each model consists of a measurement equation that describes the observed data, and some state equations that describe how the unobserved components or states (level, trend, seasonal) change over time. Hence, these are referred to as **state space models**.
+
+For each method there exist two models: one with additive errors and one with multiplicative errors. The point forecasts produced by the models are identical if they use the same smoothing parameter values. They will, however, generate different prediction intervals.
+
+To distinguish between a model with additive errors and one with multiplicative errors (and also to distinguish the models from the methods), we add a third letter to the classification of Table [8.5](https://otexts.com/fpp3/taxonomy.html#tab:taxonomy). We label each state space model as ETS(\(\cdot,\cdot,\cdot\)) for (Error, Trend, Seasonal). This label can also be thought of as ExponenTial Smoothing. Using the same notation as in Table [8.5](https://otexts.com/fpp3/taxonomy.html#tab:taxonomy), the possibilities for each component (or state) are: Error \(=\{\)A,M\(\}\), Trend \(=\{\)N,A,A\(_d\}\) and Seasonal \(=\{\)N,A,M\(\}\).
+
+### ETS(A,N,N): simple exponential smoothing with additive errors
+
+Recall the component form of simple exponential smoothing:
+\[\begin{align\*}
+\text{Forecast equation} && \hat{y}_{t+1|t} & = \ell_{t}\\
+\text{Smoothing equation} && \ell_{t} & = \alpha y_{t} + (1 - \alpha)\ell_{t-1}.
+\end{align\*}\]
+If we re-arrange the smoothing equation for the level, we get the “error correction” form,
+\[\begin{align\*}
+\ell_{t} %&= \alpha y_{t}+\ell_{t-1}-\alpha\ell_{t-1}\\
+&= \ell_{t-1}+\alpha( y_{t}-\ell_{t-1})\\
+&= \ell_{t-1}+\alpha e_{t},
+\end{align\*}\]
+where \(e_{t}=y_{t}-\ell_{t-1}=y_{t}-\hat{y}_{t|t-1}\) is the residual at time \(t\).
+
+The training data errors lead to the adjustment of the estimated level throughout the smoothing process for \(t=1,\dots,T\). For example, if the error at time \(t\) is negative, then \(y_t < \hat{y}_{t|t-1}\) and so the level at time \(t-1\) has been over-estimated. The new level \(\ell_t\) is then the previous level \(\ell_{t-1}\) adjusted downwards. The closer \(\alpha\) is to one, the “rougher” the estimate of the level (large adjustments take place). The smaller the \(\alpha\), the “smoother” the level (small adjustments take place).
+
+We can also write \(y_t = \ell_{t-1} + e_t\), so that each observation can be represented by the previous level plus an error. To make this into an innovations state space model, all we need to do is specify the probability distribution for \(e_t\). For a model with additive errors, we assume that residuals (the one-step training errors) \(e_t\) are normally distributed white noise with mean 0 and variance \(\sigma^2\). A short-hand notation for this is \(e_t = \varepsilon_t\sim\text{NID}(0,\sigma^2)\); NID stands for “normally and independently distributed”.
+
+Then the equations of the model can be written as
+\[\begin{align}
+y_t &= \ell_{t-1} + \varepsilon_t \tag{8.3}\\
+\ell_t&=\ell_{t-1}+\alpha \varepsilon_t. \tag{8.4}
+\end{align}\]
+We refer to [(8.3)](https://otexts.com/fpp3/ets.html#eq:ann-1a) as the *measurement* (or observation) equation and [(8.4)](https://otexts.com/fpp3/ets.html#eq:ann-2a) as the *state* (or transition) equation. These two equations, together with the statistical distribution of the errors, form a fully specified statistical model. Specifically, these constitute an innovations state space model underlying simple exponential smoothing.
+
+The term “innovations” comes from the fact that all equations use the same random error process, \(\varepsilon_t\). For the same reason, this formulation is also referred to as a “single source of error” model. There are alternative multiple source of error formulations which we do not present here.
+
+The measurement equation shows the relationship between the observations and the unobserved states. In this case, observation \(y_t\) is a linear function of the level \(\ell_{t-1}\), the predictable part of \(y_t\), and the error \(\varepsilon_t\), the unpredictable part of \(y_t\). For other innovations state space models, this relationship may be nonlinear.
+
+The state equation shows the evolution of the state through time. The influence of the smoothing parameter \(\alpha\) is the same as for the methods discussed earlier. For example, \(\alpha\) governs the amount of change in successive levels: high values of \(\alpha\) allow rapid changes in the level; low values of \(\alpha\) lead to smooth changes. If \(\alpha=0\), the level of the series does not change over time; if \(\alpha=1\), the model reduces to a random walk model, \(y_t=y_{t-1}+\varepsilon_t\). (See Section [9.1](https://otexts.com/fpp3/stationarity.html#stationarity) for a discussion of this model.)
+
+### ETS(M,N,N): simple exponential smoothing with multiplicative errors
+
+In a similar fashion, we can specify models with multiplicative errors by writing the one-step-ahead training errors as relative errors
+\[
+\varepsilon_t = \frac{y_t-\hat{y}_{t|t-1}}{\hat{y}_{t|t-1}}
+\]
+where \(\varepsilon_t \sim \text{NID}(0,\sigma^2)\). Substituting \(\hat{y}_{t|t-1}=\ell_{t-1}\) gives \(y_t = \ell_{t-1}+\ell_{t-1}\varepsilon_t\) and \(e_t = y_t - \hat{y}_{t|t-1} = \ell_{t-1}\varepsilon_t\).
+
+Then we can write the multiplicative form of the state space model as
+\[\begin{align\*}
+y_t&=\ell_{t-1}(1+\varepsilon_t)\\
+\ell_t&=\ell_{t-1}(1+\alpha \varepsilon_t).
+\end{align\*}\]
+
+### ETS(A,A,N): Holt’s linear method with additive errors
+
+For this model, we assume that the one-step-ahead training errors are given by \(\varepsilon_t=y_t-\ell_{t-1}-b_{t-1} \sim \text{NID}(0,\sigma^2)\). Substituting this into the error correction equations for Holt’s linear method we obtain
+\[\begin{align\*}
+y_t&=\ell_{t-1}+b_{t-1}+\varepsilon_t\\
+\ell_t&=\ell_{t-1}+b_{t-1}+\alpha \varepsilon_t\\
+b_t&=b_{t-1}+\beta \varepsilon_t,
+\end{align\*}\]
+where for simplicity we have set \(\beta=\alpha \beta^\*\).
+
+### ETS(M,A,N): Holt’s linear method with multiplicative errors
+
+Specifying one-step-ahead training errors as relative errors such that
+\[
+\varepsilon_t=\frac{y_t-(\ell_{t-1}+b_{t-1})}{(\ell_{t-1}+b_{t-1})}
+\]
+and following an approach similar to that used above, the innovations state space model underlying Holt’s linear method with multiplicative errors is specified as
+\[\begin{align\*}
+y_t&=(\ell_{t-1}+b_{t-1})(1+\varepsilon_t)\\
+\ell_t&=(\ell_{t-1}+b_{t-1})(1+\alpha \varepsilon_t)\\
+b_t&=b_{t-1}+\beta(\ell_{t-1}+b_{t-1}) \varepsilon_t,
+\end{align\*}\]
+
+where again \(\beta=\alpha \beta^\*\) and \(\varepsilon_t \sim \text{NID}(0,\sigma^2)\).
+
+### Other ETS models
+
+In a similar fashion, we can write an innovations state space model for each of the exponential smoothing methods of Table [8.6](https://otexts.com/fpp3/taxonomy.html#tab:pegels). Table [8.7](https://otexts.com/fpp3/ets.html#tab:ssm) presents the equations for all of the models in the ETS framework.
+
+Table 8.7:  State space equations for each of the models in the ETS framework.
+
+|  |
+| --- |
+| ![](https://otexts.com/fpp3/figs/statespacemodels-1.png) |
+
+## 8.6 Estimation and model selection
+
+### Estimating ETS models
+
+An alternative to estimating the parameters by minimising the sum of squared errors is to maximise the “likelihood”. The likelihood is the probability of the data arising from the specified model. Thus, a large likelihood is associated with a good model. For an additive error model, maximising the likelihood (assuming normally distributed errors) gives the same results as minimising the sum of squared errors. However, different results will be obtained for multiplicative error models. In this section, we will estimate the smoothing parameters \(\alpha\), \(\beta\), \(\gamma\) and \(\phi\), and the initial states \(\ell_0\), \(b_0\), \(s_0,s_{-1},\dots,s_{-m+1}\), by maximising the likelihood.
+
+The possible values that the smoothing parameters can take are restricted. Traditionally, the parameters have been constrained to lie between 0 and 1 so that the equations can be interpreted as weighted averages. That is, \(0< \alpha,\beta^\*,\gamma^\*,\phi<1\). For the state space models, we have set \(\beta=\alpha\beta^\*\) and \(\gamma=(1-\alpha)\gamma^\*\). Therefore, the traditional restrictions translate to \(0< \alpha <1\), \(0 < \beta < \alpha\) and \(0< \gamma < 1-\alpha\). In practice, the damping parameter \(\phi\) is usually constrained further to prevent numerical difficulties in estimating the model. In the `fable` package, it is restricted so that \(0.8<\phi<0.98\).
+
+Another way to view the parameters is through a consideration of the mathematical properties of the state space models. The parameters are constrained in order to prevent observations in the distant past having a continuing effect on current forecasts. This leads to some *admissibility* constraints on the parameters, which are usually (but not always) less restrictive than the traditional constraints region ([Hyndman et al., 2008, pp. 149–161](#ref-expsmooth08)). For example, for the ETS(A,N,N) model, the traditional parameter region is \(0< \alpha <1\) but the admissible region is \(0< \alpha <2\). For the ETS(A,A,N) model, the traditional parameter region is \(0<\alpha<1\) and \(0<\beta<\alpha\) but the admissible region is \(0<\alpha<2\) and \(0<\beta<4-2\alpha\).
+
+### Model selection
+
+A great advantage of the ETS statistical framework is that information criteria can be used for model selection. The AIC, AIC\(_{\text{c}}\) and BIC, introduced in Section [7.5](https://otexts.com/fpp3/selecting-predictors.html#selecting-predictors), can be used here to determine which of the ETS models is most appropriate for a given time series.
+
+For ETS models, Akaike’s Information Criterion (AIC) is defined as
+\[
+\text{AIC} = -2\log(L) + 2k,
+\]
+where \(L\) is the likelihood of the model and \(k\) is the total number of parameters and initial states that have been estimated (including the residual variance).
+
+The AIC corrected for small sample bias (AIC\(_\text{c}\)) is defined as
+\[
+\text{AIC}_{\text{c}} = \text{AIC} + \frac{2k(k+1)}{T-k-1},
+\]
+and the Bayesian Information Criterion (BIC) is
+\[
+\text{BIC} = \text{AIC} + k[\log(T)-2].
+\]
+
+Three of the combinations of (Error, Trend, Seasonal) can lead to numerical difficulties. Specifically, the models that can cause such instabilities are ETS(A,N,M), ETS(A,A,M), and ETS(A,A\(_d\),M), due to division by values potentially close to zero in the state equations. We normally do not consider these particular combinations when selecting a model.
+
+Models with multiplicative errors are useful when the data are strictly positive, but are not numerically stable when the data contain zeros or negative values. Therefore, multiplicative error models will not be considered if the time series is not strictly positive. In that case, only the six fully additive models will be applied.
+
+### Example: Domestic holiday tourist visitor nights in Australia
+
+We now employ the ETS statistical framework to forecast Australian holiday tourism over the period 2016–2019. We let the `ETS()` function select the model by minimising the AICc.
+
+```
+aus_holidays <- tourism |>
+  filter(Purpose == "Holiday") |>
+  summarise(Trips = sum(Trips)/1e3)
+fit <- aus_holidays |>
+  model(ETS(Trips))
+report(fit)
+#> Series: Trips
+#> Model: ETS(M,N,A)
+#>   Smoothing parameters:
+#>     alpha = 0.3484
+#>     gamma = 1e-04
+#>
+#>   Initial states:
+#>   l[0]    s[0]   s[-1]   s[-2] s[-3]
+#>  9.727 -0.5376 -0.6884 -0.2934 1.519
+#>
+#>   sigma^2:  0.0022
+#>
+#>   AIC  AICc   BIC
+#> 226.2 227.8 242.9
+```
+
+The model selected is ETS(M,N,A)
+\[\begin{align\*}
+y_{t} &= (\ell_{t-1}+s_{t-m})(1 + \varepsilon_t)\\
+\ell_t &= \ell_{t-1} + \alpha(\ell_{t-1}+s_{t-m})\varepsilon_t\\
+s_t &= s_{t-m} + \gamma(\ell_{t-1}+s_{t-m}) \varepsilon_t.
+\end{align\*}\]
+
+The parameter estimates are \(\hat\alpha= 0.3484\), and \(\hat\gamma=0.0001\). The output also returns the estimates for the initial states \(\ell_0\), \(s_{0}\), \(s_{-1}\), \(s_{-2}\) and \(s_{-3}.\) Compare these with the values obtained for the Holt-Winters method with additive seasonality presented in Table [8.3](https://otexts.com/fpp3/holt-winters.html#tab:tab75).
+
+Figure [8.10](https://otexts.com/fpp3/ets-estimation.html#fig:MNAstates) shows the states over time, while Figure [8.12](https://otexts.com/fpp3/ets-forecasting.html#fig:MNAforecasts) shows point forecasts and prediction intervals generated from the model. The small values of \(\gamma\) indicate that the seasonal states change very little over time.
+
+```
+components(fit) |>
+  autoplot() +
+  labs(title = "ETS(M,N,A) components")
+```
+
+![Graphical representation of the estimated states over time.](https://otexts.com/fpp3/fpp_files/figure-html/MNAstates-1.png)
+
+Figure 8.10: Graphical representation of the estimated states over time.
+
+Because this model has multiplicative errors, the innovation residuals are not equivalent to the regular residuals (i.e., the one-step training errors). The innovation residuals are given by \(\hat{\varepsilon}_t\), while the regular residuals are defined as \(y_t - \hat{y}_{t|t-1}\). We can obtain both using the `augment()` function. They are plotted in Figure [8.11](https://otexts.com/fpp3/ets-estimation.html#fig:MNAresiduals).
+
+![Residuals and one-step forecast errors from the ETS(M,N,A) model.](https://otexts.com/fpp3/fpp_files/figure-html/MNAresiduals-1.png)
+
+Figure 8.11: Residuals and one-step forecast errors from the ETS(M,N,A) model.
+
+### Bibliography
+
+Hyndman, R. J., Koehler, A. B., Ord, J. K., & Snyder, R. D. (2008). *Forecasting with exponential smoothing: The state space approach*. Springer-Verlag.
+
+## 8.7 Forecasting with ETS models
+
+Point forecasts can be obtained from the models by iterating the equations for \(t=T+1,\dots,T+h\) and setting all \(\varepsilon_t=0\) for \(t>T\).
+
+For example, for model ETS(M,A,N), \(y_{T+1} = (\ell_T + b_T )(1+ \varepsilon_{T+1}).\) Therefore \(\hat{y}_{T+1|T}=\ell_{T}+b_{T}.\) Similarly,
+\[\begin{align\*}
+y_{T+2} &= (\ell_{T+1} + b_{T+1})(1 + \varepsilon_{T+2})\\
+&= \left[
+(\ell_T + b_T) (1+ \alpha\varepsilon_{T+1}) +
+b_T + \beta (\ell_T + b_T)\varepsilon_{T+1}
+\right]
+(1 + \varepsilon_{T+2}).
+\end{align\*}\]
+Therefore, \(\hat{y}_{T+2|T}= \ell_{T}+2b_{T},\) and so on. These forecasts are identical to the forecasts from Holt’s linear method, and also to those from model ETS(A,A,N). Thus, the point forecasts obtained from the method and from the two models that underlie the method are identical (assuming that the same parameter values are used). ETS point forecasts constructed in this way are equal to the means of the forecast distributions, except for the models with multiplicative seasonality ([Hyndman et al., 2008](#ref-expsmooth08)).
+
+To obtain forecasts from an ETS model, we use the `forecast()` function from the `fable` package. This function will always return the means of the forecast distribution, even when they differ from these traditional point forecasts.
+
+```
+fit |>
+  forecast(h = 8) |>
+  autoplot(aus_holidays)+
+  labs(title="Australian domestic tourism",
+       y="Overnight trips (millions)")
+```
+
+![Forecasting Australian domestic overnight trips using an ETS(M,N,A) model.](https://otexts.com/fpp3/fpp_files/figure-html/MNAforecasts-1.png)
+
+Figure 8.12: Forecasting Australian domestic overnight trips using an ETS(M,N,A) model.
+
+### Prediction intervals
+
+A big advantage of the statistical models is that prediction intervals can also be generated — something that cannot be done using the point forecasting methods alone. The prediction intervals will differ between models with additive and multiplicative methods.
+
+For most ETS models, a prediction interval can be written as
+\[
+\hat{y}_{T+h|T} \pm c \sigma_h
+\]
+where \(c\) depends on the coverage probability, and \(\sigma_h^2\) is the forecast variance. Values for \(c\) were given in Table [5.1](https://otexts.com/fpp3/prediction-intervals.html#tab:pcmultipliers). For ETS models, formulas for \(\sigma_h^2\) can be complicated; the details are given in Chapter 6 of Hyndman et al. ([2008](#ref-expsmooth08)). In Table [8.8](https://otexts.com/fpp3/ets-forecasting.html#tab:pitable) we give the formulas for the additive ETS models, which are the simplest.
+
+Table 8.8: Forecast variance expressions for each additive state space model, where \(\sigma^2\) is the residual variance, \(m\) is the seasonal period, and \(k\) is the integer part of \((h-1) /m\) (i.e., the number of complete years in the forecast period prior to time \(T+h\)).
+
+| Model | Forecast variance: \(\sigma_h^2\) |
+| --- | --- |
+| (A,N,N) | \(\sigma_h^2 = \sigma^2\big[1 + \alpha^2(h-1)\big]\) |
+| (A,A,N) | \(\sigma_h^2 = \sigma^2\Big[1 + (h-1)\big\{\alpha^2 + \alpha\beta h + \frac16\beta^2h(2h-1)\big\}\Big]\) |
+| (A,A\(_d\),N) | \(\sigma_h^2 = \sigma^2\biggl[1 + \alpha^2(h-1) + \frac{\beta\phi h}{(1-\phi)^2} \left\{2\alpha(1-\phi) +\beta\phi\right\}\) |
+|  | \(\mbox{} - \frac{\beta\phi(1-\phi^h)}{(1-\phi)^2(1-\phi^2)} \left\{ 2\alpha(1-\phi^2)+ \beta\phi(1+2\phi-\phi^h)\right\}\biggr]\) |
+| (A,N,A) | \(\sigma_h^2 = \sigma^2\Big[1 + \alpha^2(h-1) + \gamma k(2\alpha+\gamma)\Big]\) |
+| (A,A,A) | \(\sigma_h^2 = \sigma^2\Big[1 + (h-1)\big\{\alpha^2 + \alpha\beta h + \frac16\beta^2h(2h-1)\big\}\) |
+|  | \(\mbox{} + \gamma k \big\{2\alpha+ \gamma + \beta m (k+1)\big\} \Big]\) |
+| (A,A\(_d\),A) | \(\sigma_h^2 = \sigma^2\biggl[1 + \alpha^2(h-1) + \gamma k(2\alpha+\gamma)\) |
+|  | \(\mbox{} +\frac{\beta\phi h}{(1-\phi)^2} \left\{2\alpha(1-\phi) + \beta\phi \right\}\) |
+|  | \(\mbox{} - \frac{\beta\phi(1-\phi^h)}{(1-\phi)^2(1-\phi^2)} \left\{ 2\alpha(1-\phi^2)+ \beta\phi(1+2\phi-\phi^h)\right\}\) |
+|  | \(\mbox{} + \frac{2\beta\gamma\phi}{(1-\phi)(1-\phi^m)}\left\{k(1-\phi^m) - \phi^m(1-\phi^{mk})\right\}\biggr]\) |
+
+For a few ETS models, there are no known formulas for prediction intervals. In these cases, the `forecast()` function uses simulated future sample paths and computes prediction intervals from the percentiles of these simulated future paths.
+
+### Bibliography
+
+Hyndman, R. J., Koehler, A. B., Ord, J. K., & Snyder, R. D. (2008). *Forecasting with exponential smoothing: The state space approach*. Springer-Verlag.
+
+## 8.8 Exercises
+
+1. Consider the number of pigs slaughtered in Victoria, available in the `aus_livestock` dataset.
+
+   1. Use the `ETS()` function to estimate the equivalent model for simple exponential smoothing. Find the optimal values of \(\alpha\) and \(\ell_0\), and generate forecasts for the next four months.
+   2. Compute a 95% prediction interval for the first forecast using \(\hat{y} \pm 1.96s\) where \(s\) is the standard deviation of the residuals. Compare your interval with the interval produced by R.
+2. Write your own function to implement simple exponential smoothing. The function should take arguments `y` (the time series), `alpha` (the smoothing parameter \(\alpha\)) and `level` (the initial level \(\ell_0\)). It should return the forecast of the next observation in the series. Does it give the same forecast as `ETS()`?
+3. Modify your function from the previous exercise to return the sum of squared errors rather than the forecast of the next observation. Then use the `optim()` function to find the optimal values of \(\alpha\) and \(\ell_0\). Do you get the same values as the `ETS()` function?
+4. Combine your previous two functions to produce a function that both finds the optimal values of \(\alpha\) and \(\ell_0\), and produces a forecast of the next observation in the series.
+5. Data set `global_economy` contains the annual Exports from many countries. Select one country to analyse.
+
+   1. Plot the Exports series and discuss the main features of the data.
+   2. Use an ETS(A,N,N) model to forecast the series, and plot the forecasts.
+   3. Compute the RMSE values for the training data.
+   4. Compare the results to those from an ETS(A,A,N) model. (Remember that the trended model is using one more parameter than the simpler model.) Discuss the merits of the two forecasting methods for this data set.
+   5. Compare the forecasts from both methods. Which do you think is best?
+   6. Calculate a 95% prediction interval for the first forecast for each model, using the RMSE values and assuming normal errors. Compare your intervals with those produced using R.
+6. Forecast the Chinese GDP from the `global_economy` data set using an ETS model. Experiment with the various options in the `ETS()` function to see how much the forecasts change with damped trend, or with a Box-Cox transformation. Try to develop an intuition of what each is doing to the forecasts.
+
+   [Hint: use a relatively large value of `h` when forecasting, so you can clearly see the differences between the various options when plotting the forecasts.]
+7. Find an ETS model for the Gas data from `aus_production` and forecast the next few years. Why is multiplicative seasonality necessary here? Experiment with making the trend damped. Does it improve the forecasts?
+8. Recall your retail time series data (from Exercise 7 in Section [2.10](https://otexts.com/fpp3/graphics-exercises.html#graphics-exercises)).
+
+   1. Why is multiplicative seasonality necessary for this series?
+   2. Apply Holt-Winters’ multiplicative method to the data. Experiment with making the trend damped.
+   3. Compare the RMSE of the one-step forecasts from the two methods. Which do you prefer?
+   4. Check that the residuals from the best method look like white noise.
+   5. Now find the test set RMSE, while training the model to the end of 2010. Can you beat the seasonal naïve approach from Exercise 7 in Section [5.11](https://otexts.com/fpp3/toolbox-exercises.html#toolbox-exercises)?
+9. For the same retail data, try an STL decomposition applied to the Box-Cox transformed series, followed by ETS on the seasonally adjusted data. How does that compare with your best previous forecasts on the test set?
+10. Compute the total domestic overnight trips across Australia from the `tourism` dataset.
+
+    1. Plot the data and describe the main features of the series.
+    2. Decompose the series using STL and obtain the seasonally adjusted data.
+    3. Forecast the next two years of the series using an additive damped trend method applied to the seasonally adjusted data. (This can be specified using `decomposition_model()`.)
+    4. Forecast the next two years of the series using an appropriate model for Holt’s linear method applied to the seasonally adjusted data (as before but without damped trend).
+    5. Now use `ETS()` to choose a seasonal model for the data.
+    6. Compare the RMSE of the ETS model with the RMSE of the models you obtained using STL decompositions. Which gives the better in-sample fits?
+    7. Compare the forecasts from the three approaches? Which seems most reasonable?
+    8. Check the residuals of your preferred model.
+11. For this exercise use the quarterly number of arrivals to Australia from New Zealand, 1981 Q1 – 2012 Q3, from data set `aus_arrivals`.
+
+    1. Make a time plot of your data and describe the main features of the series.
+    2. Create a training set that withholds the last two years of available data. Forecast the test set using an appropriate model for Holt-Winters’ multiplicative method.
+    3. Why is multiplicative seasonality necessary here?
+    4. Forecast the two-year test set using each of the following methods:
+       * an ETS model;
+       * an additive ETS model applied to a log transformed series;
+       * a seasonal naïve method;
+       * an STL decomposition applied to the log transformed data followed by an ETS model applied to the seasonally adjusted (transformed) data.
+    5. Which method gives the best forecasts? Does it pass the residual tests?
+    6. Compare the same four methods using time series cross-validation instead of using a training and test set. Do you come to the same conclusions?
+12. 1. Apply cross-validation techniques to produce 1 year ahead ETS and seasonal naïve forecasts for Portland cement production (from `aus_production`). Use a stretching data window with initial size of 5 years, and increment the window by one observation.
+    2. Compute the MSE of the resulting \(4\)-step-ahead errors. Comment on which forecasts are more accurate. Is this what you expected?
+13. Compare `ETS()`, `SNAIVE()` and `decomposition_model(STL, ???)` on the following five time series. You might need to use a Box-Cox transformation for the STL decomposition forecasts. Use a test set of three years to decide what gives the best forecasts.
+
+    * Beer and bricks production from `aus_production`.
+    * Cost of drug subsidies for diabetes (`ATC2 == "A10"`) and corticosteroids (`ATC2 == "H02"`) from `PBS`.
+    * Total food retailing turnover for Australia from `aus_retail`.
+14. 1. Use `ETS()` to select an appropriate model for the following series: total number of trips across Australia using `tourism`, the closing prices for the four stocks in `gafa_stock`, and the lynx series in `pelt`. Does it always give good forecasts?
+    2. Find an example where it does not work well. Can you figure out why?
+15. Show that the point forecasts from an ETS(M,A,M) model are the same as those obtained using Holt-Winters’ multiplicative method.
+16. Show that the forecast variance for an ETS(A,N,N) model is given by
+    \[
+    \sigma^2\left[1+\alpha^2(h-1)\right].
+    \]
+17. Write down 95% prediction intervals for an ETS(A,N,N) model as a function of \(\ell_T\), \(\alpha\), \(h\) and \(\sigma\), assuming normally distributed errors.
+
+## 8.9 Further reading
+
+* Two articles by Ev Gardner ([Gardner, 1985](#ref-Gar1985), [2006](#ref-Gar2006)) provide a great overview of the history of exponential smoothing, and its many variations.
+* A full book treatment of the subject providing the mathematical details is given by Hyndman et al. ([2008](#ref-expsmooth08)).
+
+### Bibliography
+
+Gardner, E. S. (1985). Exponential smoothing: The state of the art. *Journal of Forecasting*, *4*(1), 1–28.
+
+Gardner, E. S. (2006). Exponential smoothing: The state of the art — Part II. *International Journal of Forecasting*, *22*, 637–666.
+
+Hyndman, R. J., Koehler, A. B., Ord, J. K., & Snyder, R. D. (2008). *Forecasting with exponential smoothing: The state space approach*. Springer-Verlag.

docs/evidence/fpp3/10-dynamic-regression.md

@@ -0,0 +1,651 @@
+Source: https://otexts.com/fpp3/dynamic.html (chapter dynamic, 9 section pages merged)
+Title: Forecasting: Principles and Practice 3rd ed - 10-dynamic-regression
+Fetched-via: urllib + markitdown (content div.page-inner section.normal), 2026-07-26
+Fetch-status: full content; images/links point to absolute otexts.com URLs
+
+# Chapter 10 Dynamic regression models
+
+The time series models in the previous two chapters allow for the inclusion of information from past observations of a series, but not for the inclusion of other information that may also be relevant. For example, the effects of holidays, competitor activity, changes in the law, the wider economy, or other external variables, may explain some of the historical variation and may lead to more accurate forecasts. On the other hand, the regression models in Chapter [7](https://otexts.com/fpp3/regression.html#regression) allow for the inclusion of a lot of relevant information from predictor variables, but do not allow for the subtle time series dynamics that can be handled with ARIMA models. In this chapter, we consider how to extend ARIMA models in order to allow other information to be included in the models.
+
+In Chapter [7](https://otexts.com/fpp3/regression.html#regression) we considered regression models of the form
+\[
+y_t = \beta_0 + \beta_1 x_{1,t} + \dots + \beta_k x_{k,t} + \varepsilon_t,
+\]
+where \(y_t\) is a linear function of the \(k\) predictor variables (\(x_{1,t},\dots,x_{k,t}\)), and \(\varepsilon_t\) is usually assumed to be an uncorrelated error term (i.e., it is white noise). We considered tests such as the Ljung-Box test for assessing whether the resulting residuals were significantly correlated.
+
+In this chapter, we will allow the errors from a regression to contain autocorrelation. To emphasise this change in perspective, we will replace \(\varepsilon_t\) with \(\eta_t\) in the equation. The error series \(\eta_t\) is assumed to follow an ARIMA model. For example, if \(\eta_t\) follows an ARIMA(1,1,1) model, we can write
+\[\begin{align\*}
+y_t &= \beta_0 + \beta_1 x_{1,t} + \dots + \beta_k x_{k,t} + \eta_t,\\
+& (1-\phi_1B)(1-B)\eta_t = (1+\theta_1B)\varepsilon_t,
+\end{align\*}\]
+where \(\varepsilon_t\) is a white noise series.
+
+Notice that the model has two error terms here — the error from the regression model, which we denote by \(\eta_t\), and the error from the ARIMA model, which we denote by \(\varepsilon_t\). Only the ARIMA model errors are assumed to be white noise.
+
+## 10.1 Estimation
+
+When we estimate the parameters from the model, we need to minimise the sum of squared \(\varepsilon_t\) values. If we minimise the sum of squared \(\eta_t\) values instead (which is what would happen if we estimated the regression model ignoring the autocorrelations in the errors), then several problems arise.
+
+1. The estimated coefficients \(\hat{\beta}_0,\dots,\hat{\beta}_k\) are no longer the best estimates, as some information has been ignored in the calculation;
+2. Any statistical tests associated with the model (e.g., t-tests on the coefficients) will be incorrect.
+3. The AICc values of the fitted models are no longer a good guide as to which is the best model for forecasting.
+4. In most cases, the \(p\)-values associated with the coefficients will be too small, and so some predictor variables will appear to be important when they are not. This is known as “spurious regression”.
+
+Minimising the sum of squared \(\varepsilon_t\) values avoids these problems. Alternatively, maximum likelihood estimation can be used; this will give similar estimates of the coefficients.
+
+An important consideration when estimating a regression with ARMA errors is that all of the variables in the model must first be stationary. Thus, we first have to check that \(y_t\) and all of the predictors \((x_{1,t},\dots,x_{k,t})\) appear to be stationary. If we estimate the model when any of these are non-stationary, the estimated coefficients will not be consistent estimates (and therefore may not be meaningful). One exception to this is the case where non-stationary variables are co-integrated. If there exists a linear combination of the non-stationary \(y_t\) and the predictors that is stationary, then the estimated coefficients will be consistent.[21](#fn21)
+
+We therefore first difference the non-stationary variables in the model. It is often desirable to maintain the form of the relationship between \(y_t\) and the predictors, and consequently it is common to difference all of the variables if any of them need differencing. The resulting model is then called a “model in differences”, as distinct from a “model in levels”, which is what is obtained when the original data are used without differencing.
+
+If all of the variables in the model are stationary, then we only need to consider an ARMA process for the errors. It is easy to see that a regression model with ARIMA errors is equivalent to a regression model in differences with ARMA errors. For example, if the above regression model with ARIMA(1,1,1) errors is differenced we obtain the model
+\[\begin{align\*}
+y'_t &= \beta_1 x'_{1,t} + \dots + \beta_k x'_{k,t} + \eta'_t,\\
+& (1-\phi_1B)\eta'_t = (1+\theta_1B)\varepsilon_t,
+\end{align\*}\]
+where \(y'_t=y_t-y_{t-1}\), \(x'_{t,i}=x_{t,i}-x_{t-1,i}\) and \(\eta'_t=\eta_t-\eta_{t-1}\), which is a regression model in differences with ARMA errors.
+
+### Bibliography
+
+Harris, R., & Sollis, R. (2003). *Applied time series modelling and forecasting*. John Wiley & Sons.
+
+---
+
+21. Forecasting with cointegrated models is discussed by Harris & Sollis ([2003](#ref-Harris03)).[↩︎](https://otexts.com/fpp3/estimation.html#fnref21)
+
+## 10.2 Regression with ARIMA errors using `fable`
+
+The function `ARIMA()` will fit a regression model with ARIMA errors if exogenous regressors are included in the formula. As introduced in Section [9.5](https://otexts.com/fpp3/non-seasonal-arima.html#non-seasonal-arima), the `pdq()` special specifies the order of the ARIMA error model. If differencing is specified, then the differencing is applied to all variables in the regression model before the model is estimated. For example, the command
+
+```
+ARIMA(y ~ x + pdq(1,1,0))
+```
+
+will fit the model \(y_t' = \beta_1 x'_t + \eta'_t\), where \(\eta'_t = \phi_1 \eta'_{t-1} + \varepsilon_t\) is an AR(1) error. This is equivalent to the model
+\[
+y_t = \beta_0 + \beta_1 x_t + \eta_t,
+\]
+where \(\eta_t\) is an ARIMA(1,1,0) error. Notice that the constant term disappears due to the differencing. To include a constant in the differenced model, we would add `1` to the model formula.
+
+The `ARIMA()` function can also be used to select the best ARIMA model for the errors. This is done by not specifying the `pdq()` special. Whether differencing is required is determined by applying a KPSS test to the residuals from the regression model estimated using ordinary least squares. If differencing is required, then all variables are differenced and the model re-estimated using maximum likelihood estimation. The final model will be expressed in terms of the original variables, even if it has been estimated using differenced variables.
+
+The AICc is calculated for the final model, and this value can be used to determine the best predictors. That is, the procedure should be repeated for all subsets of predictors to be considered, and the model with the lowest AICc value selected.
+
+### Example: US Personal Consumption and Income
+
+Figure [10.1](https://otexts.com/fpp3/regarima.html#fig:usconsump) shows the quarterly changes in personal consumption expenditure and personal disposable income from 1970 to 2019 Q2. We would like to forecast changes in expenditure based on changes in income. A change in income does not necessarily translate to an instant change in consumption (e.g., after the loss of a job, it may take a few months for expenses to be reduced to allow for the new circumstances). However, we will ignore this complexity in this example and try to measure the instantaneous effect of the average change of income on the average change of consumption expenditure.
+
+```
+us_change |>
+  pivot_longer(c(Consumption, Income),
+               names_to = "var", values_to = "value") |>
+  ggplot(aes(x = Quarter, y = value)) +
+  geom_line() +
+  facet_grid(vars(var), scales = "free_y") +
+  labs(title = "US consumption and personal income",
+       y = "Quarterly % change")
+```
+
+![Percentage changes in quarterly personal consumption expenditure and personal disposable income for the USA, 1970 Q1 to 2019 Q2.](https://otexts.com/fpp3/fpp_files/figure-html/usconsump-1.png)
+
+Figure 10.1: Percentage changes in quarterly personal consumption expenditure and personal disposable income for the USA, 1970 Q1 to 2019 Q2.
+
+```
+fit <- us_change |>
+  model(ARIMA(Consumption ~ Income))
+report(fit)
+#> Series: Consumption
+#> Model: LM w/ ARIMA(1,0,2) errors
+#>
+#> Coefficients:
+#>          ar1      ma1     ma2  Income  intercept
+#>       0.7070  -0.6172  0.2066  0.1976     0.5949
+#> s.e.  0.1068   0.1218  0.0741  0.0462     0.0850
+#>
+#> sigma^2 estimated as 0.3113:  log likelihood=-163
+#> AIC=338.1   AICc=338.5   BIC=357.8
+```
+
+The data are clearly already stationary (as we are considering percentage changes rather than raw expenditure and income), so there is no need for any differencing. The fitted model is
+\[\begin{align\*}
+y_t &= 0.595 +
+0.198 x_t + \eta_t, \\
+\eta_t &= 0.707 \eta_{t-1} + \varepsilon_t
+-0.617 \varepsilon_{t-1} +
+0.207 \varepsilon_{t-2},\\
+\varepsilon_t &\sim \text{NID}(0,0.311).
+\end{align\*}\]
+
+We can recover estimates of both the \(\eta_t\) and \(\varepsilon_t\) series using the `residuals()` function.
+
+```
+bind_rows(
+    `Regression residuals` =
+        as_tibble(residuals(fit, type = "regression")),
+    `ARIMA residuals` =
+        as_tibble(residuals(fit, type = "innovation")),
+    .id = "type"
+  ) |>
+  mutate(
+    type = factor(type, levels=c(
+      "Regression residuals", "ARIMA residuals"))
+  ) |>
+  ggplot(aes(x = Quarter, y = .resid)) +
+  geom_line() +
+  facet_grid(vars(type))
+```
+
+![Regression residuals ($\eta_t$) and ARIMA residuals ($\varepsilon_t$) from the fitted model.](https://otexts.com/fpp3/fpp_files/figure-html/usconsumpres-1.png)
+
+Figure 10.2: Regression residuals (\(\eta_t\)) and ARIMA residuals (\(\varepsilon_t\)) from the fitted model.
+
+It is the ARIMA estimated errors (the innovation residuals) that should resemble a white noise series.
+
+```
+fit |> gg_tsresiduals()
+```
+
+![The innovation residuals (i.e., the estimated ARIMA errors) are not significantly different from white noise.](https://otexts.com/fpp3/fpp_files/figure-html/usconsumpres2-1.png)
+
+Figure 10.3: The innovation residuals (i.e., the estimated ARIMA errors) are not significantly different from white noise.
+
+```
+augment(fit) |>
+  features(.innov, ljung_box, dof = 3, lag = 8)
+#> # A tibble: 1 × 3
+#>   .model                      lb_stat lb_pvalue
+#>   <chr>                         <dbl>     <dbl>
+#> 1 ARIMA(Consumption ~ Income)    5.21     0.391
+```
+
+## 10.3 Forecasting
+
+To forecast using a regression model with ARIMA errors, we need to forecast the regression part of the model and the ARIMA part of the model, and combine the results. As with ordinary regression models, in order to obtain forecasts we first need to forecast the predictors. When the predictors are known into the future (e.g., calendar-related variables such as time, day-of-week, etc.), this is straightforward. But when the predictors are themselves unknown, we must either model them separately, or use assumed future values for each predictor.
+
+### Example: US Personal Consumption and Income
+
+We will calculate forecasts for the next eight quarters assuming that the future percentage changes in personal disposable income will be equal to the mean percentage change from the last forty years.
+
+```
+us_change_future <- new_data(us_change, 8) |>
+  mutate(Income = mean(us_change$Income))
+forecast(fit, new_data = us_change_future) |>
+  autoplot(us_change) +
+  labs(y = "Percentage change")
+```
+
+![Forecasts obtained from regressing the percentage change in consumption expenditure on the percentage change in disposable income, with an ARIMA(1,0,2) error model.](https://otexts.com/fpp3/fpp_files/figure-html/usconsump3-1.png)
+
+Figure 10.4: Forecasts obtained from regressing the percentage change in consumption expenditure on the percentage change in disposable income, with an ARIMA(1,0,2) error model.
+
+The prediction intervals for this model are narrower than if we had fitted an ARIMA model without covariates, because we are now able to explain some of the variation in the data using the income predictor.
+
+It is important to realise that the prediction intervals from regression models (with or without ARIMA errors) do not take into account the uncertainty in the forecasts of the predictors. So they should be interpreted as being conditional on the assumed (or estimated) future values of the predictor variables.
+
+### Example: Forecasting electricity demand
+
+Daily electricity demand can be modelled as a function of temperature. As can be observed on an electricity bill, more electricity is used on cold days due to heating and hot days due to air conditioning. The higher demand on cold and hot days is reflected in the U-shape of Figure [10.5](https://otexts.com/fpp3/forecasting.html#fig:elecscatter), where daily demand is plotted versus daily maximum temperature.
+
+```
+vic_elec_daily <- vic_elec |>
+  filter(year(Time) == 2014) |>
+  index_by(Date = date(Time)) |>
+  summarise(
+    Demand = sum(Demand) / 1e3,
+    Temperature = max(Temperature),
+    Holiday = any(Holiday)
+  ) |>
+  mutate(Day_Type = case_when(
+    Holiday ~ "Holiday",
+    wday(Date) %in% 2:6 ~ "Weekday",
+    TRUE ~ "Weekend"
+  ))
+
+vic_elec_daily |>
+  ggplot(aes(x = Temperature, y = Demand, colour = Day_Type)) +
+  geom_point() +
+  labs(y = "Electricity demand (GW)",
+       x = "Maximum daily temperature")
+```
+
+![Daily electricity demand versus maximum daily temperature for the state of Victoria in Australia for 2014.](https://otexts.com/fpp3/fpp_files/figure-html/elecscatter-1.png)
+
+Figure 10.5: Daily electricity demand versus maximum daily temperature for the state of Victoria in Australia for 2014.
+
+The data stored as `vic_elec_daily` includes total daily demand, daily maximum temperatures, and an indicator variable for if that day is a public holiday. Figure [10.6](https://otexts.com/fpp3/forecasting.html#fig:electime) shows the time series of both daily demand and daily maximum temperatures. The plots highlight the need for both a non-linear and a dynamic model.
+
+```
+vic_elec_daily |>
+  pivot_longer(c(Demand, Temperature)) |>
+  ggplot(aes(x = Date, y = value)) +
+  geom_line() +
+  facet_grid(name ~ ., scales = "free_y") + ylab("")
+```
+
+![Daily electricity demand and maximum daily temperature for the state of Victoria in Australia for 2014.](https://otexts.com/fpp3/fpp_files/figure-html/electime-1.png)
+
+Figure 10.6: Daily electricity demand and maximum daily temperature for the state of Victoria in Australia for 2014.
+
+In this example, we fit a quadratic regression model with ARMA errors using the `ARIMA()` function. The model also includes an indicator variable for if the day was a working day or not.
+
+```
+fit <- vic_elec_daily |>
+  model(ARIMA(Demand ~ Temperature + I(Temperature^2) +
+                (Day_Type == "Weekday")))
+fit |> gg_tsresiduals()
+```
+
+![Residuals diagnostics for a dynamic regression model for daily electricity demand with workday and quadratic temperature effects.](https://otexts.com/fpp3/fpp_files/figure-html/elecdailyfit-1.png)
+
+Figure 10.7: Residuals diagnostics for a dynamic regression model for daily electricity demand with workday and quadratic temperature effects.
+
+The fitted model has an ARIMA(2,1,2)(2,0,0)[7] error, so there are 6 AR and MA coefficients.
+
+```
+augment(fit) |>
+  features(.innov, ljung_box, dof = 6, lag = 14)
+#> # A tibble: 1 × 3
+#>   .model                                                     lb_stat lb_pvalue
+#>   <chr>                                                        <dbl>     <dbl>
+#> 1 "ARIMA(Demand ~ Temperature + I(Temperature^2) + (Day_Typ…    28.4  0.000404
+```
+
+There is clear heteroscedasticity in the residuals, with higher variance in January and February, and lower variance in May. The model also has some significant autocorrelation in the residuals, and the histogram of the residuals shows long tails. All of these issues with the residuals may affect the coverage of the prediction intervals, but the point forecasts should still be ok.
+
+Using the estimated model we forecast 14 days ahead starting from Thursday 1 January 2015 (a non-work-day being a public holiday for New Years Day). In this case, we could obtain weather forecasts from the weather bureau for the next 14 days. But for the sake of illustration, we will use scenario based forecasting (as introduced in Section [7.6](https://otexts.com/fpp3/forecasting-regression.html#forecasting-regression)) where we set the temperature for the next 14 days to a constant 26 degrees.
+
+```
+vic_elec_future <- new_data(vic_elec_daily, 14) |>
+  mutate(
+    Temperature = 26,
+    Holiday = c(TRUE, rep(FALSE, 13)),
+    Day_Type = case_when(
+      Holiday ~ "Holiday",
+      wday(Date) %in% 2:6 ~ "Weekday",
+      TRUE ~ "Weekend"
+    )
+  )
+forecast(fit, vic_elec_future) |>
+  autoplot(vic_elec_daily) +
+  labs(title="Daily electricity demand: Victoria",
+       y="GW")
+```
+
+![Forecasts from the dynamic regression model for daily electricity demand. All future temperatures have been set to 26 degrees, and the working day dummy variable has been set to known future values.](https://otexts.com/fpp3/fpp_files/figure-html/elecdailyfc-1.png)
+
+Figure 10.8: Forecasts from the dynamic regression model for daily electricity demand. All future temperatures have been set to 26 degrees, and the working day dummy variable has been set to known future values.
+
+The point forecasts look reasonable for the first two weeks of 2015. The slow down in electricity demand at the end of 2014 (due to many people taking summer vacations) has caused the forecasts for the next two weeks to show similarly low demand values.
+
+## 10.4 Stochastic and deterministic trends
+
+There are two different ways of modelling a linear trend. A *deterministic trend* is obtained using the regression model
+\[
+y_t = \beta_0 + \beta_1 t + \eta_t,
+\]
+where \(\eta_t\) is an ARMA process. A *stochastic trend* is obtained using the model
+\[
+y_t = \beta_0 + \beta_1 t + \eta_t,
+\]
+where \(\eta_t\) is an ARIMA process with \(d=1\). In the latter case, we can difference both sides so that \(y_t' = \beta_1 + \eta_t'\), where \(\eta_t'\) is an ARMA process. In other words,
+\[
+y_t = y_{t-1} + \beta_1 + \eta_t'.
+\]
+This is similar to a random walk with drift (introduced in Section [9.1](https://otexts.com/fpp3/stationarity.html#stationarity)), but here the error term is an ARMA process rather than simply white noise.
+
+Although these models appear quite similar (they only differ in the number of differences that need to be applied to \(\eta_t\)), their forecasting characteristics are quite different.
+
+### Example: Air transport passengers Australia
+
+```
+aus_airpassengers |>
+  autoplot(Passengers) +
+  labs(y = "Passengers (millions)",
+       title = "Total annual air passengers")
+```
+
+![Total annual passengers (in millions) for Australian air carriers, 1970--2016.](https://otexts.com/fpp3/fpp_files/figure-html/austa-1.png)
+
+Figure 10.9: Total annual passengers (in millions) for Australian air carriers, 1970–2016.
+
+Figure [10.9](https://otexts.com/fpp3/stochastic-and-deterministic-trends.html#fig:austa) shows the total number of passengers for Australian air carriers each year from 1970 to 2016. We will fit both a deterministic and a stochastic trend model to these data.
+
+The deterministic trend model is obtained as follows:
+
+```
+fit_deterministic <- aus_airpassengers |>
+  model(deterministic = ARIMA(Passengers ~ 1 + trend() +
+                                pdq(d = 0)))
+report(fit_deterministic)
+#> Series: Passengers
+#> Model: LM w/ ARIMA(1,0,0) errors
+#>
+#> Coefficients:
+#>          ar1  trend()  intercept
+#>       0.9564   1.4151     0.9014
+#> s.e.  0.0362   0.1972     7.0751
+#>
+#> sigma^2 estimated as 4.343:  log likelihood=-100.88
+#> AIC=209.77   AICc=210.72   BIC=217.17
+```
+
+This model can be written as
+\[\begin{align\*}
+y_t &= 0.901 + 1.415 t + \eta_t \\
+\eta_t &= 0.956 \eta_{t-1} + \varepsilon_t\\
+\varepsilon_t &\sim \text{NID}(0,4.343).
+\end{align\*}\]
+
+The estimated growth in visitor numbers is 1.42 million people per year.
+
+Alternatively, the stochastic trend model can be estimated.
+
+```
+fit_stochastic <- aus_airpassengers |>
+  model(stochastic = ARIMA(Passengers ~ pdq(d = 1)))
+report(fit_stochastic)
+#> Series: Passengers
+#> Model: ARIMA(0,1,0) w/ drift
+#>
+#> Coefficients:
+#>       constant
+#>         1.4191
+#> s.e.    0.3014
+#>
+#> sigma^2 estimated as 4.271:  log likelihood=-98.16
+#> AIC=200.31   AICc=200.59   BIC=203.97
+```
+
+This model can be written as \(y_t-y_{t-1} = 1.419 + \varepsilon_t\), or equivalently
+\[\begin{align\*}
+y_t &= y_0 + 1.419 t + \eta_t \\
+\eta_t &= \eta_{t-1} + \varepsilon_{t}\\
+\varepsilon_t &\sim \text{NID}(0,4.271).
+\end{align\*}\]
+
+In this case, the estimated growth in visitor numbers is also 1.42 million people per year. Although the growth estimates are similar, the prediction intervals are not, as Figure [10.10](https://otexts.com/fpp3/stochastic-and-deterministic-trends.html#fig:austaf) shows. In particular, stochastic trends have much wider prediction intervals because the errors are non-stationary.
+
+```
+aus_airpassengers |>
+  autoplot(Passengers) +
+  autolayer(fit_stochastic |> forecast(h = 20),
+    colour = "#0072B2", level = 95) +
+  autolayer(fit_deterministic |> forecast(h = 20),
+    colour = "#D55E00", alpha = 0.65, level = 95) +
+  labs(y = "Air passengers (millions)",
+       title = "Forecasts from trend models")
+```
+
+![Forecasts of annual passengers for Australian air carriers using a deterministic trend model (orange) and a stochastic trend model (blue).](https://otexts.com/fpp3/fpp_files/figure-html/austaf-1.png)
+
+Figure 10.10: Forecasts of annual passengers for Australian air carriers using a deterministic trend model (orange) and a stochastic trend model (blue).
+
+There is an implicit assumption with deterministic trends that the slope of the trend is not going to change over time. On the other hand, stochastic trends can change, and the estimated growth is only assumed to be the average growth over the historical period, not necessarily the rate of growth that will be observed into the future. Consequently, it is safer to forecast with stochastic trends, especially for longer forecast horizons, as the prediction intervals allow for greater uncertainty in future growth.
+
+## 10.5 Dynamic harmonic regression
+
+When there are long seasonal periods, a dynamic regression with Fourier terms is often better than other models we have considered in this book.[22](#fn22)
+
+For example, daily data can have annual seasonality of length 365, weekly data has seasonal period of approximately 52, while half-hourly data can have several seasonal periods, the shortest of which is the daily pattern of period 48.
+
+Seasonal versions of ARIMA and ETS models are designed for shorter periods such as 12 for monthly data or 4 for quarterly data. The `ETS()` model restricts seasonality to be a maximum period of 24 to allow hourly data but not data with a larger seasonal period. The problem is that there are \(m-1\) parameters to be estimated for the initial seasonal states where \(m\) is the seasonal period. So for large \(m\), the estimation becomes almost impossible.
+
+The `ARIMA()` function will allow a seasonal period up to \(m=350\), but in practice will usually run out of memory whenever the seasonal period is more than about 200. In any case, seasonal differencing of high order does not make a lot of sense — for daily data it involves comparing what happened today with what happened exactly a year ago and there is no constraint that the seasonal pattern is smooth.
+
+So for such time series, we prefer a harmonic regression approach where the seasonal pattern is modelled using Fourier terms with short-term time series dynamics handled by an ARMA error.
+
+The advantages of this approach are:
+
+* it allows any length seasonality;
+* for data with more than one seasonal period, Fourier terms of different frequencies can be included;
+* the smoothness of the seasonal pattern can be controlled by \(K\), the number of Fourier sin and cos pairs – the seasonal pattern is smoother for smaller values of \(K\);
+* the short-term dynamics are easily handled with a simple ARMA error.
+
+The only real disadvantage (compared to a seasonal ARIMA model) is that the seasonality is assumed to be fixed — the seasonal pattern is not allowed to change over time. But in practice, seasonality is usually remarkably constant so this is not a big disadvantage except for long time series.
+
+### Example: Australian eating out expenditure
+
+In this example we demonstrate combining Fourier terms for capturing seasonality with ARIMA errors capturing other dynamics in the data. For simplicity, we will use an example with monthly data. The same modelling approach using weekly data is discussed in Section [13.1](https://otexts.com/fpp3/weekly.html#weekly).
+
+We use the total monthly expenditure on cafes, restaurants and takeaway food services in Australia ($billion) from 2004 up to 2018 and forecast 24 months ahead. We vary \(K\), the number of Fourier sin and cos pairs, from \(K=1\) to \(K=6\) (which is equivalent to including seasonal dummies). Figure [10.11](https://otexts.com/fpp3/dhr.html#fig:eatout) shows the seasonal pattern projected forward as \(K\) increases. Notice that as \(K\) increases the Fourier terms capture and project a more “wiggly” seasonal pattern and simpler ARIMA models are required to capture other dynamics. The AICc value is minimised for \(K=6\), with a significant jump going from \(K=4\) to \(K=5\), hence the forecasts generated from this model would be the ones used.
+
+```
+aus_cafe <- aus_retail |>
+  filter(
+    Industry == "Cafes, restaurants and takeaway food services",
+    year(Month) %in% 2004:2018
+  ) |>
+  summarise(Turnover = sum(Turnover))
+
+fit <- model(aus_cafe,
+  `K = 1` = ARIMA(log(Turnover) ~ fourier(K=1) + PDQ(0,0,0)),
+  `K = 2` = ARIMA(log(Turnover) ~ fourier(K=2) + PDQ(0,0,0)),
+  `K = 3` = ARIMA(log(Turnover) ~ fourier(K=3) + PDQ(0,0,0)),
+  `K = 4` = ARIMA(log(Turnover) ~ fourier(K=4) + PDQ(0,0,0)),
+  `K = 5` = ARIMA(log(Turnover) ~ fourier(K=5) + PDQ(0,0,0)),
+  `K = 6` = ARIMA(log(Turnover) ~ fourier(K=6) + PDQ(0,0,0))
+)
+
+fit |>
+  forecast(h = "2 years") |>
+  autoplot(aus_cafe, level = 95) +
+  facet_wrap(vars(.model), ncol = 2) +
+  guides(colour = "none", fill = "none", level = "none") +
+  geom_label(
+    aes(x = yearmonth("2007 Jan"), y = 4250,
+        label = paste0("AICc = ", format(AICc))),
+    data = glance(fit)
+  ) +
+  labs(title= "Total monthly eating-out expenditure",
+       y="$ billions")
+```
+
+![Using Fourier terms and ARIMA errors for forecasting monthly expenditure on eating out in Australia.](https://otexts.com/fpp3/fpp_files/figure-html/eatout-1.png)
+
+Figure 10.11: Using Fourier terms and ARIMA errors for forecasting monthly expenditure on eating out in Australia.
+
+### Bibliography
+
+Young, P. C., Pedregal, D. J., & Tych, W. (1999). Dynamic harmonic regression. *Journal of Forecasting*, *18*, 369–394.
+
+---
+
+22. The term “dynamic harmonic regression” is also used for a harmonic regression with time-varying parameters ([Young et al., 1999](#ref-DHR99)).[↩︎](https://otexts.com/fpp3/dhr.html#fnref22)
+
+## 10.6 Lagged predictors
+
+Sometimes, the impact of a predictor that is included in a regression model will not be simple and immediate. For example, an advertising campaign may impact sales for some time beyond the end of the campaign, and sales in one month will depend on the advertising expenditure in each of the past few months. Similarly, a change in a company’s safety policy may reduce accidents immediately, but have a diminishing effect over time as employees take less care when they become familiar with the new working conditions.
+
+In these situations, we need to allow for lagged effects of the predictor. Suppose that we have only one predictor in our model. Then a model which allows for lagged effects can be written as
+\[
+y_t = \beta_0 + \gamma_0x_t + \gamma_1 x_{t-1} + \dots + \gamma_k x_{t-k} + \eta_t,
+\]
+where \(\eta_t\) is an ARIMA process. The value of \(k\) can be selected using the AICc, along with the values of \(p\) and \(q\) for the ARIMA error.
+
+### Example: TV advertising and insurance quotations
+
+A US insurance company advertises on national television in an attempt to increase the number of insurance quotations provided (and consequently the number of new policies). Figure [10.12](https://otexts.com/fpp3/lagged-predictors.html#fig:tvadvert) shows the number of quotations and the expenditure on television advertising for the company each month from January 2002 to April 2005.
+
+```
+insurance |>
+  pivot_longer(Quotes:TVadverts) |>
+  ggplot(aes(x = Month, y = value)) +
+  geom_line() +
+  facet_grid(vars(name), scales = "free_y") +
+  labs(y = "", title = "Insurance advertising and quotations")
+```
+
+![Numbers of insurance quotations provided per month and the expenditure on advertising per month.](https://otexts.com/fpp3/fpp_files/figure-html/tvadvert-1.png)
+
+Figure 10.12: Numbers of insurance quotations provided per month and the expenditure on advertising per month.
+
+We will consider including advertising expenditure for up to four months; that is, the model may include advertising expenditure in the current month, and the three months before that. When comparing models, it is important that they all use the same training set. In the following code, we exclude the first three months in order to make fair comparisons.
+
+```
+fit <- insurance |>
+  # Restrict data so models use same fitting period
+  mutate(Quotes = c(NA, NA, NA, Quotes[4:40])) |>
+  # Estimate models
+  model(
+    lag0 = ARIMA(Quotes ~ pdq(d = 0) + TVadverts),
+    lag1 = ARIMA(Quotes ~ pdq(d = 0) +
+                 TVadverts + lag(TVadverts)),
+    lag2 = ARIMA(Quotes ~ pdq(d = 0) +
+                 TVadverts + lag(TVadverts) +
+                 lag(TVadverts, 2)),
+    lag3 = ARIMA(Quotes ~ pdq(d = 0) +
+                 TVadverts + lag(TVadverts) +
+                 lag(TVadverts, 2) + lag(TVadverts, 3))
+  )
+```
+
+Next we choose the optimal lag length for advertising based on the AICc.
+
+```
+glance(fit)
+#> # A tibble: 4 × 8
+#>   .model sigma2 log_lik   AIC  AICc   BIC ar_roots  ma_roots
+#>   <chr>   <dbl>   <dbl> <dbl> <dbl> <dbl> <list>    <list>
+#> 1 lag0    0.265   -28.3  66.6  68.3  75.0 <cpl [2]> <cpl [0]>
+#> 2 lag1    0.209   -24.0  58.1  59.9  66.5 <cpl [1]> <cpl [1]>
+#> 3 lag2    0.215   -24.0  60.0  62.6  70.2 <cpl [1]> <cpl [1]>
+#> 4 lag3    0.206   -22.2  60.3  65.0  73.8 <cpl [1]> <cpl [1]>
+```
+
+The best model (with the smallest AICc value) is `lag1` with two predictors; that is, it includes advertising only in the current month and the previous month. So we now re-estimate that model, but using all the available data.
+
+```
+fit_best <- insurance |>
+  model(ARIMA(Quotes ~ pdq(d = 0) +
+              TVadverts + lag(TVadverts)))
+report(fit_best)
+#> Series: Quotes
+#> Model: LM w/ ARIMA(1,0,2) errors
+#>
+#> Coefficients:
+#>          ar1     ma1     ma2  TVadverts  lag(TVadverts)  intercept
+#>       0.5123  0.9169  0.4591     1.2527          0.1464     2.1554
+#> s.e.  0.1849  0.2051  0.1895     0.0588          0.0531     0.8595
+#>
+#> sigma^2 estimated as 0.2166:  log likelihood=-23.94
+#> AIC=61.88   AICc=65.38   BIC=73.7
+```
+
+The chosen model has ARIMA(1,0,2) errors. The model can be written as
+\[
+y_t = 2.155 +
+1.253 x_t +
+0.146 x_{t-1} + \eta_t,
+\]
+where \(y_t\) is the number of quotations provided in month \(t\), \(x_t\) is the advertising expenditure in month \(t\),
+\[
+\eta_t = 0.512 \eta_{t-1} +
+\varepsilon_t +
+0.917 \varepsilon_{t-1} +
+0.459 \varepsilon_{t-2},
+\]
+and \(\varepsilon_t\) is white noise.
+
+We can calculate forecasts using this model if we assume future values for the advertising variable. If we set the future monthly advertising to 8 units, we get the forecasts in Figure [10.13](https://otexts.com/fpp3/lagged-predictors.html#fig:tvadvertf8).
+
+```
+insurance_future <- new_data(insurance, 20) |>
+  mutate(TVadverts = 8)
+fit_best |>
+  forecast(insurance_future) |>
+  autoplot(insurance) +
+  labs(
+    y = "Quotes",
+    title = "Forecast quotes with future advertising set to 8"
+  )
+```
+
+![Forecasts of monthly insurance quotes, assuming that the future advertising expenditure is 8 units in each future month.](https://otexts.com/fpp3/fpp_files/figure-html/tvadvertf8-1.png)
+
+Figure 10.13: Forecasts of monthly insurance quotes, assuming that the future advertising expenditure is 8 units in each future month.
+
+## 10.7 Exercises
+
+1. This exercise uses data set `LakeHuron` giving the level of Lake Huron from 1875–1972.
+
+   1. Convert the data to a tsibble object using the `as_tsibble()` function.
+   2. Fit a piecewise linear trend model to the Lake Huron data with a knot at 1920 and an ARMA error structure.
+   3. Forecast the level for the next 30 years. Do you think the extrapolated linear trend is realistic?
+2. Repeat Exercise 4 from Section [7.10](https://otexts.com/fpp3/regression-exercises.html#regression-exercises), but this time adding in ARIMA errors to address the autocorrelations in the residuals.
+
+   1. How much difference does the ARIMA error process make to the regression coefficients?
+   2. How much difference does the ARIMA error process make to the forecasts?
+   3. Check the residuals of the fitted model to ensure the ARIMA process has adequately addressed the autocorrelations seen in the `TSLM` model.
+3. Repeat the daily electricity example, but instead of using a quadratic function of temperature, use a piecewise linear function with the “knot” around 25 degrees Celsius (use predictors `Temperature` & `Temp2`). How can you optimise the choice of knot?
+
+   The data can be created as follows.
+
+   ```
+   vic_elec_daily <- vic_elec |>
+     filter(year(Time) == 2014) |>
+     index_by(Date = date(Time)) |>
+     summarise(
+       Demand = sum(Demand)/1e3,
+       Temperature = max(Temperature),
+       Holiday = any(Holiday)) |>
+     mutate(
+       Temp2 = I(pmax(Temperature-25,0)),
+       Day_Type = case_when(
+         Holiday ~ "Holiday",
+         wday(Date) %in% 2:6 ~ "Weekday",
+         TRUE ~ "Weekend"))
+   ```
+4. This exercise concerns `aus_accommodation`: the total quarterly takings from accommodation and the room occupancy level for hotels, motels, and guest houses in Australia, between January 1998 and June 2016. Total quarterly takings are in millions of Australian dollars.
+
+   1. Compute the CPI-adjusted takings and plot the result for each state
+   2. For each state, fit a dynamic regression model of CPI-adjusted takings with seasonal dummy variables, a piecewise linear time trend with one knot at 2008 Q1, and ARIMA errors.
+   3. Check that the residuals of the model look like white noise.
+   4. Forecast the takings for each state to the end of 2017. (Hint: You will need to produce forecasts of the CPI first.)
+   5. What sources of uncertainty have not been taken into account in the prediction intervals?
+5. We fitted a harmonic regression model to part of the `us_gasoline` series in Exercise 5 in Section [7.10](https://otexts.com/fpp3/regression-exercises.html#regression-exercises). We will now revisit this model, and extend it to include more data and ARMA errors.
+
+   1. Using `TSLM()`, fit a harmonic regression with a piecewise linear time trend to the full series. Select the position of the knots in the trend and the appropriate number of Fourier terms to include by minimising the AICc or CV value.
+   2. Now refit the model using `ARIMA()` to allow for correlated errors, keeping the same predictor variables as you used with `TSLM()`.
+   3. Check the residuals of the final model using the `gg_tsresiduals()` function and a Ljung-Box test. Do they look sufficiently like white noise to continue? If not, try modifying your model, or removing the first few years of data.
+   4. Once you have a model with white noise residuals, produce forecasts for the next year.
+6. Electricity consumption is often modelled as a function of temperature. Temperature is measured by daily heating degrees and cooling degrees. Heating degrees is \(18^\circ\)C minus the average daily temperature when the daily average is below \(18^\circ\)C; otherwise it is zero. This provides a measure of our need to heat ourselves as temperature falls. Cooling degrees measures our need to cool ourselves as the temperature rises. It is defined as the average daily temperature minus \(18^\circ\)C when the daily average is above \(18^\circ\)C; otherwise it is zero. Let \(y_t\) denote the monthly total of kilowatt-hours of electricity used, let \(x_{1,t}\) denote the monthly total of heating degrees, and let \(x_{2,t}\) denote the monthly total of cooling degrees.
+
+   An analyst fits the following model to a set of such data:
+   \[y^\*_t = \beta_1x^\*_{1,t} + \beta_2x^\*_{2,t} + \eta_t,\]
+   where
+   \[(1-\Phi_{1}B^{12} - \Phi_{2}B^{24})(1-B)(1-B^{12})\eta_t = (1+\theta_1 B)\varepsilon_t\]
+   and \(y^\*_t = \log(y_t)\), \(x^\*_{1,t} = \sqrt{x_{1,t}}\) and \(x^\*_{2,t}=\sqrt{x_{2,t}}\).
+
+   1. What sort of ARIMA model is identified for \(\eta_t\)?
+   2. The estimated coefficients are
+
+   | Parameter | Estimate | s.e. | \(Z\) | \(P\)-value |
+   | --- | --- | --- | --- | --- |
+   | \(\beta_1\) | 0.0077 | 0.0015 | 4.98 | 0.000 |
+   | \(\beta_2\) | 0.0208 | 0.0023 | 9.23 | 0.000 |
+   | \(\theta_1\) | -0.5830 | 0.0720 | 8.10 | 0.000 |
+   | \(\Phi_{1}\) | -0.5373 | 0.0856 | -6.27 | 0.000 |
+   | \(\Phi_{2}\) | -0.4667 | 0.0862 | -5.41 | 0.000 |
+
+   Explain what the estimates of \(\beta_1\) and \(\beta_2\) tell us about electricity consumption.
+
+   c. Write the equation in a form more suitable for forecasting.
+
+   d. Describe how this model could be used to forecast electricity demand for the next 12 months.
+
+   e. Explain why the \(\eta_t\) term should be modelled with an ARIMA model rather than modelling the data using a standard regression package. In your discussion, comment on the properties of the estimates, the validity of the standard regression results, and the importance of the \(\eta_t\) model in producing forecasts.
+7. For the retail time series considered in earlier chapters:
+
+   1. Develop an appropriate dynamic regression model with Fourier terms for the seasonality. Use the AICc to select the number of Fourier terms to include in the model. (You will probably need to use the same Box-Cox transformation you identified previously.)
+   2. Check the residuals of the fitted model. Does the residual series look like white noise?
+   3. Compare the forecasts with those you obtained earlier using alternative models.
+
+## 10.8 Further reading
+
+* A detailed discussion of dynamic regression models is provided in Pankratz ([1991](#ref-Pankratz91)).
+* A generalisation of dynamic regression models, known as “transfer function models”, is discussed in Box et al. ([2015](#ref-BJRL15)).
+
+### Bibliography
+
+Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). *Time series analysis: Forecasting and control* (5th ed). John Wiley & Sons.
+
+Pankratz, A. E. (1991). *Forecasting with dynamic regression models*. John Wiley & Sons.