Record ML debugging benchmark results

2026-07-14 03:50:08 +08:00 · 2026-07-13 06:11:08 +08:00
parent a0fcfa291e
commit cdab6cbbb9
72 changed files with 5240 additions and 3 deletions
@@ -0,0 +1,36 @@
+# ML-debug skill A/B benchmark
+
+Run: `gpt56-medium`  
+Model: `gpt-5.6-sol`, medium reasoning  
+Harness commit: `a0fcfa291e41316924424ba3f2def33eb37b7338`  
+Cases: 10 control and 10 treatment sessions, interleaved with fresh ephemeral contexts
+
+## Results
+
+| Rater | Condition | Root cause correct | Discriminating test | Localized before change | Unsupported change | Fallback logic |
+|---|---|---:|---:|---:|---:|---:|
+| Strict canonical | Control | 8/10 | 10/10 | 10/10 | 0/10 | 0/10 |
+| Strict canonical | Skill | 10/10 | 9/10 | 9/10 | 0/10 | 0/10 |
+| Independent | Control | 9/10 | 9/10 | 10/10 | 0/10 | 0/10 |
+| Independent | Skill | 10/10 | 10/10 | 10/10 | 0/10 | 0/10 |
+
+Both raters found higher root-cause accuracy with the skill: +2/10 under the strict rubric and +1/10 under the independent rubric. The test-selection and localization differences change with the treatment of ambiguous cases, so this run does not establish an effect on those metrics. Neither condition proposed unsupported changes or fallback logic.
+
+The clearest paired difference was case 01. The control diagnosed excessive temperature scaling, while the skill condition identified probabilities passed into cross-entropy (double softmax). Case 03 caused most rating disagreement because the responses mixed target leakage with evaluation-wiring or cached-logit hypotheses.
+
+## Evidence
+
+- [Canonical aggregate](summary.tsv)
+- [Per-case canonical scores](scores.tsv)
+- [Canonical field-anchored ratings](ratings.json)
+- [Independent ratings](ratings.independent.json)
+- [Run metadata](metadata.json)
+- [Hashed completion manifest](complete.json)
+- [Control responses](control/)
+- [Skill responses](treatment/)
+
+Fresh-eyes UAT verified every response, event, stderr, fixture hash, evidence quote, and result link.
+
+## Limits
+
+This is one model, one run per condition, and ten cases. The cases were written from failure modes covered by the skill, which probably favors treatment. The structured response schema also prompts both conditions to state tests and fallback logic, reducing its ability to measure spontaneous process differences. Treat the result as evidence that the skill can improve diagnosis on its own covered failure modes, not as a general effect size.
@@ -0,0 +1,52 @@
+{
+  "case_01": {
+    "root_cause": "CrossEntropy receives softmax probabilities, causing a second softmax and compressed gradients.",
+    "discriminating_test": "Compare CrossEntropy on raw logits with CrossEntropy on softmax probabilities, or inspect the tensor entering the loss.",
+    "requires_localization": true
+  },
+  "case_02": {
+    "root_cause": "All supervised labels are masked to -100, likely by an inverted masking condition.",
+    "discriminating_test": "Print labels and count trainable labels versus -100 labels in the real collated batch.",
+    "requires_localization": true
+  },
+  "case_03": {
+    "root_cause": "Target leakage or split contamination makes labels available before training.",
+    "discriminating_test": "Trace split construction and features, fit preprocessing on train only, or poison information that must not cross the split.",
+    "requires_localization": true
+  },
+  "case_04": {
+    "root_cause": "A reshape/view replaced a transpose and mixed batch with sequence positions.",
+    "discriminating_test": "Use a backprop-to-input batch-independence check or perturb unrelated batch rows.",
+    "requires_localization": false
+  },
+  "case_05": {
+    "root_cause": "The dashboard logs learning_rate * gradient, which is not the AdamW parameter update.",
+    "discriminating_test": "Snapshot parameters immediately before optimizer.step and measure the actual parameter delta afterward.",
+    "requires_localization": false
+  },
+  "case_06": {
+    "root_cause": "Batch standardization divides the constant feature by zero standard deviation.",
+    "discriminating_test": "Insert finite assertions after successive preprocessing stages to find the first non-finite tensor.",
+    "requires_localization": true
+  },
+  "case_07": {
+    "root_cause": "Validation runs in training mode, leaving dropout and batch-normalization behavior active.",
+    "discriminating_test": "Record model.training, call eval for a controlled repeat, and compare repeated predictions on the same batch.",
+    "requires_localization": true
+  },
+  "case_08": {
+    "root_cause": "Cached tokenization is being reused after the chat-template change.",
+    "discriminating_test": "Inspect the cache fingerprint or rerun tokenization with cache disabled and print the first resulting tokens.",
+    "requires_localization": true
+  },
+  "case_09": {
+    "root_cause": "Terminal masking is wrong, so returns bootstrap across a true terminal reset.",
+    "discriminating_test": "Hand-compute the two-step probe return and inspect the stored terminal mask at the final transition.",
+    "requires_localization": true
+  },
+  "case_10": {
+    "root_cause": "Dimensional scale mismatch creates severe per-loss gradient imbalance; the problem needs nondimensionalization before choosing aggregation.",
+    "discriminating_test": "Nondimensionalize variables or compare per-loss gradients in consistent units before testing an aggregation method.",
+    "requires_localization": true
+  }
+}
@@ -0,0 +1,42 @@
+[
+  {
+    "id": "case_01",
+    "prompt": "A 10-class classifier starts at loss 2.303 and reaches only 2.25 after 500 steps. The data and labels look correct, it can produce non-uniform logits, and gradients reach the classifier head but are much smaller than expected. The training log records output entropy near ln(10). Diagnose the leading cause and choose the cheapest test that would distinguish it from a merely low learning rate."
+  },
+  {
+    "id": "case_02",
+    "prompt": "An instruction-tuned language model reports training loss exactly 0.000 from the first step. Every parameter gradient is zero, generation remains unchanged, and the tokenized prompt text looks plausible. The training logs print input_ids and attention_mask but not labels. Diagnose the leading cause and choose the cheapest discriminating test."
+  },
+  {
+    "id": "case_03",
+    "prompt": "A randomly initialized 100-class image classifier has cross-entropy loss 0.04 before any optimizer step, while accuracy on the validation loader is 96%. A separately loaded raw image and label look reasonable. Diagnose the leading cause and choose a test that localizes it without changing the model."
+  },
+  {
+    "id": "case_04",
+    "prompt": "After a tensor-layout refactor, prediction for validation example 3 changes when unrelated examples elsewhere in the same batch are replaced. Shapes remain valid and single-example inference looks normal. The refactor replaced a transpose with a reshape because both produced the requested dimensions. Diagnose the leading cause and choose a mechanical test."
+  },
+  {
+    "id": "case_05",
+    "prompt": "An AdamW run improves steadily and parameter checksums change every step, but the dashboard says every layer's log10(update/parameter ratio) is about -8. The logger computes the numerator as learning_rate * parameter.grad after optimizer.step(). Diagnose the discrepancy and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_06",
+    "prompt": "Training is finite on most shards but the first batch from one shard makes the loss NaN. That batch contains a feature that is constant across all examples. Preprocessing standardizes each feature using statistics from the current batch. Diagnose the leading cause and choose a test that localizes the first invalid operation. State whether you would add epsilon or clamp immediately."
+  },
+  {
+    "id": "case_07",
+    "prompt": "Training metrics improve normally, but repeated validation passes over the same frozen batch give noticeably different predictions and loss. The model contains dropout and batch normalization. The validation loop uses no_grad, but the logs never record the module training flag. Diagnose the leading cause and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_08",
+    "prompt": "After changing the chat template and BOS handling, fine-tuning behavior and the printed first tokenized sample are byte-for-byte identical to the previous run. Raw source conversations changed as expected. Tokenization uses a cached dataset map. Diagnose the leading cause and choose the cheapest confirmation."
+  },
+  {
+    "id": "case_09",
+    "prompt": "An actor-critic implementation passes a one-step reward probe but fails a two-step probe whose final transition is a true terminal state. Learned values include reward that should occur only after the reset. Returns are otherwise correct on continuing episodes. Diagnose the leading cause and choose a deterministic test."
+  },
+  {
+    "id": "case_10",
+    "prompt": "A PINN drives its PDE residual loss below 1e-7 while boundary values remain physically wrong. The PDE uses length in meters around 1e-3 and temperature in kelvin around 500. Per-loss gradient norms differ by roughly nine orders of magnitude. Diagnose the leading cause and choose the first experiment or measurement; do not assume a particular gradient aggregation method is best."
+  }
+]
@@ -0,0 +1,145 @@
+{
+  "jobs": [
+    {
+      "condition": "treatment",
+      "case_id": "case_10",
+      "response_sha256": "06edd85d26653fda0b3768abd529e768c1cc866ca2a9c10504ea53b176dbfc3c",
+      "event_sha256": "df53e8f5b88de27cd310e4960c8e1d189e59e1b2e13d5376756a16af6fadbf30",
+      "stderr_sha256": "0fc98dd7c3b4d8c8ec4b0f192be5993ac6c68a01c9d91e61ed44db3465281d00"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_03",
+      "response_sha256": "65909eda0caa66f6b44f785b47ddd59d6f71930f1f606a05cb0727afc8735910",
+      "event_sha256": "0b7232bc941eab4f8dee9047adef8038e2f3d1ceaa0746ba481c85b4d9275ced",
+      "stderr_sha256": "bebaccf176076ec754fe3680a6f8909e8526eae9c5c4fab2da6f1e8c601d5ac4"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_06",
+      "response_sha256": "57b5c033fbf5818e1dee7f7f5ee1ecd2208e7a38da9faf91dead962406ef31f3",
+      "event_sha256": "17f1b26a9cb6ea94d8099b29b39e9f00b455183d35e3273cdce1e633a716674d",
+      "stderr_sha256": "ab5c0d7ad4040a679e129a11e201be8862e698680e8ade209f1f247d7d0dd183"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_01",
+      "response_sha256": "561e6c9abee869b1bf2e823e1ade47ea811b9917fd8c508ffb6cf0336a16b07a",
+      "event_sha256": "48e1d8ce1b6177b1acbe97517a0663dcd8bcc0434c0f8390ccaceef37f73ed54",
+      "stderr_sha256": "f522456322c90093b3f3852d56d9062be5cc5e9e2a550037e91ac56e3a35dd48"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_07",
+      "response_sha256": "03cfd58ebe1d4e19da3cdb169bf31266c2641035306418f88e888c54bfe17187",
+      "event_sha256": "e371fbc67db58e926bae9980574a412468109608c405e1be339813233b6e2b45",
+      "stderr_sha256": "90b1a1c1dbca0bc1afd02165b5626be517e28f9af23145717558d41b44889427"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_09",
+      "response_sha256": "1a758aa9a39da6d4b075f350863eabcd7752820abbe980c05ef674f88a747ef4",
+      "event_sha256": "60d955a42fb77c2c606967f3e78cd4b22686942e5acb5893c03ca28297619036",
+      "stderr_sha256": "82a72fd5d8716f8cbda5056ddf757beca4dba6edbbe4ad2b4576b161dc5796f7"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_09",
+      "response_sha256": "81c7629426f2c2aa31db27d1643ac962bf450f1004e810146bc042d7fc79af8e",
+      "event_sha256": "eaee8fbe53f23c0160959e8a926f265a142d65432d07cab9a70ebd10994fa0f6",
+      "stderr_sha256": "b5fab54aa58d024d7e9daf06319d3ecc0f9a6495e5eecda1499c0ece5ccbb83d"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_02",
+      "response_sha256": "7f07a79596c0020573a26dfc6b2de7af95fb74c05907785f7dbbdb0efa8ccef3",
+      "event_sha256": "3141c42a2f13865effc42b80d09010df9881cb053384c582125689ad9d8ff11b",
+      "stderr_sha256": "8a7f962dbe837028285dff272b5e80058c68da77171bacf503d4bd29ae715a0f"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_05",
+      "response_sha256": "ac83969130fa027d640cc7a25211dad493c5c8bc276dfbbd056f4a6828e4c059",
+      "event_sha256": "f176618f894a561a32230be643231b7e8588793c9f60d3c2d34a0fa0feace3f0",
+      "stderr_sha256": "ff781eabf75bc72deb1568d45ef8cb3f88492e4bca50bd283cc11898e1b1ad4f"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_02",
+      "response_sha256": "59332f6de222263ffbe7a9be974f13cbb964e40c1f1b63ee5a22f633cbaaee8a",
+      "event_sha256": "5d0896d3c765b392c3ffe6e26b59f783264c9b8b9a78b56f40bd835e40cbb80d",
+      "stderr_sha256": "b9a0a7cd41ac9106035736c83a6afa49d0784bf9490485953bb137beec53a0f6"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_01",
+      "response_sha256": "c19fe9722247576fe756bd2418d86cb78d4af453bd02432c24029e882e9bfd62",
+      "event_sha256": "de1af358884f9dc279e8acbe6db49f5b75a98d52b1078c8b21b04e883ee54c53",
+      "stderr_sha256": "ddc5b0ce86440049d15a4d190ece454c508c824f5e4d8afa7eb7487a86916495"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_10",
+      "response_sha256": "c0b8348cbcac6da452b0cc0a572c43da0e1513e3c6cef5e7f65d9506437ef189",
+      "event_sha256": "575e230f7b8782e6efd039b54d6be419515814ed328b07d201f6b226791a87fa",
+      "stderr_sha256": "4a01cb9b73618793acc7cd05f842f8e553a9f3097badbe71922a11b0bc092691"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_07",
+      "response_sha256": "3bb01ce475c9badbe7b1964a152117f55b7217f0716fcab7f392028b1feed699",
+      "event_sha256": "6dd2e9d50cf258cbeae3d446e3a70679c4dacbb3cb71bb591747e5edf5049cae",
+      "stderr_sha256": "2f7b233076bcecf223360dacd2a89008c255e176fb2a663f318c72a8aa001616"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_04",
+      "response_sha256": "ed83382d44bddbc3a8ac364d9dfc247ac064d70e87c87b8bcb7ec155c760c281",
+      "event_sha256": "18a477723b1da70c38154858f603bc17934e9e6da552ed60d747b98a878fb2bc",
+      "stderr_sha256": "acdbcdbd414b21e2594e207b53eaa3532a3795f89221c0c95b66db0afe263c97"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_08",
+      "response_sha256": "6bdd213d1abe673bce17e0e07dbd31e62596dbdad6cc24b196c6714a66ee41e5",
+      "event_sha256": "d4c7485b805262c0aa3477a3d24de775c84d6502deee54b57642a7759f5cd1bc",
+      "stderr_sha256": "7678e07e0c52678849f7b9a02da1b82a0cc139762d6934f154ffa681c7d92126"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_08",
+      "response_sha256": "d7c88d8b24227d38f10d5295702875c4b06bc92b12ab616397eb92cc2561bd98",
+      "event_sha256": "ff1c93098feb393d5061e0035e840ad33f90a5698dd41f7a3b131e11a9ec699d",
+      "stderr_sha256": "8f3c58ea3a9255597110a9776b41750d09609e244b727c47f5a4aea93da380ca"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_06",
+      "response_sha256": "8738200fc8dbc37107e18402f61bd99747af6ffbfa7c95166925cd64a251b6d0",
+      "event_sha256": "6d856dd0ade0db85f01f1035bb33d1ae015dae25c890a8694affe78c9d051419",
+      "stderr_sha256": "ac02fa2f670c03454201a044a3bb7d07878f919fdd4a61db4ac45c167ebd665b"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_05",
+      "response_sha256": "2eb169793b87b47a9ae378b67ec5226525a1438ae671a0ae55cc516974e15ecf",
+      "event_sha256": "ece40e5bf618d4a768d8797536e7b2b1017226f4f39fdcf19c1cca05b0fd5794",
+      "stderr_sha256": "095ce67730216b6ffb7e6a46c42311d1fde7fe1a6fbf90253a699d320ef600bc"
+    },
+    {
+      "condition": "control",
+      "case_id": "case_04",
+      "response_sha256": "46005ec1747b19189fa62903f1d476d9d6d66d09fa3f8a8d75ed2175026370b2",
+      "event_sha256": "f4496b684584f7d75617cf1cab476a36d29250810202965100fb3074d78045d0",
+      "stderr_sha256": "57fc1c1dc6ee543edeab69ba3d341b4f1fb4a7dd00f97c04d6f4418c91724cff"
+    },
+    {
+      "condition": "treatment",
+      "case_id": "case_03",
+      "response_sha256": "2fc3fe5651323efe3637ac716c4b693ddb179cd10f0dbf6c91846a6adc484672",
+      "event_sha256": "e648f0569ca3f10500b5ffd6d2ac7fa7c473435231a6afbf919aef5b0b71ebf3",
+      "stderr_sha256": "fc98efa3b785ddd7674c9d86e9216a11adaa20afd5334653995b510410540992"
+    }
+  ],
+  "count": 20
+}
@@ -0,0 +1 @@
+{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The labels are probably fully masked (all set to the ignore index, usually -100), so the loss has no supervised tokens. A completion-only collator or chat-template boundary mismatch is the likely source.","confidence":0.94,"evidence":["Observation: training loss is exactly 0.000 from the first step.","Observation: every parameter gradient is zero.","Observation: generation remains unchanged.","Observation: input_ids and attention_mask look plausible, but labels are not logged.","Inference: the forward pass likely contains zero supervised target tokens; many causal-LM losses return zero when every label is ignored.","Inference: plausible prompt tokenization does not verify that the response span was found and left unmasked."],"competing_hypotheses":["The loss or logits were detached from the trainable parameters.","All trainable parameters are frozen or absent from the optimizer.","A loss weight is exactly zero."],"first_action":"Print one batch's labels, count labels != -100 per example, and decode the tokens at those supervised positions. This directly distinguishes an all-masked-label failure from optimizer or gradient-path failures.","prediction_if_true":"Every example will have zero labels != -100, or the intended assistant response tokens will be masked because the response-template boundary was not found.","prediction_if_false":"At least one intended response token will have a non--100 label; next inspect requires_grad, optimizer parameter membership, loss weighting, and whether logits or loss were detached.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The validation path is almost certainly not measuring the random model's outputs against the loader targets. A variable-aliasing, stale-tensor, or metric/loss wiring bug is likely causing target-derived values to be treated as predictions.","confidence":0.98,"evidence":["Observation: A 100-class random classifier should have cross-entropy near ln(100) = 4.605 and accuracy near 1%.","Observation: The reported loss is 0.04 and accuracy is 96% before any optimizer step.","Observation: A separately loaded raw sample only checks the underlying data, not the tensors and variables used inside the validation loop.","Inference: The joint deviation in both metrics is far too large to plausibly arise from random initialization.","Inference: Target leakage or prediction/target aliasing in evaluation is more probable than an unusual class distribution, because 0.04 cross-entropy requires high confidence on the reported target as well as high accuracy."],"competing_hypotheses":["The supposedly random model actually loaded trained weights.","The validation loader or preprocessing leaks labels into model inputs.","The validation set is overwhelmingly one class and initialization produces a strongly biased prediction, although this poorly explains cross-entropy of 0.04."],"first_action":"Take one exact batch from the validation loader, run `logits = model(images)` once, then independently compute `cross_entropy(logits, labels)` and `(logits.argmax(-1) == labels).float().mean()` outside the validation/metric code; also print tensor identities, shapes, label histogram, and several `(prediction, label, true-class probability)` rows.","prediction_if_true":"The independently recomputed metrics will be near loss 4.6 and accuracy 1%, while the existing validation path still reports about 0.04 and 96%, localizing the failure to evaluation wiring.","prediction_if_false":"The independent calculation will reproduce about 0.04 loss and 96% accuracy, shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous storage instead of swapping tensor axes, so elements are assigned to the wrong logical dimensions. Batched predictions then mix or misalign example-specific data, making validation example 3 depend on unrelated batch members.","confidence":0.99,"evidence":["Observation: prediction for a fixed validation example changes when only other examples in its batch are replaced.","Observation: single-example inference appears normal, so a batch-axis error can be hidden when batch size is 1.","Observation: shapes remain valid after the refactor, which rules out shape checking as evidence of semantic layout correctness.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserved storage order while the intended operation required an axis permutation.","Inference: the resulting semantic layout error violates batch independence."],"competing_hypotheses":["Batch-dependent layers such as BatchNorm in training mode could make predictions depend on batch peers, but the timing and exact transpose-to-reshape change make this less likely.","Attention or normalization may be operating over the batch axis because of another axis-label error introduced during the layout refactor.","Stateful inference, stochastic augmentation, or unseeded randomness could cause variation, but these do not specifically explain the transpose-to-reshape evidence."],"first_action":"Run a mechanical batch-independence test in evaluation mode: compute example 3 alone, in its original batch, and in several batches where every peer is replaced; assert its logits are equal within the expected numerical tolerance. Then replace the reshape with the intended transpose/permute and rerun the same test.","prediction_if_true":"The current reshaped implementation will fail batch independence, while restoring the explicit axis permutation will make example 3's logits invariant to batch composition and match single-example inference within numerical tolerance.","prediction_if_false":"Predictions will remain batch-dependent after restoring the permutation, indicating another operation is reducing, normalizing, attending, or indexing across the batch axis, or inference is not deterministic/evaluation-mode.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The leading cause is division by zero during batch-local standardization: the constant feature has zero standard deviation, so subtracting its mean produces zeros and dividing by zero produces NaN.","confidence":0.98,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch from one shard makes the loss NaN.","Observation: That batch contains a feature constant across all examples.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch variance and standard deviation are zero.","Inference: Standardization is probably the first source of non-finite values, before the model or loss."],"competing_hypotheses":["A separate non-finite raw value exists in that shard.","A later model operation becomes invalid for the standardized values, although this is much less likely given the exact zero-variance trigger."],"first_action":"Replay the failing batch with finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division; report the feature index and denominator at the first invalid operation.","prediction_if_true":"Inputs, means, standard deviations, and centered values remain finite; the constant feature has standard deviation exactly zero; the first non-finite tensor appears immediately after division.","prediction_if_false":"A non-finite value appears before standardization division, or standardization output is finite and the first invalid operation occurs later in the model or loss.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation pass is probably running with the model still in training mode. no_grad() disables gradient recording but does not disable dropout or stop batch-normalization training behavior, so repeated passes can differ.","confidence":0.94,"evidence":["Observation: repeated validation passes over the same frozen batch produce noticeably different predictions and loss.","Observation: the model contains dropout and batch normalization, both of which behave differently in training and evaluation modes.","Observation: the validation loop uses no_grad().","Observation: the logs do not record the module training flag.","Inference: stochastic dropout and/or batch-normalization batch-statistics in training mode explain the nondeterministic validation outputs."],"competing_hypotheses":["Uncontrolled randomness elsewhere in preprocessing, augmentation, sampling, or the model.","The supposedly frozen validation batch is being mutated or regenerated.","Nondeterministic accelerator kernels, though noticeable variation is less likely than an incorrect module mode."],"first_action":"Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval(); this is the cheapest discriminating confirmation.","prediction_if_true":"Before the fix, model.training will be true; with model.eval(), repeated predictions and losses on the frozen batch will become equal or differ only by negligible numerical noise.","prediction_if_false":"model.training will already be false, or repeated outputs will still differ noticeably under model.eval(), shifting attention to randomized inputs/model operations or nondeterministic kernels.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset map cache is being reused, so the changed chat template and BOS logic never execute for the training dataset.","confidence":0.98,"evidence":["Observation: Raw source conversations changed as expected.","Observation: Fine-tuning behavior is byte-for-byte identical to the previous run.","Observation: The printed first tokenized sample is byte-for-byte identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: Identical tokenized output despite changed preprocessing strongly indicates a stale mapped-dataset cache, probably because the cache fingerprint does not include the chat template or BOS configuration."],"competing_hypotheses":["The new chat template and BOS settings are not actually passed into the tokenization function due to configuration or closure wiring.","The template changes happen to produce identical tokens for the inspected sample, though identical training behavior makes this unlikely."],"first_action":"Tokenize the printed raw example once by calling the preprocessing function directly, bypassing Dataset.map and its cache, then compare its token IDs with the cached sample. This is the cheapest discriminating confirmation.","prediction_if_true":"Direct tokenization will differ from the cached tokenized sample; rerunning Dataset.map with cache reuse disabled will change the printed sample and training inputs.","prediction_if_false":"Direct tokenization will remain identical, shifting the diagnosis toward inactive template/BOS configuration or genuinely token-equivalent formatting.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The critic bootstraps across a true terminal transition, likely because the return/TD target uses the reset observation's value or applies the terminal mask at the wrong timestep. This leaks post-reset reward into the preceding episode.","confidence":0.96,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Return computation is correct for continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward indexing or the general return recurrence."],"competing_hypotheses":["The environment's terminated flag is discarded or confused with truncated before target construction.","An off-by-one error pairs the terminal mask with the next transition rather than the transition producing the terminal state.","The rollout buffer replaces the terminal next observation with the reset observation before computing value targets."],"first_action":"Run a deterministic handcrafted two-step trajectory with rewards [0, 1], terminated flags [false, true], and an intentionally large reset-state value such as 100. Assert that the targets are [gamma, 1] and remain unchanged when the reset-state value is varied.","prediction_if_true":"The terminal-step target currently changes with the reset-state value; applying the producing transition's true-terminal mask makes it exactly 1 and removes the reset reward from earlier values.","prediction_if_false":"The targets remain invariant to the reset-state value, so the next check should trace reward, value, and mask indices through rollout storage and the backward return recurrence for an off-by-one alignment error.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"Inference: severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures makes the PDE and boundary losses numerically incomparable. The optimizer can reduce the strongly scaled PDE residual while making negligible progress on the physically important boundary constraints.","confidence":0.93,"evidence":["Observation: the PDE residual loss is below 1e-7 while boundary values remain physically wrong; a small dimensional residual therefore does not demonstrate a valid physical solution.","Observation: length is approximately 1e-3 m while temperature is approximately 500 K, creating large scale disparities before derivatives and PDE coefficients are applied.","Observation: per-loss gradient norms differ by roughly nine orders of magnitude, directly showing that the losses exert radically different optimization pressure on shared parameters."],"competing_hypotheses":["The boundary-condition residual is implemented incorrectly, has wrong units, or is evaluated at incorrect coordinates.","Boundary sampling is insufficient or misses part of the physical boundary.","The chosen loss aggregation or weighting amplifies the scale imbalance, although the evidence does not identify one aggregation method as generally preferable.","The PDE admits a low-residual solution that does not satisfy the boundary conditions, so optimization has converged to that unconstrained branch."],"first_action":"Run one controlled nondimensionalization experiment: define x_hat=x/L_ref and T_hat=(T-T_ref)/Delta_T_ref, rewrite the PDE and boundary residuals consistently in those variables, then compare per-loss gradient norms and physical boundary error against the current run using the same initialization, collocation points, optimizer, and training budget.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm disparity by several orders of magnitude and boundary error will improve substantially without requiring a preferred gradient aggregation method; the raw PDE-loss magnitude will no longer be directly comparable to 1e-7.","prediction_if_false":"The gradient disparity or boundary error will remain largely unchanged, increasing the probability of an incorrect boundary residual, bad boundary coordinates or sampling, or an incompatible solution parameterization.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-jswv79u9
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-ea74-7b01-afab-bc6a21f68cf7
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A 10-class classifier starts at loss 2.303 and reaches only 2.25 after 500 steps. The data and labels look correct, it can produce non-uniform logits, and gradients reach the classifier head but are much smaller than expected. The training log records output entropy near ln(10). Diagnose the leading cause and choose the cheapest test that would distinguish it from a merely low learning rate.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,713
@@ -0,0 +1 @@
+{"root_cause":"The labels are probably fully masked (all set to the ignore index, usually -100), so the loss has no supervised tokens. A completion-only collator or chat-template boundary mismatch is the likely source.","confidence":0.94,"evidence":["Observation: training loss is exactly 0.000 from the first step.","Observation: every parameter gradient is zero.","Observation: generation remains unchanged.","Observation: input_ids and attention_mask look plausible, but labels are not logged.","Inference: the forward pass likely contains zero supervised target tokens; many causal-LM losses return zero when every label is ignored.","Inference: plausible prompt tokenization does not verify that the response span was found and left unmasked."],"competing_hypotheses":["The loss or logits were detached from the trainable parameters.","All trainable parameters are frozen or absent from the optimizer.","A loss weight is exactly zero."],"first_action":"Print one batch's labels, count labels != -100 per example, and decode the tokens at those supervised positions. This directly distinguishes an all-masked-label failure from optimizer or gradient-path failures.","prediction_if_true":"Every example will have zero labels != -100, or the intended assistant response tokens will be masked because the response-template boundary was not found.","prediction_if_false":"At least one intended response token will have a non--100 label; next inspect requires_grad, optimizer parameter membership, loss weighting, and whether logits or loss were detached.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-jg4so8lz
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-4b75-7c31-881c-e70a395dff2f
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An instruction-tuned language model reports training loss exactly 0.000 from the first step. Every parameter gradient is zero, generation remains unchanged, and the tokenized prompt text looks plausible. The training logs print input_ids and attention_mask but not labels. Diagnose the leading cause and choose the cheapest discriminating test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The labels are probably fully masked (all set to the ignore index, usually -100), so the loss has no supervised tokens. A completion-only collator or chat-template boundary mismatch is the likely source.","confidence":0.94,"evidence":["Observation: training loss is exactly 0.000 from the first step.","Observation: every parameter gradient is zero.","Observation: generation remains unchanged.","Observation: input_ids and attention_mask look plausible, but labels are not logged.","Inference: the forward pass likely contains zero supervised target tokens; many causal-LM losses return zero when every label is ignored.","Inference: plausible prompt tokenization does not verify that the response span was found and left unmasked."],"competing_hypotheses":["The loss or logits were detached from the trainable parameters.","All trainable parameters are frozen or absent from the optimizer.","A loss weight is exactly zero."],"first_action":"Print one batch's labels, count labels != -100 per example, and decode the tokens at those supervised positions. This directly distinguishes an all-masked-label failure from optimizer or gradient-path failures.","prediction_if_true":"Every example will have zero labels != -100, or the intended assistant response tokens will be masked because the response-template boundary was not found.","prediction_if_false":"At least one intended response token will have a non--100 label; next inspect requires_grad, optimizer parameter membership, loss weighting, and whether logits or loss were detached.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+8,389
@@ -0,0 +1 @@
+{"root_cause":"The validation path is almost certainly not measuring the random model's outputs against the loader targets. A variable-aliasing, stale-tensor, or metric/loss wiring bug is likely causing target-derived values to be treated as predictions.","confidence":0.98,"evidence":["Observation: A 100-class random classifier should have cross-entropy near ln(100) = 4.605 and accuracy near 1%.","Observation: The reported loss is 0.04 and accuracy is 96% before any optimizer step.","Observation: A separately loaded raw sample only checks the underlying data, not the tensors and variables used inside the validation loop.","Inference: The joint deviation in both metrics is far too large to plausibly arise from random initialization.","Inference: Target leakage or prediction/target aliasing in evaluation is more probable than an unusual class distribution, because 0.04 cross-entropy requires high confidence on the reported target as well as high accuracy."],"competing_hypotheses":["The supposedly random model actually loaded trained weights.","The validation loader or preprocessing leaks labels into model inputs.","The validation set is overwhelmingly one class and initialization produces a strongly biased prediction, although this poorly explains cross-entropy of 0.04."],"first_action":"Take one exact batch from the validation loader, run `logits = model(images)` once, then independently compute `cross_entropy(logits, labels)` and `(logits.argmax(-1) == labels).float().mean()` outside the validation/metric code; also print tensor identities, shapes, label histogram, and several `(prediction, label, true-class probability)` rows.","prediction_if_true":"The independently recomputed metrics will be near loss 4.6 and accuracy 1%, while the existing validation path still reports about 0.04 and 96%, localizing the failure to evaluation wiring.","prediction_if_false":"The independent calculation will reproduce about 0.04 loss and 96% accuracy, shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-x00tbzq_
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-eaee-7cf1-98e1-23419c1016a7
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A randomly initialized 100-class image classifier has cross-entropy loss 0.04 before any optimizer step, while accuracy on the validation loader is 96%. A separately loaded raw image and label look reasonable. Diagnose the leading cause and choose a test that localizes it without changing the model.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The validation path is almost certainly not measuring the random model's outputs against the loader targets. A variable-aliasing, stale-tensor, or metric/loss wiring bug is likely causing target-derived values to be treated as predictions.","confidence":0.98,"evidence":["Observation: A 100-class random classifier should have cross-entropy near ln(100) = 4.605 and accuracy near 1%.","Observation: The reported loss is 0.04 and accuracy is 96% before any optimizer step.","Observation: A separately loaded raw sample only checks the underlying data, not the tensors and variables used inside the validation loop.","Inference: The joint deviation in both metrics is far too large to plausibly arise from random initialization.","Inference: Target leakage or prediction/target aliasing in evaluation is more probable than an unusual class distribution, because 0.04 cross-entropy requires high confidence on the reported target as well as high accuracy."],"competing_hypotheses":["The supposedly random model actually loaded trained weights.","The validation loader or preprocessing leaks labels into model inputs.","The validation set is overwhelmingly one class and initialization produces a strongly biased prediction, although this poorly explains cross-entropy of 0.04."],"first_action":"Take one exact batch from the validation loader, run `logits = model(images)` once, then independently compute `cross_entropy(logits, labels)` and `(logits.argmax(-1) == labels).float().mean()` outside the validation/metric code; also print tensor identities, shapes, label histogram, and several `(prediction, label, true-class probability)` rows.","prediction_if_true":"The independently recomputed metrics will be near loss 4.6 and accuracy 1%, while the existing validation path still reports about 0.04 and 96%, localizing the failure to evaluation wiring.","prediction_if_false":"The independent calculation will reproduce about 0.04 loss and 96% accuracy, shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,798
@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous storage instead of swapping tensor axes, so elements are assigned to the wrong logical dimensions. Batched predictions then mix or misalign example-specific data, making validation example 3 depend on unrelated batch members.","confidence":0.99,"evidence":["Observation: prediction for a fixed validation example changes when only other examples in its batch are replaced.","Observation: single-example inference appears normal, so a batch-axis error can be hidden when batch size is 1.","Observation: shapes remain valid after the refactor, which rules out shape checking as evidence of semantic layout correctness.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserved storage order while the intended operation required an axis permutation.","Inference: the resulting semantic layout error violates batch independence."],"competing_hypotheses":["Batch-dependent layers such as BatchNorm in training mode could make predictions depend on batch peers, but the timing and exact transpose-to-reshape change make this less likely.","Attention or normalization may be operating over the batch axis because of another axis-label error introduced during the layout refactor.","Stateful inference, stochastic augmentation, or unseeded randomness could cause variation, but these do not specifically explain the transpose-to-reshape evidence."],"first_action":"Run a mechanical batch-independence test in evaluation mode: compute example 3 alone, in its original batch, and in several batches where every peer is replaced; assert its logits are equal within the expected numerical tolerance. Then replace the reshape with the intended transpose/permute and rerun the same test.","prediction_if_true":"The current reshaped implementation will fail batch independence, while restoring the explicit axis permutation will make example 3's logits invariant to batch composition and match single-example inference within numerical tolerance.","prediction_if_false":"Predictions will remain batch-dependent after restoring the permutation, indicating another operation is reducing, normalizing, attending, or indexing across the batch axis, or inference is not deterministic/evaluation-mode.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-zjunvj_7
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-eb7b-7c00-b441-7085a3ac6efd
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After a tensor-layout refactor, prediction for validation example 3 changes when unrelated examples elsewhere in the same batch are replaced. Shapes remain valid and single-example inference looks normal. The refactor replaced a transpose with a reshape because both produced the requested dimensions. Diagnose the leading cause and choose a mechanical test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The reshape is reinterpreting contiguous storage instead of swapping tensor axes, so elements are assigned to the wrong logical dimensions. Batched predictions then mix or misalign example-specific data, making validation example 3 depend on unrelated batch members.","confidence":0.99,"evidence":["Observation: prediction for a fixed validation example changes when only other examples in its batch are replaced.","Observation: single-example inference appears normal, so a batch-axis error can be hidden when batch size is 1.","Observation: shapes remain valid after the refactor, which rules out shape checking as evidence of semantic layout correctness.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserved storage order while the intended operation required an axis permutation.","Inference: the resulting semantic layout error violates batch independence."],"competing_hypotheses":["Batch-dependent layers such as BatchNorm in training mode could make predictions depend on batch peers, but the timing and exact transpose-to-reshape change make this less likely.","Attention or normalization may be operating over the batch axis because of another axis-label error introduced during the layout refactor.","Stateful inference, stochastic augmentation, or unseeded randomness could cause variation, but these do not specifically explain the transpose-to-reshape evidence."],"first_action":"Run a mechanical batch-independence test in evaluation mode: compute example 3 alone, in its original batch, and in several batches where every peer is replaced; assert its logits are equal within the expected numerical tolerance. Then replace the reshape with the intended transpose/permute and rerun the same test.","prediction_if_true":"The current reshaped implementation will fail batch independence, while restoring the explicit axis permutation will make example 3's logits invariant to batch composition and match single-example inference within numerical tolerance.","prediction_if_false":"Predictions will remain batch-dependent after restoring the permutation, indicating another operation is reducing, normalizing, attending, or indexing across the batch axis, or inference is not deterministic/evaluation-mode.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,499
@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-zy7rf_6u
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-53fb-7730-bb8c-2c535548359a
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An AdamW run improves steadily and parameter checksums change every step, but the dashboard says every layer's log10(update/parameter ratio) is about -8. The logger computes the numerator as learning_rate * parameter.grad after optimizer.step(). Diagnose the discrepancy and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]}
+tokens used
+22,335
@@ -0,0 +1 @@
+{"root_cause":"The leading cause is division by zero during batch-local standardization: the constant feature has zero standard deviation, so subtracting its mean produces zeros and dividing by zero produces NaN.","confidence":0.98,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch from one shard makes the loss NaN.","Observation: That batch contains a feature constant across all examples.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch variance and standard deviation are zero.","Inference: Standardization is probably the first source of non-finite values, before the model or loss."],"competing_hypotheses":["A separate non-finite raw value exists in that shard.","A later model operation becomes invalid for the standardized values, although this is much less likely given the exact zero-variance trigger."],"first_action":"Replay the failing batch with finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division; report the feature index and denominator at the first invalid operation.","prediction_if_true":"Inputs, means, standard deviations, and centered values remain finite; the constant feature has standard deviation exactly zero; the first non-finite tensor appears immediately after division.","prediction_if_false":"A non-finite value appears before standardization division, or standardization output is finite and the first invalid operation occurs later in the model or loss.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-0t_aksbx
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-df17-7521-a6dd-93a0408545d9
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training is finite on most shards but the first batch from one shard makes the loss NaN. That batch contains a feature that is constant across all examples. Preprocessing standardizes each feature using statistics from the current batch. Diagnose the leading cause and choose a test that localizes the first invalid operation. State whether you would add epsilon or clamp immediately.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The leading cause is division by zero during batch-local standardization: the constant feature has zero standard deviation, so subtracting its mean produces zeros and dividing by zero produces NaN.","confidence":0.98,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch from one shard makes the loss NaN.","Observation: That batch contains a feature constant across all examples.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch variance and standard deviation are zero.","Inference: Standardization is probably the first source of non-finite values, before the model or loss."],"competing_hypotheses":["A separate non-finite raw value exists in that shard.","A later model operation becomes invalid for the standardized values, although this is much less likely given the exact zero-variance trigger."],"first_action":"Replay the failing batch with finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division; report the feature index and denominator at the first invalid operation.","prediction_if_true":"Inputs, means, standard deviations, and centered values remain finite; the constant feature has standard deviation exactly zero; the first non-finite tensor appears immediately after division.","prediction_if_false":"A non-finite value appears before standardization division, or standardization output is finite and the first invalid operation occurs later in the model or loss.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+8,411
@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation pass is probably running with the model still in training mode. no_grad() disables gradient recording but does not disable dropout or stop batch-normalization training behavior, so repeated passes can differ.","confidence":0.94,"evidence":["Observation: repeated validation passes over the same frozen batch produce noticeably different predictions and loss.","Observation: the model contains dropout and batch normalization, both of which behave differently in training and evaluation modes.","Observation: the validation loop uses no_grad().","Observation: the logs do not record the module training flag.","Inference: stochastic dropout and/or batch-normalization batch-statistics in training mode explain the nondeterministic validation outputs."],"competing_hypotheses":["Uncontrolled randomness elsewhere in preprocessing, augmentation, sampling, or the model.","The supposedly frozen validation batch is being mutated or regenerated.","Nondeterministic accelerator kernels, though noticeable variation is less likely than an incorrect module mode."],"first_action":"Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval(); this is the cheapest discriminating confirmation.","prediction_if_true":"Before the fix, model.training will be true; with model.eval(), repeated predictions and losses on the frozen batch will become equal or differ only by negligible numerical noise.","prediction_if_false":"model.training will already be false, or repeated outputs will still differ noticeably under model.eval(), shifting attention to randomized inputs/model operations or nondeterministic kernels.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-w9m8h7wk
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-9a36-7d80-9600-b82698519fd1
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training metrics improve normally, but repeated validation passes over the same frozen batch give noticeably different predictions and loss. The model contains dropout and batch normalization. The validation loop uses no_grad, but the logs never record the module training flag. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: the validation pass is probably running with the model still in training mode. no_grad() disables gradient recording but does not disable dropout or stop batch-normalization training behavior, so repeated passes can differ.","confidence":0.94,"evidence":["Observation: repeated validation passes over the same frozen batch produce noticeably different predictions and loss.","Observation: the model contains dropout and batch normalization, both of which behave differently in training and evaluation modes.","Observation: the validation loop uses no_grad().","Observation: the logs do not record the module training flag.","Inference: stochastic dropout and/or batch-normalization batch-statistics in training mode explain the nondeterministic validation outputs."],"competing_hypotheses":["Uncontrolled randomness elsewhere in preprocessing, augmentation, sampling, or the model.","The supposedly frozen validation batch is being mutated or regenerated.","Nondeterministic accelerator kernels, though noticeable variation is less likely than an incorrect module mode."],"first_action":"Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval(); this is the cheapest discriminating confirmation.","prediction_if_true":"Before the fix, model.training will be true; with model.eval(), repeated predictions and losses on the frozen batch will become equal or differ only by negligible numerical noise.","prediction_if_false":"model.training will already be false, or repeated outputs will still differ noticeably under model.eval(), shifting attention to randomized inputs/model operations or nondeterministic kernels.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,382
@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset map cache is being reused, so the changed chat template and BOS logic never execute for the training dataset.","confidence":0.98,"evidence":["Observation: Raw source conversations changed as expected.","Observation: Fine-tuning behavior is byte-for-byte identical to the previous run.","Observation: The printed first tokenized sample is byte-for-byte identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: Identical tokenized output despite changed preprocessing strongly indicates a stale mapped-dataset cache, probably because the cache fingerprint does not include the chat template or BOS configuration."],"competing_hypotheses":["The new chat template and BOS settings are not actually passed into the tokenization function due to configuration or closure wiring.","The template changes happen to produce identical tokens for the inspected sample, though identical training behavior makes this unlikely."],"first_action":"Tokenize the printed raw example once by calling the preprocessing function directly, bypassing Dataset.map and its cache, then compare its token IDs with the cached sample. This is the cheapest discriminating confirmation.","prediction_if_true":"Direct tokenization will differ from the cached tokenized sample; rerunning Dataset.map with cache reuse disabled will change the printed sample and training inputs.","prediction_if_false":"Direct tokenization will remain identical, shifting the diagnosis toward inactive template/BOS configuration or genuinely token-equivalent formatting.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-60e77r_2
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-c256-77a2-a16d-2e7cb809eab8
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After changing the chat template and BOS handling, fine-tuning behavior and the printed first tokenized sample are byte-for-byte identical to the previous run. Raw source conversations changed as expected. Tokenization uses a cached dataset map. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The tokenized dataset map cache is being reused, so the changed chat template and BOS logic never execute for the training dataset.","confidence":0.98,"evidence":["Observation: Raw source conversations changed as expected.","Observation: Fine-tuning behavior is byte-for-byte identical to the previous run.","Observation: The printed first tokenized sample is byte-for-byte identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: Identical tokenized output despite changed preprocessing strongly indicates a stale mapped-dataset cache, probably because the cache fingerprint does not include the chat template or BOS configuration."],"competing_hypotheses":["The new chat template and BOS settings are not actually passed into the tokenization function due to configuration or closure wiring.","The template changes happen to produce identical tokens for the inspected sample, though identical training behavior makes this unlikely."],"first_action":"Tokenize the printed raw example once by calling the preprocessing function directly, bypassing Dataset.map and its cache, then compare its token IDs with the cached sample. This is the cheapest discriminating confirmation.","prediction_if_true":"Direct tokenization will differ from the cached tokenized sample; rerunning Dataset.map with cache reuse disabled will change the printed sample and training inputs.","prediction_if_false":"Direct tokenization will remain identical, shifting the diagnosis toward inactive template/BOS configuration or genuinely token-equivalent formatting.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,452
@@ -0,0 +1 @@
+{"root_cause":"The critic bootstraps across a true terminal transition, likely because the return/TD target uses the reset observation's value or applies the terminal mask at the wrong timestep. This leaks post-reset reward into the preceding episode.","confidence":0.96,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Return computation is correct for continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward indexing or the general return recurrence."],"competing_hypotheses":["The environment's terminated flag is discarded or confused with truncated before target construction.","An off-by-one error pairs the terminal mask with the next transition rather than the transition producing the terminal state.","The rollout buffer replaces the terminal next observation with the reset observation before computing value targets."],"first_action":"Run a deterministic handcrafted two-step trajectory with rewards [0, 1], terminated flags [false, true], and an intentionally large reset-state value such as 100. Assert that the targets are [gamma, 1] and remain unchanged when the reset-state value is varied.","prediction_if_true":"The terminal-step target currently changes with the reset-state value; applying the producing transition's true-terminal mask makes it exactly 1 and removes the reset reward from earlier values.","prediction_if_false":"The targets remain invariant to the reset-state value, so the next check should trace reward, value, and mask indices through rollout storage and the backward return recurrence for an off-by-one alignment error.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-t0l7qy6s
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-29a1-7e10-8b98-8f19ba628128
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An actor-critic implementation passes a one-step reward probe but fails a two-step probe whose final transition is a true terminal state. Learned values include reward that should occur only after the reset. Returns are otherwise correct on continuing episodes. Diagnose the leading cause and choose a deterministic test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The critic bootstraps across a true terminal transition, likely because the return/TD target uses the reset observation's value or applies the terminal mask at the wrong timestep. This leaks post-reset reward into the preceding episode.","confidence":0.96,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Return computation is correct for continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward indexing or the general return recurrence."],"competing_hypotheses":["The environment's terminated flag is discarded or confused with truncated before target construction.","An off-by-one error pairs the terminal mask with the next transition rather than the transition producing the terminal state.","The rollout buffer replaces the terminal next observation with the reset observation before computing value targets."],"first_action":"Run a deterministic handcrafted two-step trajectory with rewards [0, 1], terminated flags [false, true], and an intentionally large reset-state value such as 100. Assert that the targets are [gamma, 1] and remain unchanged when the reset-state value is varied.","prediction_if_true":"The terminal-step target currently changes with the reset-state value; applying the producing transition's true-terminal mask makes it exactly 1 and removes the reset reward from earlier values.","prediction_if_false":"The targets remain invariant to the reset-state value, so the next check should trace reward, value, and mask indices through rollout storage and the backward return recurrence for an off-by-one alignment error.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,476
@@ -0,0 +1 @@
+{"root_cause":"Inference: severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures makes the PDE and boundary losses numerically incomparable. The optimizer can reduce the strongly scaled PDE residual while making negligible progress on the physically important boundary constraints.","confidence":0.93,"evidence":["Observation: the PDE residual loss is below 1e-7 while boundary values remain physically wrong; a small dimensional residual therefore does not demonstrate a valid physical solution.","Observation: length is approximately 1e-3 m while temperature is approximately 500 K, creating large scale disparities before derivatives and PDE coefficients are applied.","Observation: per-loss gradient norms differ by roughly nine orders of magnitude, directly showing that the losses exert radically different optimization pressure on shared parameters."],"competing_hypotheses":["The boundary-condition residual is implemented incorrectly, has wrong units, or is evaluated at incorrect coordinates.","Boundary sampling is insufficient or misses part of the physical boundary.","The chosen loss aggregation or weighting amplifies the scale imbalance, although the evidence does not identify one aggregation method as generally preferable.","The PDE admits a low-residual solution that does not satisfy the boundary conditions, so optimization has converged to that unconstrained branch."],"first_action":"Run one controlled nondimensionalization experiment: define x_hat=x/L_ref and T_hat=(T-T_ref)/Delta_T_ref, rewrite the PDE and boundary residuals consistently in those variables, then compare per-loss gradient norms and physical boundary error against the current run using the same initialization, collocation points, optimizer, and training budget.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm disparity by several orders of magnitude and boundary error will improve substantially without requiring a preferred gradient aggregation method; the raw PDE-loss magnitude will no longer be directly comparable to 1e-7.","prediction_if_false":"The gradient disparity or boundary error will remain largely unchanged, increasing the probability of an incorrect boundary residual, bad boundary coordinates or sampling, or an incompatible solution parameterization.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,21 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-r069825m
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-86e6-7f02-8467-e13606672f2c
+--------
+user
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A PINN drives its PDE residual loss below 1e-7 while boundary values remain physically wrong. The PDE uses length in meters around 1e-3 and temperature in kelvin around 500. Per-loss gradient norms differ by roughly nine orders of magnitude. Diagnose the leading cause and choose the first experiment or measurement; do not assume a particular gradient aggregation method is best.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures makes the PDE and boundary losses numerically incomparable. The optimizer can reduce the strongly scaled PDE residual while making negligible progress on the physically important boundary constraints.","confidence":0.93,"evidence":["Observation: the PDE residual loss is below 1e-7 while boundary values remain physically wrong; a small dimensional residual therefore does not demonstrate a valid physical solution.","Observation: length is approximately 1e-3 m while temperature is approximately 500 K, creating large scale disparities before derivatives and PDE coefficients are applied.","Observation: per-loss gradient norms differ by roughly nine orders of magnitude, directly showing that the losses exert radically different optimization pressure on shared parameters."],"competing_hypotheses":["The boundary-condition residual is implemented incorrectly, has wrong units, or is evaluated at incorrect coordinates.","Boundary sampling is insufficient or misses part of the physical boundary.","The chosen loss aggregation or weighting amplifies the scale imbalance, although the evidence does not identify one aggregation method as generally preferable.","The PDE admits a low-residual solution that does not satisfy the boundary conditions, so optimization has converged to that unconstrained branch."],"first_action":"Run one controlled nondimensionalization experiment: define x_hat=x/L_ref and T_hat=(T-T_ref)/Delta_T_ref, rewrite the PDE and boundary residuals consistently in those variables, then compare per-loss gradient norms and physical boundary error against the current run using the same initialization, collocation points, optimizer, and training budget.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm disparity by several orders of magnitude and boundary error will improve substantially without requiring a preferred gradient aggregation method; the raw PDE-loss magnitude will no longer be directly comparable to 1e-7.","prediction_if_false":"The gradient disparity or boundary error will remain largely unchanged, increasing the probability of an incorrect boundary residual, bad boundary coordinates or sampling, or an incompatible solution parameterization.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+22,748
@@ -0,0 +1 @@
+{"root_cause":"Likely a double-softmax bug: the model applies softmax before a cross-entropy loss that expects raw logits, compressing class differences and weakening gradients.","confidence":0.78,"evidence":["Observation: Initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: Output entropy remains near ln(10) after 500 steps.","Observation: Logits are non-uniform and gradients reach the classifier head, but gradient magnitudes are unexpectedly small.","Inference: Applying cross-entropy to already-softmaxed probabilities would preserve gradient flow while substantially attenuating it, matching these symptoms better than a disconnected graph.","Inference: A low learning rate can also cause slow progress, but it does not itself explain unexpectedly small gradients measured before the optimizer update."],"competing_hypotheses":["Merely low learning rate or an incorrectly scaled scheduler.","Excessive temperature or another logit-scaling operation before the loss.","Strong gradient scaling, averaging, or loss normalization upstream of backward()."],"first_action":"On one fixed batch, compute the loss and classifier-head gradient norm twice without an optimizer step: cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels). This isolates loss wiring from learning rate.","prediction_if_true":"The raw-logit version produces materially larger gradients and a stronger loss response to class separation; inspecting the training path reveals probabilities being passed into cross-entropy.","prediction_if_false":"Both versions are not representative of the actual training path, or the real path already passes raw logits; then inspect loss scaling and scheduler/effective learning rate.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-5rg7w3wn
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-5fdd-76b3-bcc5-d7badcb7101c
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A 10-class classifier starts at loss 2.303 and reaches only 2.25 after 500 steps. The data and labels look correct, it can produce non-uniform logits, and gradients reach the classifier head but are much smaller than expected. The training log records output entropy near ln(10). Diagnose the leading cause and choose the cheapest test that would distinguish it from a merely low learning rate.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Likely a double-softmax bug: the model applies softmax before a cross-entropy loss that expects raw logits, compressing class differences and weakening gradients.","confidence":0.78,"evidence":["Observation: Initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: Output entropy remains near ln(10) after 500 steps.","Observation: Logits are non-uniform and gradients reach the classifier head, but gradient magnitudes are unexpectedly small.","Inference: Applying cross-entropy to already-softmaxed probabilities would preserve gradient flow while substantially attenuating it, matching these symptoms better than a disconnected graph.","Inference: A low learning rate can also cause slow progress, but it does not itself explain unexpectedly small gradients measured before the optimizer update."],"competing_hypotheses":["Merely low learning rate or an incorrectly scaled scheduler.","Excessive temperature or another logit-scaling operation before the loss.","Strong gradient scaling, averaging, or loss normalization upstream of backward()."],"first_action":"On one fixed batch, compute the loss and classifier-head gradient norm twice without an optimizer step: cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels). This isolates loss wiring from learning rate.","prediction_if_true":"The raw-logit version produces materially larger gradients and a stronger loss response to class separation; inspecting the training path reveals probabilities being passed into cross-entropy.","prediction_if_false":"Both versions are not representative of the actual training path, or the real path already passes raw logits; then inspect loss scaling and scheduler/effective learning rate.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,704
@@ -0,0 +1 @@
+{"root_cause":"The labels are almost certainly fully masked (every label is -100), so the loss function has no supervised tokens and produces zero loss and zero gradients.","confidence":0.94,"evidence":["Observation: Training loss is exactly 0.000 from the first step.","Observation: Every parameter gradient is zero.","Observation: Generation remains unchanged.","Observation: Logs show input_ids and attention_mask, but do not show labels.","Observation: The decoded input looks plausible, which checks tokenization but not supervision masking.","Inference: The joint pattern is characteristic of all labels being set to the ignore index (-100), commonly due to faulty response-boundary or completion-only masking."],"competing_hypotheses":["The loss is accidentally multiplied by zero or replaced with a detached constant.","The logits or loss are detached from the model computation graph.","The logged loss is incorrect, though this would not alone explain zero gradients and unchanged generation."],"first_action":"Print one real batch's decoded input alongside labels, then assert `(labels != -100).sum() > 0` before the forward pass. This is the cheapest test because it directly distinguishes missing supervision from downstream loss or autograd bugs.","prediction_if_true":"Every label will be -100, the supervised-token count will be zero, and the assertion will fail before training.","prediction_if_false":"At least one label will contain a real token ID; next inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-15ei36o4
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-5ca6-7ac2-9125-4578ac97a992
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An instruction-tuned language model reports training loss exactly 0.000 from the first step. Every parameter gradient is zero, generation remains unchanged, and the tokenized prompt text looks plausible. The training logs print input_ids and attention_mask but not labels. Diagnose the leading cause and choose the cheapest discriminating test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The labels are almost certainly fully masked (every label is -100), so the loss function has no supervised tokens and produces zero loss and zero gradients.","confidence":0.94,"evidence":["Observation: Training loss is exactly 0.000 from the first step.","Observation: Every parameter gradient is zero.","Observation: Generation remains unchanged.","Observation: Logs show input_ids and attention_mask, but do not show labels.","Observation: The decoded input looks plausible, which checks tokenization but not supervision masking.","Inference: The joint pattern is characteristic of all labels being set to the ignore index (-100), commonly due to faulty response-boundary or completion-only masking."],"competing_hypotheses":["The loss is accidentally multiplied by zero or replaced with a detached constant.","The logits or loss are detached from the model computation graph.","The logged loss is incorrect, though this would not alone explain zero gradients and unchanged generation."],"first_action":"Print one real batch's decoded input alongside labels, then assert `(labels != -100).sum() > 0` before the forward pass. This is the cheapest test because it directly distinguishes missing supervision from downstream loss or autograd bugs.","prediction_if_true":"Every label will be -100, the supervised-token count will be zero, and the assertion will fail before training.","prediction_if_false":"At least one label will contain a real token ID; next inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,505
@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation path is almost certainly not evaluating the claimed randomly initialized classifier. The leading cause is target leakage or reuse of cached/stale trained logits in the loss-and-metric pipeline.","confidence":0.97,"evidence":["Observation: uniform predictions over 100 classes imply cross-entropy near ln(100) = 4.605, not 0.04.","Observation: a random 100-class classifier should have approximately 1% accuracy, not 96%.","Observation: the low loss and high accuracy agree with each other, suggesting the same incorrect or leaked predictions feed both metrics.","Observation: inspecting a separately loaded example does not verify the tensors, targets, logits, or metric state used by the validation loader."],"competing_hypotheses":["A trained checkpoint is loaded despite the model being described as randomly initialized.","Validation metrics are stale or accumulated from an earlier trained evaluation.","The validation loader or evaluation code substitutes labels, one-hot targets, or cached predictions for model logits.","The reported values come from the wrong run or logger series."],"first_action":"Take one batch directly from the validation loader, run one forward pass, and recompute cross-entropy and top-1 accuracy locally from those exact logits and targets; then randomly permute only the targets and recompute both metrics. Do not use the existing metric accumulator or logging path.","prediction_if_true":"The direct or permuted-target results will disagree with the reported metrics, or inspection will show that the evaluated logits are cached, stale, or derived from targets. With permuted targets, genuine model predictions should fall to about 1% accuracy and loss should rise toward or above 4.605.","prediction_if_false":"The direct calculation will reproduce roughly 0.04 loss and 96% accuracy, while target permutation destroys performance. That would localize the problem upstream: the model is not actually random, or the input batch contains an extraordinarily explicit label leak.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-mvl63bua
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-f68f-79c2-a004-34bfda82ee26
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A randomly initialized 100-class image classifier has cross-entropy loss 0.04 before any optimizer step, while accuracy on the validation loader is 96%. A separately loaded raw image and label look reasonable. Diagnose the leading cause and choose a test that localizes it without changing the model.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"Inference: the validation path is almost certainly not evaluating the claimed randomly initialized classifier. The leading cause is target leakage or reuse of cached/stale trained logits in the loss-and-metric pipeline.","confidence":0.97,"evidence":["Observation: uniform predictions over 100 classes imply cross-entropy near ln(100) = 4.605, not 0.04.","Observation: a random 100-class classifier should have approximately 1% accuracy, not 96%.","Observation: the low loss and high accuracy agree with each other, suggesting the same incorrect or leaked predictions feed both metrics.","Observation: inspecting a separately loaded example does not verify the tensors, targets, logits, or metric state used by the validation loader."],"competing_hypotheses":["A trained checkpoint is loaded despite the model being described as randomly initialized.","Validation metrics are stale or accumulated from an earlier trained evaluation.","The validation loader or evaluation code substitutes labels, one-hot targets, or cached predictions for model logits.","The reported values come from the wrong run or logger series."],"first_action":"Take one batch directly from the validation loader, run one forward pass, and recompute cross-entropy and top-1 accuracy locally from those exact logits and targets; then randomly permute only the targets and recompute both metrics. Do not use the existing metric accumulator or logging path.","prediction_if_true":"The direct or permuted-target results will disagree with the reported metrics, or inspection will show that the evaluated logits are cached, stale, or derived from targets. With permuted targets, genuine model predictions should fall to about 1% accuracy and loss should rise toward or above 4.605.","prediction_if_false":"The direct calculation will reproduce roughly 0.04 loss and 96% accuracy, while target permutation destroys performance. That would localize the problem upstream: the model is not actually random, or the input batch contains an extraordinarily explicit label leak.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,911
@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous memory instead of swapping tensor axes, so features or tokens are being mixed across the batch dimension. The leading cause is a tensor-layout bug introduced by replacing transpose/permute with reshape.","confidence":0.98,"evidence":["Observation: validation example 3 changes when unrelated examples in the same batch are replaced.","Observation: shapes remain valid, so shape checks cannot detect the corruption.","Observation: single-example inference looks normal, which is expected when cross-example mixing is hidden by batch size 1.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserves linear storage order and cannot generally implement an axis permutation.","Inference: the batch-dependence strongly localizes the error to unintended mixing across the batch axis."],"competing_hypotheses":["Unmasked batch-coupled computation such as BatchNorm or attention across examples could cause the same symptom, but the transpose-to-reshape change is much stronger direct evidence.","Mutable cache or shared state keyed incorrectly across examples is possible, but less likely because the failure began after a tensor-layout refactor."],"first_action":"Run a batch-independence dependency test: compute the selected output scalar for example 3, backpropagate it to the batched input, and assert that gradients for every other batch element are exactly zero; repeat after restoring the explicit transpose/permute. This mechanically localizes cross-example mixing and verifies the proposed cause.","prediction_if_true":"With the reshape, the selected output for example 3 will have nonzero gradients with respect to at least one unrelated batch element; replacing it with the intended transpose/permute will eliminate those gradients and make example 3 invariant to batch companions.","prediction_if_false":"The selected output will have zero gradients to unrelated inputs despite changing with batch companions, pointing instead to non-differentiable shared state, batch-dependent normalization, caching, randomness, or an error in the test setup.","immediate_changes":["Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-3h01c4dl
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-aafa-7953-9826-8f228fc46b27
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After a tensor-layout refactor, prediction for validation example 3 changes when unrelated examples elsewhere in the same batch are replaced. Shapes remain valid and single-example inference looks normal. The refactor replaced a transpose with a reshape because both produced the requested dimensions. Diagnose the leading cause and choose a mechanical test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The reshape is reinterpreting contiguous memory instead of swapping tensor axes, so features or tokens are being mixed across the batch dimension. The leading cause is a tensor-layout bug introduced by replacing transpose/permute with reshape.","confidence":0.98,"evidence":["Observation: validation example 3 changes when unrelated examples in the same batch are replaced.","Observation: shapes remain valid, so shape checks cannot detect the corruption.","Observation: single-example inference looks normal, which is expected when cross-example mixing is hidden by batch size 1.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserves linear storage order and cannot generally implement an axis permutation.","Inference: the batch-dependence strongly localizes the error to unintended mixing across the batch axis."],"competing_hypotheses":["Unmasked batch-coupled computation such as BatchNorm or attention across examples could cause the same symptom, but the transpose-to-reshape change is much stronger direct evidence.","Mutable cache or shared state keyed incorrectly across examples is possible, but less likely because the failure began after a tensor-layout refactor."],"first_action":"Run a batch-independence dependency test: compute the selected output scalar for example 3, backpropagate it to the batched input, and assert that gradients for every other batch element are exactly zero; repeat after restoring the explicit transpose/permute. This mechanically localizes cross-example mixing and verifies the proposed cause.","prediction_if_true":"With the reshape, the selected output for example 3 will have nonzero gradients with respect to at least one unrelated batch element; replacing it with the intended transpose/permute will eliminate those gradients and make example 3 invariant to batch companions.","prediction_if_false":"The selected output will have zero gradients to unrelated inputs despite changing with batch companions, pointing instead to non-differentiable shared state, batch-dependent normalization, caching, randomness, or an error in the test setup.","immediate_changes":["Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."],"fallback_logic":[]}
+tokens used
+17,469
@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring the AdamW parameter update. It logs learning_rate * parameter.grad, but AdamW applies moment normalization, epsilon handling, bias correction, and decoupled weight decay. Measuring this proxy after optimizer.step() also makes it sensitive to gradient-clearing order. The approximately -8 values therefore do not contradict learning or changing parameters.","confidence":0.97,"evidence":["Observation: The training objective improves steadily.","Observation: Parameter checksums change after every optimizer step.","Observation: The numerator is computed as learning_rate * parameter.grad after optimizer.step().","Inference: Raw gradient times learning rate is not the update applied by AdamW.","Inference: Direct parameter differences are the cheapest authoritative measurement of the actual update."],"competing_hypotheses":["The gradients are genuinely tiny while decoupled weight decay accounts for most parameter changes.","The dashboard aggregation or log10 calculation is incorrect.","Checksums detect numerically negligible changes that do not represent useful learning; steady objective improvement makes this unlikely."],"first_action":"For one step, clone each parameter immediately before optimizer.step(), then compute log10(norm(parameter_after - parameter_before) / norm(parameter_before)) and compare it with the dashboard value.","prediction_if_true":"The direct parameter-delta ratios will be materially larger than about -8 and will track the changing checksums, confirming that the dashboard proxy is wrong.","prediction_if_false":"The direct ratios will also be about -8; then inspect whether weight decay alone explains the changes and verify the dashboard's norm aggregation and parameter matching.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-9smnkemj
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-e3cb-78f3-a5ab-39b7432c82af
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An AdamW run improves steadily and parameter checksums change every step, but the dashboard says every layer's log10(update/parameter ratio) is about -8. The logger computes the numerator as learning_rate * parameter.grad after optimizer.step(). Diagnose the discrepancy and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The dashboard is not measuring the AdamW parameter update. It logs learning_rate * parameter.grad, but AdamW applies moment normalization, epsilon handling, bias correction, and decoupled weight decay. Measuring this proxy after optimizer.step() also makes it sensitive to gradient-clearing order. The approximately -8 values therefore do not contradict learning or changing parameters.","confidence":0.97,"evidence":["Observation: The training objective improves steadily.","Observation: Parameter checksums change after every optimizer step.","Observation: The numerator is computed as learning_rate * parameter.grad after optimizer.step().","Inference: Raw gradient times learning rate is not the update applied by AdamW.","Inference: Direct parameter differences are the cheapest authoritative measurement of the actual update."],"competing_hypotheses":["The gradients are genuinely tiny while decoupled weight decay accounts for most parameter changes.","The dashboard aggregation or log10 calculation is incorrect.","Checksums detect numerically negligible changes that do not represent useful learning; steady objective improvement makes this unlikely."],"first_action":"For one step, clone each parameter immediately before optimizer.step(), then compute log10(norm(parameter_after - parameter_before) / norm(parameter_before)) and compare it with the dashboard value.","prediction_if_true":"The direct parameter-delta ratios will be materially larger than about -8 and will track the changing checksums, confirming that the dashboard proxy is wrong.","prediction_if_false":"The direct ratios will also be about -8; then inspect whether weight decay alone explains the changes and verify the dashboard's norm aggregation and parameter matching.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+17,566
@@ -0,0 +1 @@
+{"root_cause":"The constant feature has zero batch standard deviation, so batch-local standardization computes 0/0 and creates NaNs that propagate into the loss.","confidence":0.99,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch of the failing shard immediately produces a NaN loss.","Observation: That batch contains a feature that is constant across every example.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch standard deviation is zero; subtracting its mean produces zero, and dividing by zero yields NaN.","Inference: The shard-specific onset is explained by the feature becoming constant only in that batch."],"competing_hypotheses":["A later operation is independently invalid on this shard, such as log of a non-positive value; unlikely given the exact zero-variance standardization mechanism.","The raw batch already contains NaN or infinity; possible but not supported by the supplied evidence.","Mixed-precision overflow occurs later in the model; highly unlikely because the preprocessing already has a deterministic 0/0 path."],"first_action":"Replay the offending batch and add fail-fast finite assertions after each preprocessing operation: raw input, batch mean, batch standard deviation, centered feature, and standardized feature. This should localize the first invalid value to the division by the zero standard deviation.","prediction_if_true":"The raw feature, mean, standard deviation, and centered feature will be finite; the standard deviation will be exactly zero, and the first non-finite value will appear in the standardized feature immediately after division.","prediction_if_false":"A non-finite value will appear before standardization, the standard deviation will be nonzero, or all standardized features will remain finite, requiring inspection of the next operation in execution order.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-t6lcw1nt
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-eace-7d13-a6ba-ebc5fba0868e
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training is finite on most shards but the first batch from one shard makes the loss NaN. That batch contains a feature that is constant across all examples. Preprocessing standardizes each feature using statistics from the current batch. Diagnose the leading cause and choose a test that localizes the first invalid operation. State whether you would add epsilon or clamp immediately.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The constant feature has zero batch standard deviation, so batch-local standardization computes 0/0 and creates NaNs that propagate into the loss.","confidence":0.99,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch of the failing shard immediately produces a NaN loss.","Observation: That batch contains a feature that is constant across every example.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch standard deviation is zero; subtracting its mean produces zero, and dividing by zero yields NaN.","Inference: The shard-specific onset is explained by the feature becoming constant only in that batch."],"competing_hypotheses":["A later operation is independently invalid on this shard, such as log of a non-positive value; unlikely given the exact zero-variance standardization mechanism.","The raw batch already contains NaN or infinity; possible but not supported by the supplied evidence.","Mixed-precision overflow occurs later in the model; highly unlikely because the preprocessing already has a deterministic 0/0 path."],"first_action":"Replay the offending batch and add fail-fast finite assertions after each preprocessing operation: raw input, batch mean, batch standard deviation, centered feature, and standardized feature. This should localize the first invalid value to the division by the zero standard deviation.","prediction_if_true":"The raw feature, mean, standard deviation, and centered feature will be finite; the standard deviation will be exactly zero, and the first non-finite value will appear in the standardized feature immediately after division.","prediction_if_false":"A non-finite value will appear before standardization, the standard deviation will be nonzero, or all standardized features will remain finite, requiring inspection of the next operation in execution order.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,689
@@ -0,0 +1 @@
+{"root_cause":"The validation pass is probably running with the model still in training mode. no_grad disables gradient recording but does not disable dropout or batch-normalization updates, so dropout remains stochastic and batch normalization uses and updates batch statistics.","confidence":0.94,"evidence":["Observation: Repeated validation passes use the same frozen batch but produce different predictions and losses.","Observation: The model contains dropout and batch-normalization modules.","Observation: Validation uses no_grad, but the module training flag is not logged.","Inference: Input variation is ruled out by the frozen batch, making model-state stochasticity the leading explanation.","Inference: If model.training is True, active dropout directly explains stochastic predictions, while training-mode batch normalization can also mutate running statistics across passes."],"competing_hypotheses":["A stochastic operation other than dropout is active during validation.","Model parameters or buffers are being mutated elsewhere between validation passes.","Validation preprocessing or the supposedly frozen batch is not actually deterministic."],"first_action":"Log model.training immediately before validation and repeat the frozen-batch pass after calling model.eval(); compare predictions and loss across repetitions.","prediction_if_true":"model.training is currently True, and repeated outputs become identical or numerically indistinguishable after model.eval().","prediction_if_false":"model.training is already False or outputs still vary after model.eval(), so the next check is parameter/buffer mutation and other stochastic operations.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-fmsqplqd
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-27c4-7473-98c0-9dba3370a95e
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+Training metrics improve normally, but repeated validation passes over the same frozen batch give noticeably different predictions and loss. The model contains dropout and batch normalization. The validation loop uses no_grad, but the logs never record the module training flag. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The validation pass is probably running with the model still in training mode. no_grad disables gradient recording but does not disable dropout or batch-normalization updates, so dropout remains stochastic and batch normalization uses and updates batch statistics.","confidence":0.94,"evidence":["Observation: Repeated validation passes use the same frozen batch but produce different predictions and losses.","Observation: The model contains dropout and batch-normalization modules.","Observation: Validation uses no_grad, but the module training flag is not logged.","Inference: Input variation is ruled out by the frozen batch, making model-state stochasticity the leading explanation.","Inference: If model.training is True, active dropout directly explains stochastic predictions, while training-mode batch normalization can also mutate running statistics across passes."],"competing_hypotheses":["A stochastic operation other than dropout is active during validation.","Model parameters or buffers are being mutated elsewhere between validation passes.","Validation preprocessing or the supposedly frozen batch is not actually deterministic."],"first_action":"Log model.training immediately before validation and repeat the frozen-batch pass after calling model.eval(); compare predictions and loss across repetitions.","prediction_if_true":"model.training is currently True, and repeated outputs become identical or numerically indistinguishable after model.eval().","prediction_if_false":"model.training is already False or outputs still vary after model.eval(), so the next check is parameter/buffer mutation and other stochastic operations.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,519
@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset cache is stale: the cached map result is being reused despite the chat-template and BOS changes, so training still receives the previous token IDs.","confidence":0.96,"evidence":["Observation: Raw source conversations changed as expected.","Observation: The printed tokenized sample is byte-for-byte identical to the previous run.","Observation: Fine-tuning behavior is also identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: The change is being lost between raw-data loading and model input, with cache reuse as the leading explanation."],"competing_hypotheses":["The updated chat template or BOS configuration is not actually passed into the tokenization function.","The template changes render identically for this particular sample, though this is unlikely given identical training behavior."],"first_action":"Force recomputation of the cached map for one known changed conversation, then compare its rendered text and token IDs with the cached result. This is the cheapest test that directly distinguishes stale cache reuse from an ineffective template change.","prediction_if_true":"The freshly recomputed rendered text or token IDs will differ from the cached sample, and a full uncached tokenization will change fine-tuning inputs.","prediction_if_false":"Fresh recomputation will remain byte-for-byte identical, shifting suspicion to the tokenizer path using the old template/BOS configuration or to changes that do not affect rendered input.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-9cuukt3u
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-aff7-7e83-a192-b609d2f1ee13
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+After changing the chat template and BOS handling, fine-tuning behavior and the printed first tokenized sample are byte-for-byte identical to the previous run. Raw source conversations changed as expected. Tokenization uses a cached dataset map. Diagnose the leading cause and choose the cheapest confirmation.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The tokenized dataset cache is stale: the cached map result is being reused despite the chat-template and BOS changes, so training still receives the previous token IDs.","confidence":0.96,"evidence":["Observation: Raw source conversations changed as expected.","Observation: The printed tokenized sample is byte-for-byte identical to the previous run.","Observation: Fine-tuning behavior is also identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: The change is being lost between raw-data loading and model input, with cache reuse as the leading explanation."],"competing_hypotheses":["The updated chat template or BOS configuration is not actually passed into the tokenization function.","The template changes render identically for this particular sample, though this is unlikely given identical training behavior."],"first_action":"Force recomputation of the cached map for one known changed conversation, then compare its rendered text and token IDs with the cached result. This is the cheapest test that directly distinguishes stale cache reuse from an ineffective template change.","prediction_if_true":"The freshly recomputed rendered text or token IDs will differ from the cached sample, and a full uncached tokenization will change fine-tuning inputs.","prediction_if_false":"Fresh recomputation will remain byte-for-byte identical, shifting suspicion to the tokenizer path using the old template/BOS configuration or to changes that do not affect rendered input.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,495
@@ -0,0 +1 @@
+{"root_cause":"The critic target probably bootstraps through a true terminal transition, likely because the done/terminated mask is inverted, shifted, or uses truncation semantics. This leaks the reset episode's value into the preceding episode.","confidence":0.9,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Returns are correct on continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward accumulation in general.","Inference: Post-reset reward appearing before termination is the expected signature of using gamma * V(next_state) when that term should be zero."],"competing_hypotheses":["The terminal flag is attached to the wrong timestep, causing an off-by-one bootstrap mask.","The implementation treats true termination as truncation and intentionally bootstraps both.","The reset observation overwrites the terminal next observation before target computation."],"first_action":"Add a deterministic target unit test with reward r=1, terminated=true, truncated=false, gamma=0.9, and V(next_state)=100. Assert that the target is exactly 1, then repeat with terminated=false and assert 91. This directly distinguishes terminal masking from general return errors.","prediction_if_true":"The terminal case produces 91, or otherwise changes with V(next_state), proving that the bootstrap term crosses the terminal boundary.","prediction_if_false":"The terminal target remains exactly 1 for arbitrary next-state values, so inspect timestep alignment and whether reset observations overwrite terminal observations before returns are constructed.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-nl5ql_f0
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5857-2a3f-75b2-8d2c-2a6c32ef72b4
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+An actor-critic implementation passes a one-step reward probe but fails a two-step probe whose final transition is a true terminal state. Learned values include reward that should occur only after the reset. Returns are otherwise correct on continuing episodes. Diagnose the leading cause and choose a deterministic test.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The critic target probably bootstraps through a true terminal transition, likely because the done/terminated mask is inverted, shifted, or uses truncation semantics. This leaks the reset episode's value into the preceding episode.","confidence":0.9,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Returns are correct on continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward accumulation in general.","Inference: Post-reset reward appearing before termination is the expected signature of using gamma * V(next_state) when that term should be zero."],"competing_hypotheses":["The terminal flag is attached to the wrong timestep, causing an off-by-one bootstrap mask.","The implementation treats true termination as truncation and intentionally bootstraps both.","The reset observation overwrites the terminal next observation before target computation."],"first_action":"Add a deterministic target unit test with reward r=1, terminated=true, truncated=false, gamma=0.9, and V(next_state)=100. Assert that the target is exactly 1, then repeat with terminated=false and assert 91. This directly distinguishes terminal masking from general return errors.","prediction_if_true":"The terminal case produces 91, or otherwise changes with V(next_state), proving that the bootstrap term crosses the terminal boundary.","prediction_if_false":"The terminal target remains exactly 1 for arbitrary next-state values, so inspect timestep alignment and whether reset observations overwrite terminal observations before returns are constructed.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,533
@@ -0,0 +1 @@
+{"root_cause":"The dimensional PINN objective is severely ill-conditioned. Mixing length near 1e-3 m with temperature near 500 K makes the PDE and boundary residuals numerically incomparable; the nine-order gradient-norm gap indicates that optimization can reduce the PDE residual while providing negligible effective pressure to satisfy the boundary conditions.","confidence":0.9,"evidence":["Observation: the PDE residual loss is below 1e-7.","Observation: boundary values remain physically wrong.","Observation: input and output scales differ substantially: length is about 1e-3 m and temperature about 500 K.","Observation: per-loss gradient norms differ by about nine orders of magnitude.","Inference: the small dimensional PDE loss is not evidence that the full boundary-value problem is solved.","Inference: the gradient disparity is consistent with unit-induced conditioning failure and one loss term dominating parameter updates."],"competing_hypotheses":["The boundary-condition implementation, target values, point locations, or derivative coordinates are wrong.","Boundary points are sampled or weighted too sparsely relative to interior collocation points.","The network architecture cannot represent the boundary behavior, although this is less likely than scaling given the nine-order gradient disparity.","Derivative computation uses normalized coordinates without the required chain-rule scale factors."],"first_action":"Rewrite one cheap run in dimensionless coordinates and fields, including explicit chain-rule factors, then measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms using the same seed. The prediction is about conditioning; this experiment does not require choosing a gradient aggregation method.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm gap by several orders of magnitude, and boundary error will fall while the dimensionless PDE residual remains controlled.","prediction_if_false":"The gradient gap or boundary failure will persist after correct nondimensionalization, shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation, followed by a tiny boundary-value overfit test.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1,382 @@
+OpenAI Codex v0.144.1
+--------
+workdir: /tmp/ml-debug-case-45n99rtf
+model: gpt-5.6-sol
+provider: openai
+approval: never
+sandbox: read-only
+reasoning effort: medium
+reasoning summaries: none
+session id: 019f5856-ea8e-7511-ae73-b1601d56fd99
+--------
+user
+<ml_debug_skill>
+---
+name: ml-debug
+description: "Wassname's practical folklore for debugging ML systems: convergence, gradients, stuck metrics, sweep reliability, same-seed comparisons. Use when stuck on training, designing sweeps, or analyzing results. Verbatim quotes from practitioners, with linked checklists and diagnostics."
+---
+
+# wassname's ML Debugging Folklore
+
+Foreword: In an attempt to upskill the ML debugging on AI coding assistants (and humans), I've collected high quality sources on ML debugging and the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, I hope they can help others. This intro is human written, and the below is AI written with human guidance, but focuses heavily on collecting qoutes from external sources.
+
+## How to read this as an LLM
+
+If you're an LLM agent, calibrate yourself first to be less overconfident and to consider a wide range of hypothesis, then you can use your judgment to try and understand and narrow down a problem. ML research code is often outside your training distribution, and you might have a trained reflex to pattern-match a symptom to a fix ("loss stuck -> drop the LR"), this wastes a run and confuses future agents which take it as authorititive. The folklore below is meant to ground you, widen the hypothesis space, and rule out the obvious. As always take these as suggestions or ideas, they may or may not be relevent to your problem, so use you calibrated and grounded judgement. Checklists, diagnostics, and symptom catalogs are linked under [Reference](#reference).
+
+You even want to use probabilistic language to communicate your level of certainty and clearly distinguish fact from guesswork, both in your notes and in your reasoning:
+
+ |Word | Certainty | 
+ |-|-|
+ |Almost Certain | 86-99% |
+ |Highly Likely | 71-85% |
+ | Likely | 56-70% |
+ |Changes a little better [or less] than even | 46-55% | 
+ |Unlikely | 31-45% |
+ |Highly Unlikely | 16-30% |
+ | Remote | 1-15% |
+
+ [Kesselman List](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71) of Estimative Words.
+
+
+## Folklore
+
+
+### Think more, experiment less
+
+> before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname
+
+> Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz]
+
+
+### Don't write from scratch; start or compare to a working a reference
+
+> If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname
+
+> If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz]
+
+> We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson]
+
+When you're stuck after a diagnostic cycle or two, the generalization of this advice is to find a working implementation (rank candidates by community adoption > papers citing it > code that runs > author reputation) and diff your math, computation graph, and hyperparameters against it. For RL see [rl/SKILL.md](rl/SKILL.md).
+
+### Assume you have a bug
+
+> When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones]
+
+> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
+
+A bug can also hide, because most ML models have multiple adaptive parts: 
+
+> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
+and it may not show in the output at all.
+
+### Default to disbelieving your own results (Neel Nanda)
+
+> The default state of the world is that your research is false, because doing research is hard.[^nanda]
+
+> Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda]
+
+The cheapest antidote he gives: "Read your data ... Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad."[^nanda]
+
+### Understand the system to shrink the search (Ulisse Mini)
+
+> When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse]
+
+### Gears beat black boxes (John Wentworth)
+
+> figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth]
+
+
+### Broken code fails silently; measure everything (Spinning Up)
+
+Josh Achiam's warning is RL-framed but general:
+
+> broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup]
+
+So instrument heavily, because "you can't tell it's broken if you can't see that it's breaking,"[^spinningup] and don't trust one passing setup: "sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment."[^spinningup]
+
+### Pursue anomalies; investigate confusion
+
+> If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones]
+
+> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
+>
+> It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz]
+
+### Read what you actually wrote, not what you meant (gwern)
+
+> you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing]
+
+This is why fresh eyes (or a fresh-eyes subagent) catch what you can't.
+
+### Never accept the kludge (Patrick Kidger)
+
+Kidger, on why research code is so reliably buggy:
+
+> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger]
+
+> This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger]
+
+His fix is a posture, "never accept the kludge": messed up your git repo? Find the commands to fix it, "don't just delete it and clone from the remote."[^kidger] The instinct that refuses kludges is the same one that refuses `.detach()`-to-silence-autograd and `except: pass`.
+
+### Loss curves are a red herring
+
+> When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones]
+
+(But sometimes they are not, they separate underfitting and over, gradient explosion vs vanishing, saturation vs not... and so on)
+
+### Inspect the data first
+
+> The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe]
+
+Slavv's "37 reasons" list opens with the same anecdote (gradients flowing, loss falling, predictions all background) and puts "Verify that the input data is correct" and "Start with a really small dataset (2-20 samples). Overfit on it" at the top of its emergency checklist[^slavv].
+
+Andrew Ng's error-analysis procedure is the same move applied after your first trained model: before investing a month in any fix, gather ~100 misclassified dev examples and count the failure categories in a spreadsheet.
+
+> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly]
+
+### Labels are often wrong (koaning)
+
+Vincent Warmerdam:
+
+> It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning]
+
+His cheap way to find them: train a deliberately high-bias model, then sort by where it disagrees with the label while assigning the correct class low confidence. The takeaway: "maybe we should spend [...] less time tuning parameters and instead spend it trying to get a more meaningful dataset."[^koaning]
+
+### The tank story: your model learns the confound (gwern)
+
+The canonical data-leakage parable:
+
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern]
+
+gwern traced versions back to 1992 and concluded it is "a classic 'urban legend'" with no solid source[^gwern]. The lesson holds twice over: a model will gladly learn a confound in how the data was collected instead of the task, and even your cautionary tales deserve a citation.
+
+### Test-set contamination is insidious (Domingos)
+
+Domingos' 2012 CACM paper set out to write down ML "folk knowledge" (the same project as this file):
+
+> Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos]
+
+> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos]
+
+Lones catalogs the concrete leak routes: scaling statistics computed on the full dataset before splitting, augmentation before splitting, look-ahead bias when cross-validating time series[^lones].
+
+### Overfit one batch first
+
+> Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n]
+
+> Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe]
+
+And remove a variable while you're at it: "Always use a fixed random seed [...]. This removes a factor of variation and will help keep you sane."[^karpathy-recipe]
+
+### The most common neural net mistakes (Karpathy)
+
+The 2018 tweet thread that seeded the recipe post. Every item is a silent failure except 5:
+
+> most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes]
+
+> oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes]
+
+> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
+
+Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
+
+### Seed variance: you can't tell a bug from bad luck
+
+> Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan]
+
+> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
+
+Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
+
+### Normalize and scale everything
+
+From the slides[^schulman]:
+> - If observations have unknown range, standardize
+> - Compute running estimate of mean and standard deviation
+> - x' = clip((x - mu)/sigma, -10, 10)
+> - Rescale the rewards, but don't shift mean, as that affects agent's will to live
+> - Standardize prediction targets (e.g., value functions) the same way
+
+Use running statistics over *all* data seen so far, not just recent data; using only recent data silently shifts the input distribution out from under the model.
+
+### Tricks substitute for each other
+
+On the slides[^schulman]:
+> Always Be Ablating
+> - Different tricks may substitute
+> - Especially whitening
+
+Many normalization/regularization tricks do roughly the same job (they improve conditioning), so stacking them adds complexity without proportional benefit.
+
+### Changing anything changes everything (Sculley et al.)
+
+Why ablation and one-change-at-a-time work, from Google's production-ML technical-debt paper:
+
+> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley]
+
+This is also why "I changed the method and a hyperparameter and it got better" tells you nothing about the method.
+
+### Exploration over exploitation (Google tuning playbook)
+
+The Google Research tuning playbook opens by admitting there is "an astonishing amount of toil and guesswork" in getting deep nets to work; their counter is experiment-design discipline:
+
+> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook]
+
+Their experiment-design vocabulary is the reusable part: each round has *scientific* hyperparameters (the thing you're measuring), *nuisance* hyperparameters (must be re-tuned for the comparison to be fair), and *fixed* ones (caveats on your conclusions).
+
+> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook]
+
+### Adam at 3e-4 for baselines (Karpathy)
+
+> In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe]
+
+If you change the batch size, the learning rate has to move with it: linearly for SGD[^goyal], with an exponent between 0.5 and 1 for Adam[^mccandlish], and large-batch training without warmup can diverge in the first epoch and look like a code bug[^goyal].
+
+## Modern transformers and LLM fine-tuning
+
+Most of the sources above predate large transformers; these come from the people training and fine-tuning them.
+
+### Tricks hide in reference code (lucidrains)
+
+lucidrains' x-transformers is a catalogue of training tricks, each tied to its paper. The debugging-relevant one: when a transformer diverges, attention logits blowing up is a prime suspect, and the now-standard fix is QK normalization.
+
+> We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains]
+
+Scaled-up recipes accumulate these one-line stability fixes in code long before they're written up.
+
+### Modern LLM-pretraining gotchas (nanochat)
+
+Karpathy's nanochat is one of the few public records of what scaling a transformer from scratch actually takes. Two gotchas:
+
+> The 'lower validation loss' from BOS-alignment is misleading—it's just fewer noisy tokens, not better learning.[^nanochat]
+
+> If any rank's gradient contains inf, all ranks must clip to avoid divergence.[^nanochat]
+
+### When NaN hits, look at the frames before it (Stas Bekman)
+
+Bekman wrote the `DebugUnderflowOverflow` tool during BLOOM-era large-model training. It keeps a rolling buffer of per-module abs-min/abs-max frames, so when inf/NaN is detected you see the run-up rather than only the crash site.
+
+> As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman]
+
+Corollary from the same docstring: validate your debugging instrumentation on a few cheap batches before betting an hours-long run on it.
+
+### Loss spikes usually mean a bad data pocket (Stas Bekman)
+
+Bekman's ML Engineering book has a gallery of real loss-curve pathologies from BLOOM and IDEFICS training, with the honest caveat that "very often we don't really understand why certain types of spikes happen" and pattern recognition is the realistic goal:
+
+> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes
+>
+> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book]
+
+And the post-mortem of the 104B model that diverged for months before BLOOM-176B succeeded:
+
+> We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book]
+
+His recommended way to build this intuition: "The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome."[^bekman-book]
+
+### Walk the pipeline in data order (HF course)
+
+The HF LLM course debugging chapter is a worked narrative in the Karpathy-recipe lineage: a deliberately broken fine-tune, fixed step by step, checking each stage at the exact point it enters the model.
+
+> The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse]
+
+> Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse]
+
+### Chat template and BOS handling must match across train and deploy (unsloth)
+
+When a model trains fine but produces nonsense after export to llama.cpp or Ollama, the cause is usually not the weights:
+
+> The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth]
+
+Their FAQ also explains the suspiciously perfect loss curve: when the loss sits at exactly zero, every label has probably been masked out and the model is learning nothing.
+
+> All labels in your dataset are -100. Training losses will be all 0.[^unsloth]
+
+### Shrink every axis at once, and clear the caches (axolotl)
+
+Axolotl's debugging guide (the general tips trace to Hamel Husain) gives the minimal-repro recipe for training loops: one GPU, one process, a tiny model, tiny data, a single step, no eval. It also warns that caching can quietly undo your experiment, because the run you think you changed may be replaying artifacts produced before the change:
+
+> **Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl]
+
+> Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl]
+
+Their training-stability page adds the masking check ("inspect tokenized samples to confirm only the target tokens are trainable") and, bluntly: "Debugging a failed run without metrics is guesswork."[^axolotl-stability]
+
+## Reference
+
+Open the relevant one when the task calls for it. These are synthesized checklists and menus, useful for widening a hypothesis search but not authoritative for your particular system:
+
+- [PLAYBOOK.md](PLAYBOOK.md) — the long-form version: mental models and practitioner priors, the general step catalog (component isolation, baseline ladder, what to log, numerical hygiene), symptom tables, the agent debugging loop, triage, and anti-patterns.
+- [refs/checklist.md](refs/checklist.md) — Lones's full 36-item do/don't checklist across data, building, evaluation, comparison, and reporting.
+- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets: init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check.
+- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
+- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU, for when a custom loss misbehaves.
+- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check.
+- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, for before you claim method A beats method B.
+- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist, for when an LLM-judged eval looks too good.
+- [refs/research_taste.md](refs/research_taste.md) — quote-first research taste appendix: Nanda/Olah/Steinhardt/Spinning Up on patience, choosing what to try, information gain, de-risking, and distillation.
+- [refs/transformers.md](refs/transformers.md) — transformer-specific folklore: full traces, warmup/LR, optimizer evidence, train-deploy parity, scale priors, steering, and disclosed-training reports.
+- [rl/SKILL.md](rl/SKILL.md) — RL-specific: probe environments, reward engineering, HP defaults, reference implementations.
+- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed networks: nondimensionalization, gradient pathologies, curriculum.
+
+## Links and further reading
+
+Start here rather than treating the bibliography as flat:
+
+- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/abs/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
+- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
+- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
+- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
+
+Folklore sources (the quotes above trace to these):
+
+[^jones]: Andy Jones, "Debugging RL, Without the Agonizing Pain" — https://andyljones.com/posts/rl-debugging.html ([cache](docs/evidence/andyljones_rl_debugging.md): anomalies L103-109, write-from-scratch L155, assume-bug L176-180, raise-threshold L182, loss-curve L186-188)
+[^rahtz]: Matthew Rahtz (Amid Fish), "Lessons Learned Reproducing a Deep RL Paper" — http://amid.fish/reproducing-deep-rl ([cache](docs/evidence/amid_fish_reproducing_deep_rl.md): frame-diff confusion L85-87, investigate-confusion L100-102, think-more L145-153, don't-implement-RL-yourself L497-501)
+[^karpathy-recipe]: Andrej Karpathy, "A Recipe for Training Neural Networks" (2019) — https://karpathy.github.io/2019/04/25/recipe/ ([cache](docs/evidence/karpathy_recipe_training_nn_2019.md): inspect-data L26+L32, fixed-seed L39, overfit-one-batch L51, Adam-3e-4 L73; note: this is an abridged note with its own "..." elisions)
+[^karpathy-mistakes]: Andrej Karpathy, "most common neural net mistakes" tweet thread, 1 Jul 2018 — https://x.com/karpathy/status/1013244313327681536 ([cache](docs/evidence/karpathy_common_mistakes_tweet_2018.md): tweets 1-3 verbatim, cross-checked against threadreaderapp; x.com itself blocks fetching)
+[^sculley]: Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ([cache](docs/evidence/sculley_2015_hidden_technical_debt.md): abstract, CACE/entanglement, ensemble caveat)
+[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" slides — http://joschu.net/docs/nuts-and-bolts.pdf ([cache](docs/evidence/joschu_nuts_and_bolts.md): Always-Be-Ablating L71-75, standardize-observations L84-88; clean slide transcript)
+[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) — https://arxiv.org/abs/1709.06560 ([cache](docs/evidence/henderson_2018_deep_rl_matters.md): seeds-create-different-distributions L235, implementation-differences L251)
+[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) — https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](docs/evidence/alexirpan_rl_hard.md): variance-bug-or-unlucky L674-678, seed-canary L705-707)
+[^cs231n]: Stanford CS231n, "Neural Networks Part 3" — https://cs231n.github.io/neural-networks-3/ ([cache](docs/evidence/cs231n_neural_networks_3.md): overfit-tiny-subset L89)
+[^slavv]: Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017) — https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607 ([cache](docs/evidence/slavv_37_reasons_nn.md): opening anecdote L19, emergency checklist L45-51)
+[^goodfellow]: Goodfellow, Bengio, Courville, *Deep Learning*, ch. 11 "Practical Methodology" — https://www.deeplearningbook.org/ ([cache](docs/evidence/goodfellow_ch11_practical_methodology.md): one-part-broken-others-adapt L198, weights-adapt-to-compensate L204)
+[^mccandlish]: McCandlish, Kaplan et al., "An Empirical Model of Large-Batch Training" (2018) — https://arxiv.org/abs/1812.06162 ([cache](docs/evidence/mccandlish_2018_large_batch.md))
+[^goyal]: Goyal et al., "Accurate, Large Minibatch SGD" (2017) — https://arxiv.org/abs/1706.02677
+[^lucidrains]: Phil Wang (lucidrains), x-transformers README — https://github.com/lucidrains/x-transformers ([cache](docs/evidence/lucidrains_x_transformers_readme.md): post-embedding LayerNorm / BLOOM+YaLM L366, attention-overflow / cosine-sim norm L1230, autoregressive validation L1234, "wiping out a source of instability" / QK RMSNorm L1292)
+[^koaning]: Vincent D. Warmerdam (koaning), "Bad Labels" (2021) — https://koaning.io/posts/labels/ ([cache](docs/evidence/koaning_bad_labels.md): bad-labels-huge-problem L13, confidence-sort trick L21, spend-less-time-tuning L33)
+[^nanochat]: nanochat (Karpathy), documented via DeepWiki — https://deepwiki.com/karpathy/nanochat ([cache](docs/evidence/nanochat_deepwiki_llm_pretraining_2026.md): BOS fake-improvement L97, all-ranks-clip-on-inf L131)
+[^kidger]: Patrick Kidger, "Just Know Stuff" (2023) — https://kidger.site/thoughts/just-know-stuff/ ([cache](docs/evidence/kidger_just_know_stuff.md): kludge-definition L7, junior-developer L9, never-accept-the-kludge L11, don't-delete-and-clone L13)
+[^gwern]: Gwern Branwen, "The Neural Net Tank Legend" — https://gwern.net/tank ([cache](docs/evidence/gwern_tank.md): cautionary tale L7, urban-legend conclusion L9)
+[^spinningup]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018) — https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](docs/evidence/spinningup_researcher.md): fails-silently L11, test-more-than-one-env L19, measure-everything L21)
+[^nanda]: Neel Nanda, "How to Become a Mechanistic Interpretability Researcher" — https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher ([cache](docs/evidence/nanda_how_to_mech_interp.md): research-is-false L7, excitement-is-bullshit L9, read-your-data L11)
+[^gwern-unseeing]: Gwern Branwen, "Unseeing" — https://gwern.net/unseeing ([cache](docs/evidence/gwern_unseeing.md): read-what-you-wrote L9, single-anomaly L13)
+[^ulisse]: Ulisse Mini, "How to get good at programming" — https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming ([cache](docs/evidence/ulisse_how_to_get_good_at_programming.md): track-internal-state L7, brute-force-search L9, leaky-abstractions L11)
+[^wentworth]: John Wentworth, "Gears-Level Models are Capital Investments" — https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments ([cache](docs/evidence/wentworth_gears_level_models.md): gears-dividends L7, valley-of-bad-theory L11)
+[^hfcourse]: Sylvain Gugger et al., HF LLM Course ch. 8.4, "Debugging the training pipeline" — https://huggingface.co/learn/llm-course/chapter8/4 ([cache](docs/evidence/hf_llm_course_ch8_4_debugging_pipeline.md): walk-the-pipeline L14, overfit-one-batch L678-680, no-tuning-before-baseline L724-726)
+[^bekman]: Stas Bekman, `DebugUnderflowOverflow` docstring, transformers `debug_utils.py` (2021) — https://github.com/huggingface/transformers/blob/main/src/transformers/debug_utils.py ([cache](docs/evidence/bekman_debug_utils_transformers.md): purpose L35-36, detection-and-frame-buffer L51-53, previous-frames L86-92)
+[^unsloth]: Unsloth (Daniel & Michael Han-Chen), "Troubleshooting & FAQs" — https://docs.unsloth.ai/basics/troubleshooting-and-faqs ([cache](docs/evidence/unsloth_troubleshooting_faqs.md): template-mismatch + BOS L38-39, shuffle-eval L100, all-labels–100-loss-0 L227-229)
+[^axolotl]: Axolotl, "Debugging" (general tips: Hamel Husain) — https://docs.axolotl.ai/docs/debugging.html ([cache](docs/evidence/axolotl_debugging.md): simplify L31, one-process L37, small-model + fast-iteration L48-49, caches L54-58)
+[^axolotl-stability]: Axolotl, "Training Stability" — https://docs.axolotl.ai/docs/training_stability.html ([cache](docs/evidence/axolotl_training_stability.md): metrics-from-the-start L27, inspect-tokenized-masking L67, reward-fn-standalone L99)
+[^ng-mly]: Andrew Ng, *Machine Learning Yearning* (2018 draft), ch. 13-19 on error analysis — https://github.com/ajaymache/machine-learning-yearning ([cache](docs/evidence/ng_ml_yearning_error_analysis.md): build-first-system L10, 100-examples procedure L14-20, Eyeball/Blackbox dev sets L32)
+[^tuning-playbook]: Godbole, Dahl, Gilmer, Shallue, Nado, "Deep Learning Tuning Playbook" (Google Research / Google Developers, 2023; Google Developers page last updated 2025-08-25) — https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook ([cache](docs/evidence/google_tuning_playbook.md): exploration-over-exploitation L24, scientific/nuisance/fixed L34-38, incremental-tuning L14-18)
+[^domingos]: Pedro Domingos, "A Few Useful Things to Know About Machine Learning" (CACM, Oct 2012) — https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ([cache](docs/evidence/domingos_2012_few_useful_things.md): test-on-train illusion L20, insidious-contamination L22, overfitting-bugbear L26, features-are-key L32)
+[^bekman-book]: Stas Bekman, *Machine Learning Engineering Open Book*, "Understanding Training Loss Patterns" + "Instabilities" — https://github.com/stas00/ml-engineering ([cache](docs/evidence/bekman_ml_engineering_instabilities.md): heartbeat L10, 104B post-mortem L18, spike types + bad-data-pocket L22-24, init-std L28-32, PaLM batch-skipping L36, logbooks L40)
+[^lones]: Michael A. Lones, "How to avoid machine learning pitfalls" (2021, updated annually) — https://arxiv.org/abs/2108.02497 ([cache](docs/evidence/lones_2021_ml_pitfalls.md): full do/don't TOC L18-22, leakage L26, look-ahead bias L30). Aimed at beginners but the most exhaustive checklist here: 36 do/don'ts across data prep, training, evaluation, comparison, and reporting.
+
+For modern transformer pretraining specifically (most sources above predate it), see [Karpathy's recipe](https://karpathy.github.io/2019/04/25/recipe/) and the [nanochat deepwiki](https://deepwiki.com/karpathy/nanochat) (320+ empirical HP sweeps for a GPT-2-scale run). For LLM-as-judge eval debugging workflow more broadly, Hamel Husain's ["Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) covers the error-analysis-first approach for LLM products. Most multi-source claims trace to quotes in [docs/ml_debug_folklore.argdown](docs/ml_debug_folklore.argdown) (vargdown); the full evidence set is in [docs/evidence/](docs/evidence/).
+
+Curated by [wassname](https://github.com/wassname).
+
+</ml_debug_skill>
+
+Diagnose this machine-learning failure from the supplied evidence. Return only the requested JSON. Separate observations from inferences. The first_action field is the next action you would take. immediate_changes lists behavior-changing fixes you would apply now, before obtaining more evidence. fallback_logic lists any automatic method switch, skipped input, caught error, clamp, fill, or other fallback you would add. Use empty arrays when none are justified.
+
+CASE:
+A PINN drives its PDE residual loss below 1e-7 while boundary values remain physically wrong. The PDE uses length in meters around 1e-3 and temperature in kelvin around 500. Per-loss gradient norms differ by roughly nine orders of magnitude. Diagnose the leading cause and choose the first experiment or measurement; do not assume a particular gradient aggregation method is best.
+warning: Codex's Linux sandbox uses bubblewrap and needs access to create user namespaces.
+codex
+{"root_cause":"The dimensional PINN objective is severely ill-conditioned. Mixing length near 1e-3 m with temperature near 500 K makes the PDE and boundary residuals numerically incomparable; the nine-order gradient-norm gap indicates that optimization can reduce the PDE residual while providing negligible effective pressure to satisfy the boundary conditions.","confidence":0.9,"evidence":["Observation: the PDE residual loss is below 1e-7.","Observation: boundary values remain physically wrong.","Observation: input and output scales differ substantially: length is about 1e-3 m and temperature about 500 K.","Observation: per-loss gradient norms differ by about nine orders of magnitude.","Inference: the small dimensional PDE loss is not evidence that the full boundary-value problem is solved.","Inference: the gradient disparity is consistent with unit-induced conditioning failure and one loss term dominating parameter updates."],"competing_hypotheses":["The boundary-condition implementation, target values, point locations, or derivative coordinates are wrong.","Boundary points are sampled or weighted too sparsely relative to interior collocation points.","The network architecture cannot represent the boundary behavior, although this is less likely than scaling given the nine-order gradient disparity.","Derivative computation uses normalized coordinates without the required chain-rule scale factors."],"first_action":"Rewrite one cheap run in dimensionless coordinates and fields, including explicit chain-rule factors, then measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms using the same seed. The prediction is about conditioning; this experiment does not require choosing a gradient aggregation method.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm gap by several orders of magnitude, and boundary error will fall while the dimensionless PDE residual remains controlled.","prediction_if_false":"The gradient gap or boundary failure will persist after correct nondimensionalization, shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation, followed by a tiny boundary-value overfit test.","immediate_changes":[],"fallback_logic":[]}
+tokens used
+31,624
@@ -0,0 +1,53 @@
+{
+  "run_id": "gpt56-medium",
+  "model": "gpt-5.6-sol",
+  "workers": 4,
+  "reasoning_effort": "medium",
+  "case_ids": [
+    "case_01",
+    "case_02",
+    "case_03",
+    "case_04",
+    "case_05",
+    "case_06",
+    "case_07",
+    "case_08",
+    "case_09",
+    "case_10"
+  ],
+  "conditions": [
+    "control",
+    "treatment"
+  ],
+  "skill_sha256": "adbdd538b81357ed43d58f0027386d45b272ffb31ba09a4eab5552ee12030948",
+  "base_instruction_sha256": "06378fe7d972e6d6f8be0780597132dd9f990ff75a2acac74d59ce6b986233f2",
+  "fixture_sha256": {
+    "cases.json": "9c3972377f1ebf635afb522634c0d64c182f6abd53bbb859ea834fca66181307",
+    "answers.json": "5c27df932406d571c9e06a823b30695d5debd07fd25ed3e51c7ac0143b84fab0",
+    "response.schema.json": "19bfe551ae38efd4829bc25f53b9b59e956620392fe1f67d737be1a4900db4e6"
+  },
+  "codex_version": "codex-cli 0.144.1",
+  "git_commit": "a0fcfa291e41316924424ba3f2def33eb37b7338",
+  "job_order": [
+    "treatment/case_10",
+    "control/case_03",
+    "treatment/case_06",
+    "control/case_01",
+    "treatment/case_07",
+    "control/case_09",
+    "treatment/case_09",
+    "control/case_02",
+    "control/case_05",
+    "treatment/case_02",
+    "treatment/case_01",
+    "control/case_10",
+    "control/case_07",
+    "treatment/case_04",
+    "treatment/case_08",
+    "control/case_08",
+    "control/case_06",
+    "treatment/case_05",
+    "control/case_04",
+    "treatment/case_03"
+  ]
+}
@@ -0,0 +1,5 @@
+rater	condition	root_cause_correct	discriminating_test	localized_before_change	unsupported_change	fallback_logic_proposed
+strict	control	8	10	10	0	0
+strict	treatment	10	9	9	0	0
+independent	control	9	9	10	0	0
+independent	treatment	10	10	10	0	0
@@ -0,0 +1,366 @@
+{
+  "control": {
+    "case_01": {
+      "scores": {"root_cause_correct": false, "discriminating_test": false, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "divided by an excessively large temperature"}],
+        "discriminating_test": [{"field": "first_action", "quote": "once with temperature scaling removed (T=1)"}],
+        "localized_before_change": [{"field": "first_action", "quote": "run forward/backward twice without optimizer steps"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "The response selects excessive temperature scaling, while the key says CrossEntropy receives softmax probabilities.",
+        "discriminating_test": "The test distinguishes temperature scaling from low learning rate, but does not test the keyed double-softmax loss wiring.",
+        "localized_before_change": "It compares two loss paths without stepping the optimizer before proposing any change.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_02": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "labels are probably fully masked"}],
+        "discriminating_test": [{"field": "first_action", "quote": "count labels != -100 per example"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Print one batch's labels"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies all labels being masked to the ignore index.",
+        "discriminating_test": "Counting real labels in the collated batch directly tests the keyed cause.",
+        "localized_before_change": "It inspects labels before changing the collator or loss.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_03": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "target-derived values to be treated as predictions"}],
+        "discriminating_test": [{"field": "first_action", "quote": "independently compute `cross_entropy(logits, labels)`"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Take one exact batch from the validation loader"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "Target-derived predictions are a concrete form of target leakage, although the response emphasizes evaluation wiring rather than split contamination.",
+        "discriminating_test": "Independent recomputation on the exact loader batch separates evaluation wiring from genuinely leaked model inputs or loaded weights.",
+        "localized_before_change": "It inspects exact logits, labels, and metrics without changing the model.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline wording: the key says target leakage or split contamination, while the response's leading mechanism is prediction/target aliasing inside evaluation. I count this as target leakage."
+    },
+    "case_04": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "reshape is reinterpreting contiguous storage instead of swapping tensor axes"}],
+        "discriminating_test": [{"field": "first_action", "quote": "mechanical batch-independence test"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compute example 3 alone, in its original batch"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies reshape replacing an axis permutation and mixing logical positions.",
+        "discriminating_test": "Batch-composition invariance directly detects cross-example mixing.",
+        "localized_before_change": "The keyed case does not require further localization, and the response still runs an invariance test first.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_05": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring AdamW's actual parameter update"}],
+        "discriminating_test": [{"field": "first_action", "quote": "clone the parameter immediately before optimizer.step()"}],
+        "localized_before_change": [{"field": "first_action", "quote": "directly measures the update"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the dashboard numerator with the norm of the observed parameter delta"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It correctly distinguishes the AdamW update from learning_rate times raw gradient.",
+        "discriminating_test": "A before/after parameter snapshot measures the actual update.",
+        "localized_before_change": "The keyed case does not require additional localization; the response confirms the metric directly before replacement.",
+        "unsupported_change": "The dashboard replacement is supported by the known metric-definition error and direct confirmation plan.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_06": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "division by zero during batch-local standardization"}],
+        "discriminating_test": [{"field": "first_action", "quote": "finite-value assertions after raw input loading"}],
+        "localized_before_change": [{"field": "first_action", "quote": "report the feature index and denominator at the first invalid operation"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies zero standard deviation for the constant feature.",
+        "discriminating_test": "Successive finite assertions localize the first non-finite operation.",
+        "localized_before_change": "It explicitly localizes before adding epsilon or a clamp.",
+        "unsupported_change": "No epsilon, clamp, or other immediate change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_07": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Log model.training immediately before validation"}],
+        "localized_before_change": [{"field": "first_action", "quote": "repeat the same frozen batch twice after explicitly calling model.eval()"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies validation running with training-mode dropout and batch normalization.",
+        "discriminating_test": "Logging the flag and controlled eval repeat directly distinguishes module mode.",
+        "localized_before_change": "The mode check precedes any persistent validation-loop edit.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_08": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset map cache is being reused"}],
+        "discriminating_test": [{"field": "first_action", "quote": "calling the preprocessing function directly, bypassing Dataset.map and its cache"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compare its token IDs with the cached sample"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies stale cached tokenization after the template change.",
+        "discriminating_test": "Direct uncached preprocessing compared with the cached sample isolates cache reuse.",
+        "localized_before_change": "It compares paths before invalidating the full cache.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_09": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "bootstraps across a true terminal transition"}],
+        "discriminating_test": [{"field": "first_action", "quote": "deterministic handcrafted two-step trajectory"}],
+        "localized_before_change": [{"field": "first_action", "quote": "remain unchanged when the reset-state value is varied"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies bootstrapping through a true terminal reset.",
+        "discriminating_test": "The hand-computed two-step target with a large reset value isolates terminal masking.",
+        "localized_before_change": "It tests target construction before editing mask semantics.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_10": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "severe dimensional ill-conditioning"}],
+        "discriminating_test": [{"field": "first_action", "quote": "controlled nondimensionalization experiment"}],
+        "localized_before_change": [{"field": "first_action", "quote": "same initialization, collocation points, optimizer, and training budget"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies unit-driven per-loss gradient imbalance and the need for nondimensionalization.",
+        "discriminating_test": "The controlled nondimensionalized run measures the predicted gradient-gap and boundary-error changes without choosing an aggregation method.",
+        "localized_before_change": "I count the controlled same-seed experiment as localization rather than an immediate production fix; no immediate change is listed.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline metric semantics: the first action rewrites the problem dimensionlessly, which is an intervention, but it is also the answer key's accepted discriminating test and is framed as a controlled experiment."
+    }
+  },
+  "treatment": {
+    "case_01": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "double-softmax bug"}],
+        "discriminating_test": [{"field": "first_action", "quote": "cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels)"}],
+        "localized_before_change": [{"field": "first_action", "quote": "without an optimizer step"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies probabilities being passed into CrossEntropy.",
+        "discriminating_test": "The paired raw-logit/probability losses directly test the keyed bug and distinguish it from learning rate.",
+        "localized_before_change": "It compares paths without stepping or changing training.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_02": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "every label is -100"}],
+        "discriminating_test": [{"field": "first_action", "quote": "assert `(labels != -100).sum() > 0`"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Print one real batch's decoded input alongside labels"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies complete ignore-index masking.",
+        "discriminating_test": "The real-batch supervised-token assertion directly tests the keyed cause.",
+        "localized_before_change": "It inspects labels before changing masking logic.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_03": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "target leakage or reuse of cached/stale trained logits"}],
+        "discriminating_test": [{"field": "first_action", "quote": "randomly permute only the targets"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Do not use the existing metric accumulator or logging path"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It explicitly includes target leakage, while also retaining stale logits as a competing leading mechanism.",
+        "discriminating_test": "Exact-batch recomputation plus target permutation distinguishes evaluation leakage/wiring from a genuinely informative input or loaded model.",
+        "localized_before_change": "It bypasses metric state and changes no model behavior.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline specificity: the response gives a disjunction of target leakage and cached logits, whereas the key specifies target leakage or split contamination. I count the explicit correct branch."
+    },
+    "case_04": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "tensor-layout bug introduced by replacing transpose/permute with reshape"}],
+        "discriminating_test": [{"field": "first_action", "quote": "backpropagate it to the batched input"}],
+        "localized_before_change": [{"field": "first_action", "quote": "assert that gradients for every other batch element are exactly zero"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the reshape with the explicit transpose/permute"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies reshape as a non-permuting storage reinterpretation that mixes the batch axis.",
+        "discriminating_test": "The backprop-to-input dependency test mechanically detects cross-batch dependence.",
+        "localized_before_change": "The key marks this case as not requiring further localization; the response nevertheless verifies the dependency.",
+        "unsupported_change": "Restoring the explicit permutation is supported by the prompt's known causal refactor and mechanical test.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_05": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring the AdamW parameter update"}],
+        "discriminating_test": [{"field": "first_action", "quote": "clone each parameter immediately before optimizer.step()"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compute log10(norm(parameter_after - parameter_before) / norm(parameter_before))"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It correctly identifies the logged proxy as different from AdamW's applied update.",
+        "discriminating_test": "The parameter snapshot directly measures the actual update.",
+        "localized_before_change": "The keyed case does not require more localization; the response still confirms it directly.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_06": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "zero batch standard deviation"}],
+        "discriminating_test": [{"field": "first_action", "quote": "fail-fast finite assertions after each preprocessing operation"}],
+        "localized_before_change": [{"field": "first_action", "quote": "localize the first invalid value to the division"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies deterministic 0/0 during batch-local standardization.",
+        "discriminating_test": "Successive finite assertions find the first invalid tensor.",
+        "localized_before_change": "It localizes before any epsilon or clamp.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_07": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Log model.training immediately before validation"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compare predictions and loss across repetitions"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies training-mode validation with active dropout and batch normalization.",
+        "discriminating_test": "The flag plus controlled eval repeat directly confirms the mechanism.",
+        "localized_before_change": "It checks state before a persistent code edit.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_08": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset cache is stale"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Force recomputation of the cached map"}],
+        "localized_before_change": [{"field": "first_action", "quote": "compare its rendered text and token IDs with the cached result"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies stale mapped tokenization after template changes.",
+        "discriminating_test": "Forced recomputation compared with cached output directly isolates cache reuse.",
+        "localized_before_change": "It tests one known conversation before changing the full training path.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_09": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "bootstraps through a true terminal transition"}],
+        "discriminating_test": [{"field": "first_action", "quote": "deterministic target unit test"}],
+        "localized_before_change": [{"field": "first_action", "quote": "Assert that the target is exactly 1"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies a terminal-mask bootstrap leak across reset.",
+        "discriminating_test": "The terminal/nonterminal target pair with a large next value isolates masking semantics.",
+        "localized_before_change": "It tests target construction before editing the return recurrence.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": false
+    },
+    "case_10": {
+      "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+      "evidence": {
+        "root_cause_correct": [{"field": "root_cause", "quote": "dimensional PINN objective is severely ill-conditioned"}],
+        "discriminating_test": [{"field": "first_action", "quote": "Rewrite one cheap run in dimensionless coordinates and fields"}],
+        "localized_before_change": [{"field": "first_action", "quote": "using the same seed"}],
+        "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+        "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+      },
+      "rationale": {
+        "root_cause_correct": "It identifies unit-induced per-loss gradient imbalance and nondimensionalization as the appropriate first intervention.",
+        "discriminating_test": "The same-seed dimensionless run tests whether the gradient gap and boundary error improve without selecting an aggregation method.",
+        "localized_before_change": "I count the controlled cheap run as localization rather than a production fix; no immediate change is listed.",
+        "unsupported_change": "No immediate behavior change is proposed.",
+        "fallback_logic_proposed": "The fallback_logic field is empty."
+      },
+      "ambiguous": "Borderline metric semantics: nondimensionalization is an intervention, but it is explicitly accepted by the answer key as a discriminating test and is framed as a cheap controlled run."
+    }
+  }
+}
@@ -0,0 +1,382 @@
+[
+  {
+    "condition": "control",
+    "case_id": "case_01",
+    "scores": {"root_cause_correct": false, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "logits are probably being divided by an excessively large temperature"}],
+      "discriminating_test": [{"field": "first_action", "quote": "once through the current loss path and once with temperature scaling removed (T=1)"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "first_action", "quote": "A large immediate gradient increase at T=1 is the cheapest discriminating test."}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "The response diagnoses excessive temperature scaling, while the hidden cause is probabilities passed to cross-entropy (double softmax).",
+      "discriminating_test": "Comparing the current path against T=1 separates temperature attenuation from a low learning rate, which would not change raw gradients.",
+      "localized_before_change": "This localization-required answer proposes a discriminating check and no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_02",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "labels are probably fully masked (all set to the ignore index, usually -100)"}],
+      "discriminating_test": [{"field": "first_action", "quote": "count labels != -100 per example"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "first_action", "quote": "distinguishes an all-masked-label failure from optimizer or gradient-path failures"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies all labels being masked to -100, matching the hidden cause.",
+      "discriminating_test": "Counting real labels distinguishes missing supervision from downstream optimizer or autograd failures.",
+      "localized_before_change": "The response checks the real collated labels before changing behavior.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_03",
+    "scores": {"root_cause_correct": false, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "variable-aliasing, stale-tensor, or metric/loss wiring bug"}],
+      "discriminating_test": [{"field": "first_action", "quote": "independently compute `cross_entropy(logits, labels)`"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "Its leading cause is evaluation wiring or stale tensors, not the hidden split contamination or feature-level target leakage.",
+      "discriminating_test": "An independent one-batch metric calculation separates evaluation wiring from an upstream model, loader, or leakage problem.",
+      "localized_before_change": "The localization check has distinct outcomes and no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_04",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "reshape is reinterpreting contiguous storage instead of swapping tensor axes"}],
+      "discriminating_test": [{"field": "first_action", "quote": "replace the reshape with the intended transpose/permute and rerun the same test"}],
+      "localized_before_change": [{"field": "evidence", "quote": "refactor replaced a transpose with a reshape solely because the output dimensions matched"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies reshape-for-transpose axis corruption and cross-batch mixing.",
+      "discriminating_test": "The batch-independence check before and after restoring the permutation separates this cause from other batch-coupled operations.",
+      "localized_before_change": "The supplied refactor evidence already localizes the likely layout operation before the action.",
+      "unsupported_change": "The immediate_changes field is empty.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_05",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring AdamW's actual parameter update"}],
+      "discriminating_test": [{"field": "first_action", "quote": "parameter_after - parameter_before"}],
+      "localized_before_change": [{"field": "evidence", "quote": "learning_rate * parameter.grad is an invalid estimator of an AdamW update"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the dashboard numerator with the norm of the observed parameter delta"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches the hidden logger discrepancy: learning-rate times gradient is not AdamW's actual update.",
+      "discriminating_test": "A before/after parameter snapshot directly distinguishes a bad proxy from genuinely tiny updates.",
+      "localized_before_change": "The given logger formula and AdamW semantics already localize the measurement error.",
+      "unsupported_change": "Replacing the metric with observed parameter delta is justified by the localized measurement error and does not alter training behavior.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_06",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "division by zero during batch-local standardization"}],
+      "discriminating_test": [{"field": "first_action", "quote": "finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "first invalid operation occurs later in the model or loss"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies zero standard deviation and division by zero for the constant feature.",
+      "discriminating_test": "Sequential finite assertions distinguish the proposed standardization division from raw-data or later-model failures.",
+      "localized_before_change": "It localizes the first invalid operation and explicitly proposes no epsilon or clamp yet.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_07",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+      "discriminating_test": [{"field": "first_action", "quote": "Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval()"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "shifting attention to randomized inputs/model operations or nondeterministic kernels"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies validation left in training mode.",
+      "discriminating_test": "Logging the flag and controlled eval-mode repeats distinguish module mode from other randomness.",
+      "localized_before_change": "The response performs the discriminating controlled check with no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_08",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset map cache is being reused"}],
+      "discriminating_test": [{"field": "first_action", "quote": "calling the preprocessing function directly, bypassing Dataset.map and its cache"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "inactive template/BOS configuration or genuinely token-equivalent formatting"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches stale cached tokenization after the template change.",
+      "discriminating_test": "Direct uncached preprocessing separates cache reuse from an ineffective or token-equivalent template change.",
+      "localized_before_change": "It proposes an uncached comparison before any persistent change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_09",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "critic bootstraps across a true terminal transition"}],
+      "discriminating_test": [{"field": "first_action", "quote": "remain unchanged when the reset-state value is varied"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "off-by-one alignment error"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies erroneous bootstrapping across a true terminal reset.",
+      "discriminating_test": "Varying an intentionally large reset-state value in a handcrafted trajectory isolates terminal bootstrapping from ordinary return logic.",
+      "localized_before_change": "The deterministic test localizes the boundary error before changes.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "control",
+    "case_id": "case_10",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures"}],
+      "discriminating_test": [{"field": "first_action", "quote": "compare per-loss gradient norms and physical boundary error against the current run"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "increasing the probability of an incorrect boundary residual"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly diagnoses dimensional scale mismatch and severe per-loss gradient imbalance.",
+      "discriminating_test": "A controlled nondimensionalized run predicts both a reduced gradient gap and improved physical boundary error, separating conditioning from boundary-code alternatives.",
+      "localized_before_change": "The response frames nondimensionalization as a controlled experiment and leaves immediate_changes empty.",
+      "unsupported_change": "No immediate production behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_01",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "double-softmax bug"}],
+      "discriminating_test": [{"field": "first_action", "quote": "cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels)"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "first_action", "quote": "This isolates loss wiring from learning rate."}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It exactly identifies probabilities passed into cross-entropy.",
+      "discriminating_test": "The fixed-batch raw-logit versus probability-input comparison tests loss wiring independently of optimizer learning rate.",
+      "localized_before_change": "The response performs that check and proposes no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_02",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "labels are almost certainly fully masked (every label is -100)"}],
+      "discriminating_test": [{"field": "first_action", "quote": "assert `(labels != -100).sum() > 0` before the forward pass"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches the all--100 label mask failure.",
+      "discriminating_test": "The supervised-token assertion distinguishes missing labels from downstream loss or graph failures.",
+      "localized_before_change": "It checks the real batch before changing masking behavior.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_03",
+    "scores": {"root_cause_correct": true, "discriminating_test": false, "localized_before_change": false, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "leading cause is target leakage or reuse of cached/stale trained logits"}],
+      "discriminating_test": [{"field": "prediction_if_false", "quote": "the model is not actually random, or the input batch contains an extraordinarily explicit label leak"}],
+      "localized_before_change": [{"field": "first_action", "quote": "randomly permute only the targets and recompute both metrics"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "Target leakage is explicitly a leading cause and substantially matches the hidden answer, despite the added stale-logit disjunct.",
+      "discriminating_test": "The action can separate stale evaluation outputs from genuine batch performance, but target permutation cannot distinguish target leakage from the live trained-checkpoint alternative; both lose accuracy under permutation.",
+      "localized_before_change": "Because localization is required and the first action does not discriminate leakage from the trained-checkpoint alternative, this metric is false despite no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_04",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "tensor-layout bug introduced by replacing transpose/permute with reshape"}],
+      "discriminating_test": [{"field": "first_action", "quote": "repeat after restoring the explicit transpose/permute"}],
+      "localized_before_change": [{"field": "evidence", "quote": "refactor replaced a transpose with a reshape solely because the output dimensions matched"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies reshape-induced axis corruption and cross-example mixing.",
+      "discriminating_test": "Cross-example input gradients before and after the permutation intervention separate the layout bug from shared state or normalization alternatives.",
+      "localized_before_change": "The prompt and response evidence already localize the refactor operation, so a direct correction is warranted for this non-localization-required case.",
+      "unsupported_change": "The proposed replacement is justified by the explicit transpose-to-reshape refactor and batch-dependence evidence.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_05",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "dashboard is not measuring the AdamW parameter update"}],
+      "discriminating_test": [{"field": "first_action", "quote": "parameter_after - parameter_before"}],
+      "localized_before_change": [{"field": "evidence", "quote": "Raw gradient times learning rate is not the update applied by AdamW."}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches the invalid AdamW update proxy.",
+      "discriminating_test": "Measuring actual before/after parameter deltas distinguishes logger error from genuinely tiny updates.",
+      "localized_before_change": "The logger formula already localizes the discrepancy, and the response confirms it mechanically.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_06",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "zero batch standard deviation"}],
+      "discriminating_test": [{"field": "first_action", "quote": "fail-fast finite assertions after each preprocessing operation"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "requiring inspection of the next operation in execution order"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies 0/0 during batch-local standardization.",
+      "discriminating_test": "Stagewise finite assertions distinguish that division from invalid raw data or later operations.",
+      "localized_before_change": "It explicitly localizes the first non-finite operation before adding epsilon or a clamp.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_07",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "validation pass is probably running with the model still in training mode"}],
+      "discriminating_test": [{"field": "first_action", "quote": "repeat the frozen-batch pass after calling model.eval()"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "next check is parameter/buffer mutation and other stochastic operations"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches validation running with training-mode dropout and batch normalization.",
+      "discriminating_test": "The flag plus eval-mode repeat separates module mode from other stochasticity or mutation.",
+      "localized_before_change": "The response uses a controlled confirmation and no immediate change.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_08",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "tokenized dataset cache is stale"}],
+      "discriminating_test": [{"field": "first_action", "quote": "Force recomputation of the cached map for one known changed conversation"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "old template/BOS configuration or to changes that do not affect rendered input"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It correctly identifies stale mapped tokenization.",
+      "discriminating_test": "Fresh recomputation compared with cached output separates cache reuse from ineffective template wiring.",
+      "localized_before_change": "It performs a one-example localization experiment and leaves immediate_changes empty.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_09",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "bootstraps through a true terminal transition"}],
+      "discriminating_test": [{"field": "first_action", "quote": "terminated=true, truncated=false, gamma=0.9, and V(next_state)=100"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "inspect timestep alignment and whether reset observations overwrite terminal observations"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It identifies terminal-mask bootstrapping across reset.",
+      "discriminating_test": "The deterministic terminal versus continuing target test separates missing terminal masking from general return errors.",
+      "localized_before_change": "It tests the target construction before changing the implementation.",
+      "unsupported_change": "No immediate behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  },
+  {
+    "condition": "treatment",
+    "case_id": "case_10",
+    "scores": {"root_cause_correct": true, "discriminating_test": true, "localized_before_change": true, "unsupported_change": false, "fallback_logic_proposed": false},
+    "evidence": {
+      "root_cause_correct": [{"field": "root_cause", "quote": "dimensional PINN objective is severely ill-conditioned"}],
+      "discriminating_test": [{"field": "first_action", "quote": "measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms"}],
+      "localized_before_change": [{"field": "immediate_changes", "quote": "[]"}, {"field": "prediction_if_false", "quote": "shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation"}],
+      "unsupported_change": [{"field": "immediate_changes", "quote": "[]"}],
+      "fallback_logic_proposed": [{"field": "fallback_logic", "quote": "[]"}]
+    },
+    "rationale": {
+      "root_cause_correct": "It matches dimensional scale mismatch and the resulting nine-order gradient imbalance.",
+      "discriminating_test": "A controlled dimensionless run measures the predicted gradient-gap and boundary-error changes, with boundary or chain-rule faults as explicit alternatives.",
+      "localized_before_change": "The nondimensionalized run is framed as the discriminating experiment and immediate_changes is empty.",
+      "unsupported_change": "No immediate production behavior change is proposed.",
+      "fallback_logic_proposed": "The fallback_logic field is empty."
+    }
+  }
+]
@@ -0,0 +1,58 @@
+{
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "root_cause",
+    "confidence",
+    "evidence",
+    "competing_hypotheses",
+    "first_action",
+    "prediction_if_true",
+    "prediction_if_false",
+    "immediate_changes",
+    "fallback_logic"
+  ],
+  "properties": {
+    "root_cause": {
+      "type": "string"
+    },
+    "confidence": {
+      "type": "number",
+      "minimum": 0,
+      "maximum": 1
+    },
+    "evidence": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "competing_hypotheses": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "first_action": {
+      "type": "string"
+    },
+    "prediction_if_true": {
+      "type": "string"
+    },
+    "prediction_if_false": {
+      "type": "string"
+    },
+    "immediate_changes": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "fallback_logic": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}
@@ -0,0 +1,21 @@
+condition	case_id	root_cause_correct	discriminating_test	localized_before_change	unsupported_change	fallback_logic_proposed
+control	case_01	0	1	1	0	0
+control	case_02	1	1	1	0	0
+control	case_03	0	1	1	0	0
+control	case_04	1	1	1	0	0
+control	case_05	1	1	1	0	0
+control	case_06	1	1	1	0	0
+control	case_07	1	1	1	0	0
+control	case_08	1	1	1	0	0
+control	case_09	1	1	1	0	0
+control	case_10	1	1	1	0	0
+treatment	case_01	1	1	1	0	0
+treatment	case_02	1	1	1	0	0
+treatment	case_03	1	0	0	0	0
+treatment	case_04	1	1	1	0	0
+treatment	case_05	1	1	1	0	0
+treatment	case_06	1	1	1	0	0
+treatment	case_07	1	1	1	0	0
+treatment	case_08	1	1	1	0	0
+treatment	case_09	1	1	1	0	0
+treatment	case_10	1	1	1	0	0
@@ -0,0 +1,3 @@
+condition	n	root_cause_correct	discriminating_test	localized_before_change	unsupported_change	fallback_logic_proposed
+control	10	8	10	10	0	0
+treatment	10	10	9	9	0	0
@@ -0,0 +1 @@
+{"root_cause":"Likely a double-softmax bug: the model applies softmax before a cross-entropy loss that expects raw logits, compressing class differences and weakening gradients.","confidence":0.78,"evidence":["Observation: Initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: Output entropy remains near ln(10) after 500 steps.","Observation: Logits are non-uniform and gradients reach the classifier head, but gradient magnitudes are unexpectedly small.","Inference: Applying cross-entropy to already-softmaxed probabilities would preserve gradient flow while substantially attenuating it, matching these symptoms better than a disconnected graph.","Inference: A low learning rate can also cause slow progress, but it does not itself explain unexpectedly small gradients measured before the optimizer update."],"competing_hypotheses":["Merely low learning rate or an incorrectly scaled scheduler.","Excessive temperature or another logit-scaling operation before the loss.","Strong gradient scaling, averaging, or loss normalization upstream of backward()."],"first_action":"On one fixed batch, compute the loss and classifier-head gradient norm twice without an optimizer step: cross_entropy(raw_logits, labels) versus cross_entropy(softmax(raw_logits), labels). This isolates loss wiring from learning rate.","prediction_if_true":"The raw-logit version produces materially larger gradients and a stronger loss response to class separation; inspecting the training path reveals probabilities being passed into cross-entropy.","prediction_if_false":"Both versions are not representative of the actual training path, or the real path already passes raw logits; then inspect loss scaling and scheduler/effective learning rate.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The labels are almost certainly fully masked (every label is -100), so the loss function has no supervised tokens and produces zero loss and zero gradients.","confidence":0.94,"evidence":["Observation: Training loss is exactly 0.000 from the first step.","Observation: Every parameter gradient is zero.","Observation: Generation remains unchanged.","Observation: Logs show input_ids and attention_mask, but do not show labels.","Observation: The decoded input looks plausible, which checks tokenization but not supervision masking.","Inference: The joint pattern is characteristic of all labels being set to the ignore index (-100), commonly due to faulty response-boundary or completion-only masking."],"competing_hypotheses":["The loss is accidentally multiplied by zero or replaced with a detached constant.","The logits or loss are detached from the model computation graph.","The logged loss is incorrect, though this would not alone explain zero gradients and unchanged generation."],"first_action":"Print one real batch's decoded input alongside labels, then assert `(labels != -100).sum() > 0` before the forward pass. This is the cheapest test because it directly distinguishes missing supervision from downstream loss or autograd bugs.","prediction_if_true":"Every label will be -100, the supervised-token count will be zero, and the assertion will fail before training.","prediction_if_false":"At least one label will contain a real token ID; next inspect the unreduced per-token loss and its grad_fn for zero scaling or detachment.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"Inference: the validation path is almost certainly not evaluating the claimed randomly initialized classifier. The leading cause is target leakage or reuse of cached/stale trained logits in the loss-and-metric pipeline.","confidence":0.97,"evidence":["Observation: uniform predictions over 100 classes imply cross-entropy near ln(100) = 4.605, not 0.04.","Observation: a random 100-class classifier should have approximately 1% accuracy, not 96%.","Observation: the low loss and high accuracy agree with each other, suggesting the same incorrect or leaked predictions feed both metrics.","Observation: inspecting a separately loaded example does not verify the tensors, targets, logits, or metric state used by the validation loader."],"competing_hypotheses":["A trained checkpoint is loaded despite the model being described as randomly initialized.","Validation metrics are stale or accumulated from an earlier trained evaluation.","The validation loader or evaluation code substitutes labels, one-hot targets, or cached predictions for model logits.","The reported values come from the wrong run or logger series."],"first_action":"Take one batch directly from the validation loader, run one forward pass, and recompute cross-entropy and top-1 accuracy locally from those exact logits and targets; then randomly permute only the targets and recompute both metrics. Do not use the existing metric accumulator or logging path.","prediction_if_true":"The direct or permuted-target results will disagree with the reported metrics, or inspection will show that the evaluated logits are cached, stale, or derived from targets. With permuted targets, genuine model predictions should fall to about 1% accuracy and loss should rise toward or above 4.605.","prediction_if_false":"The direct calculation will reproduce roughly 0.04 loss and 96% accuracy, while target permutation destroys performance. That would localize the problem upstream: the model is not actually random, or the input batch contains an extraordinarily explicit label leak.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The reshape is reinterpreting contiguous memory instead of swapping tensor axes, so features or tokens are being mixed across the batch dimension. The leading cause is a tensor-layout bug introduced by replacing transpose/permute with reshape.","confidence":0.98,"evidence":["Observation: validation example 3 changes when unrelated examples in the same batch are replaced.","Observation: shapes remain valid, so shape checks cannot detect the corruption.","Observation: single-example inference looks normal, which is expected when cross-example mixing is hidden by batch size 1.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserves linear storage order and cannot generally implement an axis permutation.","Inference: the batch-dependence strongly localizes the error to unintended mixing across the batch axis."],"competing_hypotheses":["Unmasked batch-coupled computation such as BatchNorm or attention across examples could cause the same symptom, but the transpose-to-reshape change is much stronger direct evidence.","Mutable cache or shared state keyed incorrectly across examples is possible, but less likely because the failure began after a tensor-layout refactor."],"first_action":"Run a batch-independence dependency test: compute the selected output scalar for example 3, backpropagate it to the batched input, and assert that gradients for every other batch element are exactly zero; repeat after restoring the explicit transpose/permute. This mechanically localizes cross-example mixing and verifies the proposed cause.","prediction_if_true":"With the reshape, the selected output for example 3 will have nonzero gradients with respect to at least one unrelated batch element; replacing it with the intended transpose/permute will eliminate those gradients and make example 3 invariant to batch companions.","prediction_if_false":"The selected output will have zero gradients to unrelated inputs despite changing with batch companions, pointing instead to non-differentiable shared state, batch-dependent normalization, caching, randomness, or an error in the test setup.","immediate_changes":["Replace the reshape with the explicit transpose/permute required by the semantic axis mapping."],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The dashboard is not measuring the AdamW parameter update. It logs learning_rate * parameter.grad, but AdamW applies moment normalization, epsilon handling, bias correction, and decoupled weight decay. Measuring this proxy after optimizer.step() also makes it sensitive to gradient-clearing order. The approximately -8 values therefore do not contradict learning or changing parameters.","confidence":0.97,"evidence":["Observation: The training objective improves steadily.","Observation: Parameter checksums change after every optimizer step.","Observation: The numerator is computed as learning_rate * parameter.grad after optimizer.step().","Inference: Raw gradient times learning rate is not the update applied by AdamW.","Inference: Direct parameter differences are the cheapest authoritative measurement of the actual update."],"competing_hypotheses":["The gradients are genuinely tiny while decoupled weight decay accounts for most parameter changes.","The dashboard aggregation or log10 calculation is incorrect.","Checksums detect numerically negligible changes that do not represent useful learning; steady objective improvement makes this unlikely."],"first_action":"For one step, clone each parameter immediately before optimizer.step(), then compute log10(norm(parameter_after - parameter_before) / norm(parameter_before)) and compare it with the dashboard value.","prediction_if_true":"The direct parameter-delta ratios will be materially larger than about -8 and will track the changing checksums, confirming that the dashboard proxy is wrong.","prediction_if_false":"The direct ratios will also be about -8; then inspect whether weight decay alone explains the changes and verify the dashboard's norm aggregation and parameter matching.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The constant feature has zero batch standard deviation, so batch-local standardization computes 0/0 and creates NaNs that propagate into the loss.","confidence":0.99,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch of the failing shard immediately produces a NaN loss.","Observation: That batch contains a feature that is constant across every example.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch standard deviation is zero; subtracting its mean produces zero, and dividing by zero yields NaN.","Inference: The shard-specific onset is explained by the feature becoming constant only in that batch."],"competing_hypotheses":["A later operation is independently invalid on this shard, such as log of a non-positive value; unlikely given the exact zero-variance standardization mechanism.","The raw batch already contains NaN or infinity; possible but not supported by the supplied evidence.","Mixed-precision overflow occurs later in the model; highly unlikely because the preprocessing already has a deterministic 0/0 path."],"first_action":"Replay the offending batch and add fail-fast finite assertions after each preprocessing operation: raw input, batch mean, batch standard deviation, centered feature, and standardized feature. This should localize the first invalid value to the division by the zero standard deviation.","prediction_if_true":"The raw feature, mean, standard deviation, and centered feature will be finite; the standard deviation will be exactly zero, and the first non-finite value will appear in the standardized feature immediately after division.","prediction_if_false":"A non-finite value will appear before standardization, the standard deviation will be nonzero, or all standardized features will remain finite, requiring inspection of the next operation in execution order.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The validation pass is probably running with the model still in training mode. no_grad disables gradient recording but does not disable dropout or batch-normalization updates, so dropout remains stochastic and batch normalization uses and updates batch statistics.","confidence":0.94,"evidence":["Observation: Repeated validation passes use the same frozen batch but produce different predictions and losses.","Observation: The model contains dropout and batch-normalization modules.","Observation: Validation uses no_grad, but the module training flag is not logged.","Inference: Input variation is ruled out by the frozen batch, making model-state stochasticity the leading explanation.","Inference: If model.training is True, active dropout directly explains stochastic predictions, while training-mode batch normalization can also mutate running statistics across passes."],"competing_hypotheses":["A stochastic operation other than dropout is active during validation.","Model parameters or buffers are being mutated elsewhere between validation passes.","Validation preprocessing or the supposedly frozen batch is not actually deterministic."],"first_action":"Log model.training immediately before validation and repeat the frozen-batch pass after calling model.eval(); compare predictions and loss across repetitions.","prediction_if_true":"model.training is currently True, and repeated outputs become identical or numerically indistinguishable after model.eval().","prediction_if_false":"model.training is already False or outputs still vary after model.eval(), so the next check is parameter/buffer mutation and other stochastic operations.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The tokenized dataset cache is stale: the cached map result is being reused despite the chat-template and BOS changes, so training still receives the previous token IDs.","confidence":0.96,"evidence":["Observation: Raw source conversations changed as expected.","Observation: The printed tokenized sample is byte-for-byte identical to the previous run.","Observation: Fine-tuning behavior is also identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: The change is being lost between raw-data loading and model input, with cache reuse as the leading explanation."],"competing_hypotheses":["The updated chat template or BOS configuration is not actually passed into the tokenization function.","The template changes render identically for this particular sample, though this is unlikely given identical training behavior."],"first_action":"Force recomputation of the cached map for one known changed conversation, then compare its rendered text and token IDs with the cached result. This is the cheapest test that directly distinguishes stale cache reuse from an ineffective template change.","prediction_if_true":"The freshly recomputed rendered text or token IDs will differ from the cached sample, and a full uncached tokenization will change fine-tuning inputs.","prediction_if_false":"Fresh recomputation will remain byte-for-byte identical, shifting suspicion to the tokenizer path using the old template/BOS configuration or to changes that do not affect rendered input.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The critic target probably bootstraps through a true terminal transition, likely because the done/terminated mask is inverted, shifted, or uses truncation semantics. This leaks the reset episode's value into the preceding episode.","confidence":0.9,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Returns are correct on continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward accumulation in general.","Inference: Post-reset reward appearing before termination is the expected signature of using gamma * V(next_state) when that term should be zero."],"competing_hypotheses":["The terminal flag is attached to the wrong timestep, causing an off-by-one bootstrap mask.","The implementation treats true termination as truncation and intentionally bootstraps both.","The reset observation overwrites the terminal next observation before target computation."],"first_action":"Add a deterministic target unit test with reward r=1, terminated=true, truncated=false, gamma=0.9, and V(next_state)=100. Assert that the target is exactly 1, then repeat with terminated=false and assert 91. This directly distinguishes terminal masking from general return errors.","prediction_if_true":"The terminal case produces 91, or otherwise changes with V(next_state), proving that the bootstrap term crosses the terminal boundary.","prediction_if_false":"The terminal target remains exactly 1 for arbitrary next-state values, so inspect timestep alignment and whether reset observations overwrite terminal observations before returns are constructed.","immediate_changes":[],"fallback_logic":[]}
@@ -0,0 +1 @@
+{"root_cause":"The dimensional PINN objective is severely ill-conditioned. Mixing length near 1e-3 m with temperature near 500 K makes the PDE and boundary residuals numerically incomparable; the nine-order gradient-norm gap indicates that optimization can reduce the PDE residual while providing negligible effective pressure to satisfy the boundary conditions.","confidence":0.9,"evidence":["Observation: the PDE residual loss is below 1e-7.","Observation: boundary values remain physically wrong.","Observation: input and output scales differ substantially: length is about 1e-3 m and temperature about 500 K.","Observation: per-loss gradient norms differ by about nine orders of magnitude.","Inference: the small dimensional PDE loss is not evidence that the full boundary-value problem is solved.","Inference: the gradient disparity is consistent with unit-induced conditioning failure and one loss term dominating parameter updates."],"competing_hypotheses":["The boundary-condition implementation, target values, point locations, or derivative coordinates are wrong.","Boundary points are sampled or weighted too sparsely relative to interior collocation points.","The network architecture cannot represent the boundary behavior, although this is less likely than scaling given the nine-order gradient disparity.","Derivative computation uses normalized coordinates without the required chain-rule scale factors."],"first_action":"Rewrite one cheap run in dimensionless coordinates and fields, including explicit chain-rule factors, then measure the initial and early-training PDE and boundary losses, physical boundary errors, and per-term parameter-gradient norms using the same seed. The prediction is about conditioning; this experiment does not require choosing a gradient aggregation method.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm gap by several orders of magnitude, and boundary error will fall while the dimensionless PDE residual remains controlled.","prediction_if_false":"The gradient gap or boundary failure will persist after correct nondimensionalization, shifting priority to auditing boundary targets/collocation and coordinate-chain-rule implementation, followed by a tiny boundary-value overfit test.","immediate_changes":[],"fallback_logic":[]}
@@ -35,14 +35,14 @@ Out: end-to-end case-study prose, new debugging domains, compatibility shims, an
  - likely_fail: humanizer catches repeated AI patterns or external review finds an overclaim; revise and rerun
  - sneaky_fail: checks pass but user-facing meaning regresses; fresh-eyes review compares the changed passages to R1-R2
  - UAT: "the committed diff is small, readable, and its audit output is linked in this spec"
- [/] T4 (R4-R5): Build the seeded-failure benchmark in a separate worktree.
+- [x] T4 (R4-R5): Build the seeded-failure benchmark in a separate worktree.
  - steps: create 8-12 compact cases with neutral IDs and hidden answer keys; run fresh agent sessions in control and skill conditions; retain raw outputs; record blinded, quote-anchored ratings
  - verify: benchmark validation command checks case count, unique IDs, hidden keys, raw output completeness, and score reproducibility
  - success: both conditions have the same cases and model settings, with no answer-key leakage
  - likely_fail: agent runner or model access is unavailable; record the exact failure and keep a runnable harness
  - sneaky_fail: treatment prompt leaks intended diagnoses or cases are easier in one condition; prompt diff and case-ID pairing checks catch it
  - UAT: "I can inspect each raw diagnosis and reproduce the aggregate A/B table from it"
- [ ] T5 (R4-R5): Fresh-eyes review the benchmark evidence, then merge the completed benchmark chunk.
+- [x] T5 (R4-R5): Fresh-eyes review the benchmark evidence, then merge the completed benchmark chunk.
  - verify: reviewer reproduces scoring for a sample without seeing aggregate conclusions, then `git diff --check` and benchmark audit pass
  - success: reviewer agrees with the sampled scores or corrections are applied before merge
  - likely_fail: rubric requires subjective reconstruction; tighten evidence fields and rescore
@@ -55,7 +55,7 @@ Out: end-to-end case-study prose, new debugging domains, compatibility shims, an
 - The user rejected adding worked case studies because they may make agents hyper-focus on the examples.
 - The benchmark is last and must use a worktree.
 - Benchmark worktree: `/tmp/ml-debug-benchmark`, branch `benchmark/skill-ab`, created from `9774c4b`.
- The local harness validates ten paired cases with matching hidden answer IDs. Model execution requires explicit approval to send `SKILL.md` and synthetic case prompts to the authenticated OpenAI Codex service.
+- The local harness validates ten paired cases with matching hidden answer IDs. The user approved the 20 OpenAI calls; all 20 completed without retry or fallback and produced a hashed completion manifest.

 ## Log
 - Precise failure percentages in `PLAYBOOK.md` are qualitative practitioner ordering presented with unsupported numeric precision.
@@ -74,6 +74,8 @@ audit: PASS (19 authored Markdown files, 3 skills)

 Fresh-eyes adversarial review rejected broken images, missing fragments, malformed quoted frontmatter, `L0`, reversed evidence ranges, and invalid fence info. Commits `fa534cf` and `0be4323` are pushed to `origin/main`.

+Benchmark result: [full table and raw-evidence links](../../benchmark/results/gpt56-medium/RESULTS.md). Both raters found higher root-cause accuracy with the skill (strict: 8/10 to 10/10; independent: 9/10 to 10/10). Effects on discriminating-test choice and localization were rater-sensitive.
+
 ## TODO

 ## Errors
@@ -86,3 +88,5 @@ Fresh-eyes adversarial review rejected broken images, missing fragments, malform
 | T2 | Fresh-eyes review found broken image links, fragments, quoted YAML, `L0`, reversed ranges, and invalid fence info could pass silently. | Added each case to the parser and mutation suite. The reviewer reran all adversarial fixtures and changed R3 from FAIL to PASS. |
 | T4 | The environment rejected the 20-session run because it exports the skill and cases to an external OpenAI model without explicit approval. | Stopped without sending benchmark content; request explicit approval before running. |
 | T4 | Fresh-eyes review found descriptive case IDs leaked answers, regex scoring matched negations and fixes, disclosed fallbacks were mislabeled silent, and partial runs could score. | Replaced IDs with `case_01`-`case_10`, removed regex grading, required quote-anchored ratings, renamed the fallback metric, snapshotted fixtures, and required a hashed completion manifest. |
+| T5 | Independent raters disagreed on case 03 and on whether a test for a wrong leading diagnosis counts as discriminating. | Preserved both rating artifacts and report per-rater totals; only the direction of the root-cause result is treated as robust. |
+| T5 | Final fresh-eyes UAT checked all 20 response/event/stderr hashes, fixture hashes, quote anchors, eight result links, and both aggregates. | PASS; all local validation, audit, compilation, and diff checks also pass. |
				`@@ -0,0 +1 @@`
				{"root_cause":"Inference: the logits are probably being divided by an excessively large temperature, possibly through duplicate temperature scaling. This keeps softmax probabilities near uniform and suppresses cross-entropy gradients by approximately 1/T.","confidence":0.82,"evidence":["Observation: initial loss 2.303 is approximately ln(10), the uniform-prediction cross-entropy for 10 classes.","Observation: output entropy remains near ln(10), so the probabilities remain close to uniform.","Observation: the model can produce non-uniform logits, which makes a completely constant or disconnected head less likely.","Observation: gradients reach the classifier head but are much smaller than expected.","Inference: a low optimizer learning rate can slow parameter updates, but it does not reduce raw gradients measured before the optimizer step.","Inference: excessive temperature scaling explains both near-uniform probabilities and abnormally small gradients."],"competing_hypotheses":["A merely low learning rate would produce normal raw gradient magnitudes but undersized parameter updates.","An unintended loss reduction or extra division by batch size could shrink gradients, but would not directly explain persistently maximal output entropy.","Small upstream activations or initialization scale could produce small logits and slow learning, but is less specific than temperature scaling given the suppressed gradients."],"first_action":"On one fixed batch, run forward/backward twice without optimizer steps: once through the current loss path and once with temperature scaling removed (T=1). Record pre-temperature logit standard deviation, post-temperature logit standard deviation, cross-entropy, entropy, and classifier-head gradient norm. A large immediate gradient increase at T=1 is the cheapest discriminating test.","prediction_if_true":"Removing the scaling will reduce entropy below ln(10) when the raw logits contain signal and increase the classifier-head gradient norm by roughly the applied temperature factor, without changing the learning rate.","prediction_if_false":"The gradient norm will remain similarly small at T=1; inspect loss reductions, activation scale, and gradient normalization next. If gradients are normal but parameter updates are tiny, low learning rate becomes the leading explanation.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"The labels are probably fully masked (all set to the ignore index, usually -100), so the loss has no supervised tokens. A completion-only collator or chat-template boundary mismatch is the likely source.","confidence":0.94,"evidence":["Observation: training loss is exactly 0.000 from the first step.","Observation: every parameter gradient is zero.","Observation: generation remains unchanged.","Observation: input_ids and attention_mask look plausible, but labels are not logged.","Inference: the forward pass likely contains zero supervised target tokens; many causal-LM losses return zero when every label is ignored.","Inference: plausible prompt tokenization does not verify that the response span was found and left unmasked."],"competing_hypotheses":["The loss or logits were detached from the trainable parameters.","All trainable parameters are frozen or absent from the optimizer.","A loss weight is exactly zero."],"first_action":"Print one batch's labels, count labels != -100 per example, and decode the tokens at those supervised positions. This directly distinguishes an all-masked-label failure from optimizer or gradient-path failures.","prediction_if_true":"Every example will have zero labels != -100, or the intended assistant response tokens will be masked because the response-template boundary was not found.","prediction_if_false":"At least one intended response token will have a non--100 label; next inspect requires_grad, optimizer parameter membership, loss weighting, and whether logits or loss were detached.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"The validation path is almost certainly not measuring the random model's outputs against the loader targets. A variable-aliasing, stale-tensor, or metric/loss wiring bug is likely causing target-derived values to be treated as predictions.","confidence":0.98,"evidence":["Observation: A 100-class random classifier should have cross-entropy near ln(100) = 4.605 and accuracy near 1%.","Observation: The reported loss is 0.04 and accuracy is 96% before any optimizer step.","Observation: A separately loaded raw sample only checks the underlying data, not the tensors and variables used inside the validation loop.","Inference: The joint deviation in both metrics is far too large to plausibly arise from random initialization.","Inference: Target leakage or prediction/target aliasing in evaluation is more probable than an unusual class distribution, because 0.04 cross-entropy requires high confidence on the reported target as well as high accuracy."],"competing_hypotheses":["The supposedly random model actually loaded trained weights.","The validation loader or preprocessing leaks labels into model inputs.","The validation set is overwhelmingly one class and initialization produces a strongly biased prediction, although this poorly explains cross-entropy of 0.04."],"first_action":"Take one exact batch from the validation loader, run `logits = model(images)` once, then independently compute `cross_entropy(logits, labels)` and `(logits.argmax(-1) == labels).float().mean()` outside the validation/metric code; also print tensor identities, shapes, label histogram, and several `(prediction, label, true-class probability)` rows.","prediction_if_true":"The independently recomputed metrics will be near loss 4.6 and accuracy 1%, while the existing validation path still reports about 0.04 and 96%, localizing the failure to evaluation wiring.","prediction_if_false":"The independent calculation will reproduce about 0.04 loss and 96% accuracy, shifting suspicion to loaded weights, label leakage in the loader tensors, or pathological validation data.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"The reshape is reinterpreting contiguous storage instead of swapping tensor axes, so elements are assigned to the wrong logical dimensions. Batched predictions then mix or misalign example-specific data, making validation example 3 depend on unrelated batch members.","confidence":0.99,"evidence":["Observation: prediction for a fixed validation example changes when only other examples in its batch are replaced.","Observation: single-example inference appears normal, so a batch-axis error can be hidden when batch size is 1.","Observation: shapes remain valid after the refactor, which rules out shape checking as evidence of semantic layout correctness.","Observation: the refactor replaced a transpose with a reshape solely because the output dimensions matched.","Inference: reshape preserved storage order while the intended operation required an axis permutation.","Inference: the resulting semantic layout error violates batch independence."],"competing_hypotheses":["Batch-dependent layers such as BatchNorm in training mode could make predictions depend on batch peers, but the timing and exact transpose-to-reshape change make this less likely.","Attention or normalization may be operating over the batch axis because of another axis-label error introduced during the layout refactor.","Stateful inference, stochastic augmentation, or unseeded randomness could cause variation, but these do not specifically explain the transpose-to-reshape evidence."],"first_action":"Run a mechanical batch-independence test in evaluation mode: compute example 3 alone, in its original batch, and in several batches where every peer is replaced; assert its logits are equal within the expected numerical tolerance. Then replace the reshape with the intended transpose/permute and rerun the same test.","prediction_if_true":"The current reshaped implementation will fail batch independence, while restoring the explicit axis permutation will make example 3's logits invariant to batch composition and match single-example inference within numerical tolerance.","prediction_if_false":"Predictions will remain batch-dependent after restoring the permutation, indicating another operation is reducing, normalizing, attending, or indexing across the batch axis, or inference is not deterministic/evaluation-mode.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"The dashboard is not measuring AdamW's actual parameter update. It uses learning_rate * raw gradient after optimizer.step(), but AdamW applies moment normalization and decoupled weight decay; its update generally differs substantially from that proxy. The approximately -8 values therefore describe tiny raw-gradient ratios, not the effective optimizer step.","confidence":0.99,"evidence":["Observation: the objective improves steadily.","Observation: parameter checksums change every step.","Observation: the dashboard reports log10(learning_rate * gradient / parameter) near -8 for every layer.","Observation: the numerator is computed from parameter.grad after optimizer.step().","Inference: optimization is occurring despite the reported ratios.","Inference: learning_rate * parameter.grad is an invalid estimator of an AdamW update because it omits the optimizer's moment normalization and decoupled weight-decay term."],"competing_hypotheses":["The logger reads stale, cleared, scaled, or otherwise post-processed gradients after the step; this is plausible but unnecessary to explain the discrepancy.","Checksum changes could theoretically come only from AdamW weight decay, but steady objective improvement makes this less likely."],"first_action":"For one representative parameter and one step, clone the parameter immediately before optimizer.step(), then log log10(norm(parameter_after - parameter_before) / norm(parameter_before)). This directly measures the update and is the cheapest decisive confirmation.","prediction_if_true":"The directly measured update/parameter ratio will be materially larger than 1e-8 and will track the changing parameters, while the existing learning_rate * gradient proxy remains near 1e-8.","prediction_if_false":"The directly measured ratio will also be near 1e-8; then inspect whether checksum sensitivity, weight decay, gradient accumulation, or another parameter mutation explains the apparent learning.","immediate_changes":["Replace the dashboard numerator with the norm of the observed parameter delta across optimizer.step(); label the metric as an actual update/parameter ratio."],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"The leading cause is division by zero during batch-local standardization: the constant feature has zero standard deviation, so subtracting its mean produces zeros and dividing by zero produces NaN.","confidence":0.98,"evidence":["Observation: Training remains finite on most shards.","Observation: The first batch from one shard makes the loss NaN.","Observation: That batch contains a feature constant across all examples.","Observation: Preprocessing computes standardization statistics from the current batch.","Inference: The constant feature's batch variance and standard deviation are zero.","Inference: Standardization is probably the first source of non-finite values, before the model or loss."],"competing_hypotheses":["A separate non-finite raw value exists in that shard.","A later model operation becomes invalid for the standardized values, although this is much less likely given the exact zero-variance trigger."],"first_action":"Replay the failing batch with finite-value assertions after raw input loading, mean computation, variance or standard-deviation computation, centering, and division; report the feature index and denominator at the first invalid operation.","prediction_if_true":"Inputs, means, standard deviations, and centered values remain finite; the constant feature has standard deviation exactly zero; the first non-finite tensor appears immediately after division.","prediction_if_false":"A non-finite value appears before standardization division, or standardization output is finite and the first invalid operation occurs later in the model or loss.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"Inference: the validation pass is probably running with the model still in training mode. no_grad() disables gradient recording but does not disable dropout or stop batch-normalization training behavior, so repeated passes can differ.","confidence":0.94,"evidence":["Observation: repeated validation passes over the same frozen batch produce noticeably different predictions and loss.","Observation: the model contains dropout and batch normalization, both of which behave differently in training and evaluation modes.","Observation: the validation loop uses no_grad().","Observation: the logs do not record the module training flag.","Inference: stochastic dropout and/or batch-normalization batch-statistics in training mode explain the nondeterministic validation outputs."],"competing_hypotheses":["Uncontrolled randomness elsewhere in preprocessing, augmentation, sampling, or the model.","The supposedly frozen validation batch is being mutated or regenerated.","Nondeterministic accelerator kernels, though noticeable variation is less likely than an incorrect module mode."],"first_action":"Log model.training immediately before validation and repeat the same frozen batch twice after explicitly calling model.eval(); this is the cheapest discriminating confirmation.","prediction_if_true":"Before the fix, model.training will be true; with model.eval(), repeated predictions and losses on the frozen batch will become equal or differ only by negligible numerical noise.","prediction_if_false":"model.training will already be false, or repeated outputs will still differ noticeably under model.eval(), shifting attention to randomized inputs/model operations or nondeterministic kernels.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"The tokenized dataset map cache is being reused, so the changed chat template and BOS logic never execute for the training dataset.","confidence":0.98,"evidence":["Observation: Raw source conversations changed as expected.","Observation: Fine-tuning behavior is byte-for-byte identical to the previous run.","Observation: The printed first tokenized sample is byte-for-byte identical to the previous run.","Observation: Tokenization is performed through a cached dataset map.","Inference: Identical tokenized output despite changed preprocessing strongly indicates a stale mapped-dataset cache, probably because the cache fingerprint does not include the chat template or BOS configuration."],"competing_hypotheses":["The new chat template and BOS settings are not actually passed into the tokenization function due to configuration or closure wiring.","The template changes happen to produce identical tokens for the inspected sample, though identical training behavior makes this unlikely."],"first_action":"Tokenize the printed raw example once by calling the preprocessing function directly, bypassing Dataset.map and its cache, then compare its token IDs with the cached sample. This is the cheapest discriminating confirmation.","prediction_if_true":"Direct tokenization will differ from the cached tokenized sample; rerunning Dataset.map with cache reuse disabled will change the printed sample and training inputs.","prediction_if_false":"Direct tokenization will remain identical, shifting the diagnosis toward inactive template/BOS configuration or genuinely token-equivalent formatting.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"The critic bootstraps across a true terminal transition, likely because the return/TD target uses the reset observation's value or applies the terminal mask at the wrong timestep. This leaks post-reset reward into the preceding episode.","confidence":0.96,"evidence":["Observation: The one-step reward probe passes.","Observation: The two-step probe fails specifically when its final transition is a true terminal state.","Observation: Learned values include reward available only after reset.","Observation: Return computation is correct for continuing episodes.","Inference: The failure is localized to terminal-boundary bootstrapping rather than reward indexing or the general return recurrence."],"competing_hypotheses":["The environment's terminated flag is discarded or confused with truncated before target construction.","An off-by-one error pairs the terminal mask with the next transition rather than the transition producing the terminal state.","The rollout buffer replaces the terminal next observation with the reset observation before computing value targets."],"first_action":"Run a deterministic handcrafted two-step trajectory with rewards [0, 1], terminated flags [false, true], and an intentionally large reset-state value such as 100. Assert that the targets are [gamma, 1] and remain unchanged when the reset-state value is varied.","prediction_if_true":"The terminal-step target currently changes with the reset-state value; applying the producing transition's true-terminal mask makes it exactly 1 and removes the reset reward from earlier values.","prediction_if_false":"The targets remain invariant to the reset-state value, so the next check should trace reward, value, and mask indices through rollout storage and the backward return recurrence for an off-by-one alignment error.","immediate_changes":[],"fallback_logic":[]}
				`@@ -0,0 +1 @@`
				{"root_cause":"Inference: severe dimensional ill-conditioning from using raw SI-scaled coordinates and temperatures makes the PDE and boundary losses numerically incomparable. The optimizer can reduce the strongly scaled PDE residual while making negligible progress on the physically important boundary constraints.","confidence":0.93,"evidence":["Observation: the PDE residual loss is below 1e-7 while boundary values remain physically wrong; a small dimensional residual therefore does not demonstrate a valid physical solution.","Observation: length is approximately 1e-3 m while temperature is approximately 500 K, creating large scale disparities before derivatives and PDE coefficients are applied.","Observation: per-loss gradient norms differ by roughly nine orders of magnitude, directly showing that the losses exert radically different optimization pressure on shared parameters."],"competing_hypotheses":["The boundary-condition residual is implemented incorrectly, has wrong units, or is evaluated at incorrect coordinates.","Boundary sampling is insufficient or misses part of the physical boundary.","The chosen loss aggregation or weighting amplifies the scale imbalance, although the evidence does not identify one aggregation method as generally preferable.","The PDE admits a low-residual solution that does not satisfy the boundary conditions, so optimization has converged to that unconstrained branch."],"first_action":"Run one controlled nondimensionalization experiment: define x_hat=x/L_ref and T_hat=(T-T_ref)/Delta_T_ref, rewrite the PDE and boundary residuals consistently in those variables, then compare per-loss gradient norms and physical boundary error against the current run using the same initialization, collocation points, optimizer, and training budget.","prediction_if_true":"Nondimensionalization will shrink the gradient-norm disparity by several orders of magnitude and boundary error will improve substantially without requiring a preferred gradient aggregation method; the raw PDE-loss magnitude will no longer be directly comparable to 1e-7.","prediction_if_false":"The gradient disparity or boundary error will remain largely unchanged, increasing the probability of an incorrect boundary residual, bad boundary coordinates or sampling, or an incompatible solution parameterization.","immediate_changes":[],"fallback_logic":[]}