ml-debug/SKILL.md

---
name: ml-debug
description: "Instructions for machine learning development and debugging: at each trigger point, fill a short form and show it to the user, so the evidence gets read instead of assumed. Use when writing a project's success evidence, before launching a run, after a run finishes or crashes, when stuck after two diagnostic cycles, and before reporting any result. Turns 'read your data', 'assume you have a bug', and 'compare to reference code' into artifacts the user can check."
---

# ML development: do the block that matches

The practitioner folklore behind these blocks is in [README.md](README.md), written for humans.
Agents read that prose, agree with it, and then skip it: measured on ml-bench, loading the folklore
changed a model's score by +0.023 +/- 0.044, which is nothing.

So each principle here is an instruction instead: something to write now, and to show the user, so
that skipping it is visible. Every block asks questions and none of them tells you what to fix. The
work is to make you look before you decide. The decision stays yours, because this system is
probably not in your training data.

Do not summarise this file back to anyone. Find the line below that matches what just happened, go
to that block, and do it in your next message.

| what just happened | do this now |
|---|---|
| you are about to write "everything checks out" | block 0 |
| a new project, no evidence criteria written yet | block 1 |
| a run finished and a stage changed | block 2 |
| you are about to queue or launch a run | block 3 |
| a run finished or crashed | block 4 |
| two diagnostic cycles with no progress, or a metric will not move | block 5 |
| you are about to report a result | block 6 |

Fill the forms in the chat, in the audit file, or in the guide, as each block says. Write plain
english. Use these words for every credence, from the
[Kesselman list](https://gwern.net/doc/statistics/bayes/2008-kesselman.pdf#p71):

| Almost certain | Highly likely | Likely | Even | Unlikely | Highly unlikely | Remote |
|---|---|---|---|---|---|---|
| 86-99% | 71-85% | 56-70% | 46-55% | 31-45% | 16-30% | 1-15% |

## 0. You have a bug. Check before you say the result is fine

Watch your own draft. When you are about to write "everything checks out", "nothing looks
unresolved", "no open questions", "looks good", or "the metrics look fine", stop and do block 6
first: three diagnoses with credences, five ways this is invalid, and random raw samples.

An all-clear is a claim, and it is the claim least likely to have been checked. A machine learning
system has many adaptive parts, so a broken one is often hidden by the others compensating, and the
output still looks reasonable. Raise the bar at which you say "I think this is correct". Most
research results are false, and an exciting result is more likely false than a boring one.

## 1. Write down what would convince you, then rewrite it as you learn

This lives in the project's `AGENTS.md`. Write the first draft early, while you still know almost
nothing, and expect it to be wrong. Its job at that point is to stop a null result from being
explained away later, because nobody wrote down what a null would mean.

Revise it whenever evidence changes what you believe, and say in the audit what you changed and
why. A frame you never revised is a frame you never tested. A frame you revised after seeing the
result, without saying so, is how a null becomes a success.

```markdown
GOAL: one paragraph, plain english. What is learned, and what is the contribution.
OPTIMISING: one sentence, no jargon, no symbols. What the loss actually rewards.
PERVERSE SATISFACTIONS: 2-3 plain-english ways to score well while doing nothing
  interesting (copy the input, learn the class prior, exploit the judge). Give each
  the metric or control arm that would expose it.
EVIDENCE A, qualitative: a full trace, out of sample, long enough to see it break.
  Name the arms: bare, treatment, reversed treatment, and a placebo that should not move.
EVIDENCE B, quantitative: beats a named baseline on a named metric, not chance.
ELSE: if A and B disagree, or either fails, say now what that means about the method.
```

## 2. Build the training guide as you go, two pages

If `TRAINING_GUIDE.md` does not exist, create it now with the stages you can already name, and mark
every one of them unknown. It grows one observation at a time, after each run, and it holds your
current model of the system rather than a run history. Keep it under 180 lines by replacing claims
that went stale. Past 180 lines you are appending history, nobody will read it, and the file is
dead.

Frontmatter carries only what a program can check:

```yaml
---
last_reviewed_job: 21
stages: {init: uncertain, posterior: working, writer: failing, generation: failing}
---
```

One markdown entry per stage, and each entry says what should become observable, in order:

```markdown
## Writer
Purpose: use the inferred user state to change decoder behavior.
Trained: writer. Frozen: decoder, encoder. Input: state at t. Loss: reply NLL.
Theory of change: state difference -> write difference -> logit difference -> behavior difference.
Expected sequence: 1. write leaves its initial scale. 2. write beats zero-write.
  3. swapping the state changes the logits. 4. generations differ. 5. they differ in the
  intended direction. 6. the effect survives out of sample.
Required observations: writer loss curve, update norm, zero-write arm, shuffled-state arm,
  first-token logits, long free generations.
Current evidence: quote a job and a number for each link you claim.
Missing evidence: the observations nobody has made yet.
Status: partial, likely, 60%.
Earliest unsupported link: step 5. Swaps move the logits, the direction is not established.
Main question: ...
```

Do not put a failure-mode list in this file. A list of possible causes gives an agent thirty
excuses. The expected sequence gives it one question: which link should be visible by now, and is it?

## 3. Predict the run before you queue it

Show the user this form. Queue nothing until it is filled.

| risky part | what I expect to see | what would falsify it | metric exists? |
|---|---|---|---|

If the metric that would show the effect does not exist yet, add it and then run. A run that cannot
distinguish success from failure is not worth the GPU time.

Also state which stage of the training guide this run advances, and paste the loss from overfitting
about 20 samples. Near zero, or something is wrong before the real run starts.

## 4. Audit the run, crash included, and quote the log

Run `/auditlog`, which owns the full procedure. It requires the whole log, the resolved config, the
actual data, complete raw outputs from every arm, a stage table, and hypotheses that each carry a
quote, a credence, contrary evidence, and a discriminating test.

Four things this skill adds to that audit:

1. A measurement pass before any diagnosis, written under the heading "do not interpret yet". Quote
   the log line, metric row, or sample for each technically risky part, next to what you expected to
   see there. Give every curve its value at the start, early, middle and end, because the shape
   carries the diagnosis and the final number does not. A summary without a quote does not show that
   you opened the log. A needed metric that is missing is a valid outcome: add it and run again.
2. A prediction check against the form from block 3, row by row, marked supported, contradicted, or
   unresolved.
3. The earliest unsupported link in the training guide, updated. Then edit the affected stage entry.
   If the guide grew, you appended run history instead of replacing a stale claim.
4. The case for the next experiment, made against the current unresolved question. Say why it beats
   finishing, repeating, or cancelling work already in the queue. Killing a queued sweep that no
   longer answers the question is progress. Never stop applies to the research goal, never to one
   experiment, architecture, sweep or hypothesis.

A crash is a run and gets an audit too. Then use judgement: a typo or a missing import gets fixed
and rerun immediately, while a broken idea earns no more compute until you have read the papers and
the reference code.

Anything weird gets a line in the audit, and every line ends explained or being investigated. An
anomaly you found without looking for it is a large problem, so chase it rather than hoping it goes
away.

## 5. When stuck, diff against reference code and send a bug hunt

Both of these, not one:

Reference diff. Find the most-adopted implementation of the nearest method, ranking candidates by
community adoption, then papers citing it, then code that runs. Then fill a table, one row per
feature, with their file and line in every row.

| feature | theirs (file:line) | mine | same? |
|---|---|---|---|

Cover the algorithm tweaks, the engineering tricks, the hyperparameters, and which metrics they
log, because the tricks are usually in the code and not in the paper. "No reference exists" is a
finding you must state out loud, never one you imply by skipping the table.

Subagent bug hunt. Send a fresh-eyes subagent at the module or the diff with the instruction: find
at least one bug, we all have at least one. Report what it found, including nothing. You cannot see
your own typos because you know what the code was supposed to say.

## 6. Test the result before you report it

Three artifacts, every time, for a positive result as much as a negative one:

1. Three or more diagnoses with credences, including a code bug and an invalid evaluation whenever
   they are plausible. Leave probability on an unknown cause. Do not pad the list to reach three.
   Every diagnosis carries the strongest evidence against it. If you cannot find any evidence
   against your favourite, you have not tested it, so lower the credence and say so.
2. The five most likely ways this result is invalid, each with the check that would settle it.
3. Complete raw samples, chosen at random, quoted. Say how you chose them. Samples picked because
   they look clean prove nothing.

Start from a substantial probability that a surprising result is invalid, and lower it only as
checks rule out bugs, leakage, and broken evaluation. Excitement is evidence of a bug.

## Reference

Open the one the situation calls for. These widen a hypothesis space. They are not authoritative
for your system:

- [PLAYBOOK.md](PLAYBOOK.md) -- mental models, component isolation, baseline ladder, what to log, symptom tables, triage, anti-patterns.
- [refs/checklist.md](refs/checklist.md) -- Lones's 36 do/don'ts across data, training, evaluation, comparison, reporting.
- [refs/diagnostics.md](refs/diagnostics.md) -- copy-paste snippets: init loss, overfit one batch, gradient flow, NaN hooks, leakage tracer, backprop-to-input dependency check.
- [refs/static_analysis.md](refs/static_analysis.md) -- grep patterns for silent bugs.
- [refs/loss_surface.md](refs/loss_surface.md) -- visualize a custom loss and its gradient field with synthetic tensors.
- [refs/metric_stuck.md](refs/metric_stuck.md) -- why a metric will not move, plus the structural ceiling check.
- [refs/sweeps.md](refs/sweeps.md) -- paired comparison and cross-seed reliability, before claiming A beats B.
- [refs/llm_judges.md](refs/llm_judges.md) -- judge biases, repeat draws, paired differences, when an LLM-judged eval looks too good.
- [refs/time_series.md](refs/time_series.md) -- deployment-faithful temporal evaluation and causal missing values.
- [refs/research_taste.md](refs/research_taste.md) -- patience, choosing what to try, information gain, de-risking.
- [refs/transformers.md](refs/transformers.md) -- full traces, warmup and learning rate, train-deploy parity, scale priors, steering.
- [rl/SKILL.md](rl/SKILL.md) -- probe environments, reward engineering, defaults, reference implementations.
- [pinn/SKILL.md](pinn/SKILL.md) -- nondimensionalization, gradient pathologies, curriculum.

Curated by [wassname](https://github.com/wassname).