ml-debug/SKILL.md

name, description

name

description

ml-debug

Rituals for machine learning development and debugging: forms to fill and show the user at each trigger point, so evidence gets read instead of assumed. Use when framing an ML project's success evidence, before launching a run, after a run finishes or crashes, when stuck after two diagnostic cycles, and before reporting any result. Turns 'read your data', 'assume you have a bug', and 'compare to reference code' into artifacts the user can check.

Rituals for ML development

The practitioner folklore behind every ritual here is in README.md, written for humans. Agents read that prose, agree with it, and then skip it: measured on ml-bench, loading the folklore changed a model's score by +0.023 +/- 0.044, which is nothing.

So each principle below is a ritual instead: a trigger, a form, and an artifact you show the user. A skipped ritual is then visible. The forms ask questions and never prescribe the fix, because the judgement is yours and the system is probably not in your training data.

Fill the forms in the chat, in the audit file, or in the guide, as each ritual says. Write plain english. Use these words for every credence, from the Kesselman list:

Almost certain	Highly likely	Likely	Even	Unlikely	Highly unlikely	Remote
86-99%	71-85%	56-70%	46-55%	31-45%	16-30%	1-15%

0. You have a bug. The trigger is the all-clear

This one has no phase, so watch your own draft instead. The moment you are about to write "everything checks out", "nothing looks unresolved", "no open questions", "looks good", or "the metrics look fine", stop and pay ritual 6 first: three diagnoses with credences, five ways this is invalid, and random raw samples.

An all-clear is a claim, and it is the claim least likely to have been checked. A machine learning system has many adaptive parts, so a broken one is often hidden by the others compensating, and the output still looks reasonable. Raise the bar at which you say "I think this is correct". Most research results are false, and an exciting result is more likely false than a boring one.

1. Frame the project, once, before the first real run

Write this into the project's AGENTS.md and keep it there. Without it, a null result later gets explained away, because nobody wrote down what a null would mean.

GOAL: one paragraph, plain english. What is learned, and what is the contribution.
OPTIMISING: one sentence, no jargon, no symbols. What the loss actually rewards.
PERVERSE SATISFACTIONS: 2-3 plain-english ways to score well while doing nothing
  interesting (copy the input, learn the class prior, exploit the judge). Give each
  the metric or control arm that would expose it.
EVIDENCE A, qualitative: a full trace, out of sample, long enough to see it break.
  Name the arms: bare, treatment, reversed treatment, and a placebo that should not move.
EVIDENCE B, quantitative: beats a named baseline on a named metric, not chance.
ELSE: if A and B disagree, or either fails, say now what that means about the method.

2. Keep a training guide, two pages, living

TRAINING_GUIDE.md in the project, under 180 lines. The worker maintains it. It holds the current model of the system, not a run history, so it stays small by replacing claims that went stale. Past 180 lines you are appending history, and nobody will read it, which defeats the file.

Frontmatter carries only what a program can check:

---
last_reviewed_job: 21
stages: {init: uncertain, posterior: working, writer: failing, generation: failing}
---

One markdown entry per stage, and each entry says what should become observable, in order:

## Writer
Purpose: use the inferred user state to change decoder behavior.
Trained: writer. Frozen: decoder, encoder. Input: state at t. Loss: reply NLL.
Theory of change: state difference -> write difference -> logit difference -> behavior difference.
Expected sequence: 1. write leaves its initial scale. 2. write beats zero-write.
  3. swapping the state changes the logits. 4. generations differ. 5. they differ in the
  intended direction. 6. the effect survives out of sample.
Required observations: writer loss curve, update norm, zero-write arm, shuffled-state arm,
  first-token logits, long free generations.
Current evidence: quote a job and a number for each link you claim.
Missing evidence: the observations nobody has made yet.
Status: partial, likely, 60%.
Earliest unsupported link: step 5. Swaps move the logits, the direction is not established.
Main question: ...

Do not put a failure-mode list in this file. A list of possible causes gives an agent thirty excuses. The expected sequence gives it one question: which link should be visible by now, and is it?

3. Before a run

Show the user this form. Queue nothing until it is filled.

risky part	what I expect to see	what would falsify it	metric exists?

If the metric that would show the effect does not exist yet, add it and then run. A run that cannot distinguish success from failure is not worth the GPU time.

Also state which stage of the training guide this run advances, and paste the loss from overfitting about 20 samples. Near zero, or something is wrong before the real run starts.

4. After every run, including a crash

Run /auditlog, which owns the full procedure. It requires the whole log, the resolved config, the actual data, complete raw outputs from every arm, a stage table, and hypotheses that each carry a quote, a credence, contrary evidence, and a discriminating test.

Four things this skill adds to that audit:

1. A measurement pass before any diagnosis, written under the heading "do not interpret yet". Quote the log line, metric row, or sample for each technically risky part, next to what you expected to see there. Give every curve its value at the start, early, middle and end, because the shape carries the diagnosis and the final number does not. A summary without a quote does not show that you opened the log. A needed metric that is missing is a valid outcome: add it and run again.
2. A prediction check against the form from ritual 3, row by row, marked supported, contradicted, or unresolved.
3. The earliest unsupported link in the training guide, updated. Then edit the affected stage entry. If the guide grew, you appended run history instead of replacing a stale claim.
4. The case for the next experiment, made against the current unresolved question. Say why it beats finishing, repeating, or cancelling work already in the queue. Killing a queued sweep that no longer answers the question is progress. Never stop applies to the research goal, never to one experiment, architecture, sweep or hypothesis.

A crash is a run and gets an audit too. Then use judgement: a typo or a missing import gets fixed and rerun immediately, while a broken idea earns no more compute until you have read the papers and the reference code.

Anything weird gets a line in the audit, and every line ends explained or being investigated. An anomaly you found without looking for it is a large problem, so chase it rather than hoping it goes away.

5. When stuck, after two diagnostic cycles or a metric that will not move

Both of these, not one:

Reference diff. Find the most-adopted implementation of the nearest method, ranking candidates by community adoption, then papers citing it, then code that runs. Then fill a table, one row per feature, with their file and line in every row.

feature	theirs (file:line)	mine	same?

Cover the algorithm tweaks, the engineering tricks, the hyperparameters, and which metrics they log, because the tricks are usually in the code and not in the paper. "No reference exists" is a finding you must state out loud, never one you imply by skipping the table.

Subagent bug hunt. Send a fresh-eyes subagent at the module or the diff with the instruction: find at least one bug, we all have at least one. Report what it found, including nothing. You cannot see your own typos because you know what the code was supposed to say.

6. Before reporting any result

Three artifacts, every time, for a positive result as much as a negative one:

1. Three or more diagnoses with credences, including a code bug and an invalid evaluation whenever they are plausible. Leave probability on an unknown cause. Do not pad the list to reach three. Every diagnosis carries the strongest evidence against it. If you cannot find any evidence against your favourite, you have not tested it, so lower the credence and say so.
2. The five most likely ways this result is invalid, each with the check that would settle it.
3. Complete raw samples, chosen at random, quoted. Say how you chose them. Samples picked because they look clean prove nothing.

Start from a substantial probability that a surprising result is invalid, and lower it only as checks rule out bugs, leakage, and broken evaluation. Excitement is evidence of a bug.

Reference

Open the one the situation calls for. These widen a hypothesis space. They are not authoritative for your system:

• PLAYBOOK.md -- mental models, component isolation, baseline ladder, what to log, symptom tables, triage, anti-patterns.
• refs/checklist.md -- Lones's 36 do/don'ts across data, training, evaluation, comparison, reporting.
• refs/diagnostics.md -- copy-paste snippets: init loss, overfit one batch, gradient flow, NaN hooks, leakage tracer, backprop-to-input dependency check.
• refs/static_analysis.md -- grep patterns for silent bugs.
• refs/loss_surface.md -- visualize a custom loss and its gradient field with synthetic tensors.
• refs/metric_stuck.md -- why a metric will not move, plus the structural ceiling check.
• refs/sweeps.md -- paired comparison and cross-seed reliability, before claiming A beats B.
• refs/llm_judges.md -- judge biases, repeat draws, paired differences, when an LLM-judged eval looks too good.
• refs/time_series.md -- deployment-faithful temporal evaluation and causal missing values.
• refs/research_taste.md -- patience, choosing what to try, information gain, de-risking.
• refs/transformers.md -- full traces, warmup and learning rate, train-deploy parity, scale priors, steering.
• rl/SKILL.md -- probe environments, reward engineering, defaults, reference implementations.
• pinn/SKILL.md -- nondimensionalization, gradient pathologies, curriculum.

Curated by wassname.