mirror of
https://github.com/wassname/ml_debug.git
synced 2026-09-06 16:50:17 +08:00
follow the skill spec: references/ not refs/, and namespaced subskill names
- refs/ -> references/, the folder name the Agent Skills spec uses and the one Hermes skips when it walks for nested skills. - rl and pinn declared name: rl and name: pinn, which are global names in a flat skill namespace. Now ml-debug-rl and ml-debug-pinn. They also called themselves sub-skills of 'ml-debugging', which is not this skill's name. - Drop the dead link to SKILL_old.md. It moved into gitignored slop/, so the link was broken for anyone who cloned. - Route references/llm_judge_litreview.md, the one reference SKILL.md never named. - Description leads with the trigger situations. Hermes truncates it to 57 chars. Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
@@ -17,5 +17,5 @@ Quoting them is grounding data; rewording them injects assistant bias.
|
||||
block preserves the author's reasoning.
|
||||
- Put lower-relevance sources in "See also" rather than forcing a synthetic
|
||||
narrative around them.
|
||||
- In `SKILL.md`, link to reference docs like `refs/research_taste.md` instead
|
||||
- In `SKILL.md`, link to reference docs like `references/research_taste.md` instead
|
||||
of copying a long assistant-written summary.
|
||||
|
||||
+11
-11
@@ -16,7 +16,7 @@ How to *think* when generating hypotheses or deciding what to investigate next.
|
||||
|
||||
**4. Bias-variance via learning curves.**[^cs229][^fsdl] Plot train and val error vs dataset size (or steps). Both high and converging together = high bias (too simple, wrong features, or a capacity-reducing bug). Train low, val high = high variance (overfitting). Val flat even with 10x more data = not a data problem, fix the model.
|
||||
|
||||
**5. Structural ceiling: can the parameterization even express what you want?** Sometimes a metric is stuck not because the optimizer fails but because the architecture literally cannot represent the target. Quick check: disable the loss term entirely; if the metric reaches the same value, the loss never moved it. Worked example in [refs/metric_stuck.md](refs/metric_stuck.md).
|
||||
**5. Structural ceiling: can the parameterization even express what you want?** Sometimes a metric is stuck not because the optimizer fails but because the architecture literally cannot represent the target. Quick check: disable the loss term entirely; if the metric reaches the same value, the loss never moved it. Worked example in [references/metric_stuck.md](references/metric_stuck.md).
|
||||
|
||||
### Where to look first
|
||||
|
||||
@@ -36,7 +36,7 @@ For RL, add reward scale/sign as a top-3 issue, and episode-boundary handling (d
|
||||
|
||||
| Signal | Likely meaning | Check |
|
||||
|--------|----------------|-------|
|
||||
| Init loss << expected (e.g. 0.01 vs 2.3) | Leakage or a shortcut: the model "knows" the answer at init | Are labels in the input? Is test data in train? A trivial feature? Localize with Wassname's NaN-poisoning tracer or backprop-to-input check ([refs/diagnostics.md](refs/diagnostics.md)) |
|
||||
| Init loss << expected (e.g. 0.01 vs 2.3) | Leakage or a shortcut: the model "knows" the answer at init | Are labels in the input? Is test data in train? A trivial feature? Localize with Wassname's NaN-poisoning tracer or backprop-to-input check ([references/diagnostics.md](references/diagnostics.md)) |
|
||||
| After training, replacing real inputs with shuffled or random inputs barely changes predictions or the metric | The model may not use the intended input signal; this does not identify the cause | Inspect preprocessing, model wiring, label leakage, and task bias |
|
||||
| Predicts the same class for everything | Class imbalance (100:1 -> "always predict majority") | Label-count check; weighted loss or resample |
|
||||
| Val much worse than train from the start | Distribution shift between splits | Same preprocessing? Same time period? Same source? |
|
||||
@@ -131,7 +131,7 @@ Unfortunately, agents need these procedural mindset-shifts spelled out. This is
|
||||
|
||||
Roughly in this order, though the point is the underlying mindset:
|
||||
|
||||
**Collect clues before theorizing.** Read the traceback and logs. Run static analysis ([refs/static_analysis.md](refs/static_analysis.md)) and the cheap diagnostics ([refs/diagnostics.md](refs/diagnostics.md): data sanity check, init-loss check, overfit-one-batch). If you catch yourself proposing a fix before you've looked at anything, stop.
|
||||
**Collect clues before theorizing.** Read the traceback and logs. Run static analysis ([references/static_analysis.md](references/static_analysis.md)) and the cheap diagnostics ([references/diagnostics.md](references/diagnostics.md): data sanity check, init-loss check, overfit-one-batch). If you catch yourself proposing a fix before you've looked at anything, stop.
|
||||
|
||||
**Hold several hypotheses at once; resist converging early.** Unless the cause is already obvious (a traceback usually points right at it), generate at least three genuinely different hypotheses before ranking any, so you don't marry the first one. Use the five lenses in Mental models. Put a rough credence/prior on each, including an explicit unknown bucket when useful. Then sanity-check yourself with:
|
||||
- *Bug*: a boring implementation/data/loss bug, with high prior until checked.
|
||||
@@ -172,13 +172,13 @@ def debug(symptom):
|
||||
Rough order to consider, not authoritative; it may not fit your project. Stop when a question fits.
|
||||
|
||||
1. Exception/traceback? Read it, fix it, done.
|
||||
2. Loss NaN/Inf? Attach NaN hooks ([refs/diagnostics.md](refs/diagnostics.md)) or insert `assert torch.isfinite(x).all()` after successive stages. Find the first invalid value before changing the math. Common causes include log(0), 0/0, and exp(large).
|
||||
2. Loss NaN/Inf? Attach NaN hooks ([references/diagnostics.md](references/diagnostics.md)) or insert `assert torch.isfinite(x).all()` after successive stages. Find the first invalid value before changing the math. Common causes include log(0), 0/0, and exp(large).
|
||||
3. Init loss wrong? Check the data pipeline and loss; check for double softmax; check labels match the output format. A low init loss makes leakage or a shortcut plausible; localize it before changing the model.
|
||||
4. Can't overfit one batch? Gradient-flow check: None grads -> disconnected layer; all-zero grads -> dead layer / detach. Check autograd breakers and optimizer step order.
|
||||
5. Loss stuck from step 0 but you *can* overfit one batch? LR too low (try 10x), frozen params (check `requires_grad`), wrong loss.
|
||||
6. Loss decreases then explodes? LR too high (try 0.1x), log the pre-clip grad norm, hunt numerical instability.
|
||||
7. Training performance good but validation performance poor? First check for a train/validation mismatch or an evaluation bug. If those checks pass, overfitting is likely.
|
||||
8. Train loss fine but the metric is bad? Loss-metric misalignment ([refs/metric_stuck.md](refs/metric_stuck.md)).
|
||||
8. Train loss fine but the metric is bad? Loss-metric misalignment ([references/metric_stuck.md](references/metric_stuck.md)).
|
||||
9. Outputs constant? Mode collapse: class imbalance, all-zero init, dead ReLUs, look at confidence-sorted errors.
|
||||
10. Slow but not stuck? Not a bug. Consider batch size, depth/width, data quality.
|
||||
|
||||
@@ -201,12 +201,12 @@ These are the overconfident reflexes the "calibrate" section warns about, made c
|
||||
|
||||
Look these up when the symptom calls for them; they're kept out of the main flow on purpose.
|
||||
|
||||
- [refs/loss_surface.md](refs/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU. For when a custom loss misbehaves.
|
||||
- [refs/metric_stuck.md](refs/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check (is the optimizer failing, or can the parameterization not express it?).
|
||||
- [refs/sweeps.md](refs/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, so a result is "reliably better" not "a lucky seed."
|
||||
- [refs/llm_judges.md](refs/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist.
|
||||
- [refs/static_analysis.md](refs/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
|
||||
- [refs/diagnostics.md](refs/diagnostics.md) — copy-paste diagnostic snippets (init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check).
|
||||
- [references/loss_surface.md](references/loss_surface.md) — visualize a loss surface and its gradient field with synthetic tensors, no model or GPU. For when a custom loss misbehaves.
|
||||
- [references/metric_stuck.md](references/metric_stuck.md) — "why won't this metric move?" plus the structural-ceiling check (is the optimizer failing, or can the parameterization not express it?).
|
||||
- [references/sweeps.md](references/sweeps.md) — same-seed paired comparison and cross-seed t-stat reliability, so a result is "reliably better" not "a lucky seed."
|
||||
- [references/llm_judges.md](references/llm_judges.md) — LLM-as-a-judge biases (position, verbosity, self-preference) and the mitigation checklist.
|
||||
- [references/static_analysis.md](references/static_analysis.md) — grep patterns for silent bugs (shape mismatches, autograd breakers, double softmax, step ordering, leakage).
|
||||
- [references/diagnostics.md](references/diagnostics.md) — copy-paste diagnostic snippets (init-loss check, overfit-one-batch, gradient-flow check, NaN hooks, NaN-poisoning leakage tracer, backprop-to-input dependency check, class-imbalance check).
|
||||
- [rl/SKILL.md](rl/SKILL.md) — RL-specific debugging: probe environments, reward engineering, HP defaults, reference implementations.
|
||||
- [pinn/SKILL.md](pinn/SKILL.md) — physics-informed-network debugging: nondimensionalization, gradient pathologies, curriculum.
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Or paste `SKILL.md` into your system prompt / context when debugging.
|
||||
|
||||
- **[SKILL.md](SKILL.md)** -- what an agent loads: the folklore turned into instructions, each with a trigger, a form to fill, and an artifact to show the user. "Assume you have a bug" becomes "send a subagent to find one and report what it found". This is a bet that a form gets filled where a principle gets skipped, and it is untested. The bet is worth making because the folklore version measured no gain (below), and because forms have their own failure mode: they get filled with plausible content that nobody checked.
|
||||
|
||||
- **[PLAYBOOK.md](PLAYBOOK.md)** -- the synthesized long-form: mental models, practitioner priors, step catalogs, symptom tables, the agent debugging loop, triage, and anti-patterns. Menus of hypotheses distilled from the same sources, not quotes. Deeper one-off tricks (loss-surface analysis, stuck-metric diagnosis, sweep reliability) live in [refs/](refs/).
|
||||
- **[PLAYBOOK.md](PLAYBOOK.md)** -- the synthesized long-form: mental models, practitioner priors, step catalogs, symptom tables, the agent debugging loop, triage, and anti-patterns. Menus of hypotheses distilled from the same sources, not quotes. Deeper one-off tricks (loss-surface analysis, stuck-metric diagnosis, sweep reliability) live in [references/](references/).
|
||||
|
||||
- **[docs/evidence/](docs/evidence/)** -- frozen local copies of source material (blog posts, talks, papers, reddit threads). Claims here link back to exact quotes.
|
||||
|
||||
@@ -164,7 +164,7 @@ The 2018 tweet thread that seeded the recipe post. Every item is a silent failur
|
||||
|
||||
> 6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes]
|
||||
|
||||
Number 6 is the bug the backprop-to-input dependency check catches mechanically ([refs/diagnostics.md](refs/diagnostics.md)).
|
||||
Number 6 is the bug the backprop-to-input dependency check catches mechanically ([references/diagnostics.md](references/diagnostics.md)).
|
||||
|
||||
### Seed variance: you can't tell a bug from bad luck
|
||||
|
||||
@@ -172,7 +172,7 @@ Number 6 is the bug the backprop-to-input dependency check catches mechanically
|
||||
|
||||
> Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan]
|
||||
|
||||
Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([refs/sweeps.md](refs/sweeps.md)).
|
||||
Henderson confirmed it quantitatively: splitting 10 same-config runs (differing only in seed) into two groups of five produces "statistically different distributions just from varying random seeds."[^henderson] This is why one good run proves nothing ([references/sweeps.md](references/sweeps.md)).
|
||||
|
||||
### Normalize and scale everything
|
||||
|
||||
@@ -327,7 +327,7 @@ The one question that turns "am I overconfident" into something answerable:
|
||||
|
||||
> **How reliable is my experiment?** Ask yourself: "How surprised would I be if it turned out to be complete bullshit due to a bug, error, noise, misunderstanding, etc.?" Investigate the most uncertain bits[^nanda-papers]
|
||||
|
||||
And from an unpublished Nanda draft quoted in [refs/research_taste.md](refs/research_taste.md), so
|
||||
And from an unpublished Nanda draft quoted in [references/research_taste.md](references/research_taste.md), so
|
||||
weaker provenance than his published posts:
|
||||
|
||||
> Insufficient Skepticism: Missing simple alternative explanations, methodological flaws, or bugs. Explicitly list alternatives. Get others (especially mentors) to red team your plans before you run them. Actively try to break your hypothesis. Ask "What observation would make me abandon this?"[^nanda-taste]
|
||||
@@ -570,10 +570,10 @@ validation set is measuring overfitting to errors:
|
||||
|
||||
Start here rather than treating the bibliography as flat:
|
||||
|
||||
- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/pdf/2108.02497), with its full do/don't list extracted in [refs/checklist.md](refs/checklist.md).
|
||||
- **Beginner / broad checklist:** Lones, ["How to avoid machine learning pitfalls"](https://arxiv.org/pdf/2108.02497), with its full do/don't list extracted in [references/checklist.md](references/checklist.md).
|
||||
- **Debugging a neural net:** Karpathy, ["A Recipe for Training Neural Networks"](https://karpathy.github.io/2019/04/25/recipe/).
|
||||
- **Designing tuning experiments:** Google, [Deep Learning Tuning Playbook](https://developers.google.com/machine-learning/guides/deep-learning-tuning-playbook).
|
||||
- **Transformer and LLM runs:** [refs/transformers.md](refs/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
|
||||
- **Transformer and LLM runs:** [references/transformers.md](references/transformers.md), then the HF, Axolotl, Unsloth, nanochat, and Bekman sources below.
|
||||
|
||||
Folklore sources (the quotes above trace to these):
|
||||
|
||||
@@ -612,7 +612,7 @@ Folklore sources (the quotes above trace to these):
|
||||
[^deeprlhacks]: William Falcon, "DeepRLHacks", attendee notes on Schulman's "Nuts and Bolts of Deep RL Research" -- https://github.com/williamFalcon/DeepRLHacks ([cache](docs/evidence/williamfalcon_deeprl_hacks.md): random-noise-not-signal, observations-usable). Secondary source; the primary slide deck is `[^schulman]`.
|
||||
[^nanda-mindsets]: Neel Nanda, "My Research Process: Key Mindsets" -- https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/cbBwwm4jW6AZctymL ([cache](docs/evidence/nanda_research_process_key_mindsets.md): insufficient-skepticism-feels-like-research, mass-on-unlisted-hypotheses)
|
||||
[^nanda-papers]: Neel Nanda, "Highly Opinionated Advice on How to Write ML Papers" -- https://www.lesswrong.com/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers ([cache](docs/evidence/nanda_highly_opinionated_ml_paper_writing.md): how-reliable-is-my-experiment)
|
||||
[^nanda-taste]: Neel Nanda, "My Model of the Research Process", unpublished shared draft, as quoted in [refs/research_taste.md](refs/research_taste.md) (insufficient-skepticism, actively-seek-alternatives). Draft quality, weaker provenance than the published posts.
|
||||
[^nanda-taste]: Neel Nanda, "My Model of the Research Process", unpublished shared draft, as quoted in [references/research_taste.md](references/research_taste.md) (insufficient-skepticism, actively-seek-alternatives). Draft quality, weaker provenance than the published posts.
|
||||
[^nanda-draft]: Neel Nanda, "My Model of the Research Process", unpublished shared draft -- https://docs.google.com/document/d/1YMkeMrhqsWxZcNDD9CIUWEK_DAOegeufnbc79U2hycg/edit ([cache](docs/evidence/nanda_research_process_shared_draft.md): all-numbers-are-meaningless). This passage never made it into the published post.
|
||||
[^sanh]: Victor Sanh, "Simple considerations for simple people building fancy neural networks" (HF, 2021) -- https://huggingface.co/blog/simple-considerations ([cache](docs/evidence/sanh_simple_considerations_hf_2021.md): decent-performance-without-crashing, read-the-tokenizer-output, 4e2-is-a-symptom, pre-training questions)
|
||||
[^steinhardt]: Jacob Steinhardt, "Research as a Stochastic Decision Process" -- https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html ([cache](docs/evidence/steinhardt_research_stochastic_decision_process.md): 0.1%-of-implementations, high-standard-for-ruling-out, months-of-approaches-one-cause)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: ml-debug
|
||||
description: "Machine learning debugging exercises, each under a quote from a practitioner. If this loaded, do the exercise for your situation and show the result in your reply. Invoke it yourself. Triggers: read the log, the run finished, it crashed, queue a run, the loss is not going down, the metric will not move, is this result real, does A beat B, a spike or anything weird in the log, and any moment you are about to write that a result looks fine."
|
||||
description: "Debug an ML run: read the log, it crashed, the loss will not go down, the metric will not move, is this result real, does A beat B, a spike or anything weird in the log, about to queue a run, or about to write that a result looks fine. Machine learning debugging exercises, each under a quote from a practitioner. Do the exercise for your situation and show the result in your reply. Invoke it yourself."
|
||||
---
|
||||
|
||||
In an attempt to upskill the machine learning debugging on AI coding assistants (and humans), I've collected high quality sources on how to debug machine learning projects, focusing on the mindset and the "taste". When I started ML I went searching for discussions on best practices, and started a few discussions of my own and they helped me a lot, over the years I've collected good ones. I hope they can help others, as well as help in auto research setups. This intro is human written, and the below is AI written with human guidance. - wassname
|
||||
@@ -376,18 +376,20 @@ a punchy section-ending epigram, the third of its kind in the exercises. Written
|
||||
Sources and more quotes: [README.md](README.md). Longer material, open the one you need:
|
||||
|
||||
- [PLAYBOOK.md](PLAYBOOK.md) -- mental models, component isolation, baseline ladder, what to log, symptom tables.
|
||||
- [refs/checklist.md](refs/checklist.md) -- Lones's 36 do/don'ts.
|
||||
- [refs/diagnostics.md](refs/diagnostics.md) -- snippets: init loss, overfit one batch, gradient flow, NaN hooks, leakage tracer.
|
||||
- [refs/static_analysis.md](refs/static_analysis.md) -- grep patterns for silent bugs.
|
||||
- [refs/loss_surface.md](refs/loss_surface.md) -- visualise a custom loss and its gradient field.
|
||||
- [refs/metric_stuck.md](refs/metric_stuck.md) -- why a metric will not move, structural ceiling check.
|
||||
- [refs/sweeps.md](refs/sweeps.md) -- paired comparison and cross-seed reliability.
|
||||
- [refs/llm_judges.md](refs/llm_judges.md) -- judge biases, repeat draws, paired differences.
|
||||
- [refs/time_series.md](refs/time_series.md) -- temporal evaluation and causal missing values.
|
||||
- [refs/research_taste.md](refs/research_taste.md) -- patience, information gain, de-risking.
|
||||
- [refs/transformers.md](refs/transformers.md) -- full traces, warmup, train-deploy parity, steering.
|
||||
- [rl/SKILL.md](rl/SKILL.md), [pinn/SKILL.md](pinn/SKILL.md) -- domain specifics.
|
||||
- [SKILL_old.md](SKILL_old.md) -- the previous procedural version (P1-P5), kept until reviewed.
|
||||
- [references/checklist.md](references/checklist.md) -- Lones's 36 do/don'ts.
|
||||
- [references/diagnostics.md](references/diagnostics.md) -- snippets: init loss, overfit one batch, gradient flow, NaN hooks, leakage tracer.
|
||||
- [references/static_analysis.md](references/static_analysis.md) -- grep patterns for silent bugs.
|
||||
- [references/loss_surface.md](references/loss_surface.md) -- visualise a custom loss and its gradient field.
|
||||
- [references/metric_stuck.md](references/metric_stuck.md) -- why a metric will not move, structural ceiling check.
|
||||
- [references/sweeps.md](references/sweeps.md) -- paired comparison and cross-seed reliability.
|
||||
- [references/llm_judges.md](references/llm_judges.md) -- judge biases, repeat draws, paired differences.
|
||||
- [references/llm_judge_litreview.md](references/llm_judge_litreview.md) -- the papers behind the judge advice.
|
||||
- [references/time_series.md](references/time_series.md) -- temporal evaluation and causal missing values.
|
||||
- [references/research_taste.md](references/research_taste.md) -- patience, information gain, de-risking.
|
||||
- [references/transformers.md](references/transformers.md) -- full traces, warmup, train-deploy parity, steering.
|
||||
- [rl/SKILL.md](rl/SKILL.md), [pinn/SKILL.md](pinn/SKILL.md) -- domain specifics. These two are
|
||||
also skills in their own right, `ml-debug-rl` and `ml-debug-pinn`, so an agent that scans
|
||||
subdirectories can load one on its own.
|
||||
|
||||
## Sign off
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Unused quotes from the ml-debug evidence cache
|
||||
|
||||
Mined from `/home/wassname/.agents/skills/ml-debug/docs/evidence/` (about 40 cached sources) and
|
||||
`/home/wassname/.agents/skills/ml-debug/refs/`. Every quote here was checked against
|
||||
`/home/wassname/.agents/skills/ml-debug/references/`. Every quote here was checked against
|
||||
`/home/wassname/.agents/skills/ml-debug/README.md` and is not used there. Line numbers were
|
||||
verified by grep on a distinctive substring; long source lines are single wrapped paragraphs, so
|
||||
one line number can hold a long quote.
|
||||
@@ -82,7 +82,7 @@ Why it lands: seed noise alone can clear a significance bar. So one A-versus-B g
|
||||
Why it lands: turns "am I overconfident" into one answerable question with a calibration target, and points the next action at the least reliable step rather than the most interesting one.
|
||||
|
||||
## My Model of the Research Process (shared draft), as quoted in the skill's own topic note -- Neel Nanda
|
||||
- file: /home/wassname/.agents/skills/ml-debug/refs/research_taste.md:134
|
||||
- file: /home/wassname/.agents/skills/ml-debug/references/research_taste.md:134
|
||||
- failure modes: 1, 3
|
||||
- epistemic context: quoted from an unpublished Google Doc draft, so weaker provenance than the published posts by the same author.
|
||||
|
||||
@@ -224,7 +224,7 @@ Why it lands: two modes at once. Hypotheses 2 and 3 can be hypothesis 1 wearing
|
||||
Why it lands: a symptom-to-cause table where every symptom has two or three candidates and only one of them is a learning rate. It is a ready-made hypothesis-2-and-3 generator for the moment the agent reaches for the knob.
|
||||
|
||||
## My Model of the Research Process (shared draft), as quoted in the skill's own topic note -- Neel Nanda
|
||||
- file: /home/wassname/.agents/skills/ml-debug/refs/research_taste.md:120
|
||||
- file: /home/wassname/.agents/skills/ml-debug/references/research_taste.md:120
|
||||
- failure modes: 3
|
||||
- epistemic context: unpublished draft quoted in a local topic note; weaker provenance than the published posts.
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ summarizer produced from a web page, and nobody has read the paper. On
|
||||
2026-08-15 I re-pulled the five [ID] entries the litreview depends on and two
|
||||
of the five carried a wrong number, so treat the remaining 11 as roughly 2-in-5
|
||||
wrong until each is checked against raw text. Do not promote an [ID] number
|
||||
into SKILL.md or refs/ without re-pulling the paper first.
|
||||
into SKILL.md or references/ without re-pulling the paper first.
|
||||
|
||||
## "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — Zheng et al. (LMSYS), NeurIPS 2023 — https://arxiv.org/pdf/2306.05685
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@ Source: https://arxiv.org/pdf/2411.00640 (Evan Miller, Anthropic, Nov 2024) + ht
|
||||
Title: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
|
||||
Fetched-via: r.jina.ai on the arXiv PDF and the Anthropic post, 2026-08-16
|
||||
Fetch-status: verbatim from full PDF text (math notation mangled by the PDF-to-markdown pass; prose is clean)
|
||||
Used-by: refs/llm_judges.md (repeat draws, temperature, paired differences)
|
||||
Used-by: references/llm_judges.md (repeat draws, temperature, paired differences)
|
||||
|
||||
# Adding Error Bars to Evals (excerpts)
|
||||
|
||||
|
||||
+6
-6
@@ -1,12 +1,12 @@
|
||||
---
|
||||
name: pinn
|
||||
description: "PINN (Physics-Informed Neural Network) training best practices and debugging. Use when building, debugging, or optimizing PINNs for PDEs, ODEs, or physics-constrained learning problems. Sub-skill of ml-debugging."
|
||||
name: ml-debug-pinn
|
||||
description: "PINN (Physics-Informed Neural Network) training best practices and debugging. Use when building, debugging, or optimizing PINNs for PDEs, ODEs, or physics-constrained learning problems. Sub-skill of the ml-debug skill."
|
||||
---
|
||||
|
||||
|
||||
# PINN Training Best Practices
|
||||
|
||||
Consolidated from: NeuralPDE.jl tests/docs, ConFIG repo, Wang et al. 2021, Rathore et al. 2024 (ICML), ml_debug folklore, and practical experience. Heat-exchanger-specific notes in [refs/heat_exchanger.md](refs/heat_exchanger.md).
|
||||
Consolidated from: NeuralPDE.jl tests/docs, ConFIG repo, Wang et al. 2021, Rathore et al. 2024 (ICML), ml_debug folklore, and practical experience. Heat-exchanger-specific notes in [references/heat_exchanger.md](references/heat_exchanger.md).
|
||||
|
||||
Epistemic status: Patterns confirmed across multiple sources. Where sources disagree, noted. Paper claims marked with credence estimates.
|
||||
|
||||
@@ -18,7 +18,7 @@ PINNs are complex. Before trusting a PINN, work up the complexity ladder and com
|
||||
|
||||
**Make complexity pay rent.** If a fancier model doesn't improve on the simpler one, the added physics/architecture is either wrong, badly scaled, or unnecessary.
|
||||
|
||||
Build a complexity ladder for your problem (see [refs/heat_exchanger.md](refs/heat_exchanger.md) for a heat exchanger example). At each level, brainstorm:
|
||||
Build a complexity ladder for your problem (see [references/heat_exchanger.md](references/heat_exchanger.md) for a heat exchanger example). At each level, brainstorm:
|
||||
- What assumption am I adding/relaxing?
|
||||
- What does this buy me (lower RMSE, new physics captured)?
|
||||
- What breaks if I simplify further?
|
||||
@@ -214,7 +214,7 @@ ConFIG and UPGrad are both reasonable candidates when the losses cannot be repla
|
||||
> "The proposed approach consistently outperforms a standard PINN-based collocation method."
|
||||
> Source: https://arxiv.org/pdf/2104.08426, Abstract and Section 1
|
||||
> Evidence: evidence/sukumar2022_exact_bc_distance.md
|
||||
> Domain-specific failure modes and hard BC examples: see [refs/heat_exchanger.md](refs/heat_exchanger.md).
|
||||
> Domain-specific failure modes and hard BC examples: see [references/heat_exchanger.md](references/heat_exchanger.md).
|
||||
|
||||
---
|
||||
|
||||
@@ -242,7 +242,7 @@ For 2D problems with radial integrals: use a regular grid in r (including r=0 an
|
||||
|
||||
## 6. Property Mappings & Multi-Episode Training
|
||||
|
||||
> Domain-specific: differentiable EoS wrapping (REFPROP/PCHIP), IC handling for plant data, multi-episode training. See [refs/heat_exchanger.md](refs/heat_exchanger.md).
|
||||
> Domain-specific: differentiable EoS wrapping (REFPROP/PCHIP), IC handling for plant data, multi-episode training. See [references/heat_exchanger.md](references/heat_exchanger.md).
|
||||
|
||||
---
|
||||
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: rl
|
||||
description: "RL-specific debugging: probe environments, reward engineering, diagnostics, hyperparameter defaults, and reference implementations. Sub-skill of ml-debugging. Use when debugging reinforcement learning systems."
|
||||
name: ml-debug-rl
|
||||
description: "RL-specific debugging: probe environments, reward engineering, diagnostics, hyperparameter defaults, and reference implementations. Sub-skill of the ml-debug skill. Use when debugging reinforcement learning systems."
|
||||
---
|
||||
|
||||
# RL-Specific Debugging
|
||||
|
||||
Reference in New Issue
Block a user