mirror of
https://github.com/wassname/ml-debug.git
synced 2026-09-09 11:27:03 +08:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9ffbeff014 | ||
|
|
bdb70ba7b1 | ||
|
|
1fb188e923 | ||
|
|
4112134bfd |
@@ -98,7 +98,7 @@ When you're stuck after a diagnostic cycle or two, the generalization of this ad
|
|||||||
|
|
||||||
> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
|
> What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones]
|
||||||
|
|
||||||
A bug can also hide, because most ML models have multiple adaptive parts:
|
A bug can also hide, because most ML models have multiple adaptive parts:
|
||||||
|
|
||||||
> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
|
> "If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow],
|
||||||
and it may not show in the output at all.
|
and it may not show in the output at all.
|
||||||
@@ -356,8 +356,8 @@ instead.
|
|||||||
From William Falcon's attendee notes on Schulman's talk, so a secondary source rather than
|
From William Falcon's attendee notes on Schulman's talk, so a secondary source rather than
|
||||||
Schulman's own text[^deeprlhacks]:
|
Schulman's own text[^deeprlhacks]:
|
||||||
|
|
||||||
> 4. Think your algorithm is working but you're actually seeing random noise.
|
> 4. Think your algorithm is working but you're actually seeing random noise.
|
||||||
> - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
|
> - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
|
||||||
|
|
||||||
Nanda on why no internal warning fires:
|
Nanda on why no internal warning fires:
|
||||||
|
|
||||||
@@ -486,7 +486,7 @@ reports his two real bugs on that environment were a terminal-flag masking error
|
|||||||
broadcast, neither of which any of these can reach[^reddit-rl]:
|
broadcast, neither of which any of these can reach[^reddit-rl]:
|
||||||
|
|
||||||
> Things I've tried (but maybe not systematically enough):
|
> Things I've tried (but maybe not systematically enough):
|
||||||
>
|
>
|
||||||
> * Different initial LRs
|
> * Different initial LRs
|
||||||
> * Different optimizers
|
> * Different optimizers
|
||||||
> * Different number of hidden layers/units
|
> * Different number of hidden layers/units
|
||||||
@@ -512,7 +512,7 @@ wanted (the "demon-strated" break is an OCR artifact in the cached copy):
|
|||||||
notes[^deeprlhacks]:
|
notes[^deeprlhacks]:
|
||||||
|
|
||||||
> 2. Make sure observations usable:
|
> 2. Make sure observations usable:
|
||||||
> - See if YOU could control the system by using the same observations you give the agent.
|
> - See if YOU could control the system by using the same observations you give the agent.
|
||||||
> - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
|
> - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
|
||||||
|
|
||||||
For LLM work, the data you have to read is the tokenized data:
|
For LLM work, the data you have to read is the tokenized data:
|
||||||
|
|||||||
@@ -36,8 +36,8 @@ If either mode matters most to you, this cache needs a new source, not more mini
|
|||||||
- failure modes: 1
|
- failure modes: 1
|
||||||
- epistemic context: secondary source, attendee notes on Schulman's talk rather than Schulman's own text; the primary slide deck is cached separately as joschu_nuts_and_bolts.md.
|
- epistemic context: secondary source, attendee notes on Schulman's talk rather than Schulman's own text; the primary slide deck is cached separately as joschu_nuts_and_bolts.md.
|
||||||
|
|
||||||
> 4. Think your algorithm is working but you're actually seeing random noise.
|
> 4. Think your algorithm is working but you're actually seeing random noise.
|
||||||
> - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
|
> - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds.
|
||||||
|
|
||||||
Why it lands: a confident cross-task ranking read off three copies of one algorithm. It is the shortest demonstration that a conclusion can feel fully supported by a plot and be supported by nothing.
|
Why it lands: a confident cross-task ranking read off three copies of one algorithm. It is the shortest demonstration that a conclusion can feel fully supported by a plot and be supported by nothing.
|
||||||
|
|
||||||
@@ -287,7 +287,7 @@ Why it lands: sweeping the legible knobs is brute-force search wearing a lab coa
|
|||||||
- epistemic context: LOW CREDIBILITY. Anonymous reddit self-report from a self-described non-expert. Its value is as a specimen of the failure mode, not as advice, and it should not be quoted as authority.
|
- epistemic context: LOW CREDIBILITY. Anonymous reddit self-report from a self-described non-expert. Its value is as a specimen of the failure mode, not as advice, and it should not be quoted as authority.
|
||||||
|
|
||||||
> Things I've tried (but maybe not systematically enough):
|
> Things I've tried (but maybe not systematically enough):
|
||||||
>
|
>
|
||||||
> * Different initial LRs
|
> * Different initial LRs
|
||||||
> * Different optimizers
|
> * Different optimizers
|
||||||
> * Different number of hidden layers/units
|
> * Different number of hidden layers/units
|
||||||
@@ -328,7 +328,7 @@ Why it lands: a healthy-looking curve produced by a swimmer curling up and flail
|
|||||||
- epistemic context: secondary attendee notes; the matching primary slide is "Atari: can you see game features in downsampled image?" in the cached joschu_nuts_and_bolts.md.
|
- epistemic context: secondary attendee notes; the matching primary slide is "Atari: can you see game features in downsampled image?" in the cached joschu_nuts_and_bolts.md.
|
||||||
|
|
||||||
> 2. Make sure observations usable:
|
> 2. Make sure observations usable:
|
||||||
> - See if YOU could control the system by using the same observations you give the agent.
|
> - See if YOU could control the system by using the same observations you give the agent.
|
||||||
> - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
|
> - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way.
|
||||||
|
|
||||||
Why it lands: turns "read the data" into a pass/fail test that takes a minute. If you cannot do the task from the model's inputs, no hyperparameter will save it.
|
Why it lands: turns "read the data" into a pass/fail test that takes a minute. If you cannot do the task from the model's inputs, no hyperparameter will save it.
|
||||||
|
|||||||
Reference in New Issue
Block a user