wassname/ml_debug
1
0
Fork 0
mirror of https://github.com/wassname/ml_debug.git synced 2026-07-25 13:20:37 +08:00
Code Issues Packages Projects Releases Wiki Activity

rl: quote Spinning Up (Achiam) on silent failure and bug-first debugging

Browse Source 
Spinning Up as a Deep RL Researcher was only a bare code link; it's the
canonical RL-researcher guide and its debugging advice is gold. Cache the
rigour/debugging sections verbatim and quote the sharpest lines in the RL
sub-skill: "broken RL code almost always fails silently", "if it doesn't work,
assume there's a bug", "measure everything ... you can't tell it's broken if
you can't see that it's breaking", and test on more than one env. Add to RL
sources.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-06-02 21:04:55 +08:00
co-authored by Claudypoo
parent ee4e9a5caa
commit a602ea5a0e
2 changed files with 34 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rl/SKILL.md
+3
View File
@@ -110,6 +110,8 @@ The allure of writing from scratch is real but the self-correction mechanisms in
The canonical demonstration is "The 37 Implementation Details of PPO" [Huang et al. 2022]: reproducing PPO meant matching 37 separate details (13 core, 9 Atari, 9 continuous-control, 5 LSTM, 1 MultiDiscrete), each linked to its exact line of code, "which is not done in academic papers." If your PPO underperforms a reference, you are probably missing some of these, not lacking a better idea.
Spinning Up [Achiam 2018] states the rationale bluntly: "broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task." Hence "if it doesn't work, assume there's a bug. Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters." Two habits people skip: "measure everything ... you can't tell it's broken if you can't see that it's breaking," and test on more than one env, because "sometimes things will work in one environment even when you have a breaking bug."
References: spinning-up (OpenAI), stable-baselines3, cleanrl (single-file per algo), OpenSpiel (multi-agent).
**10. Don't over-interpret noise.** [Schulman 2017, Henderson 2018, Irpan 2018]
@@ -152,6 +154,7 @@ Sometimes (rarely) you don't. Schulman:
- McCandlish & Kaplan, "An Empirical Model of Large-Batch Training" (2018): https://arxiv.org/abs/1812.06162
- Slav Ivanov, "37 Reasons why your Neural Network is not working" (2017): https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607
- Huang et al., "The 37 Implementation Details of PPO" (ICLR Blog Track 2022): https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ ([cache](../docs/evidence/cleanrl_37_ppo_details.md))
- Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018): https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([cache](../docs/evidence/spinningup_researcher.md))
### Reference implementations
- OpenAI Spinning Up: https://github.com/openai/spinningup