wassname
ee4e9a5caa
Gather debugging folklore from more practitioners, each a verbatim quote
checked against a cached source copy (footnoted with line numbers):
- koaning (Vincent Warmerdam), "Bad Labels": benchmark labels are often wrong;
find them with confidence-sorted errors.
- gwern, the tank-detection legend: the canonical data-leakage parable, plus
the scout-mindset twist that it's a likely-unsourced urban legend.
- Patrick Kidger, "Just Know Stuff": why research code is buggy ("kludge ...
bugs that don't cripple things only because some other bug stops them") and
"never accept the kludge". Plus a one-line jaxtyping pointer for shape bugs.
- nanochat (Karpathy): BOS-alignment fake metric improvement; all-ranks must
clip on inf (a multi-GPU bug single-GPU testing hides).
- cleanrl "37 Implementation Details of PPO" -> RL sub-skill, as the canonical
proof that reference-impl details (not ideas) decide whether PPO works.
Trim the lucidrains item to one quote (it had ballooned). Add wassname credit
+ companion-gist link. All 20 footnotes resolve.
Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
1.1 KiB
Just Know Stuff (how to achieve success in an ML PhD) — Patrick Kidger
Source: https://kidger.site/thoughts/just-know-stuff/ (2023-01-26). Cached excerpt from the "Software development" section, verbatim.
Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.
This is a systemic professional failing. As an (applied) ML researcher, the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.
So, how to improve? First of all, never accept the kludge.
You've messed up your Git repo? Figure out the commands to fix it... don't just delete it and clone from the remote.
Focus on writing clean code, based around orthogonal abstractions. When the code starts getting messy - and it will - be willing to refactor your code into something more legible. Avoid both spaghetti code and ravioli code.
When the documentation is inadequate, look at their source code.