Add randomized ML debugging fortunes

Co-Authored-By: PI[k3] <288921227+claudypoo@users.noreply.github.com>

SKILL.md

@@ -339,9 +339,11 @@ Keep the list in `docs/JARGON.md` when working in a long loop.
 ## Sign off
 
 Before writing "looks fine", "works", "no effect", or "found the bug", paste the log lines that
-show it. Then end the reply with one quote from this file in an ASCII speech balloon, said by an
+show it. Then choose one random line without loading the whole file: `shuf -n 1 fortune.txt`.
+End the reply with it as a clearly separate, random ASCII speech-balloon sign-off, said by an
 animal of your choice other than a cow, drawn by hand, holding a unicode tool that fits the
-exercise you did (🔧 🔍 🪛 🧪). Say who said the quote, so the reader can find the rest of it.
-The balloon lets the reader see at a glance that this file was read.
+exercise you did (🔧 🔍 🪛 🧪). Preserve the speaker attribution; where the canonical harvested
+record names only its source file, name that file instead. The fortune is not evidence for the
+diagnosis.
 
 Curated by [wassname](https://github.com/wassname).

fortune.txt

@@ -0,0 +1,313 @@
+broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task. -- Josh Achiam
+If one part is broken, the other parts can adapt and still achieve roughly acceptable performance. -- Goodfellow, Bengio and Courville
+The challenge lies in the fact that you can make these mistakes, train a model without it ever crashing, and still get a decent performance... -- Clara Sanh
+Trying an experiment and seeing it fail gives little information by itself. If X is a high-level conceptual approach, a more correct conclusion is: I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work. -- Jacob Steinhardt
+Insufficient skepticism doesn't feel like insufficient skepticism from the inside. It just feels like doing research. -- Neel Nanda
+Read your data. Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad. -- Neel Nanda
+What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking: OK, I think this is correct. -- Andy Jones
+The CNN has learned to detect a metal token that radiology technicians place on the patient in the corner of the image field of view at the time they capture the image. -- Zech et al.
+Apparently meaningless identifier columns were the most important predictors. [...] the university only filled out much of this information after a grant application was accepted. -- Howard and Gugger
+Excitement is evidence of bullshit: generally, most true results are not exciting, but a fair amount of false results are. -- Neel Nanda
+If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky. -- Alex Irpan
+It ended up taking me 6 weeks to reproduce results, thanks to several software bugs. The question is, why did it take so long to find these bugs? -- Dan Rahtz
+QUIT THINKING AND LOOK. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+CHANGE ONE THING AT A TIME. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+IF YOU DIDN'T FIX IT, IT AIN'T FIXED. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Don't let your instruments overwhelm your system. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+If you ever see a plot or a behaviour that just seems weird, chase right after it! Do not — do not — just hope it goes away. -- Andy Jones
+The cool extra functionality you were planning to write today might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead. -- Andy Jones
+Don't be tempted to write an adaptive reward scaling scheme. It's extra nonstationarity. Just hand-scale. -- Andy Jones
+If you're new to RL, writing things from scratch is the most catastrophically self-sabotaging thing you can do. -- Andy Jones
+When their RL implementation doesn't work, people are often keen to adjust their network architecture or hyperparameters. They're reluctant to say they've got a bug. Most often, it turns out they've got a bug. -- Andy Jones
+The default state of the world is that your research is false, because doing research is hard. -- Neel Nanda
+Figuring out a system's gears takes extra work up-front, but yields dividends forever. The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system. -- John Wentworth
+You can't find typos in your own writing without a great deal of effort because you know what it's supposed to say. -- Gwern Branwen
+Even a single anomaly, apparently trivial in itself, can indicate the everyday mental model is not just a little bit wrong, but fundamentally wrong. -- Gwern Branwen
+Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so. -- Patrick Kidger
+The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. -- Andrej Karpathy
+Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort. -- Andrew Ng
+Overfit a single batch of only a few examples. If they do not [overfit], there is a bug somewhere and we cannot continue to the next stage. -- Andrej Karpathy
+When someone's RL implementation isn't working, people copy-paste a screenshot of their loss curve because they know they want a pretty, exponentially-decaying loss curve. The shape of your loss curve says very little about where in your code you've messed up. -- Andy Jones
+The quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. -- Wang et al., ACL 2024
+If there are NaNs, we should not drop them, else we end up comparing different sample sets and it's invalid. A might be a single easy sample, and B might be all 128 hard samples. Of course A looks much better, but actually it failed on the vast majority of samples. -- wassname
+All labels in your dataset are -100. Training losses will be all 0. -- Unsloth troubleshooting FAQ
+Don't just do the first experiment that pops into your head. Think about the key ways the hypothesis could be false, and how you could test that. -- Neel Nanda
+Do ablations on your fancy method. It's easy for people to have a fancy method with lots of moving parts, when many actually are unnecessary. -- Neel Nanda
+Don't reinvent the wheel. A common mistake in mech interp is doing something that's already been done. We have LLM-powered literature reviews now. You have way less of an excuse. Check first! -- Neel Nanda
+Good writing is simple. There's a tendency towards verbosity or trying to make things sound more complex and fancy than they actually are, so they feel impressive. I think this is a highly ineffective strategy. -- Neel Nanda
+The standard hypothesis testing framework can be misleading: most of your probability mass should normally be on something I haven't thought of yet. -- Neel Nanda
+A perfect fit can always be obtained by using a model with enough parameters. Over-fitting a model to data is just as bad as failing to identify a systematic pattern in the data. -- Hyndman and Athanasopoulos, *Forecasting: Principles and Practice*
+We made exactly the same mistake in one of my projects on insect recognition. [...] The learned classifier was surprisingly good. But a saliency map revealed that it was reading the bubble patterns and ignoring the specimens. I was so embarrassed that I had made the oldest mistake in the book. Lesson: always randomize even if you don't know what you are controlling for! -- Thomas G. Dietterich, quoted in Gwern's *Tank* evidence collection
+The entropy of your policy network's outputs usually starts near 1, then rapidly falls for a while, then flattens out for the rest of training. If it drops to zero, your agent has collapsed into some — likely myopic — policy, and isn't exploring any more. -- Andy Jones
+Bugs are just one more source of noise and your neural net is going to try its damnedest to pull the signal out of that mess you're feeding it. -- Andy Jones
+Don't try to debug your implementation by just running it on your full task. That might take days! That way madness lies. -- Andy Jones
+I missed a multithreading bug for several months by ignoring a small but mysterious decay in frames per second. -- Dan Rahtz
+Your misconfigured neural net will throw exceptions only if you're lucky; most of the time it will train but silently work a bit worse. -- Andrej Karpathy
+A fast and furious approach to training neural networks does not work and only leads to suffering. -- Andrej Karpathy
+You can't tell it's broken if you can't see that it's breaking. -- Josh Achiam
+You can think up thousands of possible reasons for a failure. You can see only the actual cause. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Don't stop when you hear the pump. Go down to the basement and find out which pump. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Build instrumentation in. Use source code debuggers, debug logs, status messages, flashing lights, and rotten egg odors. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Remove the changes that didn't do what you expected. They probably did something you didn't expect. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Compare it with a good one. If the bad ones all have something that the good ones don't, you're onto the problem. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Check that it's really your fix that fixed it. Wubba! might not be the thing that did the trick. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Know that it never just goes away by itself. Make it come back by using the original Make It Fail methods. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Fix the cause. Tear out the useless eight-track deck before you burn out another transformer. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. RL is unstable enough that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system. -- Dan Rahtz
+We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance. -- Henderson et al., *Deep RL That Matters*
+When good programmers debug hard problems fast, it's usually because they understand the system well enough to track the important internal state in their head, letting them drastically reduce the solution space they're searching over. -- Ulisse Mini
+It seems important to really commit yourself to always investigate whenever you notice confusion. -- Dan Rahtz
+It turns out that bad labels are a huge problem in many popular benchmark datasets. -- Vincent Warmerdam
+Doing well on the training set is easy: just memorize the examples. The most common mistake among machine learning beginners is to test on the training data and have the illusion of success. -- Pedro Domingos
+Contamination of your classifier by test data can occur in insidious ways, for example if you use test data to tune parameters and do a lot of tuning. -- Pedro Domingos
+Most common neural net mistakes: you didn't try to overfit a single batch first; you forgot to toggle train/eval mode; you forgot to zero_grad before backward; you passed softmaxed outputs to a loss that expects raw logits. -- Andrej Karpathy
+Thinking view() and permute() are the same thing. -- Andrej Karpathy
+Rescale the rewards, but don't shift mean, as that affects agent's will to live. -- John Schulman, *Nuts and Bolts of Deep RL*
+Changing Anything Changes Everything. -- Sculley et al., *Hidden Technical Debt in Machine Learning Systems*
+Switching to the BOS dataloader changes the validation loss and makes all previous experiments not comparable in absolute value. The loss appears lower but this is fake to some extent. -- Andrej Karpathy, nanochat experiment log
+The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites. -- Stas Bekman
+The best way to debug an error that arises in trainer.train() is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve. -- Hugging Face course
+Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. Don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline. -- Hugging Face course
+Eliminate concurrency: restrict the number of processes to 1 for both training and data preprocessing. -- Axolotl debugging guide
+How reliable is my experiment? Ask yourself: How surprised would I be if it turned out to be complete bullshit due to a bug, error, noise, misunderstanding, etc.? Investigate the most uncertain bits. -- Neel Nanda
+Actively seek alternatives: what are the simplest explanations? What known circuits or phenomena could be involved? What would a strong skeptic argue? -- Neel Nanda
+This doesn't seem like it will work or I feel less motivated after trying a few things along this line that didn't work are not ruling out an idea. -- Jacob Steinhardt
+I had all the data necessary to make this realization a couple weeks in but had failed to do so. -- Jacob Steinhardt
+Most importantly, there is no point of launching 1000 runs with different hyperparameters: it is delusional to expect to get your biggest jumps of performance by simply tuning a few values. -- Clara Sanh
+Third, and perhaps most important for building skill, you must notice when you're going into brute-force search mode, and then take action by investing time in understanding the underlying system. -- Ulisse Mini
+Pro-tip: when you work with language, have a serious look at the outputs of the tokenizers. I can't count the number of lost hours I spent trying to reproduce results because something went wrong with the tokenization. -- Clara Sanh
+Error analysis can often help you figure out how promising different directions are. It might result in your team spending a month only to realize afterward that it resulted in little benefit. -- Andrew Ng
+If you are doing distributed training, print samples of your dataset in each process and triple-check that you get the same thing. -- Hugging Face course
+A valuable intuition: by default, all numbers are meaningless because we lack any scale to compare them. -- Neel Nanda
+If the loss or metric on your initial model is very different from the value you expect for random predictions, double-check how your loss or metric is computed: there is probably a bug there. -- Hugging Face course
+If a machine learning model can become state of the art by squeezing another 0.5% out of a validation set one has to wonder: are we really making a better model? Or are we creating a model that is better able to overfit on the bad labels? -- Vincent Warmerdam
+Most numerical errors manifest as all your metrics going weird at the same time: your loss exploding, your KL div collapsing, your rewards oscillating. From the outside, you can tell something is wrong but you've no idea what is wrong or where to start looking. -- Andy Jones
+If you arrive in RL expecting a garbage fire, you might just stay zen throughout. -- Andy Jones
+Iteration speed is a huge determinant of debugging speed. Running a test should take at most as long as it takes you to make a potential fix: a few seconds. -- Andy Jones
+Find tests that cut your system in half in some way, and tell you which half the problem is in. -- Andy Jones
+The wise thing to do is to look under the streetlight, or to look in the dark. Best moral I've heard for it is: it depends. -- Andy Jones
+Make sure you can walk before you try running. -- Andy Jones
+If it doesn't work, assume there's a bug. Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it's a bug. -- Josh Achiam
+Sometimes things will work in one environment even when you have a breaking bug. -- Josh Achiam
+Measure everything. Do a lot of instrumenting to see what's going on under-the-hood. -- Josh Achiam
+Backprop plus SGD does not magically make your network work. Batch norm does not magically make it converge faster. And just because you can formulate your problem as RL doesn't mean you should. -- Andrej Karpathy
+If you insist on using the technology without understanding how it works you are likely to fail. -- Andrej Karpathy
+What we try to prevent very hard is the introduction of a lot of unverified complexity at once, which is bound to introduce bugs or misconfigurations that will take forever to find, if ever. -- Andrej Karpathy
+The unambiguously correct place to visualize your data is immediately before y_hat = model(x). This is the only source of truth. -- Andrej Karpathy
+It is a depressing fact that your network will typically still train okay because it will learn to ignore data from the other examples. -- Andrej Karpathy
+You will have hypotheses that are wrong, experiments that are inconclusive, beautiful methods that lose to dumb baselines, etc. This is totally fine and normal. -- Neel Nanda
+It is easy to be sloppy in the name of speed and introduce many bugs that cost you time in the long-run. -- Neel Nanda
+LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation. -- Shankar et al.
+A ruler in a biopsy image can be correlated with malignancy because dermatologists use rulers for lesions that are a cause for concern. The algorithm doesn't know why, so it could misinterpret a random ruler sighting as grounds to diagnose cancer. -- Ricardo Novoa, quoted in Gwern's *Tank* evidence collection
+Rewarding each timestep without the pancake on the floor teaches the agent to hurl the pancake into the air as hard as possible. -- Christine Barron, quoted in Gwern's *Tank* evidence collection
+If you've learned nothing in 2 hours, pivot to another approach. If 2–3 approaches were dead ends, it's fine to just pick another problem. -- Neel Nanda
+It's all in the log. Well, the instrumentation is in the log, but what the tester saw and didn't like is not. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+They were ready to take him to the loony bin, when they noticed he wasn't wearing shoes. While he may be accused of being insane for working in a hardware lab with bare feet, he wasn't hallucinating about the bug. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+A problem with garbage characters proved to be correlated with the times that Fred was on duty. It turns out that Fred had a big gut, which would press on the keyboard when he reached up for the coffeepot. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Never trust your memory with a detail — write it down. The details you didn't think were important will prove to be the critical ones. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+The horror of that moment, the King went on, I shall never, never forget! You will, though, the Queen said, if you don't make a memorandum of it. -- Lewis Carroll, quoted by David J. Agans
+Just because you pay people $50 an hour doesn't mean that they know how to debug something. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+When you think you've fixed an engineering design, take the fix out. Make sure it's broken again. Put the fix back in. Make sure it's fixed again. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Everyone wants to believe that the bug just went away. Guess what? It will. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+If you have to ship it, ship it with a trap to catch it when it happens in the field. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Logs and other system-generated audit trails are much more reliable than users. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+When users report an error, they often give you the answer they assume is true instead of looking at the failure. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Reassembling any more than is absolutely necessary before testing makes it probable that you have not fixed the problem and will have to disassemble everything again, with a probability that increases in proportion to the amount of reassembly effort involved. -- Goldberg's Corollary to Murphy's Law, quoted by David J. Agans
+You may expect a wiring error to stop a terminal from ever working. It might work poorly because an unconnected blue wire and purple wire coupled enough signal across a hundred feet of cable. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Divide and Conquer is the only rule that actually involves finding the problem. All the others are just to help you follow this one. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+See the failure. The senior engineer saw the real failure and was able to find the cause. The junior guys thought they knew what the failure was and fixed something that wasn't broken. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Guess only to focus the search. Go ahead and guess that the memory timing is bad, but look at it before you build a timing fixer. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Grab the brass bar with both hands. If you try to fix the nuke without knowing what's wrong first, you may have an underwater Chernobyl on your hands. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+Don't assume that it was the wires and send that dirty fuel filter back onto the road. -- David J. Agans, *Debugging: The 9 Indispensable Rules*
+If each run takes 10 hours, you can easily waste a lot of time. Last run didn't work? OK, I think it's this thing. A week later, you still haven't solved the problem. -- Dan Rahtz
+If you have rapid feedback, you can narrow down the hypothesis space a lot faster by trying things than thinking carefully. -- Dan Rahtz
+When ruling out ideas, it is important to hold oneself to a high standard. -- Jacob Steinhardt
+If a result is exciting and cool, it's even more likely to be false than normal. -- Neel Nanda
+One common way an experiment fails is that it turns out to be more entangled than expected: all of the approaches you try might have the same underlying failure. -- Jacob Steinhardt
+Error goes up: commonly, this is due to a flipped sign somewhere in the loss function or gradient. Error explodes: usually a numerical issue, but can be a high learning rate. -- FSDL course
+Visualize the model in action. Directly observing the machine learning model performing its task will help determine whether the quantitative performance numbers it achieves seem reasonable. -- Goodfellow, Bengio and Courville
+By reaching a local optimum, learning curves can indicate successful optimization when the returns are not qualitatively representative of learning the desired behaviour. -- Henderson et al., *Deep RL That Matters*
+A graph of 7 tasks with 3 algorithms can look like one algorithm is best on all problems, but turn out to be the same algorithm with different random seeds. -- William Falcon
+The learning rate is a nuisance hyperparameter: we can only fairly compare models if it is tuned separately for each model. -- Google Tuning Playbook
+In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate. -- Andrej Karpathy
+The loss never went up in the first place. It was under-reporting loss due to exactly repeated data; it reached data it hadn't seen before and started reporting correctly. -- Stas Bekman
+The problem when you encounter an error in trainer.train() is that it could come from multiple sources. -- Hugging Face course
+Only when you manage to pass the overfitting test can you be sure that your model can actually learn something. -- Hugging Face course
+A clear condition that training works is that the model fits one batch, with the correct labels, at the expected loss. -- Clara Sanh
+If your loss or metric differs greatly from random predictions, check the loss function: the label can be wrong, the inputs can be wrong, or you might have a bug. -- Hugging Face course
+The standard hypothesis-testing framework has an implicit frame of being able to list all the hypotheses. But most of your probability mass should normally be on something I haven't thought of yet. -- Neel Nanda
+The first step is just making time to stop and ask yourself: do I endorse what I'm doing, and could I be doing something better? -- Neel Nanda
+Instability to random seed is like a canary in a coal mine. If pure randomness leads to this much variance between runs, imagine how much an actual difference in code could make. -- Alex Irpan
+Measure samples before the model sees them. Model inputs are the source of truth; upstream plots can lie. -- Andrej Karpathy
+UNDERSTAND THE SYSTEM MAKE IT FAIL QUIT THINKING AND LOOK DIVIDE AND CONQUER CHANGE ONE THING AT A TIME KEEP AN AUDIT TRAIL CHECK THE PLUG GET A FRESH VIEW IF YOU DIDN'T FIX IT, IT AIN'T FIXED -- curated in README.md
+**Quit Thinking and Look**: You can think up thousands of possible reasons for a failure. You can see only the actual cause. -- curated in README.md
+See the failure. The senior engineer saw the real failure and was able to find the cause. The junior guys thought they knew what the failure was and fixed something that wasn't broken. See the details. Don't stop when you hear the pump. Go down to the basement and find out which pump. Build instrumentation in. Use source code debuggers, debug logs, status messages, flashing lights, and rotten egg odors. Add instrumentation on. Use analyzers, scopes, meters, metal detectors, electrocardiography machines, and soap bubbles. Don't be afraid to dive in. So it's production software. It's broken, and you'll have to open it up to fix it. Watch out for Heisenberg. Don't let your instruments overwhelm your system. Guess only to focus the search. Go ahead and guess that the memory timing is bad, but look at it before you build a timing fixer. -- curated in README.md
+**Change One Thing at a Time**: You need some predictability in your life. Remove the changes that didn't do what you expected. They probably did something you didn't expect. -- curated in README.md
+Isolate the key factor. Don't change the watering schedule if you're looking for the effect of the sunlight. Grab the brass bar with both hands. If you try to fix the nuke without knowing what's wrong first, you may have an underwater Chernobyl on your hands. Change one test at a time. I knew my VGA capture phase was broken because nothing else was changing. Compare it with a good one. If the bad ones all have something that the good ones don't, you're onto the problem. Determine what you changed since the last time it worked. My friend had changed the cartridge on the turntable, so that was a good place to start. -- curated in README.md
+**If You Didn't Fix It, It Ain't Fixed**: And now that you have all these techniques, there's no excuse for leaving it unfixed. -- curated in README.md
+Check that it's really fixed. Don't assume that it was the wires and send that dirty fuel filter back onto the road. Check that it's really your fix that fixed it. "Wubba!" might not be the thing that did the trick. Know that it never just goes away by itself. Make it come back by using the original Make It Fail methods. If you have to ship it, ship it with a trap to catch it when it happens in the field. Fix the cause. Tear out the useless eight-track deck before you burn out another transformer. Fix the process. Don't settle for just cleaning up the oil. Fix the way you design machines. -- curated in README.md
+before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname -- curated in README.md
+Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz] -- curated in README.md
+If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname -- curated in README.md
+If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz] -- curated in README.md
+We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson] -- curated in README.md
+When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones] -- curated in README.md
+What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones] -- curated in README.md
+"If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow], -- curated in README.md
+The default state of the world is that your research is false, because doing research is hard.[^nanda] -- curated in README.md
+Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda] -- curated in README.md
+When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse] -- curated in README.md
+figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth] -- curated in README.md
+broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup] -- curated in README.md
+If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones] -- curated in README.md
+It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz] -- curated in README.md
+It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz] -- curated in README.md
+you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing] -- curated in README.md
+Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger] -- curated in README.md
+This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger] -- curated in README.md
+When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones] -- curated in README.md
+The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe] -- curated in README.md
+Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly] -- curated in README.md
+It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning] -- curated in README.md
+A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern] -- curated in README.md
+Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos] -- curated in README.md
+Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos] -- curated in README.md
+Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n] -- curated in README.md
+Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe] -- curated in README.md
+most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes] -- curated in README.md
+oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes] -- curated in README.md
+6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes] -- curated in README.md
+Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan] -- curated in README.md
+Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan] -- curated in README.md
+- If observations have unknown range, standardize - Compute running estimate of mean and standard deviation - x' = clip((x - mu)/sigma, -10, 10) - Rescale the rewards, but don't shift mean, as that affects agent's will to live - Standardize prediction targets (e.g., value functions) the same way -- curated in README.md
+Always Be Ablating - Different tricks may substitute - Especially whitening -- curated in README.md
+**Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley] -- curated in README.md
+Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook] -- curated in README.md
+The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook] -- curated in README.md
+In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe] -- curated in README.md
+We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains] -- curated in README.md
+Do note that switching to the BOS dataloader changes the validation loss and makes all previous experiments not comparable in absolute value of the loss, because we have a lot fewer "confusing" tokens in the train/val batches. [...] Therefore, the loss appears lower but this is "fake" to some extent.[^nanochat] -- curated in README.md
+Original implementation clipped local gradients before sync. Since this codebase doesn't use DDP (gradient sync is in the optimizers), each rank was clipping based on its own local norm.[^nanochat] -- curated in README.md
+As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman] -- curated in README.md
+In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes -- curated in README.md
+The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book] -- curated in README.md
+We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book] -- curated in README.md
+The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse] -- curated in README.md
+Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse] -- curated in README.md
+The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth] -- curated in README.md
+All labels in your dataset are -100. Training losses will be all 0.[^unsloth] -- curated in README.md
+**Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl] -- curated in README.md
+Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl] -- curated in README.md
+4. Think your algorithm is working but you're actually seeing random noise. - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds. -- curated in README.md
+Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research.[^nanda-mindsets] -- curated in README.md
+**The challenge lies in the fact that you can make these mistakes, train a model without it ever crashing, and still get a decent performance…**[^sanh] -- curated in README.md
+-   It is all well and good to make comparisons of validation error rates estimated on a finite validation set using fastidious statistical tests, but often the trial variance alone can produce statistically significant differences between two different trained models that use the same hyperparameter settings.[^tuning-playbook] -- curated in README.md
+**How reliable is my experiment?** Ask yourself: "How surprised would I be if it turned out to be complete bullshit due to a bug, error, noise, misunderstanding, etc.?" Investigate the most uncertain bits[^nanda-papers] -- curated in README.md
+Insufficient Skepticism: Missing simple alternative explanations, methodological flaws, or bugs. Explicitly list alternatives. Get others (especially mentors) to red team your plans before you run them. Actively try to break your hypothesis. Ask "What observation would make me abandon this?"[^nanda-taste] -- curated in README.md
+**Trying an experiment and seeing it fail gives little information by itself.** When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work".[^steinhardt] -- curated in README.md
+When ruling out ideas, it is important to hold oneself to a high standard. "This doesn't seem like it will work" or "I feel less motivated after trying a few things along this line that didn't work" are _not_ ruling out an idea.[^steinhardt] -- curated in README.md
+When a machine learning system performs poorly, it is usually difficult to tell whether the poor performance is intrinsic to the algorithm itself or whether there is a bug in the implementation of the algorithm. Machine learning systems are difficult to debug for various reasons.[^goodfellow] -- curated in README.md
+It ended up taking me 6 weeks to reproduce results, thanks to several software bugs. The question is, why did it take so long to find these bugs?[^irpan] -- curated in README.md
+**Result:** This was not an out-of-the-box win for nanochat even with a mild attempt over a few hours at a bit of tuning and debugging. The idea itself is intuitively appealing. Might come back around later to try harder later.[^nanochat] -- curated in README.md
+Our specific recommendations to researchers include: 1. Computing standard errors of the mean using the Central Limit Theorem 2. When questions are drawn in related groups, computing clustered standard errors 3. Reducing variance by resampling answers and by analyzing next-token probabilities 4. When two models are being compared, conducting statistical inference on the question-level paired differences, rather than the population-level summary statistics 5. Using power analysis to determine whether an eval (or a random subsample) is capable of testing a hypothesis of interest[^miller] -- curated in README.md
+If you keep that strategy when each run takes 10 hours, though, you can easily waste a *lot* of time. Last run didn’t work? OK, I think it’s this thing. Let’s set off another run to check. Coming back the next morning: still doesn’t work? OK, maybe it’s this other thing. Let’s set off another run. A week later, you still haven’t solved the problem.[^rahtz] -- curated in README.md
+than forming hypotheses. Why spend 15 minutes carefully considering everything that could be causing what you see when you can check the first idea that jumps to mind in a fraction of that (and gather more evidence in the process)? To put it another way: if you have rapid feedback, you can narrow down the hypothesis space a lot faster by trying things than thinking carefully.[^rahtz] -- curated in README.md
+The standard hypothesis testing framework can be misleading here, because it has an implicit frame of being able to list all the hypotheses. But actually, most of your probability mass should normally be on “something I haven’t thought of yet”[^nanda-mindsets] -- curated in README.md
+If trying to explain something mysterious, novice researchers often neglect simple, dumb hypotheses like “maybe MLP0 is incredibly important on *every* input, and there’s nothing special going on with my prompt”[^nanda] -- curated in README.md
+Importantly, it is often not obvious that multiple approaches to a problem all have the same issue. In the past, I have spent months trying different approaches to a problem before finally stepping back and realizing that they were all failing for the same reason. Moreover, I had all the data necessary to make this realization a couple weeks in but had failed to do so.[^steinhardt] -- curated in README.md
+* **Error goes up**: Commonly, this is due to a flip sign somewhere in the loss function/gradient. * **Error explodes**: This is usually a numerical issue but can also be caused by a high learning rate. * **Error oscillates**: You can lower the learning rate and inspect the data for shuffled labels or incorrect data augmentation. * **Error plateaus**: You can increase the learning rate and get rid of regulation. Then you can inspect the loss function and the data pipeline for correctness.[^fsdl] -- curated in README.md
+Actively Seek Alternatives: Explicitly brainstorm other ways your observations could be explained. What are the simplest explanations? What known circuits or phenomena could be involved? What would a strong skeptic argue?[^nanda-taste] -- curated in README.md
+**If it doesn’t work, assume there’s a bug.** Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it’s a bug. Bad hyperparameters can significantly degrade RL performance, but if you’re using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue.[^spinningup] -- curated in README.md
+For example, perhaps you forgot to flip your labels when you left-right flipped the image during data augmentation. Your net can still (shockingly) work pretty well because your network can internally learn to detect flipped images and then it left-right flips its predictions. Or maybe your autoregressive model accidentally takes the thing it’s trying to predict as an input due to an off-by-one bug. Or you tried to clip your gradients but instead clipped the loss, causing the outlier examples to be ignored during training. Or you initialized your weights from a pretrained checkpoint but didn’t use the original mean. Or you just screwed up the settings for regularization strengths, learning rate, its decay rate, model size, etc.[^karpathy-recipe] -- curated in README.md
+Most importantly, there is no point of launching 1000 runs with different hyperparameters (or architecture tweaks like activation functions): **compare a couple of runs with different hyperparameters to get an idea of which hyperparameters have the highest impact** but in general, it is delusional to expect to get your biggest jumps of performance by simply tuning a few values. For instance, if your best performing model is trained with a learning rate of 4e2, there is probably something more fundamental happening inside your neural network and you want to identify and understand this behavior so that you can re-use this knowledge outside of your current specific context.[^sanh] -- curated in README.md
+Once the algorithm was partially working, they would attain higher performance by looking for remaining bugs, both by reviewing the code carefully, and by collecting metrics such as average policy entropy to perform sanity-checks, rather than just tune hyperparameters.[^olsson] -- curated in README.md
+Third, and perhaps most important for building skill,[[1]](https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming#fn289bs9hi65b)you must **notice** when you're going into brute-force search mode, and then **take action** by investing time in understanding the underlying system, until both the problem and solution make sense.[^ulisse] -- curated in README.md
+Things I've tried (but maybe not systematically enough):  * Different initial LRs * Different optimizers * Different number of hidden layers/units * Shared pi/V NN body (with diff output layers) vs not * Changing amount of entropy * Adding correlated noise * Using TD residual instead of MC version * Clipping the gradient * Different gamma values -- curated in README.md
+Visualize the model in action: When training a model to detect objects in images, view some images with the detections proposed by the model displayed superimposed on the image. When training a generative model of speech, listen to some of the speech samples it produces. This may seem obvious, but it is easy to fall into the practice of looking only at quantitative performance measurements like accuracy or log-likelihood. Directly observing the machine learning model performing its task will help to determine whether the quantitative performance numbers it achieves seem reasonable. Evaluation bugs can be some of the most devastating bugs because they can mislead you into believing your system is performing well when it is not.[^goodfellow] -- curated in README.md
+By reaching a local optimum, learning curves can indicate successful optimization of the policy over time, when in reality the returns achieved are not qualitatively representative of learning the desired behaviour, as demon-strated in video replays of the learned policy 5. Therefore, it is important to show not only returns but demonstrations of the learned policy in action.[^henderson] -- curated in README.md
+2. Make sure observations usable: - See if YOU could control the system by using the same observations you give the agent. - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way. -- curated in README.md
+Pro-tip: when you work with language, have a serious **look at the outputs of the tokenizers**. I can’t count the number of lost hours I spent trying to reproduce results (and sometimes my own old results) because something went wrong with the tokenization.[^sanh] -- curated in README.md
+Error analysis can often help you figure out how promising different directions are. I’ve seen many engineers reluctant to carry out error analysis. It often feels more exciting to just jump in and implement some idea, rather than question if the idea is worth the time investment. This is a common mistake: It might result in your team spending a month only to realize afterward that it resulted in little benefit.[^ng-mly] -- curated in README.md
+⚠️ If you are doing distributed training, print samples of your dataset in each process and triple-check that you get the same thing. One common bug is to have some source of randomness in the data creation that makes each process have a different version of the dataset.[^hfcourse] -- curated in README.md
+-   Although in many cases the primary objective of our experiments only requires considering the validation error of each trial, we must be careful when reducing each trial to a single number because it can hide important details about what’s going on below the surface. -   For every study, we always look at the **training curves** (training error and validation error plotted versus training step over the duration of training) of at least the best few trials.[^tuning-playbook] -- curated in README.md
+(I missed a multithreading bug for several months by ignoring a small but mysterious decay in frames per second.)[^rahtz] -- curated in README.md
+There was no real spike in the two earlier runs. The loss never went up in the first place. In both resumes it was under-reporting loss due to an exactly repeated data and then it reached data it hasn't seen before and started reporting correctly. In other words it was overfitting and reporting a false loss.[^bekman-book] -- curated in README.md
+**Do ablations on your fancy method**: It's easy for people to have a fancy method with lots of moving parts, when many actually are unnecessary. You should always try removing one part and see if the method breaks. Do this for each part. *   For example, the [original unlearning method](https://arxiv.org/abs/2403.03218v1) in the [RMU paper](https://arxiv.org/abs/2403.03218) claimed it was based on finding a meaningful steering vector, until follow-up work found that it was just about adding a vector with really high norm that broke the model, and a random vector performed just as well.[^nanda] -- curated in README.md
+The only way to find out what needs work is to implement something quickly, -- curated in README.md
+and find out what parts break.[^cs229] -- curated in README.md
+Figure 15.5: An autoencoder trained with mean squared error for a robotics task has failed to reconstruct a ping pong ball. The existence of the ping pong ball and all its spatial coordinates are important underlying causal factors that generate the image and are relevant to the robotics task. Unfortunately, the autoencoder has limited capacity, and the training with mean squared error did not identify the ping pong ball as being salient enough to encode.[^goodfellow-ch15] -- curated in README.md
+One of the key drivers of progress in mech interp is an openness to qualitative research: summary statistics lose a ton of information. What can we learn by actually looking deeply into what's happening?[^nanda] -- curated in README.md
+1. **Test reward function standalone**: Run it outside training with known inputs to verify it returns nonzero values.[^axolotl-stability] -- curated in README.md
+In most cases, we do not know a priori what the intended behavior of the algorithm is. In fact, the entire point of using machine learning is that it will discover useful behavior that we were not able to specify ourselves. If we train a neural network on a new classification task and it achieves 5 percent test error, we have no straightforward way of knowing if this is the expected behavior or suboptimal behavior.[^goodfellow] -- curated in README.md
+A valuable intuition to have in mind is that, by default, all numbers are meaningless because we lack any scale to compare them. E.g. if a probe gets 95% classification accuracy on some task, is this good? Is this bad? Hard to say without knowing more! Baselines are one way to get context to compare against.[^nanda-draft] -- curated in README.md
+You might be temped to keep track of the difference \(\mid f’\_a - f’\_n \mid \) or its square and define the gradient check as failed if that difference is above a threshold. However, this is problematic. For example, consider the case where their difference is 1e-4. This seems like a very appropriate difference if the two gradients are about 1.0, so we’d consider the two gradients to match. But if the gradients were both on order of 1e-5 or lower, then we’d consider 1e-4 to be a huge difference and likely a failure.[^cs231n] -- curated in README.md
+*   How would a random predictor perform (especially in classification problems)? Dataset can be unbalanced… *   What would the loss look like for a random predictor? *   What is (are) the best metric(s) to measure progress on my task? *   What are the limits of this metric? If it’s perfect, what can I conclude? What can’t I conclude?[^sanh] -- curated in README.md
+If the loss/metric you get on your initial model is very different from the loss/metric you would expect for random predictions, double-check the way your loss or metric is computed, as there is probably a bug there. If you are using several losses that you add at the end, make sure they are of the same scale.[^hfcourse] -- curated in README.md
+5.   **Rule of thumb: 400 episodic return in breakout**: Check if your PPO could obtain 400 episodic return in breakout. We have found this to be a practical rule of thumb to determine the fidelity of online PPO implementations in GitHub. Often we found PPO repositories not able to do this, and we know they probably do not match all implementation details of `openai/baselines`’ PPO.[^ppo37] -- curated in README.md
+The issue here isn't just that we might have bad labels in our training set, the issue is that it appears in the validation set. If a machine learning model can become state of the art by squeezing another 0.5% out of a validation set one has to wonder. Are we really making a better model? Or are we creating a model that is better able to overfit on the bad labels?[^koaning] -- curated in README.md
+broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task. -- Achiam -- curated in SKILL.md
+If one part is broken, the other parts can adapt and still achieve roughly acceptable performance -- Goodfellow, Bengio and Courville -- curated in SKILL.md
+Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem -- Godbole, Dahl, Gilmer, Shallue and Nado -- curated in SKILL.md
+Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research. -- Nanda -- curated in SKILL.md
+Read your data. Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad. -- Nanda -- curated in SKILL.md
+How would a random predictor perform (especially in classification problems)? [...] What would the loss look like for a random predictor? [...] What are the limits of this metric? If it's perfect, what can I conclude? What can't I conclude? -- Sanh -- curated in SKILL.md
+**NEVER STOP**: Once the experiment loop has begun (after the initial setup), do NOT pause to ask the human if you should continue. Do NOT ask 'should I keep going?' or 'is this a good stopping point?'. The human might be asleep, or gone from a computer and expects you to continue working *indefinitely* until you are manually stopped. You are autonomous. If you run out of ideas, think harder — read papers referenced in the code, re-read the in-scope files for new angles, try combining previous near-misses, try more radical architectural changes. The loop runs until the human interrupts you, period. -- Karpathy, [autoresearch/program.md](https://github.com/karpathy/autoresearch/blob/master/program.md) -- curated in SKILL.md
+Build it up as you go, don't think you can build it ahead of time. Be focused on a strong mental model of what options you have (including architectural changes and losses) that you think should affect what metrics in the logs. -- wassname -- curated in SKILL.md
+Before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possibility and brainstorm the cheapest tests that may narrow them down. -- wassname -- curated in SKILL.md
+If you are stuck, find a working reference implementation and compare it to yours. If nothing jumps out, try a bisection search: adapt their code wholesale, then half their features, and so on. -- wassname -- curated in SKILL.md
+Summarise your concept and pseudocode and do an external review in scientist mode. Perhaps describe the forward and backward pass as mermaid too. -- wassname -- curated in SKILL.md
+The CNN has learned to detect a metal token that radiology technicians place on the patient in the corner of the image field of view at the time they capture the image. -- Zech et al. -- curated in SKILL.md
+Apparently meaningless identifier columns were the most important predictors. [...] the university only filled out much of this information *after* a grant application was accepted. -- Howard and Gugger -- curated in SKILL.md
+by default, all numbers are meaningless because we lack any scale to compare them. E.g. if a probe gets 95% classification accuracy on some task, is this good? Is this bad? Hard to say without knowing more! -- Nanda -- curated in SKILL.md
+If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky. -- Irpan -- curated in SKILL.md
+It ended up taking me 6 weeks to reproduce results, thanks to several software bugs. The question is, why did it take so long to find these bugs? -- Rahtz -- curated in SKILL.md
+Don't be tempted to write an adaptive reward scaling scheme. It's extra nonstationarity. Just hand-scale. -- Andy Jones -- curated in rl/SKILL.md
+If you're new to RL, writing things from scratch is the most catastrophically self-sabotaging thing you can do. -- Andy Jones -- curated in rl/SKILL.md
+Rathore et al. 2024: "the estimate of the κ grows polynomially with nres" -- but this is in raw units. Nondimensionalization reduces the effective condition number by making all PDE coefficients O(1). -- curated in pinn/SKILL.md
+Wang et al. propose a modified MLP with multiplicative interactions. With `U = φ(XW1 + b1)`, `V = φ(XW2 + b2)` two nonlinear encodings of the input (φ = tanh) and a per-layer gate `Z(k) = φ(H(k)Wz,k + bz,k)` computed from the hidden state, the update is `H(k+1) = (1 - Z(k)) * U + Z(k) * V`. Authors claim a ~3x decrease in the leading Hessian eigenvalue. -- curated in pinn/SKILL.md
+Factorize each neuron's weight vector as w = s * w_unit, where s is a trainable scalar and w_unit is the unit-normalized direction. This changes the optimization geometry so the loss surface has better-conditioned local minima. "Predictions obtained by RWF are in excellent agreement with ground truth, while other weight parameterizations result in poor or non-physical approximations." -- curated in pinn/SKILL.md
+Used in the PirateNet architecture alongside causal training, sequence-to-sequence, and Fourier features. Simple to implement as a custom parameterization on Linear layers. -- curated in pinn/SKILL.md
+Instead of data-augmenting with transformed copies, bake symmetries directly into the architecture so every model in the function space is automatically invariant/equivariant. For turbulence closure (Reynolds stress from velocity gradients), custom tensor layers enforce Galilean invariance by construction. "The Galilean invariant model is more accurate than the other models" and generalizes better across flow configurations. -- curated in pinn/SKILL.md
+Lecture: Brunton, S. "AI/ML+Physics Part 3 - Designing an Architecture." https://www.youtube.com/watch?v=fiX8c-4K0-Q Key distinction: invariance (output unchanged by transformation, e.g., energy is frame-invariant) vs equivariance (output transforms same way as input, e.g., stress tensor rotates with frame). Equivariant architectures are more general. If your PDE has known symmetries (translation, rotation, scaling), enforce them architecturally rather than hoping the optimizer discovers them. **Caveat**: This works best for local closure terms (Reynolds stress, turbulence models) and unbounded/periodic domains where the global symmetry holds everywhere. If your domain has boundary conditions that break the symmetry (e.g., a wall breaks rotational invariance), enforcing the symmetry globally in the architecture will prevent the solution from satisfying the BCs -- the architecture will be fighting the problem. In bounded domains, use symmetry-enforcing architectures only for terms where the symmetry genuinely holds (e.g., the constitutive relation), not for the full solution field. Libraries like `e3nn` implement this but add significant computational overhead. -- curated in pinn/SKILL.md
+Rathore et al. 2024 (ICML, credence ~80%): "Adam+L-BFGS attains 14.2x smaller L2RE than Adam on convection and 6.07x smaller than L-BFGS on wave." Tested on 3 PDEs (convection, reaction, wave), 5 seeds, widths 50-400. -- curated in pinn/SKILL.md
+"on the convection PDE, a loss of 10^-3 yields an L2RE around 10^-1, but decreasing the loss by a factor of 100 to 10^-5 yields an L2RE around 10^-2, a 10x improvement." -- curated in pinn/SKILL.md
+"L-BFGS stops in these cases without reaching a critical point: the gradient norm is around 10^-2 or 10^-3. The gradient still contains useful information for improving the loss." -- curated in pinn/SKILL.md
+Cause: strong Wolfe line search fails, step size goes to zero. Fix: switch to NNCG (Armijo only) or restart with different LR. -- curated in pinn/SKILL.md
+Theorem 8.4 (Section 8.2): condition number = Omega(nres^alpha) with alpha > 1/2, given eigenvalues of A o K_inf decaying as O(j^-2alpha). nres typically ranges 1e3 to 1e4. Separately, measured condition numbers near a solution are often > 1e4 (Section 6.2, Figure 3). -- curated in pinn/SKILL.md
+L2 norm (MSE) on residuals: default; promotes smooth, low-frequency solutions. L1 norm (MAE) on residuals: more robust to outlier collocation errors and sharp gradients (shocks) since it doesn't square-penalize large pointwise residuals. This is distinct from L1 *regularization on equation coefficients*, which is what SINDy and sparse equation discovery use to promote parsimony (few active terms). Don't conflate the two: L1 residual = robust fitting; L1 coefficient regularization = sparse model selection. For standard PINNs with a known PDE, L2 is correct. L1 residual loss is worth trying if you have shocks or suspect outlier collocation points. -- curated in pinn/SKILL.md
+Wang et al. 2021 (credence ~80%): "the gradients corresponding to the boundary loss term Lub(θ) in each layer are sharply concentrated around zero and overall attain significantly smaller values than the gradients corresponding to the PDE residual loss Lr(θ)." Shown via per-layer histograms of back-propagated gradients; the paper does not quantify the gap in orders of magnitude. -- curated in pinn/SKILL.md
+Wang et al. 2021: "many eigenvalues of the residual-loss Hessian are extremely large up to 1e5" while the boundary-loss Hessian eigenvalues stay small, so the gradient-flow stiffness is dominated by the residual term. This is an absolute magnitude, not a condition number; Wang never reports one. -- curated in pinn/SKILL.md
+For a condition number, use Rathore Figure 3: outlier eigenvalues > 1e4 (convection), > 1e3 (reaction), > 1e5 (wave). -- curated in pinn/SKILL.md
+Adaptively weight each loss term inversely proportional to its gradient magnitude. EMA of gradient statistics for stability. -- curated in pinn/SKILL.md
+NeuralPDE.jl implements this as `GradientScaleAdaptiveLoss`. -- curated in pinn/SKILL.md
+Instead of summing loss gradients (which can cancel), project them into a conflict-free direction. ConFIG: unit-normalize per-loss gradients, solve least-squares for combined direction, rescale by projection lengths. -- curated in pinn/SKILL.md
+Key: must compute per-loss gradients separately (zero_grad + backward for each). Summing raw losses defeats the purpose. M-ConFIG: momentum variant, updates only one loss's gradient per step. Use with SGD, not Adam (momentum conflict). -- curated in pinn/SKILL.md
+Standard PINNs use penalized (soft) constraints: add physics as a loss term. The alternative is constrained optimization: minimize data error while exactly satisfying the physics constraints. "With a loss function you're not exactly satisfying your constraints. With constrained optimization you are." -- curated in pinn/SKILL.md
+Physics-informed DMD (Baddoo et al. 2021) is the cleanest example: restrict the DMD matrix to a symmetry-preserving manifold (Hermitian, symplectic, etc.) via the Procrustes problem. KKT closed-form solutions exist because DMD is linear in its parameters -- the constraint is linear in both the output and the parameters simultaneously. Baddoo et al. 2021. "Physics-informed dynamic mode decomposition." Proc. R. Soc. A. https://arxiv.org/pdf/2112.04307 **Critical caveat for PINNs**: A BC like u(0)=0 is affine in the output u, but it is nonlinear in the NN weights theta. Closed-form KKT does NOT apply to neural network parameters. For NN-based PINNs, the two options for hard constraints are: (a) architectural -- multiply output by a distance function that satisfies the BC (Section 4 item 8), or (b) Augmented Lagrangian Methods (ALM), which are iterative and substantially more complex than Adam. Constrained optimization is most practical for linear models (DMD, SINDy, linear state-space) where the parameters enter linearly. -- curated in pinn/SKILL.md
+When the PINN fails on hard PDE regimes (high convection coefficient, strong reaction), don't start there. Start with easy parameters (small coefficient), train to convergence, then warm-start and increase to the target regime. 1-2 orders of magnitude improvement over naive training. "The curriculum training approach achieves significantly better errors, as well as lower variance in the error." (From Figure E.2 showing 10 seeds) -- curated in pinn/SKILL.md
+For time-dependent PDEs: train on a short time window, predict next state, step forward. Don't train on full space-time at once. "Posing the problem as seq2seq learning results in significantly lower error. The difference is particularly striking for reaction and reaction-diffusion cases, where seq2seq decreases error by almost two orders of magnitude." -- curated in pinn/SKILL.md
+NeuralPDE.jl calls this time-marching; see `WeightedIntervalTraining`. Note: these failures are not due to limited NN expressivity -- the architecture has enough capacity. The problem is optimization difficulty from the soft PDE constraint. -- curated in pinn/SKILL.md
+Standard PINNs trained by gradient descent are implicitly biased toward minimizing residuals at *later* times before even fitting the initial conditions -- violating physical causality. The NTK analysis shows the residual at time t is influenced more by residuals at later t' > t than earlier ones. This makes PINNs fail on chaotic/turbulent systems. Fix: weight each temporal residual point by wi = exp(-epsilon * sum_j<i R_j(theta)), where R_j is the accumulated residual before time i. This forces earlier times to converge first before the loss "turns on" at later times. "10-100x improvements in accuracy compared to competing approaches. First time PINNs succeeded on chaotic Lorenz, Kuramoto-Sivashinsky, and 2D Navier-Stokes in turbulent regime." -- curated in pinn/SKILL.md
+Key difference from seq2seq/curriculum: causal weighting works within a single continuous training, without requiring separate time windows or changing the PDE coefficients. Can be combined with seq2seq for further gains. Sensitivity: epsilon controls the steepness of the causal weights. Too small = residuals at later times turn on too early. Too large = training stalls on early time steps. Anneal epsilon during training. -- curated in pinn/SKILL.md
+Instead of penalizing BC violations (soft), multiply the PINN output by a distance function phi(x) that is zero on the boundary. Then u(x) = phi(x) * NN(x) satisfies BCs exactly by construction. "We eliminate modeling error associated with the satisfaction of boundary conditions. The sole contribution to the loss function is from the residual error at interior collocation points." "The proposed approach consistently outperforms a standard PINN-based collocation method." -- curated in pinn/SKILL.md
+Domain-specific failure modes and hard BC examples: see [references/heat_exchanger.md](references/heat_exchanger.md). -- curated in pinn/SKILL.md
+Domain-specific: differentiable EoS wrapping (REFPROP/PCHIP), IC handling for plant data, multi-episode training. See [references/heat_exchanger.md](references/heat_exchanger.md). -- curated in pinn/SKILL.md
+ConFIG authors report improvements over PCGrad and an Adam baseline on Burgers, Schrodinger, Kovasznay, and Beltrami. This is author-reported evidence, not a general comparison with UPGrad. -- curated in pinn/SKILL.md
+Jagtap et al. 2020. "Extended physics-informed neural networks (XPINNs): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations." Commun. Comput. Phys. https://arxiv.org/pdf/2005.11025 Credence ~70%: Multiple citations, implemented in DeepXDE. Enables parallelization; each subdomain network is smaller and easier to optimize. Key: interface residuals must be added as additional loss terms. Continuity of u and its normal derivative across interfaces. Useful when the solution has different character in different regions (e.g., different phases, boundary layers). -- curated in pinn/SKILL.md
+Toscano et al. 2024: PIKANs "lead to smaller models and may also contribute to lowering computational cost while maintaining good accuracy." -- curated in pinn/SKILL.md
+Credence ~40%: New, no independent replication. Other authors focus on improving PINNs within the MLP framework, not validating PIKANs as an alternative. Interesting but unproven. -- curated in pinn/SKILL.md
+the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. [...] e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator.[^wang] -- curated in references/llm_judges.md
+Did it time out, or was it reasoning behind a buffer for longer than you think? Check the event timestamps and transport state. -- wassname -- curated in references/llm_judges.md
+Note from wassname's LLM to other LLMs: when you SHOW the human any of this (a judge trace, a compared A/B pair, a sample, a prompt design), reproduce it in FULL and verbatim, prompts, separators, special tokens, and the reasoning/thinking block included. Your trained instinct is to digest and trim; resist it, that instinct destroys exactly the boilerplate (system prompt, special tokens, separators) that the human needs to debug. Formatting may only HIGHLIGHT or REORGANISE: bold or underline the diff, split into a table, use sections, but never drop, elide with "...", or paraphrase. When comparing variants, show ALL of them this way, not a representative one. Link the source file so they can open the raw (`results/runs/.../x.jsonl`, ideally `path:line`). "Show me" means reproduce, not describe. Trim only when explicitly asked. -- CLAUDE -- curated in references/llm_judges.md
+"If there are NaNs, we should not drop them, else we end up comparing different sample sets and it's invalid. A might be a single easy sample, and B might be all 128 hard samples. Of course A looks much better, but actually it failed on the vast majority of samples." - wassname, lightly edited for spelling -- curated in references/llm_judges.md
+Across the 36-model result set, the model-average first-shown pick rate is 64.3%, with a median of 65.4%. **The model-average absolute first-position lift is 15.7 percentage points.** So the aggregate pattern is not a subtle tie-breaker: the displayed order materially changes many judgments. -- curated in references/llm_judge_litreview.md
+The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. **While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions.** Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications. -- curated in references/llm_judge_litreview.md
+We find evidence of position bias, which is especially prevalent in smaller LLM labelers (see Appendix B). **To mitigate the effect of position bias, two inferences are made for every pair of candidates, where the order in which candidates are presented to the LLM is reversed for the second inference.** The results from both inferences are then averaged to obtain the final preference distribution. -- curated in references/llm_judge_litreview.md
+As observed in the figure, models larger than 7B exhibit significantly less self-preference bias compared to those of 7B or smaller. **For example, the DBG score of Qwen2.5-0.5B-Instruct is 41.7%. In contrast, the DBG score of Qwen2.5-14B-Instruct is only 2.1%.** This suggests that LLM judging tasks should utilize larger models to obtain more accurate and unbiased judgment results. -- curated in references/llm_judge_litreview.md
+Empirical results demonstrate that JudgeLRM not only surpasses proprietary models like GPT-4 and DeepSeek-R1 but also outperforms SFT and RL baselines of comparable sizes, **with an average improvement of 8.14% in F1 score over SFT counterparts.** -- curated in references/llm_judge_litreview.md
+We observe an initial increase (similar to (Muennighoff et al., 2025; Aggarwal & Welleck, 2025)) in accuracy as the average thinking budget increases. **For example, in Figure 2(a), accuracy increases from 82.2% to 87.3% as the average number of thinking tokens increases from 385 to 1100.** However, this trend does not continue indefinitely. -- curated in references/llm_judge_litreview.md
+# Some env for reasoning effort if you using litellm https://github.com/BerriAI/litellm/blob/main/litellm/constants.py#L81 DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET=24576 DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET=8192 DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET=1024 -- curated in references/llm_judge_litreview.md
+**Results confirm that accuracy gains plateau early and, in some configurations, decline at high sample counts** — a pattern inconsistent with diminishing returns alone and more consistent with noise introduction on problems that were already solved. This suggests self-consistency should be reserved for genuinely difficult problems rather than applied as a default scaling strategy. -- curated in references/llm_judge_litreview.md
+On MATH-500, Flash-Lite accuracy improved through approximately 10 sampled paths before plateauing and then declining slightly beyond 15, as shown in Figure 2. **This decline is notable: it suggests that once a model reliably solves most problems, additional samples introduce occasional wrong reasoning paths that the aggregator cannot fully suppress.** -- curated in references/llm_judge_litreview.md
+While they perform well in short contexts (<1K), performance degrades significantly as context length increases. **At 32K, for instance, 11 models drop below 50% of their strong short-length baselines.** Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. -- curated in references/llm_judge_litreview.md
+We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. **In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.** Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models. -- curated in references/llm_judge_litreview.md

scripts/harvest_curated_fortunes.py

@@ -0,0 +1,66 @@
+from __future__ import annotations
+
+import argparse
+import re
+from pathlib import Path
+
+SOURCES = (
+    Path("README.md"),
+    Path("SKILL.md"),
+    Path("rl/SKILL.md"),
+    Path("pinn/SKILL.md"),
+    Path("references/llm_judges.md"),
+    Path("references/llm_judge_litreview.md"),
+)
+
+METADATA = re.compile(r"^(Source|Evidence|Credence|Code|Implication):")
+
+
+def normalized(text: str) -> str:
+    return re.sub(r"[^a-z0-9]+", "", text.lower())
+
+
+def quotes(path: Path) -> list[str]:
+    records: list[str] = []
+    lines: list[str] = []
+
+    def flush() -> None:
+        if lines:
+            text = " ".join(lines)
+            records.append(f"{text} -- curated in {path}")
+            lines.clear()
+
+    for line in path.read_text().splitlines():
+        if not line.startswith("> "):
+            flush()
+            continue
+        text = line[2:].strip()
+        if METADATA.match(text):
+            flush()
+            continue
+        lines.append(text)
+    flush()
+    return records
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("output", type=Path)
+    args = parser.parse_args()
+
+    seen: set[str] = set()
+    records: list[str] = []
+    for path in SOURCES:
+        for record in quotes(path):
+            key = normalized(record.rsplit(" -- curated in ", 1)[0])
+            if key not in seen:
+                seen.add(key)
+                records.append(record)
+
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text("\n".join(records) + "\n")
+    print(f"{len(records)} curated quote blocks")
+
+
+if __name__ == "__main__":
+    main()

slop/research/curated-fortunes.txt

@@ -0,0 +1,172 @@
+UNDERSTAND THE SYSTEM MAKE IT FAIL QUIT THINKING AND LOOK DIVIDE AND CONQUER CHANGE ONE THING AT A TIME KEEP AN AUDIT TRAIL CHECK THE PLUG GET A FRESH VIEW IF YOU DIDN'T FIX IT, IT AIN'T FIXED -- curated in README.md
+**Quit Thinking and Look**: You can think up thousands of possible reasons for a failure. You can see only the actual cause. -- curated in README.md
+See the failure. The senior engineer saw the real failure and was able to find the cause. The junior guys thought they knew what the failure was and fixed something that wasn't broken. See the details. Don't stop when you hear the pump. Go down to the basement and find out which pump. Build instrumentation in. Use source code debuggers, debug logs, status messages, flashing lights, and rotten egg odors. Add instrumentation on. Use analyzers, scopes, meters, metal detectors, electrocardiography machines, and soap bubbles. Don't be afraid to dive in. So it's production software. It's broken, and you'll have to open it up to fix it. Watch out for Heisenberg. Don't let your instruments overwhelm your system. Guess only to focus the search. Go ahead and guess that the memory timing is bad, but look at it before you build a timing fixer. -- curated in README.md
+**Change One Thing at a Time**: You need some predictability in your life. Remove the changes that didn't do what you expected. They probably did something you didn't expect. -- curated in README.md
+Isolate the key factor. Don't change the watering schedule if you're looking for the effect of the sunlight. Grab the brass bar with both hands. If you try to fix the nuke without knowing what's wrong first, you may have an underwater Chernobyl on your hands. Change one test at a time. I knew my VGA capture phase was broken because nothing else was changing. Compare it with a good one. If the bad ones all have something that the good ones don't, you're onto the problem. Determine what you changed since the last time it worked. My friend had changed the cartridge on the turntable, so that was a good place to start. -- curated in README.md
+**If You Didn't Fix It, It Ain't Fixed**: And now that you have all these techniques, there's no excuse for leaving it unfixed. -- curated in README.md
+Check that it's really fixed. Don't assume that it was the wires and send that dirty fuel filter back onto the road. Check that it's really your fix that fixed it. "Wubba!" might not be the thing that did the trick. Know that it never just goes away by itself. Make it come back by using the original Make It Fail methods. If you have to ship it, ship it with a trap to catch it when it happens in the field. Fix the cause. Tear out the useless eight-track deck before you burn out another transformer. Fix the process. Don't settle for just cleaning up the oil. Fix the way you design machines. -- curated in README.md
+before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possiblity and brainstorm the cheapest tests that may narrow them down. - wassname -- curated in README.md
+Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to *pour* time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you've seen so far. Spend as much time as you need, even if it takes 30 minutes, or an hour. Reserve experiments for once you've fleshed out the hypothesis space as thoroughly as possible and know which pieces of evidence would allow you to best distinguish between the different possibilities.[^rahtz] -- curated in README.md
+If you are stuck, find a working reference implementation and compare it to yours. Relvent as the hyperparameters, model, data but especially subtle things like algorithm tweaks, and engineering tricks. If nothing jumps out, the fastest way might be to try a bisection search. Here you adapt their code wholesale and try the quickest test you can. If their code works then try again with half their features and so on. Eventuall you narrow down the features that are nessesary - wassname -- curated in README.md
+If you're doing anything that involves an RL algorithm as a component in a larger system, don't try and implement the RL algorithm yourself. [...] RL is unstable enough at the moment that you'll never be sure whether your system doesn't work because of a bug in your RL implementation or because of a bug in your larger system.[^rahtz] -- curated in README.md
+We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.[^henderson] -- curated in README.md
+When their RL implementation doesn't work, people are often keen to either (a) adjust their network architecture or (b) adjust their hyperparameters. On the other hand, they're reluctant to say they've got a bug. Most often, it turns out they've got a bug. Why bugs are so much more common in RL code is discussed above, but there's another advantage to assuming you've got a bug: bugs are a damn sight faster to find and fix than validating that your new architecture is an improvement over the old one.[^jones] -- curated in README.md
+What I'm advocating for here is not a blind faith in the buginess of your code, but for dramatically raising the threshold at which you start thinking 'OK, I think this is correct.'[^jones] -- curated in README.md
+"If one part is broken, the other parts can adapt and still achieve roughly acceptable performance" [^goodfellow], -- curated in README.md
+The default state of the world is that your research is false, because doing research is hard.[^nanda] -- curated in README.md
+Excitement is evidence of bullshit: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal![^nanda] -- curated in README.md
+When good programmers debug hard problems fast, it's usually because they understand the system well enough to *track the important internal state* in their head, letting them drastically *reduce the solution space they're searching over.*[^ulisse] -- curated in README.md
+figuring out a system's gears takes extra work up-front, but yields dividends forever. [...] The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system[^wentworth] -- curated in README.md
+broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task.[^spinningup] -- curated in README.md
+If you ever see a plot or a behaviour that just *seems weird*, chase right after it! Do not - do *not* - just 'hope it goes away'. Chasing anomalies is one of the most powerful ways to debug your system, because if you've noticed a problem without having had to go look for it, that means it's a *really big problem*. [...] It's really tempting to think that the cool extra functionality you were planning to write today [...] might just magically fix this anomalous behaviour. It won't. Give up on your plan for the day and chase the anomaly instead.[^jones] -- curated in README.md
+It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz] -- curated in README.md
+It seems important to really commit yourself to *always* investigate whenever you notice confusion.[^rahtz] -- curated in README.md
+you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' [...] or even 'read it upside down'. That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote.[^gwern-unseeing] -- curated in README.md
+Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.[^kidger] -- curated in README.md
+This is a systemic professional failing. [...] the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.[^kidger] -- curated in README.md
+When someone's RL implementation isn't working, they *luuuuuurv* to copy-paste a screenshot of their loss curve to you. They do this because they know they want a pretty, exponentially-decaying loss curve, and they know what they have *isn't that*. The problem with using the loss curve as an indicator of correctness is somewhat that it's not reliable, but mostly because it doesn't localise errors. The shape of your loss curve says very little about where in your code you've messed up, and so says very little about what you need to change to get things working.[^jones] -- curated in README.md
+The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. [...] The outliers especially almost always uncover some bugs in data quality or preprocessing.[^karpathy-recipe] -- curated in README.md
+Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort.[^ng-mly] -- curated in README.md
+It turns out that bad labels are a *huge* problem in many popular benchmark datasets.[^koaning] -- curated in README.md
+A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day.[^gwern] -- curated in README.md
+Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success.[^domingos] -- curated in README.md
+Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.)[^domingos] -- curated in README.md
+Overfit a tiny subset of data. Lastly and most importantly, before training on the full dataset try to train on a tiny portion (e.g. 20 examples) of your data and make sure you can achieve zero cost. For this experiment it's also best to set regularization to zero [...]. Unless you pass this sanity check with a small dataset it is not worth proceeding to the full dataset.[^cs231n] -- curated in README.md
+Overfit a single batch of only a few examples (e.g. as little as two). [...] If they do not, there is a bug somewhere and we cannot continue to the next stage.[^karpathy-recipe] -- curated in README.md
+most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)[^karpathy-mistakes] -- curated in README.md
+oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters[^karpathy-mistakes] -- curated in README.md
+6) thinking view() and permute() are the same thing (& incorrectly using view)[^karpathy-mistakes] -- curated in README.md
+Look, there's variance in supervised learning too, but it's rarely this bad. If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky.[^irpan] -- curated in README.md
+Instability to random seed is like a canary in a coal mine. If pure randomness is enough to lead to this much variance between runs, imagine how much an actual difference in the code could make.[^irpan] -- curated in README.md
+- If observations have unknown range, standardize - Compute running estimate of mean and standard deviation - x' = clip((x - mu)/sigma, -10, 10) - Rescale the rewards, but don't shift mean, as that affects agent's will to live - Standardize prediction targets (e.g., value functions) the same way -- curated in README.md
+Always Be Ablating - Different tricks may substitute - Especially whitening -- curated in README.md
+**Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. [...] No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak.[^sculley] -- curated in README.md
+Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation".[^tuning-playbook] -- curated in README.md
+The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).[^tuning-playbook] -- curated in README.md
+In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate.[^karpathy-recipe] -- curated in README.md
+We are nearing the point of wiping out a source of transformer training instability with one simple intervention.[^lucidrains] -- curated in README.md
+Do note that switching to the BOS dataloader changes the validation loss and makes all previous experiments not comparable in absolute value of the loss, because we have a lot fewer "confusing" tokens in the train/val batches. [...] Therefore, the loss appears lower but this is "fake" to some extent.[^nanochat] -- curated in README.md
+Original implementation clipped local gradients before sync. Since this codebase doesn't use DDP (gradient sync is in the optimizers), each rank was clipping based on its own local norm.[^nanochat] -- curated in README.md
+As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.[^bekman] -- curated in README.md
+In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes -- curated in README.md
+The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites.[^bekman-book] -- curated in README.md
+We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference.[^bekman-book] -- curated in README.md
+The best way to debug an error that arises in `trainer.train()` is to manually go through this whole pipeline to see where things went awry. The error is then often very easy to solve.[^hfcourse] -- curated in README.md
+Hyperparameter tuning is always emphasized as being the hardest part of machine learning, but it's just the last step to help you gain a little bit on the metric. [...] don't launch into a time-consuming and costly hyperparameter search until you have something that beats the baseline you have on your dataset.[^hfcourse] -- curated in README.md
+The most common cause of this error is using an **incorrect chat template**. It's essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. [...] It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses![^unsloth] -- curated in README.md
+All labels in your dataset are -100. Training losses will be all 0.[^unsloth] -- curated in README.md
+**Eliminate concurrency**: Restrict the number of processes to 1 for both training and data preprocessing[^axolotl] -- curated in README.md
+Axolotl caches certain steps and so does the underlying HuggingFace trainer. You may want to clear some of these caches when debugging.[^axolotl] -- curated in README.md
+4. Think your algorithm is working but you're actually seeing random noise. - Example: Graph of 7 tasks with 3 algorithms and looks like 1 algorithm might be doing best on all problems, but turns out they're all the same algorithm with DIFFERENT random seeds. -- curated in README.md
+Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research.[^nanda-mindsets] -- curated in README.md
+**The challenge lies in the fact that you can make these mistakes, train a model without it ever crashing, and still get a decent performance…**[^sanh] -- curated in README.md
+-   It is all well and good to make comparisons of validation error rates estimated on a finite validation set using fastidious statistical tests, but often the trial variance alone can produce statistically significant differences between two different trained models that use the same hyperparameter settings.[^tuning-playbook] -- curated in README.md
+**How reliable is my experiment?** Ask yourself: "How surprised would I be if it turned out to be complete bullshit due to a bug, error, noise, misunderstanding, etc.?" Investigate the most uncertain bits[^nanda-papers] -- curated in README.md
+Insufficient Skepticism: Missing simple alternative explanations, methodological flaws, or bugs. Explicitly list alternatives. Get others (especially mentors) to red team your plans before you run them. Actively try to break your hypothesis. Ask "What observation would make me abandon this?"[^nanda-taste] -- curated in README.md
+**Trying an experiment and seeing it fail gives little information by itself.** When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work".[^steinhardt] -- curated in README.md
+When ruling out ideas, it is important to hold oneself to a high standard. "This doesn't seem like it will work" or "I feel less motivated after trying a few things along this line that didn't work" are _not_ ruling out an idea.[^steinhardt] -- curated in README.md
+When a machine learning system performs poorly, it is usually difficult to tell whether the poor performance is intrinsic to the algorithm itself or whether there is a bug in the implementation of the algorithm. Machine learning systems are difficult to debug for various reasons.[^goodfellow] -- curated in README.md
+It ended up taking me 6 weeks to reproduce results, thanks to several software bugs. The question is, why did it take so long to find these bugs?[^irpan] -- curated in README.md
+**Result:** This was not an out-of-the-box win for nanochat even with a mild attempt over a few hours at a bit of tuning and debugging. The idea itself is intuitively appealing. Might come back around later to try harder later.[^nanochat] -- curated in README.md
+Our specific recommendations to researchers include: 1. Computing standard errors of the mean using the Central Limit Theorem 2. When questions are drawn in related groups, computing clustered standard errors 3. Reducing variance by resampling answers and by analyzing next-token probabilities 4. When two models are being compared, conducting statistical inference on the question-level paired differences, rather than the population-level summary statistics 5. Using power analysis to determine whether an eval (or a random subsample) is capable of testing a hypothesis of interest[^miller] -- curated in README.md
+If you keep that strategy when each run takes 10 hours, though, you can easily waste a *lot* of time. Last run didn’t work? OK, I think it’s this thing. Let’s set off another run to check. Coming back the next morning: still doesn’t work? OK, maybe it’s this other thing. Let’s set off another run. A week later, you still haven’t solved the problem.[^rahtz] -- curated in README.md
+than forming hypotheses. Why spend 15 minutes carefully considering everything that could be causing what you see when you can check the first idea that jumps to mind in a fraction of that (and gather more evidence in the process)? To put it another way: if you have rapid feedback, you can narrow down the hypothesis space a lot faster by trying things than thinking carefully.[^rahtz] -- curated in README.md
+The standard hypothesis testing framework can be misleading here, because it has an implicit frame of being able to list all the hypotheses. But actually, most of your probability mass should normally be on “something I haven’t thought of yet”[^nanda-mindsets] -- curated in README.md
+If trying to explain something mysterious, novice researchers often neglect simple, dumb hypotheses like “maybe MLP0 is incredibly important on *every* input, and there’s nothing special going on with my prompt”[^nanda] -- curated in README.md
+Importantly, it is often not obvious that multiple approaches to a problem all have the same issue. In the past, I have spent months trying different approaches to a problem before finally stepping back and realizing that they were all failing for the same reason. Moreover, I had all the data necessary to make this realization a couple weeks in but had failed to do so.[^steinhardt] -- curated in README.md
+* **Error goes up**: Commonly, this is due to a flip sign somewhere in the loss function/gradient. * **Error explodes**: This is usually a numerical issue but can also be caused by a high learning rate. * **Error oscillates**: You can lower the learning rate and inspect the data for shuffled labels or incorrect data augmentation. * **Error plateaus**: You can increase the learning rate and get rid of regulation. Then you can inspect the loss function and the data pipeline for correctness.[^fsdl] -- curated in README.md
+Actively Seek Alternatives: Explicitly brainstorm other ways your observations could be explained. What are the simplest explanations? What known circuits or phenomena could be involved? What would a strong skeptic argue?[^nanda-taste] -- curated in README.md
+**If it doesn’t work, assume there’s a bug.** Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it’s a bug. Bad hyperparameters can significantly degrade RL performance, but if you’re using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue.[^spinningup] -- curated in README.md
+For example, perhaps you forgot to flip your labels when you left-right flipped the image during data augmentation. Your net can still (shockingly) work pretty well because your network can internally learn to detect flipped images and then it left-right flips its predictions. Or maybe your autoregressive model accidentally takes the thing it’s trying to predict as an input due to an off-by-one bug. Or you tried to clip your gradients but instead clipped the loss, causing the outlier examples to be ignored during training. Or you initialized your weights from a pretrained checkpoint but didn’t use the original mean. Or you just screwed up the settings for regularization strengths, learning rate, its decay rate, model size, etc.[^karpathy-recipe] -- curated in README.md
+Most importantly, there is no point of launching 1000 runs with different hyperparameters (or architecture tweaks like activation functions): **compare a couple of runs with different hyperparameters to get an idea of which hyperparameters have the highest impact** but in general, it is delusional to expect to get your biggest jumps of performance by simply tuning a few values. For instance, if your best performing model is trained with a learning rate of 4e2, there is probably something more fundamental happening inside your neural network and you want to identify and understand this behavior so that you can re-use this knowledge outside of your current specific context.[^sanh] -- curated in README.md
+Once the algorithm was partially working, they would attain higher performance by looking for remaining bugs, both by reviewing the code carefully, and by collecting metrics such as average policy entropy to perform sanity-checks, rather than just tune hyperparameters.[^olsson] -- curated in README.md
+Third, and perhaps most important for building skill,[[1]](https://www.lesswrong.com/posts/LTypqBMTSmRrrhb2v/how-to-get-good-at-programming#fn289bs9hi65b)you must **notice** when you're going into brute-force search mode, and then **take action** by investing time in understanding the underlying system, until both the problem and solution make sense.[^ulisse] -- curated in README.md
+Things I've tried (but maybe not systematically enough):  * Different initial LRs * Different optimizers * Different number of hidden layers/units * Shared pi/V NN body (with diff output layers) vs not * Changing amount of entropy * Adding correlated noise * Using TD residual instead of MC version * Clipping the gradient * Different gamma values -- curated in README.md
+Visualize the model in action: When training a model to detect objects in images, view some images with the detections proposed by the model displayed superimposed on the image. When training a generative model of speech, listen to some of the speech samples it produces. This may seem obvious, but it is easy to fall into the practice of looking only at quantitative performance measurements like accuracy or log-likelihood. Directly observing the machine learning model performing its task will help to determine whether the quantitative performance numbers it achieves seem reasonable. Evaluation bugs can be some of the most devastating bugs because they can mislead you into believing your system is performing well when it is not.[^goodfellow] -- curated in README.md
+By reaching a local optimum, learning curves can indicate successful optimization of the policy over time, when in reality the returns achieved are not qualitatively representative of learning the desired behaviour, as demon-strated in video replays of the learned policy 5. Therefore, it is important to show not only returns but demonstrations of the learned policy in action.[^henderson] -- curated in README.md
+2. Make sure observations usable: - See if YOU could control the system by using the same observations you give the agent. - Example: Look at preprocessed images yourself to make sure you don't remove necessary details or hinder the algorithm in a certain way. -- curated in README.md
+Pro-tip: when you work with language, have a serious **look at the outputs of the tokenizers**. I can’t count the number of lost hours I spent trying to reproduce results (and sometimes my own old results) because something went wrong with the tokenization.[^sanh] -- curated in README.md
+Error analysis can often help you figure out how promising different directions are. I’ve seen many engineers reluctant to carry out error analysis. It often feels more exciting to just jump in and implement some idea, rather than question if the idea is worth the time investment. This is a common mistake: It might result in your team spending a month only to realize afterward that it resulted in little benefit.[^ng-mly] -- curated in README.md
+⚠️ If you are doing distributed training, print samples of your dataset in each process and triple-check that you get the same thing. One common bug is to have some source of randomness in the data creation that makes each process have a different version of the dataset.[^hfcourse] -- curated in README.md
+-   Although in many cases the primary objective of our experiments only requires considering the validation error of each trial, we must be careful when reducing each trial to a single number because it can hide important details about what’s going on below the surface. -   For every study, we always look at the **training curves** (training error and validation error plotted versus training step over the duration of training) of at least the best few trials.[^tuning-playbook] -- curated in README.md
+(I missed a multithreading bug for several months by ignoring a small but mysterious decay in frames per second.)[^rahtz] -- curated in README.md
+There was no real spike in the two earlier runs. The loss never went up in the first place. In both resumes it was under-reporting loss due to an exactly repeated data and then it reached data it hasn't seen before and started reporting correctly. In other words it was overfitting and reporting a false loss.[^bekman-book] -- curated in README.md
+**Do ablations on your fancy method**: It's easy for people to have a fancy method with lots of moving parts, when many actually are unnecessary. You should always try removing one part and see if the method breaks. Do this for each part. *   For example, the [original unlearning method](https://arxiv.org/abs/2403.03218v1) in the [RMU paper](https://arxiv.org/abs/2403.03218) claimed it was based on finding a meaningful steering vector, until follow-up work found that it was just about adding a vector with really high norm that broke the model, and a random vector performed just as well.[^nanda] -- curated in README.md
+The only way to find out what needs work is to implement something quickly, -- curated in README.md
+and find out what parts break.[^cs229] -- curated in README.md
+Figure 15.5: An autoencoder trained with mean squared error for a robotics task has failed to reconstruct a ping pong ball. The existence of the ping pong ball and all its spatial coordinates are important underlying causal factors that generate the image and are relevant to the robotics task. Unfortunately, the autoencoder has limited capacity, and the training with mean squared error did not identify the ping pong ball as being salient enough to encode.[^goodfellow-ch15] -- curated in README.md
+One of the key drivers of progress in mech interp is an openness to qualitative research: summary statistics lose a ton of information. What can we learn by actually looking deeply into what's happening?[^nanda] -- curated in README.md
+1. **Test reward function standalone**: Run it outside training with known inputs to verify it returns nonzero values.[^axolotl-stability] -- curated in README.md
+In most cases, we do not know a priori what the intended behavior of the algorithm is. In fact, the entire point of using machine learning is that it will discover useful behavior that we were not able to specify ourselves. If we train a neural network on a new classification task and it achieves 5 percent test error, we have no straightforward way of knowing if this is the expected behavior or suboptimal behavior.[^goodfellow] -- curated in README.md
+A valuable intuition to have in mind is that, by default, all numbers are meaningless because we lack any scale to compare them. E.g. if a probe gets 95% classification accuracy on some task, is this good? Is this bad? Hard to say without knowing more! Baselines are one way to get context to compare against.[^nanda-draft] -- curated in README.md
+You might be temped to keep track of the difference \(\mid f’\_a - f’\_n \mid \) or its square and define the gradient check as failed if that difference is above a threshold. However, this is problematic. For example, consider the case where their difference is 1e-4. This seems like a very appropriate difference if the two gradients are about 1.0, so we’d consider the two gradients to match. But if the gradients were both on order of 1e-5 or lower, then we’d consider 1e-4 to be a huge difference and likely a failure.[^cs231n] -- curated in README.md
+*   How would a random predictor perform (especially in classification problems)? Dataset can be unbalanced… *   What would the loss look like for a random predictor? *   What is (are) the best metric(s) to measure progress on my task? *   What are the limits of this metric? If it’s perfect, what can I conclude? What can’t I conclude?[^sanh] -- curated in README.md
+If the loss/metric you get on your initial model is very different from the loss/metric you would expect for random predictions, double-check the way your loss or metric is computed, as there is probably a bug there. If you are using several losses that you add at the end, make sure they are of the same scale.[^hfcourse] -- curated in README.md
+5.   **Rule of thumb: 400 episodic return in breakout**: Check if your PPO could obtain 400 episodic return in breakout. We have found this to be a practical rule of thumb to determine the fidelity of online PPO implementations in GitHub. Often we found PPO repositories not able to do this, and we know they probably do not match all implementation details of `openai/baselines`’ PPO.[^ppo37] -- curated in README.md
+The issue here isn't just that we might have bad labels in our training set, the issue is that it appears in the validation set. If a machine learning model can become state of the art by squeezing another 0.5% out of a validation set one has to wonder. Are we really making a better model? Or are we creating a model that is better able to overfit on the bad labels?[^koaning] -- curated in README.md
+broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task. -- Achiam -- curated in SKILL.md
+If one part is broken, the other parts can adapt and still achieve roughly acceptable performance -- Goodfellow, Bengio and Courville -- curated in SKILL.md
+Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem -- Godbole, Dahl, Gilmer, Shallue and Nado -- curated in SKILL.md
+Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research. -- Nanda -- curated in SKILL.md
+Read your data. Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad. -- Nanda -- curated in SKILL.md
+How would a random predictor perform (especially in classification problems)? [...] What would the loss look like for a random predictor? [...] What are the limits of this metric? If it's perfect, what can I conclude? What can't I conclude? -- Sanh -- curated in SKILL.md
+**NEVER STOP**: Once the experiment loop has begun (after the initial setup), do NOT pause to ask the human if you should continue. Do NOT ask 'should I keep going?' or 'is this a good stopping point?'. The human might be asleep, or gone from a computer and expects you to continue working *indefinitely* until you are manually stopped. You are autonomous. If you run out of ideas, think harder — read papers referenced in the code, re-read the in-scope files for new angles, try combining previous near-misses, try more radical architectural changes. The loop runs until the human interrupts you, period. -- Karpathy, [autoresearch/program.md](https://github.com/karpathy/autoresearch/blob/master/program.md) -- curated in SKILL.md
+Build it up as you go, don't think you can build it ahead of time. Be focused on a strong mental model of what options you have (including architectural changes and losses) that you think should affect what metrics in the logs. -- wassname -- curated in SKILL.md
+Before acting plan by writing multiple competing hypotheses: consider the most likely failure but also some of: a subtle failure, a perverse failure, a possible bug, and an unknown. Put a rough credence on each. Finally write down what you expect to see differently for success vs each possibility and brainstorm the cheapest tests that may narrow them down. -- wassname -- curated in SKILL.md
+If you are stuck, find a working reference implementation and compare it to yours. If nothing jumps out, try a bisection search: adapt their code wholesale, then half their features, and so on. -- wassname -- curated in SKILL.md
+Summarise your concept and pseudocode and do an external review in scientist mode. Perhaps describe the forward and backward pass as mermaid too. -- wassname -- curated in SKILL.md
+The CNN has learned to detect a metal token that radiology technicians place on the patient in the corner of the image field of view at the time they capture the image. -- Zech et al. -- curated in SKILL.md
+Apparently meaningless identifier columns were the most important predictors. [...] the university only filled out much of this information *after* a grant application was accepted. -- Howard and Gugger -- curated in SKILL.md
+by default, all numbers are meaningless because we lack any scale to compare them. E.g. if a probe gets 95% classification accuracy on some task, is this good? Is this bad? Hard to say without knowing more! -- Nanda -- curated in SKILL.md
+If my supervised learning code failed to beat random chance 30% of the time, I'd have super high confidence there was a bug in data loading or training. If my reinforcement learning code does no better than random, I have no idea if it's a bug, if my hyperparameters are bad, or if I simply got unlucky. -- Irpan -- curated in SKILL.md
+It ended up taking me 6 weeks to reproduce results, thanks to several software bugs. The question is, why did it take so long to find these bugs? -- Rahtz -- curated in SKILL.md
+Don't be tempted to write an adaptive reward scaling scheme. It's extra nonstationarity. Just hand-scale. -- Andy Jones -- curated in rl/SKILL.md
+If you're new to RL, writing things from scratch is the most catastrophically self-sabotaging thing you can do. -- Andy Jones -- curated in rl/SKILL.md
+Rathore et al. 2024: "the estimate of the κ grows polynomially with nres" -- but this is in raw units. Nondimensionalization reduces the effective condition number by making all PDE coefficients O(1). -- curated in pinn/SKILL.md
+Wang et al. propose a modified MLP with multiplicative interactions. With `U = φ(XW1 + b1)`, `V = φ(XW2 + b2)` two nonlinear encodings of the input (φ = tanh) and a per-layer gate `Z(k) = φ(H(k)Wz,k + bz,k)` computed from the hidden state, the update is `H(k+1) = (1 - Z(k)) * U + Z(k) * V`. Authors claim a ~3x decrease in the leading Hessian eigenvalue. -- curated in pinn/SKILL.md
+Factorize each neuron's weight vector as w = s * w_unit, where s is a trainable scalar and w_unit is the unit-normalized direction. This changes the optimization geometry so the loss surface has better-conditioned local minima. "Predictions obtained by RWF are in excellent agreement with ground truth, while other weight parameterizations result in poor or non-physical approximations." -- curated in pinn/SKILL.md
+Used in the PirateNet architecture alongside causal training, sequence-to-sequence, and Fourier features. Simple to implement as a custom parameterization on Linear layers. -- curated in pinn/SKILL.md
+Instead of data-augmenting with transformed copies, bake symmetries directly into the architecture so every model in the function space is automatically invariant/equivariant. For turbulence closure (Reynolds stress from velocity gradients), custom tensor layers enforce Galilean invariance by construction. "The Galilean invariant model is more accurate than the other models" and generalizes better across flow configurations. -- curated in pinn/SKILL.md
+Lecture: Brunton, S. "AI/ML+Physics Part 3 - Designing an Architecture." https://www.youtube.com/watch?v=fiX8c-4K0-Q Key distinction: invariance (output unchanged by transformation, e.g., energy is frame-invariant) vs equivariance (output transforms same way as input, e.g., stress tensor rotates with frame). Equivariant architectures are more general. If your PDE has known symmetries (translation, rotation, scaling), enforce them architecturally rather than hoping the optimizer discovers them. **Caveat**: This works best for local closure terms (Reynolds stress, turbulence models) and unbounded/periodic domains where the global symmetry holds everywhere. If your domain has boundary conditions that break the symmetry (e.g., a wall breaks rotational invariance), enforcing the symmetry globally in the architecture will prevent the solution from satisfying the BCs -- the architecture will be fighting the problem. In bounded domains, use symmetry-enforcing architectures only for terms where the symmetry genuinely holds (e.g., the constitutive relation), not for the full solution field. Libraries like `e3nn` implement this but add significant computational overhead. -- curated in pinn/SKILL.md
+Rathore et al. 2024 (ICML, credence ~80%): "Adam+L-BFGS attains 14.2x smaller L2RE than Adam on convection and 6.07x smaller than L-BFGS on wave." Tested on 3 PDEs (convection, reaction, wave), 5 seeds, widths 50-400. -- curated in pinn/SKILL.md
+"on the convection PDE, a loss of 10^-3 yields an L2RE around 10^-1, but decreasing the loss by a factor of 100 to 10^-5 yields an L2RE around 10^-2, a 10x improvement." -- curated in pinn/SKILL.md
+"L-BFGS stops in these cases without reaching a critical point: the gradient norm is around 10^-2 or 10^-3. The gradient still contains useful information for improving the loss." -- curated in pinn/SKILL.md
+Cause: strong Wolfe line search fails, step size goes to zero. Fix: switch to NNCG (Armijo only) or restart with different LR. -- curated in pinn/SKILL.md
+Theorem 8.4 (Section 8.2): condition number = Omega(nres^alpha) with alpha > 1/2, given eigenvalues of A o K_inf decaying as O(j^-2alpha). nres typically ranges 1e3 to 1e4. Separately, measured condition numbers near a solution are often > 1e4 (Section 6.2, Figure 3). -- curated in pinn/SKILL.md
+L2 norm (MSE) on residuals: default; promotes smooth, low-frequency solutions. L1 norm (MAE) on residuals: more robust to outlier collocation errors and sharp gradients (shocks) since it doesn't square-penalize large pointwise residuals. This is distinct from L1 *regularization on equation coefficients*, which is what SINDy and sparse equation discovery use to promote parsimony (few active terms). Don't conflate the two: L1 residual = robust fitting; L1 coefficient regularization = sparse model selection. For standard PINNs with a known PDE, L2 is correct. L1 residual loss is worth trying if you have shocks or suspect outlier collocation points. -- curated in pinn/SKILL.md
+Wang et al. 2021 (credence ~80%): "the gradients corresponding to the boundary loss term Lub(θ) in each layer are sharply concentrated around zero and overall attain significantly smaller values than the gradients corresponding to the PDE residual loss Lr(θ)." Shown via per-layer histograms of back-propagated gradients; the paper does not quantify the gap in orders of magnitude. -- curated in pinn/SKILL.md
+Wang et al. 2021: "many eigenvalues of the residual-loss Hessian are extremely large up to 1e5" while the boundary-loss Hessian eigenvalues stay small, so the gradient-flow stiffness is dominated by the residual term. This is an absolute magnitude, not a condition number; Wang never reports one. -- curated in pinn/SKILL.md
+For a condition number, use Rathore Figure 3: outlier eigenvalues > 1e4 (convection), > 1e3 (reaction), > 1e5 (wave). -- curated in pinn/SKILL.md
+Adaptively weight each loss term inversely proportional to its gradient magnitude. EMA of gradient statistics for stability. -- curated in pinn/SKILL.md
+NeuralPDE.jl implements this as `GradientScaleAdaptiveLoss`. -- curated in pinn/SKILL.md
+Instead of summing loss gradients (which can cancel), project them into a conflict-free direction. ConFIG: unit-normalize per-loss gradients, solve least-squares for combined direction, rescale by projection lengths. -- curated in pinn/SKILL.md
+Key: must compute per-loss gradients separately (zero_grad + backward for each). Summing raw losses defeats the purpose. M-ConFIG: momentum variant, updates only one loss's gradient per step. Use with SGD, not Adam (momentum conflict). -- curated in pinn/SKILL.md
+Standard PINNs use penalized (soft) constraints: add physics as a loss term. The alternative is constrained optimization: minimize data error while exactly satisfying the physics constraints. "With a loss function you're not exactly satisfying your constraints. With constrained optimization you are." -- curated in pinn/SKILL.md
+Physics-informed DMD (Baddoo et al. 2021) is the cleanest example: restrict the DMD matrix to a symmetry-preserving manifold (Hermitian, symplectic, etc.) via the Procrustes problem. KKT closed-form solutions exist because DMD is linear in its parameters -- the constraint is linear in both the output and the parameters simultaneously. Baddoo et al. 2021. "Physics-informed dynamic mode decomposition." Proc. R. Soc. A. https://arxiv.org/pdf/2112.04307 **Critical caveat for PINNs**: A BC like u(0)=0 is affine in the output u, but it is nonlinear in the NN weights theta. Closed-form KKT does NOT apply to neural network parameters. For NN-based PINNs, the two options for hard constraints are: (a) architectural -- multiply output by a distance function that satisfies the BC (Section 4 item 8), or (b) Augmented Lagrangian Methods (ALM), which are iterative and substantially more complex than Adam. Constrained optimization is most practical for linear models (DMD, SINDy, linear state-space) where the parameters enter linearly. -- curated in pinn/SKILL.md
+When the PINN fails on hard PDE regimes (high convection coefficient, strong reaction), don't start there. Start with easy parameters (small coefficient), train to convergence, then warm-start and increase to the target regime. 1-2 orders of magnitude improvement over naive training. "The curriculum training approach achieves significantly better errors, as well as lower variance in the error." (From Figure E.2 showing 10 seeds) -- curated in pinn/SKILL.md
+For time-dependent PDEs: train on a short time window, predict next state, step forward. Don't train on full space-time at once. "Posing the problem as seq2seq learning results in significantly lower error. The difference is particularly striking for reaction and reaction-diffusion cases, where seq2seq decreases error by almost two orders of magnitude." -- curated in pinn/SKILL.md
+NeuralPDE.jl calls this time-marching; see `WeightedIntervalTraining`. Note: these failures are not due to limited NN expressivity -- the architecture has enough capacity. The problem is optimization difficulty from the soft PDE constraint. -- curated in pinn/SKILL.md
+Standard PINNs trained by gradient descent are implicitly biased toward minimizing residuals at *later* times before even fitting the initial conditions -- violating physical causality. The NTK analysis shows the residual at time t is influenced more by residuals at later t' > t than earlier ones. This makes PINNs fail on chaotic/turbulent systems. Fix: weight each temporal residual point by wi = exp(-epsilon * sum_j<i R_j(theta)), where R_j is the accumulated residual before time i. This forces earlier times to converge first before the loss "turns on" at later times. "10-100x improvements in accuracy compared to competing approaches. First time PINNs succeeded on chaotic Lorenz, Kuramoto-Sivashinsky, and 2D Navier-Stokes in turbulent regime." -- curated in pinn/SKILL.md
+Key difference from seq2seq/curriculum: causal weighting works within a single continuous training, without requiring separate time windows or changing the PDE coefficients. Can be combined with seq2seq for further gains. Sensitivity: epsilon controls the steepness of the causal weights. Too small = residuals at later times turn on too early. Too large = training stalls on early time steps. Anneal epsilon during training. -- curated in pinn/SKILL.md
+Instead of penalizing BC violations (soft), multiply the PINN output by a distance function phi(x) that is zero on the boundary. Then u(x) = phi(x) * NN(x) satisfies BCs exactly by construction. "We eliminate modeling error associated with the satisfaction of boundary conditions. The sole contribution to the loss function is from the residual error at interior collocation points." "The proposed approach consistently outperforms a standard PINN-based collocation method." -- curated in pinn/SKILL.md
+Domain-specific failure modes and hard BC examples: see [references/heat_exchanger.md](references/heat_exchanger.md). -- curated in pinn/SKILL.md
+Domain-specific: differentiable EoS wrapping (REFPROP/PCHIP), IC handling for plant data, multi-episode training. See [references/heat_exchanger.md](references/heat_exchanger.md). -- curated in pinn/SKILL.md
+ConFIG authors report improvements over PCGrad and an Adam baseline on Burgers, Schrodinger, Kovasznay, and Beltrami. This is author-reported evidence, not a general comparison with UPGrad. -- curated in pinn/SKILL.md
+Jagtap et al. 2020. "Extended physics-informed neural networks (XPINNs): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations." Commun. Comput. Phys. https://arxiv.org/pdf/2005.11025 Credence ~70%: Multiple citations, implemented in DeepXDE. Enables parallelization; each subdomain network is smaller and easier to optimize. Key: interface residuals must be added as additional loss terms. Continuity of u and its normal derivative across interfaces. Useful when the solution has different character in different regions (e.g., different phases, boundary layers). -- curated in pinn/SKILL.md
+Toscano et al. 2024: PIKANs "lead to smaller models and may also contribute to lowering computational cost while maintaining good accuracy." -- curated in pinn/SKILL.md
+Credence ~40%: New, no independent replication. Other authors focus on improving PINNs within the MLP framework, not validating PIKANs as an alternative. Interesting but unproven. -- curated in pinn/SKILL.md
+the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. [...] e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator.[^wang] -- curated in references/llm_judges.md
+Did it time out, or was it reasoning behind a buffer for longer than you think? Check the event timestamps and transport state. -- wassname -- curated in references/llm_judges.md
+Note from wassname's LLM to other LLMs: when you SHOW the human any of this (a judge trace, a compared A/B pair, a sample, a prompt design), reproduce it in FULL and verbatim, prompts, separators, special tokens, and the reasoning/thinking block included. Your trained instinct is to digest and trim; resist it, that instinct destroys exactly the boilerplate (system prompt, special tokens, separators) that the human needs to debug. Formatting may only HIGHLIGHT or REORGANISE: bold or underline the diff, split into a table, use sections, but never drop, elide with "...", or paraphrase. When comparing variants, show ALL of them this way, not a representative one. Link the source file so they can open the raw (`results/runs/.../x.jsonl`, ideally `path:line`). "Show me" means reproduce, not describe. Trim only when explicitly asked. -- CLAUDE -- curated in references/llm_judges.md
+"If there are NaNs, we should not drop them, else we end up comparing different sample sets and it's invalid. A might be a single easy sample, and B might be all 128 hard samples. Of course A looks much better, but actually it failed on the vast majority of samples." - wassname, lightly edited for spelling -- curated in references/llm_judges.md
+Across the 36-model result set, the model-average first-shown pick rate is 64.3%, with a median of 65.4%. **The model-average absolute first-position lift is 15.7 percentage points.** So the aggregate pattern is not a subtle tie-breaker: the displayed order materially changes many judgments. -- curated in references/llm_judge_litreview.md
+The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. **While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions.** Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications. -- curated in references/llm_judge_litreview.md
+We find evidence of position bias, which is especially prevalent in smaller LLM labelers (see Appendix B). **To mitigate the effect of position bias, two inferences are made for every pair of candidates, where the order in which candidates are presented to the LLM is reversed for the second inference.** The results from both inferences are then averaged to obtain the final preference distribution. -- curated in references/llm_judge_litreview.md
+As observed in the figure, models larger than 7B exhibit significantly less self-preference bias compared to those of 7B or smaller. **For example, the DBG score of Qwen2.5-0.5B-Instruct is 41.7%. In contrast, the DBG score of Qwen2.5-14B-Instruct is only 2.1%.** This suggests that LLM judging tasks should utilize larger models to obtain more accurate and unbiased judgment results. -- curated in references/llm_judge_litreview.md
+Empirical results demonstrate that JudgeLRM not only surpasses proprietary models like GPT-4 and DeepSeek-R1 but also outperforms SFT and RL baselines of comparable sizes, **with an average improvement of 8.14% in F1 score over SFT counterparts.** -- curated in references/llm_judge_litreview.md
+We observe an initial increase (similar to (Muennighoff et al., 2025; Aggarwal & Welleck, 2025)) in accuracy as the average thinking budget increases. **For example, in Figure 2(a), accuracy increases from 82.2% to 87.3% as the average number of thinking tokens increases from 385 to 1100.** However, this trend does not continue indefinitely. -- curated in references/llm_judge_litreview.md
+# Some env for reasoning effort if you using litellm https://github.com/BerriAI/litellm/blob/main/litellm/constants.py#L81 DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET=24576 DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET=8192 DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET=1024 -- curated in references/llm_judge_litreview.md
+**Results confirm that accuracy gains plateau early and, in some configurations, decline at high sample counts** — a pattern inconsistent with diminishing returns alone and more consistent with noise introduction on problems that were already solved. This suggests self-consistency should be reserved for genuinely difficult problems rather than applied as a default scaling strategy. -- curated in references/llm_judge_litreview.md
+On MATH-500, Flash-Lite accuracy improved through approximately 10 sampled paths before plateauing and then declining slightly beyond 15, as shown in Figure 2. **This decline is notable: it suggests that once a model reliably solves most problems, additional samples introduce occasional wrong reasoning paths that the aggregator cannot fully suppress.** -- curated in references/llm_judge_litreview.md
+While they perform well in short contexts (<1K), performance degrades significantly as context length increases. **At 32K, for instance, 11 models drop below 50% of their strong short-length baselines.** Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. -- curated in references/llm_judge_litreview.md
+We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. **In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.** Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models. -- curated in references/llm_judge_litreview.md