wassname/ml_debug
1
0
Fork 0
mirror of https://github.com/wassname/ml_debug.git synced 2026-06-27 01:00:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add research taste evidence appendix

Browse Source
This commit is contained in:
wassname
2026-06-25 09:33:34 +08:00
parent 3f3f95a3b4
commit 8fc2c0bbd0
10 changed files with 647 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evidence/nanda_highly_opinionated_ml_paper_writing.md
+67
View File
@@ -0,0 +1,67 @@
# Highly Opinionated Advice on How to Write ML Papers - Neel Nanda (2025-05-12)
Source: https://www.lesswrong.com/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers
Author: Neel Nanda
Date: 12th May 2025
Fetch-status: excerpted from LessWrong HTML via browser.
Use: distillation and paper-writing evidence. This is adjacent to the research-process sequence, and directly useful when turning messy findings into a public artifact.
## Why this matters for agents
This post is the operational version of the distillation stage: compress the research into a few claims, red-team the evidence, write to inform rather than persuade, and spend disproportionate care on the abstract, intro, figures, and limitations.
## Quotes
> The essence of an ideal paper is the narrative: a short, rigorous and evidence-based technical story you tell, with a takeaway the readers care about.
> The first step is to compress your research into these claims.
> Experimental Evidence: This is absolutely crucial to get right and aggressively red-team, it’s how you resist the temptation of elegant but false narratives.
> Inform, not persuade: Avoid the trap of overclaiming or ignoring limitations.
> Your research only matters if people read, understand, and build upon it.
> At its core, a paper should present a narrative of one to three specific concrete claims that you believe to be true, that build to some useful takeaway(s).
> Readers will rarely take away more than a few sentences of content. Choose those sentences carefully.
> Generally, stronger statements make for more interesting papers, but require higher standards of evidence - resist the temptation to overclaim for clicks!
> Warning: Before moving into paper-writing mode, it's crucial to verify that your evidence is actually correct.
> Novelty means it expands our knowledge.
> Rigorous, at-scale replications of shaky results, negative results of seemingly promising hypotheses, and high-quality failed replications of popular papers are all very valuable contributions.
> A particularly important thing to get right is extensive red-teaming: you should spend a good amount of your time, both during the original research and now, red teaming your narrative.
> Good experiments distinguish between hypotheses.
> This skepticism and sanity checking is especially key for particularly surprising or novel bits of evidence.
> Ablation studies: When a paper introduces a complex new method, there are often several moving parts.
> Track pre/post-hoc analysis.
> Quality Over Quantity: Try to prioritise having at least one really compelling and hard to deny experiment, over a bunch of mediocre ones.
> Baselines are Crucial.
> The subtlety of baselines: It's not enough to just have them; you must strive to have the strongest possible baselines.
> The Guiding Question for Evidence: Ultimately, the question to ask about your evidence is: "Should this update a reader's beliefs about my claims?"
> Reproducibility & Publishing code: Rigour can be in the eye of the beholder: if readers cannot understand or verify it for themselves, it’s far harder to consider it rigorous.
> A key challenge in paper writing is the illusion of transparency - you have spent months steeped in the context of this research project.
## Source graph
Links visible in this post worth follow-up:
- Research process sequence: https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL
- Jakob Foerster writing advice: https://www.jakobfoerster.com/
- Jacob Steinhardt writing/research advice: https://cs.stanford.edu/~jsteinhardt/
- Refusal is mediated by a single direction: https://arxiv.org/abs/2406.11717
- Nanda grokking work: https://arxiv.org/abs/2301.05217
- Paper writing checklist: Google Docs link visible in post, not cached.
docs/evidence/nanda_research_process_explore_understand_distill.md
+51
View File
@@ -0,0 +1,51 @@
# How I Think About My Research Process: Explore, Understand, Distill - Neel Nanda (2025-04-26)
Source: https://www.lesswrong.com/posts/hjMy4ZxS5ogA9cTYK/how-i-think-about-my-research-process-explore-understand
Mirror/sequence URL visible on page: https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/hjMy4ZxS5ogA9cTYK
Author: Neel Nanda
Date: 26th Apr 2025
Fetch-status: excerpted from LessWrong HTML via browser plus cross-checked against local shared draft.
Use: research-process / research-taste evidence, especially for agents deciding what mode of work they are in.
## Why this matters for agents
Nanda frames empirical research as stage-dependent. A model should not demand a crisp hypothesis when the right stage is exploration; it should not accept weak, cherry-picked evidence when the task has moved into understanding or distillation.
## Quotes
> This guide focuses more on the strategic (high-level direction, when to give up or pivot, etc) and tactical (what to do next, how to prioritise, etc) aspects of research - the "how to think about it" rather than just the "how to do it." Some of skills (coding, reading papers, understanding ML/mech interp concepts) are vital for how to do it, but not in scope here.
> How to get started? Strategic and tactical thinking are hard skills, and it is rare to be any good at them when starting out at research (or ever tbh). The best way to learn them is by trying things, making predictions, seeing what you get right or wrong (i.e., getting feedback from reality), and iterating.
> I see research as breaking down into a few stages:
>
> 1. Ideation - Choose a problem/domain to focus on
> 2. Exploration - Gain Surface area
> 1. North star: Gain information
> 3. Understanding - Test Hypotheses
> 1. North star: Convince yourself of a key hypothesis
> 4. Distillation - Compress, Refine, Communicate
> 1. North star: Compress your research findings into concise, rigorous truth that you can communicate to the world
> At the start, your understanding of the problem is often vague. Naively, it’s easy to think of research as being about testing specific hypotheses, but in practice you often start out not even knowing the right questions to ask, or the most promising directions. The exploration stage is about moving past this.
> Not having a clear goal/next step doesn’t mean that you don’t need to prioritise! Prioritise for information gain.
> Frequently ask yourself “am I getting enough information per unit time?” If you haven’t learned anything recently, shake it up.
> The mark of a good researcher is a deep commitment to skepticism of your results.
> A great experiment elegantly, and conclusively distinguishes between several plausible hypotheses, validates non-trivial predictions made by one hypothesis, and is tractable to implement in practice.
> The north star here is to distill your research findings into concise, rigorous truth that you can communicate to the world.
> Write to inform, not persuade - if you are clear (a high bar), and your results are interesting, people will likely appreciate your work.
## Source graph
High-value links inside or adjacent to this post:
- ARENA curriculum: https://arena-chapter1-transformer-interp.streamlit.app/
- Nanda paper reading list: https://www.alignmentforum.org/posts/NfFST5Mio7BCAQHPA/an-extremely-opinionated-annotated-list-of-my-favourite
- Chris Olah, research taste: https://colah.github.io/notes/taste/
- Nanda Othello research process write-up: https://www.alignmentforum.org/s/nhGNHyJHbrofpPbRG/p/TAz44Lb9n9yf52pv8
- Nanda standards post: https://www.neelnanda.io/blog/35-standards
docs/evidence/nanda_research_process_key_mindsets.md
+49
View File
@@ -0,0 +1,49 @@
# My Research Process: Key Mindsets - Truth-Seeking, Prioritisation, Moving Fast - Neel Nanda (2025-04-27)
Source: https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/cbBwwm4jW6AZctymL
Author: Neel Nanda
Date: 27th Apr 2025
Fetch-status: excerpted from LessWrong HTML via browser plus cross-checked against local shared draft.
Use: research-process evidence for truth-seeking, prioritization, speed, and action under uncertainty.
## Why this matters for agents
This is the most directly agent-steering post in the sequence. It says the research process needs active skepticism, explicit prioritization, fast feedback loops, and the ability to act under uncertainty without waiting for a perfect next step.
## Quotes
> I think the most important mindsets are:
> * Truth-seeking: By default, many research insights will be false - finding truth is hard. It’s not enough to just know this, you must put in active effort to be skeptical and resist bias, lest you risk your research being worthless.
> * Prioritisation: You have finite time, and a lot of possible actions. Your project will live or die according to whether you pick good ones.
> * Moving fast: You have finite time and a lot to do. This doesn’t just mean “push yourself to go faster” - there’s a lot of ways to eliminate inefficiency without sacrificing quality.
> Insufficient skepticism doesn't feel like insufficient skepticism from the inside. It just feels like doing research.
> This means that you must be putting in constant active effort into ensuring your results are robust. This must be integrated into part of your research process - if you’re not, then there’s a good chance your results are BS.
> The standard hypothesis testing framework can be misleading here, because it has an implicit frame of being able to list all the hypotheses. But actually, most of your probability mass should normally be on “something I haven’t thought of yet”.
> Here the Bayesian frame is often helpful. It’s generally overkill to put explicit numbers on everything, but it reminds me to ask the question “was this observation more likely under hypothesis A or B”, not just whether it was predicted by my favourite hypothesis.
> Fundamentally, good prioritisation is about having a clear goal (north star) in mind.
> The first step is just making time to stop and ask yourself “do I endorse what I’m doing, and could I be doing something better?”
> Prioritising and executing are different mental modes and should not be done simultaneously. Keep them separate, and make time to regularly reflect, and time to lock-in and execute on a plan without stressing about if it’s the best plan.
> Tight feedback loops are crucial: A key thing to track when doing research is your feedback loops.
> A corollary of this is that you should (often) do fast experiments first. It is far better to do a quick and dirty experiment to get some preliminary signs of life than an extremely long and expensive experiment that will produce conclusive data but only after weeks of work.
> Fail fast. One of the largest time sinks possible is investing weeks to months of effort into a failed research direction. Thus, a key question to ask yourself is: if this direction is doomed, how could I discover this as fast as humanly possible?
> A crucial mindset is being able to do something anyway, despite being so uncertain.
## Source graph
High-value links inside this post:
- Stop pressing the try-harder button: https://www.neelnanda.io/blog/mini-blog-post-6-stop-pressing-the-try-harder-button
- Negative results for SAEs on downstream tasks: https://www.alignmentforum.org/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks
- Five-minute timers: https://www.neelnanda.io/blog/post-28-on-creativity-the-joys-of-5-minute-timers
- Weekly review / reflection: https://www.neelnanda.io/blog/39-reflection
- Jacob Steinhardt, Research as a Stochastic Decision Process: https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
docs/evidence/nanda_research_process_research_taste.md
+49
View File
@@ -0,0 +1,49 @@
# My Research Process: Understanding and Cultivating Research Taste - Neel Nanda (2025-05-01)
Source: https://www.lesswrong.com/posts/Ldrss6o3tiKT6NdMm/my-research-process-understanding-and-cultivating-research
Author: Neel Nanda
Date: 1st May 2025
Fetch-status: excerpted from LessWrong HTML via browser plus cross-checked against local shared draft.
Use: core research-taste evidence, especially for deciding whether this should become a separate skill.
## Why this matters for agents
This post gives the boundary: research taste is not just picking ideas. It is judgment under long feedback loops across problem choice, exploration, experiment design, and distillation. It also explains why taste is learnable but slow: the feedback data is sparse.
## Quotes
> What is research taste? As I define it, research taste is far broader than just picking the right problem at the outset. Research is full of key decisions that will affect the future of the project, without an obvious way to find the right answer: from choosing the research problem itself, to identifying which anomalies are and are not worth exploring, distinguishing an experiment that will be compelling from one that’ll have inconclusive results, etc.
> I think of taste as the set of intuitions and good judgment that guide a researcher’s decisions throughout the research process, any time an ambiguous or open-ended decision like this arises.
> The core problem is you just don't get that much data. Generally the shorter a feedback loop is the more data you will get. By definition research taste is about things that are not immediately obvious.
> I think the main way to speed it up is by getting more data, and by being more sample efficient about the data that you have.
> When you have made a research decision and you eventually get feedback, do a post-mortem analyzing what did and did not work and why and what general themes you could look at in future.
> As discussed, I define research taste broadly: it's the collection of intuitions and judgments that guide good decision-making throughout a research project, especially where feedback loops are long, and the search space is large and open-ended.
> Exploration: A tactical sense for which experiments yield the most insight, recognizing interesting anomalies versus noise, knowing when to dig deeper or move on from a thread.
> Understanding: Designing creative, elegant experiments that cleanly distinguish hypotheses, judging the plausibility and explanatory power of different theories, identifying crucial assumptions or potential confounds.
> Communication & Distillation: Identifying the core, communicable claims within messy findings, structuring a compelling and true narrative, anticipating audience confusion, knowing what makes a result impactful to others.
> The ideal is strategic conviction: the ability to adopt a confident mindset to maintain momentum, while regularly zooming out to reflect and maintaining the capacity for zoomed-out skepticism and the willingness to update or abandon course based on evidence.
> Keep a research log. Ask why things worked or failed. Was it luck, execution, or a fundamental judgment call (taste)?
> Papers are a biased dataset (publication bias!), but still useful.
> Research taste isn't magic. It's a complex set of intuitions and frameworks built incrementally through experience, reflection, and learning from others. It governs the crucial, often implicit, decisions that shape a research project's success.
> Focus first on mastering the skills with shorter feedback loops – coding, running experiments, analyzing data, clearly communicating simple results.
## Source graph
High-value links inside this post:
- Chris Olah, research taste / supervised data framing: https://colah.github.io/notes/taste/
- Weekly reviews: https://www.neelnanda.io/blog/39-reflection
- Activation patching paper: https://arxiv.org/abs/2309.16042
- Gears-level model reference: https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments
docs/evidence/nanda_research_process_shared_draft.md
+75
View File
@@ -0,0 +1,75 @@
# Shared Publicly: My Model of the Research Process - Neel Nanda draft/local copy
Source file: /home/wassname/Downloads/[Shared Publicly] My Model of the Research Process_ Explore, Understand, Distill.md
Author shown in content: Neel Nanda
Date: not stated in local file; contains published posts dated 2025-04-26, 2025-04-27, and 2025-05-01 plus expanded stage-guide material.
Fetch-status: local user-provided/downloaded markdown. Treat as a shared draft/local copy, not identical to the public LessWrong pages.
Use: practical stage guide for agents. This is the most operational source for ideation, exploration, understanding, distillation, failure modes, and mentor role.
## Why this matters for agents
The published posts establish the frame. This local draft contains the useful agent checklist: when to ideate, when to explore, what counts as surface area, how to test hypotheses, how to refine evidence, and when to go back a stage.
## Quotes
> You can't do research without a question or a domain. Ideation is about finding fertile ground. It might be quick, eg deferring to a mentor, or it might involve significant exploration itself, with explorations of many unpromising domains before you settle on one.
> While research taste is important, there are many other crucial skills, and research taste itself comprises several distinct abilities that shouldn't be naively conflated. Rather than focusing solely on research taste, I’ve tried to break down the research process into concrete and specific skills.
> Chris Olah has an excellent short post on what research taste is and exercises to learn it. In this spirit, for each of the aspects of the below, I highly recommend predicting a mentor’s answer before asking.
> Ideation ends when you have a clear enough question or domain that you can start generating concrete experiments to run.
> Leverage Mentors: Especially early on, it’s fine to let someone else do the work here, i.e. have a mentor recommend a problem.
> This is basically borrowing someone else’s research taste, and IMO is one of the most valuable things I do for my mentees.
> Goal: Gain understanding of the problem/domain, start to identify and crystallise interesting hypotheses.
> Your north star is information gained per unit time/effort.
> Crucially, Exploration is not about testing a specific hypothesis. Exploration is about gaining enough of an understanding of a domain that you know what the interesting hypotheses even are.
> It’s OK to be confused: It’s totally normal to spend a large fraction of this stage feeling pretty confused about what’s going on. This is fine and does not mean that you’re failing! The key question is whether you feel like you are learning things and becoming less confused.
> Reach for a tool that might show you something interesting, and can be employed fast. Don’t hold yourself to the standard of tools that you’re confident are good.
> Notice Weirdness: This is critical. Pay close attention to results that are surprising, counter-intuitive, inconsistent, or just feel off. Ask "Why?" relentlessly.
> Research Log: Keep a detailed log (daily or per session). Note down: goals for the session, what you tried, observations (especially weird ones!), links to code/plots (eg to notebooks or git commits or saved plots), brief thoughts/interpretations, ideas for next steps.
> Mentorship Role: Suggesting initial explorations & relevant resources, distinguishing genuinely weird results from known artifacts, providing sanity checks, helping prioritize which weirdness to pursue first.
> Goal: Rigorously testing specific, plausible hypotheses.
> Design High Information Experiments: Design experiments specifically to differentiate between your main hypothesis and the most plausible alternatives. Ask: "What prediction does H1 make that H2 contradicts?" Think like a Bayesian: what evidence is most likely under H1 relative to H2?
> Avoid the mistake of looking for evidence predicted by H1 that’s also predicted by a bunch of other things!
> Use appropriate baselines - e.g. it’s not enough to show that your technique helps to lower a model’s performance on harmful tasks. Does a random vector do worse?
> Actively Seek Alternatives: Explicitly brainstorm other ways your observations could be explained. What are the simplest explanations? What known circuits or phenomena could be involved? What would a strong skeptic argue?
> Mentorship Role: Aggressively red teaming hypotheses and experimental designs. Suggesting crucial alternative hypotheses or experiments. Helping interpret confusing results. Conveying conceptual frameworks to make sense of findings. Pushing for higher standards of rigor and clarity.
> Goal: Distill all the messy insights from your research into concise, rigorous truth to communicate it to the world.
> Select Strongest Evidence: To start, choose the clearest, most convincing experiments, visualizations, and analyses that directly support your main claims.
> Acknowledge limitations: Inevitably, your results will have some limitations - edge cases, ways your evidence could be wrong, etc. I strongly encourage you to discuss these clearly and prominently in a write-up, even if you don’t have good counters to it.
> Your goal is to inform not persuade.
> The truth is what it is, and you should strive to understand it, even if it is inconvenient.
## Source graph
Links visible in this local draft worth follow-up:
- Chris Olah, research taste: https://colah.github.io/notes/taste/
- Jacob Steinhardt, Research as a Stochastic Decision Process: https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
- Nanda paper reading list: https://www.alignmentforum.org/posts/NfFST5Mio7BCAQHPA/an-extremely-opinionated-annotated-list-of-my-favourite
- Nanda Othello research process: https://www.alignmentforum.org/s/nhGNHyJHbrofpPbRG/p/TAz44Lb9n9yf52pv8
- Nanda five-minute timers: https://www.neelnanda.io/blog/post-28-on-creativity-the-joys-of-5-minute-timers
- Nanda weekly reflection: https://www.neelnanda.io/blog/39-reflection
- Negative results for SAEs: https://www.alignmentforum.org/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks
- Research Debt: referenced by name in local draft; URL not included in visible excerpt.
docs/evidence/olah_research_taste_exercises.md
+42
View File
@@ -0,0 +1,42 @@
# Research Taste Exercises - Chris Olah (2021-01-09)
Source: https://colah.github.io/notes/taste/
Author: Chris Olah
Date: Posted Jan 9, 2021
Fetch-status: excerpted from HTML via browser.
Use: direct source for research-taste training exercises; cited by Nanda's shared draft and public taste post.
## Why this matters for agents
Olah gives concrete exercises for getting more feedback on taste without spending months executing every idea. This is a good reference for an agent asked to help a researcher improve project selection or calibrate idea quality.
## Quotes
> One of the most important aspects of growing as a researcher is developing research taste -- roughly, the ability to chose good problems to work on.
> I think the fundamental issue is that actually testing whether a research idea you come up with is good is very expensive. Often it takes months, so you only really get a few pieces of feedback on your taste every year.
> Many of the following exercises are really strategies for getting (proxy) feedback on more research ideas faster.
> Write down a list of research ideas. Have a mentor you respect rate each idea 1-10. Discuss ideas where you disagree with them after reflection.
> Pay attention when other people try ideas you’ve had. How did the results compare with your expectations?
> Interview researchers around you on their taste. Why do they work on the problems they do? How do they pick problems? What’s their “big picture” of research?
> Critically consider your research taste, and the community taste around you. Your taste is likely very influenced by your research cluster (your collaborators, advisor, etc).
> Failure Mode 1: Getting overly attached to one research direction / falling into sunk costs.
> Failure mode 2: Lack of research knowledge / intimacy.
> Theoretical knowledge is table stakes for research taste. You can’t have research taste in a vacuum.
> Failure mode 3: Environment not aligned with your interests.
## Source graph
Links visible in the post worth follow-up:
- Hamming, You and Your Research: linked via YouTube.
- Michael Nielsen, Principles of Effective Research: https://michaelnielsen.org/blog/principles-of-effective-research/
- Andy Matuschak taste-related thread: linked as Twitter, may need archival route.
docs/evidence/olsson_80000hours_ml_engineering_ai_safety.md
+35
View File
@@ -0,0 +1,35 @@
# ML Engineering for AI Safety & Robustness - Catherine Olsson and 80,000 Hours (2018-11)
Source: https://80000hours.org/articles/ml-engineering-career-transition-guide/
Authors: Catherine Olsson and the 80,000 Hours team
Date: Published November 2018; update note visible Feb 2022
Fetch-status: excerpted from HTML via browser.
Use: source-graph evidence from Spinning Up's "Other Resources" section; useful for research-engineer skill acquisition, less central to research taste.
## Why this matters for agents
This source is more about becoming useful on ML research teams than choosing research ideas. Its most relevant claim is that implementing and debugging foundational algorithms is a high-value learning path, with easy environments, metrics, and reference-code scrutiny.
## Quotes
> Technical AI safety is a multifaceted area of research, with many sub-questions in areas such as reward learning, robustness, and interpretability.
> Not all of these questions are best tackled with abstract mathematics research; some can be approached with concrete coding experiments and machine learning (ML) prototypes.
> Once you know the 101-level basics of ML, the next thing to learn is how to implement and debug ML algorithms.
> Breadth of experience is not important here: you don’t need to read all the latest papers, or master an extensive reading list. You also don’t need to do novel research or come up with new algorithms.
> What you do need is to get your hands dirty implementing and debugging ML algorithms, and to build evidence for job interviews that you have some experience doing this.
> The most straightforward way to gain this experience is to choose a subfield of ML relevant to a lab you’re interested in. Then read a few dozen of the subfield’s key papers, and reimplement a few of the foundational algorithms that the papers are based on or reference most frequently.
> For each algorithm, they would first test on very easy environments, and then move to more difficult environments.
> Once the algorithm was partially working, they would attain higher performance by looking for remaining bugs, both by reviewing the code carefully, and by collecting metrics such as average policy entropy to perform sanity-checks, rather than just tune hyperparameters.
> Most importantly, he was able to implement and debug ML algorithms, going from math in a paper to running code.
## Source graph
This page was linked from Spinning Up's "Other Resources" section. It points to Josh Achiam's Key Papers in Deep RL list and a Daniel Ziegler self-study path. It is useful background for training agents to value implementation and debugging practice, but probably secondary for a dedicated research-taste skill.
docs/evidence/spinningup_research_source_graph.md
+69
View File
@@ -0,0 +1,69 @@
# Spinning Up as a Deep RL Researcher - source graph and research-taste excerpts
Primary source: https://spinningup.openai.com/en/latest/spinningup/spinningup.html
Author: Joshua Achiam, OpenAI
Date: October 13th, 2018
Related local cache: docs/evidence/spinningup_researcher.md
Fetch-status: excerpted from Spinning Up HTML via browser; source graph cross-checked against existing local evidence files where present.
Use: RL research-process evidence, especially for source graph, fair comparisons, seeds, preregistration, and ablations.
## Why this matters for agents
Spinning Up is not just an RL textbook page. Its researcher page is a compact research apprenticeship guide. It points to the same battle-tested debugging and reproducibility references already cached in this repo, then adds project selection and rigorous comparison advice.
## Quotes
> If you’re an aspiring deep RL researcher, you’ve probably heard all kinds of things about deep RL by this point. You know that it’s hard and it doesn’t always work. That even when you’re following a recipe, reproducibility is a challenge. And that if you’re starting from scratch, the learning curve is incredibly steep.
> In particular, this will outline a useful curriculum for increasing raw knowledge, while interleaving it with the odds and ends that lead to better research.
> Write your own implementations. You should implement as many of the core deep RL algorithms from scratch as you can, with the aim of writing the shortest correct implementation of each.
> Simplicity is critical. You should organize your efforts so that you implement the simplest algorithms first, and only gradually introduce complexity.
> Don’t overfit to existing implementations either. Study existing implementations for inspiration, but be careful not to overfit to the engineering details of those implementations.
> Iterate fast in simple environments. To debug your implementations, try them with simple environments where learning should happen quickly.
> Your ideal experiment turnaround-time at the debug stage is <5 minutes (on your local machine) or slightly longer but not much.
> Start by exploring the literature to become aware of topics in the field.
> Use the related work section and citations to find closely-related papers and do a deep dive in the literature. You’ll start to figure out where the unsolved problems are and where you can make an impact.
> There are a many different ways to start thinking about ideas for projects, and the frame you choose influences how the project might evolve and what risks it will face.
> Avoid reinventing the wheel. When you come up with a good idea that you want to start testing, that’s great! But while you’re still in the early stages with it, do the most thorough check you can to make sure it hasn’t already been done.
> Under no circumstances handicap the baseline!
> Beware of random seeds making things look stronger or weaker than they really are, so run everything for many random seeds (at least 3, but if you want to be thorough, do 10 or more).
> This is to enforce a weak form of preregistration: you use the tuning stage to come up with your hypotheses, and you use the final runs to come up with your conclusions.
> Check each claim separately. Another critical aspect of doing research is to run an ablation analysis.
## Source graph
Spinning Up intro references, with local status:
- Alex Irpan, Deep Reinforcement Learning Doesn't Work Yet: https://www.alexirpan.com/2018/02/14/rl-hard.html. Local cache: docs/evidence/alexirpan_rl_hard.md.
- Islam et al., Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control: https://arxiv.org/abs/1708.04133. Not separately cached; discussed/cited inside Henderson local cache.
- Henderson et al., Deep Reinforcement Learning that Matters: https://arxiv.org/abs/1709.06560. Local cache: docs/evidence/henderson_2018_deep_rl_matters.md.
- Matthew Rahtz, Lessons Learned Reproducing a Deep RL Paper: http://amid.fish/reproducing-deep-rl. Local cache: docs/evidence/amid_fish_reproducing_deep_rl.md.
- David Silver UCL RL course: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html. Not cached.
- Berkeley Deep RL course: http://rll.berkeley.edu/deeprlcourse/. Not cached.
- Deep RL Bootcamp lectures: https://sites.google.com/view/deep-rl-bootcamp/lectures. Reddit index cache: docs/evidence/reddit_deeprl_bootcamp_2017_75m5vd.md.
- John Schulman, Nuts and Bolts of Deep RL: http://joschu.net/docs/nuts-and-bolts.pdf. Local cache: docs/evidence/joschu_nuts_and_bolts.md.
- Tim Rocktaschel et al., Advice for Short-term Machine Learning Research Projects: https://rockt.github.io/2018/08/29/msc-advice.html. Not cached.
- Catherine Olsson / 80,000 Hours, ML Engineering for AI Safety & Robustness: https://80000hours.org/articles/ml-engineering-career-transition-guide/. Not cached.
## Likely follow-up cache candidates
Priority 1:
- Chris Olah, research taste: short and directly named by Nanda.
- Jacob Steinhardt, Research as a Stochastic Decision Process: directly named by Nanda for prioritization.
- Tim Rocktaschel et al., short-term ML research projects: directly named by Spinning Up for research growth.
Priority 2:
- David Silver/UCL, Berkeley Deep RL, Deep RL Bootcamp: curriculum material, less directly research-taste except via RL mastery.
- Catherine Olsson/80k: career/field-entry framing; useful if the skill expands beyond project-level research taste.
docs/evidence/steinhardt_research_stochastic_decision_process.md
+43
View File
@@ -0,0 +1,43 @@
# Research as a Stochastic Decision Process - Jacob Steinhardt
Source: https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
Author: Jacob Steinhardt
Date: not visible in fetched HTML
Fetch-status: excerpted from HTML via browser.
Use: research-prioritization evidence; cited by Nanda's Key Mindsets post.
## Why this matters for agents
Steinhardt gives a crisp formal-ish rule for research prioritization: reduce uncertainty as fast as possible. This is useful for agents deciding which experiment, baseline, prototype, or sanity check to run first.
## Quotes
> Below I analyze how to approach a project that has many somewhat independent sources of uncertainty (we can often think of these as multiple "steps" or "parts" that each have some probability of success).
> We will eventually see that a good principle is to "reduce uncertainty at the fastest possible rate".
> This reveals that harder tasks should not necessarily be prioritized. Rather, we should prioritize tasks that are more likely to fail (so that we remove the risk of them failing) but also tasks that take less time.
> Do the components in order from most informative per unit time to least informative per unit time.
> De-risk all components (to the extent feasible), then execute.
> Specifically, for each task we want a cheap way to obtain high confidence about whether that task will be feasible. This is called "de-risking".
> We are often either in "de-risking mode" (determining if the problem is infeasible as quickly as possible) or "execution mode" (assuming the problem is feasible and trying to solve it quickly).
> The counterpart to ceilings are baselines--simple or off-the-shelf methods that give a quick lower bound on achievable accuracy.
> Together with ceilings, they delineate a range of possible performance, which helps us interpret our core results.
> I often think about possible approaches to a problem as an exponentially branching search tree.
> Whenever something doesn't work, I ask why it didn't work. My goal is to avoid trying similar things that will fail for the same reason.
> Compared to other people I know, I try harder and earlier to show that my ideas can't work to solve a problem.
> We often try easier tasks first, when instead we should try the most informative tasks first.
## Source graph
This is a standalone blog post. It links to concepts like Poisson arrival processes, but the skill-relevant content is the prioritization/de-risking frame above.
refs/research_taste.md
+167
View File
@@ -0,0 +1,167 @@
# Research taste and research-process folklore
Appendix to the [ML Debugging skill](../SKILL.md).
Use this when the task is not only "why is this broken?" but "what should we try next, what would teach us the most, or how do we turn messy research into claims?" Debugging asks what is false in the current system. Research taste asks which next observation is worth buying with time.
This is an evidence map, not a finished new skill. It preserves the source language and points to the local evidence cache so a later high-context pass can decide what belongs in `SKILL.md`.
## The research loop
Neel Nanda's sequence gives the cleanest stage model: ideation, exploration, understanding, and distillation.[^nanda-explore]
> I see research as breaking down into a few stages:
>
> 1. Ideation - Choose a problem/domain to focus on
> 2. Exploration - Gain Surface area
> 3. Understanding - Test Hypotheses
> 4. Distillation - Compress, Refine, Communicate
For agents, the first diagnostic question is "what stage are we in?" A confused exploration stage should optimize for surface area, not premature proof. An understanding stage should optimize for discriminating hypotheses. A distillation stage should turn evidence into claims a skeptical outsider can audit.
> Not having a clear goal/next step doesn't mean that you don't need to prioritise! Prioritise for information gain.[^nanda-explore]
> The mark of a good researcher is a deep commitment to skepticism of your results.[^nanda-explore]
## Taste is trainable, but feedback is sparse
Nanda's taste post and Olah's taste exercises agree on the bottleneck: project-level feedback is slow, so you need proxy feedback and deliberate reflection.[^nanda-taste][^olah-taste]
> Research taste isn't magic. It's a complex set of intuitions and frameworks built incrementally through experience, reflection, and learning from others.[^nanda-taste]
> The core problem is you just don't get that much data.[^nanda-taste]
> Many of the following exercises are really strategies for getting (proxy) feedback on more research ideas faster.[^olah-taste]
Practical folklore:
- Write down research ideas and predict what a mentor will say before asking.
- When surprised, paraphrase the mentor's reasoning until the missing heuristic is explicit.
- Review decisions after reality answers: was the outcome luck, execution, or taste?
- Read papers as offline data, but remember publication bias and hidden jank.
## Ideation
The shared Nanda draft is the best operational guide for choosing a problem. It is local/downloaded evidence, so treat it as a draft rather than a canonical public post.[^nanda-draft]
> You can't do research without a question or a domain. Ideation is about finding fertile ground.[^nanda-draft]
> Ideation ends when you have a clear enough question or domain that you can start generating concrete experiments to run.[^nanda-draft]
> This is basically borrowing someone else's research taste, and IMO is one of the most valuable things I do for my mentees.[^nanda-draft]
Folklore:
- Early in a field, using a mentor's project can be rational. Originality is not the first bottleneck.
- If no mentor exists, extend a paper you like or use a vetted open-problems list.
- A bad problem can sink a project before any debugging skill matters.
## Exploration
Exploration is for building surface area, not defending a hypothesis. This is where fast, partial, qualitative, and even cherry-picked observations can be useful, as long as you do not later launder them into proof.[^nanda-draft]
> Your north star is information gained per unit time/effort.[^nanda-draft]
> Crucially, Exploration is not about testing a specific hypothesis.[^nanda-draft]
> Reach for a tool that might show you something interesting, and can be employed fast.[^nanda-draft]
> Notice Weirdness: This is critical.[^nanda-draft]
This connects directly to the ML-debugging folklore: Rahtz says confusion was the clue, Jones says anomalies should be chased, and Spinning Up says simple environments with sub-5-minute turnaround are ideal for debugging.[^rahtz][^spinningup-graph]
> It was only by following that confusion and realising that taking the difference between frames zeroed out the background that gave the hint of a problem with normalization.[^rahtz]
> Your ideal experiment turnaround-time at the debug stage is <5 minutes (on your local machine) or slightly longer but not much.[^spinningup-graph]
## Understanding
Understanding starts when you have candidate hypotheses. The question becomes: what evidence would distinguish them?
> Design High Information Experiments: Design experiments specifically to differentiate between your main hypothesis and the most plausible alternatives.[^nanda-draft]
> Avoid the mistake of looking for evidence predicted by H1 that's also predicted by a bunch of other things![^nanda-draft]
> Use appropriate baselines.[^nanda-draft]
Steinhardt gives the matching prioritization rule: do the action that reduces uncertainty fastest, not the one that feels easiest or most complete.[^steinhardt]
> Do the components in order from most informative per unit time to least informative per unit time.[^steinhardt]
> De-risk all components (to the extent feasible), then execute.[^steinhardt]
For RL and unstable ML experiments, Spinning Up, Henderson, Schulman, and Irpan converge on the same practical discipline: strong baselines, many seeds, ablations, and instrumentation.[^spinningup-graph][^henderson][^schulman][^irpan]
> Under no circumstances handicap the baseline![^spinningup-graph]
> Always Be Ablating.[^schulman]
> Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful.[^henderson]
## Distillation and paper writing
Nanda's paper-writing post is the most direct source for the distillation stage. It is not only writing advice; it is a test of whether the research has compressed into claims and evidence.[^nanda-paper]
> The essence of an ideal paper is the narrative: a short, rigorous and evidence-based technical story you tell, with a takeaway the readers care about.[^nanda-paper]
> The first step is to compress your research into these claims.[^nanda-paper]
> Inform, not persuade: Avoid the trap of overclaiming or ignoring limitations.[^nanda-paper]
> Readers will rarely take away more than a few sentences of content. Choose those sentences carefully.[^nanda-paper]
> Warning: Before moving into paper-writing mode, it's crucial to verify that your evidence is actually correct.[^nanda-paper]
> The Guiding Question for Evidence: Ultimately, the question to ask about your evidence is: "Should this update a reader's beliefs about my claims?"[^nanda-paper]
Folklore:
- Compress to one to three claims. If you cannot, you probably do not know the contribution yet.
- Red-team the narrative because elegant stories are where false research hides.
- Move weak or peripheral evidence to appendices. Keep the main text for the hard-to-deny evidence.
- Write to inform; persuasion pressure is how limitations vanish.
## For agents
The agent version is simple:
1. State the current stage: ideation, exploration, understanding, or distillation.
2. Name the north star for that stage.
3. Propose the next action by information gained per unit time.
4. Say what would change your mind before running it.
5. Preserve proof: logs, plots, commits, quotes, or tables.
If the user is AFK, continue through the loop. If the evidence invalidates the original hypothesis, update the plan rather than defending the old narrative.
## See also / source graph
Most relevant sources cached for this reference:
- Neel Nanda, research-process sequence: [explore/understand/distill](../docs/evidence/nanda_research_process_explore_understand_distill.md), [key mindsets](../docs/evidence/nanda_research_process_key_mindsets.md), [research taste](../docs/evidence/nanda_research_process_research_taste.md), [shared draft](../docs/evidence/nanda_research_process_shared_draft.md), [paper writing](../docs/evidence/nanda_highly_opinionated_ml_paper_writing.md).
- Chris Olah, [Research Taste Exercises](../docs/evidence/olah_research_taste_exercises.md): proxy feedback, mentor ratings, research intimacy.
- Jacob Steinhardt, [Research as a Stochastic Decision Process](../docs/evidence/steinhardt_research_stochastic_decision_process.md): information rate, de-risking, ceilings, baselines.
- Joshua Achiam / OpenAI Spinning Up, [research source graph](../docs/evidence/spinningup_research_source_graph.md) and [original cache](../docs/evidence/spinningup_researcher.md): RL apprenticeship, fair comparisons, seeds, preregistration, ablations.
- Matthew Rahtz, [Lessons Learned Reproducing a Deep RL Paper](../docs/evidence/amid_fish_reproducing_deep_rl.md): confusion, long iteration times, think more before expensive runs.
- Henderson et al., [Deep Reinforcement Learning that Matters](../docs/evidence/henderson_2018_deep_rl_matters.md): seed variance, implementation differences, reproducibility reporting.
- John Schulman, [Nuts and Bolts of Deep RL Research](../docs/evidence/joschu_nuts_and_bolts.md): small test problems, health indicators, multiple seeds, ablations.
- Alex Irpan, [Deep Reinforcement Learning Doesn't Work Yet](../docs/evidence/alexirpan_rl_hard.md): realistic expectations, sample inefficiency, seed variance.
Less central but useful:
- Catherine Olsson / 80,000 Hours, [ML Engineering for AI Safety & Robustness](../docs/evidence/olsson_80000hours_ml_engineering_ai_safety.md): implementation/debugging as research-engineer apprenticeship.
- Tim Rocktaschel et al., Advice for Short-term Machine Learning Research Projects: linked by Spinning Up but not cached yet.
- Islam et al., Reproducibility of Benchmarked Deep RL Tasks: linked by Spinning Up; not separately cached, but discussed in Henderson.
- David Silver UCL RL course, Berkeley Deep RL course, and Deep RL Bootcamp: curriculum links from Spinning Up; useful for background, less directly research-taste.
[^nanda-explore]: Neel Nanda, "How I Think About My Research Process: Explore, Understand, Distill" (2025-04-26) - https://www.lesswrong.com/posts/hjMy4ZxS5ogA9cTYK/how-i-think-about-my-research-process-explore-understand ([cache](../docs/evidence/nanda_research_process_explore_understand_distill.md)).
[^nanda-key]: Neel Nanda, "My Research Process: Key Mindsets - Truth-Seeking, Prioritisation, Moving Fast" (2025-04-27) - https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/cbBwwm4jW6AZctymL ([cache](../docs/evidence/nanda_research_process_key_mindsets.md)).
[^nanda-taste]: Neel Nanda, "My Research Process: Understanding and Cultivating Research Taste" (2025-05-01) - https://www.lesswrong.com/posts/Ldrss6o3tiKT6NdMm/my-research-process-understanding-and-cultivating-research ([cache](../docs/evidence/nanda_research_process_research_taste.md)).
[^nanda-draft]: Neel Nanda, shared/local draft, "My Model of the Research Process" - source file `/home/wassname/Downloads/[Shared Publicly] My Model of the Research Process_ Explore, Understand, Distill.md` ([cache](../docs/evidence/nanda_research_process_shared_draft.md)).
[^nanda-paper]: Neel Nanda, "Highly Opinionated Advice on How to Write ML Papers" (2025-05-12) - https://www.lesswrong.com/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers ([cache](../docs/evidence/nanda_highly_opinionated_ml_paper_writing.md)).
[^olah-taste]: Chris Olah, "Research Taste Exercises" (2021-01-09) - https://colah.github.io/notes/taste/ ([cache](../docs/evidence/olah_research_taste_exercises.md)).
[^steinhardt]: Jacob Steinhardt, "Research as a Stochastic Decision Process" - https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html ([cache](../docs/evidence/steinhardt_research_stochastic_decision_process.md)).
[^spinningup-graph]: Joshua Achiam, "Spinning Up as a Deep RL Researcher" (OpenAI, 2018-10-13) - https://spinningup.openai.com/en/latest/spinningup/spinningup.html ([research cache](../docs/evidence/spinningup_research_source_graph.md), [debugging cache](../docs/evidence/spinningup_researcher.md)).
[^rahtz]: Matthew Rahtz, "Lessons Learned Reproducing a Deep Reinforcement Learning Paper" (2018) - http://amid.fish/reproducing-deep-rl ([cache](../docs/evidence/amid_fish_reproducing_deep_rl.md)).
[^henderson]: Henderson et al., "Deep Reinforcement Learning that Matters" (AAAI 2018) - https://arxiv.org/abs/1709.06560 ([cache](../docs/evidence/henderson_2018_deep_rl_matters.md)).
[^schulman]: John Schulman, "Nuts and Bolts of Deep RL Research" (2016) - http://joschu.net/docs/nuts-and-bolts.pdf ([cache](../docs/evidence/joschu_nuts_and_bolts.md)).
[^irpan]: Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (2018) - https://www.alexirpan.com/2018/02/14/rl-hard.html ([cache](../docs/evidence/alexirpan_rl_hard.md)).