mirror of
https://github.com/wassname/ml-debug.git
synced 2026-09-09 11:27:03 +08:00
evidence: full text for nine LessWrong and research-blog caches
All nine said 'excerpted from HTML via browser' and held 141-684 words. LessWrong posts refetched through the markdown API, the rest through jina. All 99 previously quoted passages, and the 46 quotes SKILL.md and refs/research_taste.md take from them, verify against the new full texts.
This commit is contained in:
wassname
@@ -3,59 +3,13 @@
|
||||
Source: https://www.lesswrong.com/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers
|
||||
Author: Neel Nanda
|
||||
Date: 12th May 2025
|
||||
Fetch-status: excerpted from LessWrong HTML via browser.
|
||||
Fetch-status: full post text, fetched 2026-08-15 from the LessWrong markdown API (`/api/post/eJGptPbbFPZGLpjsp`), comments and site navigation stripped. (CLAUDE agent)
|
||||
Use: distillation and paper-writing evidence. This is adjacent to the research-process sequence, and directly useful when turning messy findings into a public artifact.
|
||||
|
||||
## Why this matters for agents
|
||||
|
||||
This post is the operational version of the distillation stage: compress the research into a few claims, red-team the evidence, write to inform rather than persuade, and spend disproportionate care on the abstract, intro, figures, and limitations.
|
||||
|
||||
## Quotes
|
||||
|
||||
> The essence of an ideal paper is the narrative: a short, rigorous and evidence-based technical story you tell, with a takeaway the readers care about.
|
||||
|
||||
> The first step is to compress your research into these claims.
|
||||
|
||||
> Experimental Evidence: This is absolutely crucial to get right and aggressively red-team, it’s how you resist the temptation of elegant but false narratives.
|
||||
|
||||
> Inform, not persuade: Avoid the trap of overclaiming or ignoring limitations.
|
||||
|
||||
> Your research only matters if people read, understand, and build upon it.
|
||||
|
||||
> At its core, a paper should present a narrative of one to three specific concrete claims that you believe to be true, that build to some useful takeaway(s).
|
||||
|
||||
> Readers will rarely take away more than a few sentences of content. Choose those sentences carefully.
|
||||
|
||||
> Generally, stronger statements make for more interesting papers, but require higher standards of evidence - resist the temptation to overclaim for clicks!
|
||||
|
||||
> Warning: Before moving into paper-writing mode, it's crucial to verify that your evidence is actually correct.
|
||||
|
||||
> Novelty means it expands our knowledge.
|
||||
|
||||
> Rigorous, at-scale replications of shaky results, negative results of seemingly promising hypotheses, and high-quality failed replications of popular papers are all very valuable contributions.
|
||||
|
||||
> A particularly important thing to get right is extensive red-teaming: you should spend a good amount of your time, both during the original research and now, red teaming your narrative.
|
||||
|
||||
> Good experiments distinguish between hypotheses.
|
||||
|
||||
> This skepticism and sanity checking is especially key for particularly surprising or novel bits of evidence.
|
||||
|
||||
> Ablation studies: When a paper introduces a complex new method, there are often several moving parts.
|
||||
|
||||
> Track pre/post-hoc analysis.
|
||||
|
||||
> Quality Over Quantity: Try to prioritise having at least one really compelling and hard to deny experiment, over a bunch of mediocre ones.
|
||||
|
||||
> Baselines are Crucial.
|
||||
|
||||
> The subtlety of baselines: It's not enough to just have them; you must strive to have the strongest possible baselines.
|
||||
|
||||
> The Guiding Question for Evidence: Ultimately, the question to ask about your evidence is: "Should this update a reader's beliefs about my claims?"
|
||||
|
||||
> Reproducibility & Publishing code: Rigour can be in the eye of the beholder: if readers cannot understand or verify it for themselves, it’s far harder to consider it rigorous.
|
||||
|
||||
> A key challenge in paper writing is the illusion of transparency - you have spent months steeped in the context of this research project.
|
||||
|
||||
## Source graph
|
||||
|
||||
Links visible in this post worth follow-up:
|
||||
@@ -65,3 +19,560 @@ Links visible in this post worth follow-up:
|
||||
- Refusal is mediated by a single direction: https://arxiv.org/abs/2406.11717
|
||||
- Nanda grokking work: https://arxiv.org/abs/2301.05217
|
||||
- Paper writing checklist: Google Docs link visible in post, not cached.
|
||||
|
||||
## Full post
|
||||
|
||||
TL;DR
|
||||
-----
|
||||
|
||||
* **The essence of an ideal paper** is the **narrative**: a short, rigorous and evidence-based technical story you tell, with a takeaway the readers care about
|
||||
* **What?** A narrative is fundamentally about a contribution to our body of knowledge: **one to three specific novel claims** that fit within a cohesive theme
|
||||
* **Why?** You need **rigorous empirical evidence** that convincingly supports your claims
|
||||
* **So what?** Why should the reader care?
|
||||
* What is the **motivation**, the problem you’re trying to solve, the way it all fits in the bigger picture?
|
||||
* What is the **impact**? Why does your takeaway matter? The **north star** of a paper is ensuring the reader **understands** and **remembers** the narrative, and **believes** that the paper’s evidence supports it
|
||||
* The first step is to **compress your research** into these claims.
|
||||
* The paper must **clearly motivate these claims, explain them on an intuitive and technical level**, and **contextualise what’s novel** in terms of the prior literature
|
||||
* This is the role of the abstract & introduction
|
||||
* **Experimental Evidence**: This is absolutely crucial to get right and aggressively red-team, it’s how you resist the temptation of elegant but false narratives.
|
||||
* **Quality > Quantity**: find compelling experiments, not a ton of vaguely relevant ones.
|
||||
* **The experiments and results must be explained in full technical detail** \- start high-level in the intro/abstract, show results in figures, and get increasingly detailed in the main body and appendix.
|
||||
* **Ensure researchers can check your work** \- provide sufficient detail to be replicated
|
||||
* **Define key terms and techniques** \- readers have less context than you think.
|
||||
* **Write iteratively**: Write abstract -> bullet point outline -> introduction -> first full draft -> repeat
|
||||
* Get feedback and reflect after each stage
|
||||
* Spend comparable amounts of time on each of: the abstract, the intro, the figures, and everything else - they have about the same number\_of\_readers * time\_to\_read
|
||||
* **Inform, not persuade**: Avoid the trap of overclaiming or ignoring limitations. Scientific integrity may get you less hype, but gains respect from the researchers who matter.
|
||||
* **Precision, not obfuscation**: Use jargon where needed to precisely state your point, but not for the sake of sounding smart. Use simple language wherever possible.
|
||||
|
||||
|
||||
|
||||
***Case study**: The abstract of* [*refusal is mediated by a single direction*](https://arxiv.org/abs/2406.11717?), broken down into the purpose of each sentence
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
**Your research only matters if people read, understand, and build upon it**. This means that **writing a good paper is a critical part of the research process**. Further, the process of writing forces you to clarify your own thinking in ways that often reveal gaps or new insights - I’ve often only properly understood an idea after writing it up. Yet, to many, writing feels less fun than research and is treated as an after thought - **a common but critical mistake**.
|
||||
|
||||
In my experience supervising 20+ papers and reading/appreciating/being annoyed by a bunch more, I've developed my own opinionated framework for what I think makes a paper good and how to approach the writing process. I try to lay this out in this post, along with a bunch of concrete advice.[^wwk8u16jdjf] This post assumes you’ve already done a bunch of technical research, and focuses on how to effectively share it with the world, [see my other posts](/api/post/hjMy4ZxS5ogA9cTYK) for advice on the research part.
|
||||
|
||||
**Caveat**: I mostly have experience with writing mechanistic interpretability papers and this advice is written with that flavour. I expect much of it to generalise to the rest of ML and some to generalise to other fields but it's hard for me to say. Further, this is very much my personal opinionated, and optimised more for truth-seeking than getting into conferences[^5kbv6xekx6j]. See other great advice [here](https://www.jakobfoerster.com/how-to-ml-paper) and [here](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html).
|
||||
|
||||
**Caveat 2**: There are many reasonable objections to the academic paper as the format for communicating research. Alas, engaging with those is outside the scope of this post, it’s long enough as it is.
|
||||
|
||||
The Essence of a Paper
|
||||
----------------------
|
||||
|
||||
At its core, a paper should **present a narrative** of **one to three specific concrete claims** that you believe to be true, that **build to some useful takeaway**(s). Everything else in the paper exists to support this narrative. The second pillar of the paper is **rigorous evidence for why they are true** \- obviously there will always be some chance you’re wrong, but they should be compelling and believable, without obvious glaring flaws.
|
||||
|
||||
1. **Communicate the key idea**
|
||||
1. **Motivate** why someone should care about them
|
||||
2. **Contextualize** them in existing literature
|
||||
2. **Communicate them precisely**, with all relevant technical detail, terminology and background context
|
||||
3. **Provide sufficient evidence** to support them
|
||||
|
||||
### Crafting a Narrative
|
||||
|
||||
One of the critical steps that can make or break a paper is crafting a narrative. What does this actually mean? And how can you do it?
|
||||
|
||||
Research is about discovering new things and pushing forward our frontier of existing knowledge. I view a paper as something that finds insight and provides compelling evidence behind it. The way to tell when you could start writing a paper is when you have **learned something insightful**, in a way that could be **made legible to someone else**.
|
||||
|
||||
**This is far easier said than done**. A research project is often a mess of fun results, confusions, insights, and remaining mysteries. Even when you’ve made enough progress to write it up, you will likely have a great deal of tacit knowledge, interesting rabbit holes, dangling threads, etc - **projects rarely feel done**.
|
||||
|
||||
I find that converting a project into a great narrative is **a subtle and difficult skill**, one of the many facets of [**research taste**](/api/post/Ldrss6o3tiKT6NdMm). It’s something I’ve gotten much, much better at over time, and it’s hard to say what the best way to get better at it is, beyond experience. One exercise I’d recommend is taking papers you know, and trying to write down what their narrative is, and ask yourself what its strengths and weaknesses are. If at all possible, consult mentors/more experienced researchers for advice. But if you need to come up with one yourself, the right questions to ask look like:
|
||||
|
||||
* Which of these results would be most exciting to show someone?
|
||||
* Actually show someone your findings and ask what they're most interested in
|
||||
* What seems particularly important?
|
||||
* Why should anyone care about this work?
|
||||
* What was hard about what you did, that perhaps no one else has done?
|
||||
|
||||
A good, compelling narrative comes with motivation and impact. The key points to be sure to cover:
|
||||
|
||||
* The context of your insight
|
||||
* The problem you're trying to solve
|
||||
* Why this matters
|
||||
* What you have shown
|
||||
* Why the reader should believe it
|
||||
* What the insight is
|
||||
|
||||
Why do you need this kind of compressed narrative? Often there’s far more insight in a research project than can be contained in this structure. But it is impossible to convey this level of nuance in a paper. Readers will rarely take away more than a few sentences of content. **Choose those sentences carefully**. These are the insights shared by your paper, **your contribution to the literature**. These are the specific, concrete claims that you want to communicate - you cannot reliably communicate much more. You will need to compress your research findings down into a handful of claims, prioritise those, and accept that you may need to drop a bunch of other detail, or move it to appendices. If you don’t deliberately de-prioritise some details, then something else will get dropped, which may have been far more important.
|
||||
|
||||
What do I mean by claims? For example:
|
||||
|
||||
* "Method X is the best approach on task Y (according to metric Z)"
|
||||
* "A substantial part of the model's behavior in scenario A is explained by simple explanation B"
|
||||
* "Technique C can fail in scenario D if conditions E and F hold"
|
||||
|
||||
One important claim, with sufficiently strong evidence, can be enough for a great paper! If you want multiple claims, I strongly recommend **choosing claims that fit together in a cohesive theme** \- papers are far easier to understand, praise, share, etc if there is a **coherent narrative**, not just a grab-bag of unconnected ideas.
|
||||
|
||||
Depending on the strength of the evidence, you can adjust the confidence of a claim:
|
||||
|
||||
* **Existence-proof claims**: "We found at least one example where X happens" (like the indirect object identification paper providing an existence-proof for self-repair)
|
||||
* **Systematic claims**: "X generally happens across a wide range of contexts" or "X is common"
|
||||
* **Hedged claims**: “There is compelling/suggestive/tentative evidence that X is true”
|
||||
* **Narrow claims**: “X is the best method for specific situations V & W, if your goal is objective Y”
|
||||
* **Guarantees**: “X is always true”[^xjcznpnz657]
|
||||
|
||||
|
||||
Generally, stronger statements make for more interesting papers, but require higher standards of evidence - resist the temptation to overclaim for clicks!
|
||||
|
||||
### When to Start?
|
||||
|
||||
Another thorny question is: When should you stop doing research and start writing up your research? This is a hard and subtle question that is, in many ways, a matter of [research taste](/api/post/Ldrss6o3tiKT6NdMm), but here is my general guide:
|
||||
|
||||
1. Write down a list of things you've learned
|
||||
2. Review that list carefully, and ideally show it to someone else
|
||||
3. Ask yourself how comfortable you would be defending the claim that you have provided meaningful, positive evidence for these results
|
||||
4. Think about reasons why others might care about this
|
||||
5. Focus on things you've done that have been hard or non-trivial and look for exciting elements
|
||||
|
||||
But generally, this is unfortunately just a hard thing to tell when starting out, and gets far easier with time and experience. If you can consult a more experienced researcher, definitely do.
|
||||
|
||||
A more meta piece of advice when starting out is to try to choose projects where the narrative will be pretty obvious, e.g. method X beats SOTA method Y in domain Z on metric W
|
||||
|
||||
**Warning**: Before moving into paper-writing mode, **it's crucial to verify that your evidence is actually correct**. An unfortunate fact is that **many published papers are basically false or wildly misleading**. Don't let this happen to you! Carefully check your critical experiments and, if possible, re-implement them through alternate pathways. Ideally, verify all experiments worth mentioning in the paper, or at least 75% of them.
|
||||
|
||||
### Novelty
|
||||
|
||||
A common and confusing requirement for papers is that the results be novel, something that is not covered before. What exactly does this mean? Science is fundamentally about **building a large body of knowledge**. This means that your work exists in the context of what has come before. **Novelty means it expands our knowledge**.
|
||||
|
||||
The conventional definition of novelty can be annoying and, in my opinion, focuses too much on shininess and doesn't capture the more important aspect of whether our knowledge has expanded. Another way to put this is: Should I assign different probabilities to propositions I care about after observing the results of this paper?
|
||||
|
||||
Rigorous, at-scale replications of shaky results, negative results of seemingly promising hypotheses, and high-quality failed replications of popular papers are all very valuable contributions. I would personally consider these novel because they expand our knowledge. However, the revealed preferences of many reviewers and researchers suggest they do not feel the same way. Such is life.
|
||||
|
||||
I don’t want to go too far re criticising novelty: there are many cases where I am uninterested in a paper due to lack of novelty. This primarily occurs with methods that I expect to work when applied in standard settings, and I assign a high probability of success, so the project provides few bits of information. While *knowing* that such a method failed could be interesting, projects can also fail due to researcher incompetence or bad luck. Therefore, it is difficult to draw meaningful conclusions without evidence of researcher competence.
|
||||
|
||||
Leaving that aside, novelty can be hard to communicate. Given a paper on its own, it's difficult to tell what is and is not supposed to be novel:
|
||||
|
||||
* Are the techniques used innovative or just standard techniques?
|
||||
* Does the claim represent a deep conceptual breakthrough?
|
||||
* Is it a very simple extension of standard ideas?
|
||||
* Is it a natural consequence of a more ambitious claim put forward in a different piece of work?
|
||||
|
||||
The main way to address this is to be extremely clear about what is and is not novel, especially in the introduction and related work, and to liberally cite the most relevant papers and explain why your work is and is not different.
|
||||
|
||||
How to find out what came before? **Use a large language model**. If you’re not already familiar with a relevant literature, LLMs are pretty great at doing quick literature reviews, e.g. [Gemini Deep Research](https://gemini.google/overview/deep-research/?hl=en-GB)[^29hhedam5nx]. Reading the literature yourself is much better of course, but takes way, way longer and should have been done at the start project.
|
||||
|
||||
One reason this is very important is that, depending on what’s claimed as novel, the same paper could be perceived as either inappropriately arrogant or making a modest incremental contribution, depending on how the claims are presented.
|
||||
|
||||
Contextualizing your work within existing literature is **particularly crucial for experienced researchers** who are familiar with the field. Clear explanation in the introduction helps them quickly engage with your work and see what’s interesting, else it blurs into all other superficially similar papers they’ve read and doesn’t seem worth the effort.
|
||||
|
||||
There are a few problems with novelty as it is traditionally thought of
|
||||
|
||||
* **Novelty is often overemphasized** \- it incentivises going for ambitious but shaky claims over simple and rigorous insights.
|
||||
* This can mean that if there's an existing paper that provides a preliminary but shaky case for a claim, going and doing it properly can seem less exciting, even though this is in some ways a more useful scientific contribution, as it establishes a confident foundation for others to build upon.
|
||||
* Another complex question arises when you have legitimate complaints about prior work, and your work superficially looks derivative, but this is because you identified a significant methodological flaw or bug.
|
||||
* I recommend being clear that you have criticism, but it's important to remain professional while explaining what was flawed and why this matters, and how your work resolves it, without critiquing the authors or their motivations.
|
||||
* There are various social norms that are kind of annoying, such as citing being obliged to cite the first instance of a concept (even if later iterations are much clearer) and referencing a ton of vaguely relevant work even if it adds nothing to the paper - people can get offended if not cited. But this doesn’t detract from all the ways that citations genuinely strengthen a paper
|
||||
|
||||
I personally prefer to just do work that is optimised for scientific value, and shoe-horn it into a peer review friendly lens at the end, if applicable. But I’m in a fortunate position here, and there are real career incentives around getting published.
|
||||
|
||||
Two example papers of mine where being clear about novelty was tricky:
|
||||
|
||||
* In [my Othello work](https://arxiv.org/abs/2309.00941), I built directly on Kenneth Lee's paper that showed an Othello plane model had a world model found with non-linear probes. My contribution was demonstrating that it could be found with linear probes, which was interesting for a bunch of reasons to do with the linear representation hypothesis, but I needed to be careful to *not* claim credit for anything Kenneth did
|
||||
* In [my refusal paper](https://arxiv.org/abs/2406.11717?), our key result was that refusal is mediated by a single direction. But it was *not* novel to find that *a* concept was linearly represented, the significant part was doing it for refusal: a particularly interesting concept.
|
||||
* Other work had loosely tried to do this for refusal, but had less compelling results, so we had to explain why our’s was better (much larger effect sizes, more models, downstream tasks, etc)
|
||||
* We also showed that we could now jailbreak the model by removing this direction from the weights - the novelty was less that we could jailbreak models (that's already known to be easy with finetuning), but that we could do it with interpretability tools, and so cheaply, one of the first practical applications of interpretability (even if, you know, not quite for safety...)
|
||||
|
||||
### Rigorous Supporting Evidence
|
||||
|
||||
**A paper is worth little unless it can** ***convince***[^c844hgv7nzl] **the reader of its key claims**. To do this, you need evidence. In machine learning, this typically means experiments. Below, I discuss how I think about what good experimental evidence looks like - see [my research process sequence](/api/post/hjMy4ZxS5ogA9cTYK) for more advice.
|
||||
|
||||
With claims the priority is being able to communicate the intuitions to everyone, but with experiments **the priority is being able to justify it in full technical detail to an engaged, skeptical reader**. You also want to explain what’s going on intuitively, to support your claims, but this is less key than actually having good, legitimate evidence.
|
||||
|
||||
A particularly important thing to get right is **extensive red-teaming**: you should spend a good amount of your time, both during the original research and now, red teaming your narrative. One of the main traps introduced by the framing of “find a great narrative” is the temptation to ignore inconvenient contradictory results - don’t let this happen to you. Tips:
|
||||
|
||||
* Assume you've made a mistake - what is that mistake? Assume there's a hole in your case that your evidence supports your grand narrative - where is that hole? Try to break it apart.
|
||||
* Try to get other researchers, especially more experienced ones, to weigh in.
|
||||
* Make sure to extensively discuss limitations. If you notice issues, design and perform new experiments to test for them. This is all the more important the more ambitious or surprising your claims are.
|
||||
* When I read a paper with a bold claim, I have a strong prior that it is false, and I am constantly looking for holes. If I identify one, and the authors have not checked whether that is a real flaw, I will generally move on.
|
||||
* However, if they have preempted me and provide sufficient evidence that I can be confident it's not a flaw, then those papers can be incredibly exciting and insightful.
|
||||
|
||||
What does good evidence look like?
|
||||
|
||||
* **Good experiments distinguish between hypotheses**: Often, you will have several plausible hypotheses for some phenomena. The point of an experiment is to have results that vary significantly depending on which is true (i.e. that provide Bayesian evidence) - if the results vary enough, and the experiment is sufficiently reliable, then one good experiment can falsify many hypotheses
|
||||
* Can you trust your results?:
|
||||
* **How reliable is my experiment?** Ask yourself: "How surprised would I be if it turned out to be complete bullshit due to a bug, error, noise, misunderstanding, etc.?" Investigate the most uncertain bits
|
||||
* **How noisy is my experiment?** If you ran similar experiments several times, how confident would you be that the results would be consistent? What is your sample size? What is your standard deviation? Are your results clearly distinguishable from noise? (There's a whole host of statistical theory here; your favourite LLM can probably effectively teach you about the basics.)
|
||||
* **Statistical rigour**: If you're doing some type of frequentist test[^n6o4myp4h8r], you probably shouldn't use p < .05 as a threshold. In stats heavy fields, like the social sciences, papers that report their central finding at .01 < p < .05, usually fail to replicate. If you're doing an exploratory approach, you should be skeptical of any result that isn't p < .001, as the number of possible hypotheses is vast.
|
||||
|
||||
* Prior work discussing replicability is very strict on this point: "One prior study of 103 replication attempts \[in psychology\] indeed found a 74% replication rate for findings reported at p ≤ .005 and a 28% replication rate for findings at .005 < p < .05 (Gordon et al., 2021)". There are also various statistical reasons why true findings usually won't produce .01 < p < .05.[^i21bv7c6wsm]
|
||||
|
||||
* This skepticism and sanity checking is especially key for particularly surprising or novel bits of evidence. Wherever possible, I will try to re-implement a key experiment from scratch or try to get at the same evidence via a somewhat different route, just to make sure that I'm not missing something crucial.
|
||||
* **Ablation studies**: When a paper introduces a complex new method, there are often several moving parts. For example, they may make changes A, B, and C to standard practice. If they then only evaluate the standard method or the method with all three changes, it's impossible to tell which changes are actually effective and necessary. It's good practice to remove one change at a time, observe its effect, and then repeat this process for each change.
|
||||
* **Unknown Unknowns:** How confident are you that there isn't some alternative explanation for your results that you're missing?
|
||||
* This is a gnarly one. You'll want to think hard about it, ideally ask other people for feedback and get their perspectives. However, ultimately, you may sometimes just need to move on after a reasonable effort and accept that you may have missed something.
|
||||
* Avoiding Misleading Evidence (Cherry-Picking and Post-Hoc Analysis):
|
||||
* **Was this cherry-picked?** Researchers can, accidentally or purposefully, produce evidence that looks more compelling than it actually is. One classic way is cherry-picking: presenting only the examples that look most compelling. This is particularly dangerous with qualitative evidence, like case studies.
|
||||
* While qualitative evidence can be extremely valuable, it’s important to note *how* cherry-picked it was. Ideally, provide randomly selected examples for context to give a fairer picture.
|
||||
* The main exception is if your claim is an existence proof. In this case, one example suffices, if it’s a trustworthy result.
|
||||
* **Track pre/post-hoc analysis.** It's important to clearly track which experimental results were obtained *before* versus *after* you formulated your claim. Post-hoc analysis (interpreting results after they're seen) is inherently less impressive than predictions confirmed by pre-specified experiments.
|
||||
* Be aware that even complex predictions suggested by a hypothesis can turn out to be correct for the wrong reasons
|
||||
* For example, in [a toy model of universality](https://arxiv.org/abs/2302.03025), I came up with the key representation theory-based algorithm I thought the network would follow before we got our key pieces of empirical evidence. I felt very confident. However, follow-up work found that a different explanation, which also involved representation theory, was what was actually occurring.
|
||||
* **Quality Over Quantity:** Try to prioritise having at least one really compelling and hard to deny experiment, over a bunch of mediocre ones.
|
||||
* If you do have many experiments, often some are more compelling than others. Highlight the ones that most strongly support your claims in the main text and consider moving others to an appendix or referencing them more briefly.
|
||||
* **Diverse Lines of Evidence Are Robust:** On the flip side, it can be far better to have several *qualitatively different* lines of evidence all pointing to the same conclusion, rather than many very similar experiments that all use similar methodologies and standards of proof.
|
||||
* Qualitatively different basically means “given the result of experiment 1, how well can I predict the result of experiment 2?”
|
||||
* This can justify putting effort into weak lines of evidence; for example, qualitative analysis of some data points can be useful supporting evidence of a quantitative study, even if insufficient to carry a paper independently, as they make it less likely that the summary statistics hid a subtle flaw.
|
||||
* Distilling Experiments:
|
||||
* Often, at the end of a project, you'll have run many experiments, some of which felt around the edges of your core claims. But by this stage, you likely have a much clearer idea of what the most promising kinds of evidence are. If practical (considering time and resources), consider going back to run a more conclusive, decisive experiment using what you now know.
|
||||
* This could also involve scaling up: using more models, larger sample sizes, sweeping hyperparameters more thoroughly, running on more diverse datasets, etc.
|
||||
* Baselines are Crucial:
|
||||
* A common mistake is for people to try to show a technique works by demonstrating it gets "decent" results, rather than showing it achieves *better* results than plausible alternatives that people might have used or are standard in the field.
|
||||
* Implicitly you’re supporting the weak claim “method X works at all” not “method X is actually worth using in practice”
|
||||
* Sadly this is especially prevalent in fields like mechanistic interpretability, where the comparative need for qualitative evidence can lead to neglecting more rigorous and systematic quantitative comparisons against strong baselines - the best papers have both qualitative and quantitative evidence.
|
||||
* **The subtlety of baselines:** It's not enough to just *have* them; you must strive to have the *strongest* possible baselines. Put meaningful effort into making them good. Often, a "competitor" method can seem weak but can significantly improve with proper hyperparameter tuning, prompt engineering, or appropriate scaffolding.
|
||||
* There's a natural bias to invest more effort in making one's "cool, shiny" new technique look good than in optimizing "boring" baselines. Resist this. Rigorous comparison to strong baselines is critical for good science and for genuinely persuading informed readers.
|
||||
* The Guiding Question for Evidence:
|
||||
* Ultimately, the question to ask about your evidence is: "Should this update a reader's beliefs about my claims?" not "Does this fit the stereotypical picture of a rigorous academic paper?" While the latter often correlates with the former, your primary goal is genuine persuasion through sound evidence.
|
||||
* Eg if reading several dataset examples by hand is genuinely strong evidence of your claim, just report that and justify why it’s great evidence!
|
||||
* **Reproducibility & Publishing code**: Rigour can be in the eye of the beholder: if readers cannot understand or verify it for themselves, it’s far harder to consider it rigorous. So your paper will be made substantially more useful by providing more detail about your exact methods.
|
||||
* A particularly useful approach is sharing your code. This enables others to build on your work and clarifies any ambiguities left in the paper (there will always be some). More broadly, it provides transparency into your exact process.
|
||||
* If you have time, you should:
|
||||
* Ensure the codebase runs on a fresh machine
|
||||
* Write a helpful README that includes links to key resources like model weights or datasets (which can be easily hosted on Hugging Face)
|
||||
* Create a Python notebook demonstrating how to run the key components
|
||||
|
||||
Tragically, the world is complicated, and there is often no single clear recipe to deal with all edge cases in research. These considerations are guidelines to help navigate that complexity
|
||||
|
||||
### Paper Structure Summary
|
||||
|
||||
How are these claims and experiments translated into a paper?
|
||||
|
||||
* **Abstract**: Motivate the paper, present your narrative and the impact: explain your key claims and why you believe they are true - be as concise and high-level as possible, while still getting the point across. Give enough information for a reader to understand the key takeaways of your paper and whether to read further or not - they often won’t read further, so it’s key that they still get the gist!
|
||||
* The reader is coming in from a cold-start, and may have no idea what your paper is about - you need to help them orient *fast*, and indicate what “genre” your paper fits into
|
||||
* The rest of your paper exists to support the abstract
|
||||
* **Introduction**: Basically an extended abstract that fleshes out your narrative - explain your key claims, motivate them, contextualise them in the *key* parts of the existing literature. Explain your key experiments and the results and why this supports your claims. Ensure the reader leaves understanding the narrative of the paper, and whether to read further or not - they often won’t. This is basically a self-contained paper summary, don’t worry about “spoiling” the paper or repetition - with a complex idea, you want to repeat it in varied ways so that it sticks.
|
||||
* The introduction sets the structure of the rest of the paper
|
||||
* **Main content**: This is where the real technical detail lives. Clearly and precisely explain background concepts and results, your precise claims, what exactly you did for your experiments (in *full* detail, using appendices if need be, relevant baselines, etc), the results, what they mean and their implications, etc.
|
||||
* This should be tightly planned to support the key claims, not sprawling and comprehensive. **For section and subsection you should have a clear answer for how it contributes to the narrative**, and would be damaging to remove.
|
||||
* I recommend first planning out clear opening and closing sentences of each paragraph: what does the paragraph show and how does it fit into the paper?
|
||||
* **Figures**: Figures and tables are a key medium for communicating experimental results. Diagrams are great for communicating key ideas and claims. Put a lot of effort into your figures.
|
||||
* Good captions are also crucial - you need to given context on what the figure shows, the nuance and intended interpretation, and key technical detail. Ideally the reader will understand everything from just the figure *and* just the caption, though this is ambitious
|
||||
* **Related work**: This is a mini literature review - I generally put this after the main content and don’t think it’s super important. Giving context on a few key similar papers and how your work differs is crucial, but typically done in the intro.
|
||||
* **Discussion** (/limitations/conclusion/etc): A place to put all the high-level reflections - limitations of your work, future work, key implications, etc. This is not essential, but is nice if you have something worthwhile to say. Acknowledging key limitations is very important, and papers that don’t do this are substantially weaker and less useful (in my opinion).
|
||||
* **Appendices**: Everything else - in the main paper you need to care about being concise, but here you can do whatever you want. Often you want to briefly discuss something in the main paper and move all technical detail to the appendix. Appendices are pretty low stakes and rarely read except by superfans, so don’t stress them too much.
|
||||
|
||||
Analysing My Grokking Work
|
||||
--------------------------
|
||||
|
||||
This is all pretty abstract. To concretise this, let’s look at [my grokking paper](https://arxiv.org/abs/2301.05217) through this lens. I’ve broken it down into claims, evidence, context and motivation, with commentary thrown in. This is somewhat stylised for pedagogical reasons, but hopefully useful!
|
||||
|
||||
* **Meta**: This was a challenging paper to write!
|
||||
* There was significant technical detail to our claims and evidence, largely unfamiliar to readers - mech interp was very new, and we did something weird and novel. We needed to communicate our claims (the algorithm) and our experimental evidence, *and* justify why the evidence was believable, since there were no standard methods to follow
|
||||
* A good diagram was critical to explaining the algorithm:
|
||||
|
||||
|
||||
|
||||
*Figure 1 from the paper, and lead image in the tweet thread*
|
||||
|
||||
* This was more like two papers - the reverse engineering, and the study of circuit formation, and we needed to compress both into the same page limit. Fortunately, they did fit a cohesive theme
|
||||
|
||||
**Structure**:
|
||||
|
||||
* **Claim**: We fully reverse engineered a tiny transformer trained on modular addition
|
||||
* **Meta**: This is a general claim, but about a specific model
|
||||
* **Context**: We needed to explain the entire notion of reverse-engineering a model from its weights, as readers may not have been familiar. The motivation for why this is interesting is pretty obvious, but the goal is easy to misunderstand
|
||||
* **Evidence**: We show this with several lines of evidence: activation and weight analysis, and causal interventions
|
||||
* **Claim**: This circuit forms gradually, well before the point of sudden grokking -> grokking is a gradual from memorisation followed by removing memorisation
|
||||
* **Context**: The entire notion of grokking as covered in prior work
|
||||
* **Meta**: This is critical context to understand my paper, but is also prior work - I need to explain enough detail for unfamiliar readers to follow, but without excessive repetition, and cite it for them to see further details.
|
||||
* **Motivation**: Grokking is a big deal and surprised many people. We show it’s fairly different from what people think.
|
||||
* **Evidence**: We show this by designing convincing progress measures to track the circuit, and show that they shift well before grokking
|
||||
|
||||
The Writing Process: Compress then Iteratively Expand
|
||||
-----------------------------------------------------
|
||||
|
||||
***Note**: Check out* [*my paper writing checklist*](https://docs.google.com/document/d/1AoF6bPJp-muWnsZLMmfcxo1fmAu1izUzZXDFHar-35o/edit?tab=t.0) *for a concrete to-do list of what I recommend doing here*
|
||||
|
||||
So, you have a list of claims and key experiments. Now, all you need to do is write the paper! I recommend an iterative process - start with a bullet point narrative, then a full bullet point outline, then flesh it out into prose, taking time to reflect at each stage. See above or the next section for more details on the actual structure of a paper, here I just try to convey the high-level strategy
|
||||
|
||||
A key challenge in paper writing is the **illusion of transparency** \- you have spent months steeped in the context of this research project. **You have tons of context, your reader does not**. You’ll need to help them understand exactly what you are doing, in the large space of all possible ML papers, and address all the misconceptions and possible misunderstandings, even though to you it all feels obvious. This is a difficult skill - wherever possible, get extensive feedback from others to address
|
||||
|
||||
**Spend far more time on early sections**: Realistically, tons of people read the title of your paper, many read the abstract, some read the introduction/skim the figures, and occasionally they read the whole thing. This means **you should spend about the same amount of time on each of: the abstract, the intro, the figures, and everything else**[^8swe1i37vnm]. (I’m only half joking)
|
||||
|
||||
### Compress
|
||||
|
||||
You should **start by compressing your work as much as possible**. Some tips:
|
||||
|
||||
* Verbally describe it to someone.
|
||||
* Bonus: Ask them what was most interesting, or to repeat it back to you
|
||||
* Plan out a talk.
|
||||
* Give your research notes to an LLM and ask it to summarize the key points.
|
||||
* After each, think about what's missing, what's extraneous, what’s inaccurate or misleading, and iterate.
|
||||
|
||||
This compression step is crucial because it forces you to identify:
|
||||
|
||||
1. The 1-3 concrete claims you believe to be true
|
||||
2. Why these claims matter (brief motivation)
|
||||
3. The crucial experimental evidence for each claim (ideally 1-3 key experiments per claim)
|
||||
|
||||
Next, critically evaluate this compressed version:
|
||||
|
||||
* Do your experiments genuinely support your claims?
|
||||
* Are there ways your experimental evidence could be flawed or misinterpreted?
|
||||
* Could the evidence be true but the claim false? How?
|
||||
|
||||
As part of this process, write down common misconceptions, limitations, or ways someone might over-update on your work.
|
||||
|
||||
### Iteratively Expand
|
||||
|
||||
Once you have a compressed list of bullet points that you are satisfied with, you should start iteratively expanding and developing them. After each step, stop, reflect, read through, and edit - rushing a step can lead to a lot of wasted time at the next step.
|
||||
|
||||
If you have a research supervisor/mentor, it is very valuable to get feedback at each stage - I find it way faster *and* easier to give feedback on a narrative or bullet point outline than being sent 8 pages of dense prose! Even if you don’t have a mentor, try to get feedback from *someone*[^amgw22xkub].
|
||||
|
||||
1. Start with the **compressed bullet point narrative** \- make sure you’re happy that this captures the narrative you want!
|
||||
2. Write a **bullet point outline of the introduction** \- **the north star here is to communicate what your claims are**, exactly, (including which parts are novel vs building on prior work), **why they matter**, and a high-level idea of **why they are true**
|
||||
1. This involves more detail, key citations to the literature, more detailed motivations, etc. Generally it won’t get too technical, but can involve explaining a few crucial concepts.
|
||||
2. Flow matters a lot here! Try to get feedback on how it feels to an unfamiliar reader, and how cohesive it feels
|
||||
3. Write a **bullet point outline of the full paper** \- covering the key experiments, results, methodology, background, limitations, etc
|
||||
1. **The north star is to convince a skeptical, engaged reader that your claims are true** \- give them enough information to understand your experiments and the results
|
||||
2. **A good outline is tight and minimal** \- every part of it should have a clear role in the overall narrative. If you don’t have a good answer to “what goes wrong if I cut this”, you should cut it.
|
||||
3. **Good figures are crucial** to communicate results - plan these out, but leave making them to step 4
|
||||
4. This can include writing the related work, or you can leave that to the end.
|
||||
4. **Results**: Collect key experimental results and make first draft figures to show. Does this convincingly support your narrative? What’s missing? Which parts and complex and need more exposition, vs standard/unsurprising and can be sped through?
|
||||
1. You can’t always get this done in advance, but it’s *much* better if you do - more time to refine, iterate, etc.
|
||||
5. **First draft**: Flesh this out into prose and full technical detail
|
||||
1. If you have writer’s block, try giving an LLM your outline, some relevant papers, and asking for a first draft. *Do not* just copy this into your paper, but I find that sometimes LLMs have good ideas, and that frustration with poor quality LLM write-ups can be a great way to break through writer’s block.
|
||||
6. **Edit it**: Repeatedly pass over your first draft (and get feedback), clean it up, polish it, make the narrative as tight and clear as possible, cut out extraneous fluff, make the figures glorious, etc
|
||||
1. This is worth spending a *lot* of time on, it can make a big difference!
|
||||
|
||||
The Anatomy of a Paper
|
||||
----------------------
|
||||
|
||||
OK, so what actually goes into a paper? What are the key components you’ll need to write, and what is the point of each?
|
||||
|
||||
### Abstract
|
||||
|
||||
*Check out the annotated abstract earlier for a concrete breakdown*
|
||||
|
||||
An abstract should **give a cold-start reader a sense of what the paper is about** \- what sub-field, what type of paper, what key motivating questions, etc. This is a key manifestation of the illusion of transparency: you know exactly what your project is about but to your reader there is a large space of possibilities, and without any context may have completely incorrect priors and wildly misinterpret.
|
||||
|
||||
People will often leave your abstract then move on, unless strongly compelled - it’s a big deal to get right, and deserves high polish
|
||||
|
||||
A common approach is:
|
||||
|
||||
* First sentence: Something uncontroversially true that clearly states which part of ML you're focused on (e.g., "Thinking models have recently become state-of-the-art across many reasoning tasks.")
|
||||
* Second sentence: Something that makes clear there's a need, something unknown, or a problem for your paper to solve (e.g., "The transition to reasoning models raises novel challenges for interpretability.") - this should convey (some of) the motivation
|
||||
|
||||
Now the reader is situated, you need to *concisely* communicate your claims. Again, illusion of transparency - they often won’t know your techniques, the work you’re building on, key ideas, etc. **Abstracts should be as accessible as possible** \- use simple language as much as you can
|
||||
|
||||
* Sentence 3: State the crucial contribution of this paper and why it is exciting - you’ll need to lose nuance, this is OK.
|
||||
* Optional: Sentence 4 should provide clarifying details on that claim, such as its meaning and how evidence could be provided if not obvious.
|
||||
* Include key definitions for any necessary jargon, though jargon should be avoided if possible, unless it’s standard in the field and useful to contextualise the paper within the field.
|
||||
* Each of the next few sentences should focus on either key experimental evidence or additional important claims. These can sometimes overlap, where a specific claim being true also supports the main claim.
|
||||
* Try to have 1 sentence per idea - this forces you to be concise, without getting overwhelming.
|
||||
* If possible, include a concrete metric or result in any of the above that gives readers a sense that your results are real and substantial.
|
||||
* This can look like folding in key evidence of a claim into the sentence introducing the claim.
|
||||
|
||||
Finally, close with motivation:
|
||||
|
||||
* Final 1-2 sentences: Wrap up by reminding readers why the paper matters/is a big deal, its implications, and how it fits into the broader context.
|
||||
* This is also a good place to clearly state your standard of evidence, whether your work is:
|
||||
* A preliminary step towards…
|
||||
* Shows that method X should be used in practice
|
||||
* Shows that practitioners should take care when using method Y
|
||||
* Establishes best practices for Z
|
||||
* Provides compelling evidence that…
|
||||
|
||||
### Introduction
|
||||
|
||||
The introduction is broadly similar to the abstract but more extended and in-depth. I proceed in roughly this order:
|
||||
|
||||
* Paragraph 1: **Context** \- What topic are we studying, what is the key motivating question, and why does it matter?
|
||||
* Optionally: 1 sentence on how our contribution answers it
|
||||
* It’s good to liberally cite papers here to establish things like ‘this is a real field’, ‘this problem matters’, ‘people are interested in it and have tried (and failed) to solve it/have solved variants’
|
||||
* Paragraph 2: **Technical background** \- what do we know about this problem? What are the established techniques our paper rests on? Etc
|
||||
* It’s good to cite liberally here to establish that what you’re using are standard methods and concepts, and to give the reader more context.
|
||||
* Here and in paragraph 1 you want to better situate your problem in the broader strategic picture of the field. Why does this matter? What other work has been done here, and why is it inadequate?
|
||||
* Paragraph 3: **Key contribution** \- What exactly is our main claim? Add key nuance, detail, context, etc.
|
||||
* Paragraph 3.5[^38hw5w3ydeg]: **Our case** \- summarise the most critical evidence we provide that our main claim is true
|
||||
|
||||
* \[Optional\]: More paragraphs for a second or third claim and the key case
|
||||
* Paragraph 4: **Impact** \- What should you take away from this paper? What are the implications, why is it a big deal, who should take different actions as a result of the results, etc. This may be emphasising practical utility, pushing forwards basic science, correcting common misconceptions, etc.
|
||||
* **Contributions**: End with a bullet point list of concise descriptions of your key claims, ideally with concise descriptions of key evidence
|
||||
* You want something a reader can look at and decide if they’re impressed/interested
|
||||
|
||||
Note: Citing here isn't about performatively covering all the relevant papers[^qfem2sure9]. It's about providing the context a reader needs to understand why your work is interesting and how it's limited. I try to have at least one citation for each step in an important argument, eg why
|
||||
|
||||
The introduction is where you have room to define key terms and concepts required to understand your claims, especially if they're somewhat technical.
|
||||
|
||||
It's often good to explicitly end with a bullet-point list of your contributions, which are basically just the concise claims you believe to be true, potentially with brief references to the supporting evidence.
|
||||
|
||||
### Figures
|
||||
|
||||
Figures are incredibly important. Having clear graphs can be the difference between a very clear and easy-to-read paper and an incomprehensible mess.
|
||||
|
||||
To create a good figure:
|
||||
|
||||
* Ask yourself, "What exactly is the information I would like someone to take away from this?" It's not just about finding the list of numbers output by your experiments and shoving them into some standard plotting software, you want to carefully choose a visualisation that emphasises the desired information and takeaway.
|
||||
* Ask yourself, "Why does this experiment tie back to my core claims? How would I like the reader to interpret these results? Which parts do I want to draw their attention to?"
|
||||
* Consider annotating a graph or, if there's one particularly important line, emphasising it
|
||||
* E.g. make it dark while the others are light and low opacity, or all the other ones of the same color.
|
||||
* Include standard elements like axis titles, a clear caption that explains what the figure is, how to interpret it, or at least where in the text they should look to understand what's going on.
|
||||
* Make sure the axis title and ticks are large enough to read, and have a good clear legend.
|
||||
* Often you can compress a fair amount of information into one graph - for example, using different sizes and shapes of markers on a scatterplot, different colors, etc.
|
||||
* For heatmaps, if your data is positive and starts at zero, use a color scale that is white at zero and dark at the max (in plotly, "blues" is good). If your data is positive and negative with zero as a meaningful neutral point, use a color scale where zero is white (in plotly, "RdBu" is good).
|
||||
* Avoid having reds and greens conveying key information, 4% of people are red-green colourblind
|
||||
|
||||
It can work well to combine several key graphs into one figure and make it your figure 1. E.g.:
|
||||
|
||||
[Language models represent space and time](https://arxiv.org/pdf/2310.02207):
|
||||
|
||||
|
||||
|
||||
[Not all features are one-dimensionally linear](https://openreview.net/pdf?id=d63a4AM4hb):
|
||||
|
||||
|
||||
|
||||
Another kind of figure is an explanatory diagram rather than a graph. This can be a high-effort but very effective figure one, that gives people a sense of roughly what is happening in the paper. This should be something that would catch people's eye if you put it as the first image in a tweet thread about your paper. Some diagrams I liked (intentionally at several different levels of effortful):
|
||||
|
||||
[Emergent Misalignment](https://www.emergent-misalignment.com/):
|
||||
|
||||
|
||||
|
||||
[On the Biology of a Large Language Model](https://transformer-circuits.pub/2025/attribution-graphs/biology.html):
|
||||
|
||||
|
||||
|
||||
[CoT in the wild is not always faithful:](https://arxiv.org/pdf/2503.08679)
|
||||
|
||||
|
||||
|
||||
[Refusal is mediated by a single direction](https://arxiv.org/abs/2406.11717?):
|
||||
|
||||
|
||||
|
||||
[My grokking modular addition work](https://arxiv.org/abs/2301.05217):
|
||||
|
||||
|
||||
|
||||
### **Main Body (Background, Methods and Results)**
|
||||
|
||||
Most of the actual paper, by word count, should be about **communicating your experiments and results in precise technical detail**. To do good science, it is important that researchers can understand exactly what you did and what you observed, so they can draw their own conclusions rather than needing to take things on faith. For example, in interpretability, there are ways that a method can give completely useless answers if misapplied, so it’s crucial that I know if a paper did that, even though the detail might seem totally unimportant to the authors!
|
||||
|
||||
Ideally, you want to communicate the information at several different layers of abstraction. It's your job to ensure that readers understand:
|
||||
|
||||
* The key background context required to disambiguate and understand your work - key terms, techniques, etc[^camk72o22gu][^dj77nx985wb]
|
||||
|
||||
* What your results are and how to interpret those results and their significance
|
||||
* What you actually did for your experiments
|
||||
* Why this was reasonable/well motivated/relevant to your claims
|
||||
* The specifics of various technical choices you made, and their implications for how to understand the results.
|
||||
|
||||
For structure, here’s a good default:
|
||||
|
||||
* **Background**: to explain the relevant context and terms - in particular, please define terminology and crucial techniques!
|
||||
* If pressed for space, you can put a glossary of key terms/definitions as an appendix, I always appreciate this
|
||||
* If you’re defining something new for this paper, put this in a section which is clearly *not* about reviewing known things (a new section or separate subsection)
|
||||
* **Methods**: Explain the methods you used and why they are relevant to the problem
|
||||
* **Results**: Specify exactly how the methods are applied as experiments, and what the results are
|
||||
* If you have a bunch of experiments using fairly similar methods, put each in a different subsection
|
||||
|
||||
If the experiments for each claim are more boutique, or if there are several claims with different styles of evidence, then I try to give each type of evidence its own section while explaining how it ties back to the overarching theme, rather than a methods -> results section. People will forget about the first method before they see its results.
|
||||
|
||||
### Discussion
|
||||
|
||||
Explaining the limitations of your work is a crucial part of scientific good practice. The goal of a paper is to contribute to our body of knowledge. Readers must understand the limitations of the evidence you provide to have a calibrated sense of what knowledge they have learned. And it's important that you put a good faith effort into documenting limitations because you know far more about your work than the readers, so they may miss things.
|
||||
|
||||
There is a common mistake of trying to make your work sound maximally exciting. Generally, the people whose opinions you most care about are competent researchers who can see through this kind of thing. And I generally have a much higher opinion of a piece of work if it clearly acknowledges its limitations up front. I’m not sure if this makes it easier or harder to get published.
|
||||
|
||||
This is also the place to discuss broader implications and general takeaways, future work you’d be excited about, reflections, etc.
|
||||
|
||||
Some people have conclusions too. Personally, I think conclusions are often kind of useless; the introduction should have explained this well. You can skip it
|
||||
|
||||
### Related Work
|
||||
|
||||
Generally, related work is often treated like a bit of an annoyance and afterthought. The feeling that you need to cite lots of things that aren't actually relevant can be annoying, but sometimes there is very important work that has done similar things to you, and a reader might have seen that and wonder why your paper is interesting.
|
||||
|
||||
It's very important to clearly explain why what you did is different or, if what you did is not very different, either acknowledge this ("that was parallel work") or explain why your work is still slightly interesting in this context, or how you fixed a mistake in prior work (stated politely).
|
||||
|
||||
But that said, there’s a lot of annoying norms here and related works often add little value IMO - needing to cite a lot so you look like you’ve put in enough effort, covering minor or obscure things that aren’t particularly relevant, citing low quality works to be polite, making sure to cite the first instance of each thing, etc. Contextualising in the literature is important, but ideally I’ve already covered it in the introduction.
|
||||
|
||||
Related work is often put as the second section of the paper. Personally, I generally prefer it to be the penultimate section. I think related work should only be upfront if it plays an important role in motivating the paper - if your paper is very heavily tied to the surrounding literature, plugging a gap, correcting a mistake, or unlocking a new capability that would enhance various bits of prior work.
|
||||
|
||||
### Appendices
|
||||
|
||||
Appendices are weird. They're basically the place you put everything that doesn't fit into the main paper. One way to think about it is that you're actually writing a much longer than nine-page paper - the main body *and* the appendices - but you've chosen a highlights reel for the first nine pages where you put all the absolutely key information. You place all the less crucial information in the appendices for readers to pick and choose from as they see fit.
|
||||
|
||||
In general, the crucial scarce resource you must manage is the reader's time and attention. The main body should be aggressively prioritized to make the most of this, be engaging, and communicate the most important pieces of information. But if you have a lot more to say than you can fit in there, then that's what appendices are for. A truly interested reader can go and take a look, though most won't.
|
||||
|
||||
Generally, appendices are held to a notably lower standard than the main body and will be read far less, so you should not feel obliged to put in meaningful effort polishing them. This is the standard solution to the dilemma when you want to include full technical detail but have done some fairly complex and convoluted work that just won't realistically fit.
|
||||
|
||||
Common Pitfalls and How to Avoid Them
|
||||
-------------------------------------
|
||||
|
||||
### Obsessing Over Publishability
|
||||
|
||||
Peer review is notoriously terrible for seeking truth. Reviewers often have biases, like favoring work that feels novel and shiny and exciting, or that doesn't feel weird or too new, or that doesn’t seriously challenge their existing beliefs. This has been shown in [rigorous RCTs](https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/), where NeurIPS 2021 gave some papers two sets of reviewers and compared their decisions. The results… aren’t great:
|
||||
|
||||
|
||||
|
||||
I personally think that, at least in safety, doing good work that people respect matters more than getting into conferences, though both are nice. I’ve generally had fairly good results with just trying to write high-integrity work that explains why I believe it is interesting, and just trying to do good science and the work that I think is highest impact, even if it doesn't fit the academic mold.
|
||||
|
||||
But it’s pretty plausible to me that many of the people reading this are not in such a fortunate position, and that getting first author papers into top conferences would be a meaningful career boost, especially your first 1-2 papers. The strategy I generally recommend for my mentees is to spend most of the project doing the best scientific work they can. Then, as we approach the end of the project, we figure out how to wrap it up in a maximally conference-friendly package while writing and submitting it. If we did anything that made the work noticeably worse, we can undo it before uploading to Arxiv.
|
||||
|
||||
### Unnecessary Complexity and Verbosity
|
||||
|
||||
Papers are seen as prestigious, formal, and highly intellectual artifacts. As a result, there's a tendency towards verbosity or trying to make things sound more complex and fancy than they actually are, so they *feel* impressive. I think this is a highly ineffective strategy. If I don’t understand a paper, I generally ignore it and move on, or assume it’s BS in the absence of strong evidence to the contrary. Often, the best papers just take some very simple techniques and apply them carefully and well. There’s a real elegance to being simple and effective.
|
||||
|
||||
People need to understand a paper in order to appreciate it and build on it and think it is interesting (except for superficial Twitter clickbait). Generally, you want to be precise, but within the constraint of being precise, be as simple and accessible as possible. Try to use plain language and minimize jargon except where the jargon is needed to precisely convey your meaning. You get points for quality technical insights, not for sounding fancy. Verbosity and overly complex language and jargon is actively detrimental to your paper’s prospects, IMO.
|
||||
|
||||
### Not Prioritizing the Writing Process
|
||||
|
||||
People often do not prioritize writing. They treat it like an annoying afterthought and do all the fun bits like running experiments, and leave it to the last minute. This is a mistake. Again, your work only matters if people read and understand it. Writing quality majorly affects clarity and engagement. Writing is absolutely crucial and is a major multiplier on the impact of your work.
|
||||
|
||||
I typically recommend that people switch from [understanding mode to distillation](/api/post/hjMy4ZxS5ogA9cTYK) and paper writing a month before a conference deadline, if at all possible. You should want to spend a lot of your time iterating on a write-up, getting feedback, trying to make it clearer, thinking about weaknesses, etc.
|
||||
|
||||
Tacit Knowledge and Beyond
|
||||
--------------------------
|
||||
|
||||
One irritation I have about the standard paper structure is that it heavily incentivizes being rigorous and maximally objective and defensible. Obviously, there are significant advantages to this, but I think that often a lot of the most valuable insights from a research project come in the form of tacit knowledge.
|
||||
|
||||
This might be:
|
||||
|
||||
* This was hard and here are the steps we had to follow to get it to work.
|
||||
* Here are some ways we noticed our experiments catching fire and what we did to fix them.
|
||||
* Here's my fuzzy intuition of what's going on in the big picture - I can't fully defend it, but I'm reasonably confident this is true after several months of screwing around in this domain.
|
||||
* Here’s something I misunderstood for months before it suddenly clicked
|
||||
* Here’s a common misconception in this domain, or way people often misunderstand or overreact to our results
|
||||
* Here’s my advice to anyone replicating this work, especially how to find hyper-parameters and deal with the fiddly bits
|
||||
* Fleshing out a plan for future work directions you find particularly exciting.
|
||||
|
||||
I think this is really important, and I find it a real shame that this is often just discarded. I am personally a big fan of putting this kind of stuff as appendix A or as an accompanying blog post, where you can take as many liberties as you like.
|
||||
|
||||
Conclusion
|
||||
----------
|
||||
|
||||
Your research will only matter if people read it, understand it, engage with it, and ideally believe it. This means that good paper writing is a crucial skill, but often neglected.
|
||||
|
||||
The core process should be to find the concise claims you believe to be true, the strongest experimental evidence that you believe builds a robust case for these claims, and use this to craft a coherent narrative. Then flesh this out into a bullet point outline of the overall post, reflect on it, and ideally get feedback, and iteratively expand.
|
||||
|
||||
Again, this is a highly opinionated post about how I personally think about the process and philosophy of paper writing. I'm sure many researchers will strongly disagree with me on many important points, and the correct approach will vary significantly by field and norms.
|
||||
|
||||
[^wwk8u16jdjf]: Note: I am in no way claiming that I follow this advice, especially in blog posts - this is my attempt to paint a platonic ideal, and advise on how to be a better person than I. Personally, I find actually writing academic papers pretty frustrating and much prefer blog posts
|
||||
|
||||
[^5kbv6xekx6j]: I think writing great papers certainly helps, and if you’re new I recommend just trying to write the best paper you can, but there’s still a lot of depressingly perverse incentives from ML peer review
|
||||
|
||||
[^xjcznpnz657]: I mention this purely for completeness. I have never seen a convincing guarantee in deep learning, neural networks are far too squishy
|
||||
|
||||
[^29hhedam5nx]: OpenAI’s is also good, but Google’s is free!
|
||||
|
||||
[^c844hgv7nzl]: Well, at least “provide enough evidence to somewhat update their beliefs”, convince is a fairly high bar
|
||||
|
||||
[^n6o4myp4h8r]: Which, admittedly, is fairly rare in ML as far as I’m aware
|
||||
|
||||
[^i21bv7c6wsm]: Thanks to Paul Bogdan for these points on p-values
|
||||
|
||||
[^8swe1i37vnm]: Also the title, though I haven’t figured out how to productively spend 20% of my time on that yet…
|
||||
|
||||
[^amgw22xkub]: Paper swaps are a great way to get feedback - find someone else also working on a paper and offer to give each other feedback. Even if you have less time before the paper deadline, this tends to be a mutually beneficial trade.
|
||||
|
||||
[^38hw5w3ydeg]: This can be its own paragraph, or part of paragraph 3
|
||||
|
||||
[^qfem2sure9]: Save that for the related work section…
|
||||
|
||||
[^camk72o22gu]: If something is super widespread knowledge, no need to cover it, but err towards defining things. E.g. I wouldn’t bother defining the transformer architecture or an LLM, but I would define a sparse autoencoder or steering vector
|
||||
|
||||
[^dj77nx985wb]: If this is too long, you can move most of it to an appendix
|
||||
|
||||
@@ -1,15 +1,957 @@
|
||||
# How to Become a Mechanistic Interpretability Researcher — Neel Nanda
|
||||
|
||||
Source: https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher (also on LessWrong, same post id). Verbatim excerpts cached for the research-taste section.
|
||||
Source: https://www.alignmentforum.org/posts/jP9KDyMkchuv6tHwm/how-to-become-a-mechanistic-interpretability-researcher (also on LessWrong, same post id)
|
||||
Author: Neel Nanda
|
||||
Date: 2nd Sep 2025 (post marked "Last updated Sept 2 2025")
|
||||
Fetch-status: full post text, fetched 2026-08-15 from the LessWrong markdown API (`/api/post/jP9KDyMkchuv6tHwm`), comments and site navigation stripped. (CLAUDE agent)
|
||||
Use: research-taste section evidence; truth-seeking, reading your data, ablations, and skeptic-proofing.
|
||||
|
||||
---
|
||||
## Full post
|
||||
|
||||
> **Skepticism/Truth-seeking:** The default state of the world is that your research is false, because doing research is hard. Your north star should always be to find *true* insights
|
||||
*Last updated Sept 2 2025*
|
||||
|
||||
> **Excitement is evidence of bullshit**: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it's even more likely to be false than normal!
|
||||
*Note - if you want to pursue a career in this kind of research, apply to my MATS stream! Apps aren't currently open,* [*sign up here to be notified*](https://neelnanda.io/mats-notifications)
|
||||
|
||||
> **Read your data**: A fantastic use of time, especially during the exploration phase, is just actually reading the data you're working with, or model chains of thought and responses. [...] Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad.
|
||||
TL;DR
|
||||
-----
|
||||
|
||||
> A useful exercise is imagining you're talking to a really obnoxious skeptic who keeps complaining that they don't believe you and coming up with arguments for why your thing is wrong. What could you do such that they don't have a leg to stand on?
|
||||
* This post is about the mindset and process I recommend if you want to *do* mechanistic interpretability research. I aim to give a clear sense of direction, so give opinionated advice and concrete recommendations.
|
||||
* Mech interp is high-leverage, impactful, and learnable on your own with short feedback loops and modest compute.
|
||||
* **Learn the minimum viable basics, then do research.** Mech interp is an empirical science
|
||||
* Three stages:
|
||||
* [**Learn the ropes**](/api/home#Stage_1__Learning_the_Ropes) **(≤1 month)** learn the essentials, go breadth-first;
|
||||
* [**Learn with research mini-projects**](/api/home#Stage_2__Practicing_Research_with_Mini_Projects) practice basic research skills with 1-5 day mini projects, focus on fast feedback loop skills;
|
||||
* [**Work up to full projects**](/api/home#Stage_3__Working_Up_To_Full_Research_Projects), do 1-2 week research sprints, continue the best ones. Explore deeper skills and the mindset of a great researcher.
|
||||
* [**Stage 1:**](/api/home#Stage_1__Learning_the_Ropes) **Learning the Ropes**
|
||||
* **Breadth over depth; get a good baseline not perfection**
|
||||
* **Learn the basics**: [Code a transformer from scratch](/api/home#Machine_Learning___Transformer_Basics), [key mech interp techniques](/api/home#Mechanistic_Interpretability_Techniques), [the landscape of the field](/api/home#Using_LLMs_for_Learning), [linear algebra intuitions](/api/home#Machine_Learning___Transformer_Basics), [how to write mech interp code](/api/home#Mechanistic_Interpretability_Coding___Tooling) ([ARENA is your friend](https://arena-chapter1-transformer-interp.streamlit.app/))
|
||||
* **Get your hands dirty**: Do *not* just read things. Mech interp is a fundamentally empirical science
|
||||
* **Move on after a month**. Don’t expect to feel “done” or to have covered *all* of the ropes, learn more when needed. You won’t stumble across great research insights without starting to do something real
|
||||
* [**Use LLMs extensively**](/api/home#Using_LLMs_for_Learning) \- they’re not perfect, but are better at mech interp than you right now! They’re a crucial learning tool (when used right!)
|
||||
* [**Unpacking the research process**](/api/home#The_Big_Picture__Learning_the_Craft_of_Research):
|
||||
* [Many skills](/api/home#Unpacking_the_Research_Process), categorise them by the feedback loops.
|
||||
* Fast skills (minutes-hours) like write/run/debug experiments
|
||||
* Slow (weeks) like how to prioritise and when to pivot
|
||||
* Very slow (months) like generating good research ideas
|
||||
* **Do** ***not*** **try to learn all skills at once**. Focus on fast/medium skills first, then slowly expand
|
||||
* [4 phases of research](/api/post/hjMy4ZxS5ogA9cTYK): finding an idea (**ideation**) -\> building intuition and hunches (**exploration**) -\> testing hypotheses (**understanding**) -\> refining and writing up (**distillation**)
|
||||
* [**Stage 2:**](/api/home#Stage_2__Practicing_Research_with_Mini_Projects) **Mini projects** (1-5 days each for 2-4 weeks)
|
||||
* [Exploration mindset](/api/home#Practicing_Exploration): **Maximise information gain per unit time**, learn how to get unstuck. You don't need a plan, so long as you're learning
|
||||
* [Understanding mindset](/api/home#Practicing_Understanding): **Every research result is false until proven otherwise**. The more exciting a result is, the more likely it is to be false. Be your own greatest critic
|
||||
* Idea quality (ideation) and write-ups (distillation) aren't the priority yet; **taste and prioritization are learned by doing things**.
|
||||
* Having good research ideas takes forever to learn, **to choose early projects, cheat**! [Pick well scoped projects](/api/home#Choose_A_Project), eg extending a paper (ideas)
|
||||
* [**Use LLMs extensively**](/api/home#Using_LLMs_for_Research_Code) \- they should speed up your research/coding a *lot* (if you know how to use them properly!)
|
||||
* [**Stage 3:**](/api/home#Stage_3__Working_Up_To_Full_Research_Projects) **Towards full projects**
|
||||
* **Work in 1-2 week sprints**, post-mortem after each, pivot to another project unless it's going *great*
|
||||
* [**Slower skills**](/api/home#Deepening_Your_Skills)**and** [**key mindsets**](/api/home#Key_Research_Mindsets): careful skepticism, awareness of the literature, prioritization, high productivity
|
||||
* [**Do good science**](/api/home#Doing_Good_Science), not flashy science \- be honest about limitations, give proof you're not cherry picking, read your data, do the simple things that work, use real baselines.
|
||||
* [**Write-up**](/api/home#Write_up_your_work_) **your work**! Distill it into a narrative, then iteratively expand it to a write-up
|
||||
* **Good public work is** [**your best credential**](/api/home#Why_aim_for_public_output_) \- for careers, PhDs, finding mentors, etc
|
||||
* **Writing is not an afterthought** \- make time for it. [The reader will understand less than you think](/api/home#Common_mistakes)
|
||||
* **Practice** [**generating research ideas**](/api/home#Practicing_Ideation). If possible, try to imitation learn [a mentor's research taste.](/api/home#Research_Taste_Exercises)
|
||||
* [Avoid fads](/api/home#Avoiding_Fads), and think about [what’s new and exciting in mech interp](/api/home#What_s_New_In_Mech_Interp_)
|
||||
* [**Proactively reach out to mentors**](/api/home#Advice_on_finding_a_mentor) Everything is *much* easier with a good mentor. Cold email, apply for mentoring programs, etc.
|
||||
* Reach out to researchers who'll have time, not the most famous
|
||||
* **Careers:** If you want to work in the field, apply for things! [Jobs](/api/home#Where_to_apply), [mentoring programs](/api/home#Mentoring_programs), [funding](/api/home#Applying_for_grants), [academic labs](/api/home#Relevant_Academic_Labs).
|
||||
* Bonus thoughts: [what do hiring managers look for](/api/home#What_do_hiring_managers_look_for), [what does a good research mentor actually do](/api/home#So_what_does_a_research_mentor_actually_do_), and [should you do a PhD](/api/home#Should_you_do_a_PhD_)?
|
||||
* I also give various thoughts on how I'm thinking about the field nowadays, and what I’ve changed my mind about. I separate these from the practical advice, so you can take it or leave it.
|
||||
* NEW (Dec 1 25): See related posts on [the much more pragmatic approach I am taking to research](https://neelnanda.io/vision) and [research directions we think are promising](https://neelnanda.io/agenda)
|
||||
* Covering: [how I currently define the field](/api/home#Interlude__What_is_mech_interp_), why I'm [pessimistic on ambitious reverse engineering, and excited about more pragmatic approaches](/api/home#A_Pragmatic_Vision_for_Mech_Interp), [what recent work I am excited about](/api/home#What_s_New_In_Mech_Interp_) and recommend building on.
|
||||
|
||||
> **Do ablations on your fancy method**: It's easy for people to have a fancy method with lots of moving parts, when many actually are unnecessary. You should always try removing one part and see if the method breaks. Do this for each part.
|
||||
Introduction
|
||||
------------
|
||||
|
||||
Mechanistic interpretability (mech interp) is, in my incredibly biased opinion, one of the most exciting research areas out there. We have these incredibly complex AI models that we don't understand, yet there are tantalizing signs of real structure inside them. Even partial understanding of this structure opens up a world of possibilities, yet is neglected by 99% of machine learning researchers. There’s so much to do!
|
||||
|
||||
I think mech interp is an unusually easy field to learn about on your own: there’s a lot of educational materials, you don’t need too much compute, and there’s short feedback loops. But if you're new, it can feel pretty intimidating to get started. This is my updated guide on how to skill up, get involved, reach the point where you can do actual research, and some advice on how to go from there to a career/academic role in the field.
|
||||
|
||||
This guide is deliberately highly opinionated. My goal is to convey a productive mindset and concrete steps that I think will work well, and give a sense of direction, rather than trying to give a fully broad overview or perfect advice. (And many of the links are to my own work because that's what I know best. Sorry!)
|
||||
|
||||
### High-Level Framing
|
||||
|
||||
My core philosophy for getting into mech interp is this: learn the absolute minimal basics as quickly as possible, and then immediately transition to learning by doing research.
|
||||
|
||||
The goal is not to read every paper before you touch research. When doing research you'll notice gaps and go back to learn more. But being grounded in a project will give you vastly more direction to guide your learning, and contextualise why anything you’re learning actually matters. You just want enough grounding to start a project with some understanding of what you’re doing.
|
||||
|
||||
Don't stress about the research quality at first, or having the perfect project idea. Key skills, like [research taste](/api/sequence/5GT3yoYM9gRmMEKqL/post/Ldrss6o3tiKT6NdMm) and the ability to prioritize, take time to develop. Gaining experience—even messy experience—will teach you the basics like how to run and interpret experiments, which in turn help you learn the high-level skills.
|
||||
|
||||
I break this down into three stages:
|
||||
|
||||
1. [**Learning the ropes**](/api/home#Stage_1__Learning_the_Ropes), where you work through the basics breadth first, and after at most a month, move on to stage 2
|
||||
2. [**Practicing research with mini-projects**](/api/home#Stage_2__Practicing_Research_with_Mini_Projects). Work on throwaway, 1-5 day research projects. Focus on practicing the basic research skills with the fastest feedback loops, don’t stress about having the best ideas, or writing them up. After 2-4 weeks, move on to stage 3
|
||||
3. [**Work up to full-projects**](/api/home#Stage_3__Working_Up_To_Full_Research_Projects): work in 1-2 week sprints. After each, do a post-mortem and pivot to something else, *unless* it was going great and has momentum. Eventually, you should end up working on something longer-term. Start thinking about the deeper skills and research mindsets, practice having good ideas, and prioritize making good public write-ups of sprints that went well
|
||||
|
||||
Stage 1: Learning the Ropes
|
||||
---------------------------
|
||||
|
||||
Your goal here is learning the basics: how to write experiments with a mech interp library, understanding the key concepts, getting the lay of the land.
|
||||
|
||||
Your aim is learning enough that the rest of your learning can be done via doing research, *not* finishing learning. Prioritize ruthlessly. **After max 1 month**[^nifk1wb1jum]**, move on to stage 2**. I’ve flagged which parts of this I think are essential, vs just nice to have.
|
||||
|
||||
**Do not just read papers** \- a common mistake among academic types is to spend months reading as many papers as they can get their hands on before writing code. Don’t do it. Mech interp is an empirical science, getting your hands dirty gives key context for your learning. Intersperse reading papers with doing coding tutorials or small research explorations. See [my research walkthroughs](https://www.youtube.com/playlist?list=PL7m7hLIqA0hr4dVOgjNwP2zjQGVHKeB7T) for an idea of what tiny exploratory projects can look like.
|
||||
|
||||
LLMs are a key tool - see [the section below](/api/home#h.ab01gbohcxm5) for advice on using them well
|
||||
|
||||
### **Machine Learning & Transformer Basics**
|
||||
|
||||
*Assuming you already know basic Python and introductory ML concepts.*
|
||||
|
||||
* Maths:
|
||||
* **Linear Algebra is King (Essential):** You need to think in vectors and matrices fluently. This is by far the highest value set of generic math you should learn to do mech interp or ML research.
|
||||
* *Resource:* 3Blue1Brown's[Essence of Linear Algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab).
|
||||
* **Highly recommended**: Put [A Mathematical Framework For Transformer Circuits](https://transformer-circuits.pub/2021/framework/index.html) in the context window and have the LLM generate exercises to test your intuitions about transformer internals.
|
||||
* LLMs are great for checking whether linear algebra actually clicks. Try summarizing what you've learned and the links between different concepts and ask an LLM whether you are correct. For example:
|
||||
* Ensure you understand SVD and why it works
|
||||
* What does changing basis mean and why does it matter
|
||||
* Key ways a low rank and full rank matrix differ
|
||||
* **Other Bits:** Basic probability, info theory, optimization, vector calculus.
|
||||
* Use an LLM tutor to quiz your understanding on the parts most relevant to transformers
|
||||
* Generally don’t bother learning other areas of maths (unless doing it for fun!)
|
||||
* Practical ML with PyTorch: (Essential)
|
||||
* Code a simple Transformer (like GPT-2) from scratch. ARENA Chapter 1.1 is a great coding tutorial[^ue9pdw6v8rj]
|
||||
|
||||
* This builds intuitions for mech interp *and* on using PyTorch.
|
||||
|
||||
* I have two video tutorials on this, starting from the basics - [start here](https://www.youtube.com/watch?v=bOYE6E8JrtU&list=PL7m7hLIqA0hoIUPhC26ASCVs_VrqcDpAz) if you’re not sure what to do!
|
||||
|
||||
* And use LLMs to fill in any background things you’re missing, like PyTorch basics
|
||||
|
||||
* Cloud GPUs:
|
||||
* You’ll need to be able to run language models, which (typically) needs a GPU
|
||||
* You can start with Google Colab to get started fast, but it’ll be very constraining to use long-term. Learn to rent and use a cloud GPU.
|
||||
* Newer Macbook Pros, or computers with powerful gaming GPUs may also be able to run LLMs locally
|
||||
* *Resource:* ARENA has a[guide](https://arena-chapter0-fundamentals.streamlit.app/#vm-setup-instructions). I like[runpod.io](http://runpod.io) as a provider;[vast.ai](http://vast.ai/) is cheaper.
|
||||
* nnsight also lets you do some [interpretability on certain models they host themselves](https://nnsight.net/notebooks/tutorials/get_started/start_remote_access/), including LLaMA 3 405B, which can be a great way to work with larger models.
|
||||
|
||||
### Mechanistic Interpretability Techniques
|
||||
|
||||
A lot of mech interp research looks like knowing the right technique to apply and in what context. This is a key thing to prioritise getting your head around when starting out. You’ll learn this with a mix of reading educational materials and doing coding tutorials like ARENA (discussed in next sub-section).
|
||||
|
||||
* [Ferrando et al](https://arxiv.org/abs/2405.00208) is a good **overview** of the key techniques - it’s long enough that you shouldn’t prioritise reading it in full, but it’s a great reference
|
||||
* Put it in a LLM context window and ask questions, or to write you exercises
|
||||
* **Essential**: Make sure you understand these **core techniques**, well enough that you can code it up yourself on a simple model like GPT-2 Small[^hh6mwdeo4zm]:
|
||||
|
||||
* Activation Patching
|
||||
|
||||
* Linear Probes
|
||||
|
||||
* Using Sparse Autoencoders (SAEs) (you only need to write code that uses an SAE, not trains one)
|
||||
|
||||
* Max Activating Dataset Examples
|
||||
|
||||
* Nice-to-have:
|
||||
|
||||
* Steering Vectors
|
||||
|
||||
* Direct Logit Attribution (DLA) (a simpler version is called logit lens)
|
||||
|
||||
* **Key exercise**: Describe each technique to an LLM with Ferrando et al in the context window and ask for feedback. Iterate until you get it all right.
|
||||
|
||||
* Use an anti-sycophancy prompt to get real feedback, by pretending someone else wrote your answer, e.g. “I saw someone claim this, it seems pretty off to me, can you help me give them direct but constructive feedback on what they missed? \[insert your description\]”
|
||||
|
||||
* Remember that there’s a bunch of valuable **black-box interpretability** techniques! (ie that don’t use the model’s internals) You can often correctly guess a model’s algorithm by reading its chain of thought. Careful variation of the prompt is a powerful way to causally test hypotheses.
|
||||
* They’re an additional tool. Often the correct first step in an investigation is just talking to the model and bunch and observing its behaviour. Don’t be a purist and dismiss them as “not rigorous” - they have uses and flaws, just like any other technique.
|
||||
* One [project I supervised](/api/post/wnzkjSmrgWZaBa2aC) on interpreting “self-preservation” in frontier models started with simple black-box techniques, and it just worked, we never needed anything fancier.
|
||||
* Understand fancier black-box techniques like [token forcing](https://arxiv.org/abs/2312.12321) (aka prefill attacks) where you put words in a model’s mouth.
|
||||
|
||||
### Mechanistic Interpretability Coding & Tooling
|
||||
|
||||
* **Goal:** Get comfortable running experiments and "playing" with model internals. Get the engineering basics down[^sxyjce3nii]. Get your hands dirty.
|
||||
|
||||
* **ARENA**: ARENA has [a set of fantastic coding tutorials by Callum McDougall](https://arena-chapter1-transformer-interp.streamlit.app/), you should just go do these. But there’s tons, so **prioritize ruthlessly**.
|
||||
* **Essential**: **Chapter 1.2** (Interpretability Basics – prioritize the first 3 sections on tooling, direct observation, and patching).
|
||||
* *Recommended:* 1.4.1 (Causal Interventions & Activation Patching – this is a core technique).
|
||||
* *Worthwhile*: 1.3.2 (Sparse Autoencoders (SAEs) – Skim or Skip section 1, the key thing to get from the rest is an intuition for what SAEs are, strengths and weaknesses, and how to use an open source SAE. Don’t worry about training them).
|
||||
* **Tooling** (**Essential**)**:** Get proficient with at least one mech interp library, this is what you’ll use to run experiments.
|
||||
* [TransformerLens](https://github.com/TransformerLensOrg/TransformerLens): best for small models <=9B where you want to write more complex interpretability experiments, or work with many models at once.
|
||||
* As of early Sept 2025, TransformerLens [v3](https://github.com/TransformerLensOrg/TransformerLens/releases/tag/v3.0.0a5) is in alpha, works well with large models and is far more flexible.
|
||||
* [nnsight](http://nnsight.net/): More performant, works well on larger models, it’s just a wrapper around standard LLM libraries like HuggingFace transformers
|
||||
* **LLM APIs**: Learn how to use an LLM API to call an LLM programmatically. This is super useful for measuring qualitative things about some data, and for generating synthetic datasets
|
||||
* I like [openrouter.ai](http://openrouter.ai) which lets you access almost all the important LLMs from a single place. GPT5 and Gemini are reasonably priced and good defaults, they have a range of sizes
|
||||
* Cerebras and Groq have *way* higher throughput than normal providers, and serve a handful of open source models, they may be worth checking out.
|
||||
* Exercise: Make a happiness steering vector (for e.g. GPT-2 Small) by having an LLM via an API generate 32 happy prompts and 32 sad prompts, and taking the difference in mean activations[^kte6u8splw] (e.g. the residual stream at the middle layer). Add this vector to the model’s residual stream[^2ob115pcmet] while generating responses to some example prompts, and use an LLM API to rate how happy they seem, and see this score go up when steering.
|
||||
|
||||
* **Open source LLMs**: You’ll want to work a lot with open source LLMs, as the thing you’re trying to interpret. The best open source LLM changes a lot
|
||||
* As of early Sept 2025, Qwen3 is a good default model family. Each model has reasoning and non-reasoning mode, there’s a good range of sizes, and most are dense[^1b9r0ass7sd]
|
||||
|
||||
* Gemma 3 and LLaMA 3.3 are decent non-reasoning models. I’ve heard bad things about gpt-oss and LLaMA 4
|
||||
|
||||
* *Gotcha:* The different open source LLMs often have different tokenizations and formats for chat or reasoning tokens. Using the wrong token format can only somewhat degrade performance and may be hard to notice while corrupting your results - keep an eye out, try hard to find where this might be documented, and sanity check by e.g. comparing to official evals
|
||||
|
||||
### Understanding the literature
|
||||
|
||||
Your priority is to understand the concepts and the basics, but you want a sense for the landscape of the field, so you should practice reading at least some papers.
|
||||
|
||||
* Remember, **breadth over depth**. Skim things, get a sense of what's out there, and only dive into the things that are most interesting.
|
||||
* You should be heavily using **LLMs** here. Give them something you're considering reading and get a summary, ask questions about the work, summarise your understanding to it and ask for feedback (with an anti-sycophancy prompt).
|
||||
* If you aren't able to verify yourself, cross-reference by asking multiple LLMs and making sure they all say consistent things.
|
||||
* Here’s [a list of my favourite papers](/api/post/NfFST5Mio7BCAQHPA) (as of mid 2024) with summaries and opinions
|
||||
* Do *not* try to read all of these in full. Skim summaries, skim abstracts, pick a few to explore deeper with an LLM, *then* decide if you want to read the full paper.
|
||||
* [My YouTube Channel](https://www.youtube.com/@neelnanda2469):[Paper walkthroughs](https://www.youtube.com/watch?v=KV5gbOmHbjU&list=PL7m7hLIqA0hpsJYYhlt1WbHHgdfRLM2eY&pp=gAQB), [recordings of myself doing research](https://www.youtube.com/watch?v=LP_NTmMvp10&list=PL7m7hLIqA0hr4dVOgjNwP2zjQGVHKeB7T), and talks.
|
||||
* [Open Problems In Mechanistic Interpretability](https://arxiv.org/abs/2501.16496) is a decent recent literature review, that a lot of top mech interp people were involved in
|
||||
* Be warned that the paper basically consists of a bunch of opinionated and disagreeable researchers writing their own sections and often having strong takes. Don’t defer to it too much, but it's a good way to quickly assess what's out there.
|
||||
* **Deep dives**: You should read at least one paper carefully and in full. This is a useful skill that you will use in research projects where there’s a handful of extremely relevant papers to your project
|
||||
* This is much more than just reading the words! You should write out a summary, try to understand the surrounding context with LLM help, be able to describe why the paper exists, the motivation, the problem it's trying to solve, etc.
|
||||
* Aim for a barbell strategy: put minimal effort into most papers and a lot of effort into a few.
|
||||
* **LLMs**: LLMs are a super useful tool for exploring the literature, but easy to shoot yourself in the foot with.
|
||||
* As a search engine over the literature (especially with some lit reviews in context, or a starting paper), basically doing a lit review, finding relevant work for a question you have, etc.
|
||||
* As a tool to help you skim a paper - put the paper in the context window[^bzop9pji3nl] then get a summary, ask it questions, etc
|
||||
|
||||
* If you’re concerned about hallucinations, you can ask it to support answers with quotes (and verify these are real and make sense), or give its answer to another LLM and ask for harsh critique of all the inaccuracies. Honestly, I often don’t bother though, frontier reasoning models are pretty good now.
|
||||
* As a tool to help with deep dives - you need to actually read the paper, but I recommend having the LLM chat open as you read with the paper in the context and asking it questions, for context, etc every time you get confused.
|
||||
|
||||
### Using LLMs for Learning
|
||||
|
||||
*Note: I expect this section to go out of date fast! Written early Sept 2025*
|
||||
|
||||
LLMs are a super useful tool for learning, especially in a new field. While they struggle to beat experts, they often beat novices. If you aren’t using them regularly throughout this process, I’d guess you’re leaving a bunch of value on the table.
|
||||
|
||||
But LLMs have weird flaws and strengths, and it’s worth being intentional about how you use them:
|
||||
|
||||
* **Use a good model**: The best paid models are way better than e.g. free ChatGPT. Don't be a cheapskate; if you can, get a $20/month subscription, it makes a big difference. Gemini 2.5 Pro, Claude 4.1 Opus with extended thinking, and GPT-5 Thinking are all reasonable. (do *not* use non-thinking GPT-5 or anything older like GPT-4o, reasoning models are a big upgrade)
|
||||
* If you can’t get a subscription, Gemini 2.5 Pro is also available for free, and is the best.
|
||||
* Use Gemini 2.5 Pro via [AI Studio](https://aistudio.google.com/prompts/new_chat), it’s way better than the main Gemini interface and has much nicer rate limits for free users. Always use compare mode (the button in the header with two arrows) to see two responses in parallel from Pro
|
||||
* See [thoughts](/api/post/jP9KDyMkchuv6tHwm?commentId=jDzbZGnjWDMsNjDPQ) from my MATS alum Paul Bogdan comparing different LLMs for learning, and why he currently prefers Gemini
|
||||
* **System Prompts:** System prompts make a big difference - be concrete and specific about what you want, and how you want it done.
|
||||
* LLMs are good at this: I'll just ramble at one about what the task is, my criteria, the failure modes I don't want, and then it’ll just write the prompt for me
|
||||
* If the prompt doesn’t work, tell the LLM what it did wrong, and see if it can rewrite the prompt for you.
|
||||
* **Merge perspectives**:
|
||||
* Ask a Q to multiple different frontier LLMs, give LLM B’s response to LLM A and ask it to assess the strengths and weaknesses then merge.
|
||||
* If a point is in both original responses, it’s probably not a hallucination
|
||||
* If you want to fact check an LLM’s answer, give it to another LLM with an anti-sycophancy prompt
|
||||
* **Anti-Sycophancy Prompts:** LLMs are bad at giving critical feedback. Frame your request so the sycophantic thing to do is to be critical, by pretending someone else wrote the thing you want feedback on.
|
||||
* *"A friend wrote this explanation and asked for brutally honest feedback. They'll be offended if I hold back. Please help me give them the most useful feedback."*
|
||||
* *"I saw someone claiming this, but it seems pretty dumb to me. What do you think?"*
|
||||
* *“Some moron wrote this thing, and I find this really annoying. Please write me a brutal but truthful response”*
|
||||
* Learn actively, not passively:
|
||||
* **Summarize** your understanding back to the LLM in your own words and ask for critical feedback. Do this every time you read a paper or learn about a new concept
|
||||
* Try having it teach you **socratically**. Note: you can probably design a better system prompt than the official “study mode”
|
||||
* Ask the LLM to **generate exercises** to test your understanding, including maths and coding exercises as appropriate.
|
||||
* Gemini can make multiple choice quizzes, which some enjoy
|
||||
* Coding exercises can be requested with accompanying tests, and template code with blank functions for you to fill out, a la the ARENA tutorials.
|
||||
* **Context engineering:** Modern LLMs are much more useful with relevant info in context. If you give them the paper in question, or source code of the relevant library[^207k0k5nobb], they’ll be far more helpful.
|
||||
|
||||
* See [this folder](https://drive.google.com/drive/u/0/folders/1GfrgKJwndk-twnJ8K7Ba-TE9i_8wBWAU) for a bunch of saved context files for mech interp queries. If you don’t know what you need, just use [this default file](https://drive.google.com/file/d/18cF3lkU17_elUSv0zk8KSVejM1jGfNnz/view?usp=drive_link).
|
||||
|
||||
* I recommend Gemini 2.5 Pro (1M context window) via[aistudio.google.com](http://aistudio.google.com/); the UI is better. Always turn compare mode on, you get two answers in parallel
|
||||
|
||||
* **Voice dictation**: If you dictate to your LLM, via free speech-to-text software, and run it with no editing, it’ll understand fine. I personally find this much easier, especially when brain-dumping.
|
||||
* [Superwhisper](http://superwhisper.com) on Mac is great; Superwhisper is not currently available on Windows, but Windows users can use [Whispr Flow](https://wisprflow.ai/).
|
||||
* **Coding**: LLM tools like Cursor are great for coding, but *not* if your goal is to learn. For things like ARENA, only let yourself use browser-based LLMs, and only use them as a tutor. Don’t copy and paste code, your goal is to learn not complete exercises.
|
||||
|
||||
Interlude: What is mech interp?
|
||||
-------------------------------
|
||||
|
||||
*Feel free to skip to the* “[*what should I do next*](/api/home#The_Big_Picture__Learning_the_Craft_of_Research)” part
|
||||
|
||||
At this point it’s worth reflecting on what mech interp actually *is*. What are we even doing here? There isn't a consensus definition on how exactly to define mechanistic interpretability, and different researchers will give very different takes. But *my* working definition is as follows[^979wnkvgpa4].
|
||||
|
||||
* **Interpretability** is the study of understanding models, gaining insight into their behavior, the cognition inside of them, why and how they work, etc. This is the important part and the heart of the field.
|
||||
* **Mechanistic** means using the internals of the model, the weights and activations
|
||||
* So **mechanistic interpretability** is any approach to understanding the model that uses its internals.
|
||||
* This is distinct from some other worthwhile directions, like **black box interpretability**, understanding models without using the internals, and **model internals**, using the internals of the model for other things like steering vectors.
|
||||
|
||||
**Why this definition?** To do impactful research, it's often good to find the directions that other people are missing. I think of most of machine learning as non-mechanistic non-interpretability. 99% of ML research just looks at the inputs and outputs to models, and treats its north star as controlling their behavior. Progress is defined by making a number go up, not to explain why it works. This has been very successful, but IMO leaves a lot of value on the table. Mechanistic interpretability is about doing better than this, and has achieved a bunch of cool stuff, like [teaching grandmasters how to play chess better by interpreting AlphaZero](https://arxiv.org/abs/2310.16410).
|
||||
|
||||
**Why care?** Obviously, our goal is not “do things if and only if they fit the above definition”, but I find it a useful one. To discuss this, let’s first consider our actual goals here. To me, **the ultimate goal is to make human-level AI systems (or beyond) safer**. I do mech interp because I think we’ll find enough understanding of what happens inside a model to be pragmatically useful here (also, because mech interp is fun!): to better understand how they work, detect if they're lying to us, detect and diagnose unexpected failure modes, etc. But people’s goals vary, e.g. real-world usefulness today, aesthetic beauty, or scientific insight. It’s worth thinking about what yours are.
|
||||
|
||||
Some implications of this framing worth laying out:
|
||||
|
||||
* My ultimate **north star is pragmatism** \- achieve enough understanding to be (reliably) useful. Subgoals like “completely reverse engineer the model” are just means to an end.
|
||||
* One of my big shifts in research prioritization in recent years is concluding that **reverse engineering is not the right aim**. Instead, I think we should just more directly try to do pragmatic work that enables us to do useful things using internals. I discuss this shift more [later on](/api/home#A_Pragmatic_Vision_for_Mech_Interp).
|
||||
* This is a **broad definition**. Historically, the field has focused on more specific agendas, like ambitious reverse engineering of models. But I think we shouldn’t limit ourselves, there’s many other important and neglected directions and the field is large enough to cover a lot of ground[^3zw26zes9dx]
|
||||
|
||||
* It’s about **understanding**, not just using internals - model internals methods like steering vectors can be useful for shaping a model’s behaviour, but compete with many powerful methods like prompting and fine-tuning. Very few areas of ML can achieve understanding
|
||||
* **Don’t be a purist** \- using internals is a means to an end. If black-box methods are the right tool, use them
|
||||
|
||||
The Big Picture: Learning the Craft of Research
|
||||
-----------------------------------------------
|
||||
|
||||
So, you've gone through the tutorials, you understand the core concepts, and you can write some basic experimental code. Now comes the hard part: learning how to actually do mech interp research[^7cxhc64szn8].
|
||||
|
||||
This is an inherently difficult thing to learn, of course. But IMO people often misunderstand what they need to do here, try to learn everything at once, or more generally make life unnecessarily hard for themselves. The key is to break the process down, understand the different skills involved, and focus on **learning the pieces with the fastest feedback loops first**.
|
||||
|
||||
I suggest breaking this down into two stages[^9wj0u0qz3q].
|
||||
|
||||
**Stage 2**: working on a bunch of throwaway mini projects of 1-5 days each. Don't stress about choosing the best projects or producing public output. The goal is to learn the skills with the fastest feedback loops.
|
||||
|
||||
**Stage 3:** After a few weeks of these, start to be more ambitious: paying more attention to how you choose your projects, gaining the subtler skills, and how to write things up. I still recommend working iteratively, in one to two week sprints, but ending up with longer-term projects if things go well.
|
||||
|
||||
Note: Unlike stage 1 to 2, the transition from stages two to three should be fairly gradual as you take on larger projects and become more ambitious. A good default would be after three to four weeks in stage two, but you don’t need to have a big formal shift.
|
||||
|
||||
**Mentorship**: A good mentor is a major accelerator, and finding one should be a major priority for you. In the careers section, I provide advice on [how to go about finding a good mentor](/api/home#Advice_on_finding_a_mentor), and [how concretely they can add value](/api/home#So_what_does_a_research_mentor_actually_do_). In the rest of the post I'll write most of it assuming you do not have a mentor and then flag the ways to use a mentor where appropriate.
|
||||
|
||||
### Unpacking the Research Process
|
||||
|
||||
I find it helpful to think of research as a cycle of four distinct stages. Read [my blog post on the research proces](/api/post/hjMy4ZxS5ogA9cTYK) for full details, but in brief:
|
||||
|
||||
* **Ideation:** You choose a research problem or a general domain to focus on.
|
||||
* **Exploration:** You may not have a specific hypothesis yet; you’re just trying to figure out the right questions to ask, and build deeper intuition for the domain. Your north star is to gain information and surface area.
|
||||
* **Understanding:** This begins when you have a concrete hypothesis, and some intuitive understanding of the domain. Your north star is to convince yourself that the hypothesis is true or false.
|
||||
* **Distillation:** Once you’re convinced, your north star is to compress your findings into concise, rigorous truth that you can communicate to the world - create enough experimental evidence to convince others, write it up clearly, and share it.
|
||||
|
||||
Underpinning these stages is a host of skills, best separated by how quickly you can apply them and get feedback. We learn by doing things and getting feedback, so you’ll learn the fast ones much more quickly. I put a rough list and categorization below.
|
||||
|
||||
My general advice is **to prioritize learning these in order of feedback loops**. If it seems like you need a slow skill to get started, like the taste to choose a good research problem, find a way to cheat rather than stressing about not having that skill (e.g. doing an incremental extension to a paper, getting one from a mentor, etc).
|
||||
|
||||
* Fast Loop (minutes-hours):
|
||||
* Planning and writing experiment code
|
||||
* **Medium**: Designing great experiments
|
||||
* **Medium**: Knowing when to write hacky vs. quality code.
|
||||
* Running/debugging experiments
|
||||
* **Medium/Slow**: Spotting and fixing subtle bugs (e.g., you got your tokenization subtly wrong, you didn’t search hyper-parameters well enough, etc)
|
||||
* Interpreting the results of a single experiment.
|
||||
* **Medium**: Understanding whether your results support your conclusions
|
||||
* **Slow**: Spotting subtle interpretability illusions where your results don't actually support your claims
|
||||
* Medium Loop (days):
|
||||
* Developing a conceptual understanding of mech interp
|
||||
* **Slow**: Noticing and fixing your own subtle confusions
|
||||
* **Slow**: Build a deep knowledge of the literature
|
||||
* Knowing how to explore without getting stuck
|
||||
* Writing up results
|
||||
* **Slow**: Communicating your work in a way that’s genuinely clear to people.
|
||||
* **Slow**: Communicating why your work is *interesting* to people
|
||||
* Slow Loop (weeks):
|
||||
* Prioritizing which experiment to do next
|
||||
* Knowing when to continue with a research direction or pivot to another angle of attack/another project
|
||||
* Identifying bad research ideas, *without* doing a project on them first
|
||||
* Very Slow Loop (months):
|
||||
* Coming up with good research ideas. This is the core of "research taste."
|
||||
|
||||
Your progression should be simple: First, focus on the fast/medium skills behind exploration and understanding with throwaway projects. Then, graduate to end-to-end projects where you can intentionally practice the deeper skills, and practice ideation and distillation too.
|
||||
|
||||
### What is research taste?
|
||||
|
||||
A particularly important and fuzzy type of skill is called research taste. I basically think of this as the bundle of intuitions you get with enough research experience that let you do things like come up with good ideas, predict if an idea is promising, have conviction in good research directions, etc. Check out [my post on the topic](/api/post/Ldrss6o3tiKT6NdMm) for more thoughts.
|
||||
|
||||
I broadly think you should just ignore it for now, find ways to compensate for not having much yet, and focus on learning the fast-medium skills, and this will give you a much better base for learning it. In particular, it's much faster to learn with a mentor, so if you don't have a mentor at the start, you should prioritize other things.
|
||||
|
||||
But you want to learn it eventually, so it's good to be mindful of it throughout, and look for opportunities to practice and learn lessons. I recommend treating it as a nice-to-have but not stressing about it
|
||||
|
||||
Note, one important trap here is that having good taste can often manifest as having confidence and conviction in some research direction. But often novice researchers develop this confidence and conviction significantly *before* they develop the ability to not be confident in bad ideas. It’s often a good learning experience to once or twice pursue a thing you feel really convinced is going to be epic and then discover you're wrong, so it's not that bad an outcome, especially in stage 2 (mini-projects) but be warned.
|
||||
|
||||
Stage 2: Practicing Research with Mini-Projects
|
||||
-----------------------------------------------
|
||||
|
||||
With that big picture in mind, let's get our hands dirty. You want to do a series of ~1-5 day mini-projects, for maybe 2-4 weeks. The goal right now is to learn the craft, not to produce groundbreaking research.
|
||||
|
||||
Focus on practicing exploration and understanding and gaining the fast/medium skills, leave aside ideation and distillation for now. If you produce something cool and want to write it up, great! But that’s a nice-to-have, not a priority.
|
||||
|
||||
Once you finish a mini-project, remember to do a post-mortem. Spend at least an hour analyzing: what did you do? What did you try? What worked? What didn't? What mistakes did you make? What would you do differently if doing this again? And how can you integrate this into your research strategy going forwards?
|
||||
|
||||
### Choose A Project
|
||||
|
||||
Some suggested starter projects
|
||||
|
||||
* **Replicate and Extend a Paper:** A classic for a reason. Replicate a key result, then extend it. Suggestions:
|
||||
* [Refusal is mediated by a single direction](https://arxiv.org/abs/2406.11717)
|
||||
* Extending papers can vary a lot in difficulty. For example, applying the method to study refusal on a new model is easy as you can reuse the same data, while applying it to a new concept is harder.
|
||||
* Skills: practicing activation patching and steering vectors.
|
||||
* [Thought Anchors](http://thought-anchors.com): apply these reasoning model interpretability methods to new types of prompts, or explore some prompts using the linked interface, or see if you can improve on the methods/invent your own.
|
||||
* Skills: reasoning model interpretability, using LLM APIs, and working with modern models
|
||||
* Replicate the truth probes in [Geometry of Truth](https://arxiv.org/abs/2310.06824) on a more modern model and try applying them in more interesting settings. How well do they generalise? Can you break them? If so, can you fix this?
|
||||
* Skills: probing, supervised learning, dataset creation
|
||||
* Play around with something interesting:
|
||||
* Use [Neuronpedia's attribution graphs](https://www.neuronpedia.org/gemma-2-2b/graph) to form a hypothesis about Gemma 2B, then use other methods (e.g. prompting) to verify it.
|
||||
* Skills: Attribution graphs, scientific mindset, prompting
|
||||
* Play with [Bartosz Cywiński's taboo models](https://huggingface.co/collections/bcywinski/gemma-2-9b-it-taboo-6826efbb186dfce0616dd174) that have a secret word programmed in and test as many methods as you can to find it.
|
||||
* If you’re feeling ambitious: train your own models with a more complex secret, and try to interpret those.
|
||||
* Skills: Logit lens, SAEs, black box methods
|
||||
* Explore [the models](https://github.com/clarifying-EM/model-organisms-for-EM) from the [emergent](https://www.emergent-misalignment.com/) [misalignment](/api/post/gLDSqQm8pwNiq7qst) [papers](https://openai.com/index/emergent-misalignment/).
|
||||
* Skills: steering vectors, SAEs, maybe fine-tuning
|
||||
* Pick some prompts from [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://arxiv.org/abs/2503.08679) and try to gain a deeper understanding of what’s happening
|
||||
* Skills: Open ended exploration, using whichever tools seem appropriate
|
||||
|
||||
Those cover two kinds of starter projects:
|
||||
|
||||
* **Understanding-heavy**, where you take a well-known domain and try to test a hypothesis there (e.g. extending a paper you’ve read closely)
|
||||
* Note that you still want to do *some*
|
||||
* **Exploration-heavy**, where you take some phenomena (a technique, a model, a phenomena, etc) play around with it, and try to understand what’s going on.
|
||||
* Exploration-heavy projects are often a less familiar style, so make sure to do some of those!
|
||||
|
||||
Common mistakes:
|
||||
|
||||
* People often get hung up on finding the “best” project. Sadly, that’s not going to happen. Instead, just do something and see what happens - better ideas and inspiration come with time.
|
||||
* Don't get too attached to your first project. It was probably badly chosen! These are throwaway projects, just move on once you’re not learning as much.
|
||||
* Conversely, don't flit between ideas so much that you never build your "getting unstuck" toolkit.
|
||||
* Avoid compute-heavy and/papers (e.g., training cross-layer transcoders) or highly technical papers (e.g., Sparse Feature Circuits).
|
||||
|
||||
### Practicing Exploration
|
||||
|
||||
The idea of exploration as a phase in itself often trips up people new to mech interp. They feel like they always need to have a plan, a clear thing they're doing at any given point, etc. In my experience, you will often spend more than half of a project trying to figure out what the hell is happening and what you think your plan is. This is totally fine!
|
||||
|
||||
You don't need a plan. It's okay to be confused. However, this does *not* mean you should just screw around. Your North Star: gain information and surface area[^xw1ra5pqnd] on the problem. Your job is to take actions that maximise information gained per unit time. If you've learned nothing in 2 hours, pivot to another approach. If 2-3 approaches were dead ends, it’s fine to just pick another problem.
|
||||
|
||||
I have [several research walkthroughs on my YouTube channel](https://www.youtube.com/watch?v=LP_NTmMvp10&list=PL7m7hLIqA0hr4dVOgjNwP2zjQGVHKeB7T) that I think demonstrates the mindset of exploration. What I think is an appropriate speed to be moving. E.g. I think you should aim to make a new plot every few minutes (or faster!) if experiments don't take too long to run.
|
||||
|
||||
A common difficulty is feeling “stuck” and not knowing what to do. IMO, this is largely a skill issue. Here's my recommended toolkit when this happens:
|
||||
|
||||
* Use "gain surface area" techniques, things that can surface new ideas and connections and just give you raw data to work with: look at the model's output/chain-of-thought, change the prompt, probe for a concept, look at an SAE/attribution graph, read examples from your dataset, try logit lens or steering, etc.
|
||||
* Set a [5-minute timer](https://www.neelnanda.io/blog/post-28-on-creativity-the-joys-of-5-minute-timers) and brainstorm things you're curious about or directions to try.
|
||||
* If you’re confused/curious about something, set a [5 minute timer](https://www.neelnanda.io/blog/post-28-on-creativity-the-joys-of-5-minute-timers) and brainstorm what could be happening.
|
||||
|
||||
Other advice:
|
||||
|
||||
* Before any >30 minute experiment, stop and brainstorm alternatives. Is this *really* the fastest way to gain information?
|
||||
* It's totally fine to pause for half a day to go learn some key background knowledge.
|
||||
* Get in the habit of keeping a research log of your findings and a "highlights" doc for the really cool stuff.
|
||||
* If applicable, it can be cool to have your research log be a slack/discord channel
|
||||
* Remember: when exploring and thinking through how to explain mysterious phenomena, most of your probability mass should be on "something I haven't thought of yet."
|
||||
* Practice following your curiosity, but be aware that it’ll often lead you astray at first. When it does, pay attention! What can you learn from this?
|
||||
|
||||
### Practicing Understanding
|
||||
|
||||
If exploration goes well, you'll start to form hunches about the problem. E.g. thinking that you are successfully (linearly) probing for some concept. Or that you found a direction that mediates refusal. Or that days of the week are represented as a circle in a 2D subspace.
|
||||
|
||||
Once you have this, you want to go to figure out if it's actually true. Be warned, the feeling of “being really convinced that it's true” is very different from actually being true. Part of being a good researcher is being good enough at testing and falsifying your pet hypotheses that, when you fail to falsify one, there’s a good chance that it's true. But you're probably not there yet.
|
||||
|
||||
Note: While I find it helpful to think of these as discrete stages, often you'll be flitting back and forth. A great way to explore is coming up with guesses and micro-hypotheses about what's going on, running a quick experiment to test them, and integrating the results into your understanding of the problem, going back to the drawing board.
|
||||
|
||||
Your North Star: convince yourself a hypothesis is true or false. The key mindset is skepticism. Advice:
|
||||
|
||||
* Before testing a hypothesis, set a five-minute timer and brainstorm, "What are the ways this could be false?"
|
||||
* Alternatively, write out the best possible case for your hypothesis and see where the argument feels weak.
|
||||
* Try using an LLM with an anti-sycophancy prompt ("My friend wrote this and wants brutal feedback...") to red-team your arguments - it probably won’t work, but might be helpful
|
||||
* Or set a 5 minute timer and brainstorm alternative explanations for your observations
|
||||
|
||||
You then want to convert these flaws and alternative hypotheses into concrete experiments. **Experiment design is a deep skill**. Honestly, I'm not sure how to teach it other than through experience. But one recommendation is to pay close attention to the experiments in papers you admire and analyze what made them so clever and effective. I also recommend that, every time you feel like you’ve (approximately) proven or falsified a hypothesis, adding them to a running doc of “things I believe to be true” with hypotheses, experiments, and results.
|
||||
|
||||
### Using LLMs for Research Code
|
||||
|
||||
In my opinion, coding is one of the domains where LLMs are most obviously useful. It was very striking to me how much better my MATS scholars were six months ago than 12 months ago, and I think a good chunk of this is attributable by them having much better LLMs to use. If you are not using LLMs as a core part of your coding workflow, I think you're making a mistake.
|
||||
|
||||
* **Use**[**Cursor**](http://cursor.com/): It's VS Code with fantastic AI integration. Make sure to add the docs for libraries with @ so the AI has context. The $20/month plan is worth it, if possible, and there’s a [free student version](https://cursor.com/students).
|
||||
* Claude Code is tempting but bad for learning and iteration. I’d use it for throwaway things and first drafts - if the draft has a bunch of bugs, go read the code yourself/throw it away and start again. Cursor facilitates reading the AI’s code better than Claude code does IMO
|
||||
* **A caveat:** If learning a new library (like in ARENA), first try writing things yourself. Use the LLM when stuck, not to replace the learning process.
|
||||
* Later on, when thinking about writing up results, if key experiments were mostly vibe-coded, I recommend re-implementing them by hand to make sure no dumb LLM bugs slipped in.
|
||||
|
||||
Interlude: What’s New In Mechanistic Interpretability?
|
||||
------------------------------------------------------
|
||||
|
||||
Feel free to skip to the “[*what should I do next*](/api/home#Stage_3__Working_Up_To_Full_Research_Projects)” part
|
||||
|
||||
Things move fast in mechanistic interpretability. Newcomers to the field who've kept up from afar are often pretty out of date. Here's what I think you need to know, again, filtered through my own opinions and biases.
|
||||
|
||||
### Avoiding Fads
|
||||
|
||||
This interlude is particularly important because **the field often has fads**: lines of research that are very popular for a year or so, make some progress and find many limitations, and then the field moves on. But if you’re new, and catching up on the literature, you might not realise. I often see people new to the field working on older things, that I don’t think are too productive to work on any more. Historical fads include:
|
||||
|
||||
* Interpreting toy models trained on algorithmic tasks (e.g. my [grokking work](https://arxiv.org/abs/2301.05217))
|
||||
* I no longer recommend working on this, as I think we basically know that “sometimes models trained on algorithmic tasks are interpretable”, and they’re sufficiently artificial and divorced from real models that I am pessimistic about deeper and more specific insights generalising
|
||||
* Circuit analysis via causal interventions on model components (e.g. the [IOI paper](https://arxiv.org/abs/2211.00593))
|
||||
* This is slightly more complicated. I think that's worth learning about, and techniques like activation and attribution patching are genuinely useful.
|
||||
* But the core problem is that once you got a sparse subgraph of a model responsible for a task, there wasn't really a “what next?”. This didn't tend to result in deeper insight because the nodes (eg layers or maybe attention heads) weren't monosemantic, and it was often more complicated than naive stories suggested but we didn’t have the tools to dig deeper.
|
||||
* It was pretty cool to see that this was possible at all, but there have been more than enough works in this area that the bar for a novel contribution is now much higher.
|
||||
* Simply identifying a circuit is no longer enough; you need to use that circuit to reveal a deeper, non-obvious property of the model. I recommend exploring [attribution-graph style approaches](https://www.neuronpedia.org/graph/info)
|
||||
* We're at the tail end of a fad of incremental [sparse autoencoder research](https://transformer-circuits.pub/2023/monosemantic-features)[^tq4gws0zq69] (i.e. focusing on simple uses and refinements of the basic technique)
|
||||
|
||||
* Calling this one a fad is probably more controversial (if only because it's more recent).
|
||||
|
||||
* The *specific* thing I am critiquing is the spate of papers, including ones I was involved in, that are about incremental improvements to the sparse autoencoder architecture, or initial demonstrations that you can apply SAEs to do things, or picking some downstream task and seeing what SAEs do on it.
|
||||
|
||||
* I think this made some sense when it seemed like SAEs could be a total gamechanger for the field, and where we were learning things from each new such paper. I think this moment has passed; I do not think they were a gamechanger in the way that I hoped they might be. See [more of my thoughts here](/api/post/4uXCAJNuPKtKBsi28).
|
||||
|
||||
* I am *not* discouraging work on the following:
|
||||
|
||||
* Attribution graph-based circuit analysis, which I don't think has played out yet - see [a recent overview of that sub-field I co-wrote](https://www.neuronpedia.org/graph/info).
|
||||
|
||||
* Trying meaningfully different approaches to dictionary learning (eg [SPD](https://arxiv.org/abs/2506.20790) or [ITDA](https://arxiv.org/abs/2505.17769)), or things targeted to fix conceptual limitations of current techniques (eg [Matryoshka](https://arxiv.org/abs/2503.17547)).
|
||||
|
||||
* Using SAEs as a tool, whether as part of a broader project investigating weird phenomena in model biology, or as a baseline/approach on some downstream task. The key is that the project’s motivation should *not* just be “what if we used SAEs for X?” unless there’s a good argument
|
||||
|
||||
* I particularly recommend them for tasks where you don’t know exactly what you’re looking for, e.g. trying to explore some mysterious phenomena
|
||||
|
||||
|
||||
Note that I am putting this after stage 2 because I think that **for initial throwaway projects you should** ***not*** **be stressing about novelty and avoiding fads** \- your goal is just to learn. But as we move into stage 3 you should start to be a bit more mindful about choosing more exciting/impactful projects where possible.
|
||||
|
||||
Also, take these as nudges and recommendations, not as instructions. If there's a direction you believe in that fits the things I'm critiquing, maybe I'm just wrong, maybe your thing is an exception, go wild, see what happens.
|
||||
|
||||
### What’s New In Mech Interp?
|
||||
|
||||
OK, so those are my hot takes on what not to do. What should you do? I think that some really cool new opportunities have opened up in mech interp over the last year, and newcomers may not have come across these. Here are some of the key themes in my favorite papers over the last year, that I’d love to see readers build on:
|
||||
|
||||
1. **Downstream tasks**. If our North Star is to be able to do useful things with interpretability, a great way to test our techniques is on real-world tasks, to see if they beat baselines or enable us to do a new thing
|
||||
1. Even if we can't directly test on the tasks we most care about, if models can let us do new things in *some* domains, that's reasonable evidence that we're learning something real in a way that is hard to fake.
|
||||
2. One of my favourite examples is **auditing games**: [Marks et al](https://arxiv.org/abs/2503.10965). is a fantastic paper that trained a model to have a hidden goal and then had different teams compete to find this goal, some with interpretability techniques, some without. Hypothesis: If interpretability does something real, then interp teams should win.
|
||||
1. This study itself was a bit ambiguous (the mech interp teams were 1st and 3rd, a black box team was 2nd), but I think this is a fantastic way to test interpretability techniques, and has a ton of potential for follow-on work evaluating interpretability in different kinds of games.
|
||||
2. The general idea is just that there is a model with some known hidden property and different techniques are used to see which can elicit it. Things like having different teams, allowed to use different techniques, etc. is just one implementation.
|
||||
2. **Model organisms**: The auditing games paper was made possible by the fact that they were able to make a model with a hidden goal[^cdmsagzbqkp], a model organism to study. In general, we’re collecting techniques like [synthetic document fine-tuning](https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/) to make really interesting model organisms.
|
||||
|
||||
1. This kind of thing has a lot of potential! If we want to make a lie detector, a core challenge is that we don’t know how to test if it works or not. But if we can insert beliefs or deceptive behaviours into a model, many more projects become possible
|
||||
|
||||
2. A great intro project is playing around with open source model organisms, e.g. from [Cywinski et al](https://arxiv.org/abs/2505.14352)
|
||||
|
||||
3. **Practice on the real AGI Safety problems**: Historically, interpretability could only practice on very dull toy problems like [modular addition](https://arxiv.org/abs/2301.05217). But we now have models that exhibit complex behaviors that seem genuinely relevant to safety concerns, and we can just study them directly, making it far easier to make real progress.
|
||||
1. E.g. [Rajamanoharan et al](/api/post/wnzkjSmrgWZaBa2aC) debunking assumed self-preservation, and [Goldowsky-Dill et al](https://www.apolloresearch.ai/research/deception-probes) probing for deception
|
||||
2. Weird behaviours: models can [insider trade then lie about it](https://www.apolloresearch.ai/research/deception-probes), [tell when they’re being evaluated](https://www.apolloresearch.ai/blog/claude-sonnet-37-often-knows-when-its-in-alignment-evaluations) (and act differently), [fake alignment](https://arxiv.org/abs/2412.14093), [reward hack](https://metr.org/blog/2025-06-05-recent-reward-hacking/), and more.
|
||||
4. **Real-World Uses of Interpretability**: Model interpretability-based techniques are starting to have genuine uses in frontier language models!
|
||||
1. [Linear probes](https://arxiv.org/abs/1610.01644), one of the simplest possible techniques, are a highly competitive way to [cheaply monitor systems](https://alignment.anthropic.com/2025/cheap-monitors/) for things like users trying to make bioweapons.
|
||||
2. I find it incredibly cool that interpretability can actually be useful, and kind of embarrassing that only a decade-old technique seems very helpful. Someone should do something about that. Maybe that someone could be you!
|
||||
3. This needs a very different kind of research: careful evaluation, comparison to strong baselines, and refinement of methods
|
||||
5. **Attribution graph-based circuit analysis**. The core problem with trying to analyze circuits in terms of things like a model's attention heads and layers is that often these things don't actually have a clear meaning. [Attribution graphs](https://transformer-circuits.pub/2025/attribution-graphs/methods.html) use techniques like [transcoders](https://arxiv.org/abs/2406.11944), popularized in [Anthropic's model biology](https://transformer-circuits.pub/2025/attribution-graphs/biology.html) work, to approximate models with a computational graph with meaningful nodes.
|
||||
1. See this [cross-org blog post](https://www.neuronpedia.org/graph/info) for the ongoing follow-on work across the community, and an open problems list I co-wrote![^p0f0m03b55r]
|
||||
|
||||
2. You can make and analyse your own attribution graphs on [Neuronpedia](https://www.neuronpedia.org/gemma-2-2b/graph)
|
||||
6. **Understanding model failures**: Models often do weird things. If we were any good at interpretability, we should be able to understand these. Recently, we’ve seen signs of life!
|
||||
1. [Meng et al](https://transluce.org/observability-interface) on why some models think 9.8 < 9.11
|
||||
2. A line of work studying [emergent misalignment](https://www.emergent-misalignment.com/) \- why training models on narrowly evil tasks like writing insecure code turns them into Nazis - has found some insights. [Wang et al](https://arxiv.org/abs/2506.19823) found this was driven by sparse autoencoder latents[^g12d8d1lqu] associated with movie villains, and in [Turner et al](/api/post/gLDSqQm8pwNiq7qst) we found that the model *could* have learned the narrow solution, but this was in some sense less “efficient” and “stable”
|
||||
|
||||
7. **Automated interpretability**: Using LLMs to automate interpretability. We saw signs of life on this from Bills et al and [Shaham et al](https://arxiv.org/abs/2404.14394), but LLMs are actually good now! It’s now possible to make basic interpretability agents that can do things like [solve auditing games](https://alignment.anthropic.com/2025/automated-auditing/)[^0td6a2gxwht]. And interpretability agents are the worst they’ll ever be[^5bdglmkdzr].
|
||||
|
||||
8. **Reasoning model interpretability**: All current frontier models are reasoning models—models that are trained with reinforcement learning to think[^wuxdh4f7kh] for a while before producing an answer. In my opinion, this requires a major rethinking of many existing interpretability approaches[^3qxoen8tddk], and calls for exploring new paradigms. IMO this is currently being neglected by the field, but will become a big deal.
|
||||
|
||||
1. In [Bogdan et al](http://thought-anchors.com), we explored what a possible paradigm could look like. Notably, there are far more interesting and sophisticated black box techniques with reasoning models, like resampling the second half of the chain of thought, or every time the model says a specific kind of sentence, deleting and regenerating that sentence.
|
||||
|
||||
|
||||
### A Pragmatic Vision for Mech Interp
|
||||
|
||||
Attentive readers may notice that the list above focuses on work to do with understanding the more qualitative high-level properties of models, and not ambitious reverse engineering. This is largely because, in my opinion, the former has gone great, while we have not seen much progress towards the fundamental blockers on the latter.
|
||||
|
||||
I used to be very excited about ambitious reverse engineering, but I currently think that the dream of completely reverse engineering a model down to something human understandable seems basically doomed. My interpretation of the research so far is that models have some human understandable high-level structure that drives important actions, and a very long tail of increasingly niche and irrelevant heuristics and biases. For pragmatic purposes, these can be largely ignored, but not if we want things like guarantees, or to claim that we have understood most of a model. I think that trying to understand as much as we can is still a reasonable proxy for getting to the point of being pragmatically useful, but think it’s historically been too great a focus of the field, and many other approaches seem more promising if our ultimate goals are pragmatic.
|
||||
|
||||
In some ways, this has actually made me more optimistic about interpretability ultimately being useful for AGI safety! Ambitious reverse engineering would be awesome but was always a long shot. But I think we've seen some real results for pragmatic approaches to mechanistic interpretability, and feel fairly confident we are going to be able to do genuinely useful things that are hard to achieve with other methods.
|
||||
|
||||
Stage 3: Working Up To Full Research Projects
|
||||
---------------------------------------------
|
||||
|
||||
Once you have a few mini-projects done, you should start being more ambitious. You want to think about gaining the deeper (medium/slow) skills, and exploring ideation and distillation.
|
||||
|
||||
However, you should still expect projects to often fail, and want to lean into breadth over depth and avoid getting bogged down in an unsuccessful project you can’t bear to give up on. To resolve this tension, I recommend **working in 1-2 week sprints**. At the end of each sprint, reflect and make a deliberate decision: **continue, or pivot?** The default should be to pivot unless the project feels truly promising. It’s great to give up on things, if it means you spend your time even better! But if it’s going great, by all means continue.
|
||||
|
||||
This strategy should mean that you eventually end up working on something longer-term when you find something *good*, but don't just get bogged down in the first ambitious idea you tried.
|
||||
|
||||
I recommend reviewing the list of skills earlier and just for each one, reflecting for a bit on how on top of it you think you feel and how you could intentionally practice it in your next project. Then after each sprint, before deciding whether to pivot, take an hour or two to do a post-mortem: what did you learn, what progress did you make on different skills, and what would you do differently next time? Your goal is to learn, and you learn much better if you make time to actually process your accumulated data!
|
||||
|
||||
### Key Research Mindsets
|
||||
|
||||
One way to decompose your learning is to think about research mindsets: the traits and mindsets a good researcher needs to have, that cut across many of these stages. See [my blog post on the topic for more](/api/post/cbBwwm4jW6AZctymL), but here's a brief view of how I'm currently thinking about it.
|
||||
|
||||
1. **Skepticism/Truth-seeking:** The default state of the world is that your research is false, because doing research is hard. Your north star should always be to find *true* insights[^lm5ixkfuzk]
|
||||
|
||||
1. It generally doesn't come naturally to people to constantly aggressively think about all the ways their work could be false and make a good faith effort to test it. You can learn to do better than this, but it often takes practice.
|
||||
|
||||
2. This is crucial in understanding, somewhat important in exploration, and crucial in distillation.
|
||||
|
||||
3. A common mistake is to grasp at straws to find a “positive” result, thinking that nothing else is worth sharing.
|
||||
|
||||
1. In my opinion, negative or inconclusive results that are well-analyzed are much better than a poorly supported positive result. I’ll often think well of someone willing to release nuanced negative results, and poorly of someone who pretends their results are better than they are.
|
||||
|
||||
2. **Prioritization:** Your time is scarce. Research involves making a bunch of decisions that are essentially searching through a high-dimensional space. The difference between a great and a mediocre researcher is being able to make these decisions well.
|
||||
1. If you have a good mentor, you can lean on them for this at first, but you will need to learn how to do this yourself eventually.
|
||||
2. This is absolutely crucial in exploration and ideation, but fairly important throughout.
|
||||
3. A good way to learn this one is to reflect on decisions you've made after the fact, eg in a sprint post-mortem, and think about how you could have made them better, and what generalisable lessons to take to the future
|
||||
3. **Productivity**[^idab8074tka]**:** The best researchers I've worked with get more than twice as much done as the merely good ones. Part of this is good research taste and making good prioritization decisions, but part of this is just being good at getting shit done.
|
||||
|
||||
1. Now, this doesn't necessarily mean pushing yourself until the point of burnout by working really long hours. Or cutting corners and being sloppy. This is about productivity integrated over the long term.
|
||||
|
||||
1. For example, sometimes the most productive thing to do is to hold off on starting work, set a 5 minute timer, brainstorm possible things to do next, and then pick the best idea
|
||||
|
||||
2. This takes many forms, and the highest priority for you:
|
||||
|
||||
1. Know when to write good code without bugs, to avoid wasting time debugging later, and when to write a hacky thing that just works.
|
||||
|
||||
2. Know the right keyboard shortcuts to move fast when coding.
|
||||
|
||||
3. Know when to ask for help and have people who can help you get unblocked where appropriate.
|
||||
|
||||
4. Be good at managing your time and tasks so that once you've decided what the highest priority thing to work on is, you in fact go and work on it.
|
||||
|
||||
5. Be able to make time to achieve deep focus on the key problems.
|
||||
|
||||
3. Exercise: Occasionally **audit your time**. Use a tool like [Toggl](http://toggl.com) for a day or two to log what you're doing, then reflect: where did time go? What was inefficient? How could I do this 10% faster next time?
|
||||
|
||||
1. The goal isn't to feel guilty, but to spot opportunities for improvement, like making a utility function for a tedious task.
|
||||
|
||||
4. **Knowing the literature**: At this point, there’s a lot of accumulated wisdom (and a lot of BS) in prior papers, in mech interp and beyond.
|
||||
1. This cuts across all stages:
|
||||
1. In ideation, you don’t want to accidentally reinvent the wheel. And often great ideas are inspired by prior work
|
||||
2. In exploration, you want to be able to spot connections, borrow interesting techniques, etc
|
||||
3. In understanding, you want to know the right standards of proof to check for, the best techniques to use, alternative hypotheses (that may have been raised in other works), etc
|
||||
4. In distillation, when writing a paper you’re expected to be able to contextualise it relative to existing work (i.e. write a related work section[^wpekmwudkpd]) which is important for other researchers knowing whether to care. And if you don’t know the standard methods of proof, key baselines everyone will ask about, key gotchas to check for etc, no one will believe your work.
|
||||
|
||||
2. LLMs are an incredibly useful tool here. GPT-5 thinking or Claude 4 with web search are both pretty useful tools here, as are the slower but more comprehensive deep research tools (Note that Google's is available for free, as of the time of writing)
|
||||
1. I recommend using these regularly and creatively throughout a project.
|
||||
2. You don't necessarily need to go and read the works that get surfaced, but even just having LLM summaries can get you more awareness of what's out there, and over time you'll build this into deeper knowledge.
|
||||
3. Of course, when there *does* seem to be a very relevant paper to your work, you should go do a deep dive and read it properly, not just relying on LLM summaries.
|
||||
4. Don’t stress - deep knowledge of the literature takes time to build. But you want to ensure you’re on an upwards gradient here, rather than assuming the broader literature is useless
|
||||
5. On the flip side, many papers *are* highly misleading/outright false, so please don’t just critically believe them![^1bau7vsh9tk]
|
||||
|
||||
|
||||
Okay, so how does this all tie back to the stages of research? Now you're going to be thinking about all four. We'll start by talking about how to deepen your existing skills with exploration and understanding, and then we'll talk about what practicing ideation and actually writing up your work should look like.
|
||||
|
||||
### Deepening Your Skills
|
||||
|
||||
You’ll still be exploring and understanding, but with a greater focus on rigor and the slower skills. In addition to the thoughts when discussing mindsets above, here’s some more specific advice
|
||||
|
||||
* **Deeper Exploration** is about internalizing the mindset of maximising productivity, which here means maximising information gain per unit time. Always ask, "Am I learning something?"
|
||||
* *Avoid Rabbit Holes:* A common mistake is finding one random anomaly and zooming in on it. Knowing when to pivot is crucial. Set a timer every hour or two to zoom out and ask if you’re making progress.
|
||||
* I recommend any time you notice yourself feeling a bit stuck or distracted or off track, setting a five minute timer and thinking about what could I be doing next, what should I be doing next, and am I doing the most important thing?
|
||||
* *Avoid Spreading Yourself Too Thin:* Doing lots of things superficially means none of them will be interesting.
|
||||
* If you have spent more than five hours without learning something new, you should probably try a different approach
|
||||
* And if you have spent more than two days without learning something new, you should seriously consider pivoting and doing something else.
|
||||
* To practice prioritization, be intentional about your decisions: write down *why* you think an experiment is the right call, and later reflect on whether you were right. This makes your intuitions explicit and easier to update.
|
||||
* **Deeper Understanding** is about practicing skepticism and building a bulletproof case. Red-team your results relentlessly.
|
||||
* Some experiments are much more impactful and informative than others! Don't just do the first experiment that pops into your head. Think about the key ways the hypothesis *could* be false, and how you could test that. Or about whether a skeptic could explain away a positive experimental results
|
||||
* A useful exercise is imagining you're talking to a really obnoxious skeptic who keeps complaining that they don't believe you and coming up with arguments for why your thing is wrong. What could you do such that they don't have a leg to stand on?
|
||||
* Of course, there's also an element of prioritization. Sometimes a shallow case that could be wrong is the right thing to aim for, if you’re working on an unimportant side claim/something that seems super plausible on priors, at which point you should just move on and do something else more interesting.
|
||||
* Exercise: To practice spotting subtle illusions, try red-teaming papers you read, thinking about potential flaws, and ideally run the experiments yourself.
|
||||
|
||||
### Doing Good Science
|
||||
|
||||
* **Avoid cherry-picking**: Researchers can, accidentally or purposefully, produce evidence that looks more compelling than it actually is. One classic way is cherry-picking: presenting only the examples that look most compelling.
|
||||
* When you write up work, always include some randomly selected examples, especially if you present extensive qualitative analysis of specific things. It's fine to put this in the appendix if space is scarce, but it should be there.
|
||||
* **Use baselines**: A common mistake is for people to try to show a technique works by demonstrating it gets 'decent' results, rather than showing it achieves better results than plausible alternatives that people might have used or are standard in the field. If you want people to e.g. use your cool steering vector results you need to show it beats changing the system prompt.
|
||||
* **Don’t sandbag your baselines**: Similarly, it's easy to put in much more effort finding good hyperparameters for your technique than for your baselines. Try to make sure you're achieving comparable results with your baselines that prior work in the field has.
|
||||
* **Do ablations on your fancy method**: It's easy for people to have a fancy method with lots of moving parts, when many actually are unnecessary. You should always try removing one part and see if the method breaks. Do this for each part.
|
||||
* For example, the [original unlearning method](https://arxiv.org/abs/2403.03218v1) in the [RMU paper](https://arxiv.org/abs/2403.03218) claimed it was based on finding a meaningful steering vector, until follow-up work found that it was just about adding a vector with really high norm that broke the model, and a random vector performed just as well.
|
||||
* **(Informally) pre-register claims**: It's important to clearly track which experimental results were obtained before versus after you formulated your claim. Post-hoc analysis (interpreting results after they're seen) is inherently less impressive than predictions confirmed by pre-specified experiments
|
||||
* **Be reproducible**: Where practical, share your code, data and models.
|
||||
* If you have time, make sure that it runs on a fresh machine and include a helpful readme that links to key model weights and datasets.
|
||||
* This both means others can check if your work is true and makes it more likely people will believe and build on your work[^adytzr5d7y] because they can see replications that are more likely to exist and because it's now low friction.
|
||||
|
||||
* **Simplicity:** Bias towards trying the simple, obvious methods first. Fancy techniques can be a trap. Good research is pragmatic, not about showing off.
|
||||
* If you’re designing a fancy technique/experiment, each new detail is one more thing that can break
|
||||
* If trying to explain something mysterious, novice researchers often neglect simple, dumb hypotheses like “maybe MLP0 is incredibly important on *every* input, and there’s nothing special going on with my prompt”
|
||||
* **Be qualitative** ***and*** **quantitative**: One of the major drivers of progress of modern machine learning is being quantitative, having benchmarks and showing that a technique increases numbers on them. One of the key drivers of progress in mech interp is an openness to qualitative research: summary statistics lose a ton of information. What can we learn by actually looking deeply into what's happening?
|
||||
* In my opinion, the best research tries to get the best of both worlds. It tries to understand what's happening via qualitative analysis and then validates it with more quantitative methods. If your paper only does one, it’s probably missing out
|
||||
* **Read your data**: A fantastic use of time, especially during the exploration phase, is just actually reading the data you're working with, or model chains of thought and responses.
|
||||
* Often, the quality of the data is a crucial driver of the results of your experiments. Often, it is quite bad.
|
||||
* Sometimes most of the work of a project is in noticing flaws in your data and making a better data set. Time figuring this out is extremely well spent.
|
||||
* Ditto, include random examples of the data in an appendix for readers to do spot checks of their own.
|
||||
* **Don’t reinvent the wheel**: A common mistake in mech interp is doing something that's already been done[^va7mhfkrhm]. We have LLM-powered literature reviews now. You have way less of an excuse. Check first!
|
||||
|
||||
* **Excitement is evidence of bullshit**: Generally, most true results are not exciting, but a fair amount of false results are. So from a Bayesian perspective, if a result is exciting and cool, it’s even more likely to be false than normal!
|
||||
* Resist the impulse to get really excited! The correct attitude to exciting results is deep skepticism until you have tried really hard to falsify it and run out of ideas.
|
||||
* **Get the stats right**: There's a lot of traps to avoid when it comes to things like having the right sample size, knowing what you can and cannot infer from the evidence you have, etc. One of my MATS alumni, Paul Bogdan, has a great blog post on this [here](/api/post/GxhtzqMwdTHo6326y).
|
||||
|
||||
### Practicing Ideation
|
||||
|
||||
Okay, so you want to actually come up with good research ideas to work on. What does this look like? I recommend breaking this down into **generating ideas** and then **evaluating** them to find the best ones.
|
||||
|
||||
To generate ideas, I'd often start with just taking a blank doc, blocking out at least an hour, and then just writing down as many ideas as you can come up with. Aim for quantity over quality. Go for at least 20.
|
||||
|
||||
There are other things you can do to help with generation:
|
||||
|
||||
* Throughout your previous sprints, every time you had an idle curiosity or noticed something weird, write it down in one massive long-running doc.
|
||||
* Likewise, when reading papers, note down confusions, curiosities, obviousnesses to do.
|
||||
|
||||
Okay, so now you have a big list. What does finding the best ones look like?
|
||||
|
||||
* Ideally, if you have a mentor or at least collaborators, you can just ask them to rate them.
|
||||
* If you do this, rate them yourself privately out of 10 before you look at their responses. Compare them and every time you have substantially different numbers, talk to the mentor and try to figure out why your intuitions disagree. This is a great source of supervised data for research taste.
|
||||
* Even if you don’t have a mentor, I think that just going through, rating each idea yourself based on gut feel and sorting is as good a way to prune down a long list as any
|
||||
* For the top few, I recommend trying to answer a few questions about them.
|
||||
* What would success look like here?
|
||||
* How surprised would I be if I did this for a month and nothing interesting had happened?
|
||||
* What skills does this require? Do I have them/could I easily gain them?
|
||||
* What models, data, computational resources, etc. does this require?
|
||||
* How does this compare to what the most relevant prior work did? Can I check for prior work and see if anything relevant comes up?
|
||||
|
||||
**Research Taste Exercises**
|
||||
|
||||
Gaining research taste is slow because the feedback loops are long. You can accelerate it with exercises that give you faster, proxy feedback. (Credit to [Chris Olah for inspiration here](https://colah.github.io/notes/taste/))
|
||||
|
||||
* If you have a mentor, query their taste for fast data and try to imitate it. Concretely:
|
||||
* Before each meeting, write a list of questions, then try to write up predictions for what the mentor will say, then actually ask the mentor, see what happens, and compare. If there are discrepancies, chat to the mentor and try to understand why.
|
||||
* Likewise, if the mentor makes a suggestion or asks a question you didn't expect, try to ask questions about where the thought came from.
|
||||
* Regularly paraphrase back to the mentor in your own words what you think they're saying, and then ask them to correct anything you're wrong about[^tt0owz8koks]
|
||||
|
||||
* **Learning from papers as "offline data":** When you read a paper, don't just passively consume it. Read the introduction, then stop. Try to predict what methods they used and what their key results will be. Then, continue reading and see how your predictions compare. Analyze why the authors made different choices. This trains your intuition on a much larger and faster dataset than your own research.
|
||||
|
||||
It’s also worth dwelling on what research taste actually is. [See my post](/api/post/Ldrss6o3tiKT6NdMm) for more, but I break it down as follows:
|
||||
|
||||
1. **Intuition (System 1):** This is the fast, gut-level feeling - what people normally think of when they say research taste. A sense of curiosity, excitement, boredom, or skepticism about a direction, experiment, or result.
|
||||
2. **Conceptual Framework (System 2)**: This is deep domain knowledge and understanding of underlying principles.
|
||||
3. **Strategic Big Picture**: Understanding the broader context of the field. What problems are important? What are the major open questions? What approaches have been tried? What constitutes a novel contribution?
|
||||
|
||||
### Write up your work!
|
||||
|
||||
At this stage, you should be thinking seriously about how to write up your work. Often, writing up work is the first time you really understand what a project has been about, or you identify key limitations, or experiments you forgot to do. You should check out [my blog post on writing ML papers](/api/post/eJGptPbbFPZGLpjsp) for much more detailed thoughts (which also apply to high-effort blog posts!) but I'll try to summarize them below.
|
||||
|
||||
**Why aim for public output?**
|
||||
|
||||
If producing something public is intimidating, for now, you can start by just writing up a private Google Doc and maybe share it with some friends or collaborators. But I heavily encourage people to aim for public output where they can. Generally, your research will not matter if no one reads it. The goal of research is to contribute to the sum of human[^e3252d8idmr] knowledge. And if no one understands what you did, then it doesn't matter.
|
||||
|
||||
Further, if you want to pursue a career in the space, whether a job, a PhD, or just informally working with mentors, **public research output is your best credential**. It's very clear and concrete proof that you are competent, can execute on research and do interesting things, and this is exactly the kind of evidence people care about seeing if they're trying to figure out whether they should work with you, pay attention to what you're saying, etc. It doesn’t matter if you wrote it in a prestigious PhD program or as a random independent researcher, if it’s good enough then people care.
|
||||
|
||||
There are a few options for what this can look like:
|
||||
|
||||
* A blog post (e.g. on a personal blog or LessWrong) - the simplest and least formal kind
|
||||
* An Arxiv paper - much more legible than a blog post, and honestly not much extra effort if you have a high-quality blog post[^8354hd0flji]
|
||||
|
||||
* A workshop paper[^9oppcf0ftrh] (i.e. something you submit for peer review to a workshop, typically part of a major ML conference, the bar is much lower than for a conference paper)
|
||||
|
||||
* A conference paper (the equivalent of top journals in ML, there’s a reasonably high quality bar[^fmh579omuc6], but also a *lot* of noise[^f09vsa4w37e])
|
||||
|
||||
|
||||
If this all seems overwhelming, starting out with blog posts is fine, but I think people generally overestimate the bar for arxiv or workshop papers - if you think you learned something cool in a project, this is totally worth turning into a paper!
|
||||
|
||||
**How to write stuff up?**
|
||||
|
||||
The core of a paper is the narrative. Readers will not take away more than a few sentences worth of content. Your job is to make sure these are the right handful of sentences and make sure the reader is convinced of them.
|
||||
|
||||
You want to distill your paper down into one to three key claims (your contribution), the evidence you provide that the contribution is true, the motivation for why a reader should care about them, and work all of this into a coherent narrative.
|
||||
|
||||
**Iterate**: I'm a big fan of writing things iteratively. You first figure out the contribution and narrative. You then write a condensed summary, the abstract (in a blog post, this should be a TL;DR/executive summary - also very important!). You then write a bullet point outline of the paper: what points you want to cover, what evidence you want to provide, how you intend to build up to that evidence, how you want to structure and order things, etc. If you have mentors or collaborators, the bullet point outline is often the best time to get feedback. Or the narrative formation stage, if you have an engaged mentor. Then write the introduction, and make sure you’re happy with that. Then (or even before the intro) make the figures - figures are incredibly important! Then flesh it out into prose. People spend a *lot* more time reading the abstract and the intro than the main body, especially when you account for all the people who read the abstract and then stop. So you should spend a lot more time per unit word on those.
|
||||
|
||||
**LLMs**: I think LLMs are a really helpful writing tool. They're super useful for getting feedback, especially if writing in an unfamiliar style like an academic ML paper may be for you. Remember to use anti-sycophanty prompts so you get real feedback. However, it's often quite easy to tell when you're reading LLM written slop. So use them as a tool, but don't just have them write the damn thing for you. But if you e.g. have writer’s block, having an LLM help you brainstorm or produce a first draft for inspiration, and can be very helpful.
|
||||
|
||||
**Common mistakes**
|
||||
|
||||
* **The reader does not have context**: Your paper will be clear in your head, because you have just spent weeks to months steeped in this research project. The reader has not. You will overestimate how clear things are to the reader, and so you should be massively erring in the other direction and spelling everything out as blatantly as possible.
|
||||
* **This is an incredibly common mistake** \- assume it will happen to you
|
||||
* The main solution is to get feedback from people with enough research context that they can actually engage and who are also willing to give you substantial negative feedback.
|
||||
* Notice the feeling of surprise when people are confused by something you thought was clear. Try to understand why they were confused and iterate on fixing it until it's clear.
|
||||
* **Writing is not an afterthought**: People often do not prioritize writing. They treat it like an annoying afterthought and do all the fun bits like running experiments, and leave it to the last minute.
|
||||
* **Acknowledge limitations**: There is a common mistake of trying to make your work sound maximally exciting. Generally, the people whose opinions you most care about are competent researchers who can see through this kind of thing
|
||||
* **Good writing is simple**: There's a tendency towards verbosity or trying to make things sound more complex and fancy than they actually are, so they feel impressive. I think this is a highly ineffective strategy
|
||||
* **Remember to motivate things**: It will typically not be obvious to the reader why your paper matters or is interesting. They do not have the context you do. It is your job to convince them, ideally in the abstract or perhaps intro, why they should care about your work, lest they just give up and stop reading.
|
||||
|
||||
Mentorship, Collaboration and Sharing Your Work
|
||||
-----------------------------------------------
|
||||
|
||||
A common theme in the above is that it's incredibly useful to have a mentor, or at least collaborators. Here I'll try to unpack that and give advice about how to go about finding one.
|
||||
|
||||
Though it's also worth saying that many mentors are not actually great researchers and may have bad research taste or research taste that's not very well suited to mech interp. What you do about this is kind of up to you.
|
||||
|
||||
### So what does a research mentor actually do?
|
||||
|
||||
A good mentor is an incredible accelerator. Dysfunctional as academia is, there is a reason it works under the apprenticeship-like system of PhD students and supervisors. When I started supervising, I was very surprised at how much of a difference a weekly check in could make! Here’s my best attempt to breakdown how a good mentor can add value:
|
||||
|
||||
* **Suggest research ideas** when you're starting out, letting you bypass the hardest skill (ideation) to focus on execution.
|
||||
* **Help you prioritize** which experiments to run, lending you their more experienced judgment, so you get more done.
|
||||
* **When to pivot**: if your research direction isn’t working out, having a mentor to pressure you to pivot can be extremely valuable[^7ruxx269r2s]
|
||||
|
||||
* **Provide supervised data for research taste**: For the slow/very-slow skills like coming up with research ideas, and prioritization, a *far* faster way to gain them at first is by learning to mimic your mentor’s.
|
||||
* **Act as an interface to the literature**: pointing you to the relevant work before you've built up deep knowledge yourself. Flagging standard baselines, standard metrics, relevant techniques, prior work so you don’t reinvent the wheel, etc.
|
||||
* **Red-team your results**, helping you spot subtle interpretability illusions and flaws in your reasoning that you're too close to see.
|
||||
* **Point out skills you're missing** that you didn't even notice were skills. Generally guide your learning and help you prioritize
|
||||
* **Walk you through communicating your work**, helping you distill your findings and present them clearly to the world.
|
||||
* **Motivation/accountability**: Many find it extremely helpful to have someone, even if very hands-off, who they present work to, so they feel motivated and accountable (especially if they e.g. want to impress the mentor, want a job, etc. Of course, these also increase stress!)
|
||||
* To those prone to analysis paralysis, being able to defer to a mentor on uncertain decisions can be highly valuable
|
||||
* **References**: Having a mentor who can vouch for your skill is very helpful, especially if they know people who may be hiring you in future.
|
||||
|
||||
### Advice on finding a mentor
|
||||
|
||||
Here are some suggested ways to get some mentorship while transitioning into the field. I discuss higher commitment ways, like doing a PhD or getting a research job, below.
|
||||
|
||||
Note: whatever you do to find a mentor, having evidence that you can do research yourself, that is, public output that demonstrates ability to self-motivate and put in effort, and ideally demonstrates actually interesting research findings, is incredibly helpful and should be a priority.
|
||||
|
||||
**Mentoring programs**
|
||||
|
||||
I think mentoring programs like [MATS](http://matsprogram.org) are an incredibly useful way into the field, you typically do a full-time, several month program where you write a paper, with weekly check-ins with a more experienced researcher. Your experience will vary wildly depending on mentor quality, but at least for my MATS scholars, often people totally new to mech interp can publish a top conference paper in a few months. See [my MATS application doc](https://tinyurl.com/neel-mats-app) for a bunch more details.
|
||||
|
||||
There’s **a wide range of backgrounds** among people who do them and get value - people totally new to a field, people with 1+ years of interpretability research experience who want to work with a more experienced mentor, young undergrads, mid-career professionals (including a handful of professors), and more.
|
||||
|
||||
My [MATS 10.0 applications](https://tinyurl.com/neel-mats-app) are open, due **Dec 23 2025**
|
||||
|
||||
Other programs (which I think are generally lower quality than MATS, but often still worth applying to depending on the mentor)
|
||||
|
||||
* *Full-time/In-person:*[MATS](https://www.matsprogram.org/), [Pivotal](https://www.pivotal-research.org/fellowship), [LASR](https://www.lasrlabs.org/), [PIBBSS](https://pibbss.ai/fellowship/)
|
||||
* *Part-time/Remote:*[SPAR](https://sparai.org/), [MARS](https://www.cambridgeaisafety.org/mars)
|
||||
|
||||
### Sending A Good Cold Email
|
||||
|
||||
You can also take matters into your own hands and try to convince someone to be your mentor. Reaching out to people, ideally via a warm introduction, but even just via a cold email, can be highly effective. However, I get lots of cold emails and I think many are not very effective, so here's some advice:
|
||||
|
||||
* **Don't just email the most prominent people**. A lot of people will just email the most prominent people in the field and ask for mentorship. This is a bad plan! These people are very busy and they also get lots of emails. I just reflexively respond to any email requesting mentorship with “please apply to my MATS cohort”.
|
||||
* However, there are lots of less prominent people who can provide a bunch of useful mentorship. These people are much more likely to be excited to get a cold email, to have time to engage, potentially even the spare capacity to properly mentor a project.
|
||||
* I think that many people who've recently joined my team or people who worked on a great paper with me during MATS are able to add a lot of value to people new to the field. And I would recommend reaching out to them!
|
||||
* For example, Josh Engels, a new starter on my team, said he would happily receive more cold emails (as of early Sept 2025).
|
||||
* As a general heuristic, email first authors of papers, not fancy last authors.
|
||||
* **Start small**: Don't email someone you've never interacted with before asking if they want to kind of officially mentor you on some project. That's a big commitment.
|
||||
* It's much better to be like, I'd be interested in having a chat about your paper or my work building on your paper.
|
||||
* Or just asking if they're down to have a chat giving you feedback on some project ideas, etc.
|
||||
* And if this goes well, it may organically turn into a more long-term mentoring relationship!
|
||||
* **Proof of work**: Demonstrate that you are actually interested in this person specifically, not just spamming tons of people.
|
||||
* Show that you've engaged with their work, say something intelligent about it, have some questions.
|
||||
* In the era of LLMs, this is less of a costly signal that you've actually taken an interest in this person specifically than it used to be, admittedly
|
||||
* But linking to some research you did building on their work I think is still reasonably costly, and very flattering to people.
|
||||
* **Prioritize aggressively**. Assume the reader will stop reading at any moment, so put your most critical and impressive information first.
|
||||
* **Explain who you are**: If you're emailing someone who gets more emails than they have capacity to respond to, they're going to be prioritizing. A key input into this is just who you are, what have you done, have you done something interesting that shows promise, do you have relevant credentials, etc. I personally find it very helpful if people just say the most impressive things about them in the first sentence or two.
|
||||
* To do this without seeming arrogant, you could try: "I'm sure you must get many of these emails. So to help you prioritise, here's some key info about me"
|
||||
* Use **bolding** for key phrases to make your email easily skimmable.
|
||||
* **Be concise**. One thing I would often appreciate is a short blurb summarizing your request with a link to a longer document for details if I'm interested.
|
||||
* **Quick requests**: Generally, my flow when reading emails is that I will either immediately respond or never look at it again. I'm a lot more likely to immediately respond if I can do so quickly. If you do want to email a busy person, have a clear, concrete question up front that they might be able to help with.
|
||||
|
||||
### Community & collaborators
|
||||
|
||||
Much easier than finding a mentor is finding collaborators, other people to work on the same project with, or just other people also trying to learn more about mech interp, who you can chat with and give each other feedback:
|
||||
|
||||
* **In-Person:** Local AI Safety hubs (London, Bay Area, etc.), University groups, ML conferences (e.g., the[NeurIPS Mech Interp workshop](http://mechinterpworkshop.com/) I co-organize), EAG/EAGx conferences.
|
||||
* If you’re a student, see if there’s a lab at your university that has some people interested in interpretability. There may be interested PhD students even if no professor works on it
|
||||
* **Online**: These are also good places to meet people! I recommend sharing work for feedback, or just asking about who’s interested in what you’re interested in, and trying to DM the people who engage/seem interested, and seeing what happens
|
||||
* [Open Source Mechanistic Interpretability Slack](https://www.neelnanda.io/osmi-slack-invite)
|
||||
* [Eleuther Discord](https://discord.gg/nHS4YxmfeM) (interpretability-general)
|
||||
* [Mech Interp Discord](https://discord.gg/ysVfhCfCKw)
|
||||
|
||||
**Staying up to date**: Another common question is how to stay up to date with the field. Honestly, I think that people new to the field should not worry that much about this. Most new papers are irrelevant, including the ones that there is hype around. But it's good to stay a little bit in the loop. Note that the community has substantial parts both in academia and outside, which are often best kept up with in different ways.
|
||||
|
||||
* LessWrong and the AlignmentForum are a reasonable place to keep up to date with the less academic half
|
||||
* Twitter is a confusing, chaotic place that is an okay place to keep up with both. It's a bit unclear who the right people to follow.
|
||||
* [Chris Olah](http://x.com/ch402) doesn't tweet much, but it's high quality when he does.
|
||||
* [I will tweet](http://x.com/neelnanda5) about all of my interpretability work and sometimes others.
|
||||
|
||||
Careers
|
||||
-------
|
||||
|
||||
### Where to apply
|
||||
|
||||
* Anthropic’s interpretability team roles: [research scientist](https://job-boards.greenhouse.io/anthropic/jobs/4020159008), [research engineer](https://job-boards.greenhouse.io/anthropic/jobs/4020305008), [research manager](https://job-boards.greenhouse.io/anthropic/jobs/4009173008)
|
||||
* [OpenAI's interpretability team roles](https://openai.com/careers/research-engineer-scientist-interpretability)
|
||||
* My team at Google DeepMind will hopefully be [hiring in early 2026](https://deepmind.google/about/careers/#open-roles)! Watch this space
|
||||
* [Transluce](https://transluce.org/) \-\- a nonprofit research lab
|
||||
* [Goodfire](https://www.goodfire.ai/) \-\- a mech interp startup that are [hiring a bunch](https://www.goodfire.ai/careers).
|
||||
* They [recently raised a $50 million Series A](https://www.goodfire.ai/blog/announcing-our-50m-series-a) and as of the time of writing are trying to both have people focused on products, and people focused on more fundamental research
|
||||
* The UK government's AI Security Institute's interpretability team ([not currently hiring](https://www.aisi.gov.uk/careers#open-roles))
|
||||
|
||||
**Applying for grants**
|
||||
|
||||
For people trying to get into mech interp via the safety community, there are some funders around open to giving career transition grants to people trying to upskill in a new field like mech interp. Probably the best place I know of is [Open Philanthropy's Early Career Funding.](https://www.openphilanthropy.org/career-development-and-transition-funding/)
|
||||
|
||||
### Explore Other AI Safety Areas
|
||||
|
||||
Mech interp isn't the only game in town! There’s other important areas of safety like Evals, AI Control, and Scalable Oversight, the latter two in particular seem neglected compared to mech interp. The[GDM AGI Safety Approach](https://arxiv.org/pdf/2504.01849) gives an overview of different parts of the field. If you’re doing this for safety reasons, I’d check if there’s other, more neglected subfields, that also appeal to you!
|
||||
|
||||
### What do hiring managers look for
|
||||
|
||||
Leaving aside things that apply to basically all roles, like whether this person has a good personality fit (which often just means looking out for red flags), here’s my sense of what hiring managers in interpretability are often looking for.
|
||||
|
||||
A useful mental model is that from a hiring manager's perspective, they're making an uncertain bet with little information in a somewhat adversarial environment. Each applicant wants to present themselves as the perfect fit. This means managers need to rely on signals that are hard to fake. But it’s quite difficult to get that much info on a person before you actually go and work with them a bunch.
|
||||
|
||||
Your goal as a candidate is to provide compelling, hard-to-fake evidence of your skills. The best way to do that is to simply do good research and share it publicly. If your research track record is good enough, interviews may just act as a check for red flags and to verify that you can actually write code and run experiments well.
|
||||
|
||||
Key skills:
|
||||
|
||||
* **Research Skills:** A track record of completing end-to-end projects is the best signal. Papers are a great way to show this.
|
||||
* **Research taste**: The ability to come up with great research ideas *and* drive them to completion is rare and very valuable.
|
||||
* **Experiment design**: Can they design good experiments and make their research ideas concrete and convert them into actions?
|
||||
* **Conceptual Understanding of Mech Interp:** Do you get the key ideas and know the literature?
|
||||
* **Productivity and Conscientiousness:** This is a very hard one to interview for, but incredibly important. A public track record of doing interesting things is a good signal, as are strong references from trusted sources[^slnwemz4grq].
|
||||
|
||||
* **Engineering Skills:** Can you work fluently in a Python notebook? Can you write experiment code fast and well? Can you get things done? Do you understand the standard gotchas?
|
||||
* **Deep engineering skill**: Beyond hacking together experiments, can you navigate large, complex codebases, write maintainable code, design complex software projects, etc?
|
||||
* This is much more important if doing research inside a larger lab or tech company than as an independent researcher or academic.
|
||||
* One of the most common reasons we don't hire seemingly promising researchers onto my team is because they lack sufficiently strong engineering skills.
|
||||
* Obviously, LLMs are substantially changing the game when it comes to engineering skills, but I think deep engineering skills will be much harder to automate than shallow ones, unfortunately.
|
||||
* Unfortunately, I don’t have great advice on how to gain these other than working in larger and more complex codebases and learning how to cope. Pair programming with more experienced programmers can be a great way to transfer tacit knowledge
|
||||
* **Skepticism**: Can you constructively engage with research and critically evaluate it? In particular, can you do this to your own research? Good researchers need to be able to do work that is true.
|
||||
|
||||
### Should you do a PhD?
|
||||
|
||||
I don't have a PhD (and think I would have had a far less successful career if I had tried to get one) so I'm somewhat biased. But it's a common question. Here are the strongest arguments I’ve heard in favour:
|
||||
|
||||
* You get extremely high **autonomy**. If you want to spend years going deep on a niche topic that no industry lab would fund, a PhD is one of the only ways to do it.
|
||||
* It's a great environment to cultivate the ability to **set your own research agenda**. This is a crucial and difficult skill that is harder to learn in industry, where agendas are often set from the top down (though this varies a lot between team).
|
||||
|
||||
And here are the reasons I think it's often a bad idea:
|
||||
|
||||
* The opportunity cost is immense. You could spend 4-6 years gaining direct, relevant experience in an industry lab.
|
||||
* Academic incentives can be misaligned with doing impactful research, e.g. pressure to publish meaning you’re discouraged from admitting to the limitations of your work.
|
||||
* The quality of supervision varies wildly, and a bad supervisor can make your life miserable.
|
||||
* Quality of life: The pay is generally terrible, which may or may not matter to you, and you may only get places in a different city/country than you’d prefer.
|
||||
|
||||
But with all those caveats in mind, it’s definitely the right option for some! My overall take:
|
||||
|
||||
* The key thing that matters is mentorship, being in an environment where you are working with a better researcher, and learning from them.
|
||||
* PhDs are often a good way of getting this. But if you can gain this by another way, plausibly you should go to that instead. PhDs have a lot of downsides too.
|
||||
* Generally, the variance between supervisors and between managers in industry will dominate the academia versus industry differences, and thus you should pay a lot of attention to who exactly would be managing you.
|
||||
* For a PhD, try to speak to your potential supervisor’s students in a private setting. If they say pretty bad things, that's a good reason not to go for the supervisor.
|
||||
* A common mistake is optimising for the most prestigious and famous supervisor when you often want to go for the ones who will have the most time for you, which anti-correlates.
|
||||
* A common mistake is people feeling they need to *finish* PhDs. But if you sincerely believe that the point of a PhD is to be a learning environment, then why would the formal end of the PhD be the optimal time to leave? It's all kind of arbitrary.
|
||||
* IMO, at least every six months, you should seriously evaluate what other opportunities you have, try applying for some things and be emotionally willing leave if a better opportunity comes along (taking into account switching costs).
|
||||
* Note that often you can just take a year's leave of absence and resume at will.
|
||||
|
||||
### Relevant Academic Labs
|
||||
|
||||
I’m a big fan of the work coming out of these two, they seem like great places to work:
|
||||
|
||||
* David Bau (Northeastern)
|
||||
* Martin Wattenberg & Fernanda Viegas (Harvard)
|
||||
|
||||
Other labs that seem like good places to do interpretability research (note that this is not trying to be a comprehensive list!):
|
||||
|
||||
* Yonatan Belinkov (Technion)
|
||||
* Jacob Andreas (MIT)
|
||||
* Jacob Steinhardt (Berkeley)
|
||||
* Ellie Pavlick (Brown)
|
||||
* Victor Veitch (UChicago)
|
||||
* Robert West (EPFL)
|
||||
* Roger Grosse (Toronto)
|
||||
* Mor Geva (Tel Aviv)
|
||||
* Sarah Wiegreffe (Maryland)
|
||||
* Aaron Mueller (Boston University)
|
||||
|
||||
*Thanks a lot to Arthur Conmy, Paul Bogdan, Bilal Chughtai, Julian Minder, Callum McDougall, Josh Engels, Clement Dumas, Bart Bussmann for valuable feedback*
|
||||
|
||||
[^nifk1wb1jum]: Note that I mean a full working month here. So something like 200 working hours. If you're only able to do this part-time, it's fine to take longer. If you're really focused on it, or have a head-start, then move on faster.
|
||||
|
||||
[^ue9pdw6v8rj]: If you want something even more approachable, one of my past MATS scholars recommends getting GPT-5 thinking to produce coding exercises (eg a Python script with empty functions, and good tests), for an easier way in.
|
||||
|
||||
[^hh6mwdeo4zm]: It’s fine for this coding to need a bunch of LLM help and documentation/tutorial looking up, this isn’t a memory test. The key thing is being able to correctly explain the core of each technique to a friend/LLM.
|
||||
|
||||
[^sxyjce3nii]: Note: This curriculum aims to get you started on independent research. This is often good enough for academic labs, but the engineering bar for most industry labs is significantly higher, as you’ll need to work in a large complex codebase with hundreds of other researchers. But those skills take much longer to gain.
|
||||
|
||||
[^kte6u8splw]: You want to exclude the first token of the prompt when collecting activations, it’s a weird attention sink and often has high norm/is anomalous in many ways
|
||||
|
||||
[^2ob115pcmet]: Gotcha: Remember to try a bunch of coefficients for the vector when adding it. This is a crucial hyper-parameter and steered model behaviour varies a lot depending on its value
|
||||
|
||||
[^1b9r0ass7sd]: Mixture of expert models, where there are many parameters, and only a fraction light up for each token, are a pain for interpretability research. Larger models means you'll need to get more/larger GPUs which is expensive and unwieldy. Favor working with dense models where possible.
|
||||
|
||||
[^bzop9pji3nl]: You can download then upload the PDF to the model, or just select all and copy and paste from the PDF to the chat window. No need to correct the formatting issues, LLMs are great at ignoring weird formatting artifacts
|
||||
|
||||
[^207k0k5nobb]: repo2txt.com is a useful tool for concatenating a Github repo into a single txt file
|
||||
|
||||
[^979wnkvgpa4]: If you would like other perspectives, check out Open Problems in Mechanistic Interpretability (broad lit review from many leading researchers, recent), or Interpretability Dreams (from Anthropic, 2 years old)
|
||||
|
||||
[^3zw26zes9dx]: And for reasons we’ll discuss later, now feel much more pessimistic about the ambitious reverse engineering direction
|
||||
|
||||
[^7cxhc64szn8]: Even if you already have a research background in another field, mechanistic interpretability is sufficiently different that you should expect to need to relearn at least some of your instincts. This stage remains very relevant to you, though you can hopefully learn faster.
|
||||
|
||||
[^9wj0u0qz3q]: The rest of this piece will be framed around approaching learning research like this and why I think it is a reasonable process. Obviously, there is not one true correct way to learn research! When I e.g. critique something as a “mistake”, interpret this as “I often see people do this and think it’s suboptimal for them”, not “there does not exist a way of learning research where this is a good idea
|
||||
|
||||
[^xw1ra5pqnd]: My term for associated knowledge, understanding, intuition, etc.
|
||||
|
||||
[^tq4gws0zq69]: Read my thoughts on SAEs here. There’s still useful work to be done, but it’s an oversubscribed area, and our bar should be higher. They are a useful tool, but not as promising as I once hoped.
|
||||
|
||||
[^cdmsagzbqkp]: This was using a technique called synthetic document fine-tuning (and some other creativity on top), which basically lets you insert false beliefs into a model by generating a bunch of fictional documents where those beliefs are true and fine-tuning the model on them.
|
||||
|
||||
[^p0f0m03b55r]: We chose problems we’re excited to see worked on, while trying to avoid fad-like dynamics
|
||||
|
||||
[^g12d8d1lqu]: Latents refer to the hidden units of the SAE. These were originally termed “features”, but that term is also used to mean “the interpretable concept the latent refers to”, so I use a different term to minimise confusion.
|
||||
|
||||
[^0td6a2gxwht]: One of my MATS scholars make a working GPT-5 model diffing agent in a day
|
||||
|
||||
[^5bdglmkdzr]: This is the one line in the post without a “as of early Sept 2025” disclaimer, this feels pretty evergreen
|
||||
|
||||
[^wuxdh4f7kh]: Note: "think" or "chain of thought" are terrible terms. It's far more useful to think of the chain of thought as a scratchpad that a model with very limited short-term memory can choose to use or ignore.
|
||||
|
||||
[^3qxoen8tddk]: Reasoning models break a lot of standard interpretability techniques because now the computational graph goes through the discrete, non-differentiable, and random operation of sampling thousands of times. Most interpretability techniques focus on studying a single forward pass.
|
||||
|
||||
[^lm5ixkfuzk]: Not just, e.g., ones you can publish on.
|
||||
|
||||
[^idab8074tka]: I called this moving fast in the blog post, but I think that may have confused some people.
|
||||
|
||||
[^wpekmwudkpd]: Though often this is done well with just a good introduction
|
||||
|
||||
[^1bau7vsh9tk]: And having a well-known researcher as co-author is not sufficient evidence to avoid this, alas. I’m sure at least one paper I’ve co-authored in the past year or two is substantially false
|
||||
|
||||
[^adytzr5d7y]: It's strongly in your interests for people to build on your work because that makes your original work look better, in addition to being just pretty cool to see people engage deeply with your stuff.
|
||||
|
||||
[^va7mhfkrhm]: Note that deliberately reproducing work, or trying to demonstrate the past work is shoddy, is completely reasonable. You just need to not accidentally reinvent the wheel.
|
||||
|
||||
[^tt0owz8koks]: This is generally a good thing to do regardless of whether you’re focused on research taste or not!
|
||||
|
||||
[^e3252d8idmr]: And, nowadays, LLM knowledge too I guess?
|
||||
|
||||
[^8354hd0flji]: Note that you’ll need someone who’s written several Arxiv papers to endorse you. cs.LG is the typical category for ML papers.
|
||||
|
||||
[^9oppcf0ftrh]: Note that you can submit something to a workshop and to a conference, so long as the workshop is “non-archival”
|
||||
|
||||
[^fmh579omuc6]: A conference paper is a fair bit more effort, and you generally want to be working with someone who understands the academic conventions and shibboleths and the various hoops you should be jumping through. But I think this can be a nice thing to aim for, especially if you're starting out and need credentials, though mech interp cares less about peer review than most academic subfields.
|
||||
|
||||
[^f09vsa4w37e]: See this NeurIPS experiment showing that half the spotlight papers would be rejected by an independent reviewing council
|
||||
|
||||
[^7ruxx269r2s]: This is one of the most valuable things I do for my MATS scholars, IMO.
|
||||
|
||||
[^slnwemz4grq]: Unfortunately, standard reference culture, especially in the US, is to basically lie, and the amount of lying varies between contexts, rendering references mostly useless unless from a cultural context the hiring manager understands or ideally from people they know and trust. This is one of the reasons that doing AI safety mentoring programs like MATS can be extremely valuable, because often your mentor will know people who might then go on to hire you, which makes you a lower risk hire from their perspective.
|
||||
|
||||
@@ -4,43 +4,13 @@ Source: https://www.lesswrong.com/posts/hjMy4ZxS5ogA9cTYK/how-i-think-about-my-r
|
||||
Mirror/sequence URL visible on page: https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/hjMy4ZxS5ogA9cTYK
|
||||
Author: Neel Nanda
|
||||
Date: 26th Apr 2025
|
||||
Fetch-status: excerpted from LessWrong HTML via browser plus cross-checked against local shared draft.
|
||||
Fetch-status: full post text, fetched 2026-08-15 from the LessWrong markdown API (`/api/post/hjMy4ZxS5ogA9cTYK`), comments and site navigation stripped. (CLAUDE agent)
|
||||
Use: research-process / research-taste evidence, especially for agents deciding what mode of work they are in.
|
||||
|
||||
## Why this matters for agents
|
||||
|
||||
Nanda frames empirical research as stage-dependent. A model should not demand a crisp hypothesis when the right stage is exploration; it should not accept weak, cherry-picked evidence when the task has moved into understanding or distillation.
|
||||
|
||||
## Quotes
|
||||
|
||||
> This guide focuses more on the strategic (high-level direction, when to give up or pivot, etc) and tactical (what to do next, how to prioritise, etc) aspects of research - the "how to think about it" rather than just the "how to do it." Some of skills (coding, reading papers, understanding ML/mech interp concepts) are vital for how to do it, but not in scope here.
|
||||
|
||||
> How to get started? Strategic and tactical thinking are hard skills, and it is rare to be any good at them when starting out at research (or ever tbh). The best way to learn them is by trying things, making predictions, seeing what you get right or wrong (i.e., getting feedback from reality), and iterating.
|
||||
|
||||
> I see research as breaking down into a few stages:
|
||||
>
|
||||
> 1. Ideation - Choose a problem/domain to focus on
|
||||
> 2. Exploration - Gain Surface area
|
||||
> 1. North star: Gain information
|
||||
> 3. Understanding - Test Hypotheses
|
||||
> 1. North star: Convince yourself of a key hypothesis
|
||||
> 4. Distillation - Compress, Refine, Communicate
|
||||
> 1. North star: Compress your research findings into concise, rigorous truth that you can communicate to the world
|
||||
|
||||
> At the start, your understanding of the problem is often vague. Naively, it’s easy to think of research as being about testing specific hypotheses, but in practice you often start out not even knowing the right questions to ask, or the most promising directions. The exploration stage is about moving past this.
|
||||
|
||||
> Not having a clear goal/next step doesn’t mean that you don’t need to prioritise! Prioritise for information gain.
|
||||
|
||||
> Frequently ask yourself “am I getting enough information per unit time?” If you haven’t learned anything recently, shake it up.
|
||||
|
||||
> The mark of a good researcher is a deep commitment to skepticism of your results.
|
||||
|
||||
> A great experiment elegantly, and conclusively distinguishes between several plausible hypotheses, validates non-trivial predictions made by one hypothesis, and is tractable to implement in practice.
|
||||
|
||||
> The north star here is to distill your research findings into concise, rigorous truth that you can communicate to the world.
|
||||
|
||||
> Write to inform, not persuade - if you are clear (a high bar), and your results are interesting, people will likely appreciate your work.
|
||||
|
||||
## Source graph
|
||||
|
||||
High-value links inside or adjacent to this post:
|
||||
@@ -49,3 +19,109 @@ High-value links inside or adjacent to this post:
|
||||
- Chris Olah, research taste: https://colah.github.io/notes/taste/
|
||||
- Nanda Othello research process write-up: https://www.alignmentforum.org/s/nhGNHyJHbrofpPbRG/p/TAz44Lb9n9yf52pv8
|
||||
- Nanda standards post: https://www.neelnanda.io/blog/35-standards
|
||||
|
||||
## Full post
|
||||
|
||||
*This is the first post in a sequence about how I think about and break down my research process. Post 2 is coming soon.*
|
||||
|
||||
*Thanks to Oli Clive-Griffin, Paul Bogdan, Shivam Raval and especially to Jemima Jones for help and feedback, and to my co-author Gemini 2.5 Pro - putting 200K tokens of past blog posts and a long voice memo in the context window is OP.*
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
Research, especially in a young and rapidly evolving field like mechanistic interpretability (mech interp), can often feel messy, confusing, and intimidating. Where do you even start? How do you know if you're making progress? When do you double down, and when do you pivot?
|
||||
|
||||
These are far from settled questions, but I’ve supervised 20+ papers by now, and have developed my own mental model of the research process that I find helpful. This isn't *the* definitive way to do research (and I’d love to hear other people’s perspectives!) but it's a way that has worked for me and others.
|
||||
|
||||
My goal here is to demystify the process by breaking it down into stages and offering some practical advice on common pitfalls and productive mindsets for each stage. I’ve also tried to be concrete about what the various facets of ‘being a good researcher’ actually mean, like ‘research taste’ ([see post 3](/api/post/Ldrss6o3tiKT6NdMm)). I’ve written this post for a mech interp audience, but hopefully it is useful for any empirical science with short feedback loops, and possibly even beyond that.
|
||||
|
||||
This guide focuses more on the *strategic* (high-level direction, when to give up or pivot, etc) and *tactical* (what to do next, how to prioritise, etc) aspects of research – the "how to think about it" rather than just the "how to do it." Some of skills (coding, reading papers, understanding ML/mech interp concepts) are vital for how to do it, but not in scope here (I recommend the [ARENA curriculum](https://arena-chapter1-transformer-interp.streamlit.app/) and [my paper reading list](/api/post/NfFST5Mio7BCAQHPA) if you need to skill up).
|
||||
|
||||
How to get started? Strategic and tactical thinking are hard skills, and it is rare to be any good at them when starting out at research (or ever tbh). The best way to learn them is by trying things, making predictions, seeing what you get right or wrong (i.e., getting feedback from reality), and iterating. Mentorship can substantially speed up this process by providing ["supervised data" to learn from](https://colah.github.io/notes/taste/), but either way you ultimately learn by doing.
|
||||
|
||||
I’ve erred towards making this post comprehensive, which may make it somewhat overwhelming. You do *not* need to try to remember everything in here! Instead think of it more as a guide for the high level things to keep in mind, and a source of advice for what to do at each stage. And, obviously, this is massively flavoured by my own subjective experience and may not generalise to you - I’d love to hear what other researchers think.
|
||||
|
||||
**A cautionary note:** Research is hard. Expect frustration, dead ends, and failed hypotheses. Imposter syndrome is common. Focus on the process and what you're learning. Take breaks, the total change to productive time is typically positive. Find sustainable ways to work. [Your standards are likely too high](https://www.neelnanda.io/blog/35-standards).
|
||||
|
||||
The key stages
|
||||
--------------
|
||||
|
||||
I see research as breaking down into a few stages:
|
||||
|
||||
1. **Ideation - Choose a problem/domain to focus on**
|
||||
2. **Exploration - Gain Surface area**
|
||||
1. **North star**: Gain information
|
||||
3. **Understanding - Test Hypotheses**
|
||||
1. **North star**: Convince *yourself* of a key hypothesis
|
||||
4. **Distillation - Compress, Refine, Communicate**
|
||||
1. **North star**: Compress your research findings into concise, rigorous truth that you can communicate to the world
|
||||
|
||||
### Ideation (Stage 1): Choose a problem
|
||||
|
||||
* This can vary from a long, high-effort exploration across areas looking for a promising angle, to just being handed a problem by a mentor.
|
||||
* Replicating and extending an existing paper can be a good starting point, especially if you don’t have an existing mentor.
|
||||
* This stage is crucial, but if you have a mentor (or other high quality source of suggestions, like someone else’s research agenda) it can be quick to just lean on them.
|
||||
* It's important to understand how your work fits into the existing literature: what is already known about the problem and what remains open.
|
||||
* Where possible, for your first project or two, lean on a mentor for guidance and just read a few key papers. Building deep knowledge of a literature takes time, and is easier once you have some hands-on experience.
|
||||
* Google/OpenAI Deep Research is invaluable for literature reviews, especially in unfamiliar domains.
|
||||
* Doing this well yourself and choosing a good problem often requires "**research taste**", and is the most commonly discussed aspect, but [is **just one facet of what research taste means**](/api/post/Ldrss6o3tiKT6NdMm) \- research taste also covers the following:
|
||||
* Exploration: **Noticing** **when an anomaly is interesting** and should be investigated, vs boring and to be ignored
|
||||
* Understanding: **Designing great experiments** that precisely distinguish hypotheses. This often stems from having a deep enough conceptual understanding to intuit *why* a hypothesis is true
|
||||
* Distillation: Having the taste to **identify the most interesting and defensible narrative**, and what to deprioritise.
|
||||
* On a broader level, I see research taste as being about an intuitive understanding of what good research looks like, to both guide high level strategy and tactical decisions in practice, informed by a deep understanding of the domain, familiarity with what good and bad research looks like, and the high level strategic picture of which problems actually matter.
|
||||
|
||||
### Exploration (Stage 2): Gain surface area
|
||||
|
||||
* *Examples:* [*My research streams*](https://www.youtube.com/watch?v=m8tzXelUTLo&list=PL7m7hLIqA0hr4dVOgjNwP2zjQGVHKeB7T), and [*my Othello research process write-up*](/api/sequence/nhGNHyJHbrofpPbRG/post/TAz44Lb9n9yf52pv8)
|
||||
* At the start, your understanding of the problem is often vague. Naively, it’s easy to think of research as being about testing specific hypotheses, but in practice you often start out not even knowing the right questions to ask, or the most promising directions. The exploration stage is about moving past this.
|
||||
* E.g. starting with “what changes in an LLM during chat fine-tuning?” or even “I’m sure there’s something interesting about how chat models behave, let’s mess around and find out”
|
||||
* **Your north star is just to gain information** \- do exploratory experiments, visualise data, follow your curiosity, prioritise moving fast.
|
||||
* Junior researchers often get stuck in the early stages of a project and don’t know what to do next. In my opinion this is because **they** **think they are in the understanding stage, but are actually in the exploration stage**.
|
||||
* That is, they think they ought to have a clear goal, and hypothesis, and obvious next step, and feel bad when they don’t. But this is totally fine and normal!
|
||||
* The solution is to have a toolkit of standard ways to gain surface area, brainstorm experiments that might teach something interesting, and be comfortable exploring a bunch and hoping something interesting happens.
|
||||
* Not having a clear goal/next step doesn’t mean that you don’t need to prioritise! **Prioritise for information gain**.
|
||||
* Try to do a lot of experiments (and don’t be a perfectionist about finding the ‘best’ experiments!), visualise things in many different ways, ensure you’re always learning.
|
||||
* Frequently ask yourself “**am I getting enough information per unit time**?” If you haven’t learned anything recently, shake it up.
|
||||
* Having fast feedback loops and powerful, flexible tooling is absolutely crucial here.
|
||||
* Note: **In the long-term exploration should feel like play** \- be fascinated by a problem, follow your curiosity, try to understand it deeply, zooming out when you get bored, etc (though it's still worth checking in on whether you're in a rabbit hole). But this isn't something you should worry about at first, as it needs well calibrated intuitions, which take time.
|
||||
* Note: often most of the work in the exploration was about **discovering the right kinds of questions to be asking**, e.g. that where information was stored is an important and interesting question, crystallising that into a precise hypothesis is often easy after that.
|
||||
* This both means ‘identify the right questions to ask’, but also gain a **deeper understanding and intuition of the domain** so you can design experiments that make sense, and **build a more gears-level model** of why a certain question may or may not be true.
|
||||
* A key practical tip is to **keep a highlights doc** of particularly interesting results, this makes it easier to spot connections
|
||||
|
||||
### Understanding (Stage 3): Test Hypotheses
|
||||
|
||||
* This stage begins when you understand the problem domain enough to **have some specific hypotheses that you think are interesting** \- hypotheses you can write down, and have some idea of what evidence you could find to show if they’re true or false.
|
||||
* E.g. “do chat models store summarised information about the user prompt in the <end\_of\_turn> special token?”
|
||||
* Your north star is to **gain evidence for and against these hypotheses**
|
||||
* Here the prioritisation is a mix of goal-directed and exploratory - you often need to briefly dip back into explore mode as you realise your hypothesis was ill-posed, your experiment didn’t make sense, you get weird and anomalous results, etc.
|
||||
* This stage is much closer to what people imagine when thinking about research.
|
||||
* Frequently ask yourself “**what am I learning and is it relevant?**”
|
||||
* The mark of a good researcher is a deep commitment to **skepticism of your results**.
|
||||
* You’ll have hypotheses that are wrong, experiments that are inconclusive, beautiful methods that lose to dumb baselines, etc. This is totally fine and normal, and a part of the natural process of science, but emotionally can be pretty hard to accept.
|
||||
* This *sounds* obvious, but in practice this requires constant active effort, and if you are not actively doing this you’ll inevitably fall into traps. Always seek alternative explanations, seek and implement strong baselines, check for bugs, etc.
|
||||
* A surprisingly deep and nuanced skill is **designing good experiments**. I think of this as one facet of “[research taste](/api/post/Ldrss6o3tiKT6NdMm)”
|
||||
* A great experiment elegantly, and conclusively distinguishes between several plausible hypotheses, validates non-trivial predictions made by one hypothesis, and is tractable to implement in practice.
|
||||
* This is an ideal rarely reached in practice but helpful to have in mind
|
||||
* My internal experience when generating good experiments is often that I try to simulate the world where hypothesis X is true, think through what this would mean and all the various implications of this, and notice if any can be turned into good experiments.
|
||||
* When reading papers, pay attention to the key experiments that their core claims hinge upon and ask yourself what made it important and how you might've thought of that experiment.
|
||||
|
||||
### Distillation (Stage 4): Compress, Refine, Communicate
|
||||
|
||||
* This stage begins when you have **enough evidence for** ***you*** **to be fairly convinced that your hypotheses are true/false**
|
||||
* The north star here is to **distill your research findings** into**concise, rigorous truth** that you can **communicate to the world**
|
||||
* **Compress** your work into some concrete, well-scoped claims - something you could list in a few bullet points. Compress it as far as you can without losing the message. Readers will not take away more than a few claims.
|
||||
* How would you explain your work to a peer? How would you write a lightning talk?
|
||||
* **Refine** your evidence into a rigorous case for each key claim, enough to be persuasive to a skeptical observer
|
||||
* This is persuasive in the sense of “actually provide strong evidence”, not just writing well enough that people don’t notice flaws! This means sanity checks, statistical robustness, and strong baselines.
|
||||
* Note that this is a higher bar than convincing yourself, both since you’re aiming for a more skeptical observer and you need to make all the key evidence you’ve seen legible to an outsider.
|
||||
* You should spend a lot of time on red-teaming here - what could you be missing? What alternative hypotheses could explain your observations? What experiments could distinguish between them? Etc
|
||||
* **Communicate** these with a clear and concise write-up - make clear what your points are, what evidence you provide, and its limitations. Write to inform, not persuade - if you are clear (a high bar), and your results are interesting, people will likely appreciate your work.
|
||||
* The form of write-up doesn’t really matter - Arxiv paper, blog post, peer-reviewed paper, etc. It doesn’t need to be polished, it just needs to present the evidence clearly, and to have strong enough evidence to meaningfully inform someone’s opinion
|
||||
* **People often under-rate this stage** and think doing the write-up is wasting time better spent on research, and can be left to the last minute. I think it’s actually a great use of time, at least for the first draft! I typically recommend my scholars make a start on distillation a month before conference deadlines.
|
||||
* Writing things up forces you to clarify your understanding to yourself. You also often notice holes and missing experiments. A common anecdote is that people didn’t really understand their project until they wrote it up.
|
||||
* If you don’t communicate your research well, it’s very hard to have an impact with it! (or to get recognition and career capital)
|
||||
* Conversely, **people often over-rate this stage** and default to writing a paper with the main goal of getting accepted to a conference. This has obvious advantages, but can also lead to warped thinking if you’re thinking about it from the start.
|
||||
* E.g. choosing questions that look good rather than being important, or focusing on forms of evidence that reviewers will like or understand, rather than ruthlessly focusing on actually establishing what’s true.
|
||||
* Sometimes you’ll discover that actually things are way messier than thought. It’s important to acknowledge this, rather than denying inconvenient truths! **Your ultimate goal is to find truth, not to produce an exciting paper**. You may need to go back to understanding or even exploration - this is totally fine and normal, and does not mean you’ve screwed anything up.
|
||||
|
||||
*Next up:* [*Post 2 of the sequence*](/api/post/cbBwwm4jW6AZctymL), on key research mindsets
|
||||
|
||||
@@ -3,42 +3,13 @@
|
||||
Source: https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/cbBwwm4jW6AZctymL
|
||||
Author: Neel Nanda
|
||||
Date: 27th Apr 2025
|
||||
Fetch-status: excerpted from LessWrong HTML via browser plus cross-checked against local shared draft.
|
||||
Fetch-status: full post text, fetched 2026-08-15 from the LessWrong markdown API (`/api/post/cbBwwm4jW6AZctymL`), comments and site navigation stripped. (CLAUDE agent)
|
||||
Use: research-process evidence for truth-seeking, prioritization, speed, and action under uncertainty.
|
||||
|
||||
## Why this matters for agents
|
||||
|
||||
This is the most directly agent-steering post in the sequence. It says the research process needs active skepticism, explicit prioritization, fast feedback loops, and the ability to act under uncertainty without waiting for a perfect next step.
|
||||
|
||||
## Quotes
|
||||
|
||||
> I think the most important mindsets are:
|
||||
> * Truth-seeking: By default, many research insights will be false - finding truth is hard. It’s not enough to just know this, you must put in active effort to be skeptical and resist bias, lest you risk your research being worthless.
|
||||
> * Prioritisation: You have finite time, and a lot of possible actions. Your project will live or die according to whether you pick good ones.
|
||||
> * Moving fast: You have finite time and a lot to do. This doesn’t just mean “push yourself to go faster” - there’s a lot of ways to eliminate inefficiency without sacrificing quality.
|
||||
|
||||
> Insufficient skepticism doesn't feel like insufficient skepticism from the inside. It just feels like doing research.
|
||||
|
||||
> This means that you must be putting in constant active effort into ensuring your results are robust. This must be integrated into part of your research process - if you’re not, then there’s a good chance your results are BS.
|
||||
|
||||
> The standard hypothesis testing framework can be misleading here, because it has an implicit frame of being able to list all the hypotheses. But actually, most of your probability mass should normally be on “something I haven’t thought of yet”.
|
||||
|
||||
> Here the Bayesian frame is often helpful. It’s generally overkill to put explicit numbers on everything, but it reminds me to ask the question “was this observation more likely under hypothesis A or B”, not just whether it was predicted by my favourite hypothesis.
|
||||
|
||||
> Fundamentally, good prioritisation is about having a clear goal (north star) in mind.
|
||||
|
||||
> The first step is just making time to stop and ask yourself “do I endorse what I’m doing, and could I be doing something better?”
|
||||
|
||||
> Prioritising and executing are different mental modes and should not be done simultaneously. Keep them separate, and make time to regularly reflect, and time to lock-in and execute on a plan without stressing about if it’s the best plan.
|
||||
|
||||
> Tight feedback loops are crucial: A key thing to track when doing research is your feedback loops.
|
||||
|
||||
> A corollary of this is that you should (often) do fast experiments first. It is far better to do a quick and dirty experiment to get some preliminary signs of life than an extremely long and expensive experiment that will produce conclusive data but only after weeks of work.
|
||||
|
||||
> Fail fast. One of the largest time sinks possible is investing weeks to months of effort into a failed research direction. Thus, a key question to ask yourself is: if this direction is doomed, how could I discover this as fast as humanly possible?
|
||||
|
||||
> A crucial mindset is being able to do something anyway, despite being so uncertain.
|
||||
|
||||
## Source graph
|
||||
|
||||
High-value links inside this post:
|
||||
@@ -47,3 +18,149 @@ High-value links inside this post:
|
||||
- Five-minute timers: https://www.neelnanda.io/blog/post-28-on-creativity-the-joys-of-5-minute-timers
|
||||
- Weekly review / reflection: https://www.neelnanda.io/blog/39-reflection
|
||||
- Jacob Steinhardt, Research as a Stochastic Decision Process: https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
|
||||
|
||||
## Full post
|
||||
|
||||
*This is post 2 of a sequence on my framework for doing and thinking about research.* [*Start here*](/api/sequence/5GT3yoYM9gRmMEKqL/post/hjMy4ZxS5ogA9cTYK).
|
||||
|
||||
Before I get into what exactly to do at each stage of the research process, it’s worth reflecting on the key mindsets that are crucial throughout the process, and how they should manifest at each stage.
|
||||
|
||||
I think the most important mindsets are:
|
||||
|
||||
* ***Truth-seeking***: By default, many research insights will be false - finding truth is hard. It’s not enough to just know this, **you must put in active effort to be skeptical and resist bias**, lest you risk your research being worthless.
|
||||
* ***Prioritisation***: You have finite time, and a *lot* of possible actions. **Your project will live or die according to whether you pick good ones.**
|
||||
* ***Moving fast***: You have finite time and a lot to do. This doesn’t just mean “push yourself to go faster” - **there’s a lot of ways to eliminate inefficiency without sacrificing quality**.
|
||||
* In particular, you must **learn to act without knowing the “correct” next step**, and avoid analysis paralysis.
|
||||
|
||||
**Warning**: It is extremely hard to be anywhere near perfect on one of these mindsets, let alone all three. I’m trying to describe an ideal worth aiming towards, but you should be realistic about the amount of mistakes you will make - I certainly am nowhere near the ideal on any of these! **Please interpret this post as a list of ideals to aim for, not something to beat yourself up about failing to meet.**
|
||||
|
||||
Truth Seeking
|
||||
-------------
|
||||
|
||||
Our ultimate goal in doing research is to uncover the truth about what’s really going on in the domain of interest. The truth exists, whether I like it or not, and being a good researcher is about understanding it regardless.
|
||||
|
||||
* This *sounds* pretty obvious. Who doesn't like truth? It’s easy to see this section, dismiss it as obvious and move on. But in practice this is extremely hard to achieve.
|
||||
* We have [many biases](https://en.wikipedia.org/wiki/List_of_cognitive_biases) that cut against finding truth
|
||||
* Insufficient skepticism doesn't *feel* like insufficient skepticism from the inside. It just feels like doing research.
|
||||
* This means that **you must be putting in constant active effort into ensuring your results are robust**. This **must be integrated into part of your research process** \- if you’re not, then there’s a good chance your results are BS.
|
||||
* “[Just try harder to be skeptical](https://www.neelnanda.io/blog/mini-blog-post-6-stop-pressing-the-try-harder-button)” is empirically a fairly ineffective strategy
|
||||
* One of the most common reasons I dismiss a paper is because I see a simple and boring explanation for the author’s observations, and they didn’t test for it - this often renders the results basically worthless.
|
||||
* I’d estimate that at least 50% of papers are basically useless due to insufficient skepticism
|
||||
|
||||
**What does putting in active effort actually mean**?
|
||||
|
||||
This takes different forms for the different stages:
|
||||
|
||||
* For exploration, the key failure mode is **not being creative enough when thinking about hypotheses**, getting attached to one or two ideas, and missing out on what’s actually going on.
|
||||
* Resist the urge to move on to the understanding stage the moment you have a plausible hypothesis - are there any unexplained anomalies? Could you do more experiments to gain more surface area first? What other hypotheses could explain your results? Etc
|
||||
* The standard hypothesis testing framework can be misleading here, because it has an implicit frame of being able to list all the hypotheses. But actually, most of your probability mass should normally be on “something I haven’t thought of yet”
|
||||
* You should regularly zoom out and look for alternative hypotheses for your observations. Asking another researcher, especially a mentor is a great source of perspective, asking LLMs is very cheap and can be effective.
|
||||
* That said, I still often find it helpful to think in a Bayesian way when doing research - if I have two hypotheses, how likely was some piece of evidence under each, and how should I update? Exploration often finds scattered pieces of inconclusive evidence, and there’s a skill to integrating them well.
|
||||
* **It’s not too bad if you end up believing false things for a bit**, the key thing is to move fast and reflexively try to falsify any beliefs you form, so you don’t get stuck in a rabbit hole based on false premises. This means it’s totally fine to investigate case studies and qualitative data, e.g. a deep dive into a single prompt.
|
||||
* If you’re getting lots of (diverse) information per unit time you’ll notice any issues.
|
||||
* **It is also an issue if you are** ***too*** **skeptical** and don’t let yourself explore the implications of promising but unproven hypotheses, as this is crucial to designing good experiments
|
||||
* For understanding, you want to be careful and precise about **what your experiments** ***actually*** **show you**, **alternative explanations** for your results, whether your **experiments make sense on a conceptual level**, etc.
|
||||
* Here the Bayesian frame is often helpful. It’s generally overkill to put explicit numbers on everything, but it reminds me to ask the question “**was this observation** ***more*** **likely under hypothesis A or B**”, not just whether it was predicted by my favourite hypothesis
|
||||
* In exploration it’s OK to be somewhat qualitative and case study focused, but here you want to be more quantitative. If you must do **qualitative case studies**, do them on **randomly sampled things**, (or at least several examples, if your sampling space is small) )since it’s *so* easy to implicitly cherry-pick
|
||||
* The one exception is if your hypothesis is “there exists at least one example of phenomenon X”, e.g. ‘[we found multidimensional SAE latents](https://arxiv.org/abs/2405.14860)’.
|
||||
* For distillation, in addition to the above, it’s important to **avoid the temptations of choosing a narrative that looks good**, rather than the best way to communicate the truth.
|
||||
* E.g. [**publishing**](https://arxiv.org/abs/2502.16681) [**negative**](https://arxiv.org/abs/2410.19278) [**results**](/api/post/4uXCAJNuPKtKBsi28)
|
||||
* While it can be emotionally hard to acknowledge to *myself* that my results are negative, mechanistic interpretability has a healthy culture and **I’ve gotten nothing but positive feedback for publishing negative results**.
|
||||
* E.g. **exaggerating results** or stating an **overconfident narrative** to seem more publishable.
|
||||
* I find it pretty easy to tell when a paper is doing this - generally you should care more about impressing the more experienced researchers in a field, who are least likely to be fooled by this! So I don’t even think it’s a good selfish strategy.
|
||||
* E.g. not acknowledging and discussing key limitations.
|
||||
* If I notice a key limitation that a paper has not addressed or acknowledged, I think far less of the paper.
|
||||
* If a paper discusses limitations, and provides a nuanced partial rebuttal, I think well of it.
|
||||
|
||||
Prioritisation
|
||||
--------------
|
||||
|
||||
Ultimately, time is scarce. The space of possible actions you can take when doing research is wide and open ended, and some are far more valuable than others. **The difference between a failed and a great research project is often prioritisation skill.** Improved prioritisation is one of the key sources of value I add as a mentor
|
||||
|
||||
* Fundamentally, good prioritisation is about having a clear goal (north star) in mind.
|
||||
* You need **good judgement** about how well different actions achieve this goal
|
||||
* You need to **actually make the time** to think about how well actions achieve this goal!
|
||||
* You need to **be ruthless** about dropping less promising directions where necessary.
|
||||
* But **beware switching costs** \- if you switch all the time without exploring anything properly you’ll learn nothing!
|
||||
* The goals at each stage are:
|
||||
* *Ideation:* **Choose a fruitful problem**
|
||||
* *Exploration*: **Gain information and surface area on the problem**
|
||||
* *Understanding*: **Find enough evidence to convince** ***you*** **of some key hypotheses**
|
||||
* *Distillation*: **Distill your research into concise, well-supported truth, and communicate this to the world.**
|
||||
* Being great at prioritisation is pretty difficult, and requires good research taste, which will take a lot of time to develop. But there’s often basic mistakes and low-hanging fruit to improve, if you just try.
|
||||
* The first step is just making time to stop and ask yourself “**do I endorse what I’m doing, and could I be doing something better**?”
|
||||
* This advice may seem obvious, but is deceptively hard to put into practice! You need regular prompts **Often it’s very easy to think of a better idea, but by default nothing prompts you to think.**
|
||||
* I like to **explicitly write goals down and regularly check in** that they’re being achieved - it sounds obvious, but you would be shocked at how effective it is to ask people if they’re doing the best thing for the project goals. I think in 3 tiers of goals:
|
||||
* Goal: What is the overall north star of the project? (generally measured in months)
|
||||
* Sub-goal: What is my current bit of the project working towards (measured in weeks)
|
||||
* Objective: What is the concrete short-term outcome I am aiming for right now (measured in days, e.g. 1 week)
|
||||
* I recommend **actually writing a plan**, and **estimate how long each step will take**, at least for the current research stage you’re in.
|
||||
* You don’t need to take it very seriously, and you’ll totally deviate a ton.
|
||||
* But **it forces you to think through the project**, notice uncertainties you could ask someone about, question if parts are really necessary to achieve your goals.
|
||||
* This is most important for understanding & distillation, though *can* be useful for exploration
|
||||
* **If you feel stuck,** [**set a 5 minute timer**](https://www.neelnanda.io/blog/post-28-on-creativity-the-joys-of-5-minute-timers) and brainstorm possible things you could do!
|
||||
* I typically wouldn’t spend more than a few hours on this
|
||||
* Unless you have a mentor giving high quality feedback - then it’s a great way to elicit their advice!
|
||||
* But even then, feel free to deviate - mentors typically have good research *priors*, but you know way more about your specific problem than them, which can be enough to make better decisions than even a very senior researcher
|
||||
* **You need to prioritise at many different layers of abstraction**, from deciding when to move on from an experiment to deciding which hypothesis to test first to deciding when to give up on testing a hypothesis and pivot to something else (or just back to exploration)
|
||||
* **Prioritising and executing are different mental modes and should not be done simultaneously**. Keep them separate, and make time to regularly reflect, and time to lock-in and execute on a plan without stressing about if it’s the best plan
|
||||
* Concrete advice: Work to a schedule where you regularly (ideally at least once a day, and with extended reflection at least once a week), zoom out and check that what you’re doing is your highest priority. E.g. work in pomodoros
|
||||
* **Having a** [**weekly review**](https://www.neelnanda.io/blog/39-reflection) **can be incredibly useful** \- where you zoom out and check in on what’s going on, any current issues, etc. Some useful prompts:
|
||||
* What is my goal right now?
|
||||
* What progress have I made towards that goal?
|
||||
* What’s consumed the most time recently?
|
||||
* What’s blocked me?
|
||||
* What mistakes have I made, and how could I systematically change my approach so it doesn’t happen again in future?
|
||||
* What am I currently confused about?
|
||||
* Am I missing something?
|
||||
* See Jacob Steinhardt’s [excellent blog post on research prioritisation](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html).
|
||||
* **Warning**: Different people need to hear different advice! (An eternal issue of writing public advice…). Some get stuck in rabbit holes and need to get better at moving on. Others get caught in analysis paralysis and never do *anything*, because they’re always waiting for the (non-existent) perfect opportunity.
|
||||
* **Real prioritisation is about a careful balance between exploration and exploitation**.
|
||||
* You probably know which failure mode you tend towards. **Please focus on the advice relevant to you, and ignore the rest**!
|
||||
|
||||
Moving Fast
|
||||
-----------
|
||||
|
||||
A core aspect of taking action in general is being able to move fast. Researchers vary a lot in their rate of productive output, and it gets very high in the best people - this is something I value a lot in potential hires.
|
||||
|
||||
This isn’t just about working long hours or cutting corners - there’s a lot of skill to **having fast feedback loops**, **noticing and fixing inefficiency** where appropriate, and **being able to take action or reflect where appropriate**. In some ways this is just another lens onto prioritisation.
|
||||
|
||||
* **Tight feedback loops are crucial**: A key thing to track when doing research is your feedback loops.
|
||||
* **Definition**: A **feedback loop** is the process from having an experiment idea and to results. Tight feedback loops are when the time taken is short.
|
||||
* It will make an enormous difference to your research velocity if you can get your feedback loops as tight as possible, and **this is a big priority**.
|
||||
* This is because you typically start a project confused, and **you need to repeatedly get feedback from reality to understand what’s going on**. This inherently requires a bunch of feedback loops that can’t be parallelised, so you want them to be as short as possible.
|
||||
* This is one of the big advantages of mech interp over other fields of ML - we can get much shorter feedback loops.
|
||||
* A mindset that I often find helpful is a deep-seated sense of impatience and **a feeling that something should be possible to do faster**. Sometimes I just need to accept that it will take a while, but often there is a better way, or at least a way that things can be reduced.
|
||||
* **Coding in a notebook is a lifesaver** (eg Jupyter, VS Code Interactive Mode or Colab)
|
||||
* Tips for tight feedback loops in mech interp:
|
||||
* Putting your data in a data frame rather than in a rigid plotting framework like Weights and Biases allows you to try arbitrary visualizations rapidly.
|
||||
* De-risking things on the smallest model you can, such as writing code and testing it on a small model before testing it on the model you're actually interested in.
|
||||
* Train things on fairly small amounts of data just to verify that you're seeing signs of life.
|
||||
* Sometimes there’s irreducible length, e.g. you need to train a model/SAE and this takes a while, but you can still often do something - train on less data, have evals that let you fail fast, etc.
|
||||
* **Good tooling accelerates everything**. All stages benefit from **flexible exploration tools** (e.g., interactive notebooks, libraries like TransformerLens or nnsight), efficient infrastructure for running experiments, and helpful utilities (e.g., plotting functions, data loaders).
|
||||
* Flexible tooling tightens feedback loops by shortening the time between an arbitrary creative experiment idea and results, even if it’s less efficient for any given idea.
|
||||
* The balance shifts: more flexibility needed early, more optimization/robustness potentially useful later e.g. during the distillation stage it can make sense to write a library to really easily do a specific kind of fine-tuning run that happens a ton
|
||||
* A corollary of this is that **you should (often) do fast experiments first**. It is far better to do a quick and dirty experiment to get some preliminary signs of life than an extremely long and expensive experiment that will produce conclusive data but only after weeks of work.
|
||||
* Realistically you should be prioritising by information gain per unit time.
|
||||
* This is especially important in exploration where it's hard to have a clear sense of which experiments are the most useful while estimating their tractability is pretty easy. When distilling you may know enough to be comfortable implementing a long running but conclusive experiment.
|
||||
* **Audit your time**. It's all well and good to talk about the importance of speed and moving fast, but how do you actually do this in practice? One thing that might be helpful is to log how you spend your time and then reflect on it, and ways you might be able to go faster next time.
|
||||
* For example, you could use a tool like [Toggl](https://toggl.com/) to roughly track what you're doing each day and then look back on how long everything took you and ask, "**How could I have done this faster**? Was this a good use of my time?"
|
||||
* Often it’s easy to fix inefficiencies and the hard part is noticing them - e.g. making a util function for a common tedious task, or noticing things that an LLM could automate.
|
||||
* Note: It is *not* productive to look back and feel really guilty about wasting time. **Nobody is perfect and you will always waste time**. I am advocating for maintaining a mindset of **optimism that you will be able to do even better next time**.
|
||||
* **Fail fast**. One of the largest time sinks possible is **investing weeks to months of effort into a failed research direction**. Thus, a key question to ask yourself is: if this direction is doomed, how could I discover this as fast as humanly possible?
|
||||
* I often try to think through what kind of confident predictions a hypothesis I care about makes in the understanding stage, or what fundamental assumptions make me think my domain is interesting at all in the exploration stage, and then think of the quickest and dirtiest experiments I can to test these.
|
||||
* It's often much better to have several quick and dirty experiments to attack different angles where you could fail fast than to put a lot of effort into one.
|
||||
* **Are you moving** ***too*****fast?** This is a natural pushback to the advice of ‘try hard to move fast’. It’s easy to e.g. be sloppy in the name of speed and introduce many bugs that cost you time in the long-run.
|
||||
* This is a hard balance, and I largely recommend just exploring and seeing how things go. But there *are* often things that can speed you up beyond ‘just push yourself to go harder in the moment’, which don’t have these trade-offs, like choosing the right experiments to run.
|
||||
* **Make sure you still regularly take time to think and reflect, rather than feeling pressure to constantly produce results**
|
||||
|
||||
### Taking action under uncertainty
|
||||
|
||||
A difficulty worth emphasising when trying to move fast is that there are a *lot* of possible next steps when doing research. And it’s pretty difficult to predict how they’ll go. Prioritisation remains crucial, but this means it’s also very hard, and **you will be highly uncertain about the best next step**. A crucial mindset is **being able to do something anyway, despite being so uncertain.**
|
||||
|
||||
* As a former pure mathematician, this is something I’ve struggled a fair bit with - I miss doing things grounded in pure, universal truth! But it’s learnable
|
||||
* Ultimately, you just need to accept on an emotional level that you don’t get to know the “right” answer for what to do next - **in practice, there’s no such thing as the right answer**.
|
||||
* The ideal is to strive to carefully evaluate the extremely noisy evidence, make a best guess for what to do next, and act on it, while also being self-aware enough to notice if it no longer seems the best action. This is a hard balance to achieve, but super useful if you can do it.
|
||||
* Especially when you’re starting out, this can be very low stakes: **the value of anything you do is dominated by the learning value**! If you make bad decisions you will learn and can do better next time, so it’s hard to really have a bad outcome.
|
||||
|
||||
*Next up:* [*post 3 of the sequence*](/api/post/Ldrss6o3tiKT6NdMm), on understanding & cultivating research taste
|
||||
|
||||
@@ -3,45 +3,13 @@
|
||||
Source: https://www.lesswrong.com/posts/Ldrss6o3tiKT6NdMm/my-research-process-understanding-and-cultivating-research
|
||||
Author: Neel Nanda
|
||||
Date: 1st May 2025
|
||||
Fetch-status: excerpted from LessWrong HTML via browser plus cross-checked against local shared draft.
|
||||
Fetch-status: full post text, fetched 2026-08-15 from the LessWrong markdown API (`/api/post/Ldrss6o3tiKT6NdMm`), comments and site navigation stripped. (CLAUDE agent)
|
||||
Use: core research-taste evidence, especially for deciding whether this should become a separate skill.
|
||||
|
||||
## Why this matters for agents
|
||||
|
||||
This post gives the boundary: research taste is not just picking ideas. It is judgment under long feedback loops across problem choice, exploration, experiment design, and distillation. It also explains why taste is learnable but slow: the feedback data is sparse.
|
||||
|
||||
## Quotes
|
||||
|
||||
> What is research taste? As I define it, research taste is far broader than just picking the right problem at the outset. Research is full of key decisions that will affect the future of the project, without an obvious way to find the right answer: from choosing the research problem itself, to identifying which anomalies are and are not worth exploring, distinguishing an experiment that will be compelling from one that’ll have inconclusive results, etc.
|
||||
|
||||
> I think of taste as the set of intuitions and good judgment that guide a researcher’s decisions throughout the research process, any time an ambiguous or open-ended decision like this arises.
|
||||
|
||||
> The core problem is you just don't get that much data. Generally the shorter a feedback loop is the more data you will get. By definition research taste is about things that are not immediately obvious.
|
||||
|
||||
> I think the main way to speed it up is by getting more data, and by being more sample efficient about the data that you have.
|
||||
|
||||
> When you have made a research decision and you eventually get feedback, do a post-mortem analyzing what did and did not work and why and what general themes you could look at in future.
|
||||
|
||||
> As discussed, I define research taste broadly: it's the collection of intuitions and judgments that guide good decision-making throughout a research project, especially where feedback loops are long, and the search space is large and open-ended.
|
||||
|
||||
> Exploration: A tactical sense for which experiments yield the most insight, recognizing interesting anomalies versus noise, knowing when to dig deeper or move on from a thread.
|
||||
|
||||
> Understanding: Designing creative, elegant experiments that cleanly distinguish hypotheses, judging the plausibility and explanatory power of different theories, identifying crucial assumptions or potential confounds.
|
||||
|
||||
> Communication & Distillation: Identifying the core, communicable claims within messy findings, structuring a compelling and true narrative, anticipating audience confusion, knowing what makes a result impactful to others.
|
||||
|
||||
> The ideal is strategic conviction: the ability to adopt a confident mindset to maintain momentum, while regularly zooming out to reflect and maintaining the capacity for zoomed-out skepticism and the willingness to update or abandon course based on evidence.
|
||||
|
||||
> Keep a research log. Ask why things worked or failed. Was it luck, execution, or a fundamental judgment call (taste)?
|
||||
|
||||
> Papers are a biased dataset (publication bias!), but still useful.
|
||||
|
||||
> Research taste isn't magic. It's a complex set of intuitions and frameworks built incrementally through experience, reflection, and learning from others. It governs the crucial, often implicit, decisions that shape a research project's success.
|
||||
|
||||
> Because the feedback loops for high-level strategic taste are long and noisy, don't expect to master it quickly. It's perfectly normal, and indeed expected, to rely heavily on external guidance (like mentors or established research directions) early in your career. Focus first on mastering the skills with shorter feedback loops – coding, running experiments, analyzing data, clearly communicating simple results.
|
||||
|
||||
> By actively engaging in research, deliberately reflecting on your decisions and their outcomes, and strategically leveraging the experiences of others, you can accelerate the development of your own research taste. Be patient with the process, especially the long-game aspects like problem selection. Trust that by doing the work and learning effectively from it, your intuition will improve over time.
|
||||
|
||||
## Source graph
|
||||
|
||||
High-value links inside this post:
|
||||
@@ -49,3 +17,115 @@ High-value links inside this post:
|
||||
- Weekly reviews: https://www.neelnanda.io/blog/39-reflection
|
||||
- Activation patching paper: https://arxiv.org/abs/2309.16042
|
||||
- Gears-level model reference: https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments
|
||||
|
||||
## Full post
|
||||
|
||||
*This is post 3 of a sequence on my framework for doing and thinking about research.* [*Start here*](/api/sequence/5GT3yoYM9gRmMEKqL/post/hjMy4ZxS5ogA9cTYK).
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
Spend enough time around researchers, and you'll hear talk of "research taste." It's often presented as a somewhat mystical quality distinguishing the seasoned research from the novice – an almost innate sense for which research ideas will flourish and which will fail. While I believe research taste is very real, incredibly valuable, and a key differentiator I look for, I *don't* think it's mystical or innate. Talent plays an important role, but taste is largely learned, and with the right mindset you can learn faster.
|
||||
|
||||
**What is research taste?** As I define it, research taste is far broader than just picking the right problem at the outset. Research is full of key decisions that will affect the future of the project, without an obvious way to find the right answer: from choosing the research problem itself, to identifying which anomalies are and are not worth exploring, distinguishing an experiment that will be compelling from one that’ll have inconclusive results, etc. I think of taste as **the set of intuitions and good judgment that guide a researcher’s decisions** ***throughout*** **the research process**, any time an ambiguous or open-ended decision like this arises. This can just be gut feeling, but also having conceptual frameworks you reason through, having novel ideas spark in your mind, etc.
|
||||
|
||||
**Where does taste come from?** If you're new to research, feeling like you lack "taste" is completely normal and expected. You don't need perfect judgment to start. In fact, trying to force it early on can be counterproductive. Think of training your intuition like training a network. It starts poorly initialized and needs lots of diverse, high-quality training data (i.e., research experience). With time, people often develop fairly deep and sophisticated taste, as they see enough examples of research outcomes, but this generally isn’t something people start with.
|
||||
|
||||
**How to learn it?** In my opinion, research taste is one of the hardest skills to learn for being a good researcher. To see why, let's lean more into this analogy of training a neural network. The core problem is **you just don't get that much data**. Generally the shorter a feedback loop is the more data you will get. By definition research taste is about things that are not immediately obvious. For designing a good experiment, sometimes you can get results from hours to day, but feedback on whether a research idea was good can take months!
|
||||
|
||||
I think the main way to speed it up is by **getting more data**, and by **being more sample efficient** about the data that you have. To get more data the easiest way is to **lean on sources of supervised data:** ideally **a mentor**, or **seeing what worked in papers**. You can also get more from each data point - analyse it in detail before setting the feedback, **predict your mentor’s answers before they give them**, etc. When you have made a research decision and you eventually get feedback, do a post-mortem analyzing what did and did not work and why and what general themes you could look at in future.
|
||||
|
||||
But even with all that, **expect learning taste to take a while**, especially high level strategic things like choosing a project - learning speed depends on your feedback loops, and taste has very slow ones. Further, research taste often translates poorly from other fields, or comes with counter-productive habits
|
||||
|
||||
What is Taste?
|
||||
==============
|
||||
|
||||
As discussed, I define **research taste** broadly: **it's the collection of intuitions and judgments that guide good decision-making throughout a research project,** especially where feedback loops are long, and the search space is large and open-ended.
|
||||
|
||||
I take such a broad definition, because I think that the ability to make good judgements is a fairly general skill, and improving at one facet often helps you improve at all of them, by e.g. getting better conceptual frameworks and domain knowledge.
|
||||
|
||||
While **Problem Selection** (strategic judgment about tractability and interest) is the most visible aspect, research taste also covers:
|
||||
|
||||
* **Exploration:** A tactical sense for which experiments yield the most insight, recognizing interesting anomalies versus noise, knowing when to dig deeper or move on from a thread. Does this surprising result feel like a key insight or a distracting artifact?
|
||||
* My internal experience here looks like a visceral science of excitement vs boredom or flinching away from messiness/ugliess. I tend to get excited about things that feel like unexpected structure, spark follow-up experiments, or relate to a deep curiosity I have.
|
||||
* **Understanding:** Designing creative, elegant experiments that cleanly distinguish hypotheses, judging the plausibility and explanatory power of different theories, identifying crucial assumptions or potential confounds. Is this experiment truly isolating the variable I care about? What's the simplest explanation for this data?
|
||||
* My internal experience is that I may have a beautiful hypothesis I *want* to believe, but it feels uncertain, and this creates an uncomfortable sense of instability.
|
||||
* I try to probe at where the instability comes from, what predictions are made by that potential flaw, and design an experiment to target it.
|
||||
* A good experiment design feels very clean and reliable - I would trust the results - while for a bad one I still have this shifting sense of uncertainty and being able to generate many alternative explanations
|
||||
* **Communication & Distillation:** Identifying the core, communicable claims within messy findings, structuring a compelling and *true* narrative, anticipating audience confusion, knowing what makes a result impactful *to others*. What's the single most important takeaway here? How can I present this evidence most clearly and honestly?
|
||||
* My internal experience of compression is about having a frustration and impatience with length and unnecessary conceptual detail - I want to distill the research down into what is truly important, and reach a point where I can cut no further without sacrificing something important.
|
||||
* If I’ve compressed too far, there’s a sense that there’s a missed opportunity - a really exciting thread that’s missed out.
|
||||
|
||||
Decomposing Research Taste
|
||||
--------------------------
|
||||
|
||||
Where does this "taste" come from? In my experience, it boils down to a few key ingredients:
|
||||
|
||||
1. **Intuition (System 1):** This is the fast, gut-level feeling - what people normally think of when they say research taste. A sense of curiosity, excitement, boredom, or skepticism about a direction, experiment, or result.
|
||||
1. "This feels promising," "This feels like a rabbit hole," "This anomaly seems *important*," "This explanation feels too simple/too complex."
|
||||
2. This is the part that feels most like "taste" and develops slowly through repeated exposure and feedback - when I refer to gathering data to train a network, I largely mean training your intuition.
|
||||
3. Empirically, my own recommendations based on this intuition have a decent hit rate, and experienced researchers are often fantastic (though not flawless!) at this, but this takes time.
|
||||
2. **Conceptual Framework (System 2):** This is deep domain knowledge and understanding of underlying principles.
|
||||
1. This is crucial in mech interp, especially as it’s a pre-paradigmatic field, where you can’t just memorise and apply a standard method.
|
||||
1. I’d guess it’s still important in other domains, though I am less sure
|
||||
2. For mech interp, this includes:
|
||||
1. Understanding transformer mechanics and basic facts - they’re autoregressive, the residual stream is the central object, tokens are discrete while all activations are continuous vectors, etc
|
||||
2. Key results and heuristics: like superposition or the linear representation hypothesis, or the idea that features and circuits exist at all
|
||||
3. Common techniques and where to use them and what they can tell you: patching, SAEs, probing, prompting, etc.
|
||||
1. This can get pretty deep! See [my paper on how to think about activation patching](https://arxiv.org/abs/2404.15255).
|
||||
4. Foundational knowledge of relevant adjacent fields: linear algebra, ML theory, training ML models, basic software engineering, etc
|
||||
3. This conceptual framework allows you to generate hypotheses, evaluate plausibility *explicitly*, spot inconsistencies, design sensible experiments, and explain *why* your intuition feels a certain way. It provides the structured reasoning to back up or override gut feelings.
|
||||
4. Eventually, this conceptual framework should feel like [a gears-level model](/api/tag/gears-level), where you can reason about the key moving parts, and what would make a project or experiment idea work vs fail vs be impractical.
|
||||
3. **Strategic Big Picture:** Understanding the broader context of the field. What problems are important? What are the major open questions? What approaches have been tried? What constitutes a novel contribution?
|
||||
1. My motivations for doing mech interp partly stem from making AGI safe, so the main big picture is “what work translates into better outcomes for AGI”, and being able to break this down into near-term steps.
|
||||
2. But even for less goal directed fields, where the goal is just curiosity driven basic science, there’s often a useful big picture around what advances would unlock many future advances or be a dead end, what would people care about, etc.
|
||||
3. Ideally, you dwell on the big picture enough that your intuitive sense of curiosity and excitement starts to integrate it - it’s not about overriding your curiosity with strategic obligations, it’s about aligning them so you’re excited about what matters. I see this as one input to prioritisation, among many.
|
||||
4. **Conviction & Confidence:** Research inevitably involves setbacks. A certain level of conviction – a belief in the direction, resilience to negative results – is often instrumentally useful for perseverance. It helps you push through the messy exploration phase or refine an idea that isn't working perfectly yet.
|
||||
1. Empirically, research taste also often leads to conviction - the intuitive feeling that an idea is exciting and important tends to also give motivation and focus.
|
||||
2. However, this is a **double-edged sword**. Your intuitions are not well calibrated. **Confidence doesn't mean correctness**. Generally people reach the level of having conviction far before they reach the level of having correct intuitions
|
||||
3. **The ideal is** ***strategic*** **conviction**: the ability to adopt a confident mindset to maintain momentum, while regularly zooming out to reflect and maintaining the capacity for zoomed-out skepticism and the willingness to update or abandon course based on evidence.
|
||||
4. **Track data**: Conviction is instrumentally useful, but so are correct beliefs. Generally, the best way to get calibrated is to pursue an exciting idea and see it fail in unexpected ways. Try to **write down prior predictions**, and *why* you think an idea is good, pursue it, and **reflect on what happened**.
|
||||
1. Corollary: **It’s fine to be uncalibrated at first**, this can help you get more research done and gather more data, if you’re paying attention you’ll often get over it.
|
||||
1. I often mentor people who start out by getting way too attached to flawed ideas, and don’t engage well with criticism. Seeing some of their exciting ideas fail tends to helps a lot.
|
||||
|
||||
**These components interact**. A strong conceptual framework sharpens intuition. Experience builds both intuition and framework knowledge. Strategic awareness helps channel conviction productively.
|
||||
|
||||
Cultivating Research Taste
|
||||
==========================
|
||||
|
||||
If taste is like an ML model, how can we speed up training? We want to improve the quantity (and quality) of data, and the sample efficiency of how much we learn from it.
|
||||
|
||||
* **Learning more from each data point**: You will learn something just from doing research. You'll get some feedback, some experience, and your intuitions and models will improve. But each data point is actually much richer than just a binary of success or failure!
|
||||
* My recommendation is to **make explicit predictions**, **review accuracy**, and make time to **reflect on what you missed** and how you could do better next time.
|
||||
* Keep a research log. Ask *why* things worked or failed. Was it luck, execution, or a fundamental judgment call (taste)?
|
||||
* **Reflect Deliberately:** After an experiment or project phase, ask: What worked? What didn't? What surprised me? What would I do differently next time? How does this update my model of this domain? ([Weekly reviews](https://www.neelnanda.io/blog/39-reflection) can be great for this).
|
||||
* **Getting more data**: The obvious source of data is doing research. But there are other sources too!
|
||||
* **Leverage Mentors:** This is perhaps the biggest accelerator. A mentor provides high-quality, curated "labels", insights and feedback. You can think of this as supervised data, in contrast to the slow RL of doing research yourself.
|
||||
* **Predict their advice:** Before asking your mentor ("Should I run experiment A or B?", "Is this result interesting?"), predict their answer and reasoning.
|
||||
* **Analyze surprises:** When their answer differs from your prediction, *dig into why*. What perspective, heuristic, or piece of knowledge did they use that you lacked? This is incredibly valuable training data for your internal model.
|
||||
* **Strong recommendation**: Do this by **repeatedly paraphrasing their reasoning**. Try to repeat back their arguments in your own words, and ask what you’re missing. This is an excellent way to ensure you’ve processed correctly, and often highlights misunderstandings. This is one of my most effective tactics when learning from people.
|
||||
* **Absorb their frameworks:** Listen not just to *what* they advise, but *how* they reason. What questions do they ask? What principles do they seem to operate by?
|
||||
* **Learn Critically from Papers (Offline Data):** Papers are a biased dataset (publication bias!), but still useful.
|
||||
* Read actively: Predict methods, results, and limitations before revealing them.
|
||||
* Ask *why*: Why did the authors make these choices? What alternative approaches might they have considered? What makes this paper impactful (or not)?
|
||||
* Focus on *reasoning*: Try to reconstruct the authors' thought process, not just memorize the outcome.
|
||||
* Note: **Papers are** ***very*** **often flawed**! A common mistake in new researchers is assuming that everything in a paper was reasonable or done for principled reasons. Even in great papers, there’s a lot of janky crap or flaws in there. And many papers are just inherently flawed or outright false. Critically engaging with a paper’s flaws is also very educational
|
||||
* **Collaborate and Discuss:** Talk to peers. Explain your research plans and reasoning. Listen to theirs. Critique each other's logic. Explaining forces clarity and exposes flawed assumptions. Hearing others' perspectives provides diverse 'data points'.
|
||||
* **Prioritize Projects with Clearer Feedback:** Especially early on, projects where you can test intermediate hypotheses or get partial results relatively quickly can accelerate learning more than moonshots with year-long feedback loops.
|
||||
* **Feedback loops**: The speed at which you complete each loop for each facet of taste determines how fast you learn that aspect.
|
||||
* **Short Loops/tactical taste:** Designing a specific experiment, debugging code, interpreting a single plot. Feedback is often quick (minutes to days). You'll likely improve *much* faster at skills with short feedback loops.
|
||||
* **Long Loops/strategic taste:** Choosing a research problem, deciding on a major strategic direction. Feedback might take months or even years. **Improvement here is inherently slower.**
|
||||
* **Implication:** Don't beat yourself up if your high-level strategic taste develops slower than your tactical experimental skills. This is expected.
|
||||
|
||||
I have less to say about other components of research taste like conceptual understanding or strategic picture - generally a similar mindset works there, though as it’s no longer really a black box I think it’s more straightforward, and is much easier to learn from reading papers and existing resources, and talking to mentors/experts. Conviction is more of a matter of personality and preference, in my experience.
|
||||
|
||||
Conclusion: Patience and Process
|
||||
================================
|
||||
|
||||
Research taste isn't magic. It's a complex set of intuitions and frameworks built incrementally through experience, reflection, and learning from others. It governs the crucial, often implicit, decisions that shape a research project's success.
|
||||
|
||||
Because the feedback loops for high-level strategic taste are long and noisy, don't expect to master it quickly. It's perfectly normal, and indeed expected, to rely heavily on external guidance (like mentors or established research directions) early in your career. Focus first on mastering the skills with shorter feedback loops – coding, running experiments, analyzing data, clearly communicating simple results.
|
||||
|
||||
By actively engaging in research, deliberately reflecting on your decisions and their outcomes, and strategically leveraging the experiences of others, you can accelerate the development of your own research taste. Be patient with the process, especially the long-game aspects like problem selection. Trust that by doing the work and learning effectively from it, your intuition will improve over time.
|
||||
|
||||
*Post 4, on ideation/choosing a research problem, is coming out soon - if you’re impatient you can read a draft of the whole sequence* [*here*](https://docs.google.com/document/d/1YMkeMrhqsWxZcNDD9CIUWEK_DAOegeufnbc79U2hycg/edit?tab=t.0).
|
||||
|
||||
@@ -3,40 +3,138 @@
|
||||
Source: https://colah.github.io/notes/taste/
|
||||
Author: Chris Olah
|
||||
Date: Posted Jan 9, 2021
|
||||
Fetch-status: excerpted from HTML via browser.
|
||||
Fetch-status: full post text, fetched 2026-08-15 via `curl https://r.jina.ai/https://colah.github.io/notes/taste/`, Disqus comment link stripped. (CLAUDE agent)
|
||||
Use: direct source for research-taste training exercises; cited by Nanda's shared draft and public taste post.
|
||||
|
||||
## Why this matters for agents
|
||||
|
||||
Olah gives concrete exercises for getting more feedback on taste without spending months executing every idea. This is a good reference for an agent asked to help a researcher improve project selection or calibrate idea quality.
|
||||
|
||||
## Quotes
|
||||
|
||||
> One of the most important aspects of growing as a researcher is developing research taste -- roughly, the ability to chose good problems to work on.
|
||||
|
||||
> I think the fundamental issue is that actually testing whether a research idea you come up with is good is very expensive. Often it takes months, so you only really get a few pieces of feedback on your taste every year.
|
||||
|
||||
> Many of the following exercises are really strategies for getting (proxy) feedback on more research ideas faster.
|
||||
|
||||
> Write down a list of research ideas. Have a mentor you respect rate each idea 1-10. Discuss ideas where you disagree with them after reflection.
|
||||
|
||||
> Pay attention when other people try ideas you’ve had. How did the results compare with your expectations?
|
||||
|
||||
> Interview researchers around you on their taste. Why do they work on the problems they do? How do they pick problems? What’s their “big picture” of research?
|
||||
|
||||
> Critically consider your research taste, and the community taste around you. Your taste is likely very influenced by your research cluster (your collaborators, advisor, etc).
|
||||
|
||||
> Failure Mode 1: Getting overly attached to one research direction / falling into sunk costs.
|
||||
|
||||
> Failure mode 2: Lack of research knowledge / intimacy.
|
||||
|
||||
> Theoretical knowledge is table stakes for research taste. You can’t have research taste in a vacuum.
|
||||
|
||||
> Failure mode 3: Environment not aligned with your interests.
|
||||
|
||||
## Source graph
|
||||
|
||||
Links visible in the post worth follow-up:
|
||||
- Hamming, You and Your Research: linked via YouTube.
|
||||
- Michael Nielsen, Principles of Effective Research: https://michaelnielsen.org/blog/principles-of-effective-research/
|
||||
- Andy Matuschak taste-related thread: linked as Twitter, may need archival route.
|
||||
|
||||
## Full post
|
||||
|
||||
Posted on Jan 9, 2021
|
||||
|
||||
This article is a rough note. Writing rough notes allows me share more content, since polishing takes lots of time. While I hope it's useful, it's likely lower quality and less carefully considered than my usual articles. It's very possible I wouldn't stand by this content if I thought about it more.
|
||||
|
||||
See also [twitter thread version](https://twitter.com/ch402/status/1347970296234479617).
|
||||
|
||||
One of the most important aspects of growing as a researcher is developing research taste -- roughly, the ability to chose good problems to work on. But it can be hard to explicitly work on developing taste, so I wanted to share some concrete, actionable exercises.
|
||||
|
||||
Before we start with exercises, it's worth think about why it's hard to develop research taste. I think the fundamental issue is that actually testing whether a research idea you come up with is good is very expensive. Often it takes months, so you only really get a few pieces of feedback on your taste every year. Many of the following exercises are really strategies for getting (proxy) feedback on more research ideas faster. The feedback you get is generally lower quality than actually executing the idea yourself, but these aim to get you orders of magnitude more of this noisy feedback.
|
||||
|
||||
Take all these exercises and suggestions with a grain of salt. I think I have better taste than I used to and doing some things like this helped. I also think these exercises have also helped the occasional person I mentored or managed. But I'm hardly an expert. If you have better ways to build resarch taste, please share them in the comments or [on twiter!](https://twitter.com/ch402/status/1347970296234479617)
|
||||
|
||||
One other point: I don't recommend forcing yourself to work on something you don't find exciting, even if you intellectually think it's more important. You'll be a lot less effective. Ideally, you want to get to a point where the problems you think are most important also feel most exciting.
|
||||
|
||||
## Exercises
|
||||
|
||||
_**Exercise 1:** Write down a list of research ideas. Have a mentor you respect rate each idea 1-10. Discuss ideas where you disagree with them after reflection._
|
||||
|
||||
It often takes several months of work to truly test an idea. Asking a mentor is a cheap proxy.
|
||||
|
||||
_**Exercise 2:** Pay attention when other people try ideas you’ve had. How did the results compare with your expectations?_
|
||||
|
||||
If you’re investing in brainstorming, you’ll have tens or hundreds of research ideas for each one you try. It can be emotionally uncomfortable when someone publishes an idea you had, but it’s an opportunity to get precious feedback for free.
|
||||
|
||||
_**Exercise 3:** Interview researchers around you on their taste. Why do they work on the problems they do? How do they pick problems? What’s their “big picture” of research?_
|
||||
|
||||
Bonus: If you have a great interview, you might consider writing it up with their permission.
|
||||
|
||||
(PhD students often have short meetings with researchers visiting their lab. One pattern is pitching their project to the visitor, which may not be very productive if they don't have similar interests. I suspect they’d get more out of talking about meta-research.)
|
||||
|
||||
_**Exercise 4:** Read books about the history of science. Reflect on why some researchers focused on important directions their contemporaries ignored._
|
||||
|
||||
(Kuhn’s “The Structure of Scientific Revolutions” is one of my favorite books, and you can get an audio book!)
|
||||
|
||||
_**Exercise 5:** Critically consider your research taste, and the community taste around you. Your taste is likely very influenced by your research cluster (your collaborators, advisor, etc)._
|
||||
|
||||
In what ways has your own research taste or your community's taste been wrong over the last few years?
|
||||
|
||||
Are there adjacent research “schools” with significantly different research taste? If so, try to articulate the strongest version of their view, and why you agree or disagree.
|
||||
|
||||
In what ways to you disagree with the research taste of your own community? What are ways in which you think your community might be wrong even if you aren't confident in that view?
|
||||
|
||||
What are the underlying goals of your research taste? This could simply be intrinsically wanting to understand, or just finding research fun. But there might also be ways you want to make the world a better place. Are the problems you are working on (or think are intersting) algined with those goals?
|
||||
|
||||
## Failure Modes
|
||||
|
||||
I also wanted to go through some research taste failure modes:
|
||||
|
||||
_**Failure Mode 1:** Getting overly attached to one research direction / falling into sunk costs._
|
||||
|
||||
A lot of researchers start working on one problem (often more due to circumstance than contemplated decision) and then find it hard to move on.
|
||||
|
||||
**Potential Antidote:** Set aside a week or two to step back and create a list of the other research directions you think would be most promising to work on (look at new research, talk to others, etc). Then ask what you’d want to work on if you were starting from scratch. Repeat every 1-2 years as needed.
|
||||
|
||||
_**Failure mode 2:** Lack of research knowledge / intimacy._
|
||||
|
||||
Theoretical knowledge is table stakes for research taste. You can’t have research taste in a vacuum.
|
||||
|
||||
Sometimes people fall into a trap of trying to pick the perfect problem or have a brilliant insight before getting their hands dirty. You can’t have research taste in a vacuum. You need theoretical knowledge and [research intimacy](https://twitter.com/ch402/status/1295063684545314818).
|
||||
|
||||
**Potential Antidote:** Help someone else with their project in a space you’re interested in. You’ll learn a lot and get your hands dirty without becoming overly committed to working on something long term. Ideally, by not leading the project, this should also be lower stress. Alternatively, do your own short-term projects to get your hands dirty.
|
||||
|
||||
_**Failure mode 3:** Environment not aligned with your interests._
|
||||
|
||||
I sometimes talk to PhD students who have different interests than their advisor, and are trying to contort their research interests into something their advisor will find palatable. This generally leads to ideas optimized for compromise rather than quality. This leads to suboptimal resarch, and starves you of an opportunity to build taste.
|
||||
|
||||
This can happen even when those around you nominally support you working on whatever you’re interested in. It can be challenging to pursue a direction when people around you are unenthusiastic.
|
||||
|
||||
**Potential Antidote:** Either (1) move to an environment which is aligned with your interests, or (2) temporarily buy into the interests of your group and try to exercise good taste within those interests. I’m generally pro moving to another environment once you’ve reflected a lot on what you’re interested in and have something you feel very actively excited about.
|
||||
|
||||
## Suggestions from other people
|
||||
|
||||
In the [twitter discussion](https://twitter.com/ch402/status/1347970296234479617) (and especially, a [thread by Andy Matuschak](https://twitter.com/andy_matuschak/status/1348001750221946882)), a number of people suggested other ideas. Note all of these are quite "exercises for building taste" (many are more "strategies for exercising good taste"), but I think they're very interesting:
|
||||
|
||||
* [**Rachel Prudden suggests**](https://twitter.com/RachelPrudden/status/1347973143990829056)
|
||||
|
||||
_The heuristic of imagining that another group has published the paper you have in mind. Are you excited to read it?_
|
||||
|
||||
Chris comment: I think this is a pretty good heuristic.
|
||||
* [**Andy Matuschak suggests**](https://twitter.com/andy_matuschak/status/1348001750221946882)
|
||||
|
||||
_ask the anti-Hamming question about your ideas. What are the most interesting (not important!) problems in your field? Why aren't you working on them?_
|
||||
|
||||
Chris comment: This is a great question. See also Hamming's famous ["You and Your Research"](https://www.youtube.com/watch?v=a1zDuOPkMSw) talk.
|
||||
* [**Andy also shares a failure mode:**](https://twitter.com/andy_matuschak/status/1348002769085419520)
|
||||
|
||||
_running to make an idea happen as soon as it seems tractable. It's exciting to have a viable approach—makes me wanna go do it! But if I sit with an idea longer I can often evolve a deeper variant and run with that instead._
|
||||
* **Michael Nielsen's [Principles of Effective Resarch](https://michaelnielsen.org/blog/archive/000114.html) ([via Shrey Jain](https://twitter.com/shreydjain13/status/1348023976275161089))** notes the value of paying attention to messes:
|
||||
|
||||
_When you identify such a mess, the natural inclination of many people is to shy away, to find something that is easier to understand. But a field that is a mess is really an opportunity. Chances are good that there are deep unifying and simplifying concepts still waiting to be understood and developed by someone - perhaps you._
|
||||
* [**Arthur Allshire**](https://twitter.com/arthurallshire/status/1348024259214516231)
|
||||
|
||||
_What are the bottlenecking problems in your field? Eg. in a lot of reinforcement learning research, a lot focus on slight tweaks to algorithms, but the biggest improvements have come from improving simulation for RL._
|
||||
* [**Preetum Nakkiran**](https://twitter.com/PreetumNakkiran/status/1348049533436129281)
|
||||
|
||||
_I've found that writing "project proposals" in the short term, and "research statements/plans" in the long term, are good exercises to think about motivations & goals (at diff timescales). By "project proposal", I mean: Once you decide to commit the next >1 month of time to something, write up a few pages explaining: why the question is interesting, why the answer will be valuable, and what you expect to find (w/ initial evidence). And then get feedback.
|
||||
|
||||
... Also seconding the advice to ask everyone you meet about their "big picture" research goals. This is almost never written in papers, but is much more interesting/important._
|
||||
* [**Christian Szegedy**](https://twitter.com/ChrSzegedy/status/1348055991565262849)
|
||||
|
||||
_In hindsight though, most of the errors are made in my research was when I listened to the criticisms of my respected mentors and most of the cool stuff I did were the ones those everybody called initially useless, uninteresting or "good luck!"... Generally, ideas in research need to be either ahead of the others or should defy conventional wisdom. Being in the latter camp requires less skill, but more courage._
|
||||
|
||||
Chris comment: I think there's a lot of truth to this, but beware survivorship bias and remember that different people may have different levels of scientific maturity / taste at different points in their careers. I suspect that many PhD students would find charting their own research agenda from the start very challenging, while others would thrive.
|
||||
* [**Sergey Gorbunov**](https://twitter.com/sergey_nog/status/1348049950630928387)
|
||||
|
||||
_My only advice on research taste to grad students or junior researchers is *not* to listen to your advisors/mentors/established researchers... Grad school is the only place when you get to try and fail as many times as needed. Grad students should come up with outside the box problems/solutions. (Established researches have their own agendas/way of thinking😂). The advice can similarly be used to overturn itself. 🧐_
|
||||
* [**Boaz Barak**](https://twitter.com/boazbaraktcs/status/1348097776551407618)
|
||||
|
||||
_I’m fond of “20/80 projects” where you do 80% of work to get 20% of goals:
|
||||
|
||||
Rather than asking “what’s coolest result I can get with smallest effort?” ask “can I find problem that distills the main obstacle even if end result doesn’t sound cool?_
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
I'm grateful to the countless people who've talked to me about research philosophy over the years.
|
||||
|
||||
Thanks to [Nick Cammarata](http://nickcammarata.com/) for discussion of these exercises.
|
||||
|
||||
Thanks to Ryan Saxe and Abinav for correcting typos.
|
||||
|
||||
@@ -3,33 +3,156 @@
|
||||
Source: https://80000hours.org/articles/ml-engineering-career-transition-guide/
|
||||
Authors: Catherine Olsson and the 80,000 Hours team
|
||||
Date: Published November 2018; update note visible Feb 2022
|
||||
Fetch-status: excerpted from HTML via browser.
|
||||
Fetch-status: full article text, fetched 2026-08-15 via `curl https://r.jina.ai/https://80000hours.org/articles/ml-engineering-career-transition-guide/`. (CLAUDE agent)
|
||||
Use: source-graph evidence from Spinning Up's "Other Resources" section; useful for research-engineer skill acquisition, less central to research taste.
|
||||
|
||||
## Why this matters for agents
|
||||
|
||||
This source is more about becoming useful on ML research teams than choosing research ideas. Its most relevant claim is that implementing and debugging foundational algorithms is a high-value learning path, with easy environments, metrics, and reference-code scrutiny.
|
||||
|
||||
## Quotes
|
||||
|
||||
> Technical AI safety is a multifaceted area of research, with many sub-questions in areas such as reward learning, robustness, and interpretability.
|
||||
|
||||
> Not all of these questions are best tackled with abstract mathematics research; some can be approached with concrete coding experiments and machine learning (ML) prototypes.
|
||||
|
||||
> Once you know the 101-level basics of ML, the next thing to learn is how to implement and debug ML algorithms.
|
||||
|
||||
> Breadth of experience is not important here: you don’t need to read all the latest papers, or master an extensive reading list. You also don’t need to do novel research or come up with new algorithms.
|
||||
|
||||
> What you do need is to get your hands dirty implementing and debugging ML algorithms, and to build evidence for job interviews that you have some experience doing this.
|
||||
|
||||
> The most straightforward way to gain this experience is to choose a subfield of ML relevant to a lab you’re interested in. Then read a few dozen of the subfield’s key papers, and reimplement a few of the foundational algorithms that the papers are based on or reference most frequently.
|
||||
|
||||
> For each algorithm, they would first test on very easy environments, and then move to more difficult environments.
|
||||
|
||||
> Once the algorithm was partially working, they would attain higher performance by looking for remaining bugs, both by reviewing the code carefully, and by collecting metrics such as average policy entropy to perform sanity-checks, rather than just tune hyperparameters.
|
||||
|
||||
> Most importantly, he was able to implement and debug ML algorithms, going from math in a paper to running code.
|
||||
|
||||
## Source graph
|
||||
|
||||
This page was linked from Spinning Up's "Other Resources" section. It points to Josh Achiam's Key Papers in Deep RL list and a Daniel Ziegler self-study path. It is useful background for training agents to value implementation and debugging practice, but probably secondary for a dedicated research-taste skill.
|
||||
|
||||
## Full article
|
||||
|
||||
Technical AI safety is a multifaceted area of research, with many sub-questions in areas such as reward learning, robustness, and interpretability. These will all need to be answered in order to [make sure AI development will go well for humanity](https://80000hours.org/problem-profiles/positively-shaping-artificial-intelligence/) as systems become more and more powerful.
|
||||
|
||||
Not all of these questions are best tackled with abstract mathematics research; some can be approached with concrete coding experiments and machine learning (ML) prototypes. As a result, some AI safety research teams are looking to hire a growing number of Software Engineers and ML Research Engineers.
|
||||
|
||||
Additionally, some research teams that may not think of themselves as focussed on ‘AI Safety’ per se, nonetheless work on related problems like verification of neural nets or learning from human feedback, and are often hiring engineers.
|
||||
|
||||
> _Note that this guide was written in November 2018 to complement [**an in-depth conversation on the 80,000 Hours Podcast with Catherine Olsson and Daniel Ziegler**](https://80000hours.org/podcast/episodes/olsson-and-ziegler-ml-engineering-and-safety/) on how to transition from computer science and software engineering in general into ML engineering, with a focus on alignment and safety. If you like this guide, we’d strongly encourage you to check out the podcast episode where we discuss some of the instructions here, and other relevant advice._
|
||||
|
||||
_Update Feb 2022: The need for software engineers in AI safety seems even greater today than when this post was written (e.g. see [this post](https://forum.effectivealtruism.org/posts/DDDyTvuZxoKStm92M/ai-safety-needs-great-engineers) by Andy Jones). You also don’t need as much knowledge of AI safety to enter the field as this guide implies._
|
||||
|
||||
Table of Contents
|
||||
|
||||
* [1 What are the necessary qualifications for these positions?](https://80000hours.org/articles/ml-engineering-career-transition-guide/#what-are-the-necessary-qualifications-for-these-positions)
|
||||
* [2 How can I best learn Machine Learning engineering skills if I don’t yet have the necessary experience?](https://80000hours.org/articles/ml-engineering-career-transition-guide/#how-can-i-best-learn-machine-learning-engineering-skills-if-i-dont-yet-have-the-necessary-experience)
|
||||
* [2.1 Initial investigation](https://80000hours.org/articles/ml-engineering-career-transition-guide/#initial-investigation)
|
||||
* [2.2 ML basics](https://80000hours.org/articles/ml-engineering-career-transition-guide/#ml-basics)
|
||||
* [2.3 Learn ML implementation and debugging, and speak with the team you want to join](https://80000hours.org/articles/ml-engineering-career-transition-guide/#learn-ml-implementation-and-debugging-and-speak-with-the-team-you-want-to-join)
|
||||
* [2.4 Case study: Daniel Ziegler’s ML self-study experience](https://80000hours.org/articles/ml-engineering-career-transition-guide/#case-study-daniel-zieglers-ml-self-study-experience)
|
||||
|
||||
* [3 Now apply for jobs](https://80000hours.org/articles/ml-engineering-career-transition-guide/#now-apply-for-jobs)
|
||||
* [4 Learn more](https://80000hours.org/articles/ml-engineering-career-transition-guide/#learn-more)
|
||||
|
||||
## What are the necessary qualifications for these positions?
|
||||
|
||||
**Software Engineering:** Some engineering roles on AI safety teams do _not_ require ML experience. You might already be prepared to apply to these positions if you have the following qualifications:
|
||||
|
||||
* BSc/BEng degree in computer science or another technical field (or comparable experience)
|
||||
* Strong knowledge of software engineering (as a benchmark: could pass a Google software engineering interview)
|
||||
* Interest in working on AI safety
|
||||
* (usually) Willingness to move to London or the San Francisco Bay Area
|
||||
|
||||
If you’re a software engineer with an interest in these roles, you may not need any additional preparation, and may be ready to _[apply right away](https://jobs.80000hours.org/?refinementList%5Btags\_area%5D%5B0%5D=AI+safety+%26+policy&refinementList%5Btags\_skill%5D%5B0%5D=Software+engineering&jb\_source=articles\_\_ml-engineering-career-transition-guide)_.
|
||||
|
||||
**ML Engineering and/or Research Engineering**: Some roles require experience implementing and debugging machine learning algorithms. If you don’t yet have ML implementation experience, you may be able to learn the necessary skills quickly, so long as you’re willing to spend a few months studying. Before deciding to do this, you should check that you meet all the following criteria:
|
||||
|
||||
* BSc/BEng degree in computer science or another technical field (or comparable experience)
|
||||
* Strong knowledge of software engineering (as a benchmark: could pass a Google software engineering interview)
|
||||
* Interest in working on AI safety
|
||||
* (usually) Willingness to move to London or the San Francisco Bay Area
|
||||
|
||||
## How can I best learn Machine Learning engineering skills if I don’t yet have the necessary experience?
|
||||
|
||||
### Initial investigation
|
||||
|
||||
Implementing and debugging ML algorithms is different from traditional software engineering. The following can help you determine whether you’ll like the day-to-day work:
|
||||
|
||||
* Matthew Rahtz’s blog post [Lessons Learned Reproducing a Deep Reinforcement Learning Paper](http://amid.fish/reproducing-deep-rl)
|
||||
* S. Zayd Enam’s blog post [Why is machine learning “hard”?](http://ai.stanford.edu/~zayd/why-is-machine-learning-hard.html)
|
||||
|
||||
### ML basics
|
||||
|
||||
If you don’t have any experience in machine learning, start by familiarizing yourself with the basics. If you have _some_ experience, but haven’t done a hands-on machine learning project recently, it’s also probably a good idea to brush up on the latest tools (writing TensorFlow, starting a virtual machine with a GPU, etc).
|
||||
|
||||
Although it can be difficult to find time for self-study if you’re already employed full-time or have other responsibilities, it’s far from impossible. Here are some ideas of how you might get started:
|
||||
|
||||
* Consider spending a few hours a week on an online course. We recommend either of these two:
|
||||
* The [fast.ai](http://course.fast.ai/) online course, “Practical Deep Learning For Coders, Part 1”
|
||||
* Google’s [ML Crash Course](https://developers.google.com/machine-learning/crash-course/ml-intro)
|
||||
|
||||
* If you’re employed full-time in a software engineering role, you might be able to learn ML basics without leaving your current job:
|
||||
* If you’re at a large tech company, take advantage of internal trainings, including full-time ML rotation programs.
|
||||
* Ask your manager if you can incorporate machine learning into your current role: for example, to spend 20% of your time learning ML, to see if it could improve one of the projects you work on.
|
||||
|
||||
For simple ML problems, you can get pretty far just on CPU on your laptop, but for larger problems it’s useful to buy a GPU and/or rent some cloud GPUs. You can often get some cloud computing credits through a [free trial](https://cloud.google.com/free/), [educational credits](https://aws.amazon.com/education/awseducate/) for students, or asking a friend with a startup.
|
||||
|
||||
### Learn ML implementation and debugging, and speak with the team you want to join
|
||||
|
||||
Once you know the 101-level basics of ML, the next thing to learn is how to _implement_ and _debug_ ML algorithms. (Based on the experiences of others in the community who have taken this path, we expect this to take at minimum 200 hours of focused work, and likely more if you are starting out with less experience).
|
||||
|
||||
Breadth of experience is not important here: you don’t need to read all the latest papers, or master an extensive reading list. You also don’t need to do novel research or come up with new algorithms. Nor do you need to focus on safety at this stage; in fact, focusing on well-known and established ML algorithms is probably better for your learning.
|
||||
|
||||
What you _do_ need is to get your hands dirty implementing and debugging ML algorithms, and to build evidence for job interviews that you have some experience doing this.
|
||||
|
||||
You should strongly consider _contacting the teams you’re interested in_ at this stage. Send them an email with the specifics of what you’re planning on spending your time on to get feedback on it. The manager of the team may suggest specific resources to use, and can help you avoid wasting time on extraneous skills you don’t need for the role.
|
||||
|
||||
The most straightforward way to gain this experience is to choose a subfield of ML relevant to a lab you’re interested in. Then read a few dozen of the subfield’s key papers, and reimplement a few of the foundational algorithms that the papers are based on or reference most frequently. Potential sub-fields include the following:
|
||||
|
||||
* Deep reinforcement learning
|
||||
* Defenses against adversarial examples
|
||||
* Verification and robustness proofs for neural nets
|
||||
* Interpretability & visualization
|
||||
|
||||
If it isn’t clear how to get started – for example, if you don’t have access to a GPU, or don’t know how to write TensorFlow – many of the resources in the “basics” section above have useful tips.
|
||||
|
||||
If you need to quit your job to make time for learning in this phase, but don’t have enough runway to self-fund your studies, consider applying for an [EA grant](https://www.effectivealtruism.org/grants/) when it next opens – they are open to funding career transitions such as this one.
|
||||
|
||||
### Case study: Daniel Ziegler’s ML self-study experience
|
||||
|
||||
In January 2018, Daniel had strong software engineering skills but only basic ML knowledge. He decided that he wanted to work on an AI safety team as a research engineer, so he talked to Dario Amodei (the OpenAI Safety team lead). Based on Dario’s advice, Daniel spent around six full-time weeks diving into deep reinforcement learning together with a housemate. He also spent a little time reviewing basic ML and doing supervised learning on images and text. Daniel then interviewed and became an ML engineer on the safety team.
|
||||
|
||||
Daniel and his housemate used Josh Achiam’s [Key Papers in Deep RL](https://docs.google.com/document/d/1t55CCHabmHmrJ1VRoOSmQmR9lHkClAPxOlYGuvmof2Q/edit) list to guide their efforts. They got through about 20-30 of those papers, spending maybe 1.5 hours independently reading and half an hour discussing each paper.
|
||||
|
||||
More importantly, they implemented a handful of the key algorithms in TensorFlow:
|
||||
|
||||
* Q-learning: DQN and some of its extensions, including prioritized replay and double DQN
|
||||
* Policy gradients: A2C, PPO, DDPG
|
||||
|
||||
They applied these algorithms to try to solve various [OpenAI Gym](https://github.com/openai/gym) environments, from the simple ‘Cartpole-v0’ to Atari games like ‘Breakout-v4’.
|
||||
|
||||
They spent 2-10 days on each algorithm (in parallel as experiments ran), depending on how in-depth they wanted to go. For some, they only got far enough to have a more-or-less-working implementation. For one (PPO), they tried to fix bugs and tune things for long enough to come close to the performance of the OpenAI Baselines implementation.
|
||||
|
||||
For each algorithm, they would first test on very easy environments, and then move to more difficult environments. Note that an easy environment for one algorithm may not be easy for another: for example, despite its simplicity, the Cartpole environment has a long time horizon, which can be challenging for some algorithms.
|
||||
|
||||
Once the algorithm was partially working, they would attain higher performance by looking for remaining bugs, both by reviewing the code carefully, and by collecting metrics such as average policy entropy to perform sanity-checks, rather than just tune hyperparameters. Finally, when they wanted to match the performance of Baselines, they scrutinized the Baselines implementations for small important details, such as exactly how to preprocess and normalize observations.
|
||||
|
||||
By the end of six weeks, Daniel was able to talk fluently about the key ideas in RL and the tradeoffs between different algorithms. Most importantly, he was able to implement and debug ML algorithms, going from math in a paper to running code. In retrospect, Daniel reports wishing he had spent a little more time on ML conceptual & mathematical fundamentals, but that overall this process prepared Daniel well for the interview and the role, and was particularly well-suited for OpenAI’s focus on reinforcement learning.
|
||||
|
||||
## Now apply for jobs
|
||||
|
||||
_These positions will eventually be filled, but you can find a constantly updated list of some of the most promising positions on the [80,000 Hours job board](https://80000hours.org/job-board/ai-ml-safety-research/?role-type=engineering)._
|
||||
|
||||
The following example job postings for software engineers on AI safety research teams specify that machine learning experience is _not_ required:
|
||||
|
||||
* OpenAI’s safety team is currently hiring a [software engineer](https://openai.com/jobs/#open) for a range of projects, including interfaces for human-in-the-loop AI training and collecting data for larger language models. (_Update: this job posting is now closed._)
|
||||
* MIRI is hiring [software engineers](https://intelligence.org/careers/software-engineer/).
|
||||
* Ought is hiring [research engineers](https://ought.org/careers/research-engineer?utm_campaign=80000+Hours+Job+Board&utm_source=80000+Hours+Job+Board) with a focus on candidates who are excited by functional programming, compilers, program analysis, and related topics.
|
||||
|
||||
The following example job postings _do_ expect experience with machine learning implementation:
|
||||
|
||||
* DeepMind is hiring [research engineers](https://deepmind.com/careers/jobs/1433588?utm_campaign=80000%20Hours%20Job%20Board&utm_source=80000%20Hours%20Job%20Board) for their _Technical AGI Safety_ team, _Safe and Robust AI_ team – which works on neural net verification and robustness – and potentially others as well.
|
||||
* Google AI is hiring [research software engineers](https://careers.google.com/jobs#t=sq&q=j&li=20&l=false&jlo=en-US&jcoid=7c8c6665-81cf-4e11-8fc9-ec1d6a69120c&jcoid=e43afd0d-d215-45db-a154-5386c9036525&j=research+engineer&) in locations worldwide. Although Google AI does not have an “AI Safety” team, there are research efforts focused on robustness, security, interpretability, and learning from human feedback.
|
||||
* OpenAI’s safety team is hiring [machine learning engineers](https://jobs.lever.co/openai/a0d3b158-14a0-48db-b38c-1c94bb18f69b) to work on alignment and interpretability.
|
||||
* The Center for Human Compatible AI at Berkeley is hiring [machine learning research engineers](https://humancompatible.ai/jobs#engineer) for 1-2 year visiting scholar positions to test alignment ideas for deep reinforcement learning systems.
|
||||
|
||||
When you apply to a larger organization that has multiple areas of research, specify in your application which of them you are most interested in working on. Investigate the company’s research areas in advance, in order to make sure that the areas you list are in fact ones that the company works on. For example, don’t specify “value alignment” on an application to a company that does not have any researchers working on value alignment.
|
||||
|
||||
If you find that you cannot get a role contributing to safety research right now, you might look for a role in which you can gain relevant experience, and transition to a safety position later.
|
||||
|
||||
Non-safety-related research engineering positions are also available at [other industry AI labs](https://www.google.com/search?q=machine+learning+research+engineer&ibp=htl;jobs#fpstate=tldetail&htidocid=t1jN4MUgXkPZajmvAAAAAA%3D%3D&htivrt=jobs) though these are likely to be more competitive than roles on AGI safety teams.
|
||||
|
||||
Finally, you could consider applying to a 1-year fellowship/residency program at [Google](https://careers.google.com/stories/edu-resources-programs//), [OpenAI](https://blog.openai.com/openai-fellows-interns-2019/), [Facebook](https://research.fb.com/programs/facebook-ai-research-residency-program/), [Uber](https://eng.uber.com/uber-ai-residency/), or [Microsoft](https://www.microsoft.com/en-us/research/academic-program/microsoft-ai-residency-program/).
|
||||
|
||||
## Learn more
|
||||
|
||||
* Working at a leading AI lab might cause harm. [Read more on whether it might still be a high-impact career step.](https://80000hours.org/career-reviews/working-at-an-ai-lab/)
|
||||
* This [curriculum on AI safety](https://www.eacambridge.org/technical-alignment-curriculum) (or, for something shorter, [this sequence of posts](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ) by Richard Ngo)
|
||||
* Our [in-depth conversation on the 80,000 Hours Podcast with Catherine Olsson and Daniel Ziegler](https://80000hours.org/podcast/episodes/olsson-and-ziegler-ml-engineering-and-safety/), on which this guide is based.
|
||||
* Our guide to [positively shaping the development of advanced artificial intelligence](https://80000hours.org/problem-profiles/positively-shaping-artificial-intelligence/)
|
||||
* Our [career review of an ML PhD](https://80000hours.org/career-reviews/machine-learning-phd/)
|
||||
* Our podcasts with Chris Olah on [what the hell is going on inside neural networks](https://80000hours.org/podcast/episodes/chris-olah-interpretability-research/) and [working at top AI labs without an undergrad degree](https://80000hours.org/podcast/episodes/chris-olah-unconventional-career-path/)
|
||||
|
||||
@@ -2,42 +2,229 @@
|
||||
|
||||
Source: https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
|
||||
Author: Jacob Steinhardt
|
||||
Date: not visible in fetched HTML
|
||||
Fetch-status: excerpted from HTML via browser.
|
||||
Date: page itself is undated; the fetch reports a published time of 2018-12-29
|
||||
Fetch-status: full post text, fetched 2026-08-15 via `curl https://r.jina.ai/https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html`. Equations render as image links in the source page and stay as image links here. (CLAUDE agent)
|
||||
Use: research-prioritization evidence; cited by Nanda's Key Mindsets post.
|
||||
|
||||
## Why this matters for agents
|
||||
|
||||
Steinhardt gives a crisp formal-ish rule for research prioritization: reduce uncertainty as fast as possible. This is useful for agents deciding which experiment, baseline, prototype, or sanity check to run first.
|
||||
|
||||
## Quotes
|
||||
|
||||
> Below I analyze how to approach a project that has many somewhat independent sources of uncertainty (we can often think of these as multiple "steps" or "parts" that each have some probability of success).
|
||||
|
||||
> We will eventually see that a good principle is to "reduce uncertainty at the fastest possible rate".
|
||||
|
||||
> This reveals that harder tasks should not necessarily be prioritized. Rather, we should prioritize tasks that are more likely to fail (so that we remove the risk of them failing) but also tasks that take less time.
|
||||
|
||||
> Do the components in order from most informative per unit time to least informative per unit time.
|
||||
|
||||
> De-risk all components (to the extent feasible), then execute.
|
||||
|
||||
> Specifically, for each task we want a cheap way to obtain high confidence about whether that task will be feasible. This is called "de-risking".
|
||||
|
||||
> We are often either in "de-risking mode" (determining if the problem is infeasible as quickly as possible) or "execution mode" (assuming the problem is feasible and trying to solve it quickly).
|
||||
|
||||
> The counterpart to ceilings are baselines--simple or off-the-shelf methods that give a quick lower bound on achievable accuracy.
|
||||
|
||||
> Together with ceilings, they delineate a range of possible performance, which helps us interpret our core results.
|
||||
|
||||
> I often think about possible approaches to a problem as an exponentially branching search tree.
|
||||
|
||||
> Whenever something doesn't work, I ask why it didn't work. My goal is to avoid trying similar things that will fail for the same reason.
|
||||
|
||||
> Compared to other people I know, I try harder and earlier to show that my ideas can't work to solve a problem.
|
||||
|
||||
> We often try easier tasks first, when instead we should try the most informative tasks first.
|
||||
|
||||
## Source graph
|
||||
|
||||
This is a standalone blog post. It links to concepts like Poisson arrival processes, but the skill-relevant content is the prioritization/de-risking frame above.
|
||||
|
||||
## Full post
|
||||
|
||||
## Research as a Stochastic Decision Process
|
||||
|
||||
In this post I will talk about an approach to research (and other projects that involve high uncertainty) that has substantially improved my productivity. Before implementing this approach, I made little research progress for over a year; afterwards, I completed one project every four months on average. Other changes also contributed, but I expect the ideas here to at least double your productivity if you aren't already employing a similar process.
|
||||
|
||||
Below I analyze how to approach a project that has many somewhat independent sources of uncertainty (we can often think of these as multiple "steps" or "parts" that each have some probability of success). Is it best to do these steps from easiest to hardest? From hardest to easiest? From quickest to slowest? We will eventually see that a good principle is to "reduce uncertainty at the fastest possible rate". After revealing issues with more simplistic approaches, I will articulate this principle in detail and show how to apply it. Throughout, I draw my examples primarily from problems in machine learning and mathematics, but I believe that the principles generalize to other situations as well.
|
||||
|
||||
### Warm-Up
|
||||
|
||||
Suppose you are embarking on a project with several parts, all of which must succeed for the project to succeed. For instance, a proof strategy might rely on proving several intermediate results, or an applied project might require achieving high enough speed and accuracy on several components. What is a good strategy for approaching such a project? For me, the most intuitively appealing strategy is something like the following:
|
||||
|
||||
**(Naive Strategy)**
|
||||
|
||||
Complete the components in increasing order of difficulty, from easiest to hardest.
|
||||
|
||||
This is psychologically tempting: you do what you know how to do first, which can provide a good warm-up to the harder parts of the project. This used to be my default strategy, but often the following happened: I would do all the easy parts, then get to the hard part and encounter a fundamental obstacle that required scrapping the entire plan and coming up with a new one. For instance, I might spend a while wrestling with a certain algorithm to make sure it had the statistical consistency properties I wanted, but then realize that the algorithm was not flexible enough to handle realistic use cases.
|
||||
|
||||
The work on the easy parts was mostly wasted--it wasn't that I could replace the hard part with a different hard part; rather, I needed to re-think the entire structure, which included throwing away the "progress" from solving the easy parts.
|
||||
|
||||
What might be a better strategy than the naive strategy above? Since the naive strategy has the problem that we waste effort on the easy components if the hard components are intractable, maybe it would be better to complete the components in _decreasing_ order of difficulty, starting from the hardest and moving to the easiest.
|
||||
|
||||
This _might_ be better, but our intuitive sense of hardness likely combines many factors--the likelihood that the task fails, the time it takes to complete, and perhaps others as well. Here is an example:
|
||||
|
||||
Task A is a detailed and tricky calculation, but you have done many similar calculations before and are confident that given a few days you will succeed. Task B will likely take much less time, but it is something you haven't done before (so it is more likely there will be an unforeseen difficulty or problem).
|
||||
|
||||
In this case, task B would be better to do first--if you do task A first and then B turns out doomed, you have wasted several days. Even if A also has some chance of failing (so that it is both more likely to fail and takes longer than B), we would still usually rather do B before A.
|
||||
|
||||
This reveals that harder tasks should not necessarily be prioritized. Rather, we should prioritize tasks that _are more likely to fail_(so that we remove the risk of them failing) but also tasks that _take less time_ (so that we've wasted less time if one of the tasks does fail, and also so that we get information about tasks more quickly).
|
||||
|
||||
### A Better Strategy: Sorting by Information Rate
|
||||
|
||||
We can incorporate both of the above desiderata by sorting the tasks based on which are _most informative per unit time_.
|
||||
|
||||
**(Better Strategy)**
|
||||
|
||||
Do the components in order from most informative per unit time to least informative per unit time.
|
||||
|
||||
To implement this, we need a method for quantifying informativeness. I will present two methods below--one based on _expected time saved_, and one based on _failure rate_. Rather than define these rigorously upfront, I will work through several examples, which should make the general case evident.
|
||||
|
||||
**Method 1: Expected Time Saved**
|
||||
|
||||
If an earlier step fails, we save time by not having to attempt the later steps. We should therefore complete the steps in the order that maximizes the expected value of the time that we save. We assume for now that we can actually quantify the probability that each step succeeds, as well as the time it will take. Consider the following example:
|
||||
|
||||
Example 1: All of the steps of a project have roughly equal chance of success (80%, say) but take varying amounts of time to complete.
|
||||
|
||||
In this example we would want to do the quickest task first and slowest last, since the later a task occurs, the more likely we will get to skip doing it. Sorting "easiest to hardest" is therefore correct here, but it is rare that all steps have equal success probability.
|
||||
|
||||
Example 2: An easy task has a 90% success probability and takes 30 minutes, and a hard task has a 40% success probability and takes 4 hours.
|
||||
|
||||
Here we should do the easy task first: if it fails we save 240 minutes, so 0.1 * 240 = 24 minutes in expectation; conversely if the hard task is done first and fails, we save 30 minutes, for 0.6 * 30 = 18 minutes in expectation. But if the hard task takes 2 hours or the easy task has a 95% chance of success, we should do the hard task first.
|
||||
|
||||
Thus, in this method we formalized "most informative per unit time" by looking at how much time we save (in expectation) by not having to do the tasks that occur after the first failure. Our computations assumed that we only find out if a task succeeds or fails at the end, as opposed to in the middle; however, they can be modified to take such complications into account.
|
||||
|
||||
For more than two tasks, this calculation method quickly becomes intractable: for K tasks we have to consider all K! permutations to find the best one. The next method avoids this issue.
|
||||
|
||||
**Method 2: Failure Rate**
|
||||
|
||||
This next method models the occurrence of failures as a Poisson process: if a task takes 30 minutes and has a 15% chance of failure, then there is about a 0.5% chance that the failure will occur in each minute (actually, it is slightly more than that because of overlap among the failures; the actual value is the solution p to (1-p)^30 = 0.85). Note that this differs from our previous assumption that failures can only occur at the end. This alternate model will simplify our calculations.
|
||||
|
||||
Formally, assume that the probability that we realize the task fails in the next minute is independent of how long we have been doing the task. Then the occurrence of a failure is a [Poisson arrival process](https://en.wikipedia.org/wiki/Poisson_point_process#Interpreted_as_a_point_process_on_the_real_line) and the time at which a failure occurs [follows an exponential distribution](https://en.wikipedia.org/wiki/Exponential_distribution#Applications_of_exponential_distribution) with some rate parameter  , where  tells us how frequently failures occur per unit time. Using basic properties of Poisson processes (see Appendix A), we can compute  as
|
||||
|
||||
 ,
|
||||
|
||||
where  is the success probability of the task.
|
||||
|
||||
This rate exactly tells us how quickly we will encounter failures while doing a given task. Since we would like to front-load failures as much as possible, we would always like to sort the tasks in decreasing order of their rate  .
|
||||
|
||||
Returning to Example 2, we can compute the rate  for the two tasks:
|
||||
|
||||
Task 1: 
|
||||
|
||||
Task 2: 
|
||||
|
||||
This new computation reverses our previous conclusion: The hard task has a higher rate, so is actually (slightly) better to do first! The reason for this is that the Poisson assumption implies that the higher the failure probability of a task, the faster (in expectation) we will encounter the failure. This contrasts with the previous assumption that we only encounter failures at the end of a task. We should keep in mind that both of these assumptions are likely somewhat incorrect in practice.
|
||||
|
||||
The rate method extends easily to more than two tasks, since we can simply sort tasks in order of  .
|
||||
|
||||
**An Additional Example**
|
||||
|
||||
In the case of the time-consuming but certain task A and quicker but uncertain task B, task A might take 12 hours but have a 90% chance of success, while task B takes 2 hours but has a 65% chance of success.
|
||||
|
||||
First, let's see what we get using the time saved method:
|
||||
|
||||
* A first: 0.1 * 2 = 0.2 hours
|
||||
* B first: 0.35 * 12 = 4.2 hours
|
||||
|
||||
Now suppose we use the rate method:
|
||||
|
||||
* A first: log(1/0.9)/12 = 0.009
|
||||
* B first: log(1/0.65)/2 = 0.215
|
||||
|
||||
B dominates A on _both_ failure prob and time, so doing B first looks substantially better under both methods.
|
||||
|
||||
**Caveats**
|
||||
|
||||
These numbers are all completely made up and in practice you won't be able to estimate things so well. I subjectively distinguish between different "buckets" of success probability, such as:
|
||||
|
||||
* "I am confident that this can be done and that there are no unforeseen difficulties" (~95%)
|
||||
* "I am confident that this can be done modulo Murphy's law" (~90%)
|
||||
* "I see the basic path to accomplishing this and all the steps seem like they should work" (~65%)
|
||||
* "I have the intuition that this should be possible but only have a murky view of the path" (~30%)
|
||||
|
||||
On the other hand, I tend to have much better estimates of task completion times if I've been practicing (~30% average relative error, albeit with large tails). You can get better at this within a few weeks by estimating completion times for each task and then recording the actual completion times in a daily log. You should also practice decomposing tasks into small actionable chunks, each taking roughly 20 minutes to 2 hours.
|
||||
|
||||
### A further improvement: opening up the "task" black box
|
||||
|
||||
Sorting tasks in decreasing order of failure rate is a good start; it should improve efficiency by a factor of 2-3. However, we can do _much_ better still by learning to front-load the information gained about each task. Front-loading information requires a mental finesse: rather than seeking to complete a task, we must seek information _about_ a task.
|
||||
|
||||
Specifically, for each task we want a _cheap way to obtain high confidence about whether that task will be feasible_. This is called "de-risking". The following pattern is indispensable:
|
||||
|
||||
**(Basic Pattern)**
|
||||
|
||||
De-risk all components (to the extent feasible), then execute.
|
||||
|
||||
As an example, suppose we wish to set up a dataset and then train a suitable model on that dataset. However, setting up the dataset is arduous: we must download it to a place with enough disc space, parse it into a usable format, and incorporate auxiliary data sources (like noun/verb banks for natural language processing).
|
||||
|
||||
Setting up the dataset and training the model are both time-consuming and either one could fail. Even worse, it would seem that we are forced to set up the dataset first, even though it is probably the more time-consuming task.
|
||||
|
||||
To avoid this issue, we could first download a few thousand examples. We can then examine several examples by hand, as well as compute some aggregate statistics, to assess whether the dataset has the properties we want. Ideally, this will reduce a lot of uncertainty about whether the will dataset is suitable.[1](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html#fn1)
|
||||
|
||||
### General principle: stochastic decision process
|
||||
|
||||
We can unify and extend the above insights by modeling a research project as a _stochastic decision process_. Specifically, we think of research as a multi-round game, where in each round we take some action that gives us some information; the information we get is stochastic, and well as perhaps the time needed to complete the action. We have two competing goals:
|
||||
|
||||
* Maximize probability of eventual success (don't give up if it turns out we can eventually solve the problem).
|
||||
* Minimize expected time spent (give up early if the problem is not feasible, and solve the problem quickly if it is feasible).
|
||||
|
||||
We are often either in "de-risking mode" (determining if the problem is infeasible as quickly as possible) or "execution mode" (assuming the problem is feasible and trying to solve it quickly).
|
||||
|
||||
**An aside: tooling.** This picture grows more complicated if we consider actions that could speed up a family of future actions (such as writing helpful scripts to automate tasks, or reducing the execution time of the system). Such "tooling" tasks are tricky to model, because it seems we should implement tooling as soon as we know we will eventually want it (since it speeds up things that come after it). However, this ignores that more experience often yields refined desiderata for the tools we implement. There is thus a trade-off between building tools earlier vs. building better-targeted tools. I won't say more about this here, but it is an important point to keep in mind.
|
||||
|
||||
Another complication is that our ultimate goal is often nebulous--we are not asking "is this problem possible" so much as "how interesting of a problem in this space is it feasible to solve"? But I don't think this substantially alters the above principles.
|
||||
|
||||
### Some further practical ideas
|
||||
|
||||
There are a number of useful patterns for putting the above principles into practice. I list several below.
|
||||
|
||||
**For empirical work, measuring "ceilings" (an upper bound of how high performance could possibly be) is often useful.**Example: suppose we wish to build a system with 3 components that interact in a complicated way. One of the components is difficult to implement, but we can easily substitute a "cheating" version of that component (e.g. by looking at the test set or by using information that won't be available at deployment time). We often benefit by building a prototype system that initially uses this cheating version:
|
||||
|
||||
* If the system works, we know that a sufficiently good implementation of the difficult component will yield a working system.
|
||||
* If the system doesn't work, we've saved the time of implementing the difficult component.
|
||||
|
||||
We can choose which components to cheat on initially, and which to implement fully, using the "informativeness per unit time" heuristic from above. For instance, if the ability to do well on a specific component is the major source of uncertainty in the project, cheating on it might be counterproductive (we may instead want to cheat on _everything but that component_).
|
||||
|
||||
The counterpart to ceilings are _baselines_--simple or off-the-shelf methods that give a quick lower bound on achievable accuracy. Baselines provide an important sanity check, as complicated methods often underperform simple baselines. Together with ceilings, they delineate a range of possible performance, which helps us interpret our core results.
|
||||
|
||||
**Brute force.**If we know of an easy-to-implement brute force solution and a difficult-to-implement fast solution, starting with the brute force solution has many of the same advantages as using ceilings, as long as the slower running time doesn't bottleneck prototyping. A brute force implementation also facilitates debugging the fast solution, since we can compare the outputs of the two algorithms.
|
||||
|
||||
As with ceilings, brute force is most useful when implementing the fast solution is not a major source of uncertainty (e.g. it is routine but annoying, or is one of many sources of uncertainty).
|
||||
|
||||
**For theoretical work, looking for counterexamples is useful.**The simplest example of this: if we find a counterexample to the main result we want to prove, then we need to either give up or make stronger assumptions.
|
||||
|
||||
A more nuanced (and more common) example: if we are trying to prove that  , and our current technique does this by proving  and then  , finding a counterexample to  will rule out that technique.
|
||||
|
||||
Yet more nuanced/common: if we are trying to prove that  , and our current technique applies equally well under assumptions  and  , then a counterexample to  will rule out the technique.
|
||||
|
||||
**More generally, thinking about simplified instances of a problem is often useful.** This is because it provides intuition that often suggests/rules out approaches for the original problem. Similarly to de-risking, the ability to rule out entire approaches makes this tactic invaluable from the stochastic decision process perspective.
|
||||
|
||||
**Running simulations.**If we wish to prove X, first run simulations to check if X is actually true. This is easy when assessing the behavior of a specific algorithm, as we can simply run the algorithm. Simulations can also, for instance, help reveal the asymptotics of a random process, or be used to search for small counterexamples to a conjecture.
|
||||
|
||||
### Exponentially branching search trees
|
||||
|
||||
Another important mental framework focuses on the combinatorial aspect of a decision process:
|
||||
|
||||
**(Research as branching search)**
|
||||
|
||||
I often think about possible approaches to a problem as an exponentially branching search tree: we could try X, X', or X''. Then X could be combined with Y, Y', Y'', or X' could be combined with Z or Z', etc. This exponential blow-up poses barriers to projects with more than a small number of steps unless we have a way to systematically rule out entire branches of the tree.
|
||||
|
||||
Exponential branching often occurs because there are many ways to try a particular approach--perhaps we want to bound the moment generating function, and there are many ways to attempt this; or we think data augmentation will help our model generalize, but there are many ways to augment the data. With many possibilities for each step, even a two- or three-step approach creates a huge search space. For instance, if there are 10 ways to try bounding the moment generating function, and two other similar steps, then we have to try 1000 possibilities.[2](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html#fn2)
|
||||
|
||||
If the steps _factor_--meaning they can each be solved in isolation--this might be fine (we only have to try 3*10 instead of 10^3 possibilities). However, I usually find that there is some interdependency between different steps. For a math problem, maybe how good of a bound I get from step 1 affects how hard I need to work for step 2. Or for an experiment, if any of 3 parts of the setup are wrong then the method just won't work, so I don't get signal until I've gotten a few things right simultaneously.
|
||||
|
||||
For this reason, I think it's _much_ more useful to prune branches of the search tree at the level of conceptual approaches ("can the moment generating function give me sufficient control over the distribution I care about?") than at the level of a specific instantiation ("does this particular moment generating function bound work?"). This leads to adopting several principles:
|
||||
|
||||
**Whenever something doesn't work, I ask _why_ it didn't work.** My goal is to avoid trying similar things that will fail for the same reason (or to notice that the reason why it didn't work is circumventable, and that a modified approach actually will work).
|
||||
|
||||
**Trying an experiment and seeing it fail gives little information by itself.** When an experiment fails, it is tempting to conclude "I tried X and it didn't work". However, if X is a high-level conceptual approach, then a more correct conclusion is "I tried an implementation comprising 0.1% of the possible implementations of X, and observed that that particular implementation did not work". For this reason, I am far less in favor than most people of publishing negative results, unless the negative result comes with insight into what caused the failure. In contrast to common concerns, negative results that come with such insights are [already publishable](https://acl2018.org/paper/1604/).
|
||||
|
||||
**Compared to other people I know, I try harder and earlier to show that my ideas can't work to solve a problem.** Importantly, it is often not obvious that multiple approaches to a problem all have the same issue. In the past, I have spent months trying different approaches to a problem before finally stepping back and realizing that they were all failing for the same reason. Moreover, I had all the data necessary to make this realization a couple weeks in but had failed to do so. I now save considerable time by ruling out ideas early on, and as a result I am usually bottlenecked on coming up with ideas rather than on implementing ideas.
|
||||
|
||||
**Additional Discussion**
|
||||
|
||||
In the previous section I talked about ruling out ideas. When ruling out ideas, it is important to hold oneself to a high standard. "This doesn't seem like it will work" or "I feel less motivated after trying a few things along this line that didn't work" are _not_ ruling out an idea. We could perhaps think of them as updating the probabilities that a solution lies within a given subtree of the search tree. But these updates are rarely large updates, and I find them much less reliable than a solid argument for why an approach is doomed.
|
||||
|
||||
Note that I am _not_ advocating that you should never trust your feelings. If you feel pessimistic about an approach, that is a great reason to try to show that the approach can't work! If I feel pessimistic about an approach but fail to rule out that it could work, I often then feel more optimistic.
|
||||
|
||||
I am also _not_ advocating for the failure mode of only trying low-variance ideas, or of avoiding projects that lack an obviously promising approach. Part of the point of being able to systematically rule out ideas is to enable trying ideas that only have a low probability of working, or that do not immediately yield progress.
|
||||
|
||||
### Summary
|
||||
|
||||
Many of our default intuitions about how to pursue uncertain ideas are counterproductive:
|
||||
|
||||
* We often try easier tasks first, when instead we should try the most informative tasks first.
|
||||
* We often conflate a high-level approach with a low-level instantiation of the approach.
|
||||
* We are often too slow to try to disprove our own ideas.
|
||||
|
||||
Building frameworks that reify the research process as a concrete search problem can help unearth these incorrect intuitions and replace them with systematic reasoning.
|
||||
|
||||
### Appendix A: Poisson Process Calculation
|
||||
|
||||
In a Poisson process with rate  , the probability that a failure has already occurred by time t is  , so in particular  , where  is the time to complete the task and  is the success probability of the task. If we solve for this, we get that the rate  is equal to
|
||||
|
||||
 ,
|
||||
|
||||
as claimed.
|
||||
|
||||
## Notes
|
||||
|
||||
* * *
|
||||
|
||||
1. This doesn't quite fit into the framework because if the dataset is unsuitable we can try again until we find a suitable dataset. But it could be that we try 4 datasets, they are all unsuitable, and we eventually conclude that there aren't any suitable datasets. The sort of de-risking above allows us to reach this conclusion much faster and avoid spending time trying to train a model on a broken dataset.[↩](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html#fnref1)
|
||||
|
||||
2. This is purely illustrative and in reality we can't necessarily decompose different attempts into a fixed number of discrete "ways" of attempting something.[↩](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html#fnref2)
|
||||
|
||||
@@ -1,11 +1,113 @@
|
||||
# Gears-Level Models are Capital Investments — John Wentworth
|
||||
|
||||
Source: https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments . Verbatim excerpts cached for the skill.
|
||||
Source: https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments
|
||||
Author: John Wentworth (johnswentworth)
|
||||
Date: 22nd Nov 2019
|
||||
Fetch-status: full post text, fetched 2026-08-15 from the LessWrong markdown API (`/api/post/nEBbw2Bc2CnN2RMxy`), comments and site navigation stripped. (CLAUDE agent)
|
||||
Use: evidence for preferring mechanism-level understanding over black-box tuning.
|
||||
|
||||
---
|
||||
## Full post
|
||||
|
||||
> This is a general feature of gears-level models: figuring out a system's gears takes extra work up-front, but yields dividends forever. The alternative, typically, is a black-box strategy: use a method which works without needing to understand the internals of the system. The black-box approach is cheaper for one-off tasks, but usually doesn't yield any insights which will generalize to new tasks using the same system - it's context-dependent.
|
||||
Mazes
|
||||
-----
|
||||
|
||||
On the "valley of bad theory" experiment (optimizing without understanding):
|
||||
The usual method to solve a maze is some variant of [babble-and-prune](/api/post/i42Dfoh4HtsCAfXxL): try a path, if it seems to get closer to the exit then keep going, if it hits a dead end then go back and try another path. It's a black-box method that works reasonably well on most mazes.
|
||||
|
||||
> Given the opportunity to test things out, subjects would often iterate their way to optimal settings - but they didn't iterate their way to correct theories. [...] This is black-box optimization: optimization was achieved, but insight into the system was not.
|
||||
However, there are [other methods](/api/post/CPBmbgYZpsGqkiz2R). For instance, you could start by looking for a chain of walls with only one opening, like this:
|
||||
|
||||
|
||||
|
||||
This chain of walls is a [gears-level insight](/api/post/B7P97C27rvHPz3s9B) into the maze - a piece of the internal structure which lets us better understand “how the maze works” on a low level. It’s not specific to any particular path, or to any particular start/end points - it’s a property of the maze itself. Every shortest path between two points in the maze either starts and ends on the same side of that line, or passes through the gap.
|
||||
|
||||
If we only need to solve the maze once, then looking for a chain of walls is not very useful - it could easily take as long as solving the maze! But if we need to solve the *same* maze more than once, with different start and end points… then we can spend the time finding that chain of walls just once, and re-use our knowledge over and over again. It’s a capital investment: we do some extra work up-front, and it pays out in lower costs every time we look for a path through the maze in the future.
|
||||
|
||||
This is a general feature of gears-level models: figuring out a system’s gears takes extra work up-front, but yields dividends forever. The alternative, typically, is a black-box strategy: use a method which works without needing to understand the internals of the system. The black-box approach is cheaper for one-off tasks, but usually doesn’t yield any insights which will generalize to new tasks using the same system - it’s context-dependent.
|
||||
|
||||
Marketing
|
||||
---------
|
||||
|
||||
Suppose we work with the marketing team at an online car loan refinance company, and we're tasked with optimizing the company's marketing to maximize the number of car loans the company refinances. Here's two different approaches we might take:
|
||||
|
||||
* We [a/b test](https://en.wikipedia.org/wiki/A/B_testing) hundreds of different ad spend strategies, marketing copy permutations, banner images, landing page layouts, etc. Ideally, we find a particular combination works especially well.
|
||||
* We obtain some anonymized data from a credit agency on people with car loans. Ideally, we learn something about the market - e.g. maybe subprime borrowers usually either declare bankruptcy or dramatically increase their credit score within two years of taking a loan.
|
||||
|
||||
The first strategy is black-box: we don't need to know anything about who our potential customers are, what they want, the psychology of clicking on ads, etc. We can treat our marketing pipeline as a black box and fiddle with its inputs to see what works. The second strategy is gears-level, the exact opposite of black-box: the whole point is to learn who our potential customers are, breaking open the black box and looking at the internal gears.
|
||||
|
||||
These aren't mutually exclusive, and they have different relative advantages. Some upsides of black-box:
|
||||
|
||||
* Black-box is usually cheaper and easier, since the code involved is pretty standard and we don't need to track down external data. Gears-level strategies require more custom work and finding particular data.
|
||||
* Black-box yields direct benefits when it works, whereas gears-level requires an extra step to translate whatever insights we find into actual improvements.
|
||||
|
||||
On the other hand:
|
||||
|
||||
* Gears-level insights can highlight ideas we wouldn't even have thought to try, whereas black-box just tests the things we think to test.
|
||||
* When some tests are expensive (e.g. integrating with a new ad channel), gears-level knowledge can tell us which tests are most likely to be worthwhile.
|
||||
* Black-box optimization is subject to [Goodhart](/api/post/YtvZxRpZjcFNwJecS), while gears-level insights usually are not (at least in-and-of themselves)
|
||||
* Gears-level insights are less likely subject to distribution shift. For instance, if we change ad channels, then the distribution of people seeing our ads will shift. Different ad copy will perform well, and we'd need to restart our black-box a/b testing, whereas general insights about subprime borrowers are more likely to remain valid.
|
||||
* Conversely, black-box optimizations depreciate over time. Audiences and ad channels evolve, and ads need to change with them, requiring constant re-optimization to check that old choices are still optimal.
|
||||
* By extension, gears-level insights tend to be permanent and broadly applicable, and have the potential for compound returns, whereas black-box improvements are much more context-specific and likely to shift with time.
|
||||
|
||||
In short, the black-box approach is easier, cheaper, and more directly useful - but its benefits are ephemeral and it can't find unknown unknowns. Gears-level understanding is more difficult, expensive, and risky, but it offers permanent, generalizable insights and can suggest new questions we wouldn't have thought to ask.
|
||||
|
||||
With this in mind, consider the world through the eyes of an ancient lich or [thousand-year-old vampire](/api/post/kXSETKZ3X9oidMozA). It's a worldview in which ephemeral gains are irrelevant. All that matters is permanent, generalizable knowledge - everything else will fade in time, and usually not even very much time. In this worldview, gears-level understanding is everything.
|
||||
|
||||
On the other end of the spectrum, consider the world through the eyes of a startup with six months of runway which needs to show rapid growth in order to close another round of funding. For them, black-box optimization is everything - they want fast, cheap results which don’t need to last forever.
|
||||
|
||||
Wheel with Weights
|
||||
------------------
|
||||
|
||||
There’s a [neat experiment](/api/post/gZP8t9BAg37bqxDzZ) where people are given a wheel with some weights on it, each of which can be shifted closer to/further from the center. Groups of subjects have to cooperatively find settings for the weights which minimize the time for the wheel to roll down a ramp.
|
||||
|
||||
|
||||
|
||||
Given the opportunity to test things out, subjects would often iterate their way to optimal settings - but they didn’t iterate their way to correct theories. When asked to predict how hypothetical settings would perform, subjects’ predictions didn’t improve much as they iterated. This is black-box optimization: optimization was achieved, but insight into the system was not.
|
||||
|
||||
If the problem had changed significantly - e.g. changing weight ratios/angles, ramp length/angle, etc - the optimal settings could easily change enough that subjects would need to re-optimize from scratch. On the other hand, the system is simple enough that just doing all the math is tractable - and that math would remain essentially the same if weights, angles, and lengths changed. A gears-level understanding is possible, and would reduce the cost of optimizing for new system parameters. It’s a capital investment: it only makes sense to make the investment in gears-level understanding if it will pay off on many different future problems.
|
||||
|
||||
In the experiment, subjects were under no pressure to achieve gears-level understanding - they only needed to optimize for one set of parameters. I’d predict that people would be more likely to gain understanding if they needed to find optimal weight-settings quickly for many different wheel/ramp parameters. (A close analogy is [evolution of modularity](/api/post/JBFHzfPkXHB2XfDGj): changing objectives incentivize learning general structure.)
|
||||
|
||||
Metis
|
||||
-----
|
||||
|
||||
Let’s bring in the [manioc example](/api/post/TMFNQoRZxM4CuRCY6):
|
||||
|
||||
> There's this plant, manioc, that grows easily in some places and has a lot of calories in it, so it was a staple for some indigenous South Americans since before the Europeans showed up. Traditional handling of the manioc involved some elaborate time-consuming steps that had no apparent purpose, so when the Portuguese introduced it to Africa, they didn't bother with those steps - just, grow it, cook it, eat it.
|
||||
|
||||
> The problem is that manioc's got cyanide in it, so if you eat too much too often over a lifetime, you get sick, in a way that's not easily traceable to the plant. Somehow, over probably hundreds of years, the people living in manioc's original range figured out a way to leach out the poison, without understanding the underlying chemistry - so if you asked them why they did it that way, they wouldn't necessarily have a good answer.
|
||||
|
||||
The techniques for processing manioc are a [stock](/api/post/TMFNQoRZxM4CuRCY6) [example](/api/post/Zm7WAJMTaFvuh2Wc7) of metis: traditional knowledge accumulated over generations, which doesn’t seem like it has any basis in reason or any reason to be useful. It’s black-box knowledge, where the black-box optimizer is cultural transmission and evolution. Manioc is a cautionary tale about the dangers of throwing away or ignoring black-box knowledge just because it doesn’t contain any gears.
|
||||
|
||||
In this case, building a gears-level model was *very* expensive - people had to get sick on a large scale in order to figure out that any knowledge was missing at all, and even after that it presumably took a while for scientists to come along and link the problem to cyanide content. On the other hand, now that we have that gears-level model in hand, we can quickly and easily test new cooking methods to see whether they eliminate the cyanide - our gears-level model provides generalizable insights. We can even check whether any particular dish of manioc is safe before eating it, or breed new manioc strains which contain less cyanide. Metic knowledge would have no way to do any of that - it doesn’t generalize.
|
||||
|
||||
More Examples
|
||||
-------------
|
||||
|
||||
(Note: in each of these examples, there are many other ways to formulate a black-box/gears-level approach. I just provide one possible approach for each.)
|
||||
|
||||
Pharma
|
||||
|
||||
* Black box approach: run a high-throughput assay to test the effect thousands of chemicals against low-level markers of some disease.
|
||||
* Gears-level approach: comb the literature for factors related to some disease. Run experiments holding various subsets of the factors constant while varying others, to figure out which factors mediate the effect of which others, and ultimately build up a causal graph of their interactions.
|
||||
|
||||
The black-box approach is a lot cheaper and faster, but it’s subject to Goodhart problems, won’t suggest compounds that nobody thought to test, and won’t provide any knowledge which generalizes to related diseases. If none of the chemicals tested are effective, then the black-box approach leaves no foundation to build on. The gears-level approach is much slower and more expensive, but eventually yields reliable, generalizable knowledge.
|
||||
|
||||
Financial Trading
|
||||
|
||||
* Black box approach: build a very thorough backtester, then try out every algorithm or indicator we can think of to see if any of them achieve statistically significant improvement over market performance.
|
||||
* Gears-level approach: research the trading algorithms and indicators actually used by others, then simulate markets with traders using those algorithms/indicators. Compare results against real price behavior and whatever side data can be found in order to identify missing pieces.
|
||||
|
||||
The gears-level approach is far more work, and likely won’t produce anything profitable until very late in development. On the other hand, the gears-level approach will likely generalize far better to new markets, new market conditions, etc.
|
||||
|
||||
Data Science
|
||||
|
||||
* Black box approach: train a neural network, random forest, support vector machine, or whatever generic black-box learning algorithm you like.
|
||||
* Gears-level approach: build a [probabilistic graphical model](/api/post/hzuSDMx7pd2uxFc5w). Research the subject matter to hypothesize model structure, and [statistically compare](/api/post/5mr8Qcqi6xWa6HCHw) different model structures to see which match the data best. Look for side information to confirm that the structure is correct.
|
||||
|
||||
The black box approach is subject to Goodhart and often fails to generalize. The gears-level approach is far more work, requiring domain expertise and side data and probably lots of custom code (although the recent surge of [probabilistic programming languages](https://pyro.ai/examples/svi_part_i.html) helps a lot in that department), but gears-level models ultimately give us human-understandable explanations of how the system actually works. Their internal parameters have physical meaning.
|
||||
|
||||
Takeaway
|
||||
--------
|
||||
|
||||
Building gears-level models is expensive - often prohibitively expensive. Black-box approaches are usually much cheaper and faster. But black-box approaches rarely generalize - they’re subject to Goodhart, need to be rebuilt when conditions change, don’t identify unknown unknowns, and are hard to build on top of. Gears-level models, on the other hand, offer permanent, generalizable knowledge which can be applied to many problems in the future, even if conditions shift.
|
||||
|
||||
The upfront cost of gears-level knowledge makes it an investment, and the payoff of that investment is the ability to re-use the model many times in the future.
|
||||
|
||||
Reference in New Issue
Block a user