diff --git a/docs/evidence/cleanrl_37_ppo_details.md b/docs/evidence/cleanrl_37_ppo_details.md
index 79d2504..2d7c638 100644
--- a/docs/evidence/cleanrl_37_ppo_details.md
+++ b/docs/evidence/cleanrl_37_ppo_details.md
@@ -3,23 +3,796 @@
Authors: Huang, Shengyi; Dossa, Rousslan Fernand Julien; Raffin, Antonin; Kanervisto, Anssi; Wang, Weixun.
Source: ICLR Blog Track, 2022-03-25 — https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
Code: https://github.com/vwxyzjn/ppo-implementation-details ; CleanRL: https://github.com/vwxyzjn/cleanrl
+Fetched-via: r.jina.ai reader, 2026-08-15 (CLAUDE agent)
+Fetch-status: full post text, all 37 details. Supersedes the earlier framing-passages excerpt. (CLAUDE agent)
-Excerpt cached for the ML-debugging skill (the full post is long; key framing passages below, verbatim).
+Why it matters here: the reference catalogue of undocumented implementation details that decide whether an RL reproduction works, each with a permanent link to the code line.
---
-> Instead of doing ablation studies and making recommendations on which details matter, this blog post takes a step back and focuses on reproductions of PPO's results in all accounts.
+25 Mar 2022 | [proximal-policy-optimization](https://iclr-blog-track.github.io/tags/#proximal-policy-optimization)[reproducibility](https://iclr-blog-track.github.io/tags/#reproducibility)[reinforcement-learning](https://iclr-blog-track.github.io/tags/#reinforcement-learning)[implementation-details](https://iclr-blog-track.github.io/tags/#implementation-details)[tutorial](https://iclr-blog-track.github.io/tags/#tutorial)
-> During our re-implementation, we have compiled an implementation checklist containing 37 details as follows. For each implementation detail, we display the permanent link to its code (which is not done in academic papers) and point out its literature connection.
+Jon is a first-year master’s student who is interested in reinforcement learning (RL). In his eyes, RL seemed fascinating because he could use RL libraries such as [Stable-Baselines3 (SB3)](https://github.com/DLR-RM/stable-baselines3) to train agents to play all kinds of games. He quickly recognized Proximal Policy Optimization (PPO) as a fast and versatile algorithm and wanted to implement PPO himself as a learning experience. Upon reading the paper, Jon thought to himself, “huh, this is pretty straightforward.” He then opened a code editor and started writing PPO. `CartPole-v1` from Gym was his chosen simulation environment, and before long, Jon made PPO work with `CartPole-v1`. He had a great time and felt motivated to make his PPO work with more interesting environments, such as the Atari games and MuJoCo robotics tasks. “How cool would that be?” he thought.
-The 37 details break down as:
-- 13 core implementation details
-- 9 Atari-specific implementation details
-- 9 implementation details for robotics tasks (continuous action spaces)
-- 5 LSTM implementation details
-- 1 `MultiDiscrete` action-spaces implementation detail
-- (plus 4 situational details not used in the official implementation)
+However, he soon struggled. Making PPO work with Atari and MuJoCo seemed more challenging than anticipated. Jon then looked for reference implementations online but was shortly overwhelmed: unofficial repositories all appeared to do things differently, whereas he just could not read the Tensorflow `1.x` code in the official repo. Fortunately, Jon stumbled across two recent papers that explain PPO’s implementations. “This is it!” he grinned. Failing to control his excitement, Jon started running around in the office, accidentally bumping into Sam, whom Jon knew was working on RL. They then had the following conversation:
-> Our ultimate purpose is to help people understand the PPO implementation through and through, reproduce past results with high fidelity, and facilitate customization for new research.
+* “Hey, I just read the _implementation details matter_ paper and the _what matters in on-policy RL_ paper. Fascinating stuff. I knew PPO wasn’t that easy!” Jon exclaimed.
+* “Oh yeah! PPO is tricky, and I love these two papers that dive into the nitty-gritty details.” Sam answered.
+* “Indeed. I feel I understand PPO much better now. You have been working with PPO, right? Quiz me on PPO!” Jon inquired enthusiastically.
+* “Sure. If you run the official PPO with the Atari game Breakout, the agent would get ~400 game scores in about 4 hours. Do you know how does PPO achieve that?”
+* “Hmm… That’s actually a good question. I don’t think the two papers explain that.”
+* “The procgen paper contains experiments conducted using the official PPO with LSTM. Do you know how does PPO + LSTM work?”
+* “Ehh… I haven’t read too much on PPO + LSTM” Jon admitted.
+* “The official PPO also works with `MultiDiscrete` action space where you can use multiple discrete values to describe an action. Do you know how that works?”
+* “…” Jon, speechless.
+* “Lastly, if you have only the standard tools (e.g., `numpy, gym...`) and a neural network library (e.g., `torch, jax,...`), could you code up PPO from scratch?”
+* “Ooof, I guess it’s going to be difficult. Prior papers analyzed PPO implementation details but didn’t show how these pieces are coded together. Also, I now realize their conclusions are in MuJoCo tasks and do not necessarily transfer to other games such as Atari. I feel sad now…” Jon sighed.
+* “Don’t feel bad. PPO is just a complicated beast. If anything helps, I have been making video tutorials on implementing PPO from scratch and a blog post explaining things in more depth!”
-Context: the official PPO implementation (`openai/baselines`, `ppo2`) has undergone several refactorings, so "it is important to recognize *which version* of the official implementation is worth studying." Libraries that match `ppo2`'s details closely (Stable-Baselines3, CleanRL) reproduce similar results; others report more diverse (worse) results.
+
+
+And the blog post is here! Instead of doing ablation studies and making recommendations on which details matter, this blog post takes a step back and focuses on reproductions of PPO’s results in all accounts. Specifically, this blog post complements prior work in the following ways:
+
+1. **Genealogy Analysis:** we establish what it means to reproduce the **official PPO implementation** by examining its historical revisions in the `openai/baselines` GitHub repository (the official repository for PPO). As we will show, the code in the `openai/baselines` repository has undergone several refactorings which could produce different results from the original paper. So it is important to recognize _which version_ of the official implementation is worth studying.
+2. **Video Tutorials and Single-file Implementations:** we make video tutorials on re-implementing PPO in PyTorch from scratch, matching details in the official PPO implementation to handle classic control tasks, Atari games, and MuJoCo tasks. Notably, we adopt single-file implementations in our code base, making the code quicker and easier to read. The videos are shown below:
+
+[Video 13](https://www.youtube.com/watch?v=MEt6rrxH8W4)[Video 14](https://www.youtube.com/watch?v=05RMTj-2K_Y)[Video 15](https://www.youtube.com/watch?v=BvZvx7ENZBw)
+
+1. **Implementation Checklist with References:** During our re-implementation, we have compiled an implementation checklist containing 37 details as follows. For each implementation detail, we display the permanent link to its code (which is not done in academic papers) and point out its literature connection.
+ * 13 core implementation details
+ * 9 Atari specific implementation details
+ * 9 implementation details for robotics tasks (with continuous action spaces)
+ * 5 LSTM implementation details
+ * 1 `MultiDiscrete` action spaces implementation detail
+
+2. **High-fidelity Reproduction:** To validate our re-implementation, we show that the empirical results of our implementation match closely with those of the original, in classic control tasks, Atari games, MuJoCo tasks, LSTM, and Real-time Strategy (RTS) game tasks.
+3. **Situational Implementation Details:** We also cover 4 implementation details not used in the official implementation but potentially useful on special occasions.
+
+Our ultimate purpose is to help people understand the PPO implementation through and through, reproduce past results with high fidelity, and facilitate customization for new research. To make research reproducible, we have made source code available at [https://github.com/vwxyzjn/ppo-implementation-details](https://github.com/vwxyzjn/ppo-implementation-details) and the tracked experiments available at [https://wandb.ai/vwxyzjn/ppo-details](https://wandb.ai/vwxyzjn/ppo-details)
+
+## Background
+
+PPO is a policy gradient algorithm proposed by [Schulman et al., (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017). As a refinement to Trust Region Policy Optimization (TRPO) ([Schulman et al., 2015](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015)), PPO uses a simpler clipped surrogate objective, omitting the expensive second-order optimization presented in TRPO. Despite this simpler objective, [Schulman et al., (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017) show PPO has higher sample efficiency than TRPO in many control tasks. PPO also has good empirical performance in the arcade learning environment (ALE) which contain Atari games.
+
+To facilitate more transparent research, [Schulman et al., (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017) have made the source code of PPO available in the `openai/baselines` GitHub repository with the code name `pposgd` (commit [da99706](https://github.com/openai/baselines/tree/da997060461e3cbf54ca4dc7a67081a731fb6b3b/baselines/pposgd) on 7/20/2017). Later, the `openai/baselines` maintainers have introduced a series of revisions. The key events include:
+
+1. 11/16/2017, commit [2dd7d30](https://github.com/openai/baselines/tree/2dd7d307d7d163a02b37c87c62b7949af02d99ad/baselines/ppo2): the maintainers introduced a refactored version `ppo2` and renamed `pposgd` to `ppo1`. According to a [GitHub issue](https://github.com/openai/baselines/issues/485#issuecomment-413722708), one maintainer suggests `ppo2` should offer better GPU utilization by batching observations from multiple simulation environments.
+2. 8/10/2018, commit [ea68f3b](https://github.com/openai/baselines/commits/ea68f3b7e6a20d4c6bf1e32f8fb5ce18e6ef3a89): after a few revisions, the maintainers evaluated `ppo2`, producing the [MuJoCo benchmark](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/benchmarks_mujoco1M.htm)
+3. 10/4/2018, commit [7bfbcf1](https://github.com/openai/baselines/commit/7bfbcf177eca8f46c0c0bfbb378e044539f5e061): after a few revisions, the maintainers evaluated `ppo2`, producing the [Atari benchmark](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/benchmarks_mujoco1M.htm)
+4. 1/31/2020, commit [ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998): the maintainers have merged the last commit to `openai/baselines` to date. To our knowledge, `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) is the base of many PPO-related resources:
+ 1. RL libraries such [Stable-Baselines3 (SB3)](https://github.com/DLR-RM/stable-baselines3), [pytorch-a2c-ppo-acktr-gail](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail), and [CleanRL](https://github.com/vwxyzjn/cleanrl) have built their PPO implementation to match implementation details in `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) closely.
+ 2. Recent papers ([Engstrom, Ilyas, et al., 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom); [Andrychowicz, et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz)) have examined implementation details concerning robotics tasks in `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)).
+
+In recent years, reproducing PPO’s results has become a challenging issue. The following table collects the best-reported performance of PPO in popular RL libraries in Atari and MuJoCo environments.
+
+| RL Library | GitHub Stars | Benchmark Source | Breakout | Pong | BeamRider | Hopper | Walker2d | HalfCheetah |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| [Baselines](https://github.com/openai/baselines)`pposgd` / `ppo1` ([da99706](https://github.com/openai/baselines/tree/da997060461e3cbf54ca4dc7a67081a731fb6b3b/baselines/pposgd)) | [](https://github.com/openai/baselines/stargazers) | [paper](https://arxiv.org/abs/1707.06347) ($) | 274.8 | 20.7 | 1590 | ~2250 | ~3000 | ~1750 |
+| [Baselines](https://github.com/openai/baselines)`ppo2` ([7bfbcf1](https://github.com/openai/baselines/commit/7bfbcf177eca8f46c0c0bfbb378e044539f5e061) and [ea68f3b](https://github.com/openai/baselines/commits/ea68f3b7e6a20d4c6bf1e32f8fb5ce18e6ef3a89)) | | [docs](https://github.com/openai/baselines/blob/master/benchmarks_atari10M.htm) (*) | 114.26 | 13.68 | 1299.25 | 2316.16 | 3424.95 | 1668.58 |
+| [Baselines](https://github.com/openai/baselines)`ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) | | this blog post (*) | 409.265 ± 30.98 | 20.59 ± 0.40 | 2627.96 ± 625.751 | 2448.73 ± 596.13 | 3142.24 ± 982.25 | 2148.77 ± 1166.023 |
+| [Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) | [](https://github.com/DLR-RM/stable-baselines3/stargazers) | [docs](https://github.com/DLR-RM/rl-baselines3-zoo/blob/111d03c4ce728fff51d4b1c10355ea612bc8d456/benchmark.md) (0) (^) | 398.03 ± 33.28 | 20.98 ± 0.10 | 3397.00 ± 1662.36 | 2410.43 ± 10.02 | 3478.79 ± 821.70 | 5819.09 ± 663.53 |
+| [CleanRL](https://github.com/vwxyzjn/cleanrl) | [](https://github.com/vwxyzjn/cleanrl/stargazers) | [docs](https://wandb.ai/cleanrl/cleanrl.benchmark/reports/Open-RL-Benchmark-0-6-0---Vmlldzo0MDcxOA) (1) (*) | ~402 | ~20.39 | ~2131 | ~2685 | ~3753 | ~1683 |
+| [Tianshou](https://github.com/thu-ml/tianshou) | [](https://github.com/thu-ml/tianshou/stargazers) | [paper](https://arxiv.org/pdf/2107.14171.pdf), [docs](https://github.com/thu-ml/tianshou/blob/f13e415eb0de55baca5dc0d6fae39d6a38e8bc0b/examples/atari/README.md) (5) (^) | ~400 | ~20 | - | 7337.4 ± 1508.2 | 3127.7 ± 413.0 | 4895.6 ± 704.3 |
+| [Ray/RLlib](https://github.com/ray-project/ray/tree/master/rllib/) | [](https://github.com/ray-project/ray/stargazers) | [repo](https://github.com/ray-project/rl-experiments/tree/9543891717cd0f8e137e23812229a06f8ed1c6c2) (2) (*) | 201 | - | 4480 | - | - | 9664 |
+| [SpinningUp](https://github.com/openai/spinningup) | [](https://github.com/openai/spinningupstargazers) | [docs](https://spinningup.openai.com/en/latest/spinningup/bench.html#id12) (3) (^) | - | - | - | ~2500 | ~2500 | ~3000 |
+| [ChainerRL](https://github.com/chainer/chainerrl) | [](https://github.com/chainer/chainerrl/stargazers) | [paper](https://arxiv.org/pdf/1912.03905.pdf) (4) (*) | - | - | - | 2719 ± 67 | 2994 ± 113 | 2404 ± 185 |
+| [Tonic](https://github.com/fabiopardo/tonic) | [](https://github.com/fabiopardo/tonic/stargazers) | [paper](https://arxiv.org/pdf/2011.07537.pdf) (6) (^) | - | - | - | ~2000 | ~4500 | ~5000 |
+
+(-): No publicly reported metrics available
+
+($): The experiments uses the v1 MuJoCo environments
+
+(*): The experiments uses the v2 MuJoCo environments
+
+(^): The experiments uses the v3 MuJoCo environments
+
+(0): 1M steps for MuJoCo experiments, 10M steps for Atari games, 1 random seed
+
+(1): 2M steps for MuJoCo experiments, 10M steps for Atari games, 2 random seeds
+
+(2): 25M steps and 10 workers (5 envs per worker) for Atari experiments; 44M steps and 16 workers for MuJoCo experiments; 1 random seed
+
+(3): 3M steps, PyTorch version, 10 random seeds
+
+(4): 2M steps, 10 random seeds
+
+(5): 3M steps, 10 random seeds for MuJoCo experiments; 10M steps, 1 random seed for Atari experiment
+
+(6): 5M steps, 10 random seeds
+
+We offer several observations.
+
+1. These revisions in `openai/baselines` are not without performance consequences. Reproducing PPO’s results is challenging partly because even the original implementation could produce inconsistent results.
+2. `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) and libraries matching its implementation details have reported rather similar results. In comparison, other libraries have usually reported more diverse results.
+3. Interestingly, we have found many libraries reported performance in MuJoCo tasks but not in Atari tasks.
+
+Despite the complicated situation, we have found `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) as an implementation worth studying. It obtains good performance in both Atari and MuJoCo tasks. More importantly, it also incorporates advanced features such as LSTM and treatment of the `MultiDiscrete` action space, unlocking application to more complicated games such as Real-time Strategy games. As such, we define `ppo2` ([ea25b9e](https://github.com/openai/baselines/commit/ea25b9e8b234e6ee1bca43083f8f3cf974143998)) as the **official PPO implementation** and base the remainder of this blog post on this implementation.
+
+## Reproducing the official PPO implementation
+
+In this section, we introduce five categories of implementation details and implement them in PyTorch from scratch.
+
+* 13 core implementation details
+* 9 Atari specific implementation details
+* 9 implementation details for robotics tasks (with continuous action spaces)
+* 5 LSTM implementation details
+* 1 `MultiDiscrete` implementation detail
+
+For each category (except the first one), we benchmark our implementation against the original implementation in three environments, each with three random seeds.
+
+## 13 core implementation details
+
+We first introduce the 13 core implementation details commonly used regardless of the tasks. To help understand how to code these details in PyTorch, we have prepared a line-by-line video tutorial as follows. Note that the video tutorial skips over the 12-th and 13-th implementation details during its making, hence the video has the title “11 Core Implementation Details”
+
+[Video 16](https://www.youtube.com/watch?v=MEt6rrxH8W4)
+
+1. Vectorized architecture ([common/cmd_util.py#L22](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L22)) Code-level Optimizations
+ * PPO leverages an efficient paradigm known as the **vectorized architecture** that features a single learner that collects samples and learns from multiple environments. Below is a pseudocode: ```
+envs = VecEnv(num_envs=N)
+ agent = Agent()
+ next_obs = envs.reset()
+ next_done = [0, 0, ..., 0] # of length N
+ for update in range(1, total_timesteps // (N*M)):
+ data = []
+ # ROLLOUT PHASE
+ for step in range(0, M):
+ obs = next_obs
+ done = next_done
+ action, other_stuff = agent.get_action(obs)
+ next_obs, reward, next_done, info = envs.step(
+ action
+ ) # step in N environments
+ data.append([obs, action, reward, done, other_stuff]) # store data
+
+ # LEARNING PHASE
+ agent.learn(data, next_obs, next_done) # `len(data) = N*M`
+```
+ * In this architecture, PPO first initializes a **vectorized environment**`envs` that runs $N$ (usually independent) environments either sequentially or in parallel by leveraging multi-processes. `envs` presents a synchronous interface that always outputs a batch of $N$ observations from $N$ environments, and it takes a batch of $N$ actions to step the $N$ environments. When calling `next_obs = envs.reset()`, `next_obs` gets a batch of $N$ initial observations (pronounced “next observation”). PPO also initializes an environment done flag variable `next_done` (pronounced “next done”) to an $N$-length array of zeros, where its i-th element `next_done[i]` has values of 0 or 1 which corresponds to the $i$-th sub-environment being _not done_ and _done_, respectively.
+ * Then, the vectorized architecture loops two phases: the **rollout phase** and the **learning phase**:
+ * Rollout phase : The agent samples actions for the $N$ environments and continue to step them for a fixed number of $M$ steps. During these $M$ steps, the agent continues to append relevant data in an empty list `data`. If the $i$-th sub-environment is done (terminated or truncated) after stepping with the $i$-th action `action[i]`, `envs` would set its returned `next_done[i]` to 1, auto-reset the $i$-th sub-environment and fill `next_obs[i]` with the initial observation in the new episode of the $i$-th environment.
+ * Learning phase: The agent in principal learns from the collected data in the rollout phase: `data` of length $N M$, `next_obs` and `done`. Specifically, PPO can estimate value for the next observation `next_obs` conditioned on `next_done` and calculate the advantage `advantages` and the return `returns`, both of which also has length $N M$. PPO then learns from the prepared data `[data, advantages, returns]`, which is called “fixed-length trajectory segments” by [(Schulman et al., 2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017).
+ * **It is important to understand `next_obs` and `next_done`’s role to help transition between phases**: At the end of the $j$-th rollout phase, `next_obs` can be used to estimate the value of the final state during learning phase, and in the begining of the $\left(\right. j + 1 \left.\right)$-th rollout phase, `next_obs` becomes the initial observation in `data`. Likewise, `next_done` tells if `next_obs` is actually the first observation of a new episode. This intricate design allows PPO to continue step the sub-environments, and because agent always learns from fixed-length trajectory segments after $M$ steps, PPO can train the agent even if the sub-environments never terminate or truncate. This is in principal why PPO can learn in long-horizon games that last 100,000 steps [(default truncation limit for Atari games in `gym`)](https://github.com/openai/gym/blob/a7b6462136ebaa610c8941e4da8a9c92155b04d1/gym/envs/__init__.py#L744) in a single episode.
+
+ * A common incorrect implementation is to train PPO based on episodes and setting a maximum episode horizon. Below is a pseudocode. ```
+env = Env()
+ agent = Agent()
+ for episode in range(1, num_episodes):
+ next_obs = env.reset()
+ data = []
+ for step in range(1, max_episode_horizon):
+ obs = next_obs
+ action, other_stuff = agent.get_action(obs)
+ next_obs, reward, done, info = env.step(action)
+ data.append([obs, action, reward, done, other_stuff]) # store data
+ if done:
+ break
+ agent.learn(data)
+```
+ * There are several downsides to this approach. First, it can be inefficient because the agent has to do one forward pass per environment step. Second, it does not scale to games with larger horizons such as StarCraft II (SC2). A single episode of the SC2 could last 100,000 steps, which bloats the memory requirement in this implementation.
+ * The vectorized architecture handles this 100,000 steps by learning from **fixed-length trajectory segments**. If we set $N = 2$ and $M = 100$, the agent would learn from the first 100 steps from 2 independent environments. Then, note that the `next_obs` is the 101st observation from these two environments, and the agent can keep doing rollouts and learn from the 101 to 200 steps from the 2 environments. Essentially, the agent learns partial trajectories of the episode, $M$ steps at a time.
+
+ * $N$ is the `num_envs` (decision C1) and $M * N$ is the `iteration_size` (decision C2) in [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz), who suggest increasing $N$ (such as $N = 256$) boosts the training throughput but makes the performance worse. They argued the performance deterioration was due to “shortened experience chunks” ($M$ becomes smaller due to the increase in $N$ in their setup ) and “earlier value bootstrapping.” While we agree increasing $N$ could hurt sample efficiency, we argue the evaluation should be based on wall-clock time efficiency. That is, if the algorithm terminates much sooner with a larger $N$ compared to other configurations, why not run the algorithm longer? Although being a different robotics simulator, [Brax](https://github.com/google/brax) follows this idea and can train a viable agent in similar tasks with PPO using a massive $N = 2048$ and a small $M = 20$ yet finish the training in one minute.
+ * The vectorized environments also support multi-agent reinforcement learning (MARL) environments. Below is the quote from ([gym3](https://github.com/openai/gym3)) using our notation:
+> In the simplest case, a vectorized environment corresponds to a single multiplayer game with $N$ players. If we run an RL algorithm in this environment, we are doing self-play without historical opponents. This setup can be straightforwardly extended to having $K$ concurrent games with $H$ players each, with $N = H * K$.
+
+ * For example, if there is a two-player game, we can create a vectorized environment that spawns two sub-environments. Then, the vectorized environment produces a batch of two observations, where the first observation is from player 1 and the second observation is from player 2. Next, the vectorized environment takes a batch of two actions and tells the game engine to let player 1 execute the first action and player 2 execute the second action. Consequently, PPO learns to control both player 1 and player 2 in this vectorized environment.
+ * Such MARL usage is widely adopted in games such as Gym-μRTS ([Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021)), Pettingzoo ([Terry et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Terry)), etc.
+
+2. Orthogonal Initialization of Weights and Constant Initialization of biases ([a2c/utils.py#L58)](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L58)) Neural Network Code-level Optimizations
+ * The related code is across multiple files in the `openai/baselines` library. The code for such initialization is in [a2c/utils.py#L58](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L58), when in fact it is used for other algorithms such as PPO. In general, the weights of _hidden_ layers use orthogonal initialization of weights with scaling `np.sqrt(2)`, and the biases are set to `0`, as shown in the CNN initialization for Atari ([common/models.py#L15-L26](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L15-L26)), and the MLP initialization for Mujoco ([common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103)). However, the policy output layer weights are initialized with the scale of `0.01`. The value output layer weights are initialized with the scale of `1` ([common/policies.py#L49-L63](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L49-L63)).
+ * It seems the implementation of the orthogonal initialization of `openai/baselines` ([a2c/utils.py#L20-L35](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L20-L35)) is different from that of pytorch/pytorch ([torch.nn.init.orthogonal_](https://pytorch.org/docs/stable/_modules/torch/nn/init.html#orthogonal_)). However, we consider this to be a very low-level detail that should not impact the performance.
+ * [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find orthogonal initialization to outperform the default Xavier initialization in terms of the highest episodic return achieved. Also, [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find centering the action distribution around 0 (i.e., initialize the policy output layer weights with 0.01”) to be beneficial (decision C57).
+
+3. The Adam Optimizer’s Epsilon Parameter ([ppo2/model.py#L100](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L100)) Code-level Optimizations
+ * PPO sets the epsilon parameter to `1e-5`, which is different from the default epsilon of `1e-8` in PyTorch and `1e-7` in TensorFlow. We list this implementation detail because the epsilon parameter is neither mentioned in the paper nor a configurable parameter in the PPO implementation. While this implementation detail may seem over specific,it is important that we match it for a high-fidelity reproduction.
+ * [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) perform a grid search on Adam optimizer’s parameters (decision C24, C26, C28) and recommend $\beta_{1} = 0.9$ and use the Tensorflow’s default epsilon parameter `1e-7`. [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) use the default PyTorch epsilon parameter `1e-8`.
+
+4. Adam Learning Rate Annealing ([ppo2/ppo2.py#L133-L135](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L133-L135)) Code-level Optimizations
+ * The Adam optimizer’s learning rate could be either constant or set to decay. By default, the hyper-parameters for training agents playing Atari games set the learning rate to linearly decay from `2.5e-4` to `0` as the number of timesteps increases. In MuJoCo, the learning rate linearly decays from `3e-4` to `0`.
+ * [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find adam learning rate annealing to help agents obtain higher episodic return. Also, [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) have also found learning rate annealing helpful as it increases performance in 4 out of 5 tasks examined, although the performance gains are relatively small (decision C31, figure 65).
+
+5. Generalized Advantage Estimation ([ppo2/runner.py#L56-L65](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L56-L65)) Theory
+ * Although the PPO paper uses the abstraction of advantage estimate in the PPO’s objective, the PPO implementation does use Generalized Advantage Estimation ([Schulman, 2015b](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015b)). Two important sub-details:
+ * Value bootstrap ([ppo2/runner.py#L50](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L50)): if a sub-environment is _not_ terminated nor truncated, PPO estimates the value of the next state in this sub-environment as the value target.
+ * **A note on truncation**: Almost all `gym` environments have a time limit and will truncate themselves if they run too long. For example, the `CartPole-v1` has a 500 time limit (see [link](https://github.com/openai/gym/blob/e9df4932434516c9f7956cc8010679a33835b204/gym/envs/__init__.py#L26)) and will return `done=True` if the game lasts for more than 500 steps. While the PPO implementation does not estimate value of the terminal state in the truncated environments, we (intuitively) should. Nonetheless, for high-fidelity reproduction, we did not implement the correct handling for truncated environments.
+
+ * $T D \left(\right. \lambda \left.\right)$ return estimation ([ppo2/runner.py#L65](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L65)): PPO implements the return target as `returns = advantages + values`, which corresponds to $T D \left(\right. \lambda \left.\right)$ for value estimation (where Monte Carlo estimation is a special case when $\lambda = 1$).
+
+ * [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find GAE to performan better than N-step returns (decision C6, figure 44 and 40).
+
+6. Mini-batch Updates ([ppo2/ppo2.py#L157-L166](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L157-L166)) Code-level Optimizations
+ * During the learning phase of the vectorized architecture, the PPO implementation shuffles the indices of the training data of size $N * M$ and breaks it into mini-batches to compute the gradient and update the policy.
+ * Some common mis-implementations include 1) always using the whole batch for the update, and 2) implementing mini-batches by randomly fetching from the training data (which does not guarantee all training data points are fetched).
+
+7. Normalization of Advantages ([ppo2/model.py#L139](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L139)) Code-level Optimizations
+ * After calculating the advantages based on GAE, PPO normalizes the advantages by subtracting their mean and dividing them by their standard deviation. In particular, _this normalization happens at the minibatch level instead of the whole batch level!_
+ * [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) (decision C67) find per-minibatch advantage normalization to not affect performance much (figure 35).
+
+8. Clipped surrogate objective ([ppo2/model.py#L81-L86](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L81-L86)) Theory
+ * PPO clips the objective as suggested in the paper.
+ * [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find the PPO’s clipped objective to have similar performance to TRPO’s objective when they controlled other implementation details to be the same. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find the PPO’s clipped objective to outperform vanilla policy gradient (PG), V-trace, AWR, and V-MPO in most tasks ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)).
+ * Based on the above findings, we argue PPO’s clipped objective is still a great objective because it achieves similar performance as TRPO’s objective while being computationally cheaper (i.e., without second order optimization as does in TRPO).
+
+9. Value Function Loss Clipping ([ppo2/model.py#L68-L75](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L68-L75)) Code-level Optimizations
+ * PPO clips the value function like the PPO’s clipped surrogate objective. Given the `V_{targ} = returns = advantages + values`, PPO fits the the value network by minimizing the following loss:
+
+$$
+L^{V} = max \left[\right. \left(\left(\right. V_{\theta_{t}} - V_{t a r g} \left.\right)\right)^{2} , \left(\left(\right. clip \left(\right. V_{\theta_{t}} , V_{\theta_{t - 1}} - \epsilon , V_{\theta_{t - 1}} + \epsilon \left.\right) - V_{t a r g} \left.\right)\right)^{2} \left]\right.
+$$
+
+ * [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) find no evidence that the value function loss clipping helps with the performance. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) suggest value function loss clipping even hurts performance (decision C13, figure 43).
+ * We implemented this detail because this work is more about high-fidelity reproduction of prior results.
+
+10. Overall Loss and Entropy Bonus ([ppo2/model.py#L91](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L91)) Theory
+ * The overall loss is calculated as `loss = policy_loss - entropy * entropy_coefficient + value_loss * value_coefficient`, which maximizes an entropy bonus term. Note that the policy parameters and value parameters share the same optimizer.
+ * Mnih et al. have reported this entropy bonus to improve exploration by encouraging the action probability distribution to be slightly more random.
+ * [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) overall find no evidence that the entropy term improves performance on continuous control environments (decision C13, figure 76 and 77).
+
+11. Global Gradient Clipping ([ppo2/model.py#L102-L108](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L102-L108)) Code-level Optimizations
+ * For each update iteration in an epoch, PPO rescales the gradients of the policy and value network so that the “global l2 norm” (i.e., the norm of the concatenated gradients of all parameters) does not exceed `0.5`.
+ * [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find global gradient clipping to offer a small performance boost (decision C68, figure 34).
+
+12. Debug variables ([ppo2/model.py#L115-L116](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L115-L116))
+ * The PPO implementation comes with several debug variables, which are
+ 1. `policy_loss`: the mean policy loss across all data points.
+ 2. `value_loss`: the mean value loss across all data points.
+ 3. `entropy_loss`: the mean entropy value across all data points.
+ 4. `clipfrac`: the fraction of the training data that triggered the clipped objective.
+ 5. `approxkl`: the approximate Kullback–Leibler divergence, measured by `(-logratio).mean()`, which corresponds to the `k1` estimator in John Schulman’s blog post on [approximating KL divergence](http://joschu.net/blog/kl-approx.html). This blog post also suggests using an alternative estimator `((ratio - 1) - logratio).mean()`, which is unbiased and has less variance.
+
+13. Shared and separate MLP networks for policy and value functions ([common/policies.py#L156-L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L156-L160), [baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103))Neural Network Code-level Optimizations
+ * By default, PPO uses a simple MLP network consisting of two layers of 64 neurons and Hyperbolic Tangent as the activation function. Then PPO builds a policy head and value head that share the outputs of the MLP network. Below is a pseudocode: ```
+network = Sequential(
+ layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+ Tanh(),
+ layer_init(Linear(64, 64)),
+ Tanh(),
+ )
+ value_head = layer_init(Linear(64, 1), std=1.0)
+ policy_head = layer_init(Linear(64, envs.single_action_space.n), std=0.01)
+ hidden = network(observation)
+ value = value_head(hidden)
+ action = Categorical(policy_head(hidden)).sample()
+```
+ * Alternatively, PPO could build a policy function and a value function using separate networks by toggling the `value_network='copy'` argument. Then the pseudocode looks like this: ```
+value_network = Sequential(
+ layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+ Tanh(),
+ layer_init(Linear(64, 64)),
+ Tanh(),
+ layer_init(Linear(64, 1), std=1.0),
+ )
+ policy_network = Sequential(
+ layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+ Tanh(),
+ layer_init(Linear(64, 64)),
+ Tanh(),
+ layer_init(Linear(64, envs.single_action_space.n), std=0.01),
+ )
+ value = value_network(observation)
+ action = Categorical(policy_network(observation)).sample()
+```
+
+We incorporate the first 12 details and the **separate-networks architecture** to produce a self-contained `ppo.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo.py)) that has 322 lines of code. Then, we make about [10 lines of code](https://www.diffchecker.com/07TdfFlg) change to adopt the **shared-network architecture**, resulting in a self-contained `ppo_shared.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_shared.py)) that has 317 lines of code. The following shows the file difference between the `ppo.py` (left) and `ppo_shared.py` (right).
+
+Below are the benchmarked results.
+
+
+
+> Tracked classic control experiments (click to show the interactive panel)
+
+While shared-network architecture is the default setting in PPO, the separate-networks architecture clearly outperforms in simpler environments. The shared-network architecture performs worse probably due to the competing objectives of the policy and value functions. For this reason, we implement the separate-networks architecture in the video tutorial.
+
+## 9 Atari-specific implementation details
+
+Next, we introduce the 9 Atari-specific implementation details. To help understand how to code these details in PyTorch, we have prepared a line-by-line video tutorial.
+
+[Video 17](https://www.youtube.com/watch?v=05RMTj-2K_Y)
+
+1. The Use of `NoopResetEnv` ([common/atari_wrappers.py#L12](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L12)) Environment Preprocessing
+ * This wrapper samples initial states by taking a random number (between 1 and 30) of no-ops on reset.
+ * The source of this wrapper comes from [(Mnih et al., 2015, Extended Data Table 1)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) and [Machado et al., 2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Machado2018) have suggested `NoopResetEnv` is a way to inject stochasticity to the environment.
+
+2. The Use of `MaxAndSkipEnv` ([common/atari_wrappers.py#L97](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L97)) Environment Preprocessing
+ * This wrapper skips 4 frames by default, repeats the agent’s last action on the skipped frames, and sums up the rewards in the skipped frames. Such frame-skipping technique could considerably speed up the algorithm because the environment step is computationally cheaper than the agent’s forward pass [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015).
+ * This wrapper also returns the maximum pixel values over the last two frames to help deal with some Atari game quirks [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015).
+ * The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below.
+> More precisely, the agent sees and selects actions on every $k$-th frame instead of every frame, and its last action is repeated on skipped frames. Because running the emulator forward for one step requires much less computation than having the agent select an action, this technique allows the agent to play roughly $k$ times more games without significantly increasing the runtime. We use $k = 4$ for all games. […] First, to encode a single frame we take the maximum value for each pixel color value over the frame being encoded and the previous frame. This was necessary to remove flickering that is present in games where some objects appear only in even frames while other objects appear only in odd frames, an artifact caused by the limited number of sprites Atari 2600 can display at once.
+
+3. The Use of `EpisodicLifeEnv` ([common/atari_wrappers.py#L61](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L61)) Environment Preprocessing
+ * In the games where there are a life counter such as breakout, this wrapper marks the end of life as the end of episode.
+ * The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below.
+> For games where there is a life counter, the Atari 2600 emulator also sends the number of lives left in the game, which is then used to mark the end of an episode during training.
+
+ * Interestingly, [(Bellemare et al., 2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/Bellemare2016b) Note this the wrapper could be detrimental to the agent’s performance and [Machado et al., 2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Machado2018) have suggested not using this wrapper.
+
+4. The Use of `FireResetEnv` ([common/atari_wrappers.py#L41](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L41)) Environment Preprocessing
+ * This wrapper takes the `FIRE` action on reset for environments that are fixed until firing.
+ * This wrapper is interesting because there is no literature reference to our knowledge. According to anecdotal conversations([openai/baselines#240](https://github.com/openai/baselines/issues/240)), neither people from DeepMind nor OpenAI know where this wrapper comes from. So… 
+
+5. The Use of `WarpFrame` (Image transformation) [common/atari_wrappers.py#L134](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L134)Environment Preprocessing
+ * This wrapper warps extracts the Y channel of the 210x160 pixel images and resizes it to 84x84.
+ * The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below.
+> Second, we then extract the Y channel, also known as luminance, from the RGB frame and rescale it to 84x84.
+
+ * In our implementation, we use the following wrappers to achieve the same purpose. ```
+env = gym.wrappers.ResizeObservation(env, (84, 84))
+ env = gym.wrappers.GrayScaleObservation(env)
+```
+
+6. The Use of `ClipRewardEnv` ([common/atari_wrappers.py#L125](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L125)) Environment Preprocessing
+ * This wrapper bins reward to `{+1, 0, -1}` by its sign.
+ * The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below.
+> As the scale of scores varies greatly from game to game, we clipped all positive rewards at 1 and all negative rewards at -1, leaving 0 rewards unchanged. Clipping the rewards in this manner limits the scale of the error derivatives and makes it easier to use the same learning rate across multiple games. At the same time, it could affect the performance of our agent since it cannot differentiate between rewards of different magnitude.
+
+7. The Use of `FrameStack` ([common/atari_wrappers.py#L188](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L188)) Environment Preprocessing
+ * This wrapper stacks $m$ last frames such that the agent can infer the velocity and directions of moving objects.
+ * The source of this wrapper comes from [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) as shown by the quote below.
+> The function $\theta$ from algorithm 1 described below applies this preprocessing to the $m$ most recent frames and stacks them to produce the input to the Q-function, in which $m = 4$.
+
+8. Shared Nature-CNN network for the policy and value functions ([common/policies.py#L157](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L157), [common/models.py#L15-L26](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L15-L26))Neural Network
+ * For Atari games, PPO uses the same Convolutional Neural Network (CNN) in [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015) along with the layer initialization technique mentioned earlier ([baselines/a2c/utils.py#L52-L53](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L52-L53)) to extract features, flatten the extracted features, apply a linear layer to compute the hidden features. Afterward, the policy and value functions share parameters by constructing a policy head and a value head using the hidden features. Below is a pseudocode: ```
+hidden = Sequential(
+ layer_init(Conv2d(4, 32, 8, stride=4)),
+ ReLU(),
+ layer_init(Conv2d(32, 64, 4, stride=2)),
+ ReLU(),
+ layer_init(Conv2d(64, 64, 3, stride=1)),
+ ReLU(),
+ Flatten(),
+ layer_init(Linear(64 * 7 * 7, 512)),
+ ReLU(),
+ )
+ policy = layer_init(Linear(512, envs.single_action_space.n), std=0.01)
+ value = layer_init(Linear(512, 1), std=1)
+```
+ * Such a parameter-sharing paradigm obviously computes faster when compared to setting completely separate networks, which would look like the following. ```
+policy = Sequential(
+ layer_init(Conv2d(4, 32, 8, stride=4)),
+ ReLU(),
+ layer_init(Conv2d(32, 64, 4, stride=2)),
+ ReLU(),
+ layer_init(Conv2d(64, 64, 3, stride=1)),
+ ReLU(),
+ Flatten(),
+ layer_init(Linear(64 * 7 * 7, 512)),
+ ReLU(),
+ layer_init(Linear(512, envs.single_action_space.n), std=0.01)
+ )
+ value = Sequential(
+ layer_init(Conv2d(4, 32, 8, stride=4)),
+ ReLU(),
+ layer_init(Conv2d(32, 64, 4, stride=2)),
+ ReLU(),
+ layer_init(Conv2d(64, 64, 3, stride=1)),
+ ReLU(),
+ Flatten(),
+ layer_init(Linear(64 * 7 * 7, 512)),
+ ReLU(),
+ layer_init(Linear(512, 1), std=1)
+ )
+```
+ * However, recent work suggests balancing the competing policy and value objective could be problematic, which is what methods like Phasic Policy Gradient are trying to address ([Cobbe et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe2021)).
+
+9. Scaling the Images to Range [0, 1] ([common/models.py#L19](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/models.py#L19)) Environment Preprocessing
+ * The input data has the range of [0,255], but it is divided by 255 to be in the range of [0,1].
+ * Our anecdotal experiments found this scaling important. Without it, the first policy update results in the Kullback–Leibler divergence explosion, likely due to how the layers are initialized.
+
+To run the experiments, we match the hyperparameters used in the original implementation as follows.
+
+```
+# https://github.com/openai/baselines/blob/master/baselines/ppo2/defaults.py
+def atari():
+ return dict(
+ nsteps=128, nminibatches=4,
+ lam=0.95, gamma=0.99, noptepochs=4, log_interval=1,
+ ent_coef=.01,
+ lr=lambda f : f * 2.5e-4,
+ cliprange=0.1,
+ )
+```
+
+These hyperparameters are
+
+* `nsteps` is the $M$ explained in this blog post .
+* `nminibatches` is the number of minibatches used for update (i.e., our 6th implementation detail).
+* `lam` is the GAE’s $\lambda$ parameter.
+* `gamma` is the discount factor.
+* `noptepochs` is the $K$ epochs in the original PPO paper.
+* `ent_coef` is the `entropy_coefficient` in our 10th implementation detail.
+* `lr=lambda f : f * 2.5e-4` is a learning rate schedule (i.e., our 4th implementation detail)
+* `cliprange=0.1` is the clipping parameter $\epsilon$ in the original PPO paper.
+
+Note that the number of environments parameter $N$ (i.e., `num_envs`) is set to the number of CPUs in the computer ([common/cmd_util.py#L167](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L167)), which is strange. We have chosen instead to match the `N=8` used in the paper (the paper listed the parameter as “number of actors, 8”).
+
+As shown below, we make [~40 lines of code](https://www.diffchecker.com/Dq5NfuQH) change to `ppo.py` to incorporate these 9 details, resulting in a self-contained `ppo_atari.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_atari.py)) that has 339 lines of code. The following shows the file difference between the `ppo.py` (left) and `ppo_atari.py` (right).
+
+Below are the benchmarked results.
+
+
+
+> Tracked Atari experiments (click to show the interactive panel)
+
+## 9 details for continuous action domains (e.g. Mujoco)
+
+Next, we introduce the 9 details for continuous action domains such as MuJoCo tasks. To help understand how to code these details in PyTorch, we have prepared a line-by-line video tutorial. Note that the video tutorial skips over the 4-th implementation detail during its making, hence the video has the title “8 Details for Continuous Actions”
+
+[Video 18](https://www.youtube.com/watch?v=BvZvx7ENZBw)
+
+1. Continuous actions via normal distributions ([common/distributions.py#L103-L104](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L103-L104)) Theory
+ * Policy gradient methods (including PPO) assume the continuous actions are sampled from a normal distribution. So to create such distribution, the neural network needs to output the mean and standard deviation of the continuous action.
+ * It is very popular to choose Gaussian distribution to represent the action distribution when the reinforcement learning algorithm is implemented in the environment of continuous action space. For example: [Schulman et al., (2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015) and [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016).
+
+2. State-independent log standard deviation ([common/distributions.py#L104](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L104)) Theory
+ * The implementation outputs the logits for the mean, but instead of outputting the logits for the standard deviation, it outputs the _logarithm_ of the standard deviation. In addition, this `log std` is set to be _state-independent and initialized to be 0._
+ * [Schulman et al., (2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2015) and [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016) use state-independent standard deviation, while [Haarnoja et al., (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Haarnoja2018) uses the state-dependent standard deviation, that is, the mean and standard deviation are output at the same time. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) compared two different implementations and found that the performance is very close (decision C59, figure 23).
+
+3. Independent action components ([common/distributions.py#L238-L246](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L238-L246)) Theory
+ * In many robotics tasks, it is common to have multiple scalar values to represent a continuous action. For example, the action of $a_{t} = \left[\right. a_{t}^{1} , a_{t}^{2} \left]\right. = \left[\right. 2.4 , 3.5 \left]\right.$ might mean to move left for 2.4 meters and move up 3.5 meters. However, most literature on policy gradient suggests the action $a_{t}$ would be a single scalar value. To account for this difference, PPO treats $\left[\right. a_{t}^{1} , a_{t}^{2} \left]\right.$ as probabilistically independent action components, therefore calculating $p r o b \left(\right. a_{t} \left.\right) = p r o b \left(\right. a_{t}^{1} \left.\right) \cdot p r o b \left(\right. a_{t}^{2} \left.\right)$.
+ * This approach comes from the currently commonly used assumption: Gaussian distribution with full covariance is used to represent the policy, which means that the action selection for each dimension is performed independently. When facing the environment of multi-dimensional action space, [Tavakoli, et al. (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Tavakoli2018) also believes that each action dimension should be selected independently and to achieve this goal by designing a network structure. Although our intuition tells us that there may be dependencies between action choices in different dimensions of policies in some environments, what is the optimal choice is still an open question. It is worth noting that this question has attracted the attention of the community, and began to try to model the dependencies of actions in different dimensions, such as using auto-regressive policy ([Metz, et al. (2019)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Metz2019), [Zhang, et al. (2019)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Zhang2018))
+
+4. Separate MLP networks for policy and value functions ([common/policies.py#L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L160), [baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103))Neural Network
+ * For continuous control tasks, PPO uses a simple MLP network consisting of two layers of 64 neurons and Hyperbolic Tangent as the activation function ([baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103)) for both the policy and value functions ([common/policies.py#L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L160)). Below is a pseudocode (also combining previous 3 details): ```
+value_network = Sequential(
+ layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+ Tanh(),
+ layer_init(Linear(64, 64)),
+ Tanh(),
+ layer_init(Linear(64, 1), std=1.0),
+ )
+ policy_mean = Sequential(
+ layer_init(Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
+ Tanh(),
+ layer_init(Linear(64, 64)),
+ Tanh(),
+ layer_init(Linear(64, envs.single_action_space.n), std=0.01),
+ )
+ policy_logstd = nn.Parameter(torch.zeros(1, np.prod(envs.single_action_space.shape)))
+ value = value_network(observation)
+ probs = Normal(
+ policy_mean(x),
+ policy_logstd.expand_as(action_mean).exp(),
+ )
+ action = probs.sample()
+ logprob = probs.log_prob(action).sum(1)
+```
+ * [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) find the separate policy and value networks generally lead to better performance (decision C47, figure 15).
+
+5. Handling of action clipping to valid range and storage ([common/cmd_util.py#L99-L100](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L99-L100)) Code-level Optimizations
+ * After a continuous action is sampled, such action could be invalid because it could exceed the valid range of continuous actions in the environment. To avoid this, add applies the rapper to clip the action into the valid range. However, the original unclipped action is stored as part of the episodic data ([ppo2/runner.py#L29-L31](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L29-L31)).
+ * Since the sampling of the Gaussian distribution has no boundaries, the environment usually has certain restrictions on the action space. So [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016) adopted clipping sampled actions into their bounds, [Haarnoja et al., (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Haarnoja2018) adopted invertible squashing function (tanh) to the Gaussian samples to satisfy constraints. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) Compared the two implementations and found that the tanh method is better (decision C63, figure 17). But in order to obtain consistent performance, we chose the implementation of clip. It is worth noting that [Chou 2017](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Chou2017) and [Fujita, et al. (2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Fujita2018) pointed out the bias brought by the clip method and proposed different solutions.
+
+6. Normalization of Observation ([common/vec_env/vec_normalize.py#L4](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L4)) Environment Preprocessing
+ * At each timestep, the `VecNormalize` wrapper pre-processes the observation before feeding it to the PPO agent. The raw observation was normalized by subtracting its running mean and divided by its variance.
+ * Using normalization on the input has become a well-known technique for training neural networks. [Duan et al., (2016)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Duan2016) adopted a moving average normalization for the observation to process the input of the network, which has also become the default choice for subsequent implementations. [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) experimentally determined that normalization for observation is very helpful for performance (decision C64, figure 33)
+
+7. Observation Clipping ([common/vec_env/vec_normalize.py#L39](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L39)) Environment Preprocessing
+ * Followed by the normalization of observation, the _normalized observation_ is further clipped by `VecNormalize` to a range, usually [−10, 10].
+ * [Andrychowicz, et al. (2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz) found that after normalization of observation, using observation clipping did not help performance (decision C65, figure 38), but guessed that it might be helpful in an environment with a wide range of observation.
+
+8. Reward Scaling ([common/vec_env/vec_normalize.py#L28](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L28)) Environment Preprocessing
+ * The `VecNormalize` also applies a certain discount-based scaling scheme, where the rewards are divided by the standard deviation of a rolling discounted sum of the rewards (without subtracting and re-adding the mean).
+ * [Engstrom, Ilyas, et al., (2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom) reported that reward scaling can significantly affect the performance of the algorithm and recommends the use of reward scaling.
+
+9. Reward Clipping ([common/vec_env/vec_normalize.py#L32](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L32)) Environment Preprocessing
+ * Followed by the scaling of reward, the _scaled reward_ is further clipped by `VecNormalize` to a range, usually [−10, 10].
+ * A similar approach can be found in [(Mnih et al., 2015)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Mnih2015). There is currently no clear evidence that Reward Clipping after Reward Scaling can help with learning.
+
+We make [~25 lines of code](https://www.diffchecker.com/lsy3qa5e) change to `ppo.py` to incorporate these 9 details, resulting in a self-contained `ppo_continuous_action.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_continuous_action.py)) that has 331 lines of code. The following shows the file difference between the `ppo.py` (left) and `ppo_continuous_action.py` (right).
+
+To run the experiments, we match the hyperparameters used in the original implementation as follows.
+
+```
+# https://github.com/openai/baselines/blob/master/baselines/ppo2/defaults.py
+def mujoco():
+ return dict(
+ nsteps=2048,
+ nminibatches=32,
+ lam=0.95,
+ gamma=0.99,
+ noptepochs=10,
+ log_interval=1,
+ ent_coef=0.0,
+ lr=lambda f: 3e-4 * f,
+ cliprange=0.2,
+ value_network='copy'
+ )
+```
+
+Note that `value_network='copy'` means to use the separate MLP networks for policy and value functions (i.e., the 4th implementation detail in this section). Also, the number of environments parameter $N$ (i.e., `num_envs`) is set to 1 ([common/cmd_util.py#L167](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L167)). Below are the benchmarked results.
+
+
+
+> Tracked MuJoCo experiments (click to show the interactive panel)
+
+## 5 LSTM implementation details
+
+Next, we introduce the 5 details for implementing LSTM.
+
+1. Layer initialization for LSTM layers ([a2c/utils.py#L84-L86](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L84-L86)) Neural Network
+ * The LSTM’s layers’ weights are initialized with `std=1` and biases initialized with `0`.
+
+2. Initialize the LSTM states to be zeros ([common/models.py#L179](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L179)) Neural Network
+ * The hidden and cell states of LSTM are initialized with zeros.
+
+3. Reset LSTM states at the end of the episode ([common/models.py#L141](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L141)) Theory
+ * During rollouts or training, an end-of-episode flag is passed to the agent so that it can reset The LSTM states to zeros.
+
+4. Prepare sequential rollouts in mini-batches ([a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81)) Theory
+ * Under the non-LSTM setting, the mini-batches fetch randomly-indexed training data because the ordering of the training data doesn’t matter. However, the ordering of the training data does matter in the LSTM setting. As a result, the mini-batches fetch the sequential training data from sub-environments.
+
+5. Reconstruct LSTM states during training ([a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81)) Theory
+ * The algorithm saves a copy of the LSTM states `initial_lstm_state` before rollouts. During training, the agent then sequentially reconstruct the LSTM states based on the `initial_lstm_state`. This process ensures that we reconstructed the probability distributions used in rollouts.
+
+We make [~60 lines of code](https://www.diffchecker.com/RelaUQdN) change to `ppo_atari.py` to incorporate these 5 details, resulting in a self-contained `ppo_atari_lstm.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_atari_lstm.py)) that has 385 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_atari_lstm.py` (right).
+
+To run the experiments, we use the Atari hyperparameters again and remove the frame stack (i.e., setting the number of frames stacked to 1). Below are the benchmarked results.
+
+
+
+> Tracked Atari LSTM experiments (click to show the interactive panel)
+
+## 1 `MultiDiscrete` action space detail
+
+The `MultiDiscrete` space is often useful to describe action space for more complicated games. The Gym’s official documentation explains `MultiDiscrete` action space as follows:
+
+```
+# https://github.com/openai/gym/blob/2af816241e4d7f41a000f6144f22e12c8231a112/gym/spaces/multi_discrete.py#L8-L25
+class MultiDiscrete(Space):
+ """
+ - The multi-discrete action space consists of a series of discrete action spaces with different number of actions in each
+ - It is useful to represent game controllers or keyboards where each key can be represented as a discrete action space
+ - It is parametrized by passing an array of positive integers specifying number of actions for each discrete action space
+ Note: Some environment wrappers assume a value of 0 always represents the NOOP action.
+ e.g. Nintendo Game Controller
+ - Can be conceptualized as 3 discrete action spaces:
+ 1) Arrow Keys: Discrete 5 - NOOP[0], UP[1], RIGHT[2], DOWN[3], LEFT[4] - params: min: 0, max: 4
+ 2) Button A: Discrete 2 - NOOP[0], Pressed[1] - params: min: 0, max: 1
+ 3) Button B: Discrete 2 - NOOP[0], Pressed[1] - params: min: 0, max: 1
+ - Can be initialized as
+ MultiDiscrete([ 5, 2, 2 ])
+ """
+ ...
+```
+
+Next, we introduce 1 detail for handling `MultiDiscrete` action space:
+
+1. Independent action components ([common/distributions.py#L215-L220](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L215-L220)Theory
+ * In `MultiDiscrete` action spaces, the actions are represented with multiple discrete values. For example, the action of $a_{t} = \left[\right. a_{t}^{1} , a_{t}^{2} \left]\right. = \left[\right. 0 , 1 \left]\right.$ might mean to press the up arrow key and press button A. To account for this difference, PPO treats $\left[\right. a_{t}^{1} , a_{t}^{2} \left]\right.$ as probabilistically independent action components, therefore calculating $p r o b \left(\right. a_{t} \left.\right) = p r o b \left(\right. a_{t}^{1} \left.\right) \cdot p r o b \left(\right. a_{t}^{2} \left.\right)$.
+ * AlphaStar ([Vinyals et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Vinyals2019)) and OpenAI Five ([Berner et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Berner2019)) adopts the `MultiDiscrete` action spaces. For example, OpenAI Five’s action space is essentially `MultiDiscrete([ 30, 4, 189, 81 ])`, as shown by the following quote:
+> All together this produces a combined factorized action space size of up to 30 × 4 × 189 × 81 = 1, 837, 080 dimensions
+
+We make [~36 lines of code](https://www.diffchecker.com/8fsnhwUI) change to `ppo_atari.py` to incorporate this 1 detail, resulting in a self-contained `ppo_multidiscrete.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_multidiscrete.py)) that has 335 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_multidiscrete.py` (right).
+
+To run the experiments, we use the Atari hyperparameters again and use Gym-μRTS ([Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021)) as the simulation environment.
+
+```
+def gym_microrts():
+ return dict(
+ nsteps=128, nminibatches=4,
+ lam=0.95, gamma=0.99, noptepochs=4, log_interval=1,
+ ent_coef=.01,
+ lr=lambda f : f * 2.5e-4,
+ cliprange=0.1,
+ )
+```
+
+Below are the benchmarked results.
+
+
+
+> Tracked Gym-MicroRTS experiments (click to show the interactive panel)
+
+## 4 Auxiliary implementation details
+
+Next, we introduce 4 auxiliary techniques that are not used (by default) in the official PPO implementations but are potentially useful in special situations.
+
+1. Clip Range Annealing ([ppo2/ppo2.py#L137](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L137)) Code-level Optimizations
+ * The clip coefficient of PPO can be annealed similar to how the learning rate is annealed. However, the clip range annealing is actually used by default.
+
+2. Parallellized Gradient Update ([ppo2/model.py#L131](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L131)) Code-level Optimizations
+ * The policy gradient is calculated in parallel using multiple processes, mainly used in `ppo1` and not used by default in `ppo2`. Such as paradigm could improve training time by making use of all the available processes.
+
+3. Early Stopping of the policy optimizations ([ppo/ppo.py#L269-L271](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/pytorch/ppo/ppo.py#L269-L271)) Code-level Optimizations
+ * This is not actually an implementation detail of _openai/baselines_, but rather an implementation detail in John Schulman’s [modular_rl](https://github.com/joschu/modular_rl/blob/5481b117aa30d3eb8e9ad79abce06378d60dcd45/modular_rl/ppo.py#L48) and _openai/spinningup_ ([TF 1.x](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/tf1/ppo/ppo.py#L234), [Pytorch](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/pytorch/ppo/ppo.py#L269-L271)). It can be considered as an additional mechanism to explicitly enforce the trust-region constraint, on top of the fixed hyperparameter `noptepochs` proposed in the original implementation by [Schulman et al. (2017)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Schulman2017).
+ * More specifically, it starts by tracking an approximate average KL divergence between the policy before and after one update step to its network weights. In case said KL divergence exceeds a preset threshold, the updates to the policy weights are preemptively stopped. [Dossa et al.](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Dossa2021) suggest that early stopping can serve as an alternative method to tune the number of update epochs. We also included this early stopping method in our implementation [(via `--target-kl 0.01`)](https://github.com/vwxyzjn/ppo-implementation-details/blob/eb40cbe172309dcda24a8e93a32269d819e5513d/ppo.py#L71), but toggled it off by default.
+ * Note, however, that while _openai/spinningup_ only early stops the updates to the policy, our implementation early stops both the policy and the value network updates.
+
+4. Invalid Action Masking ([Vinyals et al., 2017](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Vinyals2017); [Huang and Ontañón, 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#HuangOntanon2020)) Theory
+ * Invalid action masking is a technique employed most prominently in AlphaStar ([Vinyals et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Vinyals2019)) and OpenAI Five ([Berner et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Berner2019)) to avoid executing invalid actions in a given game state when the agents are being trained using policy gradient algorithms. Specifically, invalid action masking is implemented by replacing the logits corresponding to the invalid actions with negative infinity before passing the logits to softmax. [Huang and Ontañón, 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#HuangOntanon2020) show such a paradigm **actually makes the gradients corresponding to invalid actions zeros**. Furthermore, [Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021) demonstrated invalid action masking to be the critical technique in training agents to win against all past μRTS bots they tested.
+
+Notably, we highlight the effect of invalid action masking. We make [~30 lines of code](https://www.diffchecker.com/wBUb6Zne) change to `ppo_multidiscrete.py` to incorporate invalid action masking, resulting in a self-contained `ppo_multidiscrete_mask.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_multidiscrete_mask.py)) that has 363 lines of code. The following shows the file difference between the `ppo_multidiscrete.py` (left) and `ppo_multidiscrete_mask.py` (right).
+
+To run the experiments, we use the Atari hyperparameters again and use an older version of Gym-μRTS ([Huang et al, 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Huang2021)) as the simulation environment. Below are the benchmarked results.
+
+
+
+> Tracked Gym-MicroRTS + Action Mask experiments (click to show the interactive panel)
+
+## Results
+
+As shown under each section, our implementations match the results of the original implementation closely. This close matching also extends to other metrics such as policy and value losses. We have made an interactive HTML below for interested viewers to compare other metrics:
+
+## Recommendations
+
+During our reproduction, we have found a number of useful debugging techniques. They are as follows:
+
+1. **Seed everything**: One debugging approach is to seed everything and then observe when things start to differ from the reference implementation. So you could use the same seed for your implementation and mine, check if the observation returned by the environment is the same, then check if the sample the actions are the same. By following the steps, you would check everything to make sure they are aligned (e.g. print out `values.sum()` see if yours match the reference implementation). In the past, we have done this with the [pytorch-a2c-ppo-acktr-gail](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail) repository and ultimately figured out a bug with our implementation.
+2. **Check if `ratio=1`**: Check if the `ratio` are always 1s during the first epoch and first mini-batch update, when new and old policies are the same and therefore the `ratio` are 1s and has nothing to clip. If `ratio` are not 1s, it means there is a bug and the program has not reconstructed the probability distributions used in rollouts.
+3. **Check Kullback-Leibler (KL) divergence**: It is often useful to check if KL divergence goes too high. We have generally found the `approx_kl` stays below 0.02, and if `approx_kl` becomes too high it usually means the policy is changing too quickly and there is a bug.
+4. **Check other metrics**: As shown in the Results section, the other metrics such as policy and value losses in our implementation also closely match those in the original implementation. So if your policy loss’ curve looks very different than the reference implementation, there might be a bug.
+5. **Rule of thumb: 400 episodic return in breakout**: Check if your PPO could obtain 400 episodic return in breakout. We have found this to be a practical rule of thumb to determine the fidelity of online PPO implementations in GitHub. Often we found PPO repositories not able to do this, and we know they probably do not match all implementation details of `openai/baselines`’ PPO.
+
+If you are doing research using PPO, consider adopting the following recommendations to help improve the reproducibility of your work:
+
+1. **Enumerate implementation details used**: If you have implemented PPO as the baseline for your experiment, you should specify which implementation details you are using. Consider using bullet points to enumerate them like done in this blog post.
+2. **Release locked source code**: Always open source your code whenever possible and make sure the code runs. We suggest adopting proper dependency managers such as [poetry](https://python-poetry.org/) or [pipenv](https://pipenv.pypa.io/en/latest/) to lock your dependencies. In the past, we have encountered numerous projects that are based on `pip install -e .`, which 80% of the time would fail to run due to some obscure errors. Having a pre-built `docker` image with all dependencies installed can also help in case the dependencies packages are not hosted by package managers after deprecation.
+3. **Track experiments**: Consider using an experiment management software to track your metrics, hyperparameters, code, and others. They can boost your productivity by saving hundreds of hours spent on `matplotlib` and worrying about how to display data. Commercial solutions (usually more mature) include [Weights and Biases](https://wandb.ai/) and [Neptune](https://neptune.ai/), and open-source solutions include [Aim](https://github.com/aimhubio/aim), [ClearML](https://github.com/allegroai/clearml), [Polyaxon](https://github.com/polyaxon/polyaxon).
+4. **Adopt single-file implementation**: If your research requires more tweaking, consider implementing your algorithms using single-file implementations. This blog does this and creates standalone files for different environments. For example, our `ppo_atari.py` contains all relevant code to handle Atari games. Such a paradigm has the following benefits at the cost of duplicate and harder-to-refactor code:
+ * _Easier to see the whole picture_: Because each file is self-contained, people can easily spot all relevant implementation details of the algorithm. Such a paradigm also reduces the burden to understand how files like `env.py`, `agent.py`, `network.py` work together like in typical RL libraries.
+ * _Faster developing experience_: Usually, each file like `ppo.py` has significantly less LOC compared to RL libraries’ PPO. As a result, it’s often easier to prototype new features without having to do subclassing and refactoring.
+ * _Painless performance attribution_: If a new version of our algorithm has obtained higher performance, we know this single file is exactly responsible for the performance improvement. To attribute the performance improvement, we can simply do a `filediff` between the current and past versions, and every line of code change is made explicit to us.
+
+## Discussions
+
+## Does modularity help RL libraries?
+
+This blog post demonstrates reproducing PPO is a non-trivial effort, even though PPO’s source code is readily available for reference. Why is it the case? We think one important reason might be that **modularity disperses implementation details**.
+
+Almost all RL libraries have adopted modular design, featuring different modules / files like `env.py`, `agent.py`, `network.py`, `utils.py`, `runner.py`, etc. The nature of modularity necessarily puts implementation details into different files, which is usually great from a software engineering perspective. That is, we don’t have to know how other components work when we just work on `env.py`. Being able to treat other components as black boxes has empowered us to work on large and complicated systems for the last decades.
+
+However, this practice might clash hard with ML / RL: as the library grows, it becomes harder and harder to grasp all implementation details w.r.t an algorithm, whereas recognizing all implementation details has become increasingly important, as indicated by this blog post, [Engstrom, Ilyas, et al., 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Engstrom), and [Andrychowicz, et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Andrychowicz). So what can we do?
+
+Modular design still offers numerous benefits such as 1) easy-to-use interface, 2) integrated test cases, 3) easy to plug different components and others. To this end, good RL libraries are valuable, and we recommend them to write good documentation and refactor libraries to adopt new features. For algorithmic researchers, however, we recommend considering single-file implementations because they are straightforward to read and extend.
+
+## Is asynchronous PPO better?
+
+Not necessarily. The high-throughput variant Asynchronous PPO (APPO) ([Berner et al., 2019](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Berner2019)) has obtained more attention in recent years. APPO eliminates the idle time in the original PPO implementation (e.g., have to wait for all $N$ environments to return observations), resulting in much higher throughput, GPU and CPU utilization. However, APPO involves performance-reducing side-effects, namely stale experiences ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)), and we have found insufficient evidence to ascertain its improvement. The biggest issue is:
+
+**Underbenchmarked APPO implementation**: RLlib has an [APPO implementation](https://docs.ray.io/en/latest/rllib-algorithms.html#appo), yet its documentation contains no benchmark information and suggest “APPO is not always more efficient; it is often better to use standard PPO or IMPALA.” Sample Factory ([Petrenko et al, 2020](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Petrenko)) presents more benchmark results, but its support for Atari games is still a [work in progress](https://github.com/alex-petrenko/sample-factory/issues/51). To our knowledge, there is no APPO implementation that simultaneously works with Atari games, MuJoCo or Pybullet tasks, MultiDiscrete action spaces and with an LSTM.
+
+While APPO is intuitively valuable for CPU-intensive tasks such as Dota 2, this blog post recommends an alternative approach to speed up PPO: **make the vectorized environments really fast**. Initially, the vectorized environments are implemented in python, which is slow. More recently, researchers have proposed to use accelerated vectorized environments. For example,
+
+1. Procgen [(Cobbe et al, 2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe) uses C++ to implement native vectorized environments, resulting in much higher throughput when setting $N = 64$ ($N$ is the number of environments),
+2. [Envpool](https://github.com/sail-sg/envpool) uses C++ to offer native vectorized environments for Atari and classic control games,
+3. Nvidia’s [Isaac Gym](https://developer.nvidia.com/isaac-gym)[(Makoviychuk et al., 2021)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Makoviychuk) uses `torch` to write hardware-accelerated vectorized environments, allowing the users to spin up $N = 4096$ environments easily,
+4. Google’s [Brax](https://github.com/google/brax) uses jax to write hardware-accelerated vectorized environments, allowing the users to spin up $N = 2048$ environments easily and solve robotics tasks like `Ant` in minutes compared to hours of training in MuJoCo.
+
+In the following section, we demonstrate accelerated training with PPO + envpool in the Atari game Pong.
+
+### Solving Pong in 5 minutes with PPO + Envpool
+
+[Envpool](https://github.com/sail-sg/envpool) is a recent work that offers accelerated vectorized environments for Atari by leveraging C++ and thread pools. Our PPO gets a free and side-effects-free performance boost by simply adopting it. We make [~60 lines of code](https://www.diffchecker.com/RafLuYD6) change to `ppo_atari.py` to incorporate this 1 detail, resulting in a self-contained `ppo_atari_envpool.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_atari_envpool.py)) that has 365 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_atari_envpool.py` (right).
+
+As shown below, Envpool + PPO runs 3x faster without side effects (as in no loss of sample efficiency):
+
+
+
+
+
+> Tracked Atari + Envpool experiments (click to show the interactive panel)
+
+Two quick notes: 1) the performance deterioration in BeamRider is largely due to a degenerate random seed, and 2) Envpool uses the v5 ALE environments but has processed them the same way as the v4 ALE environments used in our previous experiments. Furthermore, by tuning the hyperparameters, we obtained a run that solves Pong in 5 mins. This performance is even comparable to IMPALA’s ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)) results:
+
+We think this raises a practical consideration: adopting async RL such as IMPALA could be more difficult than just making your vectorized environments fast.
+
+## Request for Research
+
+Given this blog post, we believe the community understands PPO better and would be in a much better place to make improvements. Here are a few suggested areas for research.
+
+1. **Alternative choices**: As we have walked through the different details of PPO, it seems that some of them result from arbitrary choices. It would be interesting to investigate alternative choices and see how such change affects results. You can find below a non-exhaustive list of tracks to explore:
+ * use of a different Atari pre-processing (as partially explored by [Machado et al., 2018)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Machado2018))
+ * use of a different distribution for continuous actions ([Beta distribution](http://proceedings.mlr.press/v70/chou17a/chou17a.pdf), squashed Gaussian, Gaussian with full covariance, …), it will most probably require some tuning
+ * use of a state-dependent standard deviation when using continuous actions (with or without backpropagation of the gradient to the whole actor network)
+ * use of a different initialization for LSTM (ones instead of zeros, random noise, learnable parameter, …), use of GRU cells instead of LSTM
+
+2. **Vectorized architecture for experience-replay-based methods**: Experience-replay-based methods such as DQN, DDPG, and SAC are less popular than PPO due to a few reasons: 1) they generally have lower throughput due to a single simulation environment (also means lower GPU utilization), and 2) they usually have higher memory requirement (e.g., DQN requires the notorious 1M sample replay buffer which could take 32GB memory). Can we apply the vectorized architecture to experience-replay-based methods? The vectorized environments intuitively should replace replay buffer because the environments could also provide uncorrelated experience.
+3. **Value function optimization**: In Phasic Policy Gradient ([Cobbe et al., 2021](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe2021)), optimizing value functions separately turns out to be important. In DQN, the prioritized experience replay significantly boosts performance. Can we apply prioritized experience replay to PPO or just on PPO’s value function?
+
+## Conclusion
+
+Reproducing PPO’s results has been difficult in the past few years. While recent works conducted ablation studies to provide insight on the implementation details, these works are not structured as tutorials and only focus on details concerning robotics tasks. As a result, reproducing PPO from scratch can become a daunting experience. Instead of introducing additional improvements or doing further ablation studies, this blog post takes a step back and focuses on delivering a thorough reproduction of PPO in all accounts, as well as aggregating, documenting, and cataloging its most salient implementation details. This blog post also points out software engineering challenges in PPO and further efficiency improvement via the accelerated vectorized environments. With these, we believe this blog post will help people understand PPO faster and better, facilitating customization and research upon this versatile RL algorithm.
+
+## Acknowledgment
+
+We thank [Weights and Biases](https://wandb.ai/) for providing a free academic license that helps us track the experiments. Shengyi would like to personally thank Angelica Pan, Scott Condron, Ivan Goncharov, Morgan McGuire, Jeremy Salwen, Cayla Sharp, Lavanya Shukla, and Aakarshan Chauhan for supporting him in making the video tutorials.
+
+### Bibliography
+
+[Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 2017 Jul 20.](http://arxiv.org/abs/1707.06347)
+
+[Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.](http://arxiv.org/abs/1707.06347)
+
+[Engstrom L, Ilyas A, Santurkar S, Tsipras D, Janoos F, Rudolph L, Madry A. Implementation matters in deep policy gradients: A case study on ppo and trpo. International Conference on Learning Representations, 2020](https://openreview.net/forum?id=r1etN1rtPB)
+
+[Andrychowicz M, Raichuk A, Stańczyk P, Orsini M, Girgin S, Marinier R, Hussenot L, Geist M, Pietquin O, Michalski M, Gelly S. What matters in on-policy reinforcement learning? a large-scale empirical study. International Conference on Learning Representations, 2021](https://openreview.net/forum?id=nIAxjsniDzg)
+
+[Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S. Human-level control through deep reinforcement learning. nature. 2015 Feb;518(7540):529-33.](https://www.nature.com/articles/nature14236)
+
+[Machado MC, Bellemare MG, Talvitie E, Veness J, Hausknecht M, Bowling M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research. 2018 Mar 19;61:523-62.](https://arxiv.org/abs/1709.06009)
+
+[Schulman J, Levine S, Abbeel P, Jordan M, Moritz P. Trust region policy optimization. In International conference on machine learning 2015 Jun 1 (pp. 1889-1897). PMLR.](http://proceedings.mlr.press/v37/schulman15)
+
+[Duan Y, Chen X, Houthooft R, Schulman J, Abbeel P. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning 2016 Jun 11 (pp. 1329-1338). PMLR.](http://proceedings.mlr.press/v48/duan16.html)
+
+[Haarnoja T, Zhou A, Abbeel P, Levine S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning 2018 Jul 3 (pp. 1861-1870). PMLR.](http://proceedings.mlr.press/v80/haarnoja18b)
+
+[Chou PW. The beta policy for continuous control reinforcement learning (Doctoral dissertation, Master’s thesis. Pittsburgh: Carnegie Mellon University). 2017.](https://www.ri.cmu.edu/wp-content/uploads/2017/06/thesis-Chou.pdf)
+
+[Fujita Y, Maeda SI. Clipped action policy gradient. In International Conference on Machine Learning 2018 Jul 3 (pp. 1597-1606). PMLR.](http://proceedings.mlr.press/v80/fujita18a.html)
+
+[Bellemare M, Srinivasan S, Ostrovski G, Schaul T, Saxton D, Munos R. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems. 2016;29:1471-9.](https://proceedings.neurips.cc/paper/2016/file/afda332245e2af431fb7b672a68b659d-Paper.pdf)
+
+[Tavakoli A, Pardo F, Kormushev P. Action branching architectures for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence 2018 Apr 29 (Vol. 32, No. 1).](https://ojs.aaai.org/index.php/AAAI/article/view/11798)
+
+[Metz L, Ibarz J, Jaitly N, Davidson J. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:1705.05035. 2017 May 14.](https://arxiv.org/abs/1705.05035)
+
+[Zhang Y, Vuong QH, Song K, Gong XY, Ross KW. Efficient entropy for policy gradient with multidimensional action space. arXiv preprint arXiv:1806.00589. 2018 Jun 2.](https://arxiv.org/abs/1806.00589)
+
+[Huang S, Ontañón S. A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006.14171. 2020 Jun 25.](https://arxiv.org/abs/2006.14171)
+
+[Huang, S., Ontan’on, S., Bamford, C., & Grela, L. Gym-μRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning. In Proceedings of the 2021 IEEE Conference on Games (CoG).](https://ieeexplore.ieee.org/document/9619076)
+
+[Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P, Oh J. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature. 2019 Nov;575(7782):350-4.](https://doi.org/10.1038/s41586-019-1724-z)
+
+[Berner C, Brockman G, Chan B, Cheung V, Dębiak P, Dennison C, Farhi D, Fischer Q, Hashme S, Hesse C, Józefowicz R. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680. 2019 Dec 13.](https://arxiv.org/abs/1912.06680)
+
+[Vinyals O, Ewalds T, Bartunov S, Georgiev P, Vezhnevets AS, Yeo M, Makhzani A, Küttler H, Agapiou J, Schrittwieser J, Quan J. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782. 2017 Aug 16.](https://arxiv.org/abs/1708.04782)
+
+[Dossa RF, Huang S, Ontañón S, Matsubara T. An Empirical Investigation of Early Stopping Optimizations in Proximal Policy Optimization. IEEE Access. 2021 Aug 23;9:117981-92.](https://ieeexplore.ieee.org/document/9520424)
+
+[Espeholt L, Soyer H, Munos R, Simonyan K, Mnih V, Ward T, Doron Y, Firoiu V, Harley T, Dunning I, Legg S. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. InInternational Conference on Machine Learning 2018 Jul 3 (pp. 1407-1416). PMLR.](https://arxiv.org/abs/1802.01561)
+
+[Petrenko A, Huang Z, Kumar T, Sukhatme G, Koltun V. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. InInternational Conference on Machine Learning 2020 Nov 21 (pp. 7652-7662). PMLR.](https://arxiv.org/abs/2006.11751)
+
+[Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., & State, G. (2021). Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. ArXiv, abs/2108.10470.](https://arxiv.org/abs/2108.10470)
+
+[Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2020, November). Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning (pp. 2048-2056). PMLR.](https://arxiv.org/abs/1912.01588)
+
+[Terry, J.K., Black, B., Hari, A., Santos, L., Dieffendahl, C., Williams, N.L., Lokesh, Y., Horsch, C., & Ravi, P. (2020). Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34..](https://arxiv.org/pdf/2009.14471.pdf)
+
+## Appendix
+
+In this appendix, we introduce one detail for reproducing PPO’s results in the procgen environments [(Cobbe et al, 2020)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#Cobbe).
+
+1. IMPALA-style Neural Network ([common/models.py#L28](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L28)) Neural Network
+ * In the [openai/train-procgen](https://github.com/openai/train-procgen) repository, the authors by default uses the IMPALA-style Neural Network ([train_procgen/train.py#L52](https://github.com/openai/train-procgen/blob/1a2ae2194a61f76a733a39339530401c024c3ad8/train_procgen/train.py#L52), also see see ([Espeholt et al., 2018](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/#IMPALA)) without the LSTM layers.
+
+We make [~60 lines of code](https://www.diffchecker.com/82aRqGuz) change to `ppo_atari.py` to incorporate these 5 details, resulting in a self-contained `ppo_procgen.py` ([link](https://github.com/vwxyzjn/ppo-implementation-details/blob/main/ppo_procgen.py)) that has 354 lines of code. The following shows the file difference between the `ppo_atari.py` (left) and `ppo_procgen.py` (right).
+
+To run the experiment, we try to match the default setting in [openai/train-procgen](https://github.com/openai/train-procgen) except that we use the `easy` distribution mode and `total_timesteps=25e6` to save compute.
+
+```
+def procgen():
+ return dict(
+ nsteps=256, nminibatches=8,
+ lam=0.95, gamma=0.999, noptepochs=3, log_interval=1,
+ ent_coef=.01,
+ lr=5e-4,
+ cliprange=0.2,
+ vf_coef=0.5, max_grad_norm=0.5,
+ )
+network = build_impala_cnn(x, depths=[16,32,32], emb_size=256)
+env = ProcgenEnv(
+ num_envs=64,
+ env_name="starpilot",
+ num_levels=0,
+ start_level=0,
+ distribution_mode="easy"
+)
+env = VecNormalize(venv=env, ob=False)
+ppo2.learn(..., total_timesteps = 25_000_000)
+```
+
+Notice that
+
+1. Learning rate annealing is turned off by default.
+2. Reward scaling and reward clipping is used.
+
+Below are the benchmarked results.
+
+
+
+> Tracked Procgen experiments (click to show the interactive panel)
+
+You will need to sign in to GitHub to add a comment! To edit or delete your comment, visit the [discussions page](https://github.com/iclr-blog-track/iclr-blog-track.github.io/discussions) and look for your comment in the right discussion.
diff --git a/docs/evidence/gwern_tank.md b/docs/evidence/gwern_tank.md
index 602d846..63d6ce3 100644
--- a/docs/evidence/gwern_tank.md
+++ b/docs/evidence/gwern_tank.md
@@ -1,9 +1,607 @@
# The Neural Net Tank Legend — Gwern Branwen
-Source: https://gwern.net/tank . Cached excerpt for the ML-debugging skill (verbatim abstract passages).
+Source: https://gwern.net/tank (page title: "The Neural Net Tank Urban Legend")
+Fetched-via: r.jina.ai reader, 2026-08-15 (CLAUDE agent)
+Fetch-status: full article text, with the site's backlinks / similar-links / bibliography nav sections trimmed. Supersedes the earlier abstract-only excerpt. (CLAUDE agent)
+
+Why it matters here: the canonical worked example of a model learning the confound in how data was collected, and gwern's demonstration that the story itself is an unsourced legend.
---
-> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day. This story is often told to warn about the limits of algorithms and importance of data collection to avoid "dataset bias"/"data leakage" where the collected data can be solved using algorithms that do not generalize to the true data distribution, but the tank story is usually never sourced.
+AI folklore tells a story about a neural network trained to detect tanks which instead learned to detect time of day; investigating, this probably never happened.
-> I collate many extent versions dating back a quarter of a century to 1992 along with two NN-related anecdotes from the 1960s; their contradictions & details indicate a classic "urban legend", with a probable origin in a speculative question in the 1960s by Edward Fredkin at an AI conference about some early NN research, which was then classified & never followed up on.
+> A cautionary tale in artificial intelligence tells about researchers training an neural network (NN) to detect tanks in photographs, succeeding, only to realize the photographs had been collected under specific conditions for tanks/non-tanks and the NN had learned something useless like time of day. This story is often told to warn about the limits of algorithms and importance of data collection to avoid “dataset bias”/“data leakage” where the collected data can be solved using algorithms that do not generalize to the true data distribution, but the tank story is usually never sourced.
+>
+>
+> I collate many extent versions dating back a quarter of a century to 1992 34ya along with two NN-related anecdotes from the 1960s; their contradictions & details indicate a classic “urban legend”, with a probable origin in a speculative question in the 1960s by Edward Fredkin at an AI conference about some early NN research, which was then classified & never followed up on.
+>
+>
+> I suggest that dataset bias is real but exaggerated by the tank story, giving a misleading indication of risks from deep learning and that it would be better to not repeat it but use real examples of dataset bias and focus on larger-scale risks like AI systems optimizing for wrong utility functions.
+
+[D](https://gwern.net/dropcap#kanzlei)eep learning’s rise over the past decade and dominance in image processing tasks has led to an explosion of applications attempting to infer high-level semantics locked up in raw sensory data like photographs. Convolutional neural networks are now applied to not just ordinary tasks like [sorting cucumbers by quality](https://cloud.google.com/blog/products/gcp/how-a-japanese-cucumber-farmer-is-using-deep-learning-and-tensorflow) but everything from predicting the best Go move to [where in the world](https://arxiv.org/abs/1602.05314#deepmind) it was taken to whether a photograph is [“interesting”](https://research.google/blog/automatic-photography-with-google-clips/ "Automatic Photography with Google Clips") or [“pretty”](https://research.google/blog/using-deep-learning-to-create-professional-level-photographs/ "Using Deep Learning to Create Professional-Level Photographs"), not to mention supercharging traditional tasks like radiology interpretation or facial recognition which have reached levels of accuracy that could only be dreamed of decades ago. With this approach of “neural net _all the things_!”, the question of to what extent the trained neural networks are useful in the real world and will do what we _want_ it to do & not what we _told_ it to do has taken on additional importance, especially given the possibility of neural networks learning to accomplish extremely inconvenient things like inferring individual human differences such as criminality or homosexuality (to give two highly controversial recent examples where the meaningfulness of claimed success have been severely questioned).
+
+In this context, a cautionary story is often told of incautious researchers decades ago who trained a NN for the military to find images of tanks, only to discover they had trained a neural network to detect something else entirely (what, precisely, that something else was varies in the telling). It would be a good & instructive story… if it were true. Is it?
+
+As it would be so useful a cautionary example for AI safety/alignment research, and was cited to that effect by Eliezer Yudkowsky but only to a secondary source, I decided to make myself useful by finding a proper primary source for it & see if there were more juicy details worth mentioning. My initial attempt failed, and I & several others failed for over more than half a decade to find any primary source (just secondary sources citing each other). I began to wonder if it was even real.
+
+Trying again more seriously, I conclude that, unfortunately, it is definitely not real as usually told: it is just an urban legend/leprechaun; and in fact, the seed of the story _could not_ have run into the issue the tank story warns about, because they correctly constructed their training dataset to avoid such issues. More broadly, considering that issue in contemporary deep learning, the issue it cautions against is real but not that important and conflated with more dangerous safety/alignment problems.
+
+## [Did It Happen?](https://gwern.net/tank#did-it-happen "Link to section: § 'Did It Happen?'")
+
+## [Versions of the Story](https://gwern.net/tank#versions-of-the-story "Link to section: § 'Versions of the Story'")
+
+Drawing on [the usual suspects](https://gwern.net/search) (Google/Google Books/Google Scholar/Libgen/LessWrong/Hacker News/Twitter) in [investigating leprechauns](https://gwern.net/leprechaun), I have compiled a large number of variants of the story; below, in reverse chronological order by decade, letting us trace the evolution of the story back towards its roots:
+
+### [2010s](https://gwern.net/tank#s "Link to section: § '2010s'")
+
+Heather Murphy, [“Why Stanford Researchers Tried to Create a ‘Gaydar’ Machine”](https://www.nytimes.com/2017/10/09/science/stanford-sexual-orientation-study.html "Why Stanford Researchers Tried to Create a ‘Gaydar’ Machine") (NYT), 2017-10-09:
+
+> _So What Did the Machines See?_ Dr.Kosinski and Mr.Wang [[Wang & Kosinski 2018](https://files.osf.io/v1/resources/hv28a/providers/osfstorage/59ab119b594d9002537d360c?action=download&version=10&direct#pdf); see also [Leuner 2019](https://gwern.net/tank#leuner-2019)/[Kosinski 2021](https://www.nature.com/articles/s41598-020-79310-1)] say that the algorithm is responding to fixed facial features, like nose shape, along with “grooming choices,” such as eye makeup. But it’s also possible that the algorithm is seeing something totally unknown. “The more data it has, the better it is at picking up patterns,” said Sarah Jamie Lewis, an independent privacy researcher who Tweeted a critique of the study. “But the patterns aren’t necessarily the ones you think that they are.” [Tomaso Poggio](https://en.wikipedia.org/wiki/Tomaso_Poggio), the director of M.I.T.’s Center for Brains, Minds and Machines, offered a classic parable used to illustrate this disconnect. The Army trained a program to differentiate American tanks from Russian tanks with 100% accuracy. Only later did analysts realized that the American tanks had been photographed on a sunny day and the Russian tanks had been photographed on a cloudy day. The computer had learned to detect brightness. Dr.Cox has spotted a version of this in his own studies of dating profiles. Gay people, he has found, tend to post higher-quality photos. Dr.Kosinski said that they went to great lengths to guarantee that such confounders did not influence their results. Still, he agreed that it’s easier to teach a machine to see than to understand what it has seen.
+
+[It is worth noting that [Arcs et al’s criticisms](https://medium.com/@blaisea/do-algorithms-reveal-sexual-orientation-or-just-expose-our-stereotypes-d998fafdf477 "Do algorithms reveal sexual orientation or just expose our stereotypes?"), such as their ‘gay version’ photographs, do not appear to have been confirmed by an [independent replication](https://arxiv.org/abs/1902.10739).]
+
+Alexander Harrowell, [“It was called a perceptron for a reason, damn it”](https://www.harrowell.org.uk/blog/2017/09/30/it-was-called-a-perceptron-for-a-reason-damn-it/), 2017-09-30:
+
+> You might think that this is rather like one of the classic optical illusions, but it’s worse than that. If you notice that you look at something this way, and then that way, and it looks different, you’ll notice something is odd. This is not something our deep learner will do. Nor is it able to identify any bias that might exist in the corpus of data it was trained on…or maybe it is. If there is any property of the training data set that is strongly predictive of the training criterion, it will zero in on that property with the ferocious clarity of Darwinism. In the 1980s, an early backpropagating neural network was set to find Soviet tanks in a pile of reconnaissance photographs. It worked, until someone noticed that the Red Army usually trained when the weather was good, and in any case the satellite could only see them when the sky was clear. The medical school at St Thomas’ Hospital in London found theirs had learned that their successful students were usually white.
+
+An interesting story with a distinct “family resemblance” is told about a NN classifying wolves/dogs, by Evgeniy Nikolaychuk, [“Dogs, Wolves, Data Science, and Why Machines Must Learn Like Humans Do”](https://medium.com/veon-careers/dogs-wolves-data-science-and-why-machines-must-learn-like-humans-do-213b08036a10 "Dogs, Wolves, Data Science, and Why Machines Must Learn Like Humans Do"), 2017-06-09:
+
+> Neural networks are designed to learn like the human brain, but we have to be careful. This is not because I’m scared of machines taking over the planet. Rather, we must make sure machines learn correctly. One example that always pops into my head is how one neural network learned to differentiate between dogs and wolves. It didn’t learn the differences between dogs and wolves, but instead learned that wolves were on snow in their picture and dogs were on grass. It learned to differentiate the two animals by looking at snow and grass. Obviously, the network learned incorrectly. What if the dog was on snow and the wolf was on grass? Then, it would be wrong.
+
+However, in his source, [“‘Why Should I Trust You?’ Explaining the Predictions of Any Classifier [LIME]”](https://arxiv.org/abs/1602.04938), Ribeiro et al 2016, they specify of their dog/wolf snow-detector NN that they “trained this _bad_ classifier intentionally, to evaluate whether subjects are able to detect it [the bad performance]” using LIME for insight into how the classifier was making its classification, concluding that “After examining the explanations, however, almost all of the subjects identified the correct insight, with much more certainty that it was a determining factor. Further, the trust in the classifier also dropped substantially.” So Nikolaychuk appears to have misremembered. (Perhaps in another 25 years students will be told in their classes of how a NN was once trained by ecologists to count wolves…)
+
+[Redditor mantrap2](https://www.reddit.com/r/MachineLearning/comments/3ailzi/suddenly_a_leopard_print_sofa_appears/csczkqg/) gives on 2015-06-20 this version of the story:
+
+> I remember this kind of thing from the 1980s: the US Army was testing image recognition seekers for missiles and was getting excellent results on Northern German tests with NATO tanks. Then they tested the same systems in other environment and there results were suddenly shockingly bad. Turns out the image recognition was keying off the trees with tank-like minor features rather than the tank itself. Putting other vehicles in the same forests got similar high hits but tanks by themselves (in desert test ranges) didn’t register. Luckily a sceptic somewhere decided to “do one more test to make sure”.
+
+Dennis Polis, _God, Science and Mind_, 2012 14ya (pg131, limited Google Books snippet, unclear what ref 44 is):
+
+> These facts refute a Neoplatonic argument for the essential immateriality of the soul, _viz._ that since the mind deals with _universal_ representations, it operates in a specifically immaterial way…So, awareness is not explained by connectionism. The results of neural net training are not always as expected. One team intended to train neural nets to recognize battle tanks in aerial photos. The system was trained using photos with and without tanks. After the training, a different set of photos was used for evaluation, and the system failed miserably—being totally incapable of distinguishing those with tanks. The system actually discriminated cloudy from sunny days. It happened that all the training photos with tanks were taken on cloudy days, while those without were on clear days.44 What does this show? That neural net training is mindless. The system had no _idea_ of the intent of the enterprise, and did what it was programmed to do without any concept of its _purpose_. As with Dawkins’ evolution simulation (p.66), the goals of computer neural nets are imposed by human programmers.
+
+Blay Whitby, [_Artificial Intelligence: A Beginner’s Guide_](https://books.google.com/books?id=TKOfhnUhgS4C "Artificial Intelligence: A Beginner's Guide")2012 14ya (pg53):
+
+> It is not yet clear how an artificial neural net could be trained to deal with “the world” or any really open-ended sets of problems. Now some readers may feel that this unpredictability is not a problem. After all, we are talking about training not programming and we expect a neural net to behave rather more like a brain than a computer. Given the usefulness of nets in unsupervised learning, it might seem therefore that we do not really need to worry about the problem being of manageable size and the training process being predictable. This is not the case; we really do need a manageable and well-defined problem for the training process to work. A famous AI urban myth may help to make this clearer.
+>
+>
+> The story goes something like this. A research team was training a neural net to recognize pictures containing tanks. (I’ll leave you to guess why it was tanks and not tea-cups.) To do this they showed it two training sets of photographs. One set of pictures contained at least one tank somewhere in the scene, the other set contained no tanks. The net had to be trained to discriminate between the two sets of photographs. Eventually, after all that back-propagation stuff, it correctly gave the output “tank” when there was a tank in the picture and “no tank” when there wasn’t. Even if, say, only a little bit of the gun was peeping out from behind a sand dune it said “tank”. Then they presented a picture where no part of the tank was visible—it was actually completely hidden behind a sand dune—and the program said “tank”.
+>
+>
+> Now when this sort of thing happens research labs tend to split along age-based lines. The young hairs say “Great! We’re in line for the Nobel Prize!” and the old heads say “Something’s gone wrong”. Unfortunately, the old heads are usually right—as they were in this case. What had happened was that the photographs containing tanks had been taken in the morning while the army played tanks on the range. After lunch the photographer had gone back and taken pictures from the same angles of the empty range. So the net had identified the most reliable single feature which enabled it to classify the two sets of photos, namely the angle of the shadows. “AM = tank, PM = no tank”. This was an extremely effective way of classifying the two sets of photographs in the training set. What it most certainly was _not_ was a program that recognizes tanks. The great advantage of neural nets is that they find their own classification criteria. The great problem is that it may not be the one you want!
+
+[Thom Blake](https://www.lesswrong.com/posts/PoDAyQMWEXBBBEJ5P/magical-categories4v4a) notes in 2011-09-20 that the story is:
+
+> Probably apocryphal. I haven’t been able to track this down, despite having heard the story both in computer ethics class and at academic conferences.
+
+[“Embarrassing mistakes in perceptron research”](https://www.webofstories.com/play/marvin.minsky/122) ([YouTube](https://www.youtube.com/watch?v=3JjDmFV_YwQ)), Marvin Minsky, 2011-01-31:
+
+> Like I had a friend in Italy who had a perceptron that looked at a visual… it had visual inputs. So, he… he had scores of music written by Bach of chorales and he had scores of chorales written by music students at the local conservatory. And he had a perceptron—a big machine—that looked at these and those and tried to distinguish between them. And he was able to train it to distinguish between the masterpieces by Bach and the pretty good chorales by the conservatory students. Well, so, he showed us this data and I was looking through it and what I discovered was that in the lower left hand corner of each page, one of the sets of data had single whole notes. And I think the ones by the students usually had four quarter notes. So that, in fact, it was possible to distinguish between these two classes of… of pieces of music just by looking at the lower left… lower right hand corner of the page. So, I told this to the… to our scientist friend and he went through the data and he said: ‘You guessed right. That’s… that’s how it happened to make that distinction.’ We thought it was very funny.
+>
+>
+> A similar thing happened here in the United States at one of our research institutions. Where a perceptron had been trained to distinguish between—this was for military purposes—It could… it was looking at a scene of a forest in which there were camouflaged tanks in one picture and no camouflaged tanks in the other. And the perceptron—after a little training—got… made a 100% correct distinction between these two different sets of photographs. Then they were embarrassed a few hours later to discover that the two rolls of film had been developed differently. And so these pictures were just a little darker than all of these pictures and the perceptron was just measuring the total amount of light in the scene. But it was very clever of the perceptron to find some way of making the distinction.
+
+### [2000s](https://gwern.net/tank#s-1 "Link to section: § '2000s'")
+
+[Eliezer Yudkowsky](https://www.yudkowsky.net/), [2008-08-24](https://www.lesswrong.com/posts/PoDAyQMWEXBBBEJ5P/magical-categories) (similarly quoted in [“Artificial Intelligence as a Negative and Positive Factor in Global Risk”](https://intelligence.org/files/AIPosNegFactor.pdf), “Artificial Intelligence in global risk” in _Global Catastrophic Risks_ 2011 15ya, & “Friendly Artificial Intelligence” in _Singularity Hypotheses_ 2013 13ya):
+
+> Once upon a time—I’ve seen this story in several versions and several places, sometimes cited as fact, but I’ve never tracked down an original source—once upon a time, I say, the US Army wanted to use neural networks to automatically detect camouflaged enemy tanks. The researchers trained a neural net on 50 photos of camouflaged tanks amid trees, and 50 photos of trees without tanks. Using standard techniques for supervised learning, the researchers trained the neural network to a weighting that correctly loaded the training set—output “yes” for the 50 photos of camouflaged tanks, and output “no” for the 50 photos of forest. Now this did not prove, or even imply, that new examples would be classified correctly. The neural network might have “learned” 100 special cases that wouldn’t generalize to new problems. Not, “camouflaged tanks versus forest”, but just, “photo-1 positive, photo-2 negative, photo-3 negative, photo-4 positive…” But wisely, the researchers had originally taken 200 photos, 100 photos of tanks and 100 photos of trees, and had used only half in the training set. The researchers ran the neural network on the remaining 100 photos, and _without further training_ the neural network classified all remaining photos correctly. Success confirmed! The researchers handed the finished work to the Pentagon, which soon handed it back, complaining that in their own tests the neural network did no better than chance at discriminating photos. It turned out that in the researchers’ data set, photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days. The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest. This parable—which might or might not be fact—illustrates one of the most fundamental problems in the field of supervised learning and in fact the whole field of Artificial Intelligence…
+
+Gordon Rugg, [_Using Statistics: A Gentle Introduction_](https://books.google.com/books?id=S9lsBnV7txoC "Using Statistics: A Gentle Introduction"), 2007-10-01 (pg114–115):
+
+> _Neural nets and genetic algorithms (including the story of the Russian tanks)_: Neural nets (or artificial neural networks, to give them their full name) are pieces of software inspired by the way the human brain works. In brief, you can train a neural net to do tasks like classifying images by giving it lots of examples, and telling it which examples fit into which categories; the neural net works out for itself what the defining characteristics are for each category. Alternatively, you can give it a large set of data and leave it to work out connections by itself, without giving it any feedback. There’s a story, which is probably an urban legend, which illustrates how the approach works and what can go wrong with it. According to the story, some NATO researchers trained a neural net to distinguish between photos of NATO and Warsaw Pact tanks. After a while, the neural net could get it right every time, even with photos it had never seen before. The researchers had gleeful visions of installing neural nets with miniature cameras in missiles, which could then be fired at a battlefield and left to choose their own targets. To demonstrate the method, and secure funding for the next stage, they organised a viewing by the military. On the day, they set up the system and fed it a new batch of photos. The neural net responded with apparently random decisions, sometimes identifying NATO tanks correctly, sometimes identifying them mistakenly as Warsaw Pact tanks. This did not inspire the powers that be, and the whole scheme was abandoned on the spot. It was only afterwards that the researchers realised that all their training photos of NATO tanks had been taken on sunny days in Arizona, whereas the Warsaw Pact tanks had been photographed on grey, miserable winter days on the steppes, so the neural net had flawlessly learned the unintended lesson that if you saw a tank on a gloomy day, then you made its day even gloomier by marking it for destruction.
+
+N. Katherine Hayles, “Computing the Human” (_Inventive Life: Approaches to the New Vitalism_, Fraser et al 2006 20ya; pg424):
+
+> While humans have for millennia used what Cariani calls ‘active sensing’—‘poking, pushing, bending’—to extend their sensory range and for hundreds of years have used prostheses to create new sensory experiences (for example, microscopes and telescopes), only recently has it been possible to construct evolving sensors and what [Cariani (1998 28ya: 718)](https://gwern.net/doc/transhumanism/1998-cariani.pdf) calls ‘internalized sensing’, that is, “bringing the world into the device” by creating internal, analog representations of the world out of which internal sensors extract newly-relevant properties’.
+>
+>
+> …Another conclusion emerges from Cariani’s call (1998 28ya) for research in sensors that can adapt and evolve independently of the epistemic categories of the humans who create them. The well-known and perhaps apocryphal story of the neural net trained to recognize army tanks will illustrate the point. For obvious reasons, the army wanted to develop an intelligent machine that could discriminate between real and pretend tanks. A neural net was constructed and trained using two sets of data, one consisting of photographs showing plywood cutouts of tanks and the other actual tanks. After some training, the net was able to discriminate flawlessly between the situations. As is customary, the net was then tested against a third data set showing pretend and real tanks in the same landscape; it failed miserably. Further investigation revealed that the original two data sets had been filmed on different days. One of the days was overcast with lots of clouds, and the other day was clear. The net, it turned out, was discriminating between the presence and absence of clouds. The anecdote shows the ambiguous potential of epistemically autonomous devices for categorizing the world in entirely different ways from the humans with whom they interact. While this autonomy might be used to enrich the human perception of the world by revealing novel kinds of constructions, it also can create a breed of autonomous devices that parse the world in radically different ways from their human trainers.
+>
+>
+> A counter-narrative, also perhaps apocryphal, emerged from the 1991 35ya Gulf War. US soldiers firing at tanks had been trained on simulators that imaged flames shooting out from the tank to indicate a kill. When army investigators examined Iraqi tanks that were defeated in battles, they found that for some tanks the soldiers had fired four to five times the amount of munitions necessary to disable the tanks. They hypothesized that the overuse of firepower happened because no flames shot out, so the soldiers continued firing. If the hypothesis is correct, human perceptions were altered in accord with the idiosyncrasies of intelligent machines, providing an example of what can happen when human-machine perceptions are caught in a feedback loop with one another.
+
+Linda Null & Julie Lobur, [_The Essentials of Computer Organization and Architecture_ (third edition)](https://books.google.com/books?id=GKgxDwAAQBAJ "Essentials of Computer Organization and Architecture"), 2003 23ya/2014 12ya (pg439–440 in 1 st edition, pg658 in 3 rd edition):
+
+> Correct training requires thousands of steps. The training time itself depends on the size of the network. As the number of perceptrons increases, the number of possible “states” also increases.
+>
+>
+> Let’s consider a more sophisticated example, that of determining whether a tank is hiding in a photograph. A neural net can be configured so that each output value correlates to exactly one pixel. If the pixel is part of the image of a tank, the net should output a one; otherwise, the net should output a zero. The input information would most likely consist of the color of the pixel. The network would be trained by feeding it many pictures with and without tanks. The training would continue until the network correctly identified whether the photos included tanks. The U.S. military conducted a research project exactly like the one we just described. One hundred photographs were taken of tanks hiding behind trees and in bushes, and another 100 photographs were taken of ordinary landscape with no tanks. Fifty photos from each group were kept “secret,” and the rest were used to train the neural network. The network was initialized with random weights before being fed one picture at a time. When the network was incorrect, it adjusted its input weights until the correct output was reached. Following the training period, the 50 “secret” pictures from each group of photos were fed into the network. The neural network correctly identified the presence or absence of a tank in each photo. The real question at this point has to do with the training—had the neural net actually learned to recognize tanks? The Pentagon’s natural suspicion led to more testing. Additional photos were taken and fed into the network, and to the researchers’ dismay, the results were quite random. The neural net could not correctly identify tanks within photos. After some investigation, the researchers determined that in the original set of 200 photos, all photos with tanks had been taken on a cloudy day, whereas the photos with no tanks had been taken on a sunny day. The neural net had properly separated the two groups of pictures, but had done so using the color of the sky to do this rather than the existence of a hidden tank. The government was now the proud owner of a very expensive neural net that could accurately distinguish between sunny and cloudy days!
+>
+>
+> This is a great example of what many consider the biggest issue with neural networks. If there are more than 10 to 20 neurons, it is impossible to understand how the network is arriving at its results. One cannot tell if the net is making decisions based on correct information, or, as in the above example, something totally irrelevant. Neural networks have a remarkable ability to derive meaning and extract patterns from data that are too complex to be analyzed by human beings. However, some people trust neural networks to be experts in their area of training. Neural nets are used in such areas as sales forecasting, risk management, customer research, undersea mine detection, facial recognition, and data validation. Although neural networks are promising, and the progress made in the past several years has led to significant funding for neural net research, many people are hesitant to put confidence in something that no human being can completely understand.
+
+David Gerhard, [“Pitch Extraction and Fundamental Frequency: History and Current Techniques”](http://sapyc.espe.edu.ec/evcarrera/DSP/pitch.pdf), Technical Report TR-CS 2003–06, November 2003 23ya:
+
+> The choice of the dimensionality and domain of the input set is crucial to the success of any connectionist model. A common example of a poor choice of input set and test data is the Pentagon’s foray into the field of object recognition. This story is probably apocryphal and many different versions exist on-line, but the story describes a true difficulty with neural nets.
+>
+>
+> As the story goes, a network was set up with the input being the pixels in a picture, and the output was a single bit, yes or no, for the existence of an enemy tank hidden somewhere in the picture. When the training was complete, the network performed beautifully, but when applied to new data, it failed miserably. The problem was that in the test data, all of the pictures that had tanks in them were taken on cloudy days, and all of the pictures without tanks were taken on sunny days. The neural net was identifying the existence or non-existence of sunshine, not tanks.
+
+[Rice lecture #24, “COMP 200: Elements of Computer Science”](https://www.clear.rice.edu/comp200/02spring/Lecture-notes/lec24.txt), 2002-03-18:
+
+> 1. Tanks in Desert Storm
+>
+>
+>
+> Sometimes you have to be careful what you train on . . .
+>
+>
+> The problem with neural nets is that you never know what features they’re actually training on. For example:
+>
+>
+> The US military tried to use neural nets in Desert Storm for tank recognition, so unmanned tanks could identify enemy tanks and destroy them. They trained the neural net on multiple images of “friendly” and enemy tanks, and eventually had a decent program that seemed to correctly identify friendly and enemy tanks.
+>
+>
+> Then, when they actually used the program in a real-world test phase with actual tanks, they found that the tanks would either shoot at nothing or shoot at everything. They certainly seemed to be incapable of distinguishing friendly or enemy tanks.
+>
+>
+> Why was this? It turns out that the images they were training on always had glamour-shot type photos of friendly tanks, with an immaculate blue sky, etc. The enemy tank photos, on the other hand, were all spy photos, not very clear, sometimes fuzzy, etc. And it was these characteristics that the neural net was training on, not the tanks at all. On a bright sunny day, the tanks would do nothing. On an overcast, hazy day, they’d start firing like crazy . . .
+
+Andrew Ilachinski, _Cellular Automata: A Discrete Universe_, 2001 25ya (pg547):
+
+> There is an telling story about how the Army recently went about teaching a backpropagating net to identify tanks set against a variety of environmental backdrops. The programmers correctly fed their multi-layer net photograph after photograph of tanks in grasslands, tanks in swamps, no tanks on concrete, and so on. After many trials and many thousands of iterations, their net finally learned all of the images in their database. The problem was that when the presumably “trained” net was tested with other images that were not part of the original training set, it failed to do any better than what would be expected by chance. What had happened was that the input/training fact set was statistically corrupt. The database consisted mostly of images that showed a tank only if there were heavy clouds, the tank itself was immersed in shadow or there was no sun at all. The Army’s neural net had indeed identified a latent pattern, but it unfortunately had nothing to do with tanks: it had effectively learned to identify the time of day! The obvious lesson to be taken away from this amusing example is that how well a net “learns” the desired associations depends almost entirely on how well the database of facts is defined. Just as Monte Carlo simulations in statistical mechanics may fall short of intended results if they are forced to rely upon poorly coded random number generators, so do backpropagating nets typically fail to achieve expected results if the facts they are trained on are statistically corrupt.
+
+[_Intelligent Data Analysis In Science_](https://gwern.net/doc/ai/nn/2000-cartwright-intelligentdataanalysisinscience.pdf), Hugh M. Cartwright 2000 26ya, pg126, writes (according to Google Books’s snippet view; Cartwright’s version appears to be a direct quote or close paraphrase of an earlier 1994 32ya chemistry paper, Goodacre et al 1994 32ya):
+
+> …television programme [_Horizon_](https://en.wikipedia.org/wiki/Horizon_(British_TV_series)); a neural network was trained to attempt to distinguish tanks from trees. Pictures were taken of forest scenes lacking military hardware and of similar but perhaps less bucolic landscapes which also contained more-or-less camouflaged battle tanks. A neural network was trained with these input data and found to differentiate successfully between tanks and trees. However, when a new set of pictures was analysed by the network, it failed to detect the tanks. After further investigation, it was found…
+
+Daniel Robert Franklin & Philippe Crochat, [`libneural` tutorial](https://web.archive.org/web/20001029201251/http://ieee.uow.edu.au/~daniel/software/libneural/BPN_tutorial/BPN_English/BPN_English/node9.html), 2000-03-23:
+
+> A neural network is useless if it only sees one example of a matching input/output pair. It cannot infer the characteristics of the input data for which you are looking for from only one example; rather, many examples are required. This is analogous to a child learning the difference between (say) different types of animals—the child will need to see several examples of each to be able to classify an arbitrary animal… It is the same with neural networks. The best training procedure is to compile a wide range of examples (for more complex problems, more examples are required) which exhibit all the different characteristics you are interested in. It is important to select examples which do not have major dominant features which are of no interest to you, but are common to your input data anyway. One famous example is of the US Army “Artificial Intelligence” tank classifier. It was shown examples of Soviet tanks from many different distances and angles on a bright sunny day, and examples of US tanks on a cloudy day. Needless to say it was great at classifying weather, but not so good at picking out enemy tanks.
+
+### [1990s](https://gwern.net/tank#s-2 "Link to section: § '1990s'")
+
+[Peter Watts’s](https://en.wikipedia.org/wiki/Peter_Watts_(author))1999 27ya SF novel [_Starfish_](https://www.rifters.com/real/STARFISH.htm) (Rifters #1) may have alluded to the tank story [in an anecdote](https://www.rifters.com/real/STARFISH.htm#bulrushes) foreshadowing its major plot twist, which also involves neural networks generalizing poorly despite appearing to work well.
+
+[“Neural Network Follies”](https://neil.fraser.name/writing/tank/), Neil Fraser, September 1998 28ya:
+
+> In the 1980s, the Pentagon wanted to harness computer technology to make their tanks harder to attack…The research team went out and took 100 photographs of tanks hiding behind trees, and then took 100 photographs of trees—with no tanks. They took half the photos from each group and put them in a vault for safe-keeping, then scanned the other half into their mainframe computer. The huge neural network was fed each photo one at a time and asked if there was a tank hiding behind the trees. Of course at the beginning its answers were completely random since the network didn’t know what was going on or what it was supposed to do. But each time it was fed a photo and it generated an answer, the scientists told it if it was right or wrong. If it was wrong it would randomly change the weightings in its network until it gave the correct answer. Over time it got better and better until eventually it was getting each photo correct. It could correctly determine if there was a tank hiding behind the trees in any one of the photos…So the scientists took out the photos they had been keeping in the vault and fed them through the computer. The computer had never seen these photos before—this would be the big test. To their immense relief the neural net correctly identified each photo as either having a tank or not having one. _Independent testing_: The Pentagon was very pleased with this, but a little bit suspicious. They commissioned another set of photos (half with tanks and half without) and scanned them into the computer and through the neural network. The results were completely random. For a long time nobody could figure out why. After all nobody understood how the neural had trained itself. Eventually someone noticed that in the original set of 200 photos, all the images with tanks had been taken on a cloudy day while all the images without tanks had been taken on a sunny day. The neural network had been asked to separate the two groups of photos and it had chosen the most obvious way to do it—not by looking for a camouflaged tank hiding behind a tree, but merely by looking at the color of the sky…This story might be apocryphal, but it doesn’t really matter. It is a perfect illustration of the biggest problem behind neural networks. Any automatically trained net with more than a few dozen neurons is virtually impossible to analyze and understand.
+
+[Tom White](https://x.com/dribnet/status/914945926266970112) attributes (in October 2017) to Marvin Minsky some version of the tank story being told in MIT classes 20 years before, ~1997 (but doesn’t specify the detailed story or version other than apparently the results were “classified”).
+
+Vasant Dhar & Roger Stein, [_Intelligent Decision Support Methods_](https://gwern.net/doc/ai/nn/1997-dhar-intelligentdecisionsupportmethods.pdf), 1997 29ya (pg98, limited Google Books snippet):
+
+> …However, when a new set of photographs were used, the results were horrible. At first the team was puzzled. But after careful inspection of the first two sets of photographs, they discovered a very simple explanation. The photos with tanks in them were all taken on sunny days, and those without the tanks were taken on overcast days. The network had _not_ learned to identify tank like images; instead, it had learned to identify photographs of sunny days and overcast days.
+
+Royston Goodacre, Mark J. Neal, & Douglas B. Kell, [“Quantitative Analysis of Multivariate Data Using Artificial Neural Networks: A Tutorial Review and Applications to the Deconvolution of Pyrolysis Mass Spectra”](https://gwern.net/doc/ai/nn/fully-connected/1996-goodacre.pdf), 1994-04-29:
+
+> …As in all other data analysis techniques, these supervised learning methods are not immune from sensitivity to badly chosen initial data (113). [113: Zupan, J. and J. Gasteiger: _Neural Networks for Chemists: An Introduction_. VCH Verlagsgesellschaft, Weinheim (1993 33ya)] Therefore the exemplars for the training set must be carefully chosen; the golden rule is “garbage in—garbage out”. An excellent example of an unrepresentative training set was discussed some time ago on the BBC television programme _Horizon_; a neural network was trained to attempt to distinguish tanks from trees. Pictures were taken of forest scenes lacking military hardware and of similar but perhaps less bucolic landscapes which also contained more-or-less camouflaged battle tanks. A neural network was trained with these input data and found to differentiate most successfully between tanks and trees. However, when a new set of pictures was analysed by the network, it failed to distinguish the tanks from the trees. After further investigation, it was found that the first set of pictures containing tanks had been taken on a sunny day whilst those containing no tanks were obtained when it was overcast. The neural network had therefore thus learned simply to recognise the weather! We can conclude from this that the training and tests sets should be carefully selected to contain representative exemplars encompassing the appropriate variance over all relevant properties for the problem at hand.
+
+Fernando Pereira, [“neural redlining”, RISKS 16(41), 1994-09-12](https://catless.ncl.ac.uk/risks/16.41.html):
+
+> Fred’s comments will hold not only of neural nets but of any decision model trained from data (eg. Bayesian models, decision trees). It’s just an instance of the old “GIGO” phenomenon in statistical modeling…Overall, the whole issue of evaluation, let alone certification and legal standing, of complex statistical models is still very much open. (This reminds me of a possibly apocryphal story of problems with biased data in neural net training. Some US defense contractor had supposedly trained a neural net to find tanks in scenes. The reported performance was excellent, with even camouflaged tanks mostly hidden in vegetation being spotted. However, when the net was tested on yet a new set of images supplied by the client, the net did not do better than chance. After an embarrassing investigation, it turned out that all the tank images in the original training and test sets had very different average intensity than the non-tank images, and thus the net had just learned to discriminate between two image intensity levels. Does anyone know if this actually happened, or is it just in the neural net “urban folklore”?)
+
+Erich Harth, [_The Creative Loop: How the Brain Makes a Mind_](https://gwern.net/doc/ai/nn/1993-harth-thecreativeloop.pdf), 1993 33ya/1995 31ya (pg158, limited Google Books snippet):
+
+> …55. The net was _trained_ to detect the presence of tanks in a landscape. The training consisted in showing the device many photographs of scene, some with tanks, some without. In some cases—as in the picture on page 143—the tank’s presence was not very obvious. The inputs to the neural net were digitized photographs;
+
+[Hubert L. Dreyfus](https://en.wikipedia.org/wiki/Hubert_Dreyfus)&[Stuart E. Dreyfus](https://en.wikipedia.org/wiki/Stuart_Dreyfus), [“What Artificial Experts Can and Cannot Do”](https://www.jefftk.com/dreyfus92.pdf), 1992 34ya:
+
+> All the “continue this sequence” questions found on intelligence tests, for example, really have more than one possible answer but most human beings share a sense of what is simple and reasonable and therefore acceptable. But when the net produces an unexpected association can one say it has failed to generalize? One could equally well say that the net has all along been acting on a different definition of “type” and that that difference has just been revealed. For an amusing and dramatic case of creative but unintelligent generalization, consider the legend of one of connectionism’s first applications. In the early days of the perceptron the army decided to train an artificial neural network to recognize tanks partly hidden behind trees in the woods. They took a number of pictures of a woods without tanks, and then pictures of the same woods with tanks clearly sticking out from behind trees. They then trained a net to discriminate the two classes of pictures. The results were impressive, and the army was even more impressed when it turned out that the net could generalize its knowledge to pictures from each set that had not been used in training the net. Just to make sure that the net had indeed learned to recognize partially hidden tanks, however, the researchers took some more pictures in the same woods and showed them to the trained net. They were shocked and depressed to find that with the new pictures the net totally failed to discriminate between pictures of trees with partially concealed tanks behind them and just plain trees. The mystery was finally solved when someone noticed that the training pictures of the woods without tanks were taken on a cloudy day, whereas those with tanks were taken on a sunny day. The net had learned to recognize and generalize the difference between a woods with and without shadows! Obviously, not what stood out for the researchers as the important difference. This example illustrates the general point that a net must share size, architecture, initial connections, configuration and socialization with the human brain if it is to share our sense of appropriate generalization
+
+Hubert Dreyfus appears to have told this story earlier in 1990 36ya or 1991 35ya, as a similar story appears in episode 4 ([German](https://www.youtube.com/watch?v=cG7v9eCq2u4&t=33m49s)) (starting 33m49s) of the BBC documentary series [_The Machine That Changed the World_](https://en.wikipedia.org/wiki/The_Machine_That_Changed_the_World_(miniseries)), broadcast 1991-11-08. Hubert L. Dreyfus, [_What Computers Still Can’t Do: A Critique of Artificial Reason_](https://gwern.net/doc/ai/1992-dreyfus-whatcomputerstillcantdo.epub), 1992 34ya, repeats the story in very similar but not quite identical wording ([Jeff Kaufman](https://www.jefftk.com/p/detecting-tanks) notes that Dreyfus drops the qualifying “legend of” description):
+
+> …But when the net produces an unexpected association, can one say that it has failed to generalize? One could equally well say that the net has all along been acting on a different definition of “type” and that that difference has just been revealed. For an amusing and dramatic case of creative but unintelligent generalization, consider one of connectionism’s first applications. In the early days of this work the army tried to train an artificial neural network to recognize tanks in a forest. They took a number of pictures of a forest without tanks and then, on a later day, with tanks clearly sticking out from behind trees, and they trained a net to discriminate the two classes of pictures. The results were impressive, and the army was even more impressed when it turned out that the net could generalize its knowledge to pictures that had not been part of the training set. Just to make sure that the net was indeed recognizing partially hidden tanks, however, the researchers took more pictures in the same forest and showed them to the trained net. They were depressed to find that the net failed to discriminate between the new pictures of trees with tanks behind them and the new pictures of just plain trees. After some agonizing, the mystery was finally solved when someone noticed that the original pictures of the forest without tanks were taken on a cloudy day and those with tanks were taken on a sunny day. The net had apparently learned to recognize and generalize the difference between a forest with and without shadows! This example illustrates the general point that a network must share our commonsense understanding of the world if it is to share our sense of appropriate generalization.
+
+Dreyfus’s _What Computers Still Can’t Do_ is listed as a revision of his 1972 54ya book, [_What Computers Can’t Do: A Critique of Artificial Reason_](https://archive.org/details/whatcomputerscan017504mbp), but the tank story is not in the 1972 54ya book, only the 1992 34ya one. (Dreyfus’s version is also quoted in the 2017 NYT article and Hillis 1996’s _Geography, Identity, and Embodiment in Virtual Reality_, pg346.)
+
+Laveen N. Kanal, [_Artificial Neural Networks and Statistical Pattern Recognition: Old and New Connections_’s](https://gwern.net/doc/ai/nn/1991-sethi-artificialneuralnetworksandstatisticalpatternrecognition.pdf) Foreword, discusses some early NN/tank research (predating not just LeCun’s convolutions but backpropagation), 1991 35ya:
+
+> …[Frank] Rosenblatt had not limited himself to using just a single Threshold Logic Unit but used networks of such units. The problem was how to train multilayer perceptron networks. A paper on the topic written by Block, Knight and Rosenblatt was murky indeed, and did not demonstrate a convergent procedure to train such networks. In 1962–63 at Philco-Ford, seeking a systematic approach to designing layered classification nets, we decided to use a hierarchy of threshold logic units with a first layer of “feature logics” which were threshold logic units on overlapping receptive fields of the image, feeding two additional levels of weighted threshold logic decision units. The weights in each level of the hierarchy were estimated using statistical methods rather than iterative training procedures [L.N. Kanal & N.C. Randall, [“Recognition System Design by Statistical Analysis”](https://gwern.net/doc/ai/1964-kanal.pdf), Proc. 19 th Conf. ACM, 1964 62ya]. We referred to the networks as two layer networks since we did not count the input as a layer. On a project to recognize tanks in aerial photography, the method worked well enough in practice that the U.S. Army agency sponsoring the project decided to classify the final reports, although previously the project had been unclassified. We were unable to publish the classified results! Then, enamored by the claimed promise of coherent optical filtering as a parallel implementation for automatic target recognition, the funding we had been promised was diverted away from our electro-optical implementation to a coherent optical filtering group. Some years later we presented the arguments favoring our approach, compared to optical implementations and trainable systems, in an article titled “Systems Considerations for Automatic Imagery Screening” by T.J. Harley, L.N. Kanal and N.C. Randall, which is included in the IEEE Press reprint volume titled [_Machine Recognition of Patterns_](https://gwern.net/doc/ai/nn/1977-agrawala-machinerecognitionofpatterns.pdf) edited by A. Agrawala 1977[1](https://gwern.net/tank#fn1). In the years which followed multilevel statistically designed classifiers and AI search procedures applied to pattern recognition held my interest, although comments in my 1974 52ya survey, “Patterns In Pattern Recognition: 1968–6 1974 52ya” [IEEE Trans. on IT, 1974 52ya], mention papers by Amari and others and show an awareness that neural networks and biologically motivated automata were making a comeback. In the last few years trainable multilayer neural networks have returned to dominate research in pattern recognition and this time there is potential for gaining much greater insight into their systematic design and performance analysis…
+
+While Kanal & Randall 1964 62ya matches in some ways, including the image counts, there is no mention of failure either in the paper or Kanal’s 1991 35ya reminiscences (rather, Kanal implies it was highly promising), there is no mention of a field deployment or additional testing which could have revealed overfitting, and given their use of binarizing, it’s not clear to me that their 2-layer algorithm even _could_ overfit to global brightness; the photos also appear to have been taken at low enough altitude for there to be no clouds, and to be taken under similar (possibly controlled) lighting conditions. The description in Kanal & Randall 1964 62ya is somewhat opaque to me, particularly of the ‘Laplacian’ they use to binarize or convert to edges, but there’s more background in their [“Semi-Automatic Imagery Screening Research Study and Experimental Investigation, Volume 1”](http://www.dtic.mil/docs/citations/AD0410261), Harley, Bryan, Kanal, Taylor & Grayum 1962 64ya ([mirror](https://gwern.net/doc/ai/1962-harley.pdf)), which indicates that in their preliminary studies they were already interested in prenormalization/preprocessing images to correct for altitude and brightness, and the Laplacian, along with silhouetting and “lineness editing”, noting that “The Laplacian operation eliminates absolute brightness scale as well as low-spatial frequencies which are of little consequence in screening operations.”[2](https://gwern.net/tank#fn2)
+
+An anonymous reader says he heard the story in 1990 36ya:
+
+> I was told about the tank recognition failure by a lecturer on my 1990 36ya Intelligent Knowledge Based Systems MSc, almost certainly [Libor Spacek](https://cmp.felk.cvut.cz/~spacelib/ "Libor Špaček homepage"), in terms of being aware of context in data sets; that being from (the former) Czechoslovakia he expected to see tanks on a motorway whereas most British people didn’t. I also remember reading about a project with DARPA funding aimed at differentiating Russian, European and US tanks where what the image recognition learned was not to spot the differences between tanks but to find trees, because of the US tank photos being on open ground and the Russian ones being in forests; that was during the same MSc course—so very similar to predicting tumours by looking for the ruler used to measure them in the photo—but I don’t recall the source (it wasn’t one of the books you cite though, it was either a journal article or another text book).
+
+### [1980s](https://gwern.net/tank#s-3 "Link to section: § '1980s'")
+
+[Chris Brew](https://x.com/cbrew/status/920088821823344640) states (2017-10-16) that he “Heard the story in 1984 42ya with pigeons instead of neural nets”.
+
+### [1960s](https://gwern.net/tank#s-4 "Link to section: § '1960s'")
+
+#### [Fredkin](https://gwern.net/tank#fredkin "Link to section: § 'Fredkin'")
+
+[Edward Fredkin](https://en.wikipedia.org/wiki/Edward_Fredkin), in [an email to Eliezer Yudkowsky](https://www.lesswrong.com/posts/5o3CxyvZ2XKawRB5w/machine-learning-and-unintended-consequences?commentId=SNHJNFN9SjNW6djgc) on 2013-02-26, recounts an interesting anecdote about the 1960s claiming to be the grain of truth behind the story; quoting Yudkowsky’s quote in full:
+
+> By the way, the story about the two pictures of a field, with and without army tanks in the picture, comes from me. I attended a meeting in Los Angeles [at [Caltech](https://en.wikipedia.org/wiki/California_Institute_of_Technology) or [RAND](https://en.wikipedia.org/wiki/RAND_Corporation)?], about half a century ago [~1963?] where someone gave a paper showing how a random net could be trained to detect the tanks in the picture. I was in the audience. At the end of the talk I stood up and made the comment that it was obvious that the picture with the tanks was made on a sunny day while the other picture (of the same field without the tanks) was made on a cloudy day. I suggested that the “neural net” had merely trained itself to recognize the difference between a bright picture and a dim picture.
+
+Fredkin doesn’t mention it, but in the early 1960s, he was highly active in computer vision R&D by founding his startup [Information International, Inc.](https://en.wikipedia.org/wiki/Information_International,_Inc.), which was located in Los Angeles and worked on Lisp, digitization, OCR and other imaging applications, much of which catered to military applications; so this provides a natural context both for Fredkin to be attending such talks and for military connections.
+
+Fredkin apparently doesn’t claim that his hypothetical was ever _proven_, or else he would have added that as well.[3](https://gwern.net/tank#fn3) Since he would likely hear about any proof in the subsequence half-century—having made the criticism so publicly & being well-connected in what was then a small field—his silence implies there never was one.
+
+## [Evaluation](https://gwern.net/tank#evaluation "Link to section: § 'Evaluation'")
+
+### [Sourcing](https://gwern.net/tank#sourcing "Link to section: § 'Sourcing'")
+
+The absence of any hard citations is striking: even when a citation is supplied, it is invariably to a relatively recent source like Dreyfus, and then the chain ends. Typically for a real story, one will find at least one or two hints of a penultimate citation and then a final definitive citation to some very difficult-to-obtain or obscure work (which then is often quite different from the popularized version but still recognizable as the original); for example, another popular cautionary AI urban legend is that the 1956 70ya[Dartmouth workshop](https://en.wikipedia.org/wiki/Dartmouth_workshop) claimed that a single graduate student working for a summer could solve computer vision (or perhaps AI in general), which is a highly distorted misleading description of the [original 1955 71ya proposal’s](https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html "'A Proposal For The Dartmouth Summer Research Project On Artificial Intelligence', McCarthy et al 1955") realistic claim that “a 2 month, 10 man study of artificial intelligence” could yield “a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.”[4](https://gwern.net/tank#fn4) Instead, everyone either disavows it as an urban legend or possibly apocryphal, or punts to someone else. (Minsky’s 2011 15ya version initially seems concrete, but while he specifically attributes the musical score story to a friend & claims to have found the trick personally, he is then as vague as anyone else about the tank story, saying it just “happened” somewhere “in the United States at one of our research institutes”, at an unmentioned institute by unmentioned people at an unmentioned date for an unmentioned branch of the military.)
+
+### [Variations](https://gwern.net/tank#variations "Link to section: § 'Variations'")
+
+> _Question to Radio Yerevan_: “Is it correct that Grigori Grigorievich Grigoriev won a luxury car at the All-Union Championship in Moscow?”
+>
+>
+> _Radio Yerevan answered_: “In principle, yes. But first of all it was not Grigori Grigorievich Grigoriev, but Vassili Vassilievich Vassiliev; second, it was not at the All-Union Championship in Moscow, but at a Collective Farm Sports Festival in Smolensk; third, it was not a car, but a bicycle; and fourth he didn’t win it, but rather it was stolen from him.”
+>
+>
+> [“Radio Yerevan Jokes”](https://web.archive.org/web/20140908045019/http://www.bratislavaguide.com/radio-yerevan-jokes) (collected by Allan Stevo)
+
+It is also interesting that not all the stories imply quite the same problem with the hypothetical NN. Dataset bias/selection effects is not the same thing as overfitting or disparate impact, but some of the story tellers don’t realize that. For example, in some stories, the NN fails when it’s tested on additional heldout data (overfitting), not when it’s tested on data from an entire different photographer or field exercise or data source (dataset bias/distributional shift). Or, Alexander Harrowell cites disparate impact in a medical school as if it were an example of the same problem, but it’s not—at least in the USA, a NN would be correct in inferring that white students are more likely to succeed, as that is a real predictor (this would be an example of how people play rather fast and loose with claims of “algorithmic bias”), and it would not necessarily be the case that, say, randomized admission of more non-white students would be certain to increase the number of successful graduates; such a scenario is, however, possible and illustrates the difference between predictive models & causal models for control & optimization, and the need for experiments/reinforcement learning.
+
+A read of all the variants together raises more questions than it answers:
+
+* Did this story happen in the 1960s, 1980s, 1990s, or during Desert Storm in the 1990s?
+
+* Was the research conducted by the US military, or researchers for another NATO country?
+
+* Were the photographs taken by satellite, from the air, on the ground, or by spy cameras?
+
+* Were the photographs of American tanks, plywood cutouts, Soviet tanks, or Warsaw Pact tanks?
+
+* Were the tanks out in the open, under cover, or fully camouflaged?
+
+* Were these photographs taken in forests, fields, deserts, swamps, or all of them?
+
+* Were the photographs taken in same place but different time of day, same place but different days, or different places entirely?
+
+* Were there 100, 200, or thousands of photographs; and how many were in the training vs validation set?
+
+* Was the input in black-and-white binary, grayscale, or color?
+
+* Was the tell-tale feature either field vs forest, bright vs dark, the presence vs absence of clouds, the presence vs absence of shadows, the length of shadows, or an accident in film development unrelated to weather entirely?
+
+* Was the NN to be used for image processing or in autonomous robotic tanks?
+
+* Was it even a NN?
+
+* Was the dataset bias caught quickly within “a few hours”, later by a suspicious team member, later still when applied to an additional set of tank photographs, during further testing producing a new dataset, much later during a live demo for military officers, or only after live deployment in the field?
+
+Almost every aspect of the tank story which _could_ vary _does_ vary.
+
+### [Origin](https://gwern.net/tank#origin "Link to section: § 'Origin'")
+
+So where does this urban legend come from? The key anecdote appears to be [Edward Fredkin’s](https://gwern.net/tank#fredkin) as it precedes all other excerpts except perhaps the research Kanal describes; Fredkin’s story does _not_ confirm the tank story as he merely speculates that brightness was driving the results, much less all the extraneous details about photographic film being accidentally overdeveloped or robot tanks going berserk or a demo failing in front of Army brass.
+
+But it’s easy to see how Fredkin’s reasonable (but never proven) question could have memetically evolved into the tank story as finally fixed into published form by Dreyfus’s article:
+
+1. **Setting**: Kanal & Randall set up their very small simple early perceptrons on some tiny binary aerial photos of tanks, in interesting early work, and Fredkin attends the talk sometime around 1960–1963 63ya
+
+2. **The Question**: Fredkin then asks in the Q&A whether the perceptron is not learning square-shapes but brightness?
+
+3. **Punting**: of course neither Fredkin nor Kanal & Randall can know on the spot whether this critique is right or wrong (perhaps that question motivated the binarized results reported in Kanal & Randall 1964 62ya, which showed that was not the case for their results?), and the question remains unanswered
+
+4. **Anecdotizing**: but someone in the audience considers that an excellent observation about methodological flaws in NN research, and perhaps they (or Fredkin) repeats the story to others, who find it useful too, and along the way, Fredkin’s _question mark_ gets dropped and the _possible_ flaw becomes an _actual_ flaw, with the punchline: “…and it turned out their NN were just detecting average brightness!”
+
+One might expect Kanal & Randall to rebut these rumors, if only by publishing additional papers on their functioning system, but by a quirk of fate, as Kanal explains in his preface, after their 1964 62ya paper, the Army liked it enough to make it classified and then they were reassigned to an entirely different task, killing progress entirely.[7](https://gwern.net/tank#fn7)
+
+5. **Proliferation**: In the absence of any counternarrative (silence is considered consent), the tank story continues spreading.
+
+6. **Mutation**: but now the story is incomplete, a joke missing most of the setup to its punchline—_how_ did these Army researchers discover the NN had tricked them and what was the brightness difference from? The various versions propose different resolutions, and likewise, appropriate details about the tank data must invented.
+
+7. [**Fixation**](https://en.wikipedia.org/wiki/Fixation_(population_genetics)): Eventually, after enough mutations, a version reaches Dreyfus, already a well-known critic of the AI establishment, who then uses it in his article/book, virally spreading it globally to pop up in random places thenceforth, and fixating it as an universally-known _ur_-text. (Further memetic mutations can and often will occur, but diligent writers & researchers will ‘correct’ variants by returning to the Dreyfus version.)
+
+One might try to write Dreyfus off as a coincidence and argue that the US Army _must_ have had so many neural net research programs going that one of the others is the real origin, but one would expect those programs to result in spinoffs, more reports, reports since declassified, etc. It’s been half a century, after all. And despite the close association of the US military with MIT and early AI work, tanks do not seem to have been a major focus of early NN research—for example, [Schmidhuber’s history](https://arxiv.org/abs/1404.7828#schmidhuber) does not mention tanks at all, and most of my paper searches kept pulling up NN papers about ‘tanks’ as in vats, such as controlling stirring/mixing tanks for chemistry. Nor is it a safe assumption that the military always has much more advanced technology than the public or private sectors; often, they can be quite behind or at the status quo.[8](https://gwern.net/tank#fn8)
+
+## [Could It Happen?](https://gwern.net/tank#could-it-happen "Link to section: § 'Could it Happen?'")
+
+Could something like the tank story (a NN learning to distinguish solely on average brightness levels) happen in 2017 with state-of-the-art techniques like convolutional neural networks (CNNs)? (After all, presumably nobody _really_ cares about what mistakes a crude perceptron may or may not have once made back in the 1960s; most/all of the story-tellers are using it for didactic effect in warning against carelessness in contemporary & future AI research/applications.) I would guess that while it could happen, it would be considerably less likely now than then for several reasons:
+
+1. a common preprocessing step in computer vision (and NNs in general) is to “whiten” the image by standardizing or transforming pixels to a normal distribution; this would tend to wipe global brightness levels, promoting invariance to illumination
+
+2. in addition to or instead of whitening, it is also common to use aggressive “data augmentation”: shifting the image by a few pixels in each direction, cropping it randomly, adjusting colors to be slightly more red/green/blue, flipping horizontally, barrel-warping it, adding JPEG compression noise/artifacts, brightening or darkening, etc.
+
+None of these transformations should affect whether an image is classifiable as “dog” or “cat”[9](https://gwern.net/tank#fn9), the reasoning goes, so the NN should learn to see past them, and generating variants during training provides additional data for free. Aggressive data augmentation would make it harder to pick up global brightness as a cheap trick.
+
+3. CNNs have built-in biases (compared to fully-connected neural networks) towards edges and other structures, rather than global averages; convolutions want to find edges and geometric patterns like little squares for tanks. (This point is particularly germane in light of the brain inspiration for convolutions & Dreyfus & Dreyfus 1992’s interpretation of the tank story.)
+
+4. image classification CNNs, due to their large sizes, are often trained on large datasets with many classes to categorize images into (canonically, ImageNet with 1000 classes over a million images; much larger datasets, such as 300 million images, have been explored and found to still offer benefits). Perforce, most of these images will not be generated by the dataset maintainer and will come from a wide variety of peoples, places, cameras, and settings, reducing any systematic biases. It would be difficult to find a cheap trick which works over many of those categories simultaneously, and the NN training will constantly erode any category-specific tricks in favor of more generalizable pattern-recognition (in part because there’s no inherent ‘modularity’ which could factor a NN into a “tank cheap trick” NN & a “everything else real pattern-recognition” NN). The power of generalizable abstractions will tend to overwhelm the shortcuts, and the more data & tasks a NN is trained on, providing greater supervision & richer insight, the more this will be the case.
+
+ * Even in the somewhat unusual case of a special-purpose binary classification CNN being trained on a few hundred images, because of the large sizes of good CNNs, it is typical to at least start with a pretrained ImageNet CNN in order to benefit from all the learned knowledge about edges & whatnot before “finetuning” on the special-purpose small dataset. If the CNN starts with a huge inductive bias towards edges etc., it will have a hard time throwing away its informative priors and focusing purely on global brightness. (Often in finetuning, the lower levels of the CNN aren’t allowed to change at all!)
+
+ * Another variant on transfer learning is to use the CNN as a feature-generator, by taking the final layers’ state computed on a specific image and using them as a vector embedding, a sort of summary of everything about the image content relevant to classification; this embedding is useful for other kinds of CNNs for purposes like style transfer (style transfer aims to warp an image towards the appearance of another image while preserving the embedding and thus presumably the content) or for GANs generating images (the discriminator can use the features to detect “weird” images which don’t make sense, thereby forcing the generator to learn what images correspond to realistic embeddings).
+
+5. CNNs would typically throw warning signs before a serious field deployment, either in diagnostics or failures to extend the results.
+
+ * One benefit of the filter setup of CNNs is that it’s easy to visualize what the lower layers are ‘looking at’; typically, CNN filters will look like diagonal or horizontal lines or curves or other simple geometric patterns. In the case of a hypothetical brightness-detector CNN, because it is not recognizing any shapes whatsoever or doing anything but trivial brightness averaging, one would expect its filters to look like random noise and definitely nothing like the usual filter visualizations. This would immediately alarm any deep learning researcher that the CNN is not learning what they thought it was learning.
+
+ * Related to filter visualization is input visualization: it’s common to generate some heatmaps of input images to see what regions of the input image are influencing the classification the most. If you are classifying “cats vs dogs”, you expect a heatmap of a cat image to focus on the cat’s head and tail, for example, and not on the painting on the living room wall behind it; if you have an image of a tank in a forest, you expect the heatmap to focus on the tank rather than trees in the corner or nothing in particular, just random-seeming pixels all over the image. If it’s not focusing on the tank at all, how is it doing the classification?, one would then wonder. ([“Picasso: A Modular Framework for Visualizing the Learning Process of Neural Network Image Classifiers”](https://arxiv.org/abs/1705.05627) ([blog](https://medium.com/merantix/picasso-a-free-open-source-visualizer-for-cnns-d8ed3a35cfc5 "Picasso: A free open-source visualizer for Convolutional Neural Networks; Cloudy with a chance of tanks")), Henderson & Rothe 2017-05-16 quote Yudkowsky 2008’s version of the tank story as a motivation for their heatmap visualization tool and demonstrate that, for example, blocking out the sky in a tank image doesn’t bother a VGG-16 CNN image classifier but block the tank’s treads does, and the heatmap focuses on the tank itself.) There are additional methods for trying to understand whether the NN has learned a potentially useful algorithm using other methods such as the previously cited LIME.
+
+6. Also related to the visualization is going beyond classification to the logical next step of “localization” or “image segmentation”: having detected an image with a tank in it _somewhere_, it is natural (especially for military purposes) to ask _where_ in the image the tank is?
+
+A CNN which is truly detecting the tank itself will lend itself to image segmentation (eg. CNN success in reaching human levels of ImageNet classification performance have also resulted in extremely good segmentation of an image by categorizing each pixel as human/dog/cat/etc.), while one learning the cheap trick of brightness will utterly fail at guessing better than chance which pixels are the tank.
+
+So, it is highly unlikely that a CNN trained via a normal workflow (data-augmented finetuning of a pretrained ImageNet CNN with standard diagnostics) would fail in this exact way or, at least, make it to a deployed system without failing.
+
+## [Could Something Like It Happen?](https://gwern.net/tank#could-something-like-it-happen "Link to section: § 'Could Something Like it Happen?'")
+
+Could something _like_ the tank story happen, in the sense of a selection-biased dataset yielding NNs which fail dismally in practice? One could imagine it happening and it surely does at least occasionally, but in practice it doesn’t seem to be a particularly serious or common problem—people routinely apply CNNs to very different contexts with considerable success.[10](https://gwern.net/tank#fn10) If it’s such a serious and common problem, one would think that people would be able to provide a wealth of real-world examples of systems deployed with dataset bias making it entirely useless, rather than repeating a fiction from 50 years ago.
+
+One of the most relevant (if unfortunately older & possibly out of date) papers I’ve read on this question of dataset bias is [“Unbiased Look at Dataset Bias”](https://gwern.net/doc/ai/dataset/2011-torralba.pdf), Torralba & Efros 2011 15ya:
+
+> Datasets are an integral part of contemporary object recognition research. They have been the chief reason for the considerable progress in the field, not just as source of large amounts of training data, but also as means of measuring and comparing performance of competing algorithms. At the same time, datasets have often been blamed for narrowing the focus of object recognition research, reducing it to a single benchmark performance number. Indeed, some datasets, that started out as data capture efforts aimed at representing the visual world, have become closed worlds unto themselves (eg. the Corel world, the Caltech101 world, the PASCAL VOC world). With the focus on beating the latest benchmark numbers on the latest dataset, have we perhaps lost sight of the original purpose?
+>
+>
+> The goal of this paper is to take stock of the current state of recognition datasets. We present a comparison study using a set of popular datasets, evaluated based on a number of criteria including: relative data bias, cross-dataset generalization, effects of closed-world assumption, and sample value. The experimental results, some rather surprising, suggest directions that can improve dataset collection as well as algorithm evaluation protocols. But more broadly, the hope is to stimulate discussion in the community regarding this very important, but largely neglected issue.
+
+They demonstrate on several datasets (including ImageNet), that it’s possible for a SVM (CNNs were not used) to guess at above chance levels what dataset an image comes from and that there are noticeable drops in accuracy when a classifier trained on one dataset is applied to ostensibly the same category in another dataset (eg. an ImageNet “car” SVM classifier applied to PASCAL’s “car” images will go from 57% to 36% accuracy). But—perhaps the glass is half-full—in none of the pairs does the performance degrade to near-zero, so despite the definite presence of dataset bias, the SVMs are still learning generalizable, transferable image classification (similarly, [Jo & Bengio 2017](https://arxiv.org/abs/1711.11561)/[Recht et al 2018](https://arxiv.org/abs/1806.00451)/[Recht et al 2019](https://arxiv.org/abs/1902.10811)[11](https://gwern.net/tank#fn11)/[Yadav & Bottou 2019](https://arxiv.org/abs/1905.10498)/[Zhang & Davison 2020](https://arxiv.org/abs/2002.02559)/[Beyer et al 2020](https://arxiv.org/abs/2006.07159#google) show a generalization gap but only a small one with typically better in-sample classifiers performing better out-of-sample, [Kornblith et al 2018](https://arxiv.org/abs/1805.08974#google) show that ImageNet resnets produce multiple new SOTAs on other image datasets using finetuning transfer learning, [Lapuschkin et al 2019](https://arxiv.org/abs/1902.10178) compares Fisher vectors (an SVM trained on SIFT features, &[BiT](https://arxiv.org/abs/1912.11370#google) is one of a number of [scaling papers](https://en.wikipedia.org/wiki/Neural_scaling_law) showing much better representations & robustness & transfer with extremely large CNNs) to CNNs on PASCAL VOC again, finding the Fishers overfit by eg. classifying horses based on copyright watermarks while the CNN nevertheless classifies it based on the correct parts, although the CNN may succumb to a different dataset bias by classifying airplanes based on having backgrounds of skies[12](https://gwern.net/tank#fn12)); and I believe we have good reason to expect our CNNs to also work in the wild.
+
+Some real instances of dataset bias, more or less (most of these were caught by standard heldout datasets and arguably aren’t the ‘tank story’ at all):
+
+* a particularly appropriate example is the unsuccessful [WWII Russian anti-tank dog program](https://en.wikipedia.org/wiki/Anti-tank_dog#Deployment_by_the_Soviet_Union): a failure, among several reasons, because the dogs were trained on Russian tanks and sought _them_ out rather than the enemy German tanks because the dogs recognized either the fuel smell or fuel canisters (diesel vs gasoline)
+
+* [“The person concept in monkeys (_Cebus apella_)”](https://gwern.net/doc/psychology/1988-damato.pdf), D’Amato & Van Sant 1988
+
+* Google Photos in June 2015 11ya caused a social-media fuss over mislabeling African-Americans as gorillas; Google did not explain how the Photos app made that mistake but it is presumably using a CNN and an example of either dataset bias (many more Caucasian/Asian faces leading to better performance on them and continued poor performance everywhere else) and/or a mis-specified loss function (the CNN optimizing a standard classification loss and responding to class imbalance or objective color similarity by preferring to guess ‘gorilla’ rather than ‘human’ to minimize loss, despite what ought to be a greater penalty for mistakenly classifying a human as an animal/object rather than vice versa). A similar issue occurred with Flickr in May 2015 11ya.
+
+* [“Gender-From-Iris or Gender-From-Mascara?”](https://arxiv.org/abs/1702.01304), Kuehlkamp et al 2017
+
+* Gidi Shperber, [“What I’ve learned from Kaggle’s fisheries competition”](https://gidishperber.medium.com/what-ive-learned-from-kaggle-s-fisheries-competition-92342f9ca779) (2017-05-01): initial application of VGG ImageNet CNNs for transfer solved the fish photograph classification problem almost immediately, but failed on the submission validation set; fish categories could be predicted from the specific boat taking the photographs
+
+* [“Leakage in data mining: Formulation, detection, and avoidance”](https://pdfs.semanticscholar.org/829e/6bcabe9cc1bd334429215404a5adaefc7ade.pdf), Kaufman et al 2011 15ya discusses the general topic and mentions a few examples from KDD-Cup
+
+* [Dan Piponi](https://x.com/sigfpe/status/919995891502551042) (2017-10-16): “Real world example from work: hospitals specialise in different injuries so CNN for diagnosis used annotations on x-rays to ID hospital.”
+
+ * A more detailed examination of X-ray saliencies: [“Confounding variables can degrade generalization performance of radiological deep learning models”](https://arxiv.org/abs/1807.00431), Zech et al 2018 ([blog](https://jrzech.medium.com/what-are-radiological-deep-learning-models-actually-learning-f97a546c5b98))
+
+* [Thomas G. Dietterich](https://x.com/tdietterich/status/1154839042623594496):
+
+> We made exactly the same mistake in one of my projects on insect recognition. We photographed 54 classes of insects. Specimens had been collected, identified, and placed in vials. Vials were placed in boxes sorted by class. I hired student workers to photograph the specimens. Naturally they did this one box at a time; hence, one class at a time. Photos were taken in alcohol. Bubbles would form in the alcohol. Different bubbles on different days. The learned classifier was surprisingly good. But a saliency map revealed that it was reading the bubble patterns and ignoring the specimens. I was so embarrassed that I had made the oldest mistake in the book (even if it was apocryphal). Unbelievable. Lesson: always randomize even if you don’t know what you are controlling for!
+
+* a possible case is Wu & Zhang 2016, [“Automated Inference on Criminality using Face Images”](https://pdfs.semanticscholar.org/1cd3/57b675a659413e8abf2eafad2a463272a85f.pdf), attempt to use CNNs to classify standardized government ID photos of Chinese people by whether the person has been arrested, the source of the criminal IDs being government publications of wanted suspects vs ordinary peoples’ IDs collected online; the photos are repeatedly described as ID photos and implied to be uniform. The use of official government ID photos taken in advance of any crime would appear to eliminate one’s immediate objections about dataset bias—certainly ID photos would be distinct in many ways from ordinary cropped promotional headshots—and so the results seem strong.
+
+In response to [harsh criticism](https://www.callingbullshit.org/case_studies/case_study_criminal_machine_learning.html "Case Study") (some of which points are more relevant & likely than the others…), Wu & Zhang admit in their response ([“Responses to Critiques on Machine Learning of Criminality Perceptions (Addendum of arXiv:1611 415ya.04135)”](https://arxiv.org/abs/1611.04135)) that the dataset is not quite as implied:
+
+> All criminal ID photos are government issued, but not mug shots. To our best knowledge, they are normal government issued ID portraits like those for driver’s license in USA. In contrast, most of the noncriminal ID style photos are taken officially by some organizations (such as real estate companies, law firms, etc.) for their websites. We stress that they are not selfies.
+
+While there is no direct replication testing the Wu & Zhang 2016 results that I know of, the inherent considerable differences between the two classes, which are not homogenous at all, make me highly skeptical.
+
+* Possible: [Winkler et al 2019](https://gwern.net/doc/ai/nn/cnn/2019-winkler.pdf) examine a commercial CNN (“Moleanalyzer-Pro”; [Haenssle et al 2018](https://gwern.net/doc/ai/nn/cnn/2018-haenssle.pdf)) for skin cancer detection. Concerned by the fact that doctors sometimes use purple markers to highlight potentially-malignant skin cancers for easier examination, they compare before/after photographs of skin cancers which have been highlighted, and find that the purple highlighting increases the probability of being classified as malignant.
+
+However, it is unclear that this is a dataset bias problem, as the existing training datasets for skin cancer are realistic and already include purple marker samples[13](https://gwern.net/tank#fn13). The demonstrated manipulation may simply reflect the CNN using purple as a proxy for human concern, which is an informative signal and desirable if it improves classification performance in the real world on real medical cases. It is possible that the training datasets are in fact biased to some degree with too much/too little purple or that use of purple differs systematically across hospitals, and those would damage performance to some degree, but that is not demonstrated by their before/after comparison. Ideally, one would run a field trial to test the CNN’s performance as a whole by using it in various hospitals and then following up on all cases to determine benign or malignant; if the classification performance drops considerably from the original training, then that implies something (possibly the purple highlighting) has gone wrong.
+
+* Possible: [Esteva et al 2011](https://gwern.net/doc/ai/nn/2017-esteva.pdf) trains a skin cancer classifier; the final CNN performs well in independent test sets. The paper does not mention this problem but [media coverage reported](https://www.thedailybeast.com/why-doctors-arent-afraid-of-better-more-efficient-ai-diagnosing-cancer) that rulers in photographs served as unintentional features:
+
+> He and his colleagues had one such problem in their their study with rulers. When dermatologists are looking at a lesion that they think might be a tumor, they’ll break out a ruler—the type you might have used in grade school—to take an accurate measurement of its size. Dermatologists tend to do this only for lesions that are a cause for concern. So in the set of biopsy images, if an image had a ruler in it, the algorithm was more likely to call a tumor malignant, because the presence of a ruler correlated with an increased likelihood a lesion was cancerous. Unfortunately, as Novoa emphasizes, the algorithm doesn’t know why that correlation makes sense, so it could easily misinterpret a random ruler sighting as grounds to diagnose cancer.
+
+It’s unclear how they detected this problem or how they fixed it. And like Winkler et al 2019, it’s unclear if this was a problem which would reduce real-world performance (are dermatologists going to stop measuring worrisome lesions?).
+
+## [Should We Tell Stories We Know Aren’t True?](https://gwern.net/tank#should-we-tell-stories-we-know-arent-true "Link to section: § 'Should We Tell Stories We Know Aren’t True?'")
+
+So the NN tank story probably didn’t happen as described, but something somewhat like it _could_ have happened and things sort of like it could happen now, and it is (as proven by its history) a catchy story to warn students with—it’s not true but it’s [“truthy”](https://en.wikipedia.org/wiki/Truthiness). Should we still mention it to journalists or in blog posts or in discussions of AI risk, as a noble lie?
+
+I think not. In general, we should promote more epistemic rigor and higher standards in an area where there is already far too much impact of fictional stories (eg. the depressing inevitability of a _Terminator_ allusion in AI risk discussions). Nor do I consider the story particularly effective from a didactic perspective: relegating dataset bias to mythical stories does not inform the listener about how common or how serious dataset bias is, nor is it helpful for researchers investigating countermeasures and diagnostics—the LIME developers, for example, are not helped by stories about Russian tanks, but need real testcases to show that their interpretability tools work & would help machine learning developers diagnose & fix dataset bias.
+
+I also fear that telling the tank story tends to promote complacency and underestimation of the state-of-the-art by implying that NNs and AI in general are toy systems which are far from practicality & cannot work in the real world (particularly the story variants which date it relatively recently), or that such systems when they fail will fail in easily diagnosed, visible, sometimes amusing ways, ways which can be diagnosed by a human comparing the photos or applying some political reasoning to the outputs; but modern NNs are powerful, are often deployed to the real world despite the spectre of dataset bias, and do not fail in blatant ways—what we actually see with deep learning are far more concerning failure modes like “adversarial examples” which are quite as inscrutable as the neural nets themselves (or AlphaGo’s one misjudged move resulting in its only loss to Lee Sedol). Adversarial examples are particularly insidious as the NN will work flawlessly in all the normal settings and contexts, only to fail totally when exposed to a custom adversarial input. More importantly, dataset bias and failure to transfer tends to be a self-limiting problem, particularly when embedded in an ongoing system or reinforcement learning agent, since if the NN is making errors based on dataset bias, it will in effect be generating new counterexample datapoints for its next iteration.
+
+## [Alternative Examples](https://gwern.net/tank#alternative-examples "Link to section: § 'Alternative examples'")
+
+> There is nothing so useless as doing efficiently that which should not be done at all.
+>
+>
+> [Peter Drucker](https://en.wikipedia.org/wiki/Peter_Drucker)
+
+The more troubling errors are ones where the goal itself, the reward function, is mis-specified or wrong or harmful.
+
+I am less worried about algorithms learning to do poorly the right thing for the wrong reasons because humans are sloppy in their data collection than I am about them learning to do well the wrong thing for the right reasons despite perfect data collection. Because RL rewards agents for doing the right thing, but not for doing the right thing for the right reasons.
+
+With errors or inefficiencies in the rest of the algorithm, training may simply be slower, or there may be more local optima which may temporarily trap the agent, or its final performance may be worse than it could be; these are bad things, but normal enough. But when the _reward function_ is wrong, the better the algorithm is, the more useless (or dangerous) it becomes at [pursuing the wrong objective](https://arxiv.org/abs/2105.14111) because [the reward hacking scales](https://arxiv.org/abs/2210.10760#openai), and this may [happen abruptly](https://arxiv.org/abs/2201.03544)! Using losses which have little to do with the true human utility function or decision context is far more common than serious dataset bias: people think about where their data is coming from, but they tend not to think about what the consequences of wrong classifications are.
+
+Such reward function problems cannot be fixed by collecting any amount of data or making data more representative of the real world, and for large-scale systems will be more harmful. And it can be hard to avoid errors: sure, in hindsight, once you’ve seen the converged reward hack, you can laugh and say “of course that particular bit of reward-shaping was wrong, how obvious now!”—but only in hindsight. Before then, the absence of the hack is just common sense: we are [blinded by our knowledge](https://gwern.net/unseeing), which is a burden optimization processes do not share.
+
+Unfortunately, I know of no particularly comprehensive lists of examples of mis-specified rewards/unexpectedly bad proxy objective functions/“reward hacking”/“wireheading”/“perverse instantiation”[14](https://gwern.net/tank#fn14) beyond [“The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities”, Lehman et al 2018](https://arxiv.org/abs/1803.03453); perhaps people can make suggestions, but a few examples I have found or recall include:
+
+* [linear programming](https://en.wikipedia.org/wiki/Linear_programming) optimization for nutritious (not necessarily palatable!) low-cost diets: [“The cost of subsistence”](https://gwern.net/doc/statistics/decision/stigler-diet/1945-stigler.pdf), Stigler 1945 81ya, [“The Diet Problem”](https://gwern.net/doc/statistics/decision/stigler-diet/1990-dantzig.pdf), Dantzig 1990 36ya, [“Stigler’s Diet Problem Revisited”](https://gwern.net/doc/statistics/decision/stigler-diet/2001-garille.pdf), Garille & Gass 2001
+
+ * SMT/SAT solvers are likewise infamous for finding strictly valid yet surprising or useless solutions, which perversity is exactly what makes them so invaluable in security/formal-verification research (for example, in RISC-V verification of exceptions, discovering that it can trigger an exception by turning on a [debug unit & setting a breakpoint](https://x.com/oe1cxw/status/957409526940094464), or using an obscure [memory mode setting](https://x.com/oe1cxw/status/958704985495175169))
+
+* boat race reward-shaping for picking up targets results in not finish race at all but going in circles to hit targets: [“Faulty Reward Functions in the Wild”](https://openai.com/research/faulty-reward-functions), OpenAI
+
+* [a PPO agent](https://www.reddit.com/r/MachineLearning/comments/18eh2hb/p_the_power_of_reinforcement_learning_look_how/) for [_Ultimate Mortal Kombat 3_](https://en.wikipedia.org/wiki/Ultimate_Mortal_Kombat_3) learned, to get past a challenging double-match it couldn’t beat normally, to whittle down the first enemy and then simply pace back & forth until the match time ran out with it technically winning & never facing the second enemy at all
+
+* a classic 3D robot-arm NN agent, in a somewhat unusual setup where the evaluator/reward function is another NN trained to predict human evaluations, learns to move the arm to a position which _looks_ like it is positioned at the goal but is actually just in between the ‘camera’ and the goal: [“Learning from Human Preferences”](https://openai.com/research/learning-from-human-preferences), Christiano et al 2017, OpenAI
+
+* reward-shaping a bicycle agent for not falling over & making progress towards a goal point (but not punishing for moving away) leads it to learn to circle around the goal in a physically stable loop: [“Learning to Drive a Bicycle using Reinforcement Learning and Shaping”](https://pdfs.semanticscholar.org/10ba/d197f1c1115005a56973b8326e5f7fc1031c.pdf), Randlov & Alstrom 1998 28ya; similar difficulties in avoiding pathological optimization were experienced by [Cook 2004](https://gwern.net/doc/reinforcement-learning/model-free/2004-cook.pdf) ([video](https://gwern.net/doc/reinforcement-learning/2004-cook-twoneuronbicycle.avi) of policy-iteration learning to spin handle-bar to stay upright).
+
+* reward-shaping a soccer robot for touching the ball caused it to learn to get to the ball and “vibrate” touching it as fast as possible: David Andre & Astro Teller in Ng et al 1999 27ya, [“Policy invariance under reward transformations: theory and application to reward shaping”](http://luthuli.cs.uiuc.edu/~daf/courses/games/AIpapers/ng99policy.pdf)
+
+* environments involving walking/running/movement and rewarding movement seem to often result in the agents learning to fall over as a local optima of speed generation, possibly bouncing around or moving at hyperspeed by exploiting any failure to conserve all quantities like energy.
+
+For example, Sims notes in one paper ([Sims 1994](https://www.karlsims.com/papers/siggraph94.pdf)) that “It is important that the physical simulation be reasonably accurate when optimizing for creatures that can move within it. Any bugs that allow energy leaks from non-conservation, or even round-off errors, will inevitably be discovered and exploited by the evolving creatures…speed is used as the selection criteria, but the vertical component of velocity is ignored. For land environments, it can be necessary to prevent creatures from generating high velocities by simply falling over.”; and if the [conservation-of-momentum](https://en.wikipedia.org/wiki/Conservation-of-momentum) was not _exact_, creatures could exploit it by evolving ‘paddles’ to paddle themselves at high velocity.
+
+Sims mentions round-off errors as a possibility, and apparently this happened: according to [Danny Hillis](https://en.wikipedia.org/wiki/Danny_Hillis), “early walking machines evolved on the Connection Machine [[CM-5](https://en.wikipedia.org/wiki/Connection_Machine#Designs)] took advantage of an obscure round-off error in the floating-point unit that the human programmers did not even know existed.” ([Taylor & Massey 2001](https://gwern.net/doc/ai/2001-taylor.pdf#page=6) attempted to reimplement Sims’s work, and had to implement a large range of checks on their creatures because they kept breaking the physics engine; [Ha 2018](https://arxiv.org/abs/1810.03779#google) encountered similar pathological behavior, like [falling over](https://x.com/hardmaru/status/1050193431857774592 "Ha 2026").)
+
+Evolving similar exploitation of rounding-off has been done by OpenAI in 2017 to turn [apparently linear neural networks into nonlinear ones](https://openai.com/research/nonlinear-computation-in-deep-linear-networks); [Jaderberg et al 2019](https://gwern.net/doc/reinforcement-learning/exploration/2019-jaderberg.pdf#deepmind)[appears to have had](https://www.science.org/content/article/artificial-intelligence-learns-teamwork-deadly-game-capture-flag "Artificial intelligence learns teamwork in a deadly game of capture the flag") a similar momentum bug in its _Quake_ simulator: “In one test, the bots invented a completely novel strategy, exploiting a bug that let teammates give each other a speed boost by shooting them in the back.”
+
+* [Popov et al 2017](https://arxiv.org/abs/1704.03073#deepmind), training a simulated robot gripper arm to stack objects like Legos, included reward shaping; pathologies included “hovering” and for a reward-shaping for lifting the bottom face of the top block upwards, DDPG learned to knock the blocks over, thereby (temporarily) elevating the bottom of the top block and receiving the reward:
+
+> We consider three different composite rewards in additional to the original sparse task reward:
+>
+>
+> 1. **_Grasp shaping_**: _Grasp brick 1_ and _Stack brick 1_, i.e.the agent receives a reward of 0.25 when the brick 1 has been grasped and a reward of 1.0 after completion of the full task.
+>
+> 2. **_Reach and grasp shaping_**: _Reach brick 1_, _Grasp brick 1_ and _Stack brick 1_, i.e.the agent receives a reward of 0.125 when being close to brick 1, a reward of 0.25 when brick 1 has been grasped, and a reward of 1.0 after completion of the full task.
+>
+> 3. **_Full composite shaping_**: the sparse reward components as before in combination with the distance-based smoothly varying components.
+>
+>
+>
+> Figure 5 shows the results of learning with the above reward functions (blue traces). The figure makes clear that learning with the sparse reward only does not succeed for the full task. Introducing an intermediate reward for grasping allows the agent to learn to grasp but learning is very slow. The time to successful grasping can be substantially reduced by giving a distance based reward component for reaching to the first brick, but learning does not progress beyond grasping. Only with an additional intermediate reward component as in continuous reach, grasp, stack the full task can be solved.
+>
+>
+> Although the above reward functions are specific to the particular task, we expect that the idea of a composite reward function can be applied to many other tasks thus allowing learning for to succeed even for challenging problems. Nevertheless, great care must be taken when defining the reward function. We encountered several unexpected failure cases while designing the reward function components: eg. reach and grasp components leading to a grasp unsuitable for stacking, agent not stacking the bricks because it will stop receiving the grasping reward before it receives reward for stacking and the agent flips the brick because it gets a grasping reward calculated with the wrong reference point on the brick. We show examples of these [in the video](https://www.youtube.com/watch?v=8QnD8ZM0YCo).
+
+* RL agents using learned model-based planning paradigms such as the model predictive control are noted to have issues with the planner essentially exploiting the learned model by choosing a plan going through the worst-modeled parts of the environment and producing unrealistic plans using teleportation, eg. Mishra et al 2017, [“Prediction and Control with Temporal Segment Models”](https://arxiv.org/pdf/1703.04070.pdf#page=3) who note:
+
+> If we attempt to solve the optimization problem as posed in (2), the solution will often attempt to apply action sequences outside the manifold where the dynamics model is valid: these actions come from a very different distribution than the action distribution of the training data. This can be problematic: the optimization may find actions that achieve high rewards under the model (by exploiting it in a regime where it is invalid) but that do not accomplish the goal when they are executed in the real environment.
+>
+>
+> …Next, we compare our method to the baselines on trajectory and policy optimization. Of interest is both the actual reward achieved in the environment, and the difference between the true reward and the expected reward under the model. If a control algorithm exploits the model to predict unrealistic behavior, then the latter will be large. We consider two tasks….Under each model, the optimization finds actions that achieve similar model-predicted rewards, but the baselines suffer from large discrepancies between model prediction and the true dynamics. Qualitatively, we notice that, on the pushing task, the optimization exploits the LSTM and one-step models to predict unrealistic state trajectories, such as the object moving without being touched or the arm passing through the object instead of colliding with it. Our model consistently performs better, and, with a latent action prior, the true execution closely matches the model’s prediction. When it makes inaccurate predictions, it respects physical invariants, such as objects staying still unless they are touched, or not penetrating each other when they collide
+
+This is similar to Sims’s issues, or current issues in training walking or running agents in environments like MuJoCo where it is easy for them to learn odd gaits like hopping ([Lillicrap et al 2016](https://arxiv.org/abs/1509.02971#deepmind) adds extra penalties for impacts to try to avoid this) or jumping (eg. [Stelmaszczyk’s](https://blog.mlreview.com/our-nips-2017-learning-to-run-approach-b80a295d3bb5 "Our 'NIPS 2017: Learning to Run' approach") attempts at reward shaping a skeleton agent) or flailing around wildly ([Heess et al 2017](https://arxiv.org/abs/1707.02286#deepmind) add random pushes/shoves to the environment to try to make the agent learn more generalizable policies) which may work quite well in the specific simulation but not elsewhere. (To some degree this is beneficial for driving exploration in poorly-understood regions, so it’s not all bad.) [Christine Barron](https://connect.unity.com/p/pancake-bot), working on a pancake-cooking robot-arm simulation, ran into reward-shaping problems: rewarding for each timestep without the pancake on the floor teaches the agent to hurl the pancake into the air as hard as possible; and for the passing-the-butter agent, rewarding for getting close to the goal produces the same close-approach-but-avoidance behavior to maximize reward.
+
+* A curious lexicographic-preference raw-RAM NES AI algorithm learns to pause the game to never lose at Tetris: Murphy 2013 13ya, [“The First Level of Super Mario Bros. is Easy with Lexicographic Orderings and Time Travel… after that it gets a little tricky”](http://tom7.org/mario/ "learnfun and playfun: A general technique for automating NES games")
+
+ * [Peter Whidden](https://www.youtube.com/watch?v=DcYLT37ImBY), using novelty rewards on _Pokemon_, observed that some of his reward-shaping backfired on him, including a penalty for losing battles—which resulted in simply refusing to press the ‘continue’ button after losing (as well as learning a RNG hack to catch a Pokemon on the first try)
+
+* RL agent in Udacity self-driving car rewarded for speed learns to spin in circles: [Matt Kelcey](https://x.com/mat_kelcey/status/886101319559335936 "mat_kelcey 2026")
+
+* NASA Mars mission planning, optimizing food/water/electricity consumption for total man-days survival, yields an optimal plan of killing 2/3 crew & keep survivor alive as long as possible: [iand675](https://lobste.rs/s/1d7whd/tales_from_trenches_ai_disaster_stories#c_le6tsr)
+
+* Doug Lenat’s [Eurisko](https://en.wikipedia.org/wiki/Eurisko) famously had issues with “parasitic” heuristics, due to the self-modifying ability, edited important results to claim credit and be rewarded, part of a class of such wireheading heuristics that Lenat made the Eurisko core unmodifiable: [“EURISKO: A program that learns new heuristics and domain concepts: the nature of heuristics III: program design and results”](https://pdfs.semanticscholar.org/24c7/4c798100d69555ace06145bc1ba4fd6df35d.pdf), Lenat 1983 43ya (pg90)
+
+* genetic algorithms for image classification evolves timing-attack to infer image labels based on hard drive storage location: https://news.ycombinator.com/item?id=6269114
+
+* training a dog to roll over results in [slamming against the wall](https://www.lesswrong.com/posts/5o3CxyvZ2XKawRB5w/machine-learning-and-unintended-consequences?commentId=tKdjcCZAtbE6vJq4v); dolphins rewarded for finding trash & dead seagulls in their tank learned to [manufacture trash & hunt living seagulls](https://www.theguardian.com/science/2003/jul/03/research.science "Why dolphins are deep thinkers: The more we study dolphins, the brighter they turn out to be") for more rewards
+
+* circuit design with genetic/evolutionary computation:
+
+ * an attempt to evolve a circuit on an FPGA, to discriminate audio tones of 1kHz & 10kHz without using any timing elements, evolved a design which depended on disconnected circuits in order to work: [“An evolved circuit, intrinsic in silicon, entwined with physics”](https://gwern.net/doc/ai/1997-thompson.pdf), Thompson 1996 30ya. (“Possible mechanisms include interactions through the power-supply wiring, or electromagnetic coupling.” The evolved circuit is sensitive to room temperature variations 23–43C, only working perfectly over the 10C range of room temperature it was exposed to during the 2 weeks of evolution. It is also sensitive to the exact location on the FPGA, degrading when shifted to a new position; further finetuning evolution fixes that, but then is vulnerable when shifted back to the original location.)
+
+ * an attempt to evolve an oscillator or a timer wound up evolving a circuit which picked up radio signals from the lab PCs (although since the circuits _did_ work at their assigned function as the human intended, should we consider this a case of ‘dataset bias’ where the ‘dataset’ is the local lab environment?): [“The evolved radio and its implications for modeling the evolution of novel sensors”](https://pdfs.semanticscholar.org/0adf/aaeebbf36f34ac97770adc2f52619a5d45c6.pdf), Jon Bird and Paul Layzell 2002
+
+* training a “minitaur” bot in simulation to carry a ball or duck on its back, CMA-ES discovers [it can drop the ball into a leg joint and then wiggle across the floor](https://blog.otoro.net/2017/11/12/evolving-stable-strategies/) without the ball ever dropping
+
+* [CycleGAN](https://arxiv.org/abs/1703.10593#bair), a cooperative GAN architecture for converting images from one genre to another (eg. horses⟺zebras), has a loss function that rewards accurate reconstruction of images from its transformed version; CycleGAN turns out to partially solve the task by, in addition to the cross-domain analogies it learns, steganographically hiding autoencoder-style data about the original image invisibly inside the transformed image to assist the reconstruction of details ([Chu et al 2017](https://arxiv.org/abs/1712.02950))
+
+A researcher in 2020 working on art colorization told me of an interesting similar behavior: his automatically-grayscaled images were failing to train the NN well, and he concluded that this was because grayscaling a color image produces many shades of gray in a way that human artists do not, and that the formula used by OpenCV for RGB → grayscale permits only a few colors to map onto any given shade of gray, enabling accurate guessing of the original color! Such issues might require learning a grayscaler, similar to superresolution needing learned downscalers ([Sun & Chen 2019](https://arxiv.org/abs/1907.12904)).
+
+* the ROUGE machine translation metric, based on matching sub-phrases, is typically used with RL techniques since it is a non-differentiable loss; [Salesforce](https://www.salesforce.com/products/einstein/ai-research/tl-dr-reinforced-model-abstractive-summarization/) ([Paulus et al 2017](https://arxiv.org/abs/1705.04304#salesforce)) notes that an effort at a ROUGE-only summarization NN produced largely gibberish summaries, and had to add in another loss function to get high-quality results
+
+* Alex Irpan [writes of 3 anecdotes](https://www.alexirpan.com/2018/02/14/rl-hard.html):
+
+> In talks with other RL researchers, I’ve heard several anecdotes about the novel behavior they’ve seen from improperly defined rewards.
+>
+>
+> * A coworker is teaching an agent to navigate a room. The episode terminates if the agent walks out of bounds. He didn’t add any penalty if the episode terminates this way. The final policy learned to be suicidal, because negative reward was plentiful, positive reward was too hard to achieve, and a quick death ending in 0 reward was preferable to a long life that risked negative reward.
+>
+> * A friend is training a simulated robot arm to reach towards a point above a table. It turns out the point was defined _with respect to the table_, and the table wasn’t anchored to anything. The policy learned to slam the table really hard, making the table fall over, which moved the target point too. The target point _just so happened_ to fall next to the end of the arm.
+>
+> * A researcher gives a talk about using RL to train a simulated robot hand to pick up a hammer and hammer in a nail. Initially, the reward was defined by how far the nail was pushed into the hole. Instead of picking up the hammer, the robot used its own limbs to punch the nail in. So, they added a reward term to encourage picking up the hammer, and retrained the policy. They got the policy to pick up the hammer…but then it threw the hammer at the nail instead of actually using it.
+>
+>
+>
+> Admittedly, these are all secondhand accounts, and I haven’t seen videos of any of these behaviors. However, none of it sounds implausible to me. I’ve been burned by RL too many times to believe otherwise…I’ve taken to imagining deep RL as a demon that’s deliberately misinterpreting your reward and actively searching for the laziest possible local optima. It’s a bit ridiculous, but I’ve found it’s actually a productive mindset to have.
+
+* [Chrabaszcz et al 2018](https://arxiv.org/abs/1802.08842): an evolutionary strategies RL in the ALE game [_Q*bert_](https://en.wikipedia.org/wiki/Q*bert) finds that it can steadily earn points by committing ‘suicide’ to lure an enemy into following it; more interestingly, it also discovers what appears to be a previously unknown bug where a sequence of jumps will, semi-randomly, permanently force the game into a state where the entire level begins flashing and the score increases rapidly & indefinitely until the game is reset ([video](https://www.youtube.com/watch?v=meE5aaRJ0Zs?t=14s))
+
+* [Lapuschkin et al 2019](https://arxiv.org/abs/1902.10178) notes a borderline case in the ALE pinball game where the ‘nudge’ ability is unlimited (unlike all real pinball machines) and a DQN can learn to score arbitrarily by the ball budging over a switch repeatedly:
+
+> The second showcase example studies neural network models (see Figure 5 for the network architecture) trained to play Atari games, here Pinball. As shown in [5], the DNN achieves excellent results beyond human performance. Like for the previous example, we construct LRP heatmaps to visualize the DNN’s decision behavior in terms of pixels of the pinball game. Interestingly, after extensive training, the heatmaps become focused on few pixels representing high-scoring switches and loose track of the flippers. A subsequent inspection of the games in which these particular LRP heatmaps occur, reveals that DNN agent firstly moves the ball into the vicinity of a high-scoring switch without using the flippers at all, then, secondly, “nudges” the virtual pinball table such that the ball infinitely triggers the switch by passing over it back and forth, without causing a tilt of the pinball table (see Figure 2b and Figure 6 for the heatmaps showing this point, and also Supplementary Video 1). Here, the model has learned to abuse the “nudging” threshold implemented through the tilting mechanism in the Atari Pinball software. From a pure game scoring perspective, it is indeed a rational choice to exploit any game mechanism that is available. In a real pinball game, however, the player would go likely bust since the pinball machinery is programmed to tilt after a few strong movements of the whole physical machine.
+
+* [“Trial without Error: Towards Safe Reinforcement Learning via Human Intervention”](https://arxiv.org/abs/1707.05173), Saunders et al 2017; the [blog writeup](https://owainevans.github.io/blog/hirl_blog.html) notes:
+
+> The Road Runner results are especially interesting. Our goal is to have the agent learn to play Road Runner without losing a single life on Level 1 of the game. Deep RL agents are known to discover a ‘Score Exploit’ in Road Runner: they learn to intentionally kill themselves in a way that (paradoxically) earns greater reward. Dying at a precise time causes the agent to repeat part of Level 1, where it earns more points than on Level 2. This is a local optimum in policy space that a human gamer would never be stuck in.
+>
+>
+> Ideally, our Blocker would prevent all deaths on Level 1 and hence eliminate the Score Exploit. However, through random exploration the agent may hit upon ways of dying that “fool” our Blocker (because they look different from examples in its training set) and hence learn a new version of the Score Exploit. In other words, the agent is implicitly performing a random search for adversarial examples for our Blocker (which is a convolutional neural net)…In Road Runner we did not achieve zero catastrophes but were able to reduce the rate of deaths per frame from 0.005 (with no human oversight at all) to 0.0001.
+
+* [Toromanoff et al 2019](https://arxiv.org/abs/1908.04683) note various bugs in the ALE games, but also a new infinite loop for maximizing scores:
+
+> Finally, we discovered that on some games the actual optimal strategy is by doing a loop over and over giving a small amount of reward. In _Elevator Action_ the agent learn to stay at the first floor and kill over and over the first enemy. This behavior cannot be seen as an actual issue as the agent is basically optimizing score but this is definitely not the intended goal. A human player would never perform this way.
+
+* [Le Paine et al 2019’s](https://arxiv.org/abs/1909.01387#deepmind)[R2D3](https://deepmind.google/discover/blog/making-efficient-use-of-demonstrations-to-solve-hard-exploration-problems/) writeup notes:
+
+> _Wall Sensor Stack_: The original Wall Sensor Stack environment had a bug that the R2D3 agent was able to exploit. We fixed the bug and verified the agent can learn the proper stacking behavior.
+>
+>
+> …Another desirable property of our approach is that our agents are able to learn to outperform the demonstrators, and in some cases even to discover strategies that the demonstrators were not aware of. In one of our tasks the agent is able to discover and exploit a bug in the environment in spite of all the demonstrators completing the task in the intended way…R2D3 performed better than our average human demonstrator on Baseball, Drawbridge, Navigate Cubes and the Wall Sensor tasks. The behavior on Wall Sensor Stack in particular is quite interesting. On this task R2D3 found a completely different strategy than the human demonstrators by exploiting a bug in the implementation of the environment. The intended strategy for this task is to stack two blocks on top of each other so that one of them can remain in contact with a wall mounted sensor, and this is the strategy employed by the demonstrators. However, due to a bug in the environment the strategy learned by R2D3 was to trick the sensor into remaining active even when it is not in contact with the key by pressing the key against it in a precise way.
+
+* [“Emergent Tool Use From Multi-Agent Autocurricula”](https://arxiv.org/abs/1909.07528#openai), Baker et al 2019:
+
+> We originally believed defending against ramp use would be the last stage of emergence in this environment; however, we were surprised to find that yet two more qualitatively new strategies emerged. After 380 million total episodes of training, the seekers learn to bring a box to the edge of the play area where the hiders have locked the ramps. The seekers then jump on top of the box and _surf_ it to the hiders’ shelter; this is possible because the environment allows agents to move together with the box regardless of whether they are on the ground or not. In response, the hiders learn to lock all of the boxes in place before building their shelter.
+
+ * [OA blog post](https://openai.com/research/emergent-tool-use#surprisingbehaviors)
+
+* Ziegler et al 2019: fine-tune trained an English text generation model based on human ratings for preference-learning; they provide a curious example of a reward specification bug. Here, the reward was accidentally negated and a new run began overnight while the devs slept; this reversal, rather than resulting in nonsense, resulted in (literally) perversely coherent behavior of emitting obscenities to maximize the new score:
+
+ * [blog](https://openai.com/index/fine-tuning-gpt-2/#_5VLCK1KHEBCzRHpnOQQ0Lj)
+
+* [Custard Smingleigh](https://x.com/smingleigh/status/1060325665671692288):
+
+> I hooked a neural network up to my [Roomba](https://en.wikipedia.org/wiki/Roomba) 650. I wanted it to learn to navigate without bumping into things, so I set up a reward scheme to encourage speed and discourage hitting the bumper sensors.
+>
+>
+> It learned to drive backwards, because there are no bumpers on the back.
+
+## [See Also](https://gwern.net/tank#see-also "Link to section: § 'See Also'")
+
+* [Why Tool AIs Want to Be Agent AIs](https://gwern.net/tool-ai)
+
+* [Surprisingly Turing-Complete](https://gwern.net/turing-complete)
+
+* [Feynman’s Maze Story](https://gwern.net/maze)
+
+## [External Links](https://gwern.net/tank#external-links "Link to section: § 'External Links'")
+
+* [“Concrete Problems in AI Safety”](https://arxiv.org/abs/1606.06565), Amodei et al 2016
+
+* [“Edge instantiation”](https://arbital.com/p/edge_instantiation/)/[“Nearest unblocked strategy”](https://arbital.com/p/nearest_unblocked/)
+
+* [“Adversarial Examples Are Not Bugs, They Are Features”](https://arxiv.org/abs/1905.02175), Ilyas et al 2019
+
+* [“Specification gaming: the flip side of AI ingenuity”](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/), Krakovna et al 2020
+
+* [“Were Armed Kangaroos Added to a Military Combat Simulation Program?”](https://www.snopes.com/fact-check/shoot-me-kangaroo-down-sport/ "Were Armed Kangaroos Added to a Military Combat Simulation Program?") (no)
+
+* **Discussion**: [/r/machinelearning](https://www.reddit.com/r/MachineLearning/comments/76qua8/d_that_urban_legend_about_neural_nets_tanks/), HN: [1](https://news.ycombinator.com/item?id=15485538), [2](https://news.ycombinator.com/item?id=36416895)
+
+* * *
+
+[](https://gwern.net/tank#footnotes "Link to section: § ‘Footnotes’")
+1. [](https://gwern.net/tank#fn1 "Link to footnote 1")
+The paper in question discusses general questions of necessary resolution, computing requirements, optics, necessary error rates, and algorithms, but doesn’t describe any implemented systems, much less experiences which resemble the tank story.[](https://gwern.net/tank#fnref1)
+
+2. [](https://gwern.net/tank#fn2 "Link to footnote 2")
+Another interesting detail from Harley et al 1962 64ya about their tank study: in discussing designing their computer ‘simulation’ of their quasi-NN algorithms, their description of the photographs on pg133 makes it sound as if the dataset was constructed from the _same_ photographs by using large-scale aerial footage and then cropping out the small squares with tanks and then corresponding small squares without tanks—so they only had to process one set of photographs, and the resulting tank/non-tank samples are inherently matched on date, weather, time of day, lighting, general location, roll of film, camera, and photographer. If true, that would make almost all the various suggested tank problem shortcuts impossible, and would be further evidence that Kanal’s project was not & could not have been a true origin of the tank story (although if it was simply _misunderstood_ and erroneously critiqued, then it could be a tiny kernel of truth from which the urban legend sprang).[](https://gwern.net/tank#fnref2)
+
+3. [](https://gwern.net/tank#fn3 "Link to footnote 3")
+Fredkin was quite a character; see [Hagar 2016](https://gwern.net/doc/cs/algorithm/information/2016-hagar.pdf). He was highly opinionated & critical, and very much an eccentric ‘outsider’ scientist & entrepreneur—one can read through his [oral history](https://archive.computerhistory.org/resources/access/text/2013/05/102630504-05-01-acc.pdf)& see that he is not reluctant to claim vindication or say ‘I told you so’, nor was he ever reluctant to tell someone that they were wrong or their research was bogus. So his story is credible, but ends at the criticism: he would surely have told Yudkowsky he had been proven right or wrong if he had ever been.[](https://gwern.net/tank#fnref3)
+
+4. [](https://gwern.net/tank#fn4 "Link to footnote 4")
+This seems entirely reasonable to me, given that hardly any AI research existed at that point. While it’s unclear what results were accomplished immediately thanks to the 1956 70ya workshop, many of the attendees would make major discoveries in AI. Attendee [Ray Solomonoff’s](https://en.wikipedia.org/wiki/Ray_Solomonoff) wife, Grace Solomonoff ([“Ray Solomonoff and the Dartmouth Summer Research Project in Artificial Intelligence, 1956”](https://raysolomonoff.com/dartmouth/dartray.pdf), 2016) describes the workshop as having vivid discussions but was compromised by getting only half its funding (so it didn’t last the summer) and attendees showing up sporadically & for short times (“Many participants only showed up for a day or even less.”); no agreement was reached on a specific project to try to tackle, although Solomonoff did write a paper there he considered important.[](https://gwern.net/tank#fnref4)
+
+5. [](https://gwern.net/tank#fn5 "Link to footnote 5")
+One commenter observes that the NN tank story and ilk appears to almost always be told about neural networks, and wonders why when dataset bias ought to be just as much a problem for other statistical/machine-learning methods like decision trees, which are capable of learning complex nonlinear problems. I could note that these anecdotes also get routinely told about genetic algorithms & evolutionary methods, so it’s not purely neural, and it might be that NNs are victims of their own success: particularly as of 2017, NNs are so powerful & flexible in some areas (like computer vision) there is little competition, and so any horror stories will probably involve NNs.[](https://gwern.net/tank#fnref5)
+
+6. [](https://gwern.net/tank#fn6 "Link to footnote 6")
+Here, the number of photographs and exactly how they were divided into training/validation sets is an oddly specific detail. This is reminiscent of religions or novels, where originally sparse and undetailed stories become elaborated and ever more detailed, with striking details added to catch the imagination. For example, the [Three Magi](https://en.wikipedia.org/wiki/Biblical_Magi) in the Christian Gospels are unnamed, but have been given by later Christians extensive fictional biographies of names ([“Names for the Nameless in the New Testament”](https://gwern.net/doc/history/1980-metzger.pdf); one of [many given names](https://en.wikipedia.org/wiki/List_of_names_for_the_biblical_nameless)), symbolism, kingdoms, contemporary successors/descendants, martyrdoms & locations of remains…[](https://gwern.net/tank#fnref6)
+
+7. [](https://gwern.net/tank#fn7 "Link to footnote 7")
+Something similar happened to [Woody Bledsoe & the best early facial recognition system](https://www.wired.com/story/secret-history-facial-recognition/); the military is not known for its efficiency or brilliance at R&D (as Fredkin’s [own autobiography](https://archive.computerhistory.org/resources/access/text/2013/05/102630504-05-01-acc.pdf#page=27) shows), and doubtless classification & other exigencies have strangled many promising projects in their cradle.
+
+One is struck, when reading through connectionist history, by the sheer level of contingency. Despite superficially seeming large & well-funded, trivial setbacks seem to compound and result in delays that fed on themselves, blocking results that (with the benefit of abundant hindsight) could have been achieved decades before.
+
+Besides Kanal & Randall or Woody Bledsoe, we can note that Alan Turing & John von Neumann &[Frank Rosenblatt](https://en.wikipedia.org/wiki/Frank_Rosenblatt) died unusually young while still actively involved in early AI & connectionism (accident/suicide, cancer, & boating accident respectively), Claude Shannon’s career was neutered by [tenure & perfectionism](https://gwern.net/review/book#shannon-late-career), and that [Walter Pitts’s](https://en.wikipedia.org/wiki/Walter_Pitts) (of the McCulloch-Pitts neuron) career was [destroyed by a false rape/seduction accusation](https://web.archive.org/web/20220927022638/https://nautil.us/the-man-who-tried-to-redeem-the-world-with-logic-235253/) (leading to destroying all his unpublished work & his death by alcoholism). Others left the field for greener pastures when it became clear there was no immediate payoff, such as [Bill Highleyman](https://www.argmin.net/p/revisiting-highleymans-data "Revisiting Highleyman's Data") or whole labs when they failed to crack the puzzle of how to train _multiple_ layer neural networks instead of single-layer perceptrons. (See [Olazaran 1993](https://gwern.net/doc/ai/nn/1993-olazaran.pdf) which makes the case that Minsky’s infamous _Perceptrons_ book was merely an obituary & the death was a decade of failure to train networks better than perceptrons, whose inherent limitations were already well-known.) Early DARPA support was shocked by the [Mansfield amendments](https://en.wikipedia.org/wiki/Mike_Mansfield#Mansfield_Amendments), and then later there was the [Lighthill Report](https://en.wikipedia.org/wiki/Lighthill_report)—both [“AI winters”](https://en.wikipedia.org/wiki/AI_winter) had much more to do with the failures of ‘GOFAI’ like expert systems & ultra-fragile demos like [SHRDLU](https://gwern.net/doc/ai/1991-winograd.pdf#page=7), but as a struggling niche, connectionism probably suffered much more than GOFAI did. We could also point to the surprising slowness of backpropagation to be applied to NNs—it seems so obvious to us now, and it’s often described as ‘simple’ or ‘just the chain rule’, yet despite [publications going back to the 1960s](https://people.idsia.ch/~juergen/who-invented-backpropagation.html) (like [Kelley 1960](https://gwern.net/doc/statistics/decision/1960-kelley.pdf)), connectionists didn’t learn their fundamental problem had been solved until [Rumelhart](https://en.wikipedia.org/wiki/David_Rumelhart)/PDP in the late 1980s (which could’ve been done easily in the 1970s).
+
+Further, when we look at DL scaling research post-2010, often scaling successes seem to come down to a single opinionated researcher far out of the mainstream who insists on spending far more GPU-time than any of their colleagues consider reasonable: Schmidhuber/Alex Krizhevsky/Ilya Sutskever, Dario Amodei & Paul Christiano etc.[](https://gwern.net/tank#fnref7)
+
+8. [](https://gwern.net/tank#fn8 "Link to footnote 8")
+One memorable example of this for me was when the Edward Snowden NSA leaks began.
+
+Surely, given previous instances like differential cryptanalysis or public-key cryptography, the NSA had any number of amazing technologies and moon math beyond the ken of the rest of us? I read many of the presentations with great interest, particularly about how they searched for individuals or data—cutting edge deep neural networks? Evolutionary algorithms? Even more exotic techniques unheard of by mere civilians? Nope—regexps, linear models, and random forests. Practical but boring. Nor did any major cryptographic breakthroughs become exposed via Snowden.
+
+Overall, the NSA corpus indicates that they had the abilities you would expect from a large group of patient programmers with no ethics and given a budget of billions of dollars to spend on a mission whose motto was “hack the planet” using a comprehensive set of methods ranging from physical breakins & bugs, theft of private keys, bribery, large-scale telecommunications tapping, implanting backdoors, purchase & discovery of unpatched vulnerabilities, & standards process subversion. Highly effective in the aggregate but little that people hadn’t expected or long speculated about in the abstract—merely carried out on a scale that people could not believe in their guts until a massive leak.[](https://gwern.net/tank#fnref8)
+
+9. [](https://gwern.net/tank#fn9 "Link to footnote 9")
+Although there are occasional exceptions where a data augmentation _doesn’t_ preserve important semantics: you wouldn’t want to use horizontal flips with street signs.[](https://gwern.net/tank#fnref9)
+
+10. [](https://gwern.net/tank#fn10 "Link to footnote 10")
+It amuses me to note when websites or tools are clearly using ImageNet CNNs, because they assume ImageNet categories or provide annotations in their metadata, or because they exhibit uncannily good recognition of dogs. Sometimes CNNs are much better than they are given credit for being and they are _assumed_ by commenters to fail on problems they actually succeed on; for example, some meme images have circulated claiming that CNNs can’t distinguish fried chickens from [Labradoodle](https://en.wikipedia.org/wiki/Labradoodle) dogs, chihuahuas from muffins, or sleeping dogs from bagels—but as amusing as the image-sets are, [Miles Brundage](https://x.com/Miles_Brundage/status/874448037929725952) reports that [Clarifai’s](https://www.clarifai.com/) CNN API has little trouble accurately distinguishing man’s worst food from man’s best friend.[](https://gwern.net/tank#fnref10)
+
+11. [](https://gwern.net/tank#fn11 "Link to footnote 11")
+Recht et al 2019’s ImageNet-v2 turns out to illustrate some [subtle issues in measuring dataset bias](https://gradientscience.org/data_rep_bias/) ([Engstrom et al 2020](https://gradientscience.org/data_rep_bias.pdf)): because of measurement error in the labels of images causing errors in the final dataset, simply comparing a classifier trained on one with its performance on the other and noting that performance fell by X% yields a misleadingly inflated estimate of ‘bias’ by attributing the combined error of both datasets to the bias. A [Rip Van Winkle](https://www.offconvex.org/2021/04/07/ripvanwinkle/) estimate of CNN overfitting indicates it must be mild—CNNs just aren’t all that algorithmically complex and thus unable to be overly-tailored to ImageNet. For much more theory on covariate shift impacts and decreases/increases in performance of NNs, see [Tripuraneni et al 2021](https://arxiv.org/abs/2111.08234).[](https://gwern.net/tank#fnref11)
+
+12. [](https://gwern.net/tank#fn12 "Link to footnote 12")
+Lapuschkin et al 2019:
+
+> The first learning machine is a model based on Fisher vectors (FV) [31, 32] trained on the PASCAL VOC 2007 19ya image dataset [33] (see §E). The model and also its competitor, a pretrained Deep Neural Network (DNN) that we fine-tune on PASCAL VOC, show both excellent state-of-the-art test set accuracy on categories such as ‘person’, ‘train’, ‘car’, or ‘horse’ of this benchmark (see Table 3). Inspecting the basis of the decisions with LRP, however, reveals for certain images substantial divergence, as the heatmaps exhibiting the reasons for the respective classification could not be more different. Clearly, the DNN’s heatmap points at the horse and rider as the most relevant features (see Figure 14). In contrast, FV’s heatmap is most focused onto the lower left corner of the image, which contains a source tag. A closer inspection of the data set (of 9963 samples [33]) that typically humans never look through exhaustively, shows that such source tags appear distinctively on horse images; a striking artifact of the dataset that so far had gone unnoticed [34]. Therefore, the FV model has ‘overfitted’ the PASCAL VOC dataset by relying mainly on the easily identifiable source tag, which incidentally correlates with the true features, a clear case of ‘Clever Hans’ behavior. This is confirmed by observing that artificially cutting the source tag from horse images significantly weakens the FV model’s decision while the decision of the DNN stays virtually unchanged (see Figure 14). If we take instead a correctly classified image of a Ferrari and then add to it a source tag, we observe that the FV’s prediction swiftly changes from ‘car’ to ‘horse’ (cf.Figure 2a) a clearly invalid decision (see §E and Figures 15–20 for further examples and analyses)… For the classification of ships the classifier is mostly focused on the presence of water in the bottom half of an image. Removing the copyright tag or the background results in a drop of predictive capabilities. A deep neural network, pre-trained in the ImageNet dataset [93], instead shows none of these shortcomings.
+
+The airplane example is a little more debatable—the presence of a lot of blue sky in airplane images seems like a valid cue to me and not necessarily cheating:
+
+> …The SpRAy analysis could furthermore reveal another ‘Clever Hans’ type behavior in our fine-tuned DNN model, which had gone unnoticed in previous manual analysis of the relevance maps. The large eigengaps in the eigenvalue spectrum of the DNN heatmaps for class “aeroplane” indicate that the model uses very distinct strategies for classifying aeroplane images (see Figure 26). A t-SNE visualization (Figure 28) further highlights this cluster structure. One unexpected strategy we could discover with the help of SpRAy is to identify aeroplane images by looking at the artificial padding pattern at the image borders, which for aeroplane images predominantly consists of uniform and structureless blue background. Note that padding is typically introduced for technical reasons (the DNN model only accepts square shaped inputs), but unexpectedly (and unwantedly) the padding pattern became part of the model’s strategy to classify aeroplane images. Subsequently we observe that changing the manner in which padding is performed has a strong effect on the output of the DNN classifier (see Figures 29–32).
+
+13. [](https://gwern.net/tank#fn13 "Link to footnote 13")
+Winkler et al 2019: “When reviewing the open-access International Skin Imaging Collaboration database, which is a source of training images for research groups, we found that a similar percent-age of melanomas (52 of 2169 [2.4%]) and nevi (214 of 9303 [2.3%]) carry skin markings. Nevertheless, it seems conceivable that either an imbalance in the distribution of skin markings in thousands of other training images that were used in the CNN tested herein or the assignment of higher weights to blue markings only in lesions with specific (though unknown) accompanying features may induce a CNN to associate skin markings with the diagnosis of melanoma. The latter hypothesis may also explain why melanoma probability scores remained almost unchanged in many marked nevi while being increased in others.”[](https://gwern.net/tank#fnref13)
+
+14. [](https://gwern.net/tank#fn14 "Link to footnote 14")
+Getting into more general economic, behavioral, or human situations would be going too far afield, but the relevant analogues are “[principal-agent problem](https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem)”, “[perverse incentives](https://en.wikipedia.org/wiki/Perverse_incentive)”, “law of [unintended consequences](https://en.wikipedia.org/wiki/Unintended_consequences)”, “[Lucas critique](https://en.wikipedia.org/wiki/Lucas_critique)”, “[Goodhart’s law](https://en.wikipedia.org/wiki/Goodhart%27s_law)”, or “[Campbell’s law](https://en.wikipedia.org/wiki/Campbell%27s_law)”; such alignment problems are only partially dealt with by having ground-truth evolutionary [‘outer’ losses](https://gwern.net/backstop), and avoiding reward hacking remains an open problem (even in theory). [Speedrun](https://en.wikipedia.org/wiki/Speedrunning) gaming communities frequently provide examples of reward-hacking, particularly when games are finished faster by exploiting bugs to [sequence break](https://en.wikipedia.org/wiki/Sequence_breaking); particularly esoteric techniques require outright hacking the [“weird machines”](https://gwern.net/turing-complete#security-implications) present in many games/devices—for example, [pannenkoek2012’s](https://en.wikipedia.org/wiki/Pannenkoek2012)[‘parallel universes’](https://pannenkoek2012.fandom.com/wiki/Parallel_Universe)[_Super Mario 64_](https://en.wikipedia.org/wiki/Super_Mario_64) hack which [avoids using any jumps](https://www.youtube.com/watch?v=kpk2tdsPh0A) by exploiting an [integer overflow](https://en.wikipedia.org/wiki/Integer_overflow) bug &[modulo](https://en.wikipedia.org/wiki/Modular_arithmetic) wraparound to accelerate Mario to near-infinite speed, passing through the entire map multiple times, in order to stop at the right place.[](https://gwern.net/tank#fnref14)
diff --git a/docs/evidence/gwern_unseeing.md b/docs/evidence/gwern_unseeing.md
index 1fc76eb..39e05a6 100644
--- a/docs/evidence/gwern_unseeing.md
+++ b/docs/evidence/gwern_unseeing.md
@@ -1,13 +1,221 @@
# Unseeing — Gwern Branwen
-Source: https://gwern.net/unseeing . Verbatim excerpts cached for the skill.
+Source: https://gwern.net/unseeing (page title: "On Seeing Through and Unseeing: The Hacker Mindset")
+Fetched-via: r.jina.ai reader, 2026-08-15 (CLAUDE agent)
+Fetch-status: full article text, with the site's backlinks / similar-links / bibliography nav sections trimmed. Supersedes the earlier two-quote excerpt. (CLAUDE agent)
+
+Why it matters here: why you cannot see your own work or data clearly, and why a single small anomaly can mean the everyday mental model is fundamentally wrong.
---
-From "Learning To Unsee" (on why you can't see your own work/data clearly):
+Defining the security/hacker mindset as extreme reductionism: ignoring the surface abstractions and limitations to treat a system as a source of parts to manipulate into a different system, with different (and usually unintended) capabilities.
-> For example, you can't find typos in your own writing without a great deal of effort because you know what it's *supposed* to say; so copyediting advice runs like 'read it out loud' or 'print it out and read it' or 'wait a week' or recite until gibberish or even 'read it upside down' (easier than it sounds). That's the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote. Similar tricks are used for learning drawing: a face is too familiar, so instead you can flip it in a mirror and try to copy it.
+> To draw some parallels here and expand [Dullien 2017](https://gwern.net/turing-complete#dullien-2017), I think [unexpected Turing-complete systems and weird machines](https://gwern.net/turing-complete) have something in common with heist movies or cons or stage magic: they all share a specific paradigm we might call the _security mindset_ or _hacker mindset_.
+>
+>
+> What they (and hacking, [speedrunning](https://en.wikipedia.org/wiki/Speedrunning), [social-engineering](https://en.wikipedia.org/wiki/Social_engineering_(security)) etc.) all have in common is that they show that the much-ballyhooed ‘hacker mindset’ is, fundamentally, a sort of reductionism run amok, where one [‘sees through’](https://gwern.net/doc/philosophy/epistemology/2012-sistery-tryingtoseethrough.html) abstractions to a manipulable reality. Like Neo in the _Matrix_—a deeply cliche analogy for hacking, but cliche because it resonates—one achieves enlightenment by seeing through the surface illusions of objects and can now see the endless lines of green code which make up the Matrix, and vice-versa. (It’s maps all the way down!)
+>
+>
+> In each case, the fundamental principle is that the hacker asks: “here I have a system _W_, which pretends to be made out of a few [_X_ s](https://github.com/kdeldycke/awesome-falsehood); however, it is **really** made out of many _Y_, which form an entirely different system, _Z_; I will now proceed to ignore the illusory _X_ and understand how _Z_ works, so I may use the _Y_ to thereby change _W_ however I like”.
-From the "Confirmation Bias" section (on anomalies):
+[A](https://gwern.net/dropcap#kanzlei)bstractions are vital, but like many living things, dangerous, because [abstractions always leak](https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/). (“You’re very clever, young man, but it’s reductionism all the way down!”) This is in some sense the opposite of a mathematician: a mathematician tries to ‘see through’ a complex system’s accidental complexity up to a simpler more-abstract more-true version which can be understood & manipulated—but for the hacker, all complexity is essential, and they are instead trying to _un_ see the simple abstract system down to the more-complex less-abstract (but also more true) version.[1](https://gwern.net/unseeing#fn1) (A mathematician might try to transform a program up into successively more abstract representations to eventually show it is trivially correct; a hacker would prefer to compile a program down into its most concrete representation to [brute force all execution paths](https://gwern.net/forking-path)& find an exploit trivially proving it incorrect.)
-> Even a single 'anomaly', apparently trivial in itself, can indicate the everyday mental model is not just a little bit wrong, but *fundamentally* wrong
+## [Confirmation Bias](https://gwern.net/unseeing#confirmation-bias "Link to section: § 'Confirmation Bias'")
+
+> [Uncle Milton Industries](https://en.wikipedia.org/wiki/Milton_Levine) has been selling [ant farms](https://en.wikipedia.org/wiki/Formicarium) to children since 1956 70ya. Some years ago, I remember opening one up with a friend. There were no actual ants included in the box. Instead, there was a card that you filled in with your address, and the company would mail you some ants.
+>
+>
+> My friend expressed surprise that you could get ants sent to you in the mail. I replied: ‘What’s really interesting is that these people will send a tube of live ants to anyone you tell them to.’
+>
+>
+> [Bruce Schneier](https://en.wikipedia.org/wiki/Bruce_Schneier), [“The Security Mindset”](https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html) (2008 18ya); cf.[DNS](https://www.tbray.org/ongoing/When/202x/2022/06/02/Dangerous-Gift), [Mormons/JVs](https://x.com/_JeanLannes/status/1687649736356982784)
+
+
+Ordinary users ask only that all their everyday examples of _Y_ s transforms into _Z_ correctly; they forget to ask whether all and _only_ correct examples of _Y_ s transform into correct _Z_ s, and whether only correct _Z_ s can be constructed to become _Y_ s. Even a single ‘anomaly’, apparently trivial in itself, can indicate the everyday mental model is not just a little bit wrong, but _fundamentally_ wrong, in the way that Newton’s theory of gravity is not merely a little bit wrong and just needs a quick patch with a fudge factor to account for [Mercury](https://en.wikipedia.org/wiki/Two-body_problem_in_general_relativity#Anomalous_precession_of_Mercury) or that NASA management’s mental model of O-rings was [not merely](https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disaster) in need of a minor increase in the thickness of the rubber gaskets[2](https://gwern.net/unseeing#fn2).
+
+## [Atoms](https://gwern.net/unseeing#atoms "Link to section: § 'Atoms'")
+
+> Every drop of blood has great talent; the original cellule seems identical in all animals, and only varied in its growth by the varying circumstance which opens now this kind of cell and now that, causing in the remote effect now horns, now wings, now scales, now hair; and the same numerical atom, it would seem, was equally ready to be a particle of the eye or brain of man, or of the claw of a tiger…The man truly conversant with life knows, against all appearances, that there is a remedy for every wrong, and that every wall is a gate.
+>
+>
+> [Ralph Waldo Emerson](https://en.wikipedia.org/wiki/Ralph_Waldo_Emerson), “Natural History Of Intellect”, 1893[3](https://gwern.net/unseeing#fn3)
+
+It’s all “atoms and the void”[4](https://gwern.net/unseeing#fn4):
+
+* In **hacking**, a computer pretends to be made out of things like ‘buffers’ and ‘lists’ and ‘objects’ with rich meaningful semantics, but really, it’s just made out of bits which mean nothing and only accidentally can be interpreted as things like ‘web browsers’ or ‘passwords’, and if you move some bits around and rewrite these other bits in a particular order and read one string of bits in a different way, now you have bypassed the password.
+
+* In [**speed running**](https://en.wikipedia.org/wiki/Speed_running) (particularly [TASes](https://en.wikipedia.org/wiki/Tool-assisted_speedrun)), a video game pretends to be made out of things like ‘walls’ and ‘speed limits’ and ‘levels which must be completed in a particular order’, but it’s really again just made out of bits and memory locations, and messing with them in particular ways, such as deliberately overloading the RAM [to cause](https://threadreaderapp.com/thread/1148361355130527748.html)[memory allocation](https://www.halopedia.org/Overload_Glitch_(Halo_3)) errors, can give you infinite ‘velocity’ or shift you into [alternate coordinate systems in the true physics](https://www.youtube.com/watch?v=wjge1bVobN0), allowing enormous movements in the supposed map, giving shortcuts to the ‘end’[5](https://gwern.net/unseeing#fn5) of the game.
+
+* in [**stealth games**](https://en.wikipedia.org/wiki/Stealth_games), players learn to unsee levels into patterns of gaps moving around over time—gaps in guard patrols or observability of light/sound—and how to dismantle the level piece by piece until they can go anywhere and do anything
+
+* In **breaking and entering**, like robbing a hotel room, people see ‘doors’ and ‘locks’ and ‘walls’, but really, they are just made out of atoms arranged in a particular order, and you can move some atoms around more easily than others, and instead of going through a ‘door’ you can just cut a hole in the [wall](https://en.wikipedia.org/wiki/Drywall)[6](https://gwern.net/unseeing#fn6) (or ceiling) and obtain access to a space. At Los Alamos, Richard Feynman, among other tactics, [obtained classified papers by reaching in underneath drawers](https://gwern.net/doc/cs/cryptography/1985-feynman-surelyyourejokingmrfeynman-ch18-safecrackermeetsafecracker.pdf)& ignoring the locks entirely.
+
+ * One analysis of the movie _[Die Hard](https://en.wikipedia.org/wiki/Die\_Hard)_, [“Nakatomi space”](https://bldgblog.com/2010/01/nakatomi-space/), highlights how it and the Israel military’s [_mouse-holing_](https://en.wikipedia.org/wiki/Mouse-holing) in the [Battle of Nablus](https://en.wikipedia.org/wiki/Battle_of_Nablus) treat buildings as kinds of machines, which can be manipulated in weird ways to move around to attack their enemies.
+
+ * That example reminds me of the [Carr & Adey](https://bodiesfromthelibrary.com/2017/10/23/seven-types-of-locked-room-mystery-part-15/) anatomy of [_locked room murder mysteries_](https://en.wikipedia.org/wiki/Locked-room_mystery), laying out a taxonomy of all the possible solutions which—like a magician’s trick—violate one’s assumptions about the locked room.
+
+For example, whether the room was always locked, locked at the right time, the murder done while in the room, the murder done _before_ everyone entered the room, it being murder rather than suicide, the supposed secure room with locked-doors having a _ceiling_ etc.[7](https://gwern.net/unseeing#fn7) (These tricks inspired [_Umineko_’s](https://en.wikipedia.org/wiki/Umineko_When_They_Cry) mysteries ([review](https://gwern.net/review/umineko)), although in it a lot of them turn out to just involve [conspirators/lying](https://07th-expansion.fandom.com/wiki/Willard's_Truths).)
+
+ * In [_lockpicking_](https://en.wikipedia.org/wiki/Lockpicking), copying a key or reverse-engineering its cuts are some of the most difficult ways to pick a lock. One can instead simply use a [bump key](https://en.wikipedia.org/wiki/Lock_bumping) to brute-force the positions of the pins in a lock, or kick the door in, or [among other door lock bypasses](https://www.youtube.com/watch?v=4YYvBLAF4T8?t=330), wiggle the bolt, or reach through a crack to open from the inside, or drill the lock. (How do you know someone hasn’t already? You _assume_ it’s the same lock as yesterday?) If all else fails, you can use a portable [hydraulic ram](https://en.wikipedia.org/wiki/Hydraulic_ram) as a spreader to shatter the frame or wall itself _around_ the door.
+
+Locks & safes have many other interesting vulnerabilities; I particularly like [Matt Blaze’s](https://en.wikipedia.org/wiki/Matt_Blaze)[master-key](https://en.wikipedia.org/wiki/Master_keying) vulnerability ([Blaze 2003](https://www.mattblaze.org/papers/mk.pdf)/[Blaze 2004 22ya a](https://www.mattblaze.org/papers/safelocks.pdf)/[Blaze 2004 22ya b](https://www.mattblaze.org/papers/humancambridgepreproc.pdf)), which uses the fact that a master-key lock is actually opening for any _combination_ of master+ordinary key cuts (ie. ‘master OR ordinary’ rather than ‘master XOR ordinary’), and so it is like a password which one can guess one letter at a time. (These papers made locksmiths so mad [they harassed Blaze into quitting](https://x.com/mattblaze/status/1553254965870841856).)
+
+* In [**stage magic**](https://en.wikipedia.org/wiki/Magic_(illusion)) (especially close-up/card/coin/pickpocketing), one believes one is continuously seeing single whole objects which must move from one place to another continuously; in reality, one is only seeing, occasionally, surfaces of many (possibly duplicate) objects, which may be moving only when you are not looking, in the opposite direction, or not moving at all. By hacking [object permanence](https://en.wikipedia.org/wiki/Object_permanence) and limited [attentional](https://en.wikipedia.org/wiki/Misdirection_(magic))[resources](https://en.wikipedia.org/wiki/Change_blindness), the stage magician shows the ‘impossible’ ([Macknik et al 2008’s Table 1](https://gwern.net/doc/psychology/cognitive-bias/illusion-of-depth/2008-macknik.pdf) lists many [folk physics](https://en.wikipedia.org/wiki/Na%C3%AFve_physics) assumptions which can be hacked). Stage magic works by exploiting our implicit beliefs that no adversary would take the trouble to so precisely exploit our heuristics and shortcuts.[8](https://gwern.net/unseeing#fn8)[9](https://gwern.net/unseeing#fn9)
+
+* In **weird machines**, you have a ‘protocol’ like SSL or x86 machine code which appear to do simple things like ‘check a cryptographic signature’ or ‘add one number in a register to another register’, but in reality, it’s a layer over far more complex realities like processor states & optimizations like speculative execution reading other parts of memory and then quickly erasing it, and these can be pasted together to execute operations and reveal secrets without ever running ‘code’ (see again Mcilroy et al 2019).
+
+Similarly, in finding hidden examples of Turing completeness, one says, ‘this system appears to be a bunch of dominoes or whatever, but actually, each one is a computational element which has unusual inputs/outputs; I will now proceed to wire a large number of them together to form a Turing machine so I can play Tetris in Conway’s Game of Life or use heart muscle cells to implement Boolean logic or run arbitrary computations in a game of _Magic: The Gathering_’.
+
+Or in side channels, you go below bits and say, ‘these bits are only approximations to the actual flow of electricity and heat in a system; I will now proceed to measure the physical system’ etc.
+
+* In **social engineering/pen testing**, people see social norms and imaginary things like ‘permission’ and ‘authority’ and ‘managers’ which ‘forbid access to facilities’, but in reality, all there is, is a piece of laminated plastic or a clipboard or certain magic words spoken; the people are merely non-computerized ways of implementing rules like ‘if laminated plastic, allow in’, and if you put on a blue piece of plastic to your shirt and you incant certain words at certain times, you can walk right past the guards.[10](https://gwern.net/unseeing#fn10)
+
+
+* Many financial or economic strategies have a certain flavor of this; [Alice Maz’s _Minecraft_ economics exploits](https://www.alicemaz.com/writing/minecraft.html) strongly reminds me of ‘seeing through’, as do many clever financial trades based on careful reading of contractual minutiae or taking seriously what are usually abstracted details like ‘taking delivery’ of futures etc
+
+* and while we’re at it, why are **puns** so [irresistible to hackers](http://www.catb.org/jargon/html/H/hacker-humor.html "‘Hacker humor’, Raymond 2003")? (Consider how omnipresent they are in _[Gödel, Escher, Bach](https://en.wikipedia.org/wiki/G%C3%B6del,\_Escher,\_Bach)_ or the [Jargon File](https://en.wikipedia.org/wiki/Jargon_File) or text adventures or…)
+
+Because computers are nothing but puns on bits, and languages are nothing but puns on letters! Puns force one to drop from the abstract semantic level to the raw syntactic level of sub-words or characters, and back up again to achieve some semantic twist—they are literally hacking language.
+
+And so on. These sorts of things can seem magical (‘how‽’), shocking (‘but—but—but that’s _cheating_!’ [the scrub](https://www.sirlin.net/articles/playing-to-win) says, who is not playing to _win_), or hilarious (in the ‘[violation of expectations](https://en.wikipedia.org/wiki/Theories_of_humor#Incongruity_theory) followed by [understanding](https://people.idsia.ch/~juergen/creativity.html)’ theory of humor) because the abstract system _W_& our verbalizations are so familiar and useful that we quickly get trapped in our dreams of abstractions, and forget that it is merely a map and not the territory, while inevitably the map has made gross simplifications and it fails to document various paths from one point to another point which we don’t want to exist.
+
+Indeed, these ‘backdoors’ _must_ exist unless carefully engineered away, because the high-level properties we rely on have no existence at the lower levels. If we explain things like ‘permission’ in terms of sequences of digital bits, we must at some point reach a level where the bits no longer express this ‘permission’, in the same way that if we explain ‘color’ or ‘smell’ by atoms, we must do so by eventually describing entities which do not look like they have any color nor have any smell; at some point, these properties must _disintegrate_ into brute facts like a circuit going one way rather than another.[11](https://gwern.net/unseeing#fn11)
+
+## [Curse of Expertise](https://gwern.net/unseeing#curse-of-expertise "Link to section: § 'Curse of Expertise'")
+
+Perversely, the more educated you are, and the more of the map you know, the worse this effect can be, because you have more to unsee (eg. in [fiction](https://gwern.net/story-of-your-life)). One must always maintain a certain contempt for [words](https://gwern.net/language)&[spooks](https://en.wikipedia.org/wiki/Max_Stirner#Philosophy).
+
+The fool can walk right in because he was too ignorant to know that’s impossible. This is why atheoretical optimization processes like animals (eg. [cats engaged in](https://gwern.net/fuzz-testing)[fuzz testing](https://en.wikipedia.org/wiki/Fuzzing)) or [SMT solvers](https://en.wikipedia.org/wiki/Satisfiability_modulo_theories) or [evolutionary AI](https://arxiv.org/abs/1803.03453) are so dumb to begin with, but in the long run can be so good at surprising us and finding ‘unreasonable’ inputs or [reward hacks](https://gwern.net/tank#alternative-examples) (analogous to the [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff)): being unable to understand the map, they can’t benefit from it like we do, but they also can’t overvalue it, and, forced to explore the territory directly to get what they want, discover new things.
+
+## [Learning To Unsee](https://gwern.net/unseeing#learning-to-unsee "Link to section: § 'Learning To Unsee'")
+
+> I don’t even see the code. All I see is blonde, brunette, redhead.
+>
+>
+> Cypher, _The Matrix_
+
+> Whoa.
+>
+>
+> Neo
+
+To escape our semantic illusions can require a determined effort to unsee them, and use of techniques to [defamiliarize](https://en.wikipedia.org/wiki/Defamiliarization) the things.
+
+For example, you can’t find typos in your own writing without a great deal of effort because you know what it’s _supposed_ to say; so copyediting advice runs like ‘read it out loud’ or ‘print it out and read it’ or ‘wait a week’ or [recite until gibberish](https://en.wikipedia.org/wiki/Semantic_satiation) or even ‘read it upside down’ (easier than it sounds). That’s the sort of thing it takes to force you to read what you actually wrote, and not what you thought you wrote. Similar tricks are used for learning drawing: a face is too familiar, so instead you can flip it in a mirror and try to copy it.
+
+The good news is that “what has been unseen cannot be seen”, and that once one _has_ been enlightened into unseeing a system, it seems hard to slip back into the original illusion. And even a little unseeing can be a prophylactic which protects against harmful illusions.
+
+## [External Links](https://gwern.net/unseeing#external-links "Link to section: § 'External Links'")
+
+* [“Security Mindset and Ordinary Paranoia”](https://www.lesswrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia); [“Security Mindset and the Logistic Success Curve”](https://www.lesswrong.com/posts/cpdsMuAHSWhWnKdog/security-mindset-and-the-logistic-success-curve)
+
+* [“How did so many _Dungeon Crawl: Stone Soup_ players miss such an obvious bug?”](https://desystemize.substack.com/p/desystemize-7 "Desystemize #7")
+
+* [“Stargate Physics 101”](https://archiveofourown.org/works/3673335)
+
+* [“The Line of Death”](https://textslashplain.com/2017/01/14/the-line-of-death/)
+
+* [“Movie-Plot Threats”](https://www.schneier.com/tag/movie-plot-threat-contests/)
+
+* [“Security is Mathematics”](https://www.daemonology.net/blog/2008-03-21-security-is-mathematics.html), Colin Percival; [“On Exactitude in Science”](https://kwarc.info/teaching/TDM/Borges.pdf), Jorge Luis Borges
+
+* [“No general method to detect fraud”](https://calpaterson.com/fraud.html "No general method to detect fraud"), Cal Peterson
+
+* [_Red Teaming: How Your Business Can Conquer the Competition by Challenging Everything_](https://www.amazon.com/Red-Teaming-Competition-Challenging-Everything/dp/1101905972), Hoffman
+
+* [_Baba Is You_](https://en.wikipedia.org/wiki/Baba_Is_You): [“No Really, There Are No Rules!”](https://www.lesswrong.com/posts/gvCwotnq2cBTYqEsS/no-really-there-are-no-rules)
+
+* [_The City & the City_](https://en.wikipedia.org/wiki/The_City_%26_the_City)
+
+* [Homograph attacks](https://en.wikipedia.org/wiki/IDN_homograph_attack)
+
+* [“_Getting Over It_ Developer Reacts to 1 Minute 24 Second Speedrun”](https://www.youtube.com/watch?v=dGU5_UUalPA)
+
+* [“The Board Game of the Alpha Nerds: Before _Risk_, before _Dungeons & Dragons_, before _Magic: The Gathering_, there was _Diplomacy_”](https://grantland.com/features/diplomacy-the-board-game-of-the-alpha-nerds/ "One writer enters international competition to play the world-conquering game that redefines what it means to be a geek (and a person)") ([WP](https://en.wikipedia.org/wiki/Diplomacy_(game)); “I still don’t know whom I should have trusted, if anyone. All I know is that I felt stupid, stressed out, humiliated, and sad.”)
+
+* **Discussion**: Reddit: [1](https://www.reddit.com/r/slatestarcodex/comments/c0nqg7/people_seem_to_think_thieves_should_lockpick_or/er6huvz/), [2](https://www.reddit.com/r/DepthHub/comments/c0uutk/ugwern_talks_about_the_hacker_mindset_in/), [3](https://www.reddit.com/r/slatestarcodex/comments/1g1lmmn/gwern_hacker_mindset_nontechnical_examples/); [Twitter](https://x.com/sonyaellenmann/status/1139752544761081858)
+
+* * *
+
+[](https://gwern.net/unseeing#footnotes "Link to section: § ‘Footnotes’")
+1. [](https://gwern.net/unseeing#fn1 "Link to footnote 1")
+‘Thinking outside the box’ can be this, but often isn’t. This is a specific pattern of reductionism, and many instances of ‘thinking outside the box’ are other patterns, like putting on another layer, or eliminating the systems in question entirely.[](https://gwern.net/unseeing#fnref1)
+
+2. [](https://gwern.net/unseeing#fn2 "Link to footnote 2")
+[Feynman](https://www.nasa.gov/history/rogersrep/v2appf.htm):
+
+> The phenomenon of accepting for flight, seals that had shown erosion and blow-by in previous flights, is very clear. The Challenger flight is an excellent example. There are several references to previous flights; the acceptance and success of these flights are taken as evidence of safety. But erosion and blowby are not what the design expected. They are warnings that something is wrong. The equipment is not operating as expected, and therefore there is a danger that it can operate with even wider deviations in the unexpected and not thoroughly understood way. The fact that this danger did not lead to catastrophe before is no guarantee that it will not the next time, unless it is completely understood. When playing Russian roulette the fact that the first shot got off safely is little comfort for the next. The origin and consequences of the erosion and blow-by were not understood. They did not occur equally on all flights and all joints; sometimes more, and sometimes less. Why not sometime, when whatever conditions determined it were right, still more leading to catastrophe?
+>
+>
+> In spite of these variations from case to case, officials behaved as if they understood it, giving apparently logical arguments to each other often depending on the “success” of previous flights…
+
+3. [](https://gwern.net/unseeing#fn3 "Link to footnote 3")
+[pg441–442](https://quod.lib.umich.edu/e/emerson/4957107.0012.001/1:15.1?rgn=div2;view=fulltext), _The complete works of Ralph Waldo Emerson: Natural history of intellect, and other papers_, Vol. 12[](https://gwern.net/unseeing#fnref3)
+
+4. [](https://gwern.net/unseeing#fn4 "Link to footnote 4")
+“By convention sweet is sweet, bitter is bitter, hot is hot, cold is cold, color is color; but in truth there are only atoms and the void.” Incidentally, [Democritus’s](https://en.wikipedia.org/wiki/Democritus) other famous quote on atomism is a pun: “For ‘Tragedy’ [_τρ**α**γωδία_] and ‘Comedy’ [_τρ**υ**γωδία_] come to be out of the same letters.” (As quoted/paraphrased by Aristotle, Book 1, [_On Generation and Corruption_](https://en.wikipedia.org/wiki/On_Generation_and_Corruption); for defense of the interpretation that this is wordplay & not merely a generic observation about alphabetic writing, see [West 1969](https://gwern.net/doc/philosophy/ontology/1969-west.pdf).)[](https://gwern.net/unseeing#fnref4)
+
+5. [](https://gwern.net/unseeing#fn5 "Link to footnote 5")
+A fictional example from _[Ender’s Game](https://en.wikipedia.org/wiki/Ender%27s\_Game)_ is worth noting: if victory in Battle School is defined by 4 soldiers at the corner of the enemy gate & someone passing through, then why not—shades of [Eurisko](https://en.wikipedia.org/wiki/Eurisko)—skip fighting entirely & go straight for the gate?[](https://gwern.net/unseeing#fnref5)
+
+6. [](https://gwern.net/unseeing#fn6 "Link to footnote 6")
+pg356 of [_A Burglar’s Guide to the City_](https://burglarsguide.com/), Geoff Manaugh 2016:
+
+> [Schatz’s](https://en.wikipedia.org/wiki/Andy_Schatz) exhortation to [players](https://en.wikipedia.org/wiki/Monaco:_What%27s_Yours_Is_Mine) to move _against_ the architecture, not with it, to uncover a scene’s possible crimes, is useful not only in the world of games. Ignoring the paths laid out by architects and even remaking a space from within are some of the most fundamental ways in which burglars misuse the built environment…In one of the most interesting moments in [Bill Mason’s](https://en.wikipedia.org/wiki/Bill_Mason_(jewel_thief))[memoir](https://www.amazon.com/Confessions-Master-Jewel-Thief-Mason/dp/0375760717 "_Confessions of a Master Jewel Thief_, Mason 2005"), he sees that architecture can be made to do what he wants it to do; it’s like watching a character in _Star Wars_ learn to use the Force.
+>
+>
+> …he explains that his intended prize was locked inside a room whose door was too closely guarded for him to slip through. Then he realizes the obvious: he has been thinking the way the hotel wanted him to think—the way the architects had hoped he would behave—looking for doors and hallways when he could simply carve a new route where he wanted it. The ensuing realization delights him. “Elated at the idea that I could cut my own door right where I needed one,” he writes, Mason simply breaks into the hotel suite adjacent to the main office. There, he flings open the closet, pushes aside the hangers, and cuts his way from one room into the other using a drywall knife. In no time at all, he has cut his “own door” through to the manager’s office, where he takes whatever he wants—departing right back through the very “door” he himself made. It is architectural surgery, pure and simple.
+>
+>
+> Later, Mason actually mocks the idea that a person would remain reliant on doors, making fun of anyone who thinks burglars, in particular, would respect the limitations of architecture. “_Surely if someone were to rob the place_,” he writes in all italics, barbed with sarcasm, “_they’d come in as respectable people would, through the door provided for the purpose. Maybe that explains why people will have 4 heavy-duty locks on a solid oak door that’s right next to a glass window_.” People seem to think they should lock-pick or kick their way through solid doors rather than just take a $14$10 2016 drywall knife and carve whole new hallways into the world. Those people are mere slaves to architecture, spatial captives in a world someone else has designed for them.
+>
+>
+> Something about this is almost unsettlingly brilliant, as if it is _nonburglars_ who have been misusing the built environment this whole time; as if it is nonburglars who have been unwilling to question the world’s most basic spatial assumptions, too scared to think past the tyranny of architecture’s long-held behavioral expectations…Because doors are often the sturdiest and most fortified parts of the wall in front of you, they are a distraction and a trap. By comparison, the wall itself is often more like tissue paper, just drywall and some 2×4s, without a lock or a chain in sight. Like clouds, apartment walls are mostly air; seen through a burglar’s eyes, they aren’t even there. Cut a hole through one and you’re in the next room in seconds.
+
+7. [](https://gwern.net/unseeing#fn7 "Link to footnote 7")
+Particularly in office buildings, ‘ceilings’ are more of [a suggestion](https://en.wikipedia.org/wiki/Dropped_ceiling) than a structure; in many other buildings, like data centers, so are [the floors](https://en.wikipedia.org/wiki/Raised_floor).[](https://gwern.net/unseeing#fnref7)
+
+8. [](https://gwern.net/unseeing#fn8 "Link to footnote 8")
+Stage magician [Teller](https://en.wikipedia.org/wiki/Teller_(magician)), of [Penn & Teller](https://en.wikipedia.org/wiki/Penn_%26_Teller), puts this well in interviews: what makes stage magic work is _hard work_. “Magic” is spending more effort than any reasonable man would. (Therefore, all magic depends on the unreasonable man.)
+
+Teller 2012 14ya, [“Teller Reveals His Secrets: The smaller, quieter half of the magician duo Penn & Teller writes about how magicians manipulate the human mind”](https://www.smithsonianmag.com/arts-culture/teller-reveals-his-secrets-100744801/):
+
+> I think you’ll see what I mean if I teach you a few principles magicians employ when they want to alter your perceptions…Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest. My partner, Penn, and I once produced 500 live cockroaches from a top hat on the desk of talk-show host [David Letterman](https://en.wikipedia.org/wiki/David_Letterman). To prepare this took weeks. We hired an entomologist who provided slow-moving, camera-friendly cockroaches (the kind from under your stove don’t hang around for close-ups) and taught us to pick the bugs up without screaming like preadolescent girls. Then we built a secret compartment out of foam-core (one of the few materials cockroaches can’t cling to) and worked out a devious routine for sneaking the compartment into the hat. More trouble than the trick was worth? To you, probably. But not to magicians.
+
+Or in his [Huttson 2015 11ya interview](http://www.magicalthinkingbook.com/2015/07/teller-of-penn-teller-on-explaining-magic-tricks/):
+
+> **Matt**: So why don’t you explain all your tricks?
+>
+>
+> **Teller**: Because the short explanation—the explanation that you’d have to do during a theatrical or TV performance—is dull and no fun. The greatest secret to making a deceptive piece of magic is you do it by the ugliest possible means. It’s complex, it’s unromantic, it’s unclever.
+>
+>
+> Because there are no big secrets. There is no safe full of magic secrets somewhere. [Jim Steinmeyer](https://en.wikipedia.org/wiki/Jim_Steinmeyer) said he thinks most of the public believes there’s a big safe that contains all the magic secrets. The biggest job for a magician, he says, is to conceal the fact that that safe is empty. Because every magic secret is just a minor modification of something that you fully understand in everyday life.
+>
+>
+> Take ‘suspending something with a thread’, for example. Everybody’s not been able to see a piece of a thread when they were trying to put it through a needle. What makes it difficult to find is lighting and background. If a magician’s using a thread on stage, say, to levitate a ball, he must use lighting and background to conceal the thread. There’s no obscure secret in that. You learned that playing in your grandmother’s sewing box.
+>
+>
+> Every magic ‘secret’ is hiding in plain sight in the everyday world. It’s not news, and eminently drab.
+
+9. [](https://gwern.net/unseeing#fn9 "Link to footnote 9")
+[Houdini’s trick of Sir Arthur Conan Doyle](https://gwern.net/doc/psychology/cognitive-bias/2006-polidoro-houdinisimpossibledemonstration.html) exemplifies these strategies.
+
+No _reasonable person_ would expect Houdini to renovate an entire room just for a trick, to have learned a [steganographic](https://en.wikipedia.org/wiki/Steganographic) code to communicate the phrase Doyle wrote on a piece of phrase to the assistant without Doyle noticing, or the assistant to manipulate a magnetic pole behind a small suspended slate board, hiding it in the viewers’ _precise_ blind spot in order to make it appear as if the chalk were hovering in mid-air & writing by itself. No reasonable person would go to such efforts to fool you. Therefore, reasonable people are fooled by Houdini’s trick.
+
+Doyle, being a merely reasonable man, did not expect any of that; and disbelieved Houdini’s statement it was merely a trick. But Doyle should have remembered Hume’s dictum: which is more likely—witnessing the paranormal, or that [somewhere in the wide world](https://gwern.net/littlewood) there was a man as cunning, careful, & compulsive as Houdini? The latter!
+
+[Olson &Raz 2020](https://gwern.net/doc/psychedelic/lsd/2020-olson-2.pdf) give further examples, and demonstrate how this can be useful for running psychology experiments.[](https://gwern.net/unseeing#fnref9)
+
+10. [](https://gwern.net/unseeing#fn10 "Link to footnote 10")
+Speaking of ‘social engineering’, why was Facebook’s success in spreading from a niche of college students to much of the world by offering such superficial social networking so surprising to so many? Perhaps its success is a hint that the underlying logic of social interactions are much more abstractable than, and not as rich & subtle as, we’d prefer to think.[](https://gwern.net/unseeing#fnref10)
+
+11. [](https://gwern.net/unseeing#fn11 "Link to footnote 11")
+[Heisenberg](https://en.wikipedia.org/wiki/Werner_Heisenberg) (as quoted in [Hanson 1962](https://gwern.net/doc/philosophy/ontology/1962-hanson.pdf)):
+
+> It is impossible to explain…qualities of matter except by tracing these back to the behavior of entities which themselves no longer possess these qualities. If atoms are really to explain the origin of color and smell of visible material bodies, then they cannot possess properties like color and smell…Atomic theory consistently denies the atom any such perceptible qualities.
+
+Hofstadter sums it up as [“Greenness disintegrates.”](https://gwern.net/doc/philosophy/ontology/1981-hofstadter.pdf#page=21)[](https://gwern.net/unseeing#fnref11)
+
+12. [](https://gwern.net/unseeing#fn12 "Link to footnote 12")
+“48. The best book on programming for the layman is _Alice in Wonderland_; but that’s because it’s the best book on anything for the layman.” —[“Epigrams on Programming”](https://gwern.net/doc/cs/algorithm/1982-perlis.pdf), Perlis 1982 44ya.[](https://gwern.net/unseeing#fnref12)
diff --git a/docs/evidence/karpathy_nn_zero_to_hero_lec4_diagnostics.md b/docs/evidence/karpathy_nn_zero_to_hero_lec4_diagnostics.md
index 5de7f03..3e48e1e 100644
--- a/docs/evidence/karpathy_nn_zero_to_hero_lec4_diagnostics.md
+++ b/docs/evidence/karpathy_nn_zero_to_hero_lec4_diagnostics.md
@@ -3,10 +3,17 @@
**Source:** Andrej Karpathy, nn-zero-to-hero lecture series
**Notebook:** lectures/makemore/makemore_part3_bn.ipynb
**URL:** https://github.com/karpathy/nn-zero-to-hero
+**Raw:** https://raw.githubusercontent.com/karpathy/nn-zero-to-hero/master/lectures/makemore/makemore_part3_bn.ipynb
**Lecture description:** "We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network."
+**Fetched-via:** curl of the raw .ipynb from GitHub, 2026-08-15, cells rendered to markdown with outputs dropped. (CLAUDE agent)
+**Fetch-status:** full notebook source. The skill-authored diagnostic reading notes come first, then the complete notebook. (CLAUDE agent)
---
+# Skill-authored reading notes
+
+The "Healthy / Bad" annotations below are written for this skill, not quoted from Karpathy. The notebook code they refer to follows in the next section.
+
## Incremental improvements documented (from notebook markdown)
```
@@ -111,3 +118,549 @@ The notebook demonstrates by construction (not just assertion) that:
3. BatchNorm → robust to poor init; normalization forces healthy activation stats
The incremental improvement log (above) makes this concrete: each targeted fix yields measurable improvement. This is the same pattern as the recipe blog post but with code and measured results.
+
+---
+
+# Full notebook source: makemore_part3_bn.ipynb
+
+---
+
+# makemore: part 3
+
+```python
+import torch
+import torch.nn.functional as F
+import matplotlib.pyplot as plt # for making figures
+%matplotlib inline
+```
+
+```python
+# read in all the words
+words = open('names.txt', 'r').read().splitlines()
+words[:8]
+```
+
+```python
+len(words)
+```
+
+```python
+# build the vocabulary of characters and mappings to/from integers
+chars = sorted(list(set(''.join(words))))
+stoi = {s:i+1 for i,s in enumerate(chars)}
+stoi['.'] = 0
+itos = {i:s for s,i in stoi.items()}
+vocab_size = len(itos)
+print(itos)
+print(vocab_size)
+```
+
+```python
+# build the dataset
+block_size = 3 # context length: how many characters do we take to predict the next one?
+
+def build_dataset(words):
+ X, Y = [], []
+
+ for w in words:
+ context = [0] * block_size
+ for ch in w + '.':
+ ix = stoi[ch]
+ X.append(context)
+ Y.append(ix)
+ context = context[1:] + [ix] # crop and append
+
+ X = torch.tensor(X)
+ Y = torch.tensor(Y)
+ print(X.shape, Y.shape)
+ return X, Y
+
+import random
+random.seed(42)
+random.shuffle(words)
+n1 = int(0.8*len(words))
+n2 = int(0.9*len(words))
+
+Xtr, Ytr = build_dataset(words[:n1]) # 80%
+Xdev, Ydev = build_dataset(words[n1:n2]) # 10%
+Xte, Yte = build_dataset(words[n2:]) # 10%
+```
+
+```python
+# MLP revisited
+n_embd = 10 # the dimensionality of the character embedding vectors
+n_hidden = 200 # the number of neurons in the hidden layer of the MLP
+
+g = torch.Generator().manual_seed(2147483647) # for reproducibility
+C = torch.randn((vocab_size, n_embd), generator=g)
+W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * (5/3)/((n_embd * block_size)**0.5) #* 0.2
+#b1 = torch.randn(n_hidden, generator=g) * 0.01
+W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.01
+b2 = torch.randn(vocab_size, generator=g) * 0
+
+# BatchNorm parameters
+bngain = torch.ones((1, n_hidden))
+bnbias = torch.zeros((1, n_hidden))
+bnmean_running = torch.zeros((1, n_hidden))
+bnstd_running = torch.ones((1, n_hidden))
+
+parameters = [C, W1, W2, b2, bngain, bnbias]
+print(sum(p.nelement() for p in parameters)) # number of parameters in total
+for p in parameters:
+ p.requires_grad = True
+```
+
+```python
+# same optimization as last time
+max_steps = 200000
+batch_size = 32
+lossi = []
+
+for i in range(max_steps):
+
+ # minibatch construct
+ ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)
+ Xb, Yb = Xtr[ix], Ytr[ix] # batch X,Y
+
+ # forward pass
+ emb = C[Xb] # embed the characters into vectors
+ embcat = emb.view(emb.shape[0], -1) # concatenate the vectors
+ # Linear layer
+ hpreact = embcat @ W1 #+ b1 # hidden layer pre-activation
+ # BatchNorm layer
+ # -------------------------------------------------------------
+ bnmeani = hpreact.mean(0, keepdim=True)
+ bnstdi = hpreact.std(0, keepdim=True)
+ hpreact = bngain * (hpreact - bnmeani) / bnstdi + bnbias
+ with torch.no_grad():
+ bnmean_running = 0.999 * bnmean_running + 0.001 * bnmeani
+ bnstd_running = 0.999 * bnstd_running + 0.001 * bnstdi
+ # -------------------------------------------------------------
+ # Non-linearity
+ h = torch.tanh(hpreact) # hidden layer
+ logits = h @ W2 + b2 # output layer
+ loss = F.cross_entropy(logits, Yb) # loss function
+
+ # backward pass
+ for p in parameters:
+ p.grad = None
+ loss.backward()
+
+ # update
+ lr = 0.1 if i < 100000 else 0.01 # step learning rate decay
+ for p in parameters:
+ p.data += -lr * p.grad
+
+ # track stats
+ if i % 10000 == 0: # print every once in a while
+ print(f'{i:7d}/{max_steps:7d}: {loss.item():.4f}')
+ lossi.append(loss.log10().item())
+```
+
+```python
+plt.plot(lossi)
+```
+
+```python
+# calibrate the batch norm at the end of training
+
+with torch.no_grad():
+ # pass the training set through
+ emb = C[Xtr]
+ embcat = emb.view(emb.shape[0], -1)
+ hpreact = embcat @ W1 # + b1
+ # measure the mean/std over the entire training set
+ bnmean = hpreact.mean(0, keepdim=True)
+ bnstd = hpreact.std(0, keepdim=True)
+```
+
+```python
+@torch.no_grad() # this decorator disables gradient tracking
+def split_loss(split):
+ x,y = {
+ 'train': (Xtr, Ytr),
+ 'val': (Xdev, Ydev),
+ 'test': (Xte, Yte),
+ }[split]
+ emb = C[x] # (N, block_size, n_embd)
+ embcat = emb.view(emb.shape[0], -1) # concat into (N, block_size * n_embd)
+ hpreact = embcat @ W1 # + b1
+ #hpreact = bngain * (hpreact - hpreact.mean(0, keepdim=True)) / hpreact.std(0, keepdim=True) + bnbias
+ hpreact = bngain * (hpreact - bnmean_running) / bnstd_running + bnbias
+ h = torch.tanh(hpreact) # (N, n_hidden)
+ logits = h @ W2 + b2 # (N, vocab_size)
+ loss = F.cross_entropy(logits, y)
+ print(split, loss.item())
+
+split_loss('train')
+split_loss('val')
+```
+
+## loss log
+
+### original:
+train 2.1245384216308594
+val 2.168196439743042
+
+### fix softmax confidently wrong:
+train 2.07
+val 2.13
+
+### fix tanh layer too saturated at init:
+train 2.0355966091156006
+val 2.1026785373687744
+
+### use semi-principled "kaiming init" instead of hacky init:
+train 2.0376641750335693
+val 2.106989622116089
+
+### add batch norm layer
+train 2.0668270587921143
+val 2.104844808578491
+
+```python
+# SUMMARY + PYTORCHIFYING -----------
+```
+
+```python
+# Let's train a deeper network
+# The classes we create here are the same API as nn.Module in PyTorch
+
+class Linear:
+
+ def __init__(self, fan_in, fan_out, bias=True):
+ self.weight = torch.randn((fan_in, fan_out), generator=g) / fan_in**0.5
+ self.bias = torch.zeros(fan_out) if bias else None
+
+ def __call__(self, x):
+ self.out = x @ self.weight
+ if self.bias is not None:
+ self.out += self.bias
+ return self.out
+
+ def parameters(self):
+ return [self.weight] + ([] if self.bias is None else [self.bias])
+
+
+class BatchNorm1d:
+
+ def __init__(self, dim, eps=1e-5, momentum=0.1):
+ self.eps = eps
+ self.momentum = momentum
+ self.training = True
+ # parameters (trained with backprop)
+ self.gamma = torch.ones(dim)
+ self.beta = torch.zeros(dim)
+ # buffers (trained with a running 'momentum update')
+ self.running_mean = torch.zeros(dim)
+ self.running_var = torch.ones(dim)
+
+ def __call__(self, x):
+ # calculate the forward pass
+ if self.training:
+ xmean = x.mean(0, keepdim=True) # batch mean
+ xvar = x.var(0, keepdim=True) # batch variance
+ else:
+ xmean = self.running_mean
+ xvar = self.running_var
+ xhat = (x - xmean) / torch.sqrt(xvar + self.eps) # normalize to unit variance
+ self.out = self.gamma * xhat + self.beta
+ # update the buffers
+ if self.training:
+ with torch.no_grad():
+ self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * xmean
+ self.running_var = (1 - self.momentum) * self.running_var + self.momentum * xvar
+ return self.out
+
+ def parameters(self):
+ return [self.gamma, self.beta]
+
+class Tanh:
+ def __call__(self, x):
+ self.out = torch.tanh(x)
+ return self.out
+ def parameters(self):
+ return []
+
+n_embd = 10 # the dimensionality of the character embedding vectors
+n_hidden = 100 # the number of neurons in the hidden layer of the MLP
+g = torch.Generator().manual_seed(2147483647) # for reproducibility
+
+C = torch.randn((vocab_size, n_embd), generator=g)
+layers = [
+ Linear(n_embd * block_size, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),
+ Linear( n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),
+ Linear( n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),
+ Linear( n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),
+ Linear( n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),
+ Linear( n_hidden, vocab_size, bias=False), BatchNorm1d(vocab_size),
+]
+# layers = [
+# Linear(n_embd * block_size, n_hidden), Tanh(),
+# Linear( n_hidden, n_hidden), Tanh(),
+# Linear( n_hidden, n_hidden), Tanh(),
+# Linear( n_hidden, n_hidden), Tanh(),
+# Linear( n_hidden, n_hidden), Tanh(),
+# Linear( n_hidden, vocab_size),
+# ]
+
+with torch.no_grad():
+ # last layer: make less confident
+ layers[-1].gamma *= 0.1
+ #layers[-1].weight *= 0.1
+ # all other layers: apply gain
+ for layer in layers[:-1]:
+ if isinstance(layer, Linear):
+ layer.weight *= 1.0 #5/3
+
+parameters = [C] + [p for layer in layers for p in layer.parameters()]
+print(sum(p.nelement() for p in parameters)) # number of parameters in total
+for p in parameters:
+ p.requires_grad = True
+```
+
+```python
+# same optimization as last time
+max_steps = 200000
+batch_size = 32
+lossi = []
+ud = []
+
+for i in range(max_steps):
+
+ # minibatch construct
+ ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)
+ Xb, Yb = Xtr[ix], Ytr[ix] # batch X,Y
+
+ # forward pass
+ emb = C[Xb] # embed the characters into vectors
+ x = emb.view(emb.shape[0], -1) # concatenate the vectors
+ for layer in layers:
+ x = layer(x)
+ loss = F.cross_entropy(x, Yb) # loss function
+
+ # backward pass
+ for layer in layers:
+ layer.out.retain_grad() # AFTER_DEBUG: would take out retain_graph
+ for p in parameters:
+ p.grad = None
+ loss.backward()
+
+ # update
+ lr = 0.1 if i < 150000 else 0.01 # step learning rate decay
+ for p in parameters:
+ p.data += -lr * p.grad
+
+ # track stats
+ if i % 10000 == 0: # print every once in a while
+ print(f'{i:7d}/{max_steps:7d}: {loss.item():.4f}')
+ lossi.append(loss.log10().item())
+ with torch.no_grad():
+ ud.append([((lr*p.grad).std() / p.data.std()).log10().item() for p in parameters])
+
+ if i >= 1000:
+ break # AFTER_DEBUG: would take out obviously to run full optimization
+```
+
+```python
+# visualize histograms
+plt.figure(figsize=(20, 4)) # width and height of the plot
+legends = []
+for i, layer in enumerate(layers[:-1]): # note: exclude the output layer
+ if isinstance(layer, Tanh):
+ t = layer.out
+ print('layer %d (%10s): mean %+.2f, std %.2f, saturated: %.2f%%' % (i, layer.__class__.__name__, t.mean(), t.std(), (t.abs() > 0.97).float().mean()*100))
+ hy, hx = torch.histogram(t, density=True)
+ plt.plot(hx[:-1].detach(), hy.detach())
+ legends.append(f'layer {i} ({layer.__class__.__name__}')
+plt.legend(legends);
+plt.title('activation distribution')
+```
+
+```python
+# visualize histograms
+plt.figure(figsize=(20, 4)) # width and height of the plot
+legends = []
+for i, layer in enumerate(layers[:-1]): # note: exclude the output layer
+ if isinstance(layer, Tanh):
+ t = layer.out.grad
+ print('layer %d (%10s): mean %+f, std %e' % (i, layer.__class__.__name__, t.mean(), t.std()))
+ hy, hx = torch.histogram(t, density=True)
+ plt.plot(hx[:-1].detach(), hy.detach())
+ legends.append(f'layer {i} ({layer.__class__.__name__}')
+plt.legend(legends);
+plt.title('gradient distribution')
+```
+
+```python
+# visualize histograms
+plt.figure(figsize=(20, 4)) # width and height of the plot
+legends = []
+for i,p in enumerate(parameters):
+ t = p.grad
+ if p.ndim == 2:
+ print('weight %10s | mean %+f | std %e | grad:data ratio %e' % (tuple(p.shape), t.mean(), t.std(), t.std() / p.std()))
+ hy, hx = torch.histogram(t, density=True)
+ plt.plot(hx[:-1].detach(), hy.detach())
+ legends.append(f'{i} {tuple(p.shape)}')
+plt.legend(legends)
+plt.title('weights gradient distribution');
+```
+
+```python
+plt.figure(figsize=(20, 4))
+legends = []
+for i,p in enumerate(parameters):
+ if p.ndim == 2:
+ plt.plot([ud[j][i] for j in range(len(ud))])
+ legends.append('param %d' % i)
+plt.plot([0, len(ud)], [-3, -3], 'k') # these ratios should be ~1e-3, indicate on plot
+plt.legend(legends);
+```
+
+```python
+@torch.no_grad() # this decorator disables gradient tracking
+def split_loss(split):
+ x,y = {
+ 'train': (Xtr, Ytr),
+ 'val': (Xdev, Ydev),
+ 'test': (Xte, Yte),
+ }[split]
+ emb = C[x] # (N, block_size, n_embd)
+ x = emb.view(emb.shape[0], -1) # concat into (N, block_size * n_embd)
+ for layer in layers:
+ x = layer(x)
+ loss = F.cross_entropy(x, y)
+ print(split, loss.item())
+
+# put layers into eval mode
+for layer in layers:
+ layer.training = False
+split_loss('train')
+split_loss('val')
+```
+
+```python
+# sample from the model
+g = torch.Generator().manual_seed(2147483647 + 10)
+
+for _ in range(20):
+
+ out = []
+ context = [0] * block_size # initialize with all ...
+ while True:
+ # forward pass the neural net
+ emb = C[torch.tensor([context])] # (1,block_size,n_embd)
+ x = emb.view(emb.shape[0], -1) # concatenate the vectors
+ for layer in layers:
+ x = layer(x)
+ logits = x
+ probs = F.softmax(logits, dim=1)
+ # sample from the distribution
+ ix = torch.multinomial(probs, num_samples=1, generator=g).item()
+ # shift the context window and track the samples
+ context = context[1:] + [ix]
+ out.append(ix)
+ # if we sample the special '.' token, break
+ if ix == 0:
+ break
+
+ print(''.join(itos[i] for i in out)) # decode and print the generated word
+```
+
+```python
+# DONE; BONUS content below, not covered in video
+```
+
+```python
+# BatchNorm forward pass as a widget
+
+from ipywidgets import interact, interactive, fixed, interact_manual
+import ipywidgets as widgets
+import scipy.stats as stats
+import numpy as np
+
+def normshow(x0):
+
+ g = torch.Generator().manual_seed(2147483647+1)
+ x = torch.randn(5, generator=g) * 5
+ x[0] = x0 # override the 0th example with the slider
+ mu = x.mean()
+ sig = x.std()
+ y = (x - mu)/sig
+
+ plt.figure(figsize=(10, 5))
+ # plot 0
+ plt.plot([-6,6], [0,0], 'k')
+ # plot the mean and std
+ xx = np.linspace(-6, 6, 100)
+ plt.plot(xx, stats.norm.pdf(xx, mu, sig), 'b')
+ xx = np.linspace(-6, 6, 100)
+ plt.plot(xx, stats.norm.pdf(xx, 0, 1), 'r')
+ # plot little lines connecting input and output
+ for i in range(len(x)):
+ plt.plot([x[i],y[i]], [1, 0], 'k', alpha=0.2)
+ # plot the input and output values
+ plt.scatter(x.data, torch.ones_like(x).data, c='b', s=100)
+ plt.scatter(y.data, torch.zeros_like(y).data, c='r', s=100)
+ plt.xlim(-6, 6)
+ # title
+ plt.title('input mu %.2f std %.2f' % (mu, sig))
+
+interact(normshow, x0=(-30,30,0.5));
+```
+
+```python
+# Linear: activation statistics of forward and backward pass
+
+g = torch.Generator().manual_seed(2147483647)
+
+a = torch.randn((1000,1), requires_grad=True, generator=g) # a.grad = b.T @ c.grad
+b = torch.randn((1000,1000), requires_grad=True, generator=g) # b.grad = c.grad @ a.T
+c = b @ a
+loss = torch.randn(1000, generator=g) @ c
+a.retain_grad()
+b.retain_grad()
+c.retain_grad()
+loss.backward()
+print('a std:', a.std().item())
+print('b std:', b.std().item())
+print('c std:', c.std().item())
+print('-----')
+print('c grad std:', c.grad.std().item())
+print('a grad std:', a.grad.std().item())
+print('b grad std:', b.grad.std().item())
+```
+
+```python
+# Linear + BatchNorm: activation statistics of forward and backward pass
+
+g = torch.Generator().manual_seed(2147483647)
+
+n = 1000
+# linear layer ---
+inp = torch.randn(n, requires_grad=True, generator=g)
+w = torch.randn((n, n), requires_grad=True, generator=g) # / n**0.5
+x = w @ inp
+# bn layer ---
+xmean = x.mean()
+xvar = x.var()
+out = (x - xmean) / torch.sqrt(xvar + 1e-5)
+# ----
+loss = out @ torch.randn(n, generator=g)
+inp.retain_grad()
+x.retain_grad()
+w.retain_grad()
+out.retain_grad()
+loss.backward()
+
+print('inp std: ', inp.std().item())
+print('w std: ', w.std().item())
+print('x std: ', x.std().item())
+print('out std: ', out.std().item())
+print('------')
+print('out grad std: ', out.grad.std().item())
+print('x grad std: ', x.grad.std().item())
+print('w grad std: ', w.grad.std().item())
+print('inp grad std: ', inp.grad.std().item())
+```
diff --git a/docs/evidence/karpathy_recipe_training_nn_2019.md b/docs/evidence/karpathy_recipe_training_nn_2019.md
index 0047b50..b30ab18 100644
--- a/docs/evidence/karpathy_recipe_training_nn_2019.md
+++ b/docs/evidence/karpathy_recipe_training_nn_2019.md
@@ -3,117 +3,123 @@
**Source:** Andrej Karpathy blog post, April 25, 2019
**URL:** https://karpathy.github.io/2019/04/25/recipe/
**Author:** Andrej Karpathy (then Stanford/OpenAI/Tesla)
+**Fetched-via:** r.jina.ai reader, 2026-08-15 (CLAUDE agent)
+**Fetch-status:** full post text. Supersedes the earlier abridged note, which paraphrased and elided with "..." inside quote marks. (CLAUDE agent)
+
+Why it matters here: the canonical staged process (data, dumb baseline, overfit, regularize, tune, squeeze) for training that fails silently instead of crashing.
---
-## Core thesis: silent failure problem
+Some few weeks ago I [posted](https://twitter.com/karpathy/status/1013244313327681536?lang=en) a tweet on “the most common neural net mistakes”, listing a few common gotchas related to training neural nets. The tweet got quite a bit more engagement than I anticipated (including a [webinar](https://www.bigmarker.com/missinglink-ai/PyTorch-Code-to-Unpack-Andrej-Karpathy-s-6-Most-Common-NN-Mistakes) :)). Clearly, a lot of people have personally encountered the large gap between “here is how a convolutional layer works” and “our convnet achieves state of the art results”.
-> "The 'possible error surface' is large, logical (as opposed to syntactic), and very tricky to unit test... a 'fast and furious' approach to training neural networks does not work and only leads to suffering."
+So I thought it could be fun to brush off my dusty blog to expand my tweet to the long form that this topic deserves. However, instead of going into an enumeration of more common errors or fleshing them out, I wanted to dig a bit deeper and talk about how one can avoid making these errors altogether (or fix them very fast). The trick to doing so is to follow a certain process, which as far as I can tell is not very often documented. Let’s start with two important observations that motivate it.
-Examples of silent failures listed:
-- Forgetting to flip labels during left-right augmentation (network learns to detect flipped images internally)
-- Off-by-one bugs in autoregressive models
-- Clipping loss instead of gradients
-- Using wrong mean from pretrained checkpoint
-- Misconfigured regularization / LR / decay
+#### 1) Neural net training is a leaky abstraction
-> "The qualities that in my experience correlate most strongly to success in deep learning are patience and attention to detail."
+It is allegedly easy to get started with training neural nets. Numerous libraries and frameworks take pride in displaying 30-line miracle snippets that solve your data problems, giving the (false) impression that this stuff is plug and play. It’s common see things like:
----
+```
+>>> your_data = # plug your awesome dataset here
+>>> model = SuperCrossValidator(SuperDuper.fit, your_data, ResNet50, SGDOptimizer)
+# conquer world here
+```
-## Stage 1: Become one with the data
+These libraries and examples activate the part of our brain that is familiar with standard software - a place where clean APIs and abstractions are often attainable. [Requests](http://docs.python-requests.org/en/master/) library to demonstrate:
-> "The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data."
+```
+>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
+>>> r.status_code
+200
+```
-**Manual inspection:**
-> "Scan through thousands of examples manually... understand distribution patterns. Look for: duplicates, corrupted images/labels, imbalances, biases. Pay attention to own classification process -- hints at needed architecture."
+That’s cool! A courageous developer has taken the burden of understanding query strings, urls, GET/POST requests, HTTP connections, and so on from you and largely hidden the complexity behind a few lines of code. This is what we are familiar with and expect. Unfortunately, neural nets are nothing like that. They are not “off-the-shelf” technology the second you deviate slightly from training an ImageNet classifier. I’ve tried to make this point in my post [“Yes you should understand backprop”](https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b) by picking on backpropagation and calling it a “leaky abstraction”, but the situation is unfortunately much more dire. Backprop + SGD does not magically make your network work. Batch norm does not magically make it converge faster. RNNs don’t magically let you “plug in” text. And just because you can formulate your problem as RL doesn’t mean you should. If you insist on using the technology without understanding how it works you are likely to fail. Which brings me to…
-**Programmatic search for outliers:**
-> "The outliers especially almost always uncover some bugs in data quality or preprocessing."
+#### 2) Neural net training fails silently
----
+When you break or misconfigure code you will often get some kind of an exception. You plugged in an integer where something expected a string. The function only expected 3 arguments. This import failed. That key does not exist. The number of elements in the two lists isn’t equal. In addition, it’s often possible to create unit tests for a certain functionality.
-## Stage 2: End-to-end pipeline & dumb baselines
+This is just a start when it comes to training neural nets. Everything could be correct syntactically, but the whole thing isn’t arranged properly, and it’s really hard to tell. The “possible error surface” is large, logical (as opposed to syntactic), and very tricky to unit test. For example, perhaps you forgot to flip your labels when you left-right flipped the image during data augmentation. Your net can still (shockingly) work pretty well because your network can internally learn to detect flipped images and then it left-right flips its predictions. Or maybe your autoregressive model accidentally takes the thing it’s trying to predict as an input due to an off-by-one bug. Or you tried to clip your gradients but instead clipped the loss, causing the outlier examples to be ignored during training. Or you initialized your weights from a pretrained checkpoint but didn’t use the original mean. Or you just screwed up the settings for regularization strengths, learning rate, its decay rate, model size, etc. Therefore, your misconfigured neural net will throw exceptions only if you’re lucky; Most of the time it will train but silently work a bit worse.
-**Fix random seed:**
-> "Always use a fixed random seed to guarantee that when you run the code twice you will get the same outcome. This removes a factor of variation and will help keep you sane."
+As a result, (and this is reeaally difficult to over-emphasize) **a “fast and furious” approach to training neural networks does not work** and only leads to suffering. Now, suffering is a perfectly natural part of getting a neural network to work well, but it can be mitigated by being thorough, defensive, paranoid, and obsessed with visualizations of basically every possible thing. The qualities that in my experience correlate most strongly to success in deep learning are patience and attention to detail.
-**Disable complexity early:**
-- Turn off data augmentation initially -- "it is just another opportunity to introduce some dumb bug"
+## The recipe
-**Verify loss at initialization:**
-> "Verify that your loss starts at the correct loss value. E.g. if you initialize your final layer correctly you should measure -log(1/n_classes) on a softmax at initialization."
+In light of the above two facts, I have developed a specific process for myself that I follow when applying a neural net to a new problem, which I will try to describe. You will see that it takes the two principles above very seriously. In particular, it builds from simple to complex and at every step of the way we make concrete hypotheses about what will happen and then either validate them with an experiment or investigate until we find some issue. What we try to prevent very hard is the introduction of a lot of “unverified” complexity at once, which is bound to introduce bugs/misconfigurations that will take forever to find (if ever). If writing your neural net code was like training one, you’d want to use a very small learning rate and guess and then evaluate the full test set after every iteration.
-**Initialize final layer bias correctly:**
-> "Regression with mean 50? Initialize bias to 50. Imbalanced dataset (1:10)? Set logit bias to predict 0.1 probability at init. Setting these correctly will speed up convergence and eliminate 'hockey stick' loss curves."
+#### 1. Become one with the data
-**Overfit a single batch:**
-> "Overfit a single batch of only a few examples (e.g. as little as two). To do so we increase the capacity of our model and verify that we can reach the lowest achievable loss (e.g. zero)... If they do not, there is a bug somewhere and we cannot continue to the next stage."
+The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. This step is critical. I like to spend copious amount of time (measured in units of hours) scanning through thousands of examples, understanding their distribution and looking for patterns. Luckily, your brain is pretty good at this. One time I discovered that the data contained duplicate examples. Another time I found corrupted images / labels. I look for data imbalances and biases. I will typically also pay attention to my own process for classifying the data, which hints at the kinds of architectures we’ll eventually explore. As an example - are very local features enough or do we need global context? How much variation is there and what form does it take? What variation is spurious and could be preprocessed out? Does spatial position matter or do we want to average pool it out? How much does detail matter and how far could we afford to downsample the images? How noisy are the labels?
-**Visualize immediately before model input:**
-> "The unambiguously correct place to visualize your data is immediately before your y_hat = model(x)... This is the only 'source of truth'. I can't count the number of times this has saved me and revealed problems in data preprocessing and augmentation."
+In addition, since the neural net is effectively a compressed/compiled version of your dataset, you’ll be able to look at your network (mis)predictions and understand where they might be coming from. And if your network is giving you some prediction that doesn’t seem consistent with what you’ve seen in the data, something is off.
-**Visualize prediction dynamics:**
-> "The 'dynamics' of how these predictions move will give you incredibly good intuition for how the training progresses. Many times it is possible to feel the network 'struggle' to fit your data if it wiggles too much in some way, revealing instabilities."
+Once you get a qualitative sense it is also a good idea to write some simple code to search/filter/sort by whatever you can think of (e.g. type of label, size of annotations, number of annotations, etc.) and visualize their distributions and the outliers along any axis. The outliers especially almost always uncover some bugs in data quality or preprocessing.
-**Backprop-to-input dependency check:**
-> "A relatively common bug I've come across... people use view instead of transpose/permute somewhere and inadvertently mix information across the batch dimension... set the loss to be something trivial like the sum of all outputs of example i, run the backward pass all the way to the input, and ensure that you get a non-zero gradient only on the i-th input."
+#### 2. Set up the end-to-end training/evaluation skeleton + get dumb baselines
-**Input-independent baseline:**
-> Train model with all inputs zeroed. "Does your model learn to extract any information out of the input at all? If not, something is wrong."
+Now that we understand our data can we reach for our super fancy Multi-scale ASPP FPN ResNet and begin training awesome models? For sure no. That is the road to suffering. Our next step is to set up a full training + evaluation skeleton and gain trust in its correctness via a series of experiments. At this stage it is best to pick some simple model that you couldn’t possibly have screwed up somehow - e.g. a linear classifier, or a very tiny ConvNet. We’ll want to train it, visualize the losses, any other metrics (e.g. accuracy), model predictions, and perform a series of ablation experiments with explicit hypotheses along the way.
----
+Tips & tricks for this stage:
-## Stage 3: Overfit
+* **fix random seed**. Always use a fixed random seed to guarantee that when you run the code twice you will get the same outcome. This removes a factor of variation and will help keep you sane.
+* **simplify**. Make sure to disable any unnecessary fanciness. As an example, definitely turn off any data augmentation at this stage. Data augmentation is a regularization strategy that we may incorporate later, but for now it is just another opportunity to introduce some dumb bug.
+* **add significant digits to your eval**. When plotting the test loss run the evaluation over the entire (large) test set. Do not just plot test losses over batches and then rely on smoothing them in Tensorboard. We are in pursuit of correctness and are very willing to give up time for staying sane.
+* **verify loss @ init**. Verify that your loss starts at the correct loss value. E.g. if you initialize your final layer correctly you should measure `-log(1/n_classes)` on a softmax at initialization. The same default values can be derived for L2 regression, Huber losses, etc.
+* **init well**. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias.
+* **human baseline**. Monitor metrics other than loss that are human interpretable and checkable (e.g. accuracy). Whenever possible evaluate your own (human) accuracy and compare to it. Alternatively, annotate the test data twice and for each example treat one annotation as prediction and the second as ground truth.
+* **input-indepent baseline**. Train an input-independent baseline, (e.g. easiest is to just set all your inputs to zero). This should perform worse than when you actually plug in your data without zeroing it out. Does it? i.e. does your model learn to extract any information out of the input at all?
+* **overfit one batch**. Overfit a single batch of only a few examples (e.g. as little as two). To do so we increase the capacity of our model (e.g. add layers or filters) and verify that we can reach the lowest achievable loss (e.g. zero). I also like to visualize in the same plot both the label and the prediction and ensure that they end up aligning perfectly once we reach the minimum loss. If they do not, there is a bug somewhere and we cannot continue to the next stage.
+* **verify decreasing training loss**. At this stage you will hopefully be underfitting on your dataset because you’re working with a toy model. Try to increase its capacity just a bit. Did your training loss go down as it should?
+* **visualize just before the net**. The unambiguously correct place to visualize your data is immediately before your `y_hat = model(x)` (or `sess.run` in tf). That is - you want to visualize _exactly_ what goes into your network, decoding that raw tensor of data and labels into visualizations. This is the only “source of truth”. I can’t count the number of times this has saved me and revealed problems in data preprocessing and augmentation.
+* **visualize prediction dynamics**. I like to visualize model predictions on a fixed test batch during the course of training. The “dynamics” of how these predictions move will give you incredibly good intuition for how the training progresses. Many times it is possible to feel the network “struggle” to fit your data if it wiggles too much in some way, revealing instabilities. Very low or very high learning rates are also easily noticeable in the amount of jitter.
+* **use backprop to chart dependencies**. Your deep learning code will often contain complicated, vectorized, and broadcasted operations. A relatively common bug I’ve come across a few times is that people get this wrong (e.g. they use `view` instead of `transpose/permute` somewhere) and inadvertently mix information across the batch dimension. It is a depressing fact that your network will typically still train okay because it will learn to ignore data from the other examples. One way to debug this (and other related problems) is to set the loss to be something trivial like the sum of all outputs of example **i**, run the backward pass all the way to the input, and ensure that you get a non-zero gradient only on the **i-th** input. The same strategy can be used to e.g. ensure that your autoregressive model at time t only depends on 1..t-1. More generally, gradients give you information about what depends on what in your network, which can be useful for debugging.
+* **generalize a special case**. This is a bit more of a general coding tip but I’ve often seen people create bugs when they bite off more than they can chew, writing a relatively general functionality from scratch. I like to write a very specific function to what I’m doing right now, get that to work, and then generalize it later making sure that I get the same result. Often this applies to vectorizing code, where I almost always write out the fully loopy version first and only then transform it to vectorized code one loop at a time.
-**Don't be a hero:**
-> "I've seen a lot of people who are eager to get crazy and creative in stacking up the lego blocks of the neural net toolbox in various exotic architectures... Resist this temptation strongly in the early stages of your project. I always advise people to simply find the most related paper and copy paste their simplest architecture that achieves good performance."
+#### 3. Overfit
-**Adam as safe starting point:**
-> "In the early stages of setting baselines I like to use Adam with a learning rate of 3e-4. In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate."
+At this stage we should have a good understanding of the dataset and we have the full training + evaluation pipeline working. For any given model we can (reproducibly) compute a metric that we trust. We are also armed with our performance for an input-independent baseline, the performance of a few dumb baselines (we better beat these), and we have a rough sense of the performance of a human (we hope to reach this). The stage is now set for iterating on a good model.
-> "For ConvNets a well-tuned SGD will almost always slightly outperform Adam, but the optimal learning rate region is much more narrow and problem-specific."
+The approach I like to take to finding a good model has two stages: first get a model large enough that it can overfit (i.e. focus on training loss) and then regularize it appropriately (give up some training loss to improve the validation loss). The reason I like these two stages is that if we are not able to reach a low error rate with any model at all that may again indicate some issues, bugs, or misconfiguration.
-**Build complexity incrementally:**
-> "If you have multiple signals to plug into your classifier I would advise that you plug them in one by one and every time ensure that you get a performance boost you'd expect. Don't throw the kitchen sink at your model at the start."
+A few tips & tricks for this stage:
-**Learning rate decay warning:**
-> "If you are re-purposing code from some other domain always be very careful with learning rate decay... your code could secretly be driving your learning rate to zero too early, not allowing your model to converge."
+* **picking the model**. To reach a good training loss you’ll want to choose an appropriate architecture for the data. When it comes to choosing this my #1 advice is: **Don’t be a hero**. I’ve seen a lot of people who are eager to get crazy and creative in stacking up the lego blocks of the neural net toolbox in various exotic architectures that make sense to them. Resist this temptation strongly in the early stages of your project. I always advise people to simply find the most related paper and copy paste their simplest architecture that achieves good performance. E.g. if you are classifying images don’t be a hero and just copy paste a ResNet-50 for your first run. You’re allowed to do something more custom later and beat this.
+* **adam is safe**. In the early stages of setting baselines I like to use Adam with a learning rate of [3e-4](https://twitter.com/karpathy/status/801621764144971776?lang=en). In my experience Adam is much more forgiving to hyperparameters, including a bad learning rate. For ConvNets a well-tuned SGD will almost always slightly outperform Adam, but the optimal learning rate region is much more narrow and problem-specific. (Note: If you are using RNNs and related sequence models it is more common to use Adam. At the initial stage of your project, again, don’t be a hero and follow whatever the most related papers do.)
+* **complexify only one at a time**. If you have multiple signals to plug into your classifier I would advise that you plug them in one by one and every time ensure that you get a performance boost you’d expect. Don’t throw the kitchen sink at your model at the start. There are other ways of building up complexity - e.g. you can try to plug in smaller images first and make them bigger later, etc.
+* **do not trust learning rate decay defaults**. If you are re-purposing code from some other domain always be very careful with learning rate decay. Not only would you want to use different decay schedules for different problems, but - even worse - in a typical implementation the schedule will be based current epoch number, which can vary widely simply depending on the size of your dataset. E.g. ImageNet would decay by 10 on epoch 30. If you’re not training ImageNet then you almost certainly do not want this. If you’re not careful your code could secretely be driving your learning rate to zero too early, not allowing your model to converge. In my own work I always disable learning rate decays entirely (I use a constant LR) and tune this all the way at the very end.
-> "In my own work I always disable learning rate decays entirely (I use a constant LR) and tune this all the way at the very end."
+#### 4. Regularize
-**First layer sanity check:**
-> "To gain additional confidence that your network is a reasonable classifier, I like to visualize the network's first-layer weights and ensure you get nice edges that make sense. If your first layer filters look like noise then something could be off."
+Ideally, we are now at a place where we have a large model that is fitting at least the training set. Now it is time to regularize it and gain some validation accuracy by giving up some of the training accuracy. Some tips & tricks:
----
+* **get more data**. First, the by far best and preferred way to regularize a model in any practical setting is to add more real training data. It is a very common mistake to spend a lot engineering cycles trying to squeeze juice out of a small dataset when you could instead be collecting more data. As far as I’m aware adding more data is pretty much the only guaranteed way to monotonically improve the performance of a well-configured neural network almost indefinitely. The other would be ensembles (if you can afford them), but that tops out after ~5 models.
+* **data augment**. The next best thing to real data is half-fake data - try out more aggressive data augmentation.
+* **creative augmentation**. If half-fake data doesn’t do it, fake data may also do something. People are finding creative ways of expanding datasets; For example, [domain randomization](https://openai.com/blog/learning-dexterity/), use of [simulation](http://vladlen.info/publications/playing-data-ground-truth-computer-games/), clever [hybrids](https://arxiv.org/abs/1708.01642) such as inserting (potentially simulated) data into scenes, or even GANs.
+* **pretrain**. It rarely ever hurts to use a pretrained network if you can, even if you have enough data.
+* **stick with supervised learning**. Do not get over-excited about unsupervised pretraining. Unlike what that blog post from 2008 tells you, as far as I know, no version of it has reported strong results in modern computer vision (though NLP seems to be doing pretty well with BERT and friends these days, quite likely owing to the more deliberate nature of text, and a higher signal to noise ratio).
+* **smaller input dimensionality**. Remove features that may contain spurious signal. Any added spurious input is just another opportunity to overfit if your dataset is small. Similarly, if low-level details don’t matter much try to input a smaller image.
+* **smaller model size**. In many cases you can use domain knowledge constraints on the network to decrease its size. As an example, it used to be trendy to use Fully Connected layers at the top of backbones for ImageNet but these have since been replaced with simple average pooling, eliminating a ton of parameters in the process.
+* **decrease the batch size**. Due to the normalization inside batch norm smaller batch sizes somewhat correspond to stronger regularization. This is because the batch empirical mean/std are more approximate versions of the full mean/std so the scale & offset “wiggles” your batch around more.
+* **drop**. Add dropout. Use dropout2d (spatial dropout) for ConvNets. Use this sparingly/carefully because dropout [does not seem to play nice](https://arxiv.org/abs/1801.05134) with batch normalization.
+* **weight decay**. Increase the weight decay penalty.
+* **early stopping**. Stop training based on your measured validation loss to catch your model just as it’s about to overfit.
+* **try a larger model**. I mention this last and only after early stopping but I’ve found a few times in the past that larger models will of course overfit much more eventually, but their “early stopped” performance can often be much better than that of smaller models.
-## Stage 4: Regularize
+Finally, to gain additional confidence that your network is a reasonable classifier, I like to visualize the network’s first-layer weights and ensure you get nice edges that make sense. If your first layer filters look like noise then something could be off. Similarly, activations inside the net can sometimes display odd artifacts and hint at problems.
-**Primary advice: get more real data**
-> "It is a very common mistake to spend a lot engineering cycles trying to squeeze juice out of a small dataset when you could instead be collecting more data. As far as I'm aware adding more data is pretty much the only guaranteed way to monotonically improve the performance of a well-configured neural network almost indefinitely."
+#### 5. Tune
-**Smaller batch size = more regularization (via batch norm):**
-> "Due to the normalization inside batch norm smaller batch sizes somewhat correspond to stronger regularization. This is because the batch empirical mean/std are more approximate versions of the full mean/std so the scale & offset 'wiggles' your batch around more."
+You should now be “in the loop” with your dataset exploring a wide model space for architectures that achieve low validation loss. A few tips and tricks for this step:
-**Dropout + batchnorm warning:**
-> "Use this [dropout] sparingly/carefully because dropout does not seem to play nice with batch normalization."
+* **random over grid search**. For simultaneously tuning multiple hyperparameters it may sound tempting to use grid search to ensure coverage of all settings, but keep in mind that it is [best to use random search instead](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf). Intuitively, this is because neural nets are often much more sensitive to some parameters than others. In the limit, if a parameter **a** matters but changing **b** has no effect then you’d rather sample **a** more throughly than at a few fixed points multiple times.
+* **hyper-parameter optimization**. There is a large number of fancy bayesian hyper-parameter optimization toolboxes around and a few of my friends have also reported success with them, but my personal experience is that the state of the art approach to exploring a nice and wide space of models and hyperparameters is to use an intern :). Just kidding.
-**Larger model + early stopping:**
-> "I've found a few times in the past that larger models will of course overfit much more eventually, but their 'early stopped' performance can often be much better than that of smaller models."
+#### 6. Squeeze out the juice
----
+Once you find the best types of architectures and hyper-parameters you can still use a few more tricks to squeeze out the last pieces of juice out of the system:
-## Stage 5: Hyperparameter tuning
+* **ensembles**. Model ensembles are a pretty much guaranteed way to gain 2% of accuracy on anything. If you can’t afford the computation at test time look into distilling your ensemble into a network using [dark knowledge](https://arxiv.org/abs/1503.02531).
+* **leave it training**. I’ve often seen people tempted to stop the model training when the validation loss seems to be leveling off. In my experience networks keep training for unintuitively long time. One time I accidentally left a model training during the winter break and when I got back in January it was SOTA (“state of the art”).
-**Random search over grid:**
-> "It is best to use random search instead [of grid search]. Intuitively, this is because neural nets are often much more sensitive to some parameters than others. In the limit, if a parameter a matters but changing b has no effect then you'd rather sample a more thoroughly than at a few fixed points multiple times."
+#### Conclusion
----
-
-## Stage 6: Squeeze performance
-
-**Don't stop early:**
-> "I've often seen people tempted to stop the model training when the validation loss seems to be leveling off. In my experience networks keep training for unintuitively long time. One time I accidentally left a model training during the winter break and when I got back in January it was SOTA."
-
-**Ensembles:**
-> "Model ensembles are a pretty much guaranteed way to gain 2% of accuracy on anything."
+Once you make it here you’ll have all the ingredients for success: You have a deep understanding of the technology, the dataset and the problem, you’ve set up the entire training/evaluation infrastructure and achieved high confidence in its accuracy, and you’ve explored increasingly more complex models, gaining performance improvements in ways you’ve predicted each step of the way. You’re now ready to read a lot of papers, try a large number of experiments, and get your SOTA results. Good luck!
diff --git a/docs/evidence/kidger_just_know_stuff.md b/docs/evidence/kidger_just_know_stuff.md
index 5688c64..44424fe 100644
--- a/docs/evidence/kidger_just_know_stuff.md
+++ b/docs/evidence/kidger_just_know_stuff.md
@@ -1,17 +1,230 @@
# Just Know Stuff (how to achieve success in an ML PhD) — Patrick Kidger
-Source: https://kidger.site/thoughts/just-know-stuff/ (2023-01-26). Cached excerpt from the "Software development" section, verbatim.
+Source: https://kidger.site/thoughts/just-know-stuff/ (2023-01-26)
+Fetched-via: r.jina.ai reader, 2026-08-15 (CLAUDE agent)
+Fetch-status: full post text. Supersedes the earlier "Software development" section excerpt. (CLAUDE agent)
+
+Why it matters here: the "never accept the kludge" posture, and the claim that most ML researchers would not pass muster as junior developers.
---
-> Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don't cripple things only because some other bug stops them from doing so.
+_Posted on January 26, 2023_
+## Introduction
-> This is a systemic professional failing. As an (applied) ML researcher, the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.
+So I recently completed my PhD in Mathematics from the University of Oxford. (Hurrah! It was so much fun.)
-> So, how to improve? First of all, never accept the kludge.
+In 2-and-a-bit years I wrote 12 papers, received 4139 GitHub stars, got 3271 Twitter followers, authored 1 textbook – doing double-duty as my thesis – and got the coveted big-tech job-offer.
-> You've messed up your Git repo? Figure out the commands to fix it... don't just delete it and clone from the remote.
+On Neural Differential Equations
-> Focus on writing clean code, based around orthogonal abstractions. When the code starts getting messy - and it will - be willing to refactor your code into something more legible. Avoid both spaghetti code and ravioli code.
+If you’re interested in a textbook on Neural Differential Equations with a smattering of scientific computing, then [my thesis is available online.](https://arxiv.org/abs/2202.02435)
-> When the documentation is inadequate, look at their source code.
+Quite a few folks seem to have looked at this, and messaged me – mostly on [Twitter](https://twitter.com/PatrickKidger) or [Mastodon](https://fosstodon.org/@PatrickKidger) – asking for advice on **how to achieve success in a machine learning PhD?**
+
+Each time my answer is: **Just Know Stuff.**
+
+Now, I don’t think “Just Know Stuff” is a terribly controversial opinion – undergraduate classes are largely based around imparting knowledge; the first year of a new PhD’s life is usually spent reading up on the literature – but from the number of questions I get it would seem that this is something worth restating.
+
+Know your field inside-out. Know as much about adjacent fields (in math, statistics, …) as you can. Don’t just know how to program; know how to do software development. Know the mathematical underpinnings your work is built upon. And so on and so on. Indeed: [possessing a technical depth of knowledge is how you come up with new ideas and learn to recognise bad ones](https://kidger.site/thoughts/how-to-handle-a-hands-off-supervisor).
+
+This does beg the follow-up question: **what is worth knowing? What is worth learning?**
+
+And the answer to _that_ is what I started repeating to all of you folks messaging me. But then that started taking up way too much time, mostly because I write way too much. So now I’m writing this post instead – this way I’ll only have to write way too much only once!
+
+The following is a highly personal list of the things I found to be useful during my PhD, and which I think are of a broad enough appeal that they probably represent a reasonable core of knowledge for those just starting an ML PhD. The following is by no mean exhaustive, and you should certainly expect to add a lot of domain-specific stuff on top of this. But perhaps the following is a useful starting point.
+
+_This list is targeted towards early-stage PhD or pre-PhD students. If you’re late-stage and reading through this thinking “yeah, of course I know this stuff”, then well… that’s the point!_
+
+## Machine learning
+
+* Know both forward- and reverse-mode autodifferentiation. (Nice reference: Appendix A of my thesis. ;) )
+ * Write some custom gradient operations in both PyTorch and JAX.
+ * Look up “optimal Jacobian accumulation” on the autodifferentiation page on Wikipedia.
+ * Optional: learn how JAX derives reverse-mode autoderivatives by combining partial evaluation, forward-mode-autodifferentiation, and transposition.
+ * Optional: why is the computation of a divergence computationally expensive using autodifferentiation? Learn Hutchinson’s trace estimator. (Why is that efficient?) Learn the [Hutch++ trace estimator](https://ram900.hosting.nyu.edu/hutchplusplus/). (Which is surprisingly poorly known.)
+
+* What is meant by Strassen’s algorithm? Learn how matrix multiplies are actually done in practice. Learn Winograd convolutions.
+* Write your own implementation of a convolutional layer. Write your own implementation of multihead attention.
+* Know the universal approximation theorem. (I recommend Leshno et al. 1993 or Pinkus 1999 as references. _Not_ the much-more-frequently cited references to older results by Cybenko or Hornik, who give much weaker results.)
+ * Optional: if you’re really keen then look up the modern line of work on [alternate universal approximation theorems](https://arxiv.org/abs/1905.08539).
+
+* Learn the basics of graph neural networks. (E.g. what is oversmoothing?) How do these generalise CNNs?
+* Learn modern Transformer architectures. Look up recent papers [(or implementations)](https://github.com/lucidrains) to see some of the more common architectural tricks. Build a toy implementation.
+* Learn U-Nets. Build a toy implementation.
+* Know how residual networks are discretised ordinary differential equations.
+* know how Gated Recurrent Units (GRUs) are also discretised differential equations.
+* Know how stochastic gradient descent is also a discretised differential equation too! (Yes, including the “stochastic”: that’s a Monte-Carlo discretisation of an expectation.) These are [gradient flows](https://francisbach.com/gradient-flows/).
+* Know what is meant by the manifold hypothesis.
+* Learn the basics of policy gradients. Implement PPO to solve cart-pole. ([Spinning up](https://spinningup.openai.com/en/latest/) is a great resource.)
+* Learn KL divergence, Wasserstein distance, MMD distance.
+* Learn normalising flows, VAEs, WGANs, score-based diffusion models. [Implement a basic score-based diffusion from scratch.](https://docs.kidger.site/equinox/examples/score_based_diffusion/)
+* Try the basics of distributed training of a model. (Over multiple GPUs; multiple computers.) Start with `jax.pmap`.
+* Know how to do hyperparameter optimisation via Bayesian optimisation. My favourite library for this is [Ax](https://ax.dev/).
+ * Optional: try doing this in a distributed fashion, with a main thread sending hyperparameter jobs to different machines, and receiving results back. (The “Service API” for Ax is the appropriate tool here.)
+
+* Learn the formulae for Adadelta, Adam, etc. What were the innovations for each optimiser? (Momentum, second moments, …) What are some of the newer ones that are now being used (Adabelief, RAdam, NAdamW, … etc. etc. – this is a flavour-of-the-month kind of field.)
+* Learn why we use first-order optimisation techniques (SGD and friends), rather than anything else. (Why not Gauss–Newton? Why not Newton–Raphson? Why not Levenberg–Marquardt?) On that note, let’s move on to…
+
+## (Elementary) scientific computing
+
+* …start by learning all of those algorithms I just mentioned as well (they’re all nonlinear solvers).
+* Learn QR decompositions, LU decompositions, SVD decompositions, Cholesky decompositions.
+ * Solve linear systems via each of the above decompositions. (Recognise that this is better than inverting a matrix.) Learn the varying computational costs and stabilities of the different ways of doing this. (SVD -> Cholesky-> QR -> LU.)
+ * Reduce linear least squares to linear solves via the normal equations. Know that this squares the condition number. Recognise that this is the textbook approach to fitting a linear model.
+
+* Learn what is meant by the [Moore–Penrose pseudoinverse](https://en.m.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse) of a matrix.
+* Learn the basics of numerical differential equation solvers:
+ * Euler’s method
+ * Heun’s method
+ * Optional: Implicit Euler method. Know that it works provided `hL < 1`, where `h` is the step size and `L` is the Lipschitz constant of the vector field. (Know the contraction mapping theorem.)
+ * Optional: other diffeq solvers, e.g. explicit Runge–Kutta methods. [This is a nice summary of when to use each.](https://docs.kidger.site/diffrax/usage/how-to-choose-a-solver/)
+
+* Know Monte-Carlo sampling. Know Quasi Monte-Carlo sampling. Know the convergence rates for both.
+* Learn what is meant by quadrature.
+* Learn Chebyshev polynomials.
+* Know the quirks of floating-point arithmetic: non-associativity, catastrophic cancellation, the impossibility of representing some integers, that you should not compare floats via equality, the meaning of numerical stability.
+ * This is the reason `expm1` and `logsumexp` exist as standalone functions.
+
+* Optional: learn wavelets.
+* Optional: sparsity.
+ * The different kinds of sparse format (CSC, COO, …);
+ * Sparse linear solvers (e.g. iterative/Krylov methods);
+ * Linear preconditioners.
+
+There’s (a lot) more scientific computing out there, but I’m writing for an ML audience here. The above is perhaps a minimum worth being conversant on.
+
+## Software development.
+
+_(Those of you deriving PAC-Bayes bounds, you might be able to skip this section. Unless you want an industry job post-PhD, that is.)_
+
+Academic software is almost always a poorly-maintained kludge of leaky abstractions, awful formatting, and bugs that don’t cripple things only because some other bug stops them from doing so.
+
+_This is a systemic professional failing._ As an (applied) ML researcher, the overwhelming majority of your time will be spent in front of a screen, staring at code. And yet most of you (yes, you) would not pass muster as a junior developer.
+
+So, how to improve? First of all, never accept the kludge.
+
+* You’ve messed up your Git repo? Figure out the commands to fix it… don’t just delete it and clone from the remote. ([https://xkcd.com/1597](https://xkcd.com/1597))
+* You’ve written messy code? Assuming you’re using Python: learn PEP8, pre-commit, Black, flake8, isort. (Or [ruff](https://github.com/charliermarsh/ruff) if you’re ahead-of-the-curve.)
+ * Feel free to steal the configs from [one of my repositories](https://github.com/patrick-kidger/equinox).
+
+* Your code is too slow? Learn a more performant language (C++, Rust, Triton) and write things there.
+* Focus on writing clean code, based around orthogonal abstractions. When the code starts getting messy – and it will – be willing to refactor your code into something more legible. Avoid both spaghetti code and ravioli code.
+
+And returning to the overall theme:
+
+* Learn Python to an advanced enough level that you know what descriptors, weak references, and metaclasses are.
+ * Learn what closures are.
+
+* Learn how to build your own Python package and push it to PyPI.
+* Learn both PyTorch and JAX.
+ * When the documentation is inadequate, look at their source code.
+ * Optional: [reimplement JAX core transforms from scratch](https://jax.readthedocs.io/en/latest/autodidax.html).
+
+* Learn some object oriented design patterns. At least as far as dependency inversion and factories.
+* Learn some C/C++.
+ * Pass-by-reference vs pass-by-copy. Pointers.
+ * Write some bindings for using these from Python. (In ML, this is easiest using PyTorch+LibTorch+pybind11.)
+ * Optional: learn some OpenMP.
+
+* Learn some Julia. Understand why multiple dispatch is so cool, and how this helps build numerical programs. Write some macros and learn what is meant by homoiconicity.
+* Learn some Haskell. Learn functional programming. Learn some type theory. (Learn the difference between a sum type and a union type.) Learn what is meant by monads. Learn what is meant by referential transparency.
+ * Optional: look up Koka and learn what is meant by algebraic effects.
+ * Optional: look up Idris or Liquid Haskell and learn what is meant by dependent types.
+
+* Learn some Common Lisp or Scheme. Understand why its code is the same as its abstract syntax tree (AST). Write some macros and _really_ understand homoiconicity.
+* What is meant by generic programming? What is meant by variadic generics? When are these helpful? ([Cough cough](https://github.com/google/jaxtyping).)
+* Learn big-O notation for computational complexity. Learn how a hash map is implemented. Look up how a Python dict is implemented. Look up the exponential memory allocation trick for continually appending to e.g. a Python list.
+* Know dynamic programming. (The classic example here is the Fibonacci numbers.) Learn to recognise when a problem can be solved this way. Recognise the equivalence between dynamic programming and caching (a la Python’s `functools.lru_cache`).
+* Have a read of programming blogs. (Personally, this is how I procrastinate from more serious work.)
+* Learn how to collaborate on code! Typically via GitHub-style pull-request workflows. We’re not going to hire you without evidence we can work with you.
+* Know how to write tests. Integrate them into a CI/CD system e.g. GitHub Actions. (Once again, feel free to [steal from one of my repos](https://github.com/patrick-kidger/diffrax/tree/main/.github/workflows/).)
+
+There’s loads more I could add here: learn some compiler theory (tail call optimisation, peephole optimisation, …). Learn distributed computing. Learn different database systems. Learn a bit about how a CPU works (L1/L2/L3 caches, CPU cycles, vectorisation, branch prediction, some basic assembly, etc.). Learn other programming languages (Nim, Zig, Dex, …) Learn when to use a few mildly nontrivial data structures (heaps, btrees, ropes, …)
+
+You don’t need to become a serious software developer. (i.e. knowing all of the above list and substantially more.) Just don’t write code that makes my eyes bleed.
+
+In nearly every respect I’d actually recommend against the university-taught courses for much of the above list. YMMV, but these are usually pretty poor. (Perhaps because they’re taught by academics… who, as already discussed, don’t usually know what they’re doing here. E.g. C++ courses that taught the `new` and `delete` operators as good practice…) Try the internet instead.
+
+I recommend reading programming forums, YouTube videos from programming conferences, and programming blogs.
+
+## Mathematics
+
+* Some basics.
+ * Convex functions (recognise that this is a way to bound a nonlinear function by an easier-to-understand linear function).
+ * Lipschitz functions (these have already appeared several times above: in WGANs, the implicit Euler method, the contraction mapping theorem).
+ * The meaning of injectivity, surjectivity, bijectivity.
+
+* Please, please: learn some probability via measure theory. You’ll start reading machine learning papers wondering how people ever express themselves precisely without it. The entire field seems to be predicated around writing things like as if that’s somehow meaningful notation.
+* Likewise, learn integration through measure theory. At least as far as Fubini’s theorem, the Leibniz Integral Rule, and what is meant by absolute continuity of measures.
+ * Optional: If you’re keen then go as far as Radon–Nikodym derivatives. (Which appears in the definition of the KL divergence, for example.)
+ * Optional: the meaning of almost-everywhere. Recognise that ReLU is almost-everywhere differentiable.
+ * Optional: Alexandrov’s Theorem.
+
+* Topology is a great topic to learn the basics of, as this underpins nearly all of modern mathematics: open sets, closed sets, compactness, continuous functions, etc.
+ * Optional: there’s some very enjoyable “counterexamples in topology” books out there, that will melt your brain into a variety of interesting shapes.
+
+* Analysis. A topic close to my heart, as this was my primary field of study at university.
+ * Real analysis, at least the basics: epsilon-delta, the definition of differentiation, that continuous functions on a compact set attain their bounds, etc.
+ * Functional analysis, once again at least the basics: at least as far as the Weierstraß Approximation Theorem.
+ * Ordinary differential equations; at least as far as linearisation around equilibria. (Probably the engineers have some good not-too-dense reference texts for these.)
+ * Fourier series.
+ * Div, grad, curl and all that.
+
+* Optional: any number of slightly more specialised, but still very widely applicable, fields. For example:
+ * Differential geometry
+ * Optimal transport.
+ * Stochastic calculus, if you do anything to do with time series. (Or score-based diffusion models.)
+ * Statistical physics.
+ * Perturbation theory. Much of machine learning is morphing into a branch of applied mathematics. And as my old fluid dynamics lecturer commented, you can’t be a card-carrying applied mathematician without knowing perturbation theory.
+
+## Statistics
+
+Actually, I’m going to admit to something here: my statistics is nowhere near as strong as I’d like it to be. I think there’s probably a lot that should be added to the following list.
+
+* All the usual introductory stuff: log-likelihoods, BLUE, cross-validation, confidence intervals, random forests, XGBoost etc. etc.
+ * Regularisation: Tikhonov/ridge/L2 regularisation, sparsity/L1 regularisation, weight decay. The equivalence between regularised maximum likelihood and maximum a-posteriori.
+
+* Variance minimisation:
+ * Antithetic sampling;
+ * Importance sampling (_cough_ Radon–Nikodym derivatives again _cough_);
+ * Quasi Monte-Carlo (again);
+ * Control variates.
+
+* Linear-time biased Monte-Carlo approximations to MMDs. Quadratic-time unbiased Monte-Carlo approximations to MMDs.
+ * It may have gone out of fashion, but the basics of kernel theory.
+
+* Markov Chain Monte-Carlo. Hamiltonian Monte-Carlo.
+ * Relatedly, Gaussian “soap bubbles” in high dimensions, and “typical sets” in MCMC. Anything to build high-dimensional intuition is great. [This](https://stanislavfort.github.io/blog/sphere-spilling-out/) is a fun example. Can you figure out what’s wrong with the final picture?
+
+## That’s a lot of stuff
+
+That’s quite a long list.
+
+Don’t expect to cover all of that in a few months; this is something that should happen over the next few years. To be precise, the above is more-or-less what I think deserves to be known by most people by the end of their PhD. You should naturally expect to also know your own subfield, whatever that is, inside-out.
+
+This list is noticeably biased towards the things I happen to be more involved in, which I guess is unsurprising.
+
+(For example I haven’t mentioned Vapnik–Chervonenkis dimensions or Gaussian processes anywhere. Some may disagree with me but I think it’s possible to get by without knowing VC dimensions these days. And I have a personal bias against Gaussian processes.)
+
+So, season to taste. Probably a few of you are reading this wondering how it could have slipped my mind to add your favourite X, Y or Z to that list! (Object detection, scaling laws for large models, subquadratic attention mechanisms, symbolic regression, …)
+
+It’s worth noting that “by the end of their PhD” is kind of an arbitrary deadline. One never really stops learning. I certainly look back what I’ve written a couple of years ago, and see noticeable improvements I would make if I were to do it again. And looking forward, I have a list of things I intend to learn more about. (Currently: algebraic effects, deeper knowledge of Rust, and microbiology.)
+
+## Interesting parts of the internet to hang out in.
+
+When it comes to Just Knowing Stuff, it’s great to get a sense of the general Zeitgeist in the ML community, and also the rest of the tech community at large. These are a few of my favourite spots:
+
+* Twitter;
+* Mastodon;
+* Hacker News;
+* /r/machinelearning
+* YouTube, in particular the recorded talks from programming conferences;
+* Programming/software blogs;
+* Forums for software you use regularly
+ * including the GitHub “issues” and “discussions” tabs
+
+## Conclusion
+
+Those of you who already have research experience, and who are reading this: what would be your personal Just Know Stuff list? Do you think mine is fair?
+
+Write your own list and/or let me know on [Twitter](https://twitter.com/PatrickKidger) or [Mastodon](https://fosstodon.org/@PatrickKidger).
diff --git a/docs/evidence/koaning_bad_labels.md b/docs/evidence/koaning_bad_labels.md
index d567ef3..8c27c85 100644
--- a/docs/evidence/koaning_bad_labels.md
+++ b/docs/evidence/koaning_bad_labels.md
@@ -1,33 +1,93 @@
# Bad Labels — Vincent D. Warmerdam (koaning)
-Source: https://koaning.io/posts/labels/ (2021-09-02). Cached copy for the ML-debugging skill.
+Source: https://koaning.io/posts/labels/ (2021-09-02)
+Fetched-via: r.jina.ai reader, 2026-08-15 (CLAUDE agent)
+Fetch-status: full post text, including the Google Emotions worked example, the confusion table and the cleanlab snippet that the earlier cached copy dropped. (CLAUDE agent)
+
+Why it matters here: benchmark labels are often wrong, and a cheap high-bias model sorted by predicted probability finds the wrong ones.
---
+
+
I write a lot of blogposts on why you need more than grid-search to properly judge a machine learning model. In this blogpost I want to demonstrate yet another reason; labels often seem to be wrong.
-What I'll describe here is also available as a course on calmcode.io.
+What I'll describe here is also available as a course on [calmcode.io](https://calmcode.io/bad-labels/introduction.html).
## Bit of Background
-It turns out that bad labels are a *huge* problem in many popular benchmark datasets. To get an impression of the scale of the issue, just go to labelerrors.com. It's an impressive project that shows problems with many popular datasets; CIFAR, MNIST, Amazon Reviews, IMDB, Quickdraw and Newsgroups just to name a few. It's part of a research paper (https://arxiv.org/abs/2103.14749) that tries to quantify how big of a problem these bad labels are.
+It turns out that bad labels are a _huge_ problem in many popular benchmark datasets. To get an impression of the scale of the issue, just go to [labelerrors.com](https://labelerrors.com/). It's an impressive project that shows problems with many popular datasets; CIFAR, MNIST, Amazon Reviews, IMDB, Quickdraw and Newsgroups just to name a few. It's part of a [research paper](https://arxiv.org/abs/2103.14749) that tries to quantify how big of a problem these bad labels are.
+
+
+
+The table from the paper gives a nice summary. It's a huge problem.
The issue here isn't just that we might have bad labels in our training set, the issue is that it appears in the validation set. If a machine learning model can become state of the art by squeezing another 0.5% out of a validation set one has to wonder. Are we really making a better model? Or are we creating a model that is better able to overfit on the bad labels?
+## Another Dataset
+
+The results from the paper didn't surprise me much, but it did get me wondering how easy it might be for me to find bad labels in a dataset myself. After a bit of searching I discovered the [Google Emotions](https://arxiv.org/abs/2005.00547) dataset. This dataset contains text from Reddit (so expect profanity) with emotion tags attached. There are 28 different tags and a single text can belong to more than one emotion
+
+The dataset also has an [paper about it](https://arxiv.org/abs/2005.00547) which explains how the dataset came to be. It explains what steps have been taken to make the dataset robust.
+
+* There are 82 raters involved n labelling this dataset. Each example should have been at least 3 people checking it. The paper mentions that all the folks who rated were from India but spoke English natively.
+* An effort was made to remove subreddits that were not safe for work or that contained too much vulgar tokens (according to a predefined word-list).
+* An effort was made to balance different subreddits such that larger subreddits wouldn't bias the dataset.
+* An effort was made to remove subreddits that didn't offer a variety of emotions.
+* An effort was made to mask names of people as well as references to religions.
+* An effort was made to, in hindsight, confirm that there is sufficient interrated correlation.
+
+All of this amounts to quite a lot of effort indeed. So how hard would it be to find bad examples here?
+
## Quick Trick
Here's a quick trick seems worthwhile. Let's say that we train a model that is very general. That means high bias, low variance. You may have a lower capacity model this way, but it will be less prone to overfit on details.
-After training such a model, it'd be interesting to see where the model disagrees with the training data. These would be valid candidates to check, but it might result in list that's a bit too long for comfort. So to save time you can can sort the data based on the `predict_proba()`-value. When the model gets it wrong, that's interesting, but when it *also* associates a very low confidence to the correct class, that's an example worth double checking.
+After training such a model, it'd be interesting to see where the model disagrees with the training data. These would be valid candidates to check, but it might result in list that's a bit too long for comfort. So to save time you can can sort the data based on the `predict_proba()`-value. When the model gets it wrong, that's interesting, but when it _also_ associates a very low confidence to the correct class, that's an example worth double checking.
+
+So I figured I would try this trick on the Google emotions dataset to see what would happen. I tried predicting a few tags chosen at random and tried using this sorting trick to to see how easy it was to find bad labels. For each tag, I would apply my sorting to see if I could find bad labels in the top 20 results.
+
+Here's some of the results:
+
+??? note "Label = 'love'" - Weird game lol - Looks like it. I didn't make it, I just found it. - Wow, you people...
+
+??? note "Label = 'not love'" - Very nice!! I love your art! What journal is this? I love the texture on the pages. - love love love this. so happy for the both of you. - I LOVE IT, I would love if they will make season 2... I really enjoyed it - I love this, my wife told me about something she read on reddit yesterday and I was like.... well just like ol [NAME] here!!!
+
+??? note "Label = 'curiosity'" - I actually enjoy doing this on my own. Am I weird? - She probably has a kid by now. - So much time saved. Not. - Didn't you just post this and people told you it was dumb and not meant for this sub?
+
+??? note "Label = 'not curiosity'" - I cant wait. I'm curious if it will give us any more insight into the incident other than what we already know. - Why do you guys hate [NAME]? I’m neutral leaning slightly positive on him. Just curious why the strong negative opinion? - What does that even mean? How does one decide right or wrong with something so vague? - Wait, this is actually a really interesting point. That could/should play a factor if he‘s a legitimate candidate. - Is it weed? I’m curious to ask if you know what weed smells like?
+
+??? note "Label = 'not excitement'" - I am inexplicably excited by [NAME]. I get so excited by how he curls passes - Omg this is so amazing ! Keep up the awesome work and have a fantastic New Year ! - I just read your list and now I can't wait, either!! Hurry up with the happy, relieved and peaceful onward and upward!! Congratulations😎 - I absolutely love that idea. I went on an anniversary trip with a couple once and it was amazing! We had so much fun. - Happy New Year! Looks like you had a great time there! Cheers! Here’s to a great 2019 hopefully in both baseball and life!
+
+??? note "Label = 'not joy'" - Happy cake day! Have a great day and year, cheers. - It's wonderful and gives me happy happy feels - Happy to hear this exciting news. Congratulations on your fun-filled morning. - It's good, good, good, good - good good good! - My son and I both enjoy taking pictures. It gives us pleasure. Part of the fun for us on vacation is taking pictures of new things.
+
+??? note "Label = 'not gratitude'" - Thanks. Nice input as always. - Thanks. I didn't quite get it from the original. Appreciate the time. - This made my hump day. Thank you good sir - Excellent work thank you for this. This is why I love Reddit. - You’re amazing thank you so much!! :)
+
+I don't know about you, but many of these examples seem wrong.
+
+## Friggin' Strange
+
+Before pointing a finger, it'd be good to admit that interpreting emotion isn't a straightforward task. At all. There's context and all sorts of cultural interpretation to consider. It's a tricky task to define well.
+
+The paper also added a disclaimer to the paper to make people aware of potential flaws in the dataset. Here's a part of it:
+
+> We are aware that the dataset contains biases and is not representative of global diversity. We are aware that the dataset contains potentially problematic content. Potential biases in the data include: Inherent biases in Reddit and user base biases, the offensive/vulgar word lists used for data filtering, inherent or unconscious bias in assessment of offensive identity labels, annotators were all native English speakers from India. All these likely affect labeling, precision, and recall for a trained model.
+
+Adding this disclaimer is fair. That said. It really feels just a bit too weird that it was _that_ easy for me to find examples that really seem so clearly wrongly labeled. I didn't run through the whole dataset, so I don't have a number on the amount of bad labels but I'm certainly worried now. Given the kind of label errors, I can certainly imagine that my grid-search results are skewed.
## What does this mean?
-The abstract of the [Northcutt et al.] paper certainly paints a clear picture of what this exercise means for state-of-the-art models:
+The abstract of the paper certainly paints a clear picture of what this exercise means for state-of-the-art models:
> We find that lower capacity models may be practically more useful than higher capacity models in real-world datasets with high proportions of erroneously labeled data. For example, on ImageNet with corrected labels: ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6%. On CIFAR-10 with corrected labels: VGG-11 outperforms VGG-19 if the prevalence of originally mislabeled test examples increases by 5%. Traditionally, ML practitioners choose which model to deploy based on test accuracy -- our findings advise caution here, proposing that judging models over correctly labeled test sets may be more useful, especially for noisy real-world datasets.
## So what now?
-More people should do check their labels more frequently. ... if you're looking for a simple place to start, check out the cleanlab project (https://github.com/cgnorthcutt/cleanlab). It's made by the same authors of the labelerrors-paper and is meant to help you find bad labels.
+More people should do check their labels more frequently. Anybody is free to try out any trick that they like, but if you're looking for a simple place to start, check out the [cleanlab project](https://github.com/cgnorthcutt/cleanlab). It's made by the same authors of the labelerrors-paper and is meant to help you find bad labels. I've used it a bunch of times and I can confirm that it's able to return relevant examples to double-check.
-For everyone; maybe we should spend a less time tuning parameters and instead spend it trying to get a more meaningful dataset.
+Here's the standard snippet that you'd need:
+
+`from cleanlab.pruning import get_noise_indices# Find label indicesordered_label_errors = get_noise_indices( s=numpy_array_of_noisy_labels, psx=numpy_array_of_predicted_probabilities, sorted_index_method='normalized_margin', # Orders label errors)# Use indices to subset dataframeexamples_df.iloc[ordered_label_errors]`
+It's not a lot of effort and it feels like such an obvious thing to check going forward. The disclaimer on the Google Emotions paper checks a lot of boxes, but imagine that in the future they'd add "we checked out labels with cleanlab before releasing it". For a dataset that's meant to become a public benchmark, it'd sure be a step worth adding.
+
+For everyone; maybe we should spend a less time tuning parameters and instead spend it trying to get a more meaningful dataset. If working at [Rasa](https://rasa.com/) is teaching me anything, it's that this would be time well spent.
diff --git a/docs/evidence/sanh_simple_considerations_hf_2021.md b/docs/evidence/sanh_simple_considerations_hf_2021.md
index 5da71f5..d28007c 100644
--- a/docs/evidence/sanh_simple_considerations_hf_2021.md
+++ b/docs/evidence/sanh_simple_considerations_hf_2021.md
@@ -3,59 +3,112 @@
**Source:** Victor Sanh, Hugging Face Blog, February 25, 2021
**URL:** https://huggingface.co/blog/simple-considerations
**Author:** Victor Sanh (Hugging Face research scientist, author of DistilBERT)
+**Fetched-via:** raw post markdown from https://raw.githubusercontent.com/huggingface/blog/main/simple-considerations.md, 2026-08-15. r.jina.ai returns HTTP 403 for huggingface.co and markitdown returns the page buried in HF site chrome. (CLAUDE agent)
+**Fetch-status:** full post text. Supersedes the earlier quote-and-bullet excerpt. The skill-authored survey of the post's outbound links is kept at the end. (CLAUDE agent)
+
+Why it matters here: a practitioner restatement of the Karpathy recipe, plus tokenizer-output checking and a list of the implementation errors that bite most often.
---
-## Core practices (overlaps heavily with Karpathy 2019 recipe)
+
-**Data first:**
-> "the very first step of building a neural network is to put aside machine learning and simply focus on your data"
+Photo by [Henry & Co.](https://unsplash.com/@hngstrm?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/builder?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)
-**Overfit test:**
-> "it is a good habit when you think you have finished implementing to overfit a small batch of examples (16 for instance). If your implementation is (nearly) correct, your model will be able to overfit and remember these examples by displaying a 0-loss (make sure you remove any form of regularization such as weight decay)."
+# 🚧 Simple considerations for simple people building fancy neural networks
-**Baselines:**
-> "Start as simple as possible to get a sense of the difficulty of your task and how well standard baselines would perform."
-> "it is sometimes hard to understand if your performance comes from a bug in your model/code or is simply limited by your model's expressiveness"
+
+As machine learning continues penetrating all aspects of the industry, neural networks have never been so hyped. For instance, models like GPT-3 have been all over social media in the past few weeks and continue to make headlines outside of tech news outlets with fear-mongering titles.
+
+
+
+
+
An article from The Guardian
+
