diff --git a/docs/evidence/bekman_ml_engineering_instabilities.md b/docs/evidence/bekman_ml_engineering_instabilities.md index 229ce33..87d33c9 100644 --- a/docs/evidence/bekman_ml_engineering_instabilities.md +++ b/docs/evidence/bekman_ml_engineering_instabilities.md @@ -1,42 +1,296 @@ -Source: https://github.com/stas00/ml-engineering — training/instabilities/README.md, training/instabilities/training-loss-patterns.md, debug/README.md (master branch) -Title: "Machine Learning Engineering Open Book" — Stas Bekman (BLOOM-176B / IDEFICS-80B training lead at HF, ex-PyTorch) -Fetched-via: curl of raw markdown from github, 2026-06-11 -Fetch-status: verbatim excerpts +Source: https://github.com/stas00/ml-engineering - training/instabilities/README.md, training/instabilities/training-loss-patterns.md, debug/README.md (master branch) +Title: "Machine Learning Engineering Open Book" - Stas Bekman (BLOOM-176B / IDEFICS-80B training lead at HF, ex-PyTorch) +Fetched-via: curl -sL of the three raw markdown files, 2026-08-15 (CLAUDE agent) +Fetch-status: verbatim, full text of all three pages, concatenated with a heading per file. Replaces the earlier excerpts (CLAUDE agent) -# ML Engineering Open Book — instabilities and loss patterns (excerpts) +# ==== training/instabilities/README.md ==== -From "Understanding Training Loss Patterns": +# Avoiding, Recovering From and Understanding Instabilities -> Training loss plot is similar to the heart beat pattern - there is the good, the bad and you-should-worry one. After studying many training loss trajectories one develops an intuition to explain various loss behaviors during one's training and how to act on those. +Sub-sections: -> I warn you that the "Understanding" in the title of this section is overloaded since very often we don't really understand why certain types of spikes happen. Here "understanding" refers to recognizing various patterns. We then usually have techniques to overcome the bad patterns and bring the training successfully to the finish line. +* [Understanding Training Loss Patterns](training-loss-patterns.md) - types of spikes, divergences, grokking moments, resumes, etc. -> Thus you will find here a gallery of training loss patterns sometimes with real explanations, but more often than not educated guesses to what might be happening. +## Learning from Training Logbooks -The pre-BLOOM 104B failure story ("A very failed training"): +The best learning is to read [Publicly available training LLM/VLM logbooks](../../resources/README.md#publicly-available-training-llmvlm-logbooks) because there you can see exactly what happened and how the problem has been overcome. -> Prior to starting BLOOM-176B training we did multiple experiments with the 104B model. We failed to figure out how to not diverge very early on. [...] As you can see many attempts were made, many techniques were applied (see chronicles). We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference. -On loss spikes ("Main types of loss spikes"): +## STD Init -> In general there are 3 types of loss spikes: 1. Fast recovering spikes 2. Slow recovering spikes 3. Not fully recovering spikes -> -> The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites. +Correctly initializing the initial distribution of the tensors can have a tremendous impact on training's stability. The `std` value isn't fixed and depends on the hidden dimension size. -From "Avoiding, Recovering From and Understanding Instabilities" — the init-std story: +This proved to be a very crucial setting in our pre-BLOOM 104B experiments and we couldn't break past the first few thousands iterations until we figured out that the 0.02 default `--init-method-std` in Megatron-LM was a way too big for our model. -> Correctly initializing the initial distribution of the tensors can have a tremendous impact on training's stability. The `std` value isn't fixed and depends on the hidden dimension size. -> -> This proved to be a very crucial setting in our pre-BLOOM 104B experiments and we couldn't break past the first few thousands iterations until we figured out that the 0.02 default `--init-method-std` in Megatron-LM was a way too big for our model. +We referred to these two sources: -(They settled on the 530B paper's `sqrt(1/(NHIDDEN*3))`: "for NHIDDEN=14336 the math was sqrt(1/(14336*3)) = 0.00482 and that's what we used. It surely wasn't the only reason why we had no stability issues during BLOOM-176B training, but I think it was one of the crucial ones.") +1. "Transformers without Tears" paper https://arxiv.org/abs/1910.05895 prescribes: `sqrt(2/(NHIDDEN*5))` -On PaLM's spikes ("'Bad' combination of data batch and model parameter state"): +2. The 530B training paper https://arxiv.org/abs/2201.11990 they used an even smaller init formula: `sqrt(1/(NHIDDEN*3))` -> PaLM team observed dozens of loss spikes at "highly irregular intervals" when training larger models. While they were not able to track down the root cause, they mitigated the issue by restarting from an earlier checkpoint and skipping potentially problematic data batches. +and decided to go with the 530B one as it leads to an even smaller init value. -On reading training logbooks: +To make it easier to compare the two formulas, they can be rewritten as: +1. `sqrt(0.4000/NHIDDEN)` +2. `sqrt(0.3333/NHIDDEN)` -> The best learning is to read Publicly available training LLM/VLM logbooks because there you can see exactly what happened and how the problem has been overcome. +Thus for `NHIDDEN=14336` the math was `sqrt(1/(14336*3)) = 0.00482` and that's what we used. It surely wasn't the only reason why we had no stability issues during BLOOM-176B training, but I think it was one of the crucial ones. -Debug section index (debug/README.md) — guides for: Debugging PyTorch programs; Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs; Network Debug; Troubleshooting NVIDIA GPUs; Underflow and Overflow Detection; plus tools (torch-distributed-gpu-test.py, NicerTrace). + +## Numerical instabilities + +See also [Detecting problematic tensor values](../../debug/pytorch.md#detecting-problematic-tensor-values) and [Underflow and Overflow Detection](../../debug/pytorch.md#underflow-and-overflow-detection) in the debugging chapter for tooling to locate `inf`/`nan` values. + +Certain mathematical operations could be unstable when dealing with low precision numbers. + +For example, please see this very interesting [PyTorch guide on numerical stability](https://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html). + +Now let's look at a specific example of this concept in action. + +During 104B training experiments where fp16 mixed precision was used - the following improvement was proposed by [Corby Rosset](https://github.com/corbyrosset) to make [self-attention more stable](https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/118). + +Specifically this [line](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/c839a8aa30731f71b3738d56009be9668508e366/megatron/model/transformer.py#L303) shows that the `norm_factor` may be multiplied after the Query * Key matrix multiplication. If the dim of Q and K are very large, the output may blow up and the `norm_factor` won't be able to save it. + +Proposal: move the `norm_factor` inward, so Q and K are scaled down before matrix multiply: +```python + matmul_result = torch.baddbmm( + matmul_result, + 1.0/math.sqrt(self.norm_factor) * query_layer.transpose(0, 1), # [b * np, sq, hn] + 1.0/math.sqrt(self.norm_factor) * key_layer.transpose(0, 1).transpose(1, 2), # [b * np, hn, sk] + beta=0.0 if alibi is None else 1.0, alpha=1.0) + + # change view to [b, np, sq, sk] + attention_scores = matmul_result.view(*output_size) +``` + +To make the operation mathematically equivalent, moving the norm factor inward requires taking sqrt again if n is a scalar, A and B matrices: +``` +n * (A dot B) === (sqrt(n) * A) dot (sqrt(n) * B) +``` + +Now A and B dimensions can be significantly larger. + +The same post-multiply scaling problem also exists one level down, in the CUDA matmul APIs themselves. + +For CUDA kernel writers [CuBlas](https://docs.nvidia.com/cuda/cublas/index.html)'s `GemmStridedBatchedEx` at the time of this writing has a similar issue. It is defined as: + +``` +C+i*strideC=αop(A+i*strideA)op(B+i*strideB)+β(C+i*strideC), for i ∈[0,batchCount−1] +``` + +The issue is that `alpha` is multiplied after the matrix-matrix multiplication is done so it can cause instability. + +## "Bad" combination of data batch and model parameter state + +PaLM team observed dozens of loss spikes at "highly irregular intervals" when training larger models. While they were not able to track down the root cause, they mitigated the issue by restarting from an earlier checkpoint and skipping potentially problematic data batches. [Section 5.1 Training instability](https://arxiv.org/pdf/2204.02311) + + +## Time-domain correlation divergence in Adam + +[A Theory on Adam Instability in Large-Scale Machine Learning](https://arxiv.org/abs/2304.09871) performs a rigorous study of divergence spikes while training LLMs at up to 546B parameters - and suggests that the time-domain correlation leads to divergence of Adam. This is triggered by the epsilon value not being small enough and gradient estimation components become similar to the epsilon. + +In section 7.1 they propose practical suggestions, the most interesting one of them is setting epsilon to 0 and possibly dealing with division by zero condition. + + +# ==== training/instabilities/training-loss-patterns.md ==== + +# Understanding Training Loss Patterns + +Training loss plot is similar to the heart beat pattern - there is the good, the bad and you-should-worry one. After studying many training loss trajectories one develops an intuition to explain various loss behaviors during one's training and how to act on those. + +I warn you that the "Understanding" in the title of this section is overloaded since very often we don't really understand why certain types of spikes happen. Here "understanding" refers to recognizing various patterns. We then usually have techniques to overcome the bad patterns and bring the training successfully to the finish line. + +Thus you will find here a gallery of training loss patterns sometimes with real explanations, but more often than not educated guesses to what might be happening. + +Please excuse the plot snapshots looking wildly different from each other as they have come from many sources over multiple years. + +## The good, the bad and the unexpected + +Let's look at some good, bad and unusual patterns. + +### A very failed training + +Prior to starting BLOOM-176B training we did multiple experiments with the [104B model](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-wide). We failed to figure out how to not diverge very early on. + +![](images/pre-bloom-104B-en-fail.png) + +As you can see many attempts were made, many techniques were applied (see [chronicles](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr8-104B-wide/chronicles.md). We think the 2 main obstacles were using fp16 and data that had a lot of garbage in it. For BLOOM-176B we switched to bf16, used much cleaner data and also added an embedding layer-norm and that made all the difference. + + +### An almost perfect training + +![](images/bloom-176B-success.png) + +The [BLOOM-176B](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml) training had a close to perfect training loss trajectory, with a single spike that has recovered in 200 steps. + +You can inspect the [TB](https://huggingface.co/bigscience/tr11-176B-logs/tensorboard) to zoom in and check other plots. + +This was the almost perfect training indeed. Lots of hard work was put into achieving this. + + +### The grokking moment + +Some time back I was doing performance testing and run a tiny global batch size of 8 on an 8×A100 node, training llama-2-7b from scratch. (w/ DeepSpeed ZeRO-3 DP using HF Transformers [Llama](https://github.com/huggingface/transformers/tree/main/src/transformers/models/llama) implementation) + +![](images/llama-7b-grokking-no-zoom.png) + +Here one can observe a rapid loss improvement from 4 to 2.5 in just 480 samples after a very steady much slower improvements. My colleague [Gautam Mittal](https://github.com/gmittal) called it the [grokking](https://en.wikipedia.org/wiki/Grok) moment. In just a handful of steps the model suddenly generalized to much better predict the next tokens. + +Normally one doesn't see such a dramatic improvement when using a much larger batch size. + +If we zoom in it took about 60 8-sample per iteration steps: + +![](images/llama-7b-grokking.png) + + + + +## Main types of loss spikes + +In general there are 3 types of loss spikes: + +1. Fast recovering spikes +2. Slow recovering spikes +3. Not fully recovering spikes + +The spikes usually happen because of a bad data pocket, either due to badly shuffled data or because it hasn't been cleaned from some garbage scraped from the websites. + +While one would suspect that the batch before the spike was the trigger, but if you were to study that batch's contents you are likely to find nothing unusual - quite often the problem starts developing many steps before and then most of the sudden it happens. But also it might not be easy to study the batch, since it could amount to a size of a book when the global batch size and the sequence lengths are huge. + + +### Fast recovering spikes + +Loss spikes can happen often and as long as they quickly bounce back to where they left off the training usually continues as if nothing happened: + +Here is an example of [the 13B pre-BLOOM training experiment](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr1-13B-base): + +![](images/pre-bloom-tr1-13B-glitch-1-2.png) + +As you can see there are many spikes, some of a huge magnitude but they have all quickly recovered. + + +### Slow recovering spikes + +Here is a slow recovering spike from the [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training: + +![](images/idefics-80b-tr-190-01-spike-recover-2023-05-30.png) + + + +### Not fully recovering spikes + + +This [104B model attempt](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-wide) spiked, started recovering but decided to not recover fully and instead started diverging + +![](images/pre-bloom-tr8-104B-glitch-1.png) + +Here is another example from the [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training: + +![](images/idefics-80b-tr-190-01-spike-2023-05-27.png) + + +### Non-spike diverging + +Here are a few examples of diverging that didn't go through a spike + +![](images/pre-bloom-tr8-104B-glitch-5.png) + +and here are a few more: + +![](images/pre-bloom-tr8-104B-glitch-7-10.png) + +as you can see each restart makes a bit of progress and then the model diverges. + +All these are from the [104B model attempts](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-wide). + + +### Multiple datasets spikes + +During the [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training we were using 2 different dataset types mixed together: + +![](images/idefics-80b-tr-190-01-losses-2023-06-04.png) + +Legend: cm4 (high), average (mid) and pmd (low) + +You can see that the loss spikes were sometimes happening simultaneously on both datasets and at other times only one of the datasets loss would spike. + +Here the model was learning two different data distributions and as you can see it was not reporting the same loss and the spike behaviors on both data distributions. The pmd datasets loss was much easier for the model than the cm4 one. + + +## Resume-related spikes + +Training resume due to a hardware crash or because a need to rollback to an earlier checkpoint due to encountering a divergence is pretty much guaranteed to happen. If your training software doesn't resume perfectly so that the model doesn't notice there was a resume various problems could be encountered. + +The most complicated challenge of resume is restoring various RNGs, getting to the DataLoader index where the previous training was restored, and dealing with various other requirements if you use complex DataLoaders that are specific to your setup. + + +### DataSampler related issues + +During [IDEFICS-80B](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicles.md) training we had a very complicated DataLoader which was suffering from image to text ratio fluctuations when the DataLoader was getting restored on resume, so we ended up having a small spike on each resume which would then recover: + +![](images/idefics-80b-tr-190-01-image2text.png) + +You can see the loss and ratio plots correlation here. As we had to resume about a dozen times we saw a lot of those spikes. + + + + + +### Impacts of repeat data + +I was training a variation of Llama2 and saw this super unusual spike that didn't diverge or recover but which switched to a new higher loss level: + +![](images/ptl-repeat-data-p1.png) + +I rolled back to just before the weird behavior occurred and restarted. The loss training progressed at the same loss level for a bit and then again spiked and shifted to a higher loss. + +![](images/ptl-repeat-data-p2.png) + +I have never seen this type of divergence before. I was scratching my head for a while and then decided to look at the bigger picture. + +[wandb](https://wandb.ai/) didn't handle resume data plotting correctly if a rollback was performed, that is it ignored all new data after the rollback until the steps of the old data have been overcome. This forced us to start a new wandb plot for every resume with a rollback so that new data is shown. And if you need to see the whole plot you have to stitch them together, which includes dead data points that are no longer true. So I did the stitching and saw this puzzle: + +![](images/ptl-repeat-data-p3.png) + +footnote: as of 2025 wandb can do this properly - [rewinding a run](https://docs.wandb.ai/models/runs/rewind) with `wandb.init(resume_from="?_step=N")` truncates the history at step `N` and lets you log forward under the same run id, and `fork_from` does the same while leaving the original run intact (wandb SDK 0.17.1+; wandb recommends forking over rewinding for performance). Two catches: it's cloud-only - Multi-tenant and Dedicated Cloud, not Self-Managed - and it needs monotonically increasing steps, so it won't work alongside a non-monotonic `define_metric()`. Self-hosting, the stitching above is still the way. + +There was no real spike in the two earlier runs. The loss never went up in the first place. In both resumes it was under-reporting loss due to an exactly repeated data and then it reached data it hasn't seen before and started reporting correctly. In other words it was overfitting and reporting a false loss. + +The cause of the problem is data repetition, and since it clearly memorised some of it, it was reporting a better loss. + +The problem came from [pytorch-lightning](https://github.com/Lightning-AI/pytorch-lightning) not handling resumes correctly wrt DataSampler automatically - basically every time you resume you start your data stream from scratch. This, of course, requires a user to somehow fix the situation. You could change the seed to somewhat ameliorate the situation and avoid the exact data sequence, but it still leaves you with repeat data, which isn't what you want for any serious training (or ablation experiments, since your observation will be invalid, if they assume [IID data distribution](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables)). + +footnote: I discussed [this issue with the PTL developers](https://github.com/Lightning-AI/pytorch-lightning/issues/18780) and they said that they tried hard to come up with a generic solution but it wasn't meant to be. So the user needs to figure it out. + +Make sure to check your training framework documentation whether it handles the DataSampler resuming correctly. Make sure you didn't discover this problem after the training has finished and you ended up training 6x times the same 50B of tokens from the planned 300B tokens seen only once each. + +Doing a couple of resumes early on before embarking on the real training should also expose if there is a problem. Albeit, if the data gets reshuffled on each resume you are unlikely to see it. It'll only be seen if the seed is the same. + + +# ==== debug/README.md ==== + +# Debugging and Troubleshooting + + +## Guides + +- [Debugging PyTorch programs](./pytorch.md) + +- [Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs](./pytorch.md#diagnosing-crashes-hangs-and-tracing-execution) + +- [Network Debug](../network/debug/) + +- [Troubleshooting NVIDIA GPUs](../compute/accelerator/nvidia/debug.md) + +- [Underflow and Overflow Detection](./pytorch.md#underflow-and-overflow-detection) + + + +## Tools + +- [Debug Tools](./tools.md) + +- [torch-distributed-gpu-test.py](./torch-distributed-gpu-test.py) - this a `torch.distributed` diagnostics + script that checks that all GPUs in the cluster (one or many nodes) can talk to each other and allocate gpu memory. + +- [NicerTrace](./NicerTrace.py) - this is an improved `trace` python module with multiple additional flags added to the constructor and more useful output. diff --git a/docs/evidence/google_tuning_playbook.md b/docs/evidence/google_tuning_playbook.md index 0a18b08..8a46ce1 100644 --- a/docs/evidence/google_tuning_playbook.md +++ b/docs/evidence/google_tuning_playbook.md @@ -1,38 +1,2160 @@ -Source: https://github.com/google-research/tuning_playbook (README.md, fetched from raw.githubusercontent.com main branch) -Title: "Deep Learning Tuning Playbook" — Varun Godbole, George E. Dahl, Justin Gilmer, Christopher J. Shallue, Zachary Nado (Google Research / Harvard), 2023 -Fetched-via: curl of raw README.md, 2026-06-11 -Fetch-status: verbatim excerpts; bullet indentation flattened in places, content unchanged +Source: https://github.com/google-research/tuning_playbook (README.md, main branch) +Title: "Deep Learning Tuning Playbook" - Varun Godbole, George E. Dahl, Justin Gilmer, Christopher J. Shallue, Zachary Nado (Google Research / Harvard), 2023 +Fetched-via: curl -sL https://raw.githubusercontent.com/google-research/tuning_playbook/main/README.md, 2026-08-15 (CLAUDE agent) +Fetch-status: verbatim, full README; the whole playbook lives in this single file. Replaces the earlier hand-picked excerpts (CLAUDE agent) -# Deep Learning Tuning Playbook (excerpts) +# Deep Learning Tuning Playbook -From "Why a tuning playbook?": +*This is not an officially supported Google product.* -> Currently, there is an astonishing amount of toil and guesswork involved in actually getting deep neural networks to work well in practice. Even worse, the actual recipes people use to get good results with deep learning are rarely documented. Papers gloss over the process that led to their final results in order to present a cleaner story, and machine learning engineers working on commercial problems rarely have time to take a step back and generalize their process. [...] There is a vast gulf between the results achieved by deep learning experts and less skilled practitioners using superficially similar methods. At the same time, these very experts readily admit some of what they do might not be well-justified. +**Varun Godbole†, George E. Dahl†, Justin Gilmer†, Christopher J. Shallue‡, Zachary Nado†** -From "The incremental tuning strategy": -> ***Summary:*** *Start with a simple configuration and incrementally make improvements while building up insight into the problem. Make sure that any improvement is based on strong evidence to avoid adding unnecessary complexity.* +† Google Research, Brain Team -> The most effective way to maximize performance is to start with a simple configuration and incrementally add features and make improvements while building up insight into the problem. +‡ Harvard University -> For each launch, we must make sure that the change is based on strong evidence – not just random chance based on a lucky configuration – so that we don't add unnecessary complexity to the training pipeline. +## Table of Contents -From "Exploration vs exploitation": +- [Who is this document for?](#who-is-this-document-for) +- [Why a tuning playbook?](#why-a-tuning-playbook) +- [Guide for starting a new project](#guide-for-starting-a-new-project) + - [Choosing the model architecture](#choosing-the-model-architecture) + - [Choosing the optimizer](#choosing-the-optimizer) + - [Choosing the batch size](#choosing-the-batch-size) + - [Choosing the initial configuration](#choosing-the-initial-configuration) +- [A scientific approach to improving model performance](#a-scientific-approach-to-improving-model-performance) + - [The incremental tuning strategy](#the-incremental-tuning-strategy) + - [Exploration vs exploitation](#exploration-vs-exploitation) + - [Choosing the goal for the next round of experiments](#choosing-the-goal-for-the-next-round-of-experiments) + - [Designing the next round of experiments](#Designing-the-next-round-of-experiments) + - [Determining whether to adopt a training pipeline change or + hyperparameter + configuration](#Determining-whether-to-adopt-a-training-pipeline-change-or-hyperparameter-configuration) + - [After exploration concludes](#After-exploration-concludes) +- [Determining the number of steps for each training run](#Determining-the-number-of-steps-for-each-training-run) + - [Deciding how long to train when training is not compute-bound](#Deciding-how-long-to-train-when-training-is-not-compute-bound) + - [Deciding how long to train when training is compute-bound](#Deciding-how-long-to-train-when-training-is-compute-bound) +- [Additional guidance for the training pipeline](#Additional-guidance-for-the-training-pipeline) + - [Optimizing the input pipeline](#Optimizing-the-input-pipeline) + - [Evaluating model performance](#evaluating-model-performance) + - [Saving checkpoints and retrospectively selecting the best checkpoint](#Saving-checkpoints-and-retrospectively-selecting-the-best-checkpoint) + - [Setting up experiment tracking](#Setting-up-experiment-tracking) + - [Batch normalization implementation details](#Batch-normalization-implementation-details) + - [Considerations for multi-host pipelines](#Considerations-for-multi-host-pipelines) +- [FAQs](#faqs) +- [Acknowledgments](#acknowledgments) +- [Citing](#citing) +- [Contributing](#contributing) -> ***Summary:*** *Most of the time, our primary goal is to gain insight into the problem.* +## Who is this document for? -> Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error. In other words, we spend most of our time on "exploration" and only a small amount on "exploitation". +This document is for engineers and researchers (both individuals and teams) +interested in **maximizing the performance of deep learning models**. We assume +basic knowledge of machine learning and deep learning concepts. -> Prioritizing insight over short term gains can help us: Avoid launching unnecessary changes that happened to be present in well-performing runs merely through historical accident. Identify which hyperparameters the validation error is most sensitive to, which hyperparameters interact the most and therefore need to be re-tuned together, and which hyperparameters are relatively insensitive to other changes and can therefore be fixed in future experiments. +Our emphasis is on the **process of hyperparameter tuning**. We touch on other +aspects of deep learning training, such as pipeline implementation and +optimization, but our treatment of those aspects is not intended to be complete. -From "Choosing the goal for the next round of experiments": +We assume the machine learning problem is a supervised learning problem or +something that looks a lot like one (e.g. self-supervised). That said, some of +the prescriptions in this document may also apply to other types of problems. -> Each round of experiments should have a clear goal and be sufficiently narrow in scope that the experiments can actually make progress towards the goal: if we try to add multiple features or answer multiple questions at once, we may not be able to disentangle the separate effects on the results. +## Why a tuning playbook? -From "Identifying scientific, nuisance, and fixed hyperparameters": +Currently, there is an astonishing amount of toil and guesswork involved in +actually getting deep neural networks to work well in practice. Even worse, the +actual recipes people use to get good results with deep learning are rarely +documented. Papers gloss over the process that led to their final results in +order to present a cleaner story, and machine learning engineers working on +commercial problems rarely have time to take a step back and generalize their +process. Textbooks tend to eschew practical guidance and prioritize fundamental +principles, even if their authors have the necessary experience in applied work +to provide useful advice. When preparing to create this document, we couldn't +find any comprehensive attempt to actually explain *how to get good results with +deep learning*. Instead, we found snippets of advice in blog posts and on social +media, tricks peeking out of the appendix of research papers, occasional case +studies about one particular project or pipeline, and a lot of confusion. There +is a vast gulf between the results achieved by deep learning experts and less +skilled practitioners using superficially similar methods. At the same time, +these very experts readily admit some of what they do might not be +well-justified. As deep learning matures and has a larger impact on the world, +the community needs more resources covering useful recipes, including all the +practical details that can be so critical for obtaining good results. -> For a given goal, all hyperparameters will be either **scientific hyperparameters**, **nuisance hyperparameters**, or **fixed hyperparameters**. Scientific hyperparameters are those whose effect on the model's performance we're trying to measure. Nuisance hyperparameters are those that need to be optimized over in order to fairly compare different values of the scientific hyperparameters. This is similar to the statistical concept of nuisance parameters. Fixed hyperparameters will have their values fixed in the current round of experiments. +We are a team of five researchers and engineers who have worked in deep learning +for many years, some of us since as early as 2006. We have applied deep learning +to problems in everything from speech recognition to astronomy, and learned a +lot along the way. This document grew out of our own experience training neural +networks, teaching new machine learning engineers, and advising our colleagues +on the practice of deep learning. Although it has been gratifying to see deep +learning go from a machine learning approach practiced by a handful of academic +labs to a technology powering products used by billions of people, deep learning +is still in its infancy as an engineering discipline and we hope this document +encourages others to help systematize the field's experimental protocols. -> The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture). +This document came about as we tried to crystalize our own approach to deep +learning and thus it represents the opinions of the authors at the time of +writing, not any sort of objective truth. Our own struggles with hyperparameter +tuning made it a particular focus of our guidance, but we also cover other +important issues we have encountered in our work (or seen go wrong). Our +intention is for this work to be a living document that grows and evolves as our +beliefs change. For example, the material on debugging and mitigating training +failures would not have been possible for us to write two years ago since it is +based on recent results and ongoing investigations. Inevitably, some of our +advice will need to be updated to account for new results and improved +workflows. We do not know the *optimal* deep learning recipe, but until the +community starts writing down and debating different procedures, we cannot hope +to find it. To that end, we would encourage readers who find issues with our +advice to produce alternative recommendations, along with convincing evidence, +so we can update the playbook. We would also love to see alternative guides and +playbooks that might have different recommendations so we can work towards best +practices as a community. Finally, any sections marked with a 🤖 emoji are places +we would like to do more research. Only after trying to write this playbook did +it become completely clear how many interesting and neglected research questions +can be found in the deep learning practitioner's workflow. -> By fixing certain hyperparameters for a set of experiments, we must accept that conclusions derived from the experiments might not be valid for other settings of the fixed hyperparameters. In other words, fixed hyperparameters create caveats for any conclusions we draw from the experiments. +## Guide for starting a new project + +Many of the decisions we make over the course of tuning can be made once at the +beginning of a project and only occasionally revisited when circumstances +change. + +Our guidance below makes the following assumptions: + +- Enough of the essential work of problem formulation, data cleaning, etc. has + already been done that spending time on the model architecture and training + configuration makes sense. +- There is already a pipeline set up that does training and evaluation, and it + is easy to execute training and prediction jobs for various models of + interest. +- The appropriate metrics have been selected and implemented. These should be + as representative as possible of what would be measured in the deployed + environment. + +### Choosing the model architecture + +***Summary:*** *When starting a new project, try to reuse a model that already +works.* + +- Choose a well established, commonly used model architecture to get working + first. It is always possible to build a custom model later. +- Model architectures typically have various hyperparameters that determine + the model's size and other details (e.g. number of layers, layer width, type + of activation function). + - Thus, choosing the architecture really means choosing a family of + different models (one for each setting of the model hyperparameters). + - We will consider the problem of choosing the model hyperparameters in + [Choosing the initial configuration](#choosing-the-initial-configuration) + and + [A scientific approach to improving model performance](#a-scientific-approach-to-improving-model-performance). +- When possible, try to find a paper that tackles something as close as + possible to the problem at hand and reproduce that model as a starting + point. + +### Choosing the optimizer + +***Summary:*** *Start with the most popular optimizer for the type of problem at +hand.* + +- No optimizer is the "best" across all types of machine learning problems and + model architectures. Even just + [comparing the performance of optimizers is a difficult task](https://arxiv.org/abs/1910.05446). + 🤖 +- We recommend sticking with well-established, popular optimizers, especially + when starting a new project. + - Ideally, choose the most popular optimizer used for the same type of + problem. +- Be prepared to give attention to **\*****all****\*** hyperparameters of the + chosen optimizer. + - Optimizers with more hyperparameters may require more tuning effort to + find the best configuration. + - This is particularly relevant in the beginning stages of a project when + we are trying to find the best values of various other hyperparameters + (e.g. architecture hyperparameters) while treating optimizer + hyperparameters as + [nuisance parameters](#identifying-scientific-nuisance-and-fixed-hyperparameters). + - It may be preferable to start with a simpler optimizer (e.g. SGD with + fixed momentum or Adam with fixed $\epsilon$, $\beta_{1}$, and + $\beta_{2}$) in the initial stages of the project and switch to a more + general optimizer later. +- Well-established optimizers that we like include (but are not limited to): + - [SGD with momentum](#what-are-the-update-rules-for-all-the-popular-optimization-algorithms) + (we like the Nesterov variant) + - [Adam and NAdam](#what-are-the-update-rules-for-all-the-popular-optimization-algorithms), + which are more general than SGD with momentum. Note that Adam has 4 + tunable hyperparameters + [and they can all matter](https://arxiv.org/abs/1910.05446)! + - See + [How should Adam's hyperparameters be tuned?](#how-should-adams-hyperparameters-be-tuned) + +### Choosing the batch size + +***Summary:*** *The batch size governs the training speed and shouldn't be used +to directly tune the validation set performance. Often, the ideal batch size +will be the largest batch size supported by the available hardware.* + +- The batch size is a key factor in determining the *training time* and + *computing resource consumption*. +- Increasing the batch size will often reduce the training time. This can be + highly beneficial because it, e.g.: + - Allows hyperparameters to be tuned more thoroughly within a fixed time + interval, potentially resulting in a better final model. + - Reduces the latency of the development cycle, allowing new ideas to be + tested more frequently. +- Increasing the batch size may either decrease, increase, or not change the + resource consumption. +- The batch size should *not be* treated as a tunable hyperparameter for + validation set performance. + - As long as all hyperparameters are well-tuned (especially the learning + rate and regularization hyperparameters) and the number of training + steps is sufficient, the same final performance should be attainable + using any batch size (see + [Shallue et al. 2018](https://arxiv.org/abs/1811.03600)). + - Please see [Why shouldn't the batch size be tuned to directly improve + validation set + performance?](#why-shouldnt-the-batch-size-be-tuned-to-directly-improve-validation-set-performance) + +#### Determining the feasible batch sizes and estimating training throughput + + +
[Click to expand] + +
+ +- For a given model and optimizer, there will typically be a range of batch + sizes supported by the available hardware. The limiting factor is usually + accelerator memory. +- Unfortunately, it can be difficult to calculate which batch sizes will fit + in memory without running, or at least compiling, the full training program. +- The easiest solution is usually to run training jobs at different batch + sizes (e.g. increasing powers of 2) for a small number of steps until one of + the jobs exceeds the available memory. +- For each batch size, we should train for long enough to get a reliable + estimate of the *training throughput* + +

training throughput = (# examples processed per second)

+ +

or, equivalently, the time per step.

+ +

time per step = (batch size) / (training throughput)

+ +- When the accelerators aren't yet saturated, if the batch size doubles, the + training throughput should also double (or at least nearly double). + Equivalently, the time per step should be constant (or at least nearly + constant) as the batch size increases. +- If this is not the case then the training pipeline has a bottleneck such as + I/O or synchronization between compute nodes. This may be worth diagnosing + and correcting before proceeding. +- If the training throughput increases only up to some maximum batch size, + then we should only consider batch sizes up to that maximum batch size, even + if a larger batch size is supported by the hardware. + - All benefits of using a larger batch size assume the training throughput + increases. If it doesn't, fix the bottleneck or use the smaller batch + size. + - **Gradient accumulation** simulates a larger batch size than the + hardware can support and therefore does not provide any throughput + benefits. It should generally be avoided in applied work. +- These steps may need to be repeated every time the model or optimizer is + changed (e.g. a different model architecture may allow a larger batch size + to fit in memory). + +
+ +#### Choosing the batch size to minimize training time + +
[Click to expand] + +
+ + +

Training time = (time per step) x (total number of steps)

+ +- We can often consider the time per step to be approximately constant for all + feasible batch sizes. This is true when there is no overhead from parallel + computations and all training bottlenecks have been diagnosed and corrected + (see the + [previous section](#determining-the-feasible-batch-sizes-and-estimating-training-throughput) + for how to identify training bottlenecks). In practice, there is usually at + least some overhead from increasing the batch size. +- As the batch size increases, the total number of steps needed to reach a + fixed performance goal typically decreases (provided all relevant + hyperparameters are re-tuned when the batch size is changed; + [Shallue et al. 2018](https://arxiv.org/abs/1811.03600)). + - E.g. Doubling the batch size might halve the total number of steps + required. This is called **perfect scaling**. + - Perfect scaling holds for all batch sizes up to a critical batch size, + beyond which one achieves diminishing returns. + - Eventually, increasing the batch size no longer reduces the number of + training steps (but never increases it). +- Therefore, the batch size that minimizes training time is usually the + largest batch size that still provides a reduction in the number of training + steps required. + - This batch size depends on the dataset, model, and optimizer, and it is + an open problem how to calculate it other than finding it experimentally + for every new problem. 🤖 + - When comparing batch sizes, beware the distinction between an example + budget/[epoch](https://developers.google.com/machine-learning/glossary#epoch) + budget (running all experiments while fixing the number of training + example presentations) and a step budget (running all experiments with + the number of training steps fixed). + - Comparing batch sizes with an epoch budget only probes the perfect + scaling regime, even when larger batch sizes might still provide a + meaningful speedup by reducing the number of training steps + required. + - Often, the largest batch size supported by the available hardware will + be smaller than the critical batch size. Therefore, a good rule of thumb + (without running any experiments) is to use the largest batch size + possible. +- There is no point in using a larger batch size if it ends up increasing the + training time. + +
+ +#### Choosing the batch size to minimize resource consumption + +
[Click to expand] + +
+ + +- There are two types of resource costs associated with increasing the batch + size: + 1. *Upfront costs*, e.g. purchasing new hardware or rewriting the training + pipeline to implement multi-GPU / multi-TPU training. + 2. *Usage costs*, e.g. billing against the team's resource budgets, billing + from a cloud provider, electricity / maintenance costs. +- If there are significant upfront costs to increasing the batch size, it + might be better to defer increasing the batch size until the project has + matured and it is easier to assess the cost-benefit tradeoff. Implementing + multi-host parallel training programs can introduce + [bugs](#considerations-for-multi-host-pipelines) and + [subtle issues](#batch-normalization-implementation-details) so it is + probably better to start off with a simpler pipeline anyway. (On the other + hand, a large speedup in training time might be very beneficial early in the + process when a lot of tuning experiments are needed). +- We refer to the total usage cost (which may include multiple different kinds + of costs) as the "resource consumption". We can break down the resource + consumption into the following components: + +

Resource consumption = (resource consumption per step) x (total number of steps)

+ +- Increasing the batch size usually allows us to + [reduce the total number of steps](#choosing-the-batch-size-to-minimize-training-time). + Whether the resource consumption increases or decreases will depend on how + the consumption per step changes. + - Increasing the batch size might *decrease* the resource consumption. For + example, if each step with the larger batch size can be run on the same + hardware as the smaller batch size (with only a small increase in time + per step), then any increase in the resource consumption per step might + be outweighed by the decrease in the number of steps. + - Increasing the batch size might *not change* the resource consumption. + For example, if doubling the batch size halves the number of steps + required and doubles the number of GPUs used, the total consumption (in + terms of GPU-hours) will not change. + - Increasing the batch size might *increase* the resource consumption. For + example, if increasing the batch size requires upgraded hardware, the + increase in consumption per step might outweigh the reduction in the + number of steps. + +
+ +#### Changing the batch size requires re-tuning most hyperparameters + +
[Click to expand] + +
+ + +- The optimal values of most hyperparameters are sensitive to the batch size. + Therefore, changing the batch size typically requires starting the tuning + process all over again. +- The hyperparameters that interact most strongly with the batch size, and therefore are most important to tune separately for each batch size, are the optimizer hyperparameters (e.g. learning rate, momentum) and the regularization hyperparameters. +- Keep this in mind when choosing the batch size at the start of a project. If + you need to switch to a different batch size later on, it might be + difficult, time consuming, and expensive to re-tune everything for the new + batch size. + +
+ +#### How batch norm interacts with the batch size + +
[Click to expand] + +
+ + +- Batch norm is complicated and, in general, should use a different batch size + than the gradient computation to compute statistics. See the + [batch norm section](#batch-normalization-implementation-details) for a + detailed discussion. + +
+ +### Choosing the initial configuration + +- Before beginning hyperparameter tuning we must determine the starting point. + This includes specifying (1) the model configuration (e.g. number of + layers), (2) the optimizer hyperparameters (e.g. learning rate), and (3) the + number of training steps. +- Determining this initial configuration will require some manually configured + training runs and trial-and-error. +- Our guiding principle is to find a simple, relatively fast, relatively + low-resource-consumption configuration that obtains a "reasonable" result. + - "Simple" means avoiding bells and whistles wherever possible; these can + always be added later. Even if bells and whistles prove helpful down the + road, adding them in the initial configuration risks wasting time tuning + unhelpful features and/or baking in unnecessary complications. + - For example, start with a constant learning rate before adding fancy + decay schedules. + - Choosing an initial configuration that is fast and consumes minimal + resources will make hyperparameter tuning much more efficient. + - For example, start with a smaller model. + - "Reasonable" performance depends on the problem, but at minimum means + that the trained model performs much better than random chance on the + validation set (although it might be bad enough to not be worth + deploying). +- Choosing the number of training steps involves balancing the following + tension: + - On the one hand, training for more steps can improve performance and + makes hyperparameter tuning easier (see + [Shallue et al. 2018](https://arxiv.org/abs/1811.03600)). + - On the other hand, training for fewer steps means that each training run + is faster and uses fewer resources, boosting tuning efficiency by + reducing the time between cycles and allowing more experiments to be run + in parallel. Moreover, if an unnecessarily large step budget is chosen + initially, it might be hard to change it down the road, e.g. once the + learning rate schedule is tuned for that number of steps. + +## A scientific approach to improving model performance + +For the purposes of this document, the ultimate goal of machine learning +development is to maximize the utility of the deployed model. Even though many +aspects of the development process differ between applications (e.g. length of +time, available computing resources, type of model), we can typically use the +same basic steps and principles on any problem. + +Our guidance below makes the following assumptions: + +- There is already a fully-running training pipeline along with a + configuration that obtains a reasonable result. +- There are enough computational resources available to conduct meaningful + tuning experiments and run at least several training jobs in parallel. + +### The incremental tuning strategy + +***Summary:*** *Start with a simple configuration and incrementally make +improvements while building up insight into the problem. Make sure that any +improvement is based on strong evidence to avoid adding unnecessary complexity.* + +- Our ultimate goal is to find a configuration that maximizes the performance + of our model. + - In some cases, our goal will be to maximize how much we can improve the + model by a fixed deadline (e.g. submitting to a competition). + - In other cases, we want to keep improving the model indefinitely (e.g. + continually improving a model used in production). +- In principle, we could maximize performance by using an algorithm to + automatically search the entire space of possible configurations, but this + is not a practical option. + - The space of possible configurations is extremely large and there are + not yet any algorithms sophisticated enough to efficiently search this + space without human guidance. +- Most automated search algorithms rely on a hand-designed *search space* that + defines the set of configurations to search in, and these search spaces can + matter quite a bit. +- The most effective way to maximize performance is to start with a simple + configuration and incrementally add features and make improvements while + building up insight into the problem. + - We use automated search algorithms in each round of tuning and + continually update our search spaces as our understanding grows. +- As we explore, we will naturally find better and better configurations and + therefore our "best" model will continually improve. + - We call it a *launch* when we update our best configuration (which may + or may not correspond to an actual launch of a production model). + - For each launch, we must make sure that the change is based on strong + evidence – not just random chance based on a lucky configuration – so + that we don't add unnecessary complexity to the training pipeline. + +At a high level, our incremental tuning strategy involves repeating the +following four steps: + +1. Identify an appropriately-scoped goal for the next round of experiments. +2. Design and run a set of experiments that makes progress towards this goal. +3. Learn what we can from the results. +4. Consider whether to launch the new best configuration. + +The remainder of this section will consider this strategy in much greater +detail. + +### Exploration vs exploitation + +***Summary:*** *Most of the time, our primary goal is to gain insight into the +problem.* + +- Although one might think we would spend most of our time trying to maximize + performance on the validation set, in practice we spend the majority of our + time trying to gain insight into the problem, and comparatively little time + greedily focused on the validation error. + - In other words, we spend most of our time on "exploration" and only a + small amount on "exploitation". +- In the long run, understanding the problem is critical if we want to + maximize our final performance. Prioritizing insight over short term gains + can help us: + - Avoid launching unnecessary changes that happened to be present in + well-performing runs merely through historical accident. + - Identify which hyperparameters the validation error is most sensitive + to, which hyperparameters interact the most and therefore need to be + re-tuned together, and which hyperparameters are relatively insensitive + to other changes and can therefore be fixed in future experiments. + - Suggest potential new features to try, such as new regularizers if + overfitting is an issue. + - Identify features that don't help and therefore can be removed, reducing + the complexity of future experiments. + - Recognize when improvements from hyperparameter tuning have likely + saturated. + - Narrow our search spaces around the optimal value to improve tuning + efficiency. +- When we are eventually ready to be greedy, we can focus purely on the + validation error even if the experiments aren't maximally informative about + the structure of the tuning problem. + +### Choosing the goal for the next round of experiments + +***Summary:*** *Each round of experiments should have a clear goal and be +sufficiently narrow in scope that the experiments can actually make progress +towards the goal.* + +- Each round of experiments should have a clear goal and be sufficiently + narrow in scope that the experiments can actually make progress towards the + goal: if we try to add multiple features or answer multiple questions at + once, we may not be able to disentangle the separate effects on the results. +- Example goals include: + - Try a potential improvement to the pipeline (e.g. a new regularizer, + preprocessing choice, etc.). + - Understand the impact of a particular model hyperparameter (e.g. the + activation function) + - Greedily minimize validation error. + +### Designing the next round of experiments + +***Summary:*** *Identify which hyperparameters are scientific, nuisance, and +fixed hyperparameters for the experimental goal. Create a sequence of studies to +compare different values of the scientific hyperparameters while optimizing over +the nuisance hyperparameters. Choose the search space of nuisance +hyperparameters to balance resource costs with scientific value.* + +#### Identifying scientific, nuisance, and fixed hyperparameters + +
[Click to expand] + +
+ +- For a given goal, all hyperparameters will be either **scientific + hyperparameters**, **nuisance hyperparameters**, or **fixed + hyperparameters**. + - Scientific hyperparameters are those whose effect on the model's + performance we're trying to measure. + - Nuisance hyperparameters are those that need to be optimized over in + order to fairly compare different values of the scientific + hyperparameters. This is similar to the statistical concept of + [nuisance parameters](https://en.wikipedia.org/wiki/Nuisance_parameter). + - Fixed hyperparameters will have their values fixed in the current round + of experiments. These are hyperparameters whose values do not need to + (or we do not want them to) change when comparing different values of + the scientific hyperparameters. + - By fixing certain hyperparameters for a set of experiments, we must + accept that conclusions derived from the experiments might not be + valid for other settings of the fixed hyperparameters. In other + words, fixed hyperparameters create caveats for any conclusions we + draw from the experiments. +- For example, if our goal is to "determine whether a model with more hidden + layers will reduce validation error", then the number of hidden layers is a + scientific hyperparameter. + - The learning rate is a nuisance hyperparameter because we can only + fairly compare models with different numbers of hidden layers if the + learning rate is tuned separately for each number of layers (the optimal + learning rate generally depends on the model architecture). + - The activation function could be a fixed hyperparameter if we have + determined in prior experiments that the best choice of activation + function is not sensitive to model depth, or if we are willing to limit + our conclusions about the number of hidden layers to only cover this + specific choice of activation function. Alternatively, it could be a + nuisance parameter if we are prepared to tune it separately for each + number of hidden layers. +- Whether a particular hyperparameter is a scientific hyperparameter, nuisance + hyperparameter, or fixed hyperparameter is not inherent to that + hyperparameter, but changes depending on the experimental goal. + - For example, the choice of activation function could be a scientific + hyperparameter (is ReLU or tanh a better choice for our problem?), a + nuisance hyperparameter (is the best 5-layer model better than the best + 6-layer model when we allow several different possible activation + functions?), or a fixed hyperparameter (for ReLU nets, does adding batch + normalization in a particular position help?). +- When designing a new round of experiments, we first identify the scientific + hyperparameters for our experimental goal. + - At this stage, we consider all other hyperparameters to be nuisance + hyperparameters. +- Next, we convert some of the nuisance hyperparameters into fixed + hyperparameters. + - With limitless resources, we would leave all non-scientific + hyperparameters as nuisance hyperparameters so that the conclusions we + draw from our experiments are free from caveats about fixed + hyperparameter values. + - However, the more nuisance hyperparameters we attempt to tune, the + greater the risk we fail to tune them sufficiently well for each setting + of the scientific hyperparameters and end up reaching the wrong + conclusions from our experiments. + - As described + [below](#striking-a-balance-between-informative-and-affordable-experiments), + we could counter this risk by increasing the computational budget, + but often our maximum resource budget is less than would be needed + to tune over all non-scientific hyperparameters. + - We choose to convert a nuisance hyperparameter into a fixed + hyperparameter when, in our judgment, the caveats introduced by fixing + it are less burdensome than the cost of including it as a nuisance + hyperparameter. + - The more a given nuisance hyperparameter interacts with the + scientific hyperparameters, the more damaging it is to fix its + value. For example, the best value of the weight decay strength + typically depends on the model size, so comparing different model + sizes assuming a single specific value of the weight decay would not + be very insightful. +- Although the type we assign to each hyperparameter depends on the + experimental goal, we have the following rules of thumb for certain + categories of hyperparameters: + - Of the various optimizer hyperparameters (e.g. the learning rate, + momentum, learning rate schedule parameters, Adam betas etc.), at least + some of them will be nuisance hyperparameters because they tend to + interact the most with other changes. + - They are rarely scientific hyperparameters because a goal like "what + is the best learning rate for the current pipeline?" doesn't give + much insight – the best setting could easily change with the next + pipeline change anyway. + - Although we might fix some of them occasionally due to resource + constraints or when we have particularly strong evidence that they + don't interact with the scientific parameters, we should generally + assume that optimizer hyperparameters must be tuned separately to + make fair comparisons between different settings of the scientific + hyperparameters, and thus shouldn't be fixed. + - Furthermore, we have no *a priori* reason to prefer one + optimizer hyperparameter value over another (e.g. they don't + usually affect the computational cost of forward passes or + gradients in any way). + - In contrast, the *choice* of optimizer is typically a scientific + hyperparameter or fixed hyperparameter. + - It is a scientific hyperparameter if our experimental goal involves + making fair comparisons between two or more different optimizers + (e.g. "determine which optimizer produces the lowest validation + error in a given number of steps"). + - Alternatively, we might make it a fixed hyperparameter for a variety + of reasons, including (1) prior experiments make us believe that the + best optimizer for our problem is not sensitive to current + scientific hyperparameters; and/or (2) we prefer to compare values + of the scientific hyperparameters using this optimizer because its + training curves are easier to reason about; and/or (3) we prefer to + use this optimizer because it uses less memory than the + alternatives. + - Hyperparameters introduced by a regularization technique are typically + nuisance hyperparameters, but whether or not we include the + regularization technique at all is a scientific or fixed hyperparameter. + - For example, dropout adds code complexity, so when deciding whether + to include it we would make "no dropout" vs "dropout" a scientific + hyperparameter and the dropout rate a nuisance hyperparameter. + - If we decide to add dropout to our pipeline based on this + experiment, then the dropout rate would be a nuisance + hyperparameter in future experiments. + - Architectural hyperparameters are often scientific or fixed + hyperparameters because architecture changes can affect serving and + training costs, latency, and memory requirements. + - For example, the number of layers is typically a scientific or fixed + hyperparameter since it tends to have dramatic consequences for + training speed and memory usage. +- In some cases, the sets of nuisance and fixed hyperparameters will depend on + the values of the scientific hyperparameters. + - For example, suppose we are trying to determine which optimizer out of + Nesterov momentum and Adam results in the lowest validation error. The + scientific hyperparameter is the `optimizer`, which takes values + `{"Nesterov_momentum", "Adam"}`. The value + `optimizer="Nesterov_momentum"` introduces the nuisance/fixed + hyperparameters `{learning_rate, momentum}`, but the value + `optimizer="Adam"` introduces the nuisance/fixed hyperparameters + `{learning_rate, beta1, beta2, epsilon}`. + - Hyperparameters that are only present for certain values of the + scientific hyperparameters are called **conditional hyperparameters**. + - We should not assume two conditional hyperparameters are the same just + because they have the same name! In the above example, the conditional + hyperparameter called `learning_rate` is a *different* hyperparameter + for `optimizer="Nesterov_momentum"` versus `optimizer="Adam"`. Its role + is similar (although not identical) in the two algorithms, but the range + of values that work well in each of the optimizers is typically + different by several orders of magnitude. + +
+ +#### Creating a set of studies + +
[Click to expand] + +
+ + +- Once we have identified the scientific and nuisance hyperparameters, we + design a "study" or sequence of studies to make progress towards the + experimental goal. + - A study specifies a set of hyperparameter configurations to be run for + subsequent analysis. Each configuration is called a "trial". + - Creating a study typically involves choosing the hyperparameters that + will vary across trials, choosing what values those hyperparameters can + take on (the "search space"), choosing the number of trials, and + choosing an automated search algorithm to sample that many trials from + the search space. Alternatively, we could create a study by specifying + the set of hyperparameter configurations manually. +- The purpose of the studies is to run the pipeline with different values of + the scientific hyperparameters, while at the same time **"optimizing away"** + (or "optimizing over") the nuisance hyperparameters so that comparisons + between different values of the scientific hyperparameters are as fair as + possible. +- In the simplest case, we would make a separate study for each configuration + of the scientific parameters, where each study tunes over the nuisance + hyperparameters. + - For example, if our goal is to select the best optimizer out of Nesterov + momentum and Adam, we could create one study in which + `optimizer="Nesterov_momentum"` and the nuisance hyperparameters are + `{learning_rate, momentum}`, and another study in which + `optimizer="Adam"` and the nuisance hyperparameters are `{learning_rate, + beta1, beta2, epsilon}`. We would compare the two optimizers by + selecting the best performing trial from each study. + - We can use any gradient-free optimization algorithm, including methods + such as Bayesian optimization or evolutionary algorithms, to optimize + over the nuisance hyperparameters, although + [we prefer](#why-use-quasi-random-search-instead-of-more-sophisticated-black-box-optimization-algorithms-during-the-exploration-phase-of-tuning) + to use quasi-random search in the + [exploration phase](#exploration-vs-exploitation) of tuning because of a + variety of advantages it has in this setting. + [After exploration concludes](#after-exploration-concludes), if + state-of-the-art Bayesian optimization software is available, that is + our preferred choice. +- In the more complicated case where we want to compare a large number of + values of the scientific hyperparameters and it is impractical to make that + many independent studies, we can include the scientific parameters in the + same search space as the nuisance hyperparameters and use a search algorithm + to sample values of *both* the scientific and nuisance hyperparameters in a + single study. + - When taking this approach, conditional hyperparameters can cause + problems since it is hard to specify a search space unless the set of + nuisance hyperparameters is the same for all values of the scientific + hyperparameters. + - In this case, + [our preference](#why-use-quasi-random-search-instead-of-more-sophisticated-black-box-optimization-algorithms-during-the-exploration-phase-of-tuning) + for using quasi-random search over fancier black-box optimization tools + is even stronger, since it ensures that we obtain a relatively uniform + sampling of values of the scientific hyperparameters. Regardless of the + search algorithm, we need to make sure somehow that it searches the + scientific parameters uniformly. + +
+ +#### Striking a balance between informative and affordable experiments + +
[Click to expand] + +
+ + +- When designing a study or sequence of studies, we need to allocate a limited + budget in order to adequately achieve the following three desiderata: + 1. Comparing enough different values of the scientific hyperparameters. + 2. Tuning the nuisance hyperparameters over a large enough search space. + 3. Sampling the search space of nuisance hyperparameters densely enough. +- The better we can achieve these three desiderata, the more insight we can + extract from our experiment. + - Comparing as many values of the scientific hyperparameters as possible + broadens the scope of the insights we gain from the experiment. + - Including as many nuisance hyperparameters as possible and allowing each + nuisance hyperparameter to vary over as wide a range as possible + increases our confidence that a "good" value of the nuisance + hyperparameters **exists** in the search space for each configuration of + the scientific hyperparameters. + - Otherwise, we might make unfair comparisons between values of the + scientific hyperparameters by not searching possible regions of the + nuisance parameter space where better values might lie for some + values of the scientific parameters. + - Sampling the search space of nuisance hyperparameters as densely as + possible increases our confidence that any good settings for the + nuisance hyperparameters that happen to exist in our search space will + be found by the search procedure. + - Otherwise, we might make unfair comparisons between values of the + scientific parameters due to some values getting luckier with the + sampling of the nuisance hyperparameters. +- Unfortunately, improvements in *any* of these three dimensions require + either increasing the number of trials, and therefore increasing the + resource cost, or finding a way to save resources in one of the other + dimensions. + - Every problem has its own idiosyncrasies and computational constraints, + so how to allocate resources across these three desiderata requires some + level of domain knowledge. + - After running a study, we always try to get a sense of whether the study + tuned the nuisance hyperparameters well enough (i.e. searched a large + enough space extensively enough) to fairly compare the scientific + hyperparameters (as described in greater detail + [below](#extracting-insight-from-experimental-results)). + +
+ +### Extracting insight from experimental results + +***Summary:*** *In addition to trying to achieve the original scientific goal of +each group of experiments, go through a checklist of additional questions and, +if issues are discovered, revise the experiments and rerun them.* + +- Ultimately, each group of experiments has a specific goal and we want to + evaluate the evidence the experiments provide toward that goal. + - However, if we ask the right questions, we will often find issues that + need to be corrected before a given set of experiments can make much + progress towards their original goal. + - If we don’t ask these questions, we may draw incorrect conclusions. + - Since running experiments can be expensive, we also want to take the + opportunity to extract other useful insights from each group of + experiments, even if these insights are not immediately relevant to the + current goal. +- Before analyzing a given set of experiments to make progress toward their + original goal, we should ask ourselves the following additional questions: + - [Is the search space large enough?](#identifying-bad-search-space-boundaries) + - If the optimal point from a study is near the boundary of the search + space in one or more dimensions, the search is probably not wide + enough. In this case, we should run another study with an expanded + search space. + - [Have we sampled enough points from the search space?](#not-sampling-enough-points-in-the-search-space) + - If not, run more points or be less ambitious in the tuning goals. + - What fraction of the trials in each study are **infeasible** (i.e. + trials that diverge, get really bad loss values, or fail to run at all + because they violate some implicit constraint)? + - When a very large fraction of points in a study are **infeasible** + we should try to adjust the search space to avoid sampling such + points, which sometimes requires reparameterizing the search space. + - In some cases, a large number of infeasible points can indicate a + bug in the training code. + - [Does the model exhibit optimization issues?](#how-can-optimization-failures-be-debugged-and-mitigated) + - [What can we learn from the training curves of the best trials?](#examining-the-training-curves) + - For example, do the best trials have training curves consistent with + problematic overfitting? +- If necessary, based on the answers to the questions above, refine the most + recent study (or group of studies) to improve the search space and/or sample + more trials, or take some other corrective action. +- Once we have answered the above questions, we can move on to evaluating the + evidence the experiments provide towards our original goal (for example, + [evaluating whether a change is useful](#detecting-whether-a-change-is-useful-with-isolation-plots)). + +#### Identifying bad search space boundaries + +
[Click to expand] + +
+ + +- A search space is suspicious if the best point sampled from it is close to + its boundary. We might find an even better point if we expanded the search + range in that direction. +- To check search space boundaries, we like to plot completed trials on what + we call **basic hyperparameter axis plots** where we plot the validation + objective value versus one of the hyperparameters (e.g. learning rate). Each + point on the plot corresponds to a single trial. + - The validation objective value for each trial should usually be the best + value it achieved over the course of training. + +

+ Example of bad search space boundaries +Example of good search space boundaries +

+ +

Figure 1: Examples of bad search space boundaries and acceptable search space boundaries.

+ +- The plots in [Figure 1](#figure-1) show the error rate (lower is better) + against the initial learning rate. +- If the best points cluster towards the edge of a search space (in some + dimension), then the search space boundaries might need to be expanded until + the best observed point is no longer close to the boundary. +- Often, a study will include "infeasible" trials that diverge or get very bad + results (marked with red Xs in the above plots). + - If all trials are infeasible for learning rates greater than some + threshold value, and if the best performing trials have learning rates + at the edge of that region, the model [may suffer from stability issues + preventing it from accessing higher learning + rates](#how-can-optimization-failures-be-debugged-and-mitigated). + +
+ +#### Not sampling enough points in the search space + +
[Click to expand] + +
+ + +- In general, + [it can be very difficult to know](#how-many-trials-are-needed-to-get-good-results-with-quasi-random-search) + if the search space has been sampled densely enough. 🤖 +- Running more trials is of course better, but comes at an obvious cost. +- Since it is so hard to know when we have sampled enough, we usually sample + what we can afford and try to calibrate our intuitive confidence from + repeatedly looking at various hyperparameter axis plots and trying to get a + sense of how many points are in the "good" region of the search space. + +
+ +#### Examining the training curves + +
[Click to expand] + +
+ + +***Summary:*** *Examining the training curves is an easy way to identify common +failure modes and can help us prioritize what actions to take next.* + +- Although in many cases the primary objective of our experiments only + requires considering the validation error of each trial, we must be careful + when reducing each trial to a single number because it can hide important + details about what’s going on below the surface. +- For every study, we always look at the **training curves** (training error + and validation error plotted versus training step over the duration of + training) of at least the best few trials. +- Even if this is not necessary for addressing the primary experimental + objective, examining the training curves is an easy way to identify common + failure modes and can help us prioritize what actions to take next. +- When examining the training curves, we are interested in the following + questions. +- Are any of the trials exhibiting **problematic overfitting?** + - Problematic overfitting occurs when the validation error starts + *increasing* at some point during training. + - In experimental settings where we optimize away nuisance hyperparameters + by selecting the "best" trial for each setting of the scientific + hyperparameters, we should check for problematic overfitting in *at + least* each of the best trials corresponding to the settings of the + scientific hyperparameters that we’re comparing. + - If any of the best trials exhibits problematic overfitting, we + usually want to re-run the experiment with additional regularization + techniques and/or better tune the existing regularization parameters + before comparing the values of the scientific hyperparameters. + - This may not apply if the scientific hyperparameters include + regularization parameters, since then it would not be surprising + if low-strength settings of those regularization parameters + resulted in problematic overfitting. + - Reducing overfitting is often straightforward using common + regularization techniques that add minimal code complexity or extra + computation (e.g. dropout, label smoothing, weight decay), so it’s + usually no big deal to add one or more of these to the next round of + experiments. + - For example, if the scientific hyperparameter is "number of hidden + layers" and the best trial that uses the largest number of hidden + layers exhibited problematic overfitting, then we would usually + prefer to try it again with additional regularization instead of + immediately selecting the smaller number of hidden layers. + - Even if none of the "best" trials are exhibiting problematic + overfitting, there might still be a problem if it occurs in *any* of + the trials. + - Selecting the best trial suppresses configurations exhibiting + problematic overfitting and favors those that do not. In other + words, it will favor configurations with more regularization. + - However, anything that makes training worse can act as a + regularizer, even if it wasn't intended that way. For example, + choosing a smaller learning rate can regularize training by + hobbling the optimization process, but we typically don't want + to choose the learning rate this way. + - So we must be aware that the "best" trial for each setting of + the scientific hyperparameters might be selected in such a way + that favors "bad" values of some of the scientific or nuisance + hyperparameters. +- Is there high step-to-step variance in the training or validation error late + in training? + - If so, this could interfere with our ability to compare different values + of the scientific hyperparameters (since each trial randomly ends on a + "lucky" or "unlucky" step) and our ability to reproduce the result of + the best trial in production (since the production model might not end + on the same "lucky" step as in the study). + - The most likely causes of step-to-step variance are batch variance (from + randomly sampling examples from the training set for each batch), small + validation sets, and using a learning rate that’s too high late in + training. + - Possible remedies include increasing the batch size, obtaining more + validation data, using learning rate decay, or using Polyak averaging. +- Are the trials still improving at the end of training? + - If so, this indicates that we are in the + ["compute bound" regime](#determining-the-number-of-steps-for-each-training-run) + and we may benefit from + [increasing the number of training steps](#Deciding-how-long-to-train-when-training-is-compute-bound) + or changing the learning rate schedule. +- Has performance on the training and validation sets saturated long before + the final training step? + - If so, this indicates that we are in the + ["not compute-bound"](#determining-the-number-of-steps-for-each-training-run) + regime and that we may be able to + [decrease the number of training steps](#deciding-how-long-to-train-when-training-is-not-compute-bound). +- Although we cannot enumerate them all, there are many other additional + behaviors that can become evident from examining the training curves (e.g. + training loss *increasing* during training usually indicates a bug in the + training pipeline). + +
+ +#### Detecting whether a change is useful with isolation plots + +
[Click to expand] + +
+ + +

+Isolation plot that investigates the best value of weight decay for ResNet-50 +trained on ImageNet. +

+ +

Figure 2: Isolation plot that investigates the best value of weight decay for ResNet-50 trained on ImageNet.

+ +- Often, the goal of a set of experiments is to compare different values of a + scientific hyperparameter. + - For example, we may want to determine the value of weight decay that + results in the best validation error. +- An **isolation plot** is a special case of the basic hyperparameter axis + plot. Each point on an isolation plot corresponds to the performance of the + *best* trial across some (or all) of the nuisance hyperparameters. + - In other words, we plot the model performance after "optimizing away" + the nuisance hyperparameters. +- An isolation plot makes it easier to perform an apples-to-apples comparison + between different values of the scientific hyperparameter. +- For example, [Figure 2](#figure-2) reveals the value of weight decay that + produces the best validation performance for a particular configuration of + ResNet-50 trained on ImageNet. + - If our goal is to determine whether to include weight decay at all, then + we would compare the best point from this plot against the baseline of + no weight decay. For a fair comparison, the baseline should also have + its learning rate equally well tuned. +- When we have data generated by (quasi)random search and are considering a + continuous hyperparameter for an isolation plot, we can approximate the + isolation plot by bucketing the x-axis values of the basic hyperparameter + axis plot and taking the best trial in each vertical slice defined by the + buckets. + +
+ +#### Automate generically useful plots + +
[Click to expand] + +
+ +- The more effort it is to generate plots, the less likely we are to look at + them as much as we should, so it behooves us to set up our infrastructure to + automatically produce as many of them as possible. +- At a minimum, we automatically generate basic hyperparameter axis plots for + all hyperparameters that we vary in an experiment. +- Additionally, we automatically produce training curves for all trials and + make it as easy as possible to find the best few trials of each study and + examine their training curves. +- There are many other potential plots and visualizations we can add that can + be useful. Although the ones described above are a good starting point, to + paraphrase Geoffrey Hinton, "Every time you plot something new, you learn + something new." + +
+ +### Determining whether to adopt a training pipeline change or hyperparameter configuration + +***Summary:*** *When deciding whether to make a change to our model or training +procedure or adopt a new hyperparameter configuration going forward, we need to +be aware of the different sources of variation in our results.* + +- When we are trying to improve our model, we might observe that a particular + candidate change initially achieves a better validation error compared to + our incumbent configuration, but find that after repeating the experiment + there is no consistent advantage. Informally, we can group the most + important sources of variation that might cause such an inconsistent result + into the following broad categories: + - **Training procedure variance**, **retrain variance**, or **trial + variance**: the variation we see between training runs that use the same + hyperparameters, but different random seeds. + - For example, different random initializations, training data + shuffles, dropout masks, patterns of data augmentation operations, + and orderings of parallel arithmetic operations, are all potential + sources of trial variance. + - **Hyperparameter search variance**, or **study variance**: the variation + in results caused by our procedure to select the hyperparameters. + - For example, we might run the same experiment with a particular + search space, but with two different seeds for quasi-random search + and end up selecting different hyperparameter values. + - **Data collection and sampling variance**: the variance from any sort of + random split into training, validation, and test data or variance due to + the training data generation process more generally. +- It is all well and good to make comparisons of validation error rates + estimated on a finite validation set using fastidious statistical tests, but + often the trial variance alone can produce statistically significant + differences between two different trained models that use the same + hyperparameter settings. +- We are most concerned about study variance when trying to make conclusions + that go beyond the level of an individual point in hyperparameters space. + - The study variance depends on the number of trials and the search space + and we have seen cases where it is larger than the trial variance as + well as cases where it is much smaller. +- Therefore, before adopting a candidate change, consider running the best + trial N times to characterize the run-to-run trial variance. + - Usually, we can get away with only recharacterizing the trial variance + after major changes to the pipeline, but in some applications we might + need fresher estimates. + - In other applications, characterizing the trial variance is too costly + to be worth it. +- At the end of the day, although we only want to adopt changes (including new + hyperparameter configurations) that produce real improvements, demanding + complete certainty that something helps isn't the right answer either. +- Therefore, if a new hyperparameter point (or other change) gets a better + result than the baseline (taking into account the retrain variance of both + the new point and the baseline as best we can), then we probably should + adopt it as the new baseline for future comparisons. + - However, we should only adopt changes that produce improvements that + outweigh any complexity they add. + +### After exploration concludes + +***Summary:*** *Bayesian optimization tools are a compelling option once we’re +done exploring for good search spaces and have decided what hyperparameters even +should be tuned at all.* + +- At some point, our priorities will shift from learning more about the tuning + problem to producing a single best configuration to launch or otherwise use. +- At this point, there should be a refined search space that comfortably + contains the local region around the best observed trial and has been + adequately sampled. +- Our exploration work should have revealed the most essential hyperparameters + to tune (as well as sensible ranges for them) that we can use to construct a + search space for a final automated tuning study using as large a tuning + budget as possible. +- Since we no longer care about maximizing our insight into the tuning + problem, many of + [the advantages of quasi-random search](#why-use-quasi-random-search-instead-of-more-sophisticated-black-box-optimization-algorithms-during-the-exploration-phase-of-tuning) + no longer apply and Bayesian optimization tools should be used to + automatically find the best hyperparameter configuration. + - [Open-Source Vizier](https://github.com/google/vizier) implements + a variety of sophisticated algorithms for tuning ML models, including + Bayesian Optimization algorithms. + - If the search space contains a non-trivial volume of divergent points + (points that get NaN training loss or even training loss many standard + deviations worse than the mean), it is important to use black box + optimization tools that properly handle trials that diverge (see + [Bayesian Optimization with Unknown Constraints](https://arxiv.org/abs/1403.5607) + for an excellent way to deal with this issue). [Open-Source Vizier](https://github.com/google/vizier) + has support for divergent points by marking trials as infeasible, although it may not use our preferred approach from [Gelbart et al.](https://arxiv.org/abs/1403.5607), depending on how it is configured. +- At this point, we should also consider checking the performance on the test + set. + - In principle, we could even fold the validation set into the training + set and retraining the best configuration found with Bayesian + optimization. However, this is only appropriate if there won't be future + launches with this specific workload (e.g. a one-time Kaggle + competition). + +## Determining the number of steps for each training run + +- There are two types of workloads: those that are compute-bound and those + that are not. +- When training is **compute-bound**, training is limited by how long we are + willing to wait and not by how much training data we have or some other + factor. + - In this case, if we can somehow train longer or more efficiently, we + should see a lower training loss and, with proper tuning, an improved + validation loss. + - In other words, *speeding up* training is equivalent to *improving* + training and the "optimal" training time is always "as long as we can + afford." + - That said, just because a workload is compute-limited doesn't mean + training longer/faster is the only way to improve results. +- When training is **not compute-bound**, we can afford to train as long as we + would like to, and, at some point, training longer doesn't help much (or + even causes problematic overfitting). + - In this case, we should expect to be able to train to very low training + loss, to the point where training longer might slightly reduce the + training loss, but will not meaningfully reduce the validation loss. + - Particularly when training is not compute-bound, a more generous + training time budget can make tuning easier, especially when tuning + learning rate decay schedules, since they have a particularly strong + interaction with the training budget. + - In other words, very stingy training time budgets might require a + learning rate decay schedule tuned to perfection in order to achieve + a good error rate. +- Regardless of whether a given workload is compute-bound or not, methods that + increase the variance of the gradients (across batches) will usually result + in slower training progress, and thus may increase the number of training + steps required to reach a particular validation loss. High gradient variance + can be caused by: + - Using a smaller batch size + - Adding data augmentation + - Adding some types of regularization (e.g. dropout) + +### Deciding how long to train when training is *not* compute-bound + +- Our main goal is to ensure we are training long enough for the model to + reach the best possible result, while avoiding being overly wasteful in the + number of training steps. +- When in doubt, err on the side of training longer. Performance should never + degrade when training longer, assuming retrospective (optimal) checkpoint + selection is used properly and checkpoints are frequent enough. +- Never tune the `max_train_steps` number in a study. Pick a value and use it + for all trials. From these trials, plot the training step that retrospective + checkpoint selection finds in order to refine the choice of + `max_train_steps`. + - For example, if the best step is always during the first 10% of + training, then the maximum number of steps is way too high. + - Alternatively, if the best step is consistently in the last 25% of + training we might benefit from training longer and re-tuning the decay + schedule. +- The ideal number of training steps can change when the architecture or data + changes (e.g. adding data augmentation). +- Below we describe how to pick an initial candidate value for + `max_train_steps` based on the number of steps necessary to "perfectly fit" + the training set using a constant learning rate. + - Note, we are not using the phrase "perfectly fit the training set" in a + precise or mathematically well-defined way. It is merely meant as an + informal descriptor to indicate a very low training loss. + - For example, when training with the log loss, absent regularization + terms, we might see the training loss keep slowly improving until we + reach floating point limits as the network weights grow without + bound and the predictions of the model on the training set become + increasingly confident. In this case, we might say the model + "perfectly fit" the training set around the time the + misclassification error reached zero on the training set. + - The starting value for `max_train_steps` we find may need to be + increased if the amount of gradient noise in the training procedure + increases. + - For example, if data augmentation or regularizers like dropout are + introduced to the model. + - It may be possible to decrease `max_train_steps` if the training process + improves somehow. + - For example, with a better tuned optimizer or a better tuned + learning rate schedule. + +#### Algorithm for picking an initial candidate for max_train_steps using a learning rate sweep + +
[Click to expand] + +
+ +- This procedure assumes it is possible to not only "perfectly" fit the + training set, but to do so using a constant learning rate schedule. +- If it is possible to perfectly fit the entire training set, then there must + exist a configuration (with some value of `max_train_steps`) that perfectly + fits the training set; find any such configuration and use its value of + `max_train_steps` as a starting point `N`. +- Run a constant learning rate sweep (i.e. grid search the learning rate) + without data augmentation and without regularization where each trial trains + for `N` steps. +- The number of steps required for the fastest trial in the sweep to reach + perfect training performance is our initial guess for `max_train_steps`. +- **NOTE:** Bad search spaces can make it possible to engage in + self-deception. + - For example, if all the learning rates in a study are too small, we + might incorrectly conclude that a very large value of `max_train_steps` + is necessary. + - At a minimum, we should check that the optimal learning rate in the + study is not at the boundary of the search space. + +
+ +### Deciding how long to train when training is compute-bound + +- In some cases, training loss keeps improving indefinitely and our patience + and computational resources become the limiting factors. +- If training loss (or even validation loss) keeps improving indefinitely, + should we always train as long as we can afford? Not necessarily. + - We might be able to tune more effectively by running a larger number of + shorter experiments and reserving the longest "production length" runs + for the models we hope to launch. + - As the training time for trials approaches our patience limit, tuning + experiments become more relevant for our potential launch candidates, + but we can complete fewer of them. + - There are probably many questions we can answer while only training for + ~10% of the production length, but there is always a risk that our + conclusions at this time limit will not apply to experiments at 20% of + the production length, let alone 100%. +- Tuning in multiple rounds with increasing, per-trial training step limits is + a sensible approach. + - We can do as many rounds as we want, but usually 1-3 are the most + practical. + - Essentially, try to obtain as much understanding of the problem as + possible using trials with a very quick turnaround time, trading off + tuning thoroughness with relevance to the final, longest runs. + - Once a given per-trial time limit has generated useful insights, we can + increase the training time and continue tuning, double-checking our + conclusions from the shorter runs as needed. +- As a starting point, we recommend two rounds of tuning: + - Round 1: Shorter runs to find good model and optimizer hyperparameters. + - Round 2: Very few long runs on good hyperparameter points to get the + final model. +- The biggest question going from `Round i` → `Round i+1` is how to + adjust learning rate decay schedules. + - One common pitfall when adjusting learning rate schedules between rounds + is using all the extra training steps with too small of a learning rate. + +#### Round 1 + +
[Click to expand] + +
+ +- Unfortunately, there is no guarantee that good hyperparameters found in + short, incomplete training are still good choices when training length is + significantly increased. However, for some kinds of hyperparameters, they + are often correlated enough for Round 1 to be useful. +- What hyperparameter values found in shorter runs do we expect to transfer to + longer training runs? For all of this, we need more research. But based on + what we know so far, here are the authors’ suspicions in order of decreasing + probability of transferring: + - Very likely to transfer + - Early training instability can be resolved in the first round of + tuning using a smaller number of training steps. Perhaps these + hyperparameters are the closest thing to a sure bet for transfer + that we have. + - Warmup length + - Initialization + - Likely to transfer + - Model architecture - A dramatic win in the model architecture will + usually transfer, but there are probably many counterexamples. + - Might transfer + - Optimization algorithm/optimizer hyperparameters - We think this + would "loosely" transfer. It’s definitely weaker than the things + above it. + - Data augmentation + - Regularization + - If it isn't possible to perfectly fit the training set, the + model might be in a regime where regularization is unlikely to + help very much. + - Unlikely to transfer + - Learning rate schedule: unlikely to transfer perfectly. + - [This paper](https://arxiv.org/abs/2203.15556) suggests that + even decay schedule transfers, but we don't believe this is true + in general. Example: Tuning sqrt decay on small # of training + steps then extending to large # will result in the majority of + training occurring at overly small steps. + - One can likely do "good enough" with most schedules in the + limit of extreme training budget, but noticeable performance + improvements can likely be seen if it is tuned. + - [Understanding Short-Horizon Bias in Stochastic + Meta-Optimization](https://arxiv.org/abs/1803.02021) describes + the dangers of trying to pick learning rates myopically. + +
+ +#### Round 2 + +
[Click to expand] + +
+ +- Run the best hyperparameter configuration from Round 1. +- **(Speculation)** 🤖 Use the extra steps to extend the period of training at + a high learning rate. + - E.g. if linear schedule then keep the length of the decay fixed from + Round 1 and extend the period of constant lr in the beginning. + - For cosine decay, just keep the base lr from Round 1 and extend + `max_train_steps` as in + [Chinchilla paper](https://arxiv.org/abs/2203.15556). +- More rounds might make sense for teams with very mature modeling and tuning + pipelines and very long and expensive production training runs, but they + will often be overkill. + - We've described how to transfer from Step 1 → Step 2. If we didn't care + about analysis time and if making efficient use of compute was the + overriding concern, then the ideal would be to exponentially increase + the length of training runs (and thus the end-to-end time to complete a + study) over many different rounds of tuning. + - At each round we systematically ensure our choices continue to hold + up. + - New ideas go through a pipeline that progressively derisks them + using increasingly long-running experiments from Step i to Step i+1. + +
+ +## Additional guidance for the training pipeline + +### Optimizing the input pipeline + +***Summary:*** *The causes and interventions of input-bound pipelines are highly +task-dependent; use a profiler and look out for common issues.* + +- Use an appropriate profiler to diagnose input-bound pipelines. For example, + [Perfetto](https://jax.readthedocs.io/en/latest/profiling.html) for JAX or + [TensorFlow profiler](https://www.tensorflow.org/guide/profiler) for + TensorFlow. +- Ultimately, the specific causes and interventions will be highly + task-dependent. Broader engineering considerations (e.g. minimizing disk + footprint) may warrant worse input pipeline performance. +- Common causes: + - Data are not colocated with the training process, causing I/O latency + (this might happen when reading training data over a network). + - Expensive online data preprocessing (consider doing this once offline + and saving). + - Unintentional synchronization barriers that interfere with data pipeline + prefetching. For example, when synchronizing metrics between the device + and host in CommonLoopUtils + ([link](https://github.com/google/CommonLoopUtils/blob/fea2518ada8814a78e1492023fd9f00edb0b0568/clu/metrics.py#L291)). +- Common tips: + - Instrument input pipeline to prefetch examples (e.g. + [tf.data.Dataset.prefetch](https://www.tensorflow.org/guide/data_performance#prefetching)) + - Remove unused features/metadata from each as early in the pipeline as + possible. + - Increase the replication of the number of jobs generating examples for + the input pipeline. For example, by using the + [tf.data service](https://www.tensorflow.org/api_docs/python/tf/data/experimental/service). + +### Evaluating model performance + +***Summary:*** *Run evaluation at larger batch sizes than training. Run +evaluations at regular step intervals, not regular time intervals.* + +#### Evaluation settings + +
[Click to expand] + +
+ +- There are several settings in which we can evaluate the performance of our + models. + - **Online evaluation** - metrics are collected when the model is serving + predictions in a production environment. + - **Offline evaluation** - metrics are collected when the model is run on + offline train/validation/test sets that are representative of the + production environment. + - **Periodic evaluations** - metrics are collected during model training + that might either be a proxy for the offline evaluation, and/or on a + subset of the data used in offline evaluation. +- Online evaluation is the gold standard, but is often impractical during the + model development phase. +- Depending on the problem, offline evaluation can be fairly involved and + computationally expensive. +- Periodic evaluations are the most practical and economical choice, but may + not fully represent the production environment. + - Our goal during periodic evaluation is to use an expedient proxy of the + offline evaluation, without sacrificing the reliability of the signal we + get during training. + +
+ +#### Setting up periodic evaluations + +
[Click to expand] + +
+ +- We run periodic evaluations during training to monitor its progress in real + time, to + [facilitate retrospective model checkpoint selection](#saving-checkpoints-and-retrospectively-selecting-the-best-checkpoint), + and so that we can + [examine the training curves at the end of training](#examining-the-training-curves). +- The simplest configuration is to perform both training and periodic + evaluations within the same compute instance, periodically alternating + between training and evaluation. + - In this case, the batch size used to perform evaluations should be *at + least* as large as the batch size used for training because model + activations don't need to be maintained during evaluation, lowering the + computational requirements per example. +- Periodic evaluations should be done at regular step intervals, not time + intervals. + - Evaluating based on time intervals can make it harder to interpret the + training curves, especially when training may suffer from preemptions of + the training jobs, network latency issues, etc. +- Periodicity in valid/test metrics (when using a shuffled + train/validation/test split) can indicate implementation bugs such as test + data having overlap with training data, or training data not being properly + shuffled. Evaluating at regular step intervals can make these issues easier + to catch. +- Partial batches can occur when the evaluation sets are not divisible by the + batch size. Ensure that the padded examples are correctly weighted to prevent + the loss function from being biased by them. Often, these padded examples + can be given a weight of zero. +- Save sufficient information per evaluation to support offline analysis. + Ideally, we would save predictions on a selection of individual examples + since they can be invaluable for debugging. + - Generating artifacts like + [SavedModels](https://www.tensorflow.org/guide/saved_model) make it easy + to do ad-hoc model inspection after evaluation jobs finish. + +
+ +#### Choosing a sample for periodic evaluation + +
[Click to expand] + +
+ +- The periodic evaluation job might not run fast enough to compute metrics on + the full offline evaluation set in a reasonable amount of time. This often + necessitates sampling data for periodic evaluation. +- We consider the following factors when constructing a sampled dataset: + - Sample size + - Check that the performance computed on the sampled dataset used by + the periodic job matches the performance on the whole offline + evaluation set, i.e. there is no skew between the sampled set and + the full dataset. + - The dataset used for periodic evaluation should be small enough that + it’s easy to generate model predictions over its entirety, but large + enough that improvements to the model can be accurately measured + (i.e. not overwhelmed by label noise). + - It should be large enough to accommodate multiple such evaluations + across trials in sequence, and still produce accurate estimates. + That is, to avoid adaptively "fitting" to the validation set over + time, in a way that doesn't generalize to a held-out test set. + However, this consideration is rarely a practical concern. + - Imbalanced datasets + - For imbalanced datasets, performance on rare classes of examples + will often be noisy. + - For datasets with a small number of examples in a class label, log + the number of examples predicted correctly to get more insight into + accuracy improvements (.05 sensitivity improvement sounds exciting, + but was it just one more example correct?). + +
+ +### Saving checkpoints and retrospectively selecting the best checkpoint + +***Summary:*** *Run training for a fixed number of steps and retrospectively +choose the best checkpoint from the run.* + +- Most deep learning frameworks support + [model checkpointing](https://flax.readthedocs.io/en/latest/api_reference/flax.training.html). + That is, the current state of the model is periodically preserved on disk. + This allows the training job to be resilient to compute instance + interruptions. +- The best checkpoint is often not the last checkpoint, particularly when the + validation set performance does not continue to increase over time but + rather fluctuates about a particular value. +- Set up the pipeline to keep track of the N best checkpoints seen so far + during training. At the end of training, model selection is then a matter of + choosing the best checkpoint seen during training. We call this + **retrospective optimal checkpoint selection**. +- Supporting prospective early stopping is usually not necessary, since we’re + pre-specifying a trial budget and are preserving the N best checkpoints seen + so far. + +### Setting up experiment tracking + +***Summary:*** *When tracking different experiments, make sure to note a number +of essentials like the best performance of a checkpoint in the study, and a +short description of the study.* + +- We've found that keeping track of experiment results in a spreadsheet has + been helpful for the sorts of modeling problems we've worked on. It often + has the following columns: + - Study name + - A link to wherever the config for the study is stored. + - Notes or a short description of the study. + - Number of trials run + - Performance on the validation set of the best checkpoint in the study. + - Specific reproduction commands or notes on what unsubmitted changes were + necessary to launch training. +- Find a tracking system that captures at least the information listed above + and is convenient for the people doing it. Untracked experiments might as + well not exist. + +### Batch normalization implementation details + +***Summary:*** *Nowadays batch norm can often be replaced with LayerNorm, but in +cases where it cannot, there are tricky details when changing the batch size or +number of hosts.* + +- Batch norm normalizes activations using their mean and variance over the + current batch, but in the multi-device setting these statistics are + different on each device unless explicitly synchronized. +- Anecdotal reports (mostly on ImageNet) say calculating these normalizing + statistics using only ~64 examples actually works better in practice (see + Ghost Batch Norm from [this paper](https://arxiv.org/abs/1705.08741)). +- Decoupling the total batch size and the number of examples used to calculate + batch norm statistics is particularly useful for batch size comparisons. +- Ghost batch norm implementations do not always correctly handle the case + where the per-device batch size > virtual batch size. In this case we'd + actually need to subsample the batch on each device in order to get the + proper number of batch norm statistic examples. +- Exponential moving averages used in test mode batch norm are just a linear + combination of training statistics, so these EMAs only need to be + synchronized before saving them in checkpoints. However, some common + implementations of batch norm do not synchronize these EMAs and only save + the EMA from the first device. + +### Considerations for multi-host pipelines + +***Summary:*** *for logging, evals, RNGs, checkpointing, and data sharding, +multi-host training can make it very easy to introduce bugs!* + +- Ensure the pipeline is only logging and checkpointing on one host. +- Make sure before evaluation or checkpointing is run, the batch norm + statistics are synchronized across hosts. +- It is critical to have RNG seeds that are the same across hosts (for model + initialization), and seeds that are different across hosts (for data + shuffling/preprocessing), so make sure to mark them appropriately. +- Sharding data files across hosts is usually recommended for improved + performance. + +## FAQs + +### What is the best learning rate decay schedule family? + +
[Click to expand] + +
+ +- It’s an open problem. It’s not clear how to construct a set of rigorous + experiments to confidently answer what the "best" LR decay schedule is. +- Although we don't know the best schedule family, we're confident that it’s + important to have some (non-constant) schedule and that tuning it matters. +- Different learning rates work best at different times during the + optimization process. Having some sort of schedule makes it more likely for + the model to hit a good learning rate. + +
+ +### Which learning rate decay should I use as a default? + +
[Click to expand] +
+ +- Our preference is either linear decay or cosine decay, and a bunch of other + schedule families are probably good too. + +
+ +### Why do some papers have complicated learning rate schedules? + +
[Click to expand] +
+ +- It’s not uncommon to see papers with complicated piecewise learning rate + (LR) decay schedules. +- Readers often wonder how the authors arrived at such a complicated schedule. +- Many complicated LR decay schedules are the result of tuning the schedule as + a function of the validation set performance in an ad hoc way: + 1. Start a single training run with some simple LR decay (or a constant + learning rate). + 2. Keep training running until the performance seems to stagnate. If this + happens, pause training. Resume it with a perhaps steeper LR decay + schedule (or smaller constant learning rate) from this point. Repeat + this process until the conference/launch deadline. +- Blithely copying the resulting *schedule* is generally not a good idea since + the best particular schedule will be sensitive to a host of other + hyperparameter choices. + - Better to copy the *algorithm* that produced the schedule, although this + is rarely possible when arbitrary human judgment produced the schedule. +- This type of validation-error-sensitive schedule is fine to use if it can be + fully automated, but human-in-the-loop schedules that are a function of + validation error are brittle and not easily reproducible, so we recommend + avoiding them. + - Before publishing results that used such a schedule, please try to make + it fully reproducible. + +
+ +### How should Adam’s hyperparameters be tuned? + +
[Click to expand] +
+ +- As discussed above, making general statements about search spaces and how + many points one should sample from the search space is very difficult. Note + that not all the hyperparameters in Adam are equally important. The + following rules of thumb correspond to different "budgets" for the number of + trials in a study. + - If < 10 trials in a study, only tune the (base) learning rate. + - If 10-25 trials, tune learning rate and $\beta_1$. + - If 25+ trials, tune the learning rate, $\beta_1$ and $\epsilon$. + - If one can run substantially more than 25 trials, additionally tune + $\beta_2$. + +
+ +### Why use quasi-random search instead of more sophisticated black box optimization algorithms during the exploration phase of tuning? + +
[Click to expand] + +- Quasi-random search (based on + [low-discrepancy sequences](https://en.wikipedia.org/wiki/Low-discrepancy_sequence)) + is our preference over fancier black box optimization tools when used as + part of an iterative tuning process intended to maximize insight into the + tuning problem (what we refer to as the "exploration phase"). Bayesian + optimization and similar tools are more appropriate for the exploitation + phase. +- Quasi-random search based on randomly shifted low-discrepancy sequences can + be thought of as "jittered, shuffled grid search", since it uniformly, but + randomly, explores a given search space and spreads out the search points + more than random search. +- The advantages of quasi-random search over more sophisticated black box + optimization tools (e.g. Bayesian optimization, evolutionary algorithms) + include: + 1. Sampling the search space non-adaptively makes it possible to change the + tuning objective in post hoc analysis without rerunning experiments. + - For example, we usually want to find the best trial in terms of + validation error achieved at any point in training. But the + non-adaptive nature of quasi-random search makes it possible to find + the best trial based on final validation error, training error, or + some alternative evaluation metric without rerunning any + experiments. + 2. Quasi-random search behaves in a consistent and statistically + reproducible way. + - It should be possible to reproduce a study from six months ago even + if the implementation of the search algorithm changes, as long as it + maintains the same uniformity properties. If using sophisticated + Bayesian optimization software, the implementation might change in + an important way between versions, making it much harder to + reproduce an old search. It isn’t always possible to roll back to an + old implementation (e.g. if the optimization tool is run as a + service). + 3. Its uniform exploration of the search space makes it easier to reason + about the results and what they might suggest about the search space. + - For example, if the best point in the traversal of quasi-random + search is at the boundary of the search space, this is a good (but + not foolproof) signal that the search space bounds should be + changed. [This section](#identifying-bad-search-space-boundaries) + goes into more depth. However, an adaptive black box optimization + algorithm might have neglected the middle of the search space + because of some unlucky early trials even if it happens to contain + equally good points, since it is this exact sort of non-uniformity + that a good optimization algorithm needs to employ to speed up the + search. + 4. Running different numbers of trials in parallel versus sequentially will + not produce statistically different results when using quasi-random + search (or other non-adaptive search algorithms), unlike with adaptive + algorithms. + 5. More sophisticated search algorithms may not always handle infeasible + points correctly, especially if they aren't designed with neural network + hyperparameter tuning in mind. + 6. Quasi-random search is simple and works especially well when many tuning + trials will be running in parallel. + - Anecdotally[^3], it is very hard for an adaptive algorithm to beat a + quasi-random search that has 2X its budget, especially when many + trials need to be run in parallel (and thus there are very few + chances to make use of previous trial results when launching new + trials). + - Without expertise in Bayesian optimization and other advanced black + box optimization methods, we might not achieve the benefits they + are, in principle, capable of providing. It is hard to benchmark + advanced black box optimization algorithms in realistic deep + learning tuning conditions. They are a very active area of current + research, and the more sophisticated algorithms come with their own + pitfalls for inexperienced users. Experts in these methods are able + to get good results, but in high-parallelism conditions the search + space and budget tend to matter a lot more. +- That said, if our computational resources only allow a small number of + trials to run in parallel and we can afford to run many trials in sequence, + Bayesian optimization becomes much more attractive despite making our tuning + results harder to interpret. + +[^3]: Ben Recht and Kevin Jamieson + [pointed out](http://www.argmin.net/2016/06/20/hypertuning/) how strong + 2X-budget random search is as a baseline (the + [Hyperband paper](https://jmlr.org/papers/volume18/16-558/16-558.pdf) + makes similar arguments), but it is certainly possible to find search + spaces and problems where state-of-the-art Bayesian optimization + techniques crush random search that has 2X the budget. However, in our + experience beating 2X-budget random search gets much harder in the + high-parallelism regime since Bayesian optimization has no opportunity to + observe the results of previous trials. + +
+ +### Where can I find an implementation of quasi-random search? + +
[Click to expand] +
+ +- [Open-Source Vizier](https://github.com/google/vizier) has an [implementation + of quasi-random search](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/quasi_random.py). Set `algorithm="QUASI_RANDOM_SEARCH"` in [this usage example](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html). +- An alternative implementation exists + [here](https://github.com/mlcommons/algorithmic-efficiency/blob/main/algorithmic_efficiency/halton.py). +- Both implementations above generate a Halton sequence for a given search space (intended to + implement a shifted, scrambled Halton sequence as recommended in + https://arxiv.org/abs/1706.03200). +- If a quasi-random search algorithm based on a low-discrepancy sequence is + not available, it is possible to substitute pseudo random uniform search + instead, although this is likely to be slightly less efficient. + - In 1-2 dimensions, grid search is also acceptable, although not in + higher dimensions (see + [Bergstra & Bengio, 2012](https://www.jmlr.org/papers/v13/bergstra12a.html)). + +
+ +### How many trials are needed to get good results with quasi-random search? + +
[Click to expand] +
+ +

+A box plot showing the importance of sampling enough +

+ +

Figure 3: A ResNet-50 was tuned on ImageNet with 100 +trials. Via bootstrapping, different amounts of tuning budget were simulated. +Box plots of the best performances for each trial budget are plotted above. + +- There is no way to answer this question in general, but we can look at + specific examples. +- As the Figure 3 shows, the number of trials in a study can have a + substantial impact on the results. + - Notice how large the interquartile ranges are when 6 trials were + sampled, versus when 20 trials were sampled. + - Even with 20 trials, it is likely that the difference between especially + lucky and unlucky studies will be larger than the typical variation + between re-trains of this model on different random seeds, with fixed + hyperparameters, which for this workload might be around +/- 0.1% on a + validation error rate of \~23%. + +

+ +### How can optimization failures be debugged and mitigated? + +
[Click to expand] +
+ + +***Summary:*** *If the model is experiencing optimization difficulties, it’s +important to fix them before trying other things. Diagnosing and correcting +training failures is an active area of research.* + +

+Changing the strides in a single residual block in a WideResnet results in training instability. +

+ + +

Figure 4: Changing the strides in a single residual block (2x2 -> 1x1) in a WideResnet results in training instability. This does not degrade performance at low learning rates, but high learning rates no longer train well due to the instability. Applying 1000 steps of learning rate warmup resolves this particular instance of instability, allowing stable training at max learning rate of .1.

+ +#### Identifying unstable workloads + +- Any workload will become unstable if the learning rate is too large. + Instability is only an issue when it forces us to use a learning rate that’s + too small. +- There are at least two types of training instability worth distinguishing: + 1. Instability at initialization/early in training. + 2. Sudden instability in the middle of training. +- We can take a systematic approach to identifying stability issues in our + workload. + 1. Do a learning rate sweep and find the best learning rate lr*. + 2. Plot training loss curves for learning rates just above lr*. + 3. If the learning rates > lr* show loss instability (loss goes up not down + during periods of training), then it is likely that fixing the + instability will result in better training. +- Log the L2 norm of the full loss gradient during training, outlier values + can result in spurious instability in the middle of training. This can + inform how to pick gradient/update clipping. + +**NOTE:** Some models show very early instability followed by a recovery that +results in slow but stable training. **Common evaluation schedules can miss +these issues by not evaluating frequently enough!** + +To check for this, we can train for an abbreviated run of just \~500 steps using +`lr = 2 * current best`, but evaluate every step. + +

+Illustration of the value of more frequent evaluations at the start of +training. +

+ +

Figure 5: Illustration of the value of more frequent evaluations at the start of training. Useful if there’s a suspicion that the model suffers from early training instability.

+ +#### Potential fixes for common instability patterns + +- Apply learning rate warmup + - Best for early training instability. +- Apply gradient clipping + - Good for both early and mid training instability, may fix some bad inits + that warmup cannot. +- Try a new optimizer + - Sometimes Adam can handle instabilities that Momentum can’t. This is an + active area of research. +- We can ensure that we’re using best practices/initializations for our model + architecture (examples below). + - Add residual connections and normalization if the model doesn't contain + it already. +- Normalization should be inside the residual. E.g. x + f(Norm(x)). +- Norm(x + f(x)) known to cause issues. +- Try initializing residual branches to 0 (e.g. + [ReZero init](https://arxiv.org/abs/2003.04887)). +- Lower the learning rate + - This is a last resort. + +#### Learning rate warmup + +

+An example of instability during a warmup period (note the horizontal axis log +scale). +

+ +

Figure 6: An example of instability during a warmup period (note the horizontal axis log scale). 40k steps of warmup was needed for successful training in this case.

+ +##### When to apply learning rate warmup + +

+Axis plot for model with instability +

+ +

Figure 7a: An example of a hyperparameter axis plot for a model exhibiting training instability. The best learning rate is at the edge of what is feasible. An "infeasible" trial is defined as one that either produces NaNs or uncharacteristically high values of the loss.

+ +

+Loss curve for model with instability +

+ +

Figure 7b: The training loss of a model trained with a learning rate where we see instability.

+ +- Figure 7a shows a hyperparameter axis plot that indicates a model + experiencing optimization instabilities, because the best learning rate is + right at the edge of instability. +- Figure 7b shows how this can be double-checked by examining the training + loss of a model trained with a learning rate either 5x or 10x larger than + this peak. If that plot shows a sudden rise in the loss after a steady + decline (e.g. at step \~10k in the figure above), then the model likely + suffers from optimization instability. + +##### How to apply learning rate warmup + +

+Beneficial effect of warmup on training instabilities +

+ +

Figure 8: Beneficial effect of learning rate warmup on addressing training instabilities.

+ +- Using the section immediately above, we assume that the practitioner has + already identified the learning rate at which the model becomes unstable. + This is the `unstable_base_learning_rate`. +- Warmup involves prepending a learning rate schedule that ramps up the + learning rate from 0 to some stable `base_learning_rate`, that is at least + one order of magnitude larger than `unstable_base_learning_rate`. The + default would be to try a `base_learning_rate` that’s 10x + `unstable_base_learning_rate`. Although note that it’d be possible to run + this entire procedure again for something like 100x + `unstable_base_learning_rate`. The specific schedule is: + - Ramp up from 0 to `base_learning_rate` over `warmup_steps`. + - Train at a constant rate for `post_warmup_steps`. +- Our goal is to find the shortest number of `warmup_steps` that allows us to + access peak learning rates that are much higher than + `unstable_base_learning_rate`. +- So for each `base_learning_rate`, we need to tune `warmup_steps` and + `post_warmup_steps`. It’s usually fine to set `post_warmup_steps` to be + `2*warmup_steps`. +- Warmup can be tuned independently of an existing decay schedule. + `warmup_steps` should be swept at a few different orders of magnitude. For + example, an example study could try [10, 103, 104, + 105]. The largest feasible point shouldn't be more than 10% of + `max_train_steps`. +- Once a `warmup_steps` that doesn't blow up training at `base_learning_rate` + has been established, it should be applied to the baseline model. + Essentially, we prepend this schedule onto the existing schedule, and use + the optimal checkpoint selection discussed above to compare this experiment + to the baseline. For example, if we originally had 10,000 `max_train_steps` + and did `warmup_steps` for 1000 steps, the new training procedure should run + for 11,000 steps total. +- If long `warmup_steps` are required for stable training (>5% of + `max_train_steps`), `max_train_steps` may need to be increased to account + for this. +- There isn't really a "typical" value across the full range of workloads. + Some models only need 100 steps, while others (particularly transformers) + may need 40k+. + +#### Gradient clipping + +

+Gradient clipping on early training instabilities +

+ +

Figure 9: Illustration of gradient clipping correcting early training instability.

+ +- Gradient clipping is most useful when large or outlier gradient issues + occur. +- Clipping can fix either early training instability (large gradient norm + early), or mid training instabilities (sudden gradient spikes mid training). +- Sometimes longer warmup periods can correct instabilities that clipping does + not: see [this section above](#How-to-apply-learning-rate-warmup). + - 🤖 What about clipping during warmup? +- The ideal clip thresholds are just above the "typical" gradient norm. +- Here’s an example of how gradient clipping could be done: + - If the norm of the gradient $\left | g \right |$ is greater than the + gradient clipping threshold $\lambda$, then do ${g}'= \lambda \times \frac{g}{\left | g \right |}$ where ${g}'$ is the new gradient. +- Log the unclipped gradient norm during training. By default, generate: + - A plot of gradient norm vs step + - A histogram of gradient norms aggregated over all steps +- Choose a gradient clipping threshold based on the 90th percentile of + gradient norms. + - The threshold will be workload dependent, but 90% is a good starting + point. If it doesn't work, this threshold can be tuned. + - 🤖 What about some sort of adaptive strategy? +- If we try gradient clipping and the instability issues remain, we can try it + harder (i.e. make the threshold smaller). +- Extremely aggressive gradient clipping is in essence a strange way of + reducing the learning rate. If we find ourselves using extremely aggressive + clipping, we probably should just cut the learning rate instead. +- We would usually consider having >50% of the updates getting clipped somehow + as "extremely aggressive". +- If we need to do extremely aggressive gradient clipping to deal with our + instability issues, then we might as well reduce the learning rate. + +
+ +### Why do you call the learning rate and other optimization parameters hyperparameters? They are not parameters of any prior distribution. + +
[Click to expand] +
+ +- It is true that the term "hyperparameter" has a precise + [meaning](https://en.wikipedia.org/wiki/Hyperparameter) in Bayesian machine + learning and referring to the learning rate and most of the other parameters + we tune in deep learning as "hyperparameters" is an abuse of terminology. +- We would prefer to use the term "metaparameter" for learning rates, + architectural parameters, and all the other things we tune in deep learning, + since it avoids the potential for confusion that comes from misusing the + word "hyperparameter" (confusion that is especially likely when discussing + Bayesian optimization where the probabilistic response surface models have + their own true hyperparameters). +- Unfortunately, although potentially confusing, the term hyperparameter has become + extremely common in the deep learning community. +- Therefore, for a document, such as this one, intended for a wide audience + that includes many people who are unlikely to be aware of this technicality, + we made the choice to contribute to one source of confusion in the + field in hopes of avoiding another. +- That said, we might make a different choice when publishing a research + paper, and we would encourage others to use "metaparameter" instead in most + contexts. + +
+ +### Why shouldn't the batch size be tuned to directly improve validation set performance? + +
[Click to expand] +
+ +- Changing the batch size *without changing any other details of the training pipeline* will often affect the validation set performance. +- However, the difference in validation set performance between two batch sizes typically goes away if the training pipeline is optimized independently for each batch size. +- The hyperparameters that interact most strongly with the batch size, and therefore are most important to tune separately for each batch size, are the optimizer hyperparameters (e.g. learning rate, momentum) and the regularization hyperparameters. + - Smaller batch sizes introduce more noise into the training algorithm due to sample variance, and this noise can have a regularizing effect. Thus, larger batch sizes can be more prone to overfitting and may require stronger regularization and/or additional regularization techniques. +- In addition, [the number of training steps may need to be adjusted](#choosing-the-batch-size-to-minimize-training-time) when changing the batch size. +- Once all these effects are taken into account, there is currently no convincing evidence that the batch size affects the maximum achievable validation performance (see [Shallue et al. 2018](https://arxiv.org/abs/1811.03600)). + +
+ +### What are the update rules for all the popular optimization algorithms? + +
[Click to expand] + +
+ +#### Stochastic gradient descent (SGD) + +$$\theta_{t+1} = \theta_{t} - \eta_t \nabla \mathcal{l}(\theta_t)$$ + +#### Momentum + +$$v_0 = 0$$ + +$$v_{t+1} = \gamma v_{t} + \nabla \mathcal{l}(\theta_t)$$ + +$$\theta_{t+1} = \theta_{t} - \eta_t v_{t+1}$$ + +#### Nesterov + +$$v_0 = 0$$ + +$$v_{t+1} = \gamma v_{t} + \nabla \mathcal{l}(\theta_t)$$ + +$$\theta_{t+1} = \theta_{t} - \eta_t( \gamma v_{t+1} + \nabla \mathcal{l}(\theta_{t}))$$ + +#### RMSProp + +$$v_0 = 1 \text{,} m_0 = 0$$ + +$$v_{t+1} = \rho v_{t} + (1 - \rho) \nabla \mathcal{l}(\theta_t)^2$$ + +$$m_{t+1} = \gamma m_{t} + \frac{\eta_t}{\sqrt{v_{t+1} + \epsilon}}\nabla \mathcal{l}(\theta_t)$$ + +$$\theta_{t+1} = \theta_{t} - m_{t+1}$$ + +#### ADAM + +$$m_0 = 0 \text{,} v_0 = 0$$ + +$$m_{t+1} = \beta_1 m_{t} + (1 - \beta_1) \nabla \mathcal{l} (\theta_t)$$ + +$$v_{t+1} = \beta_2 v_{t} + (1 - \beta_2) \nabla \mathcal{l}(\theta_t)^2$$ + +$$b_{t+1} = \frac{\sqrt{1 - \beta_2^{t+1}}}{1 - \beta_1^{t+1}}$$ + +$$\theta_{t+1} = \theta_{t} - \alpha_t \frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon} b_{t+1}$$ + +#### NADAM + +$$m_0 = 0 \text{,} v_0 = 0$$ + +$$m_{t+1} = \beta_1 m_{t} + (1 - \beta_1) \nabla \mathcal{l} (\theta_t)$$ + +$$v_{t+1} = \beta_2 v_{t} + (1 - \beta_2) \nabla \mathcal{l} (\theta_t)^2$$ + +$$b_{t+1} = \frac{\sqrt{1 - \beta_2^{t+1}}}{1 - \beta_1^{t+1}}$$ + +$$\theta_{t+1} = \theta_{t} - \alpha_t \frac{\beta_1 m_{t+1} + (1 - \beta_1) \nabla \mathcal{l} (\theta_t)}{\sqrt{v_{t+1}} + \epsilon} b_{t+1}$$ + +
+ +## Acknowledgments + +- We owe a debt of gratitude to Max Bileschi, Roy Frostig, Zelda Mariet, Stan + Bileschi, Mohammad Norouzi, Chris DuBois and Charles Sutton for reading the + manuscript and providing valuable feedback. +- We reused some experimental data for several plots that were originally + produced by Naman Agarwal for other joint research. +- We would like to thank Will Chen for invaluable advice on the presentation of the document. +- We would also like to thank Rohan Anil for useful discussions. + +## Citing + +``` +@misc{tuningplaybookgithub, + author = {Varun Godbole and George E. Dahl and Justin Gilmer and Christopher J. Shallue and Zachary Nado}, + title = {Deep Learning Tuning Playbook}, + url = {http://github.com/google-research/tuning_playbook}, + year = {2023}, + note = {Version 1.0} +} +``` + +## Contributing + +- This is not an officially supported Google product. + +- We'd love to hear your feedback! + + - If you like the playbook, please [leave a star](https://docs.github.com/en/get-started/exploring-projects-on-github/saving-repositories-with-stars#starring-a-repository)! Or email + deep-learning-tuning-playbook \[at\] googlegroups.com. Testimonials help + us justify creating more resources like this. + - If anything seems incorrect, please file an issue to start a discussion. + For questions or other messages where an issue isn't appropriate, please + open a new discussion topic on GitHub. + +- As discussed in the preamble, this is a living document. We anticipate + making periodic improvements, both small and large. If you’d like to be + notified, please watch our repository (see [instructions](https://docs.github.com/en/account-and-profile/managing-subscriptions-and-notifications-on-github/setting-up-notifications/configuring-notifications#configuring-your-watch-settings-for-an-individual-repository)). + +- Please don't file a pull request without first coordinating with the authors + via the issue tracking system. + +### Contributor License Agreement + +Contributions to this project must be accompanied by a Contributor License +Agreement (CLA). You (or your employer) retain the copyright to your +contribution; this simply gives us permission to use and redistribute your +contributions as part of the project. Head over to + to see your current agreements on file or +to sign a new one. + +You generally only need to submit a CLA once, so if you've already submitted one +(even if it was for a different project), you probably don't need to do it +again. + +### Code Reviews + +All submissions, including submissions by project members, require review. We +use GitHub pull requests for this purpose. Consult +[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more +information on using pull requests. + +### Community Guidelines + +This project follows +[Google's Open Source Community Guidelines](https://opensource.google/conduct/). diff --git a/docs/evidence/ng_ml_yearning_error_analysis.md b/docs/evidence/ng_ml_yearning_error_analysis.md index 1613c06..c130dd9 100644 --- a/docs/evidence/ng_ml_yearning_error_analysis.md +++ b/docs/evidence/ng_ml_yearning_error_analysis.md @@ -1,32 +1,1097 @@ -Source: https://github.com/ajaymache/machine-learning-yearning/blob/master/full%20book/machine-learning-yearning.pdf (mirror of the draft Andrew Ng distributed via deeplearning.ai mailing list, 2018; never formally published) -Title: "Machine Learning Yearning" (draft) — Andrew Ng, chapters 13-19 (basic error analysis) -Fetched-via: PDF downloaded from the github mirror, text extracted with pdfplumber, 2026-06-11 -Fetch-status: verbatim excerpts; line breaks rejoined +Source: https://github.com/ajaymache/machine-learning-yearning/blob/master/full%20book/machine-learning-yearning.pdf (mirror of the draft Andrew Ng distributed via the deeplearning.ai mailing list, 2018; never formally published) +Title: "Machine Learning Yearning" (draft) - Andrew Ng. Full book, 58 chapters; chapters 13-19 are the basic error analysis part this skill cites +Fetched-via: curl https://r.jina.ai/, 2026-08-15 (CLAUDE agent) +Fetch-status: verbatim, full book (118 pages). Page furniture ("Page N Machine Learning Yearning-Draft Andrew Ng" running feet) left inline. Replaces the earlier chapter 13-19 excerpts (CLAUDE agent) -# Machine Learning Yearning — basic error analysis (excerpts) +Title: machine-learning-yearning.pdf -Chapter 13, "Build your first system quickly, then iterate" (p. 29): +URL Source: https://github.com/ajaymache/machine-learning-yearning/raw/master/full%20book/machine-learning-yearning.pdf -> So don't start off trying to design and build the perfect system. Instead, build and train a basic system quickly—perhaps in just a few days. Even if the basic system is far from the "best" system you can build, it is valuable to examine how the basic system functions: you will quickly find clues that show you the most promising directions in which to invest your time. +Number of Pages: 118 -Chapter 14, "Error analysis: Look at dev set examples to evaluate ideas" (pp. 30-31): +Markdown Content: +Machine Learning Yearning is a deeplearning.ai project. -> Before investing a month on this task, I recommend that you first estimate how much it will actually improve the system's accuracy. [...] In detail, here's what you can do: -> 1. Gather a sample of 100 dev set examples that your system misclassified. I.e., examples that your system made an error on. -> 2. Look at these examples manually, and count what fraction of them are dog images. +© 2018 Andrew Ng. All Rights Reserved. -> Error analysis can often help you figure out how promising different directions are. I've seen many engineers reluctant to carry out error analysis. It often feels more exciting to just jump in and implement some idea, rather than question if the idea is worth the time investment. This is a common mistake: It might result in your team spending a month only to realize afterward that it resulted in little benefit. +> Page 2 Machine Learning Yearning-Draft Andrew Ng -> Manually examining 100 examples does not take long. Even if you take one minute per image, you'd be done in under two hours. These two hours could save you a month of wasted effort. +# Table of Contents -Chapter 15, "Evaluating multiple ideas in parallel during error analysis" (p. 32): +1 Why Machine Learning Strategy 2 How to use this book to help your team 3 Prerequisites and Notation 4 Scale drives machine learning progress 5 Your development and test sets 6 Your dev and test sets should come from the same distribution 7 How large do the dev/test sets need to be? 8 Establish a single-number evaluation metric for your team to optimize 9 Optimizing and satisficing metrics 10 Having a dev set and metric speeds up iterations 11 When to change dev/test sets and metrics 12 Takeaways: Setting up development and test sets 13 Build your first system quickly, then iterate 14 Error analysis: Look at dev set examples to evaluate ideas 15 Evaluating multiple ideas in parallel during error analysis 16 Cleaning up mislabeled dev and test set examples 17 If you have a large dev set, split it into two subsets, only one of which you look at 18 How big should the Eyeball and Blackbox dev sets be? 19 Takeaways: Basic error analysis 20 Bias and Variance: The two big sources of error 21 Examples of Bias and Variance 22 Comparing to the optimal error rate 23 Addressing Bias and Variance 24 Bias vs. Variance tradeoff 25 Techniques for reducing avoidable bias -> You can efficiently evaluate all of these ideas in parallel. I usually create a spreadsheet and fill it out while looking through ~100 misclassified dev set images. I also jot down comments that might help me remember specific examples. [...] once you start looking through examples, you will probably be inspired to propose new error categories. +> Page 3 Machine Learning Yearning-Draft Andrew Ng -Chapter 19, "Takeaways: Basic error analysis" (p. 40): +26 Error analysis on the training set 27 Techniques for reducing variance 28 Diagnosing bias and variance: Learning curves 29 Plotting training error 30 Interpreting learning curves: High bias 31 Interpreting learning curves: Other cases 32 Plotting learning curves 33 Why we compare to human-level performance 34 How to define human-level performance 35 Surpassing human-level performance 36 When you should train and test on different distributions 37 How to decide whether to use all your data 38 How to decide whether to include inconsistent data 39 Weighting data 40 Generalizing from the training set to the dev set 41 Identifying Bias, Variance, and Data Mismatch Errors 42 Addressing data mismatch 43 Artificial data synthesis 44 The Optimization Verification test 45 General form of Optimization Verification test 46 Reinforcement learning example 47 The rise of end-to-end learning 48 More end-to-end learning examples 49 Pros and cons of end-to-end learning 50 Choosing pipeline components: Data availability 51 Choosing pipeline components: Task simplicity -> When you start a new project, especially if it is in an area in which you are not an expert, it is hard to correctly guess the most promising directions. +> Page 4 Machine Learning Yearning-Draft Andrew Ng -> Carry out error analysis by manually examining ~100 dev set examples the algorithm misclassifies and counting the major categories of errors. Use this information to prioritize what types of errors to work on fixing. +52 Directly learning rich outputs 53 Error analysis by parts 54 Attributing error to one part 55 General case of error attribution 56 Error analysis by parts and comparison to human-level performance 57 Spotting a flawed ML pipeline 58 Building a superhero team - Get your teammates to read this -> Consider splitting the dev set into an Eyeball dev set, which you will manually examine, and a Blackbox dev set, which you will not manually examine. If performance on the Eyeball dev set is much better than the Blackbox dev set, you have overfit the Eyeball dev set and should consider acquiring more data for it. +> Page 5 Machine Learning Yearning-Draft Andrew Ng + +# 1 Why Machine Learning Strategy + +Machine learning is the foundation of countless important applications, including web search, email anti-spam, speech recognition, product recommendations, and more. I assume that you or your team is working on a machine learning application, and that you want to make rapid progress. This book will help you do so. + +Example: Building a cat picture startup + +Say you’re building a startup that will provide an endless stream of cat pictures to cat lovers. You use a neural network to build a computer vision system for detecting cats in pictures. But tragically, your learning algorithm’s accuracy is not yet good enough. You are under tremendous pressure to improve your cat detector. What do you do? Your team has a lot of ideas, such as: + +• Get more data: Collect more pictures of cats. + +• Collect a more diverse training set. For example, pictures of cats in unusual positions; cats with unusual coloration; pictures shot with a variety of camera settings; …. + +• Train the algorithm longer, by running more gradient descent iterations. + +• Try a bigger neural network, with more layers/hidden units/parameters. + +> Page 6 Machine Learning Yearning-Draft Andrew Ng + +• Try a smaller neural network. + +• Try adding regularization (such as L2 regularization). + +• Change the neural network architecture (activation function, number of hidden units, etc.) + +• …If you choose well among these possible directions, you’ll build the leading cat picture platform, and lead your company to success. If you choose poorly, you might waste months. How do you proceed? This book will tell you how. Most machine learning problems leave clues that tell you what’s useful to try, and what’s not useful to try. Learning to read those clues will save you months or years of development time. + +> Page 7 Machine Learning Yearning-Draft Andrew Ng + +# 2 How to use this book to help your team + +After finishing this book, you will have a deep understanding of how to set technical direction for a machine learning project. But your teammates might not understand why you’re recommending a particular direction. Perhaps you want your team to define a single-number evaluation metric, but they aren’t convinced. How do you persuade them? That’s why I made the chapters short: So that you can print them out and get your teammates to read just the 1-2 pages you need them to know. A few changes in prioritization can have a huge effect on your team’s productivity. By helping your team with a few such changes, I hope that you can become the superhero of your team! + +> Page 8 Machine Learning Yearning-Draft Andrew Ng + +# 3 Prerequisites and Notation + +If you have taken a Machine Learning course such as my machine learning MOOC on Coursera, or if you have experience applying supervised learning, you will be able to understand this text. I assume you are familiar with supervised learning : learning a function that maps from x to y, using labeled training examples (x,y). Supervised learning algorithms include linear regression, logistic regression, and neural networks. There are many forms of machine learning, but the majority of Machine Learning’s practical value today comes from supervised learning. I will frequently refer to neural networks (also known as “deep learning”). You’ll only need a basic understanding of what they are to follow this text. If you are not familiar with the concepts mentioned here, watch the first three weeks of videos in the Machine Learning course on Coursera at http://ml-class.org + +> Page 9 Machine Learning Yearning-Draft Andrew Ng + +# 4 Scale drives machine learning progress + +Many of the ideas of deep learning (neural networks) have been around for decades. Why are these ideas taking off now? Two of the biggest drivers of recent progress have been: + +• Data availability. People are now spending more time on digital devices (laptops, mobile devices). Their digital activities generate huge amounts of data that we can feed to our learning algorithms. + +• Computational scale. We started just a few years ago to be able to train neural networks that are big enough to take advantage of the huge datasets we now have. In detail, even as you accumulate more data, usually the performance of older learning algorithms, such as logistic regression, “plateaus.” This means its learning curve “flattens out,” and the algorithm stops improving even as you give it more data: + +It was as if the older algorithms didn’t know what to do with all the data we now have. If you train a small neutral network (NN) on the same supervised learning task, you might get slightly better performance: + +> Page 10 Machine Learning Yearning-Draft Andrew Ng + +Here, by “Small NN” we mean a neural network with only a small number of hidden units/layers/parameters. Finally, if you train larger and larger neural networks, you can obtain even better performance: 1 + +Thus, you obtain the best performance when you (i) Train a very large neural network, so that you are on the green curve above; (ii) Have a huge amount of data. Many other details such as neural network architecture are also important, and there has been much innovation here. But one of the more reliable ways to improve an algorithm’s performance today is still to (i) train a bigger network and (ii) get more data. + +> 1 + +This diagram shows NNs doing better in the regime of small datasets. This effect is less consistent than the effect of NNs doing well in the regime of huge datasets. In the small data regime, depending on how the features are hand-engineered, traditional algorithms may or may not do better. For example, if you have 20 training examples, it might not matter much whether you use logistic regression or a neural network; the hand-engineering of features will have a bigger effect than the choice of algorithm. But if you have 1 million examples, I would favor the neural network. + +> Page 11 Machine Learning Yearning-Draft Andrew Ng + +The process of how to accomplish (i) and (ii) are surprisingly complex. This book will discuss the details at length. We will start with general strategies that are useful for both traditional learning algorithms and neural networks, and build up to the most modern strategies for building deep learning systems. + +> Page 12 Machine Learning Yearning-Draft Andrew Ng + +# Setting up development and test sets + +Page 13 Machine Learning Yearning-Draft Andrew Ng 5 Your development and test sets + +Let’s return to our earlier cat pictures example: You run a mobile app, and users are uploading pictures of many different things to your app. You want to automatically find the cat pictures. Your team gets a large training set by downloading pictures of cats (positive examples) and non-cats (negative examples) off of different websites. They split the dataset 70%/30% into training and test sets. Using this data, they build a cat detector that works well on the training and test sets. But when you deploy this classifier into the mobile app, you find that the performance is really poor! + +What happened? You figure out that the pictures users are uploading have a different look than the website images that make up your training set: Users are uploading pictures taken with mobile phones, which tend to be lower resolution, blurrier, and poorly lit. Since your training/test sets were made of website images, your algorithm did not generalize well to the actual distribution you care about: mobile phone pictures. Before the modern era of big data, it was a common rule in machine learning to use a random 70%/30% split to form your training and test sets. This practice can work, but it’s a bad idea in more and more applications where the training distribution (website images in + +> Page 14 Machine Learning Yearning-Draft Andrew Ng + +our example above) is different from the distribution you ultimately care about (mobile phone images). We usually define: + +• Training set — Which you run your learning algorithm on. + +• Dev (development) set — Which you use to tune parameters, select features, and make other decisions regarding the learning algorithm. Sometimes also called the + +hold-out cross validation set . + +• Test set — which you use to evaluate the performance of the algorithm, but not to make any decisions regarding what learning algorithm or parameters to use. Once you define a dev set (development set) and test set, your team will try a lot of ideas, such as different learning algorithm parameters, to see what works best. The dev and test sets allow your team to quickly see how well your algorithm is doing. In other words, the purpose of the dev and test sets are to direct your team toward the most important changes to make to the machine learning system .So, you should do the following: Choose dev and test sets to reflect data you expect to get in the future and want to do well on. In other words, your test set should not simply be 30% of the available data, especially if you expect your future data (mobile phone images) to be different in nature from your training set (website images). If you have not yet launched your mobile app, you might not have any users yet, and thus might not be able to get data that accurately reflects what you have to do well on in the future. But you might still try to approximate this. For example, ask your friends to take mobile phone pictures of cats and send them to you. Once your app is launched, you can update your dev/test sets using actual user data. If you really don’t have any way of getting data that approximates what you expect to get in the future, perhaps you can start by using website images. But you should be aware of the risk of this leading to a system that doesn’t generalize well. It requires judgment to decide how much to invest in developing great dev and test sets. But don’t assume your training distribution is the same as your test distribution. Try to pick test + +> Page 15 Machine Learning Yearning-Draft Andrew Ng + +examples that reflect what you ultimately want to perform well on, rather than whatever data you happen to have for training. + +> Page 16 Machine Learning Yearning-Draft Andrew Ng + +# 6 Your dev and test sets should come from the same distribution + +You have your cat app image data segmented into four regions, based on your largest markets: (i) US, (ii) China, (iii) India, and (iv) Other. To come up with a dev set and a test set, say we put US and India in the dev set; China and Other in the test set. In other words, we can randomly assign two of these segments to the dev set, and the other two to the test set, right? Once you define the dev and test sets, your team will be focused on improving dev set performance. Thus, the dev set should reflect the task you want to improve on the most: To do well on all four geographies, and not only two. There is a second problem with having different dev and test set distributions: There is a chance that your team will build something that works well on the dev set, only to find that it does poorly on the test set. I’ve seen this result in much frustration and wasted effort. Avoid letting this happen to you. As an example, suppose your team develops a system that works well on the dev set but not the test set. If your dev and test sets had come from the same distribution, then you would have a very clear diagnosis of what went wrong: You have overfit the dev set. The obvious cure is to get more dev set data. But if the dev and test sets come from different distributions, then your options are less clear. Several things could have gone wrong: + +1. You had overfit to the dev set. + +2. The test set is harder than the dev set. So your algorithm might be doing as well as could be expected, and no further significant improvement is possible. + +> Page 17 Machine Learning Yearning-Draft Andrew Ng + +3. The test set is not necessarily harder, but just different, from the dev set. So what works well on the dev set just does not work well on the test set. In this case, a lot of your work to improve dev set performance might be wasted effort. Working on machine learning applications is hard enough. Having mismatched dev and test sets introduces additional uncertainty about whether improving on the dev set distribution also improves test set performance. Having mismatched dev and test sets makes it harder to figure out what is and isn’t working, and thus makes it harder to prioritize what to work on. If you are working on a 3rd party benchmark problem, their creator might have specified dev and test sets that come from different distributions. Luck, rather than skill, will have a greater impact on your performance on such benchmarks compared to if the dev and test sets come from the same distribution. It is an important research problem to develop learning algorithms that are trained on one distribution and generalize well to another. But if your goal is to make progress on a specific machine learning application rather than make research progress, I recommend trying to choose dev and test sets that are drawn from the same distribution. This will make your team more efficient. + +> Page 18 Machine Learning Yearning-Draft Andrew Ng + +# 7 How large do the dev/test sets need to be? + +The dev set should be large enough to detect differences between algorithms that you are trying out. For example, if classifier A has an accuracy of 90.0% and classifier B has an accuracy of 90.1%, then a dev set of 100 examples would not be able to detect this 0.1% difference. Compared to other machine learning problems I’ve seen, a 100 example dev set is small. Dev sets with sizes from 1,000 to 10,000 examples are common. With 10,000 examples, you will have a good chance of detecting an improvement of 0.1%. 2 + +For mature and important applications—for example, advertising, web search, and product recommendations—I have also seen teams that are highly motivated to eke out even a 0.01% improvement, since it has a direct impact on the company’s profits. In this case, the dev set could be much larger than 10,000, in order to detect even smaller improvements. How about the size of the test set? It should be large enough to give high confidence in the overall performance of your system. One popular heuristic had been to use 30% of your data for your test set. This works well when you have a modest number of examples—say 100 to 10,000 examples. But in the era of big data where we now have machine learning problems with sometimes more than a billion examples, the fraction of data allocated to dev/test sets has been shrinking, even as the absolute number of examples in the dev/test sets has been growing. There is no need to have excessively large dev/test sets beyond what is needed to evaluate the performance of your algorithms. + +> 2 + +In theory, one could also test if a change to an algorithm makes a statistically significant difference on the dev set. In practice, most teams don’t bother with this (unless they are publishing academic research papers), and I usually do not find statistical significance tests useful for measuring interim progress. + +> Page 19 Machine Learning Yearning-Draft Andrew Ng + +# 8 Establish a single-number evaluation metric for your team to optimize + +Classification accuracy is an example of a single-number evaluation metric : You run your classifier on the dev set (or test set), and get back a single number about what fraction of examples it classified correctly. According to this metric, if classifier A obtains 97% accuracy, and classifier B obtains 90% accuracy, then we judge classifier A to be superior. In contrast, Precision and Recall is not a single-number evaluation metric: It gives two + +> 3 + +numbers for assessing your classifier. Having multiple-number evaluation metrics makes it harder to compare algorithms. Suppose your algorithms perform as follows: + +Classifier Precision Recall A 95% 90% + +B 98% 85% + +Here, neither classifier is obviously superior, so it doesn’t immediately guide you toward picking one. + +Classifier Precision Recall F1 score A 95% 90% 92.4% + +During development, your team will try a lot of ideas about algorithm architecture, model parameters, choice of features, etc. Having a single-number evaluation metric such as accuracy allows you to sort all your models according to their performance on this metric, and quickly decide what is working best. If you really care about both Precision and Recall, I recommend using one of the standard ways to combine them into a single number. For example, one could take the average of precision and recall, to end up with a single number. Alternatively, you can compute the “F1 + +> 3 + +The Precision of a cat classifier is the fraction of images in the dev (or test) set it labeled as cats that really are cats. Its Recall is the percentage of all cat images in the dev (or test) set that it correctly labeled as a cat. There is often a tradeoff between having high precision and high recall. + +> Page 20 Machine Learning Yearning-Draft Andrew Ng + +score,” which is a modified way of computing their average, and works better than simply taking the mean. 4 + +> Classifier Precision Recall F1 score A95% 90% 92.4% +> B98% 85% 91.0% + +Having a single-number evaluation metric speeds up your ability to make a decision when you are selecting among a large number of classifiers. It gives a clear preference ranking among all of them, and therefore a clear direction for progress. As a final example, suppose you are separately tracking the accuracy of your cat classifier in four key markets: (i) US, (ii) China, (iii) India, and (iv) Other. This gives four metrics. By taking an average or weighted average of these four numbers, you end up with a single number metric. Taking an average or weighted average is one of the most common ways to combine multiple metrics into one. + +> 4 + +If you want to learn more about the F1 score, see https://en.wikipedia.org/wiki/F1_score . It is the “harmonic mean” between Precision and Recall, and is calculated as 2/((1/Precision)+(1/Recall)). + +> Page 21 Machine Learning Yearning-Draft Andrew Ng + +# 9 Optimizing and satisficing metrics + +Here’s another way to combine multiple evaluation metrics. Suppose you care about both the accuracy and the running time of a learning algorithm. You need to choose from these three classifiers: + +> Classifier Accuracy Running time A90% 80ms +> B92% 95ms +> C95% 1,500ms + +It seems unnatural to derive a single metric by putting accuracy and running time into a single formula, such as: Accuracy - 0.5*RunningTime Here’s what you can do instead: First, define what is an “acceptable” running time. Lets say anything that runs in 100ms is acceptable. Then, maximize accuracy, subject to your classifier meeting the running time criteria. Here, running time is a “satisficing metric”—your classifier just has to be “good enough” on this metric, in the sense that it should take at most 100ms. Accuracy is the “optimizing metric.” If you are trading off N different criteria, such as binary file size of the model (which is important for mobile apps, since users don’t want to download large apps), running time, and accuracy, you might consider setting N-1 of the criteria as “satisficing” metrics. I.e., you simply require that they meet a certain value. Then define the final one as the “optimizing” metric. For example, set a threshold for what is acceptable for binary file size and running time, and try to optimize accuracy given those constraints. As a final example, suppose you are building a hardware device that uses a microphone to listen for the user saying a particular “wakeword,” that then causes the system to wake up. Examples include Amazon Echo listening for “Alexa”; Apple Siri listening for “Hey Siri”; Android listening for “Okay Google”; and Baidu apps listening for “Hello Baidu.” You care about both the false positive rate—the frequency with which the system wakes up even when no one said the wakeword—as well as the false negative rate—how often it fails to wake up when someone says the wakeword. One reasonable goal for the performance of this system is + +> Page 22 Machine Learning Yearning-Draft Andrew Ng + +to minimize the false negative rate (optimizing metric), subject to there being no more than one false positive every 24 hours of operation (satisficing metric). Once your team is aligned on the evaluation metric to optimize, they will be able to make faster progress. + +> Page 23 Machine Learning Yearning-Draft Andrew Ng + +# 10 Having a dev set and metric speeds up iterations + +It is very difficult to know in advance what approach will work best for a new problem. Even experienced machine learning researchers will usually try out many dozens of ideas before they discover something satisfactory. When building a machine learning system, I will often: + +1. Start off with some idea on how to build the system. + +2. Implement the idea in code . + +3. Carry out an experiment which tells me how well the idea worked. (Usually my first few ideas don’t work!) Based on these learnings, go back to generate more ideas, and keep on iterating. This is an iterative process. The faster you can go round this loop, the faster you will make progress. This is why having dev/test sets and a metric are important: Each time you try an idea, measuring your idea’s performance on the dev set lets you quickly decide if you’re heading in the right direction. In contrast, suppose you don’t have a specific dev set and metric. So each time your team develops a new cat classifier, you have to incorporate it into your app, and play with the app for a few hours to get a sense of whether the new classifier is an improvement. This would be incredibly slow! Also, if your team improves the classifier’s accuracy from 95.0% to 95.1%, you might not be able to detect that 0.1% improvement from playing with the app. Yet a lot of progress in your system will be made by gradually accumulating dozens of these 0.1% improvements. Having a dev set and metric allows you to very quickly detect which ideas are successfully giving you small (or large) improvements, and therefore lets you quickly decide what ideas to keep refining, and which ones to discard. + +> Page 24 Machine Learning Yearning-Draft Andrew Ng + +# 11 When to change dev/test sets and metrics + +When starting out on a new project, I try to quickly choose dev/test sets, since this gives the team a well-defined target to aim for. I typically ask my teams to come up with an initial dev/test set and an initial metric in less than one week—rarely longer. It is better to come up with something imperfect and get going quickly, rather than overthink this. But this one week timeline does not apply to mature applications. For example, anti-spam is a mature deep learning application. I have seen teams working on already-mature systems spend months to acquire even better dev/test sets. If you later realize that your initial dev/test set or metric missed the mark, by all means change them quickly. For example, if your dev set + metric ranks classifier A above classifier B, but your team thinks that classifier B is actually superior for your product, then this might be a sign that you need to change your dev/test sets or your evaluation metric. There are three main possible causes of the dev set/metric incorrectly rating classifier A higher: 1. The actual distribution you need to do well on is different from the dev/test sets. Suppose your initial dev/test set had mainly pictures of adult cats. You ship your cat app, and find that users are uploading a lot more kitten images than expected. So, the dev/test set distribution is not representative of the actual distribution you need to do well on. In this case, update your dev/test sets to be more representative. + +> Page 25 Machine Learning Yearning-Draft Andrew Ng + +2. You have overfit to the dev set. The process of repeatedly evaluating ideas on the dev set causes your algorithm to gradually “overfit” to the dev set. When you are done developing, you will evaluate your system on the test set. If you find that your dev set performance is much better than your test set performance, it is a sign that you have overfit to the dev set. In this case, get a fresh dev set. If you need to track your team’s progress, you can also evaluate your system regularly—say once per week or once per month—on the test set. But do not use the test set to make any decisions regarding the algorithm, including whether to roll back to the previous week’s system. If you do so, you will start to overfit to the test set, and can no longer count on it to give a completely unbiased estimate of your system’s performance (which you would need if you’re publishing research papers, or perhaps using this metric to make important business decisions). 3. The metric is measuring something other than what the project needs to optimize. Suppose that for your cat application, your metric is classification accuracy. This metric currently ranks classifier A as superior to classifier B. But suppose you try out both algorithms, and find classifier A is allowing occasional pornographic images to slip through. Even though classifier A is more accurate, the bad impression left by the occasional pornographic image means its performance is unacceptable. What do you do? Here, the metric is failing to identify the fact that Algorithm B is in fact better than Algorithm A for your product. So, you can no longer trust the metric to pick the best algorithm. It is time to change evaluation metrics. For example, you can change the metric to heavily penalize letting through pornographic images. I would strongly recommend picking a new metric and using the new metric to explicitly define a new goal for the team, rather than proceeding for too long without a trusted metric and reverting to manually choosing among classifiers. It is quite common to change dev/test sets or evaluation metrics during a project. Having an initial dev/test set and metric helps you iterate quickly. If you ever find that the dev/test sets or metric are no longer pointing your team in the right direction, it’s not a big deal! Just change them and make sure your team knows about the new direction. + +> Page 26 Machine Learning Yearning-Draft Andrew Ng + +# 12 Takeaways: Setting up development and test sets + +• Choose dev and test sets from a distribution that reflects what data you expect to get in the future and want to do well on. This may not be the same as your training data’s distribution. + +• Choose dev and test sets from the same distribution if possible. + +• Choose a single-number evaluation metric for your team to optimize. If there are multiple goals that you care about, consider combining them into a single formula (such as averaging multiple error metrics) or defining satisficing and optimizing metrics. + +• Machine learning is a highly iterative process: You may try many dozens of ideas before finding one that you’re satisfied with. + +• Having dev/test sets and a single-number evaluation metric helps you quickly evaluate algorithms, and therefore iterate faster. + +• When starting out on a brand new application, try to establish dev/test sets and a metric quickly, say in less than a week. It might be okay to take longer on mature applications. + +• The old heuristic of a 70%/30% train/test split does not apply for problems where you have a lot of data; the dev and test sets can be much less than 30% of the data. + +• Your dev set should be large enough to detect meaningful changes in the accuracy of your algorithm, but not necessarily much larger. Your test set should be big enough to give you a confident estimate of the final performance of your system. + +• If your dev set and metric are no longer pointing your team in the right direction, quickly change them: (i) If you had overfit the dev set, get more dev set data. (ii) If the actual distribution you care about is different from the dev/test set distribution, get new dev/test set data. (iii) If your metric is no longer measuring what is most important to you, change the metric. + +> Page 27 Machine Learning Yearning-Draft Andrew Ng + +# Basic Error Analysis + +Page 28 Machine Learning Yearning-Draft Andrew Ng 13 Build your first system quickly, then iterate + +You want to build a new email anti-spam system. Your team has several ideas: + +• Collect a huge training set of spam email. For example, set up a “honeypot”: deliberately send fake email addresses to known spammers, so that you can automatically harvest the spam messages they send to those addresses. + +• Develop features for understanding the text content of the email. + +• Develop features for understanding the email envelope/header features to show what set of internet servers the message went through. + +• and more. Even though I have worked extensively on anti-spam, I would still have a hard time picking one of these directions. It is even harder if you are not an expert in the application area. So don’t start off trying to design and build the perfect system. Instead, build and train a basic system quickly—perhaps in just a few days. Even if the basic system is far from the + +> 5 + +“best” system you can build, it is valuable to examine how the basic system functions: you will quickly find clues that show you the most promising directions in which to invest your time. These next few chapters will show you how to read these clues. + +> 5 + +This advice is meant for readers wanting to build AI applications, rather than those whose goal is to publish academic papers. I will later return to the topic of doing research. + +> Page 29 Machine Learning Yearning-Draft Andrew Ng + +# 14 Error analysis: Look at dev set examples to evaluate ideas + +When you play with your cat app, you notice several examples where it mistakes dogs for cats. Some dogs do look like cats! A team member proposes incorporating 3rd party software that will make the system do better on dog images. These changes will take a month, and the team member is enthusiastic. Should you ask them to go ahead? Before investing a month on this task, I recommend that you first estimate how much it will actually improve the system’s accuracy. Then you can more rationally decide if this is worth the month of development time, or if you’re better off using that time on other tasks. In detail, here’s what you can do: + +1. Gather a sample of 100 dev set examples that your system misclassified . I.e., examples that your system made an error on. + +2. Look at these examples manually, and count what fraction of them are dog images. The process of looking at misclassified examples is called error analysis . In this example, if you find that only 5% of the misclassified images are dogs, then no matter how much you improve your algorithm’s performance on dog images, you won’t get rid of more than 5% of your errors. In other words, 5% is a “ceiling” (meaning maximum possible amount) for how much the proposed project could help. Thus, if your overall system is currently 90% accurate (10% error), this improvement is likely to result in at best 90.5% accuracy (or 9.5% error, which is 5% less error than the original 10% error). + +> Page 30 Machine Learning Yearning-Draft Andrew Ng + +In contrast, if you find that 50% of the mistakes are dogs, then you can be more confident that the proposed project will have a big impact. It could boost accuracy from 90% to 95% (a 50% relative reduction in error, from 10% down to 5%). This simple counting procedure of error analysis gives you a quick way to estimate the possible value of incorporating the 3rd party software for dog images. It provides a quantitative basis on which to decide whether to make this investment. Error analysis can often help you figure out how promising different directions are. I’ve seen many engineers reluctant to carry out error analysis. It often feels more exciting to just jump in and implement some idea, rather than question if the idea is worth the time investment. This is a common mistake: It might result in your team spending a month only to realize afterward that it resulted in little benefit. Manually examining 100 examples does not take long. Even if you take one minute per image, you’d be done in under two hours. These two hours could save you a month of wasted effort. + +Error Analysis refers to the process of examining dev set examples that your algorithm misclassified, so that you can understand the underlying causes of the errors. This can help you prioritize projects—as in this example—and inspire new directions, which we will discuss next. The next few chapters will also present best practices for carrying out error analyses. + +> Page 31 Machine Learning Yearning-Draft Andrew Ng + +# 15 Evaluating multiple ideas in parallel during error analysis + +Your team has several ideas for improving the cat detector: + +• Fix the problem of your algorithm recognizing dogs as cats. + +• Fix the problem of your algorithm recognizing great cats (lions, panthers, etc.) as house cats (pets). + +• Improve the system’s performance on blurry images. + +• …You can efficiently evaluate all of these ideas in parallel. I usually create a spreadsheet and fill it out while looking through ~100 misclassified dev set images. I also jot down comments that might help me remember specific examples. To illustrate this process, let’s look at a spreadsheet you might produce with a small dev set of four examples: + +Image Dog Great cat Blurry Comments 1 ✔ Unusual pitbull color + +2 ✔ + +3 ✔ ✔ Lion; picture taken at zoo on rainy day + +4 ✔ Panther behind tree + +% of total 25% 50% 50% + +Image #3 above has both the Great Cat and the Blurry columns checked. Furthermore, because it is possible for one example to be associated with multiple categories, the percentages at the bottom may not add up to 100%. Although you may first formulate the categories (Dog, Great cat, Blurry) then categorize the examples by hand, in practice, once you start looking through examples, you will probably be inspired to propose new error categories. For example, say you go through a dozen images and realize a lot of mistakes occur with Instagram-filtered pictures. You can go back and add a new “Instagram” column to the spreadsheet. Manually looking at examples that the algorithm misclassified and asking how/whether you as a human could have labeled the + +> Page 32 Machine Learning Yearning-Draft Andrew Ng + +picture correctly will often inspire you to come up with new categories of errors and solutions. The most helpful error categories will be ones that you have an idea for improving. For example, the Instagram category will be most helpful to add if you have an idea to “undo” Instagram filters and recover the original image. But you don’t have to restrict yourself only to error categories you know how to improve; the goal of this process is to build your intuition about the most promising areas to focus on. Error analysis is an iterative process. Don’t worry if you start off with no categories in mind. After looking at a couple of images, you might come up with a few ideas for error categories. After manually categorizing some images, you might think of new categories and re-examine the images in light of the new categories, and so on. Suppose you finish carrying out error analysis on 100 misclassified dev set examples and get the following: + +> Image Dog Great cat Blurry Comments 1✔Usual pitbull color +> 2✔ +> 3✔✔Lion; picture taken at zoo on rainy day +> 4✔Panther behind tree +> …………... +> % of total 8% 43% 61% + +You now know that working on a project to address the Dog mistakes can eliminate 8% of the errors at most. Working on Great Cat or Blurry image errors could help eliminate more errors. Therefore, you might pick one of the two latter categories to focus on. If your team has enough people to pursue multiple directions in parallel, you can also ask some engineers to work on Great Cats and others to work on Blurry images. Error analysis does not produce a rigid mathematical formula that tells you what the highest priority task should be. You also have to take into account how much progress you expect to make on different categories and the amount of work needed to tackle each one. + +> Page 33 Machine Learning Yearning-Draft Andrew Ng + +# 16 Cleaning up mislabeled dev and test set examples + +During error analysis, you might notice that some examples in your dev set are mislabeled. When I say “mislabeled” here, I mean that the pictures were already mislabeled by a human labeler even before the algorithm encountered it. I.e., the class label in an example (x,y) has an incorrect value for y. For example, perhaps some pictures that are not cats are mislabeled as containing a cat, and vice versa. If you suspect the fraction of mislabeled images is significant, add a category to keep track of the fraction of examples mislabeled: + +Image Dog Great cat Blurry Mislabeled Comments … + +98 ✔ Labeler missed cat in background + +99 ✔ + +100 ✔ Drawing of a cat; not a real cat. + +% of total 8% 43% 61% 6% + +Should you correct the labels in your dev set? Remember that the goal of the dev set is to help you quickly evaluate algorithms so that you can tell if Algorithm A or B is better. If the fraction of the dev set that is mislabeled impedes your ability to make these judgments, then it is worth spending time to fix the mislabeled dev set labels. For example, suppose your classifier’s performance is: + +• Overall accuracy on dev set.………………. 90% (10% overall error.) + +• Errors due to mislabeled examples……. 0.6% (6% of dev set errors.) + +• Errors due to other causes………………… 9.4% (94% of dev set errors) Here, the 0.6% inaccuracy due to mislabeling might not be significant enough relative to the 9.4% of errors you could be improving. There is no harm in manually fixing the mislabeled images in the dev set, but it is not crucial to do so: It might be fine not knowing whether your system has 10% or 9.4% overall error. Suppose you keep improving the cat classifier and reach the following performance: + +> Page 34 Machine Learning Yearning-Draft Andrew Ng + +• Overall accuracy on dev set.………………. 98.0% (2.0% overall error.) + +• Errors due to mislabeled examples……. 0.6%. (30% of dev set errors.) + +• Errors due to other causes………………… 1.4% (70% of dev set errors) 30% of your errors are due to the mislabeled dev set images, adding significant error to your estimates of accuracy. It is now worthwhile to improve the quality of the labels in the dev set. Tackling the mislabeled examples will help you figure out if a classifier’s error is closer to 1.4% or 2%—a significant relative difference. It is not uncommon to start off tolerating some mislabeled dev/test set examples, only later to change your mind as your system improves so that the fraction of mislabeled examples grows relative to the total set of errors. The last chapter explained how you can improve error categories such as Dog, Great Cat and Blurry through algorithmic improvements. You have learned in this chapter that you can work on the Mislabeled category as well—through improving the data’s labels. Whatever process you apply to fixing dev set labels, remember to apply it to the test set labels too so that your dev and test sets continue to be drawn from the same distribution. Fixing your dev and test sets together would prevent the problem we discussed in Chapter 6, where your team optimizes for dev set performance only to realize later that they are being judged on a different criterion based on a different test set. If you decide to improve the label quality, consider double-checking both the labels of examples that your system misclassified as well as labels of examples it correctly classified. It is possible that both the original label and your learning algorithm were wrong on an example. If you fix only the labels of examples that your system had misclassified, you might introduce bias into your evaluation. If you have 1,000 dev set examples, and if your classifier has 98.0% accuracy, it is easier to examine the 20 examples it misclassified than to examine all 980 examples classified correctly. Because it is easier in practice to check only the misclassified examples, bias does creep into some dev sets. This bias is acceptable if you are interested only in developing a product or application, but it would be a problem if you plan to use the result in an academic research paper or need a completely unbiased measure of test set accuracy. + +> Page 35 Machine Learning Yearning-Draft Andrew Ng + +# 17 If you have a large dev set, split it into two subsets, only one of which you look at + +Suppose you have a large dev set of 5,000 examples in which you have a 20% error rate. Thus, your algorithm is misclassifying ~1,000 dev images. It takes a long time to manually examine 1,000 images, so we might decide not to use all of them in the error analysis. In this case, I would explicitly split the dev set into two subsets, one of which you look at, and one of which you don’t. You will more rapidly overfit the portion that you are manually looking at. You can use the portion you are not manually looking at to tune parameters. Let ’s continue our example above, in which the algorithm is misclassifying 1,000 out of 5,000 dev set examples. Suppose we want to manually examine about 100 errors for error analysis (10% of the errors). You should randomly select 10% of the dev set and place that into what we’ll call an Eyeball dev set to remind ourselves that we are looking at it with our eyes. (For a project on speech recognition, in which you would be listening to audio clips, perhaps you would call this set an Ear dev set instead). The Eyeball dev set therefore has 500 examples, of which we would expect our algorithm to misclassify about 100. The second subset of the dev set, called the Blackbox dev set , will have the remaining 4500 examples. You can use the Blackbox dev set to evaluate classifiers automatically by measuring their error rates. You can also use it to select among algorithms or tune hyperparameters. However, you should avoid looking at it with your eyes. We use the term “Blackbox” because we will only use this subset of the data to obtain “Blackbox” evaluations of classifiers. + +> Page 36 Machine Learning Yearning-Draft Andrew Ng + +Why do we explicitly separate the dev set into Eyeball and Blackbox dev sets? Since you will gain intuition about the examples in the Eyeball dev set, you will start to overfit the Eyeball dev set faster. If you see the performance on the Eyeball dev set improving much more rapidly than the performance on the Blackbox dev set, you have overfit the Eyeball dev set. In this case, you might need to discard it and find a new Eyeball dev set by moving more examples from the Blackbox dev set into the Eyeball dev set or by acquiring new labeled data. Explicitly splitting your dev set into Eyeball and Blackbox dev sets allows you to tell when your manual error analysis process is causing you to overfit the Eyeball portion of your data. + +> Page 37 Machine Learning Yearning-Draft Andrew Ng + +# 18 How big should the Eyeball and Blackbox dev sets be? + +Your Eyeball dev set should be large enough to give you a sense of your algorithm’s major error categories. If you are working on a task that humans do well (such as recognizing cats in images), here are some rough guidelines: + +• An eyeball dev set in which your classifier makes 10 mistakes would be considered very small. With just 10 errors, it’s hard to accurately estimate the impact of different error categories. But if you have very little data and cannot afford to put more into the Eyeball dev set, it ’s better than nothing and will help with project prioritization. + +• If your classifier makes ~20 mistakes on eyeball dev examples, you would start to get a rough sense of the major error sources. + +• With ~50 mistakes, you would get a good sense of the major error sources. + +• With ~100 mistakes, you would get a very good sense of the major sources of errors. I’ve seen people manually analyze even more errors—sometimes as many as 500. There is no harm in this as long as you have enough data. Say your classifier has a 5% error rate. To make sure you have ~100 misclassified examples in the Eyeball dev set, the Eyeball dev set would have to have about 2,000 examples (since 0.05*2,000 = 100). The lower your classifier’s error rate, the larger your Eyeball dev set needs to be in order to get a large enough set of errors to analyze. If you are working on a task that even humans cannot do well, then the exercise of examining an Eyeball dev set will not be as helpful because it is harder to figure out why the algorithm didn’t classify an example correctly. In this case, you might omit having an Eyeball dev set. We discuss guidelines for such problems in a later chapter. + +> Page 38 Machine Learning Yearning-Draft Andrew Ng + +How about the Blackbox dev set? We previously said that dev sets of around 1,000-10,000 examples are common. To refine that statement, a Blackbox dev set of 1,000-10,000 examples will often give you enough data to tune hyperparameters and select among models, though there is little harm in having even more data. A Blackbox dev set of 100 would be small but still useful. If you have a small dev set, then you might not have enough data to split into Eyeball and Blackbox dev sets that are both large enough to serve their purposes. Instead, your entire dev set might have to be used as the Eyeball dev set—i.e., you would manually examine all the dev set data. Between the Eyeball and Blackbox dev sets, I consider the Eyeball dev set more important (assuming that you are working on a problem that humans can solve well and that examining the examples helps you gain insight). If you only have an Eyeball dev set, you can perform error analyses, model selection and hyperparameter tuning all on that set. The downside of having only an Eyeball dev set is that the risk of overfitting the dev set is greater. If you have plentiful access to data, then the size of the Eyeball dev set would be determined mainly by how many examples you have time to manually analyze. For example, I’ve rarely seen anyone manually analyze more than 1,000 errors. + +> Page 39 Machine Learning Yearning-Draft Andrew Ng + +# 19 Takeaways: Basic error analysis + +• When you start a new project, especially if it is in an area in which you are not an expert, it is hard to correctly guess the most promising directions. + +• So don’t start off trying to design and build the perfect system. Instead build and train a basic system as quickly as possible—perhaps in a few days. Then use error analysis to help you identify the most promising directions and iteratively improve your algorithm from there. + +• Carry out error analysis by manually examining ~100 dev set examples the algorithm misclassifies and counting the major categories of errors. Use this information to prioritize what types of errors to work on fixing. + +• Consider splitting the dev set into an Eyeball dev set, which you will manually examine, and a Blackbox dev set, which you will not manually examine. If performance on the Eyeball dev set is much better than the Blackbox dev set, you have overfit the Eyeball dev set and should consider acquiring more data for it. + +• The Eyeball dev set should be big enough so that your algorithm misclassifies enough examples for you to analyze. A Blackbox dev set of 1,000-10,000 examples is sufficient for many applications. + +• If your dev set is not big enough to split this way, just use the entire dev set as an Eyeball dev set for manual error analysis, model selection, and hyperparameter tuning. + +> Page 40 Machine Learning Yearning-Draft Andrew Ng + +# Bias and Variance + +Page 41 Machine Learning Yearning-Draft Andrew Ng 20 Bias and Variance: The two big sources of error + +Suppose your training, dev and test sets all come from the same distribution. Then you should always try to get more training data, since that can only improve performance, right? Even though having more data can’t hurt, unfortunately it doesn’t always help as much as you might hope. It could be a waste of time to work on getting more data. So, how do you decide when to add data, and when not to bother? There are two major sources of error in machine learning: bias and variance. Understanding them will help you decide whether adding data, as well as other tactics to improve performance, are a good use of time. Suppose you hope to build a cat recognizer that has 5% error. Right now, your training set has an error rate of 15%, and your dev set has an error rate of 16%. In this case, adding training data probably won’t help much. You should focus on other changes. Indeed, adding more examples to your training set only makes it harder for your algorithm to do well on the training set. (We explain why in a later chapter.) If your error rate on the training set is 15% (or 85% accuracy), but your target is 5% error (95% accuracy), then the first problem to solve is to improve your algorithm ’s performance on your training set. Your dev/test set performance is usually worse than your training set performance. So if you are getting 85% accuracy on the examples your algorithm has seen, there’s no way you’re getting 95% accuracy on examples your algorithm hasn’t even seen. Suppose as above that your algorithm has 16% error (84% accuracy) on the dev set. We break the 16% error into two components: + +• First, the algorithm’s error rate on the training set. In this example, it is 15%. We think of this informally as the algorithm’s bias . + +• Second, how much worse the algorithm does on the dev (or test) set than the training set. In this example, it does 1% worse on the dev set than the training set. We think of this informally as the algorithm’s variance .6 + +> 6 + +The field of statistics has more formal definitions of bias and variance that we won’t worry about. Roughly, the bias is the error rate of your algorithm on your training set when you have a very large training set. The variance is how much worse you do on the test set compared to the training set in + +> Page 42 Machine Learning Yearning-Draft Andrew Ng + +Some changes to a learning algorithm can address the first component of error— bias —and improve its performance on the training set. Some changes address the second component— variance —and help it generalize better from the training set to the dev/test sets. To select the most promising changes, it is incredibly useful to understand which of + +> 7 + +these two components of error is more pressing to address. Developing good intuition about Bias and Variance will help you choose effective changes for your algorithm. + +this setting. When your error metric is mean squared error, you can write down formulas specifying these two quantities, and prove that Total Error = Bias + Variance. But for our purposes of deciding how to make progress on an ML problem, the more informal definition of bias and variance given here will suffice. + +> 7 + +There are also some methods that can simultaneously reduce bias and variance, by making major changes to the system architecture. But these tend to be harder to identify and implement. + +> Page 43 Machine Learning Yearning-Draft Andrew Ng + +# 21 Examples of Bias and Variance + +Consider our cat classification task. An “ideal” classifier (such as a human) might achieve nearly perfect performance in this task. Suppose your algorithm performs as follows: + +• Training error = 1% + +• Dev error = 11% What problem does it have? Applying the definitions from the previous chapter, we estimate the bias as 1%, and the variance as 10% (=11%-1%). Thus, it has high variance . The classifier has very low training error, but it is failing to generalize to the dev set. This is also called overfitting .Now consider this: + +• Training error = 15% + +• Dev error = 16% We estimate the bias as 15%, and variance as 1%. This classifier is fitting the training set poorly with 15% error, but its error on the dev set is barely higher than the training error. This classifier therefore has high bias , but low variance. We say that this algorithm is + +underfitting .Now, consider this: + +• Training error = 15% + +• Dev error = 30% We estimate the bias as 15%, and variance as 15%. This classifier has high bias and high variance : It is doing poorly on the training set, and therefore has high bias, and its performance on the dev set is even worse, so it also has high variance. The overfitting/underfitting terminology is hard to apply here since the classifier is simultaneously overfitting and underfitting. + +> Page 44 Machine Learning Yearning-Draft Andrew Ng + +Finally, consider this: + +• Training error = 0.5% + +• Dev error = 1% This classifier is doing well, as it has low bias and low variance. Congratulations on achieving this great performance! + +> Page 45 Machine Learning Yearning-Draft Andrew Ng + +# 22 Comparing to the optimal error rate + +In our cat recognition example, the “ideal” error rate—that is, one achievable by an “optimal” classifier—is nearly 0%. A human looking at a picture would be able to recognize if it contains a cat almost all the time; thus, we can hope for a machine that would do just as well. Other problems are harder. For example, suppose that you are building a speech recognition system, and find that 14% of the audio clips have so much background noise or are so unintelligible that even a human cannot recognize what was said. In this case, even the most “optimal” speech recognition system might have error around 14%. Suppose that on this speech recognition problem, your algorithm achieves: + +• Training error = 15% + +• Dev error = 30% The training set performance is already close to the optimal error rate of 14%. Thus, there is not much room for improvement in terms of bias or in terms of training set performance. However, this algorithm is not generalizing well to the dev set; thus there is ample room for improvement in the errors due to variance. This example is similar to the third example from the previous chapter, which also had a training error of 15% and dev error of 30%. If the optimal error rate is ~0%, then a training error of 15% leaves much room for improvement. This suggests bias-reducing changes might be fruitful. But if the optimal error rate is 14%, then the same training set performance tells us that there’s little room for improvement in the classifier’s bias. For problems where the optimal error rate is far from zero, here ’s a more detailed breakdown of an algorithm ’s error. Continuing with our speech recognition example above, the total dev set error of 30% can be broken down as follows (a similar analysis can be applied to the test set error): + +• Optimal error rate (“unavoidable bias”) : 14%. Suppose we decide that, even with the best possible speech system in the world, we would still suffer 14% error. We can think of this as the “unavoidable” part of a learning algorithm ’s bias. + +> Page 46 Machine Learning Yearning-Draft Andrew Ng + +• Avoidable bias : 1%. This is calculated as the difference between the training error and the optimal error rate. 8 + +• Variance : 15%. The difference between the dev error and the training error. To relate this to our earlier definitions, Bias and Avoidable Bias are related as follows: 9 + +Bias = Optimal error rate (“unavoidable bias”) + Avoidable bias The “avoidable bias” reflects how much worse your algorithm performs on the training set than the “optimal classifier.” The concept of variance remains the same as before. In theory, we can always reduce variance to nearly zero by training on a massive training set. Thus, all variance is “avoidable” with a sufficiently large dataset, so there is no such thing as “unavoidable variance.” Consider one more example, where the optimal error rate is 14%, and we have: + +• Training error = 15% + +• Dev error = 16% Whereas in the previous chapter we called this a high bias classifier, now we would say that error from avoidable bias is 1%, and the error from variance is about 1%. Thus, the algorithm is already doing well, with little room for improvement. It is only 2% worse than the optimal error rate. We see from these examples that knowing the optimal error rate is helpful for guiding our next steps. In statistics, the optimal error rate is also called Bayes error rate , or Bayes rate. How do we know what the optimal error rate is? For tasks that humans are reasonably good at, such as recognizing pictures or transcribing audio clips, you can ask a human to provide labels then measure the accuracy of the human labels relative to your training set. This would give an estimate of the optimal error rate. If you are working on a problem that even + +> 8If this number is negative, you are doing better on the training set than the optimal error rate. This means you are overfitting on the training set, and the algorithm has over-memorized the training set. You should focus on variance reduction methods rather than on further bias reduction methods. +> 9These definitions are chosen to convey insight on how to improve your learning algorithm. These definitions are different than how statisticians define Bias and Variance. Technically, what I define here as “Bias” should be called “Error we attribute to bias”; and “Avoidable bias” should be “error we attribute to the learning algorithm’s bias that is over the optimal error rate.” +> Page 47 Machine Learning Yearning-Draft Andrew Ng + +humans have a hard time solving (e.g., predicting what movie to recommend, or what ad to show to a user) it can be hard to estimate the optimal error rate. In the section “Comparing to Human-Level Performance (Chapters 33 to 35), I will discuss in more detail the process of comparing a learning algorithm’s performance to human-level performance. In the last few chapters, you learned how to estimate avoidable/unavoidable bias and variance by looking at training and dev set error rates. The next chapter will discuss how you can use insights from such an analysis to prioritize techniques that reduce bias vs. techniques that reduce variance. There are very different techniques that you should apply depending on whether your project’s current problem is high (avoidable) bias or high variance. Read on! + +> Page 48 Machine Learning Yearning-Draft Andrew Ng + +# 23 Addressing Bias and Variance + +Here is the simplest formula for addressing bias and variance issues: + +• If you have high avoidable bias, increase the size of your model (for example, increase the size of your neural network by adding layers/neurons). + +• If you have high variance, add data to your training set. If you are able to increase the neural network size and increase training data without limit, it is possible to do very well on many learning problems. In practice, increasing the size of your model will eventually cause you to run into computational problems because training very large models is slow. You might also exhaust your ability to acquire more training data. (Even on the internet, there is only a finite number of cat pictures!) Different model architectures—for example, different neural network architectures—will have different amounts of bias/variance for your problem. A lot of recent deep learning research has developed many innovative model architectures. So if you are using neural networks, the academic literature can be a great source of inspiration. There are also many great open-source implementations on github. But the results of trying new architectures are less predictable than the simple formula of increasing the model size and adding data. Increasing the model size generally reduces bias, but it might also increase variance and the risk of overfitting. However, this overfitting problem usually arises only when you are not using regularization. If you include a well-designed regularization method, then you can usually safely increase the size of the model without increasing overfitting. Suppose you are applying deep learning, with L2 regularization or dropout, with the regularization parameter that performs best on the dev set. If you increase the model size, usually your performance will stay the same or improve; it is unlikely to worsen significantly. The only reason to avoid using a bigger model is the increased computational cost. + +> Page 49 Machine Learning Yearning-Draft Andrew Ng + +# 24 Bias vs. Variance tradeoff + +You might have heard of the “Bias vs. Variance tradeoff.” Of the changes you could make to most learning algorithms, there are some that reduce bias errors but at the cost of increasing variance, and vice versa. This creates a “trade off” between bias and variance. For example, increasing the size of your model—adding neurons/layers in a neural network, or adding input features—generally reduces bias but could increase variance. Alternatively, adding regularization generally increases bias but reduces variance. In the modern era, we often have access to plentiful data and can use very large neural networks (deep learning). Therefore, there is less of a tradeoff, and there are now more options for reducing bias without hurting variance, and vice versa. For example, you can usually increase a neural network size and tune the regularization method to reduce bias without noticeably increasing variance. By adding training data, you can also usually reduce variance without affecting bias. If you select a model architecture that is well suited for your task, you might also reduce bias and variance simultaneously. Selecting such an architecture can be difficult. In the next few chapters, we discuss additional specific techniques for addressing bias and variance. + +> Page 50 Machine Learning Yearning-Draft Andrew Ng + +# 25 Techniques for reducing avoidable bias + +If your learning algorithm suffers from high avoidable bias, you might try the following techniques: + +• Increase the model size (such as number of neurons/layers): This technique reduces bias, since it should allow you to fit the training set better. If you find that this increases variance, then use regularization, which will usually eliminate the increase in variance. + +• Modify input features based on insights from error analysis : Say your error analysis inspires you to create additional features that help the algorithm eliminate a particular category of errors. (We discuss this further in the next chapter.) These new features could help with both bias and variance. In theory, adding more features could increase the variance; but if you find this to be the case, then use regularization, which will usually eliminate the increase in variance. + +• Reduce or eliminate regularization (L2 regularization, L1 regularization, dropout): This will reduce avoidable bias, but increase variance. + +• Modify model architecture (such as neural network architecture) so that it is more suitable for your problem: This technique can affect both bias and variance. One method that is not helpful: + +• Add more training data : This technique helps with variance problems, but it usually has no significant effect on bias. + +> Page 51 Machine Learning Yearning-Draft Andrew Ng + +# 26 Error analysis on the training set + +Your algorithm must perform well on the training set before you can expect it to perform well on the dev/test sets. In addition to the techniques described earlier to address high bias, I sometimes also carry out an error analysis on the training data , following a protocol similar to error analysis on the Eyeball dev set. This can be useful if your algorithm has high bias—i.e., if it is not fitting the training set well. For example, suppose you are building a speech recognition system for an app and have collected a training set of audio clips from volunteers. If your system is not doing well on the training set, you might consider listening to a set of ~100 examples that the algorithm is doing poorly on to understand the major categories of training set errors. Similar to the dev set error analysis, you can count the errors in different categories: + +> Audio clip Loud background noise User spoke quickly Far from microphone Comments 1✔Car noise +> 2✔✔Restaurant noise +> 3✔✔User shouting across living room? +> 4✔Coffeeshop +> % of total 75% 25% 50% + +In this example, you might realize that your algorithm is having a particularly hard time with training examples that have a lot of background noise. Thus, you might focus on techniques that allow it to better fit training examples with background noise. You might also double-check whether it is possible for a person to transcribe these audio clips, given the same input audio as your learning algorithm. If there is so much background noise that it is simply impossible for anyone to make out what was said, then it might be unreasonable to expect any algorithm to correctly recognize such utterances. We will discuss the benefits of comparing your algorithm to human-level performance in a later section. + +> Page 52 Machine Learning Yearning-Draft Andrew Ng + +# 27 Techniques for reducing variance + +If your learning algorithm suffers from high variance, you might try the following techniques: + +• Add more training data : This is the simplest and most reliable way to address variance, so long as you have access to significantly more data and enough computational power to process the data. + +• Add regularization (L2 regularization, L1 regularization, dropout): This technique reduces variance but increases bias. + +• Add early stopping (i.e., stop gradient descent early, based on dev set error): This technique reduces variance but increases bias. Early stopping behaves a lot like regularization methods, and some authors call it a regularization technique. + +• Feature selection to decrease number/type of input features: This technique might help with variance problems, but it might also increase bias. Reducing the number of features slightly (say going from 1,000 features to 900) is unlikely to have a huge effect on bias. Reducing it significantly (say going from 1,000 features to 100—a 10x reduction) is more likely to have a significant effect, so long as you are not excluding too many useful features. In modern deep learning, when data is plentiful, there has been a shift away from feature selection, and we are now more likely to give all the features we have to the algorithm and let the algorithm sort out which ones to use based on the data. But when your training set is small, feature selection can be very useful. + +• Decrease the model size (such as number of neurons/layers): Use with caution. This technique could decrease variance, while possibly increasing bias. However, I don’t recommend this technique for addressing variance. Adding regularization usually gives better classification performance. The advantage of reducing the model size is reducing your computational cost and thus speeding up how quickly you can train models. If speeding up model training is useful, then by all means consider decreasing the model size. But if your goal is to reduce variance, and you are not concerned about the computational cost, consider adding regularization instead. Here are two additional tactics, repeated from the previous chapter on addressing bias: + +• Modify input features based on insights from error analysis : Say your error analysis inspires you to create additional features that help the algorithm to eliminate a particular category of errors. These new features could help with both bias and variance. In + +> Page 53 Machine Learning Yearning-Draft Andrew Ng + +theory, adding more features could increase the variance; but if you find this to be the case, then use regularization, which will usually eliminate the increase in variance. + +• Modify model architecture (such as neural network architecture) so that it is more suitable for your problem: This technique can affect both bias and variance. + +> Page 54 Machine Learning Yearning-Draft Andrew Ng + +# Learning curves + +Page 55 Machine Learning Yearning-Draft Andrew Ng 28 Diagnosing bias and variance: Learning curves + +We’ve seen some ways to estimate how much error can be attributed to avoidable bias vs. variance. We did so by estimating the optimal error rate and computing the algorithm’s training set and dev set errors. Let’s discuss a technique that is even more informative: plotting a learning curve. A learning curve plots your dev set error against the number of training examples. To plot it, you would run your algorithm using different training set sizes. For example, if you have 1,000 examples, you might train separate copies of the algorithm on 100, 200, 300, …, 1000 examples. Then you could plot how dev set error varies with the training set size. Here is an example: As the training set size increases, the dev set error should decrease. We will often have some “desired error rate” that we hope our learning algorithm will eventually achieve. For example: + +• If we hope for human-level performance, then the human error rate could be the “desired error rate.” + +• If our learning algorithm serves some product (such as delivering cat pictures), we might have an intuition about what level of performance is needed to give users a great experience. + +> Page 56 Machine Learning Yearning-Draft Andrew Ng + +• If you have worked on a important application for a long time, then you might have intuition about how much more progress you can reasonably make in the next quarter/year. Add the desired level of performance to your learning curve: You can visually extrapolate the red “dev error” curve to guess how much closer you could get to the desired level of performance by adding more data. In the example above, it looks plausible that doubling the training set size might allow you to reach the desired performance. But if the dev error curve has “plateaued” (i.e. flattened out), then you can immediately tell that adding more data won’t get you to your goal: Looking at the learning curve might therefore help you avoid spending months collecting twice as much training data, only to realize it does not help. + +> Page 57 Machine Learning Yearning-Draft Andrew Ng + +One downside of this process is that if you only look at the dev error curve, it can be hard to extrapolate and predict exactly where the red curve will go if you had more data. There is one additional plot that can help you estimate the impact of adding more data: the training error. + +> Page 58 Machine Learning Yearning-Draft Andrew Ng + +# 29 Plotting training error + +Your dev set (and test set) error should decrease as the training set size grows. But your training set error usually increases as the training set size grows. Let’s illustrate this effect with an example. Suppose your training set has only 2 examples: One cat image and one non-cat image. Then it is easy for the learning algorithms to “memorize” both examples in the training set, and get 0% training set error. Even if either or both of the training examples were mislabeled, it is still easy for the algorithm to memorize both labels. Now suppose your training set has 100 examples. Perhaps even a few examples are mislabeled, or ambiguous—some images are very blurry, so even humans cannot tell if there is a cat. Perhaps the learning algorithm can still “memorize” most or all of the training set, but it is now harder to obtain 100% accuracy. By increasing the training set from 2 to 100 examples, you will find that the training set accuracy will drop slightly. Finally, suppose your training set has 10,000 examples. In this case, it becomes even harder for the algorithm to perfectly fit all 10,000 examples, especially if some are ambiguous or mislabeled. Thus, your learning algorithm will do even worse on this training set. Let’s add a plot of training error to our earlier figures: You can see that the blue “training error” curve increases with the size of the training set. Furthermore, your algorithm usually does better on the training set than on the dev set; thus the red dev error curve usually lies strictly above the blue training error curve. Let’s discuss next how to interpret these plots. + +> Page 59 Machine Learning Yearning-Draft Andrew Ng + +# 30 Interpreting learning curves: High bias + +Suppose your dev error curve looks like this: + +We previously said that, if your dev error curve plateaus, you are unlikely to achieve the desired performance just by adding data. But it is hard to know exactly what an extrapolation of the red dev error curve will look like. If the dev set was small, you would be even less certain because the curves could be noisy. Suppose we add the training error curve to this plot and get the following: + +Now, you can be absolutely sure that adding more data will not, by itself, be sufficient. Why is that? Remember our two observations: + +> Page 60 Machine Learning Yearning-Draft Andrew Ng + +• As we add more training data, training error can only get worse. Thus, the blue training error curve can only stay the same or go higher, and thus it can only get further away from the (green line) level of desired performance. + +• The red dev error curve is usually higher than the blue training error. Thus, there’s almost no way that adding more data would allow the red dev error curve to drop down to the desired level of performance when even the training error is higher than the desired level of performance. Examining both the dev error curve and the training error curve on the same plot allows us to more confidently extrapolate the dev error curve. Suppose, for the sake of discussion, that the desired performance is our estimate of the optimal error rate. The figure above is then the standard “textbook” example of what a learning curve with high avoidable bias looks like: At the largest training set size—presumably corresponding to all the training data we have—there is a large gap between the training error and the desired performance, indicating large avoidable bias. Furthermore, the gap between the training and dev curves is small, indicating small variance. Previously, we were measuring training and dev set error only at the rightmost point of this plot, which corresponds to using all the available training data. Plotting the full learning curve gives us a more comprehensive picture of the algorithms’ performance on different training set sizes. + +> Page 61 Machine Learning Yearning-Draft Andrew Ng + +# 31 Interpreting learning curves: Other cases + +Consider this learning curve: + +Does this plot indicate high bias, high variance, or both? The blue training error curve is relatively low, and the red dev error curve is much higher than the blue training error. Thus, the bias is small, but the variance is large. Adding more training data will probably help close the gap between dev error and training error. Now, consider this: + +This time, the training error is large, as it is much higher than the desired level of performance. The dev error is also much larger than the training error. Thus, you have significant bias and significant variance. You will have to find a way to reduce both bias and variance in your algorithm. + +> Page 62 Machine Learning Yearning-Draft Andrew Ng + +# 32 Plotting learning curves + +Suppose you have a very small training set of 100 examples. You train your algorithm using a randomly chosen subset of 10 examples, then 20 examples, then 30, up to 100, increasing the number of examples by intervals of ten. You then use these 10 data points to plot your learning curve. You might find that the curve looks slightly noisy (meaning that the values are higher/lower than expected) at the smaller training set sizes. When training on just 10 randomly chosen examples, you might be unlucky and have a particularly “bad” training set, such as one with many ambiguous/mislabeled examples. Or, you might get lucky and get a particularly “good” training set. Having a small training set means that the dev and training errors may randomly fluctuate. If your machine learning application is heavily skewed toward one class (such as a cat classification task where the fraction of negative examples is much larger than positive examples), or if it has a huge number of classes (such as recognizing 100 different animal species), then the chance of selecting an especially “unrepresentative” or bad training set is also larger. For example, if 80% of your examples are negative examples (y=0), and only 20% are positive examples (y=1), then there is a chance that a training set of 10 examples contains only negative examples, thus making it very difficult for the algorithm to learn something meaningful. If the noise in the training curve makes it hard to see the true trends, here are two solutions: + +• Instead of training just one model on 10 examples, instead select several (say 3-10) different randomly chosen training sets of 10 examples by sampling with replacement 10 + +from your original set of 100. Train a different model on each of these, and compute the training and dev set error of each of the resulting models. Compute and plot the average training error and average dev set error. + +• If your training set is skewed towards one class, or if it has many classes, choose a “balanced” subset instead of 10 training examples at random out of the set of 100. For example, you can make sure that 2/10 of the examples are positive examples, and 8/10 are + +> 10 Here’s what sampling with replacement means: You would randomly pick 10 different examples out of the 100 to form your first training set. Then to form the second training set, you would again pick 10 examples, but without taking into account what had been chosen in the first training set. Thus, it is possible for one specific example to appear in both the first and second training sets. In contrast, if you were sampling without replacement , the second training set would be chosen from just the 90 examples that had not been chosen the first time around. In practice, sampling with or without replacement shouldn’t make a huge difference, but the former is common practice. +> Page 63 Machine Learning Yearning-Draft Andrew Ng + +negative. More generally, you can make sure the fraction of examples from each class is as close as possible to the overall fraction in the original training set. I would not bother with either of these techniques unless you have already tried plotting learning curves and concluded that the curves are too noisy to see the underlying trends. If your training set is large—say over 10,000 examples—and your class distribution is not very skewed, you probably won’t need these techniques. Finally, plotting a learning curve may be computationally expensive: For example, you might have to train ten models with 1,000, then 2,000, all the way up to 10,000 examples. Training models with small datasets is much faster than training models with large datasets. Thus, instead of evenly spacing out the training set sizes on a linear scale as above, you might train models with 1,000, 2,000, 4,000, 6,000, and 10,000 examples. This should still give you a clear sense of the trends in the learning curves. Of course, this technique is relevant only if the computational cost of training all the additional models is significant. + +> Page 64 Machine Learning Yearning-Draft Andrew Ng + +# Comparing to human-level performance + +Page 65 Machine Learning Yearning-Draft Andrew Ng 33 Why we compare to human-level performance + +Many machine learning systems aim to automate things that humans do well. Examples include image recognition, speech recognition, and email spam classification. Learning algorithms have also improved so much that we are now surpassing human-level performance on more and more of these tasks. Further, there are several reasons building an ML system is easier if you are trying to do a task that people can do well: + +1. Ease of obtaining data from human labelers. For example, since people recognize cat images well, it is straightforward for people to provide high accuracy labels for your learning algorithm. + +2. Error analysis can draw on human intuition. Suppose a speech recognition algorithm is doing worse than human-level recognition. Say it incorrectly transcribes an audio clip as “This recipe calls for a pear of apples,” mistaking “pair” for “pear.” You can draw on human intuition and try to understand what information a person uses to get the correct transcription, and use this knowledge to modify the learning algorithm. + +3. Use human-level performance to estimate the optimal error rate and also set a “desired error rate.” Suppose your algorithm achieves 10% error on a task, but a person achieves 2% error. Then we know that the optimal error rate is 2% or lower and the avoidable bias is at least 8%. Thus, you should try bias-reducing techniques. Even though item #3 might not sound important, I find that having a reasonable and achievable target error rate helps accelerate a team’s progress. Knowing your algorithm has high avoidable bias is incredibly valuable and opens up a menu of options to try. There are some tasks that even humans aren’t good at. For example, picking a book to recommend to you; or picking an ad to show a user on a website; or predicting the stock market. Computers already surpass the performance of most people on these tasks. With these applications, we run into the following problems: + +• It is harder to obtain labels. For example, it’s hard for human labelers to annotate a database of users with the “optimal” book recommendation. If you operate a website or app that sells books, you can obtain data by showing books to users and seeing what they buy. If you do not operate such a site, you need to find more creative ways to get data. + +> Page 66 Machine Learning Yearning-Draft Andrew Ng + +• Human intuition is harder to count on. For example, pretty much no one can predict the stock market. So if our stock prediction algorithm does no better than random guessing, it is hard to figure out how to improve it. + +• It is hard to know what the optimal error rate and reasonable desired error rate is. Suppose you already have a book recommendation system that is doing quite well. How do you know how much more it can improve without a human baseline? + +> Page 67 Machine Learning Yearning-Draft Andrew Ng + +# 34 How to define human-level performance + +Suppose you are working on a medical imaging application that automatically makes diagnoses from x-ray images. A typical person with no previous medical background besides some basic training achieves 15% error on this task. A junior doctor achieves 10% error. An experienced doctor achieves 5% error. And a small team of doctors that discuss and debate each image achieves 2% error. Which one of these error rates defines “human-level performance”? In this case, I would use 2% as the human-level performance proxy for our optimal error rate. You can also set 2% as the desired performance level because all three reasons from the previous chapter for comparing to human-level performance apply: + +• Ease of obtaining labeled data from human labelers. You can get a team of doctors to provide labels to you with a 2% error rate. + +• Error analysis can draw on human intuition. By discussing images with a team of doctors, you can draw on their intuitions. + +• Use human-level performance to estimate the optimal error rate and also set achievable “desired error rate.” It is reasonable to use 2% error as our estimate of the optimal error rate. The optimal error rate could be even lower than 2%, but it cannot be higher, since it is possible for a team of doctors to achieve 2% error. In contrast, it is not reasonable to use 5% or 10% as an estimate of the optimal error rate, since we know these estimates are necessarily too high. When it comes to obtaining labeled data, you might not want to discuss every image with an entire team of doctors since their time is expensive. Perhaps you can have a single junior doctor label the vast majority of cases and bring only the harder cases to more experienced doctors or to the team of doctors. If your system is currently at 40% error, then it doesn’t matter much whether you use a junior doctor (10% error) or an experienced doctor (5% error) to label your data and provide intuitions. But if your system is already at 10% error, then defining the human-level reference as 2% gives you better tools to keep improving your system. + +> Page 68 Machine Learning Yearning-Draft Andrew Ng + +# 35 Surpassing human-level performance + +You are working on speech recognition and have a dataset of audio clips. Suppose your dataset has many noisy audio clips so that even humans have 10% error. Suppose your system already achieves 8% error. Can you use any of the three techniques described in Chapter 33 to continue making rapid progress? If you can identify a subset of data in which humans significantly surpass your system, then you can still use those techniques to drive rapid progress. For example, suppose your system is much better than people at recognizing speech in noisy audio, but humans are still better at transcribing very rapidly spoken speech. For the subset of data with rapidly spoken speech: + +1. You can still obtain transcripts from humans that are higher quality than your algorithm’s output. + +2. You can draw on human intuition to understand why they correctly heard a rapidly spoken utterance when your system didn’t. + +3. You can use human-level performance on rapidly spoken speech as a desired performance target. More generally, so long as there are dev set examples where humans are right and your algorithm is wrong, then many of the techniques described earlier will apply. This is true even if, averaged over the entire dev/test set, your performance is already surpassing human-level performance. There are many important machine learning applications where machines surpass human level performance. For example, machines are better at predicting movie ratings, how long it takes for a delivery car to drive somewhere, or whether to approve loan applications. Only a subset of techniques apply once humans have a hard time identifying examples that the algorithm is clearly getting wrong. Consequently, progress is usually slower on problems where machines already surpass human-level performance, while progress is faster when machines are still trying to catch up to humans. + +> Page 69 Machine Learning Yearning-Draft Andrew Ng + +# Training and testing on different distributions + +Page 70 Machine Learning Yearning-Draft Andrew Ng 36 When you should train and test on different distributions + +Users of your cat pictures app have uploaded 10,000 images, which you have manually labeled as containing cats or not. You also have a larger set of 200,000 images that you downloaded off the internet. How should you define train/dev/test sets? Since the 10,000 user images closely reflect the actual probability distribution of data you want to do well on, you might use that for your dev and test sets. If you are training a data-hungry deep learning algorithm, you might give it the additional 200,000 internet images for training. Thus, your training and dev/test sets come from different probability distributions. How does this affect your work? Instead of partitioning our data into train/dev/test sets, we could take all 210,000 images we have, and randomly shuffle them into train/dev/test sets. In this case, all the data comes from the same distribution. But I recommend against this method, because about 205,000/210,000 ≈ 97.6% of your dev/test data would come from internet images, which does not reflect the actual distribution you want to do well on. Remember our recommendation on choosing dev/test sets: Choose dev and test sets to reflect data you expect to get in the future and want to do well on. Most of the academic literature on machine learning assumes that the training set, dev set and test set all come from the same distribution. In the early days of machine learning, data + +> 11 + +was scarce. We usually only had one dataset drawn from some probability distribution. So we would randomly split that data into train/dev/test sets, and the assumption that all the data was coming from the same source was usually satisfied. + +> 11 + +There is some academic research on training and testing on different distributions. Examples include “domain adaptation,” “transfer learning” and “multitask learning.” But there is still a huge gap between theory and practice. If you train on dataset A and test on some very different type of data B, luck could have a huge effect on how well your algorithm performs. (Here, “luck” includes the researcher’s hand-designed features for the particular task, as well as other factors that we just don’t understand yet.) This makes the academic study of training and testing on different distributions difficult to carry out in a systematic way. + +> Page 71 Machine Learning Yearning-Draft Andrew Ng + +But in the era of big data, we now have access to huge training sets, such as cat internet images. Even if the training set comes from a different distribution than the dev/test set, we still want to use it for learning since it can provide a lot of information. For the cat detector example, instead of putting all 10,000 user-uploaded images into the dev/test sets, we might instead put 5,000 into the dev/test sets. We can put the remaining 5,000 user-uploaded examples into the training set. This way, your training set of 205,000 examples contains some data that comes from your dev/test distribution along with the 200,000 internet images. We will discuss in a later chapter why this method is helpful. Let’s consider a second example. Suppose you are building a speech recognition system to transcribe street addresses for a voice-controlled mobile map/navigation app. You have 20,000 examples of users speaking street addresses. But you also have 500,000 examples of other audio clips with users speaking about other topics. You might take 10,000 examples of street addresses for the dev/test sets, and use the remaining 10,000, plus the additional 500,000 examples, for training. We will continue to assume that your dev data and your test data come from the same distribution. But it is important to understand that different training and dev/test distributions offer some special challenges. + +> Page 72 Machine Learning Yearning-Draft Andrew Ng + +# 37 How to decide whether to use all your data + +Suppose your cat detector’s training set includes 10,000 user-uploaded images. This data comes from the same distribution as a separate dev/test set, and represents the distribution you care about doing well on. You also have an additional 20,000 images downloaded from the internet. Should you provide all 20,000+10,000=30,000 images to your learning algorithm as its training set, or discard the 20,000 internet images for fear of it biasing your learning algorithm? When using earlier generations of learning algorithms (such as hand-designed computer vision features, followed by a simple linear classifier) there was a real risk that merging both types of data would cause you to perform worse. Thus, some engineers will warn you against including the 20,000 internet images. But in the modern era of powerful, flexible learning algorithms—such as large neural networks—this risk has greatly diminished. If you can afford to build a neural network with a large enough number of hidden units/layers, you can safely add the 20,000 images to your training set. Adding the images is more likely to increase your performance. This observation relies on the fact that there is some x —> y mapping that works well for both types of data. In other words, there exists some system that inputs either an internet image or a mobile app image and reliably predicts the label, even without knowing the source of the image. Adding the additional 20,000 images has the following effects: + +1. It gives your neural network more examples of what cats do/do not look like. This is helpful, since internet images and user-uploaded mobile app images do share some similarities. Your neural network can apply some of the knowledge acquired from internet images to mobile app images. + +2. It forces the neural network to expend some of its capacity to learn about properties that are specific to internet images (such as higher resolution, different distributions of how the images are framed, etc.) If these properties differ greatly from mobile app images, it will “use up” some of the representational capacity of the neural network. Thus there is less capacity for recognizing data drawn from the distribution of mobile app images, which is what you really care about. Theoretically, this could hurt your algorithms’ performance. + +> Page 73 Machine Learning Yearning-Draft Andrew Ng + +To describe the second effect in different terms, we can turn to the fictional character Sherlock Holmes, who says that your brain is like an attic; it only has a finite amount of space. He says that “for every addition of knowledge, you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.” 12 + +Fortunately, if you have the computational capacity needed to build a big enough neural network—i.e., a big enough attic—then this is not a serious concern. You have enough capacity to learn from both internet and from mobile app images, without the two types of data competing for capacity. Your algorithm’s “brain” is big enough that you don’t have to worry about running out of attic space. But if you do not have a big enough neural network (or another highly flexible learning algorithm), then you should pay more attention to your training data matching your dev/test set distribution. If you think you have data that has no benefit,you should just leave out that data for computational reasons. For example, suppose your dev/test sets contain mainly casual pictures of people, places, landmarks, animals. Suppose you also have a large collection of scanned historical documents: These documents don’t contain anything resembling a cat. They also look completely unlike your dev/test distribution. There is no point including this data as negative examples, because the benefit from the first effect above is negligible—there is almost nothing your neural network can learn from this data that it can apply to your dev/test set distribution. Including them would waste computation resources and representation capacity of the neural network. + +> 12 A Study in Scarlet by Arthur Conan Doyle +> Page 74 Machine Learning Yearning-Draft Andrew Ng + +# 38 How to decide whether to include inconsistent data + +Suppose you want to learn to predict housing prices in New York City. Given the size of a house (input feature x), you want to predict the price (target label y). Housing prices in New York City are very high. Suppose you have a second dataset of housing prices in Detroit, Michigan, where housing prices are much lower. Should you include this data in your training set? Given the same size x, the price of a house y is very different depending on whether it is in New York City or in Detroit. If you only care about predicting New York City housing prices, putting the two datasets together will hurt your performance. In this case, it would be better to leave out the inconsistent Detroit data. 13 + +How is this New York City vs. Detroit example different from the mobile app vs. internet cat images example? The cat image example is different because, given an input picture x, one can reliably predict the label y indicating whether there is a cat, even without knowing if the image is an internet image or a mobile app image. I.e., there is a function f(x) that reliably maps from the input x to the target output y, even without knowing the origin of x. Thus, the task of recognition from internet images is “consistent” with the task of recognition from mobile app images. This means there was little downside (other than computational cost) to including all the data, and some possible significant upside. In contrast, New York City and Detroit, Michigan data are not consistent. Given the same x (size of house), the price is very different depending on where the house is. + +> 13 + +There is one way to address the problem of Detroit data being inconsistent with New York City data, which is to add an extra feature to each training example indicating the city. Given an input x—which now specifies the city—the target value of y is now unambiguous. However, in practice I do not see this done frequently. + +> Page 75 Machine Learning Yearning-Draft Andrew Ng + +# 39 Weighting data + +Suppose you have 200,000 images from the internet and 5,000 images from your mobile app users. There is a 40:1 ratio between the size of these datasets. In theory, so long as you build a huge neural network and train it long enough on all 205,000 images, there is no harm in trying to make the algorithm do well on both internet images and mobile images. But in practice, having 40x as many internet images as mobile app images might mean you need to spend 40x (or more) as much computational resources to model both, compared to if you trained on only the 5,000 images. If you don’t have huge computational resources, you could give the internet images a much lower weight as a compromise. For example, suppose your optimization objective is squared error (This is not a good choice for a classification task, but it will simplify our explanation.) Thus, our learning algorithm tries to optimize: + +The first sum above is over the 5,000 mobile images, and the second sum is over the 200,000 internet images. You can instead optimize with an additional parameter 𝛽 : + +If you set 𝛽 =1/40, the algorithm would give equal weight to the 5,000 mobile images and the 200,000 internet images. You can also set the parameter 𝛽 to other values, perhaps by tuning to the dev set. By weighting the additional Internet images less, you don’t have to build as massive a neural network to make sure the algorithm does well on both types of tasks. This type of re-weighting is needed only when you suspect the additional data (Internet Images) has a very different distribution than the dev/test set, or if the additional data is much larger than the data that came from the same distribution as the dev/test set (mobile images). + +> Page 76 Machine Learning Yearning-Draft Andrew Ng + +# 40 Generalizing from the training set to the dev set + +Suppose you are applying ML in a setting where the training and the dev/test distributions are different. Say, the training set contains Internet images + Mobile images, and the dev/test sets contain only Mobile images. However, the algorithm is not working well: It has a much higher dev/test set error than you would like. Here are some possibilities of what might be wrong: + +1. It does not do well on the training set. This is the problem of high (avoidable) bias on the training set distribution. + +2. It does well on the training set, but does not generalize well to previously unseen data + +drawn from the same distribution as the training set . This is high variance. + +3. It generalizes well to new data drawn from the same distribution as the training set, but not to data drawn from the dev/test set distribution. We call this problem data mismatch , since it is because the training set data is a poor match for the dev/test set data. For example, suppose that humans achieve near perfect performance on the cat recognition task. Your algorithm achieves this: + +• 1% error on the training set + +• 1.5% error on data drawn from the same distribution as the training set that the algorithm has not seen + +• 10% error on the dev set In this case, you clearly have a data mismatch problem. To address this, you might try to make the training data more similar to the dev/test data. We discuss some techniques for this later. In order to diagnose to what extent an algorithm suffers from each of the problems 1-3 above, it will be useful to have another dataset. Specifically, rather than giving the algorithm all the available training data, you can split it into two subsets: The actual training set which the algorithm will train on, and a separate set, which we will call the “Training dev” set, that we will not train on. You now have four subsets of data: + +> Page 77 Machine Learning Yearning-Draft Andrew Ng + +• Training set. This is the data that the algorithm will learn from (e.g., Internet images + Mobile images). This does not have to be drawn from the same distribution as what we really care about (the dev/test set distribution). + +• Training dev set: This data is drawn from the same distribution as the training set (e.g., Internet images + Mobile images). This is usually smaller than the training set; it only needs to be large enough to evaluate and track the progress of our learning algorithm. + +• Dev set: This is drawn from the same distribution as the test set, and it reflects the distribution of data that we ultimately care about doing well on. (E.g., mobile images.) + +• Test set: This is drawn from the same distribution as the dev set. (E.g., mobile images.) Armed with these four separate datasets, you can now evaluate: + +• Training error, by evaluating on the training set. + +• The algorithm’s ability to generalize to new data drawn from the training set distribution, by evaluating on the training dev set. + +• The algorithm’s performance on the task you care about, by evaluating on the dev and/or test sets. Most of the guidelines in Chapters 5-7 for picking the size of the dev set also apply to the training dev set. + +> Page 78 Machine Learning Yearning-Draft Andrew Ng + +# 41 Identifying Bias, Variance, and Data Mismatch Errors + +Suppose humans achieve almost perfect performance (≈0% error) on the cat detection task, and thus the optimal error rate is about 0%. Suppose you have: + +• 1% error on the training set. + +• 5% error on training dev set. + +• 5% error on the dev set. What does this tell you? Here, you know that you have high variance. The variance reduction techniques described earlier should allow you to make progress. Now, suppose your algorithm achieves: + +• 10% error on the training set. + +• 11% error on training dev set. + +• 12% error on the dev set. This tells you that you have high avoidable bias on the training set. I.e., the algorithm is doing poorly on the training set. Bias reduction techniques should help. In the two examples above, the algorithm suffered from only high avoidable bias or high variance. It is possible for an algorithm to suffer from any subset of high avoidable bias, high variance, and data mismatch. For example: + +• 10% error on the training set. + +• 11% error on training dev set. + +• 20% error on the dev set. This algorithm suffers from high avoidable bias and from data mismatch. It does not, however, suffer from high variance on the training set distribution. It might be easier to understand how the different types of errors relate to each other by drawing them as entries in a table: + +> Page 79 Machine Learning Yearning-Draft Andrew Ng + +Continuing with the example of th e cat image detector, you can see that there are two different distributions of data on the x-axis. On the y-axis, we ha ve three types of error: human level error, error on examples the algorithm has trained on, and error on examples the algorithm has not trained on. We can fill in the boxes with the different types of errors we identified in the previous chapter. If you wish, you can also fill in the remaining two boxes in this table: You can fill in the upper-right box (Human level performance on Mobile Images) by asking some humans to label your mobile cat images data and measure their error. You can fill in the next box by taking the mobile cat images (Distribution B) and putting a small fraction of into the training set so that the neural network learns on it too. Then you measure the learned model’s error on that subset of data. Filling in these two additional entries may sometimes give additional insight about what the algorithm is doing on the two different distributions (Distribution A and B) of data. By understanding which types of error the algorithm suffers from the most, you will be better positioned to decide whether to focus on reducing bias, reducing variance, or reducing data mismatch. + +> Page 80 Machine Learning Yearning-Draft Andrew Ng + +# 42 Addressing data mismatch + +Suppose you have developed a speech recognition system that does very well on the training set and on the training dev set. However, it does poorly on your dev set: You have a data mismatch problem. What can you do? I recommend that you: (i) Try to understand what properties of the data differ between the training and the dev set distributions. (ii) Try to find more training data that better matches the dev set examples that your algorithm has trouble with. 14 + +For example, suppose you carry out an error analysis on the speech recognition dev set: You manually go through 100 examples, and try to understand where the algorithm is making mistakes. You find that your system does poorly because most of the audio clips in the dev set are taken within a car, whereas most of the training examples were recorded against a quiet background. The engine and road noise dramatically worsen the performance of your speech system. In this case, you might try to acquire more training data comprising audio clips that were taken in a car. The purpose of the error analysis is to understand the significant differences between the training and the dev set, which is what leads to the data mismatch. If your training and training dev sets include audio recorded within a car, you should also double-check your system’s performance on this subset of data. If it is doing well on the car data in the training set but not on car data in the training dev set, then this further validates the hypothesis that getting more car data would help. This is why we discussed the possibility of including in your training set some data drawn from the same distribution as your dev/test set in the previous chapter. Doing so allows you to compare your performance on the car data in the training set vs. the dev/test set. Unfortunately, there are no guarantees in this process. For example, if you don't have any way to get more training data that better match the dev set data, you might not have a clear path towards improving performance. + +> 14 + +There is also some research on “domain adaptation”—how to train an algorithm on one distribution and have it generalize to a different distribution. These methods are typically applicable only in special types of problems and are much less widely used than the ideas described in this chapter. + +> Page 81 Machine Learning Yearning-Draft Andrew Ng + +# 43 Artificial data synthesis + +Your speech system needs more data that sounds as if it were taken from within a car. Rather than collecting a lot of data while driving around, there might be an easier way to get this data: By artificially synthesizing it. Suppose you obtain a large quantity of car/road noise audio clips. You can download this data from several websites. Suppose you also have a large training set of people speaking in a quiet room. If you take an audio clip of a person speaking and “add” to that to an audio clip of car/road noise, you will obtain an audio clip that sounds as if that person was speaking in a noisy car. Using this process, you can “synthesize” huge amounts of data that sound as if it were collected inside a car. More generally, there are several circumstances where artificial data synthesis allows you to create a huge dataset that reasonably matches the dev set. Let’s use the cat image detector as a second example. You notice that dev set images have much more motion blur because they tend to come from cellphone users who are moving their phone slightly while taking the picture. You can take non-blurry images from the training set of internet images, and add simulated motion blur to them, thus making them more similar to the dev set. Keep in mind that artificial data synthesis has its challenges: it is sometimes easier to create synthetic data that appears realistic to a person than it is to create data that appears realistic to a computer. For example, suppose you have 1,000 hours of speech training data, but only 1 hour of car noise. If you repeatedly use the same 1 hour of car noise with different portions from the original 1,000 hours of training data, you will end up with a synthetic dataset where the same car noise is repeated over and over. While a person listening to this audio probably would not be able to tell—all car noise sounds the same to most of us—it is possible that a learning algorithm would “overfit” to the 1 hour of car noise. Thus, it could generalize poorly to a new audio clip where the car noise happens to sound different. Alternatively, suppose you have 1,000 unique hours of car noise, but all of it was taken from just 10 different cars. In this case, it is possible for an algorithm to “overfit” to these 10 cars and perform poorly if tested on audio from a different car. Unfortunately, these problems can be hard to spot. + +> Page 82 Machine Learning Yearning-Draft Andrew Ng + +To take one more example, suppose you are building a computer vision system to recognize cars. Suppose you partner with a video gaming company, which has computer graphics models of several cars. To train your algorithm, you use the models to generate synthetic images of cars. Even if the synthesized images look very realistic, this approach (which has been independently proposed by many people) will probably not work well. The video game might have ~20 car designs in the entire video game. It is very expensive to build a 3D car model of a car; if you were playing the game, you probably wouldn’t notice that you’re seeing the same cars over and over, perhaps only painted differently. I.e., this data looks very realistic to you. But compared to the set of all cars out on roads—and therefore what you’re likely to see in the dev/test sets—this set of 20 synthesized cars captures only a minuscule fraction of the world’s distribution of cars. Thus if your 100,000 training examples all come from these 20 cars, your system will “overfit” to these 20 specific car designs, and it will fail to generalize well to dev/test sets that include other car designs. When synthesizing data, put some thought into whether you’re really synthesizing a representative set of examples. Try to avoid giving the synthesized data properties that makes it possible for a learning algorithm to distinguish synthesized from non-synthesized examples—such as if all the synthesized data comes from one of 20 car designs, or all the synthesized audio comes from only 1 hour of car noise. This advice can be hard to follow. When working on data synthesis, my teams have sometimes taken weeks before we produced data with details that are close enough to the actual distribution for the synthesized data to have a significant effect. But if you are able to get the details right, you can suddenly access a far larger training set than before. + +> Page 83 Machine Learning Yearning-Draft Andrew Ng + +# Debugging inference algorithms + +Page 84 Machine Learning Yearning-Draft Andrew Ng 44 The Optimization Verification test + +Suppose you are building a speech recognition system. Your system works by inputting an audio clip A, and computing some Score A(S) for each possible output sentence S. For example, you might try to estimate Score A(S) = P( S|A), the probability that the correct output transcription is the sentence S, given that the input audio was A. + +Given a way to compute Score A(S), you still have to find the English sentence S that maximizes it: + +How do you compute the “arg max” above? If the English language has 50,000 words, then there are (50,000) N possible sentences of length N—far too many to exhaustively enumerate. So, you need to apply an approximate search algorithm, to try to find the value of S that optimizes (maximizes) Score A(S). One example search algorithm is “beam search,” which keeps only K top candidates during the search process. (For the purposes of this chapter, you don’t need to understand the details of beam search.) Algorithms like this are not guaranteed to find the value of S that maximizes Score A(S). Suppose that an audio clip A records someone saying “I love machine learning.” But instead of outputting the correct transcription, your system outputs the incorrect “I love robots.” There are now two possibilities for what went wrong: + +1. Search algorithm problem . The approximate search algorithm (beam search) failed to find the value of S that maximizes Score A(S). + +2. Objective (scoring function) problem. Our estimates for Score A(S) = P( S|A) were inaccurate. In particular, our choice of Score A(S) failed to recognize that “I love machine learning” is the correct transcription. Depending on which of these was the cause of the failure, you should prioritize your efforts very differently. If #1 was the problem, you should work on improving the search algorithm. If #2 was the problem, you should work on the learning algorithm that estimates Score A(S). Facing this situation, some researchers will randomly decide to work on the search algorithm; others will randomly work on a better way to learn values for Score A(S). But unless you know which of these is the underlying cause of the error, your efforts could be wasted. How can you decide more systematically what to work on? + +> Page 85 Machine Learning Yearning-Draft Andrew Ng + +Let S out be the output transcription (“I love robots”). Let S* be the correct transcription (“I love machine learning”). In order to understand whether #1 or #2 above is the problem, you can perform the Optimization Verification test : First, compute Score A(S*) and Score A(Sout ). Then check whether Score A(S*) > Score A(Sout ). There are two possibilities: Case 1: Score A(S*) > Score A(S out )In this case, your learning algorithm has correctly given S* a higher score than S out .Nevertheless, our approximate search algorithm chose S out rather than S*. This tells you that your approximate search algorithm is failing to choose the value of S that maximizes Score A(S). In this case, the Optimization Verification test tells you that you have a search algorithm problem and should focus on that. For example, you could try increasing the beam width of beam search. Case 2: Score A(S*) ≤ Score A(S out )In this case, you know that the way you’re computing Score A(.) is at fault: It is failing to give a strictly higher score to the correct output S* than the incorrect Sout . The Optimization Verification test tells you that you have an objective (scoring) function problem. Thus, you should focus on improving how you learn or approximate Score A(S) for different sentences S.Our discussion has focused on a single example. To apply the Optimization Verification test in practice, you should examine the errors in your dev set. For each error, you would test whether Score A(S*) > Score A(S out ). Each dev example for which this inequality holds will get marked as an error caused by the optimization algorithm. Each example for which this does not hold (Score A(S*) ≤ Score A(S out )) gets counted as a mistake due to the way you’re computing Score A(.). For example, suppose you find that 95% of the errors were due to the scoring function Score A(.), and only 5% due to the optimization algorithm. Now you know that no matter how much you improve your optimization procedure, you would realistically eliminate only ~5% of our errors. Thus, you should instead focus on improving how you estimate Score A(.). + +> Page 86 Machine Learning Yearning-Draft Andrew Ng + +# 45 General form of Optimization Verification test + +You can apply the Optimization Verification test when, given some input x, you know how to compute Score x(y) that indicates how good a response y is to an input x. Furthermore, you are using an approximate algorithm to try to find arg max y Score x(y), but suspect that the search algorithm is sometimes failing to find the maximum. In our previous speech recognition example, x=A was an audio clip, and y=S was the output transcript. Suppose y* is the “correct” output but the algorithm instead outputs y out . Then the key test is to measure whether Score x(y*) > Score x(y out ). If this inequality holds, then we blame the optimization algorithm for the mistake. Refer to the previous chapter to make sure you understand the logic behind this. Otherwise, we blame the computation of Score x(y). Let’s look at one more example. Suppose you are building a Chinese-to-English machine translation system. Your system works by inputting a Chinese sentence C, and computing some Score C(E) for each possible translation E. For example, you might use Score C(E) = P( E|C), the probability of the translation being E given that the input sentence was C.Your algorithm translates sentences by trying to compute: + +However, the set of all possible English sentences E is too large, so you rely on a heuristic search algorithm. Suppose your algorithm outputs an incorrect translation Eout rather than some correct translation E*. Then the Optimization Verification test would ask you to compute whether Score C(E* ) > Score C(Eout ). If this inequality holds, then the Score C(.) correctly recognized E* as a superior output to Eout ; thus, you would attribute this error to the approximate search algorithm. Otherwise, you attribute this error to the computation of Score C(.). It is a very common “design pattern” in AI to first learn an approximate scoring function Score x(.), then use an approximate maximization algorithm. If you are able to spot this pattern, you will be able to use the Optimization Verification test to understand your source of errors. + +> Page 87 Machine Learning Yearning-Draft Andrew Ng + +# 46 Reinforcement learning example + +Suppose you are using machine learning to teach a helicopter to fly complex maneuvers. Here is a time-lapse photo of a computer-controller helicopter executing a landing with the engine turned off. This is called an “autorotation” maneuver. It allows helicopters to land even if their engine unexpectedly fails. Human pilots practice this maneuver as part of their training. Your goal is to use a learning algorithm to fly the helicopter through a trajectory T that ends in a safe landing. To apply reinforcement learning, you have to develop a “Reward function” R(.) that gives a score measuring how good each possible trajectory T is. For example, if T results in the helicopter crashing, then perhaps the reward is R(T) = -1,000—a huge negative reward. A trajectory T resulting in a safe landing might result in a positive R(T) with the exact value depending on how smooth the landing was. The reward function R(.) is typically chosen by hand to quantify how desirable different trajectories T are. It has to trade off how bumpy the landing was, whether the helicopter landed in exactly the desired spot, how rough the ride down was for passengers, and so on. It is not easy to design good reward functions. + +> Page 88 Machine Learning Yearning-Draft Andrew Ng + +Given a reward function R(T), the job of the reinforcement learning algorithm is to control the helicopter so that it achieves max T R(T). However, reinforcement learning algorithms make many approximations and may not succeed in achieving this maximization. Suppose you have picked some reward R(.) and have run your learning algorithm. However, its performance appears far worse than your human pilot—the landings are bumpier and seem less safe than what a human pilot achieves. How can you tell if the fault is with the reinforcement learning algorithm—which is trying to carry out a trajectory that achieves max T R(T) —or if the fault is with the reward function—which is trying to measure as well as specify the ideal tradeoff between ride bumpiness and accuracy of landing spot? To apply the Optimization Verification test, let Thuman be the trajectory achieved by the human pilot, and let Tout be the trajectory achieved by the algorithm. According to our description above, Thuman is a superior trajectory to Tout . Thus, the key test is the following: Does it hold true that R(Thuman ) > R(Tout )? Case 1: If this inequality holds, then the reward function R(.) is correctly rating Thuman as superior to Tout . But our reinforcement learning algorithm is finding the inferior Tout. This suggests that working on improving our reinforcement learning algorithm is worthwhile. Case 2: The inequality does not hold: R(Thuman ) ≤ R(Tout ). This means R(.) assigns a worse score to Thuman even though it is the superior trajectory. You should work on improving R(.) to better capture the tradeoffs that correspond to a good landing. Many machine learning applications have this “pattern” of optimizing an approximate scoring function Score x(.) using an approximate search algorithm. Sometimes, there is no specified input x, so this reduces to just Score(.). In our example above, the scoring function was the reward function Score( T)=R( T), and the optimization algorithm was the reinforcement learning algorithm trying to execute a good trajectory T.One difference between this and earlier examples is that, rather than comparing to an “optimal” output, you were instead comparing to human-level performance Thuman .We assumed Thuman is pretty good, even if not optimal. In general, so long as you have some y* (in this example, Thuman ) that is a superior output to the performance of your current learning algorithm—even if it is not the “optimal” output—then the Optimization Verification test can indicate whether it is more promising to improve the optimization algorithm or the scoring function. + +> Page 89 Machine Learning Yearning-Draft Andrew Ng + +# End-to-end deep learning + +Page 90 Machine Learning Yearning-Draft Andrew Ng 47 The rise of end-to-end learning + +Suppose you want to build a system to examine online product reviews and automatically tell you if the writer liked or disliked that product. For example, you hope to recognize the following review as highly positive: This is a great mop! and the following as highly negative: This mop is low quality--I regret buying it. The problem of recognizing positive vs. negative opinions is called “sentiment classification.” To build this system, you might build a “pipeline” of two components: 1. Parser: A system that annotates the text with information identifying the most important words. For example, you might use the parser to label all the adjectives + +> 15 + +and nouns. You would therefore get the following annotated text: This is a great Adjective mop Noun !2. Sentiment classifier: A learning algorithm that takes as input the annotated text and predicts the overall sentiment. The parser’s annotation could help this learning algorithm greatly: By giving adjectives a higher weight, your algorithm will be able to quickly hone in on the important words such as “great,” and ignore less important words such as “this.” We can visualize your “pipeline” of two components as follows: There has been a recent trend toward replacing pipeline systems with a single learning algorithm. An end-to-end learning algorithm for this task would simply take as input the raw, original text “This is a great mop!”, and try to directly recognize the sentiment: + +> 15 + +A parser gives a much richer annotation of the text than this, but this simplified description will suffice for explaining end-to-end deep learning. + +> Page 91 Machine Learning Yearning-Draft Andrew Ng + +Neural networks are commonly used in end-to-end learning systems. The term “end-to-end” refers to the fact that we are asking the learning algorithm to go directly from the input to the desired output. I.e., the learning algorithm directly connects the “input end” of the system to the “output end.” In problems where data is abundant, end-to-end systems have been remarkably successful. But they are not always a good choice. The next few chapters will give more examples of end-to-end systems as well as give advice on when you should and should not use them. + +> Page 92 Machine Learning Yearning-Draft Andrew Ng + +# 48 More end-to-end learning examples + +Suppose you want to build a speech recognition system. You might build a system with three components: The components work as follows: + +1. Compute features: Extract hand-designed features, such as MFCC ( Mel-frequency cepstrum coefficients) features, which try to capture the content of an utterance while disregarding less relevant properties, such as the speaker’s pitch. + +2. Phoneme recognizer: Some linguists believe that there are basic units of sound called “phonemes.” For example, the initial “k” sound in “keep” is the same phoneme as the “c” sound in “cake.” This system tries to recognize the phonemes in the audio clip. + +3. Final recognizer: Take the sequence of recognized phonemes, and try to string them together into an output transcript. In contrast, an end-to-end system might input an audio clip, and try to directly output the transcript: So far, we have only described machine learning “pipelines” that are completely linear: the output is sequentially passed from one staged to the next. Pipelines can be more complex. For example, here is a simple architecture for an autonomous car: + +> Page 93 Machine Learning Yearning-Draft Andrew Ng + +It has three components: One detects other cars using the camera images; one detects pedestrians; then a final component plans a path for our own car that avoids the cars and pedestrians. Not every component in a pipeline has to be learned. For example, the literature on “robot motion planning” has numerous algorithms for the final path planning step for the car. Many of these algorithms do not involve learning. In contrast, and end-to-end approach might try to take in the sensor inputs and directly output the steering direction: Even though end-to-end learning has seen many successes, it is not always the best approach. For example, end-to-end speech recognition works well. But I’m skeptical about end-to-end learning for autonomous driving. The next few chapters explain why. + +> Page 94 Machine Learning Yearning-Draft Andrew Ng + +# 49 Pros and cons of end-to-end learning + +Consider the same speech pipeline from our earlier example: Many parts of this pipeline were “hand-engineered”: + +• MFCCs are a set of hand-designed audio features. Although they provide a reasonable summary of the audio input, they also simplify the input signal by throwing some information away. + +• Phonemes are an invention of linguists. They are an imperfect representation of speech sounds. To the extent that phonemes are a poor approximation of reality, forcing an algorithm to use a phoneme representation will limit the speech system’s performance. These hand-engineered components limit the potential performance of the speech system. However, allowing hand-engineered components also has some advantages: + +• The MFCC features are robust to some properties of speech that do not affect the content, such as speaker pitch. Thus, they help simplify the problem for the learning algorithm. + +• To the extent that phonemes are a reasonable representation of speech, they can also help the learning algorithm understand basic sound components and therefore improve its performance. Having more hand-engineered components generally allows a speech system to learn with less data. The hand-engineered knowledge captured by MFCCs and phonemes “supplements” the knowledge our algorithm acquires from data. When we don’t have much data, this knowledge is useful. Now, consider the end-to-end system: + +> Page 95 Machine Learning Yearning-Draft Andrew Ng + +This system lacks the hand-engineered knowledge. Thus, when the training set is small, it might do worse than the hand-engineered pipeline. However, when the training set is large, then it is not hampered by the limitations of an MFCC or phoneme-based representation. If the learning algorithm is a large-enough neural network and if it is trained with enough training data, it has the potential to do very well, and perhaps even approach the optimal error rate. End-to-end learning systems tend to do well when there is a lot of labeled data for “both ends”—the input end and the output end. In this example, we require a large dataset of (audio, transcript) pairs. When this type of data is not available, approach end-to-end learning with great caution. If you are working on a machine learning problem where the training set is very small, most of your algorithm’s knowledge will have to come from your human insight. I.e., from your “hand engineering” components. If you choose not to use an end-to-end system, you will have to decide what are the steps in your pipeline, and how they should plug together. In the next few chapters, we’ll give some suggestions for designing such pipelines. + +> Page 96 Machine Learning Yearning-Draft Andrew Ng + +# 50 Choosing pipeline components: Data availability + +When building a non-end-to-end pipeline system, what are good candidates for the components of the pipeline? How you design the pipeline will greatly impact the overall system’s performance. One important factor is whether you can easily collect data to train each of the components. For example, consider this autonomous driving architecture: You can use machine learning to detect cars and pedestrians. Further, it is not hard to obtain data for these: There are numerous computer vision datasets with large numbers of labeled cars and pedestrians. You can also use crowdsourcing (such as Amazon Mechanical Turk) to obtain even larger datasets. It is thus relatively easy to obtain training data to build a car detector and a pedestrian detector. In contrast, consider a pure end-to-end approach: To train this system, we would need a large dataset of (Image, Steering Direction) pairs. It is very time-consuming and expensive to have people drive cars around and record their steering direction to collect such data. You need a fleet of specially-instrumented cars, and a huge amount of driving to cover a wide range of possible scenarios. This makes an end-to-end system difficult to train. It is much easier to obtain a large dataset of labeled car or pedestrian images. More generally, if there is a lot of data available for training “intermediate modules” of a pipeline (such as a car detector or a pedestrian detector), then you might consider using a + +> Page 97 Machine Learning Yearning-Draft Andrew Ng + +pipeline with multiple stages. This structure could be superior because you could use all that available data to train the intermediate modules. Until more end-to-end data becomes available, I believe the non-end-to-end approach is significantly more promising for autonomous driving: Its architecture better matches the availability of data. + +> Page 98 Machine Learning Yearning-Draft Andrew Ng + +# 51 Choosing pipeline components: Task simplicity + +Other than data availability, you should also consider a second factor when picking components of a pipeline: How simple are the tasks solved by the individual components? You should try to choose pipeline components that are individually easy to build or learn. But what does it mean for a component to be “easy” to learn? + +Consider these machine learning tasks, listed in order of increasing difficulty: + +1. Classifying whether an image is overexposed (like the example above) 2. Classifying whether an image was taken indoor or outdoor 3. Classifying whether an image contains a cat 4. Classifying whether an image contains a cat with both black and white fur 5. Classifying whether an image contains a Siamese cat (a particular breed of cat) Each of these is a binary image classification task: You have to input an image, and output either 0 or 1. But the tasks earlier in the list seem much “easier” for a neural network to learn. You will be able to learn the easier tasks with fewer training examples. Machine learning does not yet have a good formal definition of what makes a task easy or hard. With the rise of deep learning and multi-layered neural networks, we sometimes say a 16 + +task is “easy” if it can be carried out with fewer computation steps (corresponding to a shallow neural network), and “hard” if it requires more computation steps (requiring a deeper neural network). But these are informal definitions. + +> 16 + +Information theory has the concept of “Kolmogorov Complexity”, which says that the complexity of a learned function is the length of the shortest computer program that can produce that function. However, this theoretical concept has found few practical applications in AI. See also: https://en.wikipedia.org/wiki/Kolmogorov_complexity + +> Page 99 Machine Learning Yearning-Draft Andrew Ng + +If you are able to take a complex task, and break it down into simpler sub-tasks, then by coding in the steps of the sub-tasks explicitly, you are giving the algorithm prior knowledge that can help it learn a task more efficiently. + +Suppose you are building a Siamese cat detector. This is the pure end-to-end architecture: + +In contrast, you can alternatively use a pipeline with two steps: + +The first step (cat detector) detects all the cats in the image. + +> Page 100 Machine Learning Yearning-Draft Andrew Ng + +The second step then passes cropped images of each of the detected cats (one at a time) to a cat species classifier, and finally outputs 1 if any of the cats detected is a Siamese cat. + +Compared to training a purely end-to-end classifier using just labels 0/1, each of the two components in the pipeline--the cat detector and the cat breed classifier--seem much easier to learn and will require significantly less data. 17 + +> 17 + +If you are familiar with practical object detection algorithms, you will recognize that they do not learn just with 0/1 image labels, but are instead trained with bounding boxes provided as part of the training data. A discussion of them is beyond the scope of this chapter. See the Deep Learning specialization on Coursera ( http://deeplearning.ai ) if you would like to learn more about such algorithms. + +> Page 101 Machine Learning Yearning-Draft Andrew Ng + +As one final example, let’s revisit the autonomous driving pipeline. + +By using this pipeline, you are telling the algorithm that there are 3 key steps to driving: (1) Detect other cars, (2) Detect pedestrians, and (3) Plan a path for your car. Further, each of these is a relatively simpler function--and can thus be learned with less data--than the purely end-to-end approach. In summary, when deciding what should be the components of a pipeline, try to build a pipeline where each component is a relatively “simple” function that can therefore be learned from only a modest amount of data. + +> Page 102 Machine Learning Yearning-Draft Andrew Ng + +# 52 Directly learning rich outputs + +An image classification algorithm will input an image x, and output an integer indicating the object category. Can an algorithm instead output an entire sentence describing the image? For example: + +x = y = “A yellow bus driving down a road with green trees and green grass in the background.” + +Traditional applications of supervised learning learned a function h:X→Y, where the output + +y was usually an integer or a real number. For example: + +Problem X YSpam classification Email Spam/Not spam (0/1) + +Image recognition Image Integer label + +Housing price prediction Features of house Price in dollars + +Product recommendation Product & user features Chance of purchase + +One of the most exciting developments in end-to-end deep learning is that it is letting us directly learn y that are much more complex than a number. In the image-captioning example above, you can have a neural network input an image ( x) and directly output a caption ( y). + +> Page 103 Machine Learning Yearning-Draft Andrew Ng + +Here are more examples: + +Problem X Y Example Citation Image captioning Image Text Mao et al., 2014 + +Machine translation English text French text Suskever et al., 2014 + +Question answering (Text,Question) pair Answer text Bordes et al., 2015 + +Speech recognition Audio Transcription Hannun et al., 2015 + +TTS Text features Audio van der Oord et al., 2016 + +This is an accelerating trend in deep learning: When you have the right (input,output) labeled pairs, you can sometimes learn end-to-end even when the output is a sentence, an image, audio, or other outputs that are richer than a single number. + +> Page 104 Machine Learning Yearning-Draft Andrew Ng + +# Error analysis by parts + +Page 105 Machine Learning Yearning-Draft Andrew Ng 53 Error analysis by parts + +Suppose your system is built using a complex machine learning pipeline, and you would like to improve the system’s performance. Which part of the pipeline should you work on improving? By attributing errors to specific parts of the pipeline, you can decide how to prioritize your work. Let’s use our Siamese cat classifier example: + +The first part, the cat detector, detects cats and crops them out of the image. The second part, the cat breed classifier, decides if it is a Siamese cat. It is possible to spend years working on improving either of these two pipeline components. How do you decide which component(s) to focus on? By carrying out error analysis by parts , you can try to attribute each mistake the algorithm makes to one (or sometimes both) of the two parts of the pipeline. For example, the algorithm misclassifies this image as not containing a Siamese cat (y=0) even though the correct label is y=1. + +Let’s manually examine what the two steps of the algorithm did. Suppose the Siamese cat detector had detected a cat as follows: + +> Page 106 Machine Learning Yearning-Draft Andrew Ng + +This means that the cat breed classifier is given the following image: + +The cat breed classifier then correctly classifies this image as not containing a Siamese cat. Thus, the cat breed classifier is blameless: It was given of a pile of rocks and outputted a very reasonable label y=0. Indeed, a human classifying the cropped image above would also have predicted y=0. Thus, you can clearly attribute this error to the cat detector. If, on the other hand, the cat detector had outputted the following bounding box: + +then you would conclude that the cat detector had done its job, and that it was the cat breed classifier that is at fault. Say you go through 100 misclassified dev set images and find that 90 of the errors are attributable to the cat detector, and only 10 errors are attributable to the cat breed classifier. You can safely conclude that you should focus more attention on improving the cat detector. + +> Page 107 Machine Learning Yearning-Draft Andrew Ng + +Further, you have now also conveniently found 90 examples where the cat detector outputted incorrect bounding boxes. You can use these 90 examples to carry out a deeper level of error analysis on the cat detector to see how to improve that. Our description of how you attribute error to one part of the pipeline has been informal so far: you look at the output of each of the parts and see if you can decide which one made a mistake. This informal method could be all you need. But in the next chapter, you’ll also see a more formal way of attributing error. + +> Page 108 Machine Learning Yearning-Draft Andrew Ng + +# 54 Attributing error to one part + +Let’s continue to use this example: + +Suppose the cat detector outputted this bounding box: + +The cat breed classifier is thus given this cropped image, whereupon it incorrectly outputs y=0, or that there is no cat in the picture. + +The cat detector did its job poorly. However, a highly skilled human could arguably still recognize the Siamese cat from the poorly cropped image. So do we attribute this error to the cat detector, or the cat breed classifier, or both? It is ambiguous. If the number of ambiguous cases like these is small, you can make whatever decision you want and get a similar result. But here is a more formal test that lets you more definitively attribute the error to exactly one part: 1. Replace the cat detector output with a hand-labeled bounding box. + +> Page 109 Machine Learning Yearning-Draft Andrew Ng + +2. Run the corresponding cropped image through the cat breed classifier. If the cat breed classifier still misclassifies it, attribute the error to the cat breed classifier. Otherwise, attribute the error to the cat detector. In other words, run an experiment in which you give the cat breed classifier a “perfect” input. There are two cases: + +● Case 1: Even given a “perfect” bounding box, the cat breed classifier still incorrectly outputs y=0. In this case, clearly the cat breed classifier is at fault. + +● Case 2: Given a “perfect” bounding box, the breed classifier now correctly outputs y=1. This shows that if only the cat detector had given a more perfect bounding box, then the overall system’s output would have been correct. Thus, attribute the error to the cat detector. By carrying out this analysis on the misclassified dev set images, you can now unambiguously attribute each error to one component. This allows you to estimate the fraction of errors due to each component of the pipeline, and therefore decide where to focus your attention. + +> Page 110 Machine Learning Yearning-Draft Andrew Ng + +# 55 General case of error attribution + +Here are the general steps for error attribution. Suppose the pipeline has three steps A, B and C, where A feeds directly into B, and B feeds directly into C. + +For each mistake the system makes on the dev set: 1. Try manually modifying A’s output to be a “perfect” output (e.g., the “perfect” bounding box for the cat), and run the rest of the pipeline B, C on this output. If the algorithm now gives a correct output, then this shows that, if only A had given a better output, the overall algorithm’s output would have been correct; thus, you can attribute this error to component A. Otherwise, go on to Step 2. 2. Try manually modifying B’s output to be the “perfect” output for B. If the algorithm now gives a correct output, then attribute the error to component B. Otherwise, go on to Step 3. 3. Attribute the error to component C. Let’s look at a more complex example: + +Your self-driving car uses this pipeline. How do you use error analysis by parts to decide which component(s) to focus on? You can map the three components to A, B, C as follows: A: Detect cars B: Detect pedestrians C: Plan path for car + +> Page 111 Machine Learning Yearning-Draft Andrew Ng + +Following the procedure described above, suppose you test out your car on a closed track and find a case where the car chooses a more jarring steering direction than a skilled driver would. In the self-driving world, such a case is usually called a scenario . You would then: 1. Try manually modifying A (detecting cars)’s output to be a “perfect” output (e.g., manually go in and tell it where the other cars are). Run the rest of the pipeline B, C as before, but allow C (plan path) to use A’s now perfect output. If the algorithm now plans a much better path for the car, then this shows that, if only A had given a better output, the overall algorithm’s output would have been better; Thus, you can attribute this error to component A. Otherwise, go on to Step 2. 2. Try manually modifying B (detect pedestrian)’s output to be the “perfect” output for B. If the algorithm now gives a correct output, then attribute the error to component B. Otherwise, go on to Step 3. 3. Attribute the error to component C. The components of an ML pipeline should be ordered according to a Directed Acyclic Graph (DAG), meaning that you should be able to compute them in some fixed left-to-right order, and later components should depend only on earlier components’ outputs. So long as the mapping of the components to the A->B->C order follows the DAG ordering, then the error analysis will be fine. You might get slightly different results if you swap A and B: A: Detect pedestrians (was previously Detect cars )B: Detect cars (was previously Detect pedestrians )C: Plan path for car But the results of this analysis would still be valid and give good guidance for where to focus your attention. + +> Page 112 Machine Learning Yearning-Draft Andrew Ng + +# 56 Error analysis by parts and comparison to human-level performance + +Carrying out error analysis on a learning algorithm is like using data science to analyze an ML system’s mistakes in order to derive insights about what to do next. At its most basic, error analysis by parts tells us what component(s) performance is (are) worth the greatest effort to improve. Say you have a dataset about customers buying things on a website. A data scientist may have many different ways of analyzing the data. She may draw many different conclusions about whether the website should raise prices, about the lifetime value of customers acquired through different marketing campaigns, and so on. There is no one “right” way to analyze a dataset, and there are many possible useful insights one could draw. Similarly, there is no one “right” way to carry out error analysis. Through these chapters you have learned many of the most common design patterns for drawing useful insights about your ML system, but you should feel free to experiment with other ways of analyzing errors as well. Let’s return to the self-driving application, where a car detection algorithm outputs the location (and perhaps velocity) of the nearby cars, a pedestrian detection algorithm outputs the location of the nearby pedestrians, and these two outputs are finally used to plan a path for the car. To debug this pipeline, rather than rigorously following the procedure you saw in the previous chapter, you could more informally ask: 1. How far is the Detect cars component from human-level performance at detecting cars? 2. How far is the Detect pedestrians component from human-level performance? + +> Page 113 Machine Learning Yearning-Draft Andrew Ng + +3. How far is the overall system’s performance from human-level performance? Here, human-level performance assumes the human has to plan a path for the car given only the outputs from the previous two pipeline components (rather than access to the camera images). In other words, how does the Plan path component’s performance compare to that of a human’s, when the human is given only the same input? If you find that one of the components is far from human-level performance, you now have a good case to focus on improving the performance of that component. Many error analysis processes work best when we are trying to automate something humans can do and can thus benchmark against human-level performance. Most of our preceding examples had this implicit assumption. If you are building an ML system where the final output or some of the intermediate components are doing things that even humans cannot do well, then some of these procedures will not apply. This is another advantage of working on problems that humans can solve--you have more powerful error analysis tools, and thus you can prioritize your team’s work more efficiently. + +> Page 114 Machine Learning Yearning-Draft Andrew Ng + +# 57 Spotting a flawed ML pipeline + +What if each individual component of your ML pipeline is performing at human-level performance or near-human-level performance, but the overall pipeline falls far short of human-level? This usually means that the pipeline is flawed and needs to be redesigned. Error analysis can also help you understand if you need to redesign your pipeline. + +In the previous chapter, we posed the question of whether each of the three components’ performance is at human level. Suppose the answer to all three questions is yes. That is: 1. The Detect cars component is at (roughly) human-level performance for detecting cars from the camera images. 2. The Detect pedestrians component is at (roughly) human-level performance for detecting cars from the camera images. 3. Compared to a human that has to plan a path for the car given only the outputs from the previous two pipeline components (rather than access to the camera images), the Plan path component’s performance is at a similar level. However, your overall self-driving car is performing significantly below human-level performance. I.e., humans given access to the camera images can plan significantly better paths for the car. What conclusion can you draw? The only possible conclusion is that the ML pipeline is flawed. In this case, the Plan path component is doing as well as it can given its inputs , but the inputs do not contain enough information. You should ask yourself what other information, other than the outputs from the two earlier pipeline components, is needed to plan paths very well for a car to drive. In other words, what other information does a skilled human driver need? + +> Page 115 Machine Learning Yearning-Draft Andrew Ng + +For example, suppose you realize that a human driver also needs to know the location of the lane markings. This suggests that you should redesign the pipeline as follows :18 + +Ultimately, if you don’t think your pipeline as a whole will achieve human-level performance, even if every individual component has human-level performance (remember that you are comparing to a human who is given the same input as the component), then the pipeline is flawed and should be redesigned. + +> 18 + +In the self-driving example above, in theory one could solve this problem by also feeding the raw camera image into the planning component. However, this would violate the design principle of “Task simplicity” described in Chapter 51, because the path planning module now needs to input a raw image and has a very complex task to solve. That’s why adding a Detect lane markings component is a better choice--it helps get the important and previously missing information about lane markings to the path planning module, but you avoid making any particular module overly complex to build/train. + +> Page 116 Machine Learning Yearning-Draft Andrew Ng + +# Conclusion + +Page 117 Machine Learning Yearning-Draft Andrew Ng 58 Building a superhero team - Get your teammates to read this + +Congratulations on finishing this book! In Chapter 2, we talked about how this book can help you become the superhero of your team. + +The only thing better than being a superhero is being part of a superhero team. I hope you’ll give copies of this book to your friends and teammates and help create other superheroes! + +Page 118 Machine Learning Yearning-Draft Andrew Ng diff --git a/docs/evidence/reports/qwen3_technical_report.md b/docs/evidence/reports/qwen3_technical_report.md index 54dabd1..06f7888 100644 --- a/docs/evidence/reports/qwen3_technical_report.md +++ b/docs/evidence/reports/qwen3_technical_report.md @@ -1,29 +1,32 @@ +Source: https://arxiv.org/html/2505.09388 (arXiv:2505.09388, Qwen Team, Alibaba) +Title: "Qwen3 Technical Report" +Fetched-via: curl https://r.jina.ai/https://arxiv.org/html/2505.09388, 2026-08-15 (CLAUDE agent) +Fetch-status: verbatim, full report through the conclusion and reference list. The previous cache stopped mid-section 4.3 (CLAUDE agent) + Title: Qwen3 Technical Report URL Source: https://arxiv.org/html/2505.09388 +Published Time: Tue, 11 Aug 2026 23:50:41 GMT + Markdown Content: -\useunder - -\ul - ###### Abstract In this work, we present Qwen3, the latest version of the Qwen model family. Qwen3 comprises a series of large language models (LLMs) designed to advance performance, efficiency, and multilingual capabilities. The Qwen3 series includes models of both dense and Mixture-of-Expert (MoE) architectures, with parameter scales ranging from 0.6 to 235 billion. A key innovation in Qwen3 is the integration of thinking mode (for complex, multi-step reasoning) and non-thinking mode (for rapid, context-driven responses) into a unified framework. This eliminates the need to switch between different models—–such as chat-optimized models (e.g., GPT-4o) and dedicated reasoning models (e.g., QwQ-32B)—–and enables dynamic mode switching based on user queries or chat templates. Meanwhile, Qwen3 introduces a thinking budget mechanism, allowing users to allocate computational resources adaptively during inference, thereby balancing latency and performance based on task complexity. Moreover, by leveraging the knowledge from the flagship models, we significantly reduce the computational resources required to build smaller-scale models, while ensuring their highly competitive performance. Empirical evaluations demonstrate that Qwen3 achieves state-of-the-art results across diverse benchmarks, including tasks in code generation, mathematical reasoning, agent tasks, etc., competitive against larger MoE models and proprietary models. Compared to its predecessor Qwen2.5, Qwen3 expands multilingual support from 29 to 119 languages and dialects, enhancing global accessibility through improved cross-lingual understanding and generation capabilities. To facilitate reproducibility and community-driven research and development, all Qwen3 models are publicly accessible under Apache 2.0. ## 1 Introduction -The pursuit of artificial general intelligence (AGI) or artificial super intelligence (ASI) has long been a goal for humanity. Recent advancements in large foundation models, e.g., GPT-4o (gpt4o), Claude 3.7 (claude3.7), Gemini 2.5 (gemini2.5), DeepSeek-V3 (deepseekv3), Llama-4 (llama4), and Qwen2.5 (qwen2.5), have demonstrated significant progress toward this objective. These models are trained on vast datasets spanning trillions of tokens across diverse domains and tasks, effectively distilling human knowledge and capabilities into their parameters. Furthermore, recent developments in reasoning models, optimized through reinforcement learning, highlight the potential for foundation models to enhance inference-time scaling and achieve higher levels of intelligence, e.g., o3 (o3), DeepSeek-R1 (r1). While most state-of-the-art models remain proprietary, the rapid growth of open-source communities has substantially reduced the performance gap between open-weight and closed-source models. Notably, an increasing number of top-tier models (llama4; deepseekv3; r1; qwen2.5) are now being released as open-source, fostering broader research and innovation in artificial intelligence. +The pursuit of artificial general intelligence (AGI) or artificial super intelligence (ASI) has long been a goal for humanity. Recent advancements in large foundation models, e.g., GPT-4o ([OpenAI 2024](https://arxiv.org/html/2505.09388#bib.bib41)), Claude 3.7 ([Anthropic 2025](https://arxiv.org/html/2505.09388#bib.bib5)), Gemini 2.5 ([DeepMind 2025](https://arxiv.org/html/2505.09388#bib.bib16)), DeepSeek-V3 ([Liu et al. 2024a](https://arxiv.org/html/2505.09388#bib.bib36)), Llama-4 ([Meta-AI 2025](https://arxiv.org/html/2505.09388#bib.bib40)), and Qwen2.5 ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), have demonstrated significant progress toward this objective. These models are trained on vast datasets spanning trillions of tokens across diverse domains and tasks, effectively distilling human knowledge and capabilities into their parameters. Furthermore, recent developments in reasoning models, optimized through reinforcement learning, highlight the potential for foundation models to enhance inference-time scaling and achieve higher levels of intelligence, e.g., o3 ([OpenAI 2025](https://arxiv.org/html/2505.09388#bib.bib44)), DeepSeek-R1 ([Guo et al. 2025](https://arxiv.org/html/2505.09388#bib.bib23)). While most state-of-the-art models remain proprietary, the rapid growth of open-source communities has substantially reduced the performance gap between open-weight and closed-source models. Notably, an increasing number of top-tier models ([Meta-AI 2025](https://arxiv.org/html/2505.09388#bib.bib40); [Liu et al. 2024a](https://arxiv.org/html/2505.09388#bib.bib36); [Guo et al. 2025](https://arxiv.org/html/2505.09388#bib.bib23); [Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)) are now being released as open-source, fostering broader research and innovation in artificial intelligence. In this work, we introduce Qwen3, the latest series in our foundation model family, Qwen. Qwen3 is a collection of open-weight large language models (LLMs) that achieve state-of-the-art performance across a wide variety of tasks and domains. We release both dense and Mixture-of-Experts (MoE) models, with the number of parameters ranging from 0.6 billion to 235 billion, to meet the needs of different downstream applications. Notably, the flagship model, Qwen3-235B-A22B, is an MoE model with a total of 235 billion parameters and 22 billion activated ones per token. This design ensures both high performance and efficient inference. -Qwen3 introduces several key advancements to enhance its functionality and usability. First, it integrates two distinct operating modes, thinking mode and non-thinking mode, into a single model. This allows users to switch between these modes without alternating between different models, e.g., switching from Qwen2.5 to QwQ (qwq). This flexibility ensures that developers and users can adapt the model's behavior to suit specific tasks efficiently. Additionally, Qwen3 incorporates thinking budgets, providing users with fine-grained control over the level of reasoning effort applied by the model during task execution. This capability is crucial to the optimization of computational resources and performance, tailoring the model's thinking behavior to meet varying complexity in real-world applications. Furthermore, Qwen3 has been pre-trained on 36 trillion tokens covering up to 119 languages and dialects, effectively enhancing its multilingual capabilities. This broadened language support amplifies its potential for deployment in global use cases and international applications. These advancements together establish Qwen3 as a cutting-edge open-source large language model family, capable of effectively addressing complex tasks across various domains and languages. +Qwen3 introduces several key advancements to enhance its functionality and usability. First, it integrates two distinct operating modes, thinking mode and non-thinking mode, into a single model. This allows users to switch between these modes without alternating between different models, e.g., switching from Qwen2.5 to QwQ ([Qwen Team 2024](https://arxiv.org/html/2505.09388#bib.bib49)). This flexibility ensures that developers and users can adapt the model’s behavior to suit specific tasks efficiently. Additionally, Qwen3 incorporates thinking budgets, providing users with fine-grained control over the level of reasoning effort applied by the model during task execution. This capability is crucial to the optimization of computational resources and performance, tailoring the model’s thinking behavior to meet varying complexity in real-world applications. Furthermore, Qwen3 has been pre-trained on 36 trillion tokens covering up to 119 languages and dialects, effectively enhancing its multilingual capabilities. This broadened language support amplifies its potential for deployment in global use cases and international applications. These advancements together establish Qwen3 as a cutting-edge open-source large language model family, capable of effectively addressing complex tasks across various domains and languages. -The pre-training process for Qwen3 utilizes a large-scale dataset consisting of approximately 36 trillion tokens, curated to ensure linguistic and domain diversity. To efficiently expand the training data, we employ a multi-modal approach: Qwen2.5-VL (qwen2.5vl) is finetuned to extract text from extensive PDF documents. We also generate synthetic data using domain-specific models: Qwen2.5-Math (qwen2.5math) for mathematical content and Qwen2.5-Coder (qwen2.5coder) for code-related data. The pre-training process follows a three-stage strategy. In the first stage, the model is trained on about 30 trillion tokens to build a strong foundation of general knowledge. In the second stage, it is further trained on knowledge-intensive data to enhance reasoning abilities in areas like science, technology, engineering, and mathematics (STEM) and coding. Finally, in the third stage, the model is trained on long-context data to increase its maximum context length from 4,096 to 32,768 tokens. +The pre-training process for Qwen3 utilizes a large-scale dataset consisting of approximately 36 trillion tokens, curated to ensure linguistic and domain diversity. To efficiently expand the training data, we employ a multi-modal approach: Qwen2.5-VL ([Bai et al. 2025](https://arxiv.org/html/2505.09388#bib.bib8)) is finetuned to extract text from extensive PDF documents. We also generate synthetic data using domain-specific models: Qwen2.5-Math ([Yang et al. 2024c](https://arxiv.org/html/2505.09388#bib.bib71)) for mathematical content and Qwen2.5-Coder ([Hui et al. 2024](https://arxiv.org/html/2505.09388#bib.bib29)) for code-related data. The pre-training process follows a three-stage strategy. In the first stage, the model is trained on about 30 trillion tokens to build a strong foundation of general knowledge. In the second stage, it is further trained on knowledge-intensive data to enhance reasoning abilities in areas like science, technology, engineering, and mathematics (STEM) and coding. Finally, in the third stage, the model is trained on long-context data to increase its maximum context length from 4,096 to 32,768 tokens. To better align foundation models with human preferences and downstream applications, we employ a multi-stage post-training approach that empowers both thinking (reasoning) and non-thinking modes. In the first two stages, we focus on developing strong reasoning abilities through long chain-of-thought (CoT) cold-start finetuning and reinforcement learning focusing on mathematics and coding tasks. In the final two stages, we combine data with and without reasoning paths into a unified dataset for further fine-tuning, enabling the model to handle both types of input effectively, and we then apply general-domain reinforcement learning to improve performance across a wide range of downstream tasks. For smaller models, we use strong-to-weak distillation, leveraging both off-policy and on-policy knowledge transfer from larger models to enhance their capabilities. Distillation from advanced teacher models significantly outperforms reinforcement learning in performance and training efficiency. -We evaluate both pre-trained and post-trained versions of our models across a comprehensive set of benchmarks spanning multiple tasks and domains. Experimental results show that our base pre-trained models achieve state-of-the-art performance. The post-trained models, whether in thinking or non-thinking mode, perform competitively against leading proprietary models and large mixture-of-experts (MoE) models such as o1, o3-mini, and DeepSeek-V3. Notably, our models excel in coding, mathematics, and agent-related tasks. For example, the flagship model Qwen3-235B-A22B achieves 85.7 on AIME'24 and 81.5 on AIME'25 (aime), 70.7 on LiveCodeBench v5 (livecodebench), 2,056 on CodeForces, and 70.8 on BFCL v3 (bfcl). In addition, other models in the Qwen3 series also show strong performance relative to their size. Furthermore, we observe that increasing the thinking budget for thinking tokens leads to a consistent improvement in the model's performance across various tasks. +We evaluate both pre-trained and post-trained versions of our models across a comprehensive set of benchmarks spanning multiple tasks and domains. Experimental results show that our base pre-trained models achieve state-of-the-art performance. The post-trained models, whether in thinking or non-thinking mode, perform competitively against leading proprietary models and large mixture-of-experts (MoE) models such as o1, o3-mini, and DeepSeek-V3. Notably, our models excel in coding, mathematics, and agent-related tasks. For example, the flagship model Qwen3-235B-A22B achieves 85.7 on AIME’24 and 81.5 on AIME’25 ([AIME 2025](https://arxiv.org/html/2505.09388#bib.bib2)), 70.7 on LiveCodeBench v5 ([Jain et al. 2024](https://arxiv.org/html/2505.09388#bib.bib30)), 2,056 on CodeForces, and 70.8 on BFCL v3 ([Yan et al. 2024](https://arxiv.org/html/2505.09388#bib.bib68)). In addition, other models in the Qwen3 series also show strong performance relative to their size. Furthermore, we observe that increasing the thinking budget for thinking tokens leads to a consistent improvement in the model’s performance across various tasks. In the following sections, we describe the design of the model architecture, provide details on its training procedures, present the experimental results of pre-trained and post-trained models, and finally, conclude this technical report by summarizing the key findings and outlining potential directions for future research. @@ -31,11 +34,11 @@ In the following sections, we describe the design of the model architecture, pro The Qwen3 series includes 6 dense models, namely Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen3-32B, and 2 MoE models, Qwen3-30B-A3B and Qwen3-235B-A22B. The flagship model, Qwen3-235B-A22B, has a total of 235B parameters with 22B activated ones. Below, we elaborate on the architecture of the Qwen3 models. -The architecture of the Qwen3 dense models is similar to Qwen2.5 (qwen2.5), including using Grouped Query Attention (GQA, gqa), SwiGLU (glu), Rotary Positional Embeddings (RoPE, rope), and RMSNorm (rmsnorm) with pre-normalization. Besides, we remove QKV-bias used in Qwen2 (qwen2) and introduce QK-Norm (pmlr-v202-dehghani23a) to the attention mechanism to ensure stable training for Qwen3. Key information on model architecture is provided in Table [1](https://arxiv.org/html/2505.09388#S2.T1 "Table 1 ‣ 2 Architecture ‣ Qwen3 Technical Report"). +The architecture of the Qwen3 dense models is similar to Qwen2.5 ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), including using Grouped Query Attention (GQA, [Ainslie et al. 2023](https://arxiv.org/html/2505.09388#bib.bib3)), SwiGLU ([Dauphin et al. 2017](https://arxiv.org/html/2505.09388#bib.bib15)), Rotary Positional Embeddings (RoPE, [Su et al. 2024](https://arxiv.org/html/2505.09388#bib.bib57)), and RMSNorm ([Jiang et al. 2023](https://arxiv.org/html/2505.09388#bib.bib31)) with pre-normalization. Besides, we remove QKV-bias used in Qwen2 ([Yang et al. 2024a](https://arxiv.org/html/2505.09388#bib.bib69)) and introduce QK-Norm ([Dehghani et al. 2023](https://arxiv.org/html/2505.09388#bib.bib17)) to the attention mechanism to ensure stable training for Qwen3. Key information on model architecture is provided in Table [1](https://arxiv.org/html/2505.09388#S2.T1 "Table 1 ‣ 2 Architecture ‣ Qwen3 Technical Report"). -The Qwen3 MoE models share the same fundamental architecture as the Qwen3 dense models. Key information on model architecture is provided in Table [2](https://arxiv.org/html/2505.09388#S2.T2 "Table 2 ‣ 2 Architecture ‣ Qwen3 Technical Report"). We follow Qwen2.5-MoE (qwen2.5) and implement fine-grained expert segmentation (deepseekmoe). The Qwen3 MoE models have 128 total experts with 8 activated experts per token. Unlike Qwen2.5-MoE, the Qwen3-MoE design excludes shared experts. Furthermore, we adopt the global-batch load balancing loss (global_balance) to encourage expert specialization. These architectural and training innovations have yielded substantial improvements in model performance across downstream tasks. +The Qwen3 MoE models share the same fundamental architecture as the Qwen3 dense models. Key information on model architecture is provided in Table [2](https://arxiv.org/html/2505.09388#S2.T2 "Table 2 ‣ 2 Architecture ‣ Qwen3 Technical Report"). We follow Qwen2.5-MoE ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)) and implement fine-grained expert segmentation ([Dai et al. 2024](https://arxiv.org/html/2505.09388#bib.bib14)). The Qwen3 MoE models have 128 total experts with 8 activated experts per token. Unlike Qwen2.5-MoE, the Qwen3-MoE design excludes shared experts. Furthermore, we adopt the global-batch load balancing loss ([Qiu et al. 2025](https://arxiv.org/html/2505.09388#bib.bib47)) to encourage expert specialization. These architectural and training innovations have yielded substantial improvements in model performance across downstream tasks. -Qwen3 models utilize Qwen's tokenizer (qwen), which implements byte-level byte-pair encoding (BBPE, gpt3; wang2020neural; sennirch2016neural) with a vocabulary size of 151,669. +Qwen3 models utilize Qwen’s tokenizer ([Bai et al. 2023](https://arxiv.org/html/2505.09388#bib.bib7)), which implements byte-level byte-pair encoding (BBPE, [Brown et al. 2020](https://arxiv.org/html/2505.09388#bib.bib10); [Wang et al. 2020](https://arxiv.org/html/2505.09388#bib.bib60); [Sennrich et al. 2016](https://arxiv.org/html/2505.09388#bib.bib53)) with a vocabulary size of 151,669. Table 1: Model architecture of Qwen3 dense models. @@ -59,11 +62,11 @@ In this section, we describe the construction of our pretraining data, the detai ### 3.1 Pre-training Data -Compared with Qwen2.5 (qwen2.5), we have significantly expanded the scale and diversity of our training data. Specifically, we collected twice as many pre-training tokens—covering three times more languages. All Qwen3 models are trained on a large and diverse dataset consisting of 119 languages and dialects, with a total of 36 trillion tokens. This dataset includes high-quality content in various domains such as coding, STEM (Science, Technology, Engineering, and Mathematics), reasoning tasks, books, multilingual texts, and synthetic data. +Compared with Qwen2.5 ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), we have significantly expanded the scale and diversity of our training data. Specifically, we collected twice as many pre-training tokens—covering three times more languages. All Qwen3 models are trained on a large and diverse dataset consisting of 119 languages and dialects, with a total of 36 trillion tokens. This dataset includes high-quality content in various domains such as coding, STEM (Science, Technology, Engineering, and Mathematics), reasoning tasks, books, multilingual texts, and synthetic data. -To further expand the pre-training data corpus, we first employ the Qwen2.5-VL model (qwen2.5vl) to perform text recognition on a large volume of PDF-like documents. The recognized text is then refined using the Qwen2.5 model (qwen2.5), which helps improve its quality. Through this two-step process, we are able to obtain an additional set of high-quality text tokens, amounting to trillions in total. Besides, we employ Qwen2.5 (qwen2.5), Qwen2.5-Math (qwen2.5math), and Qwen2.5-Coder (qwen2.5coder) models to synthesize trillions of text tokens in different formats, including textbooks, question-answering, instructions, and code snippets, covering dozens of domains. Finally, we further expand the pre-training corpus by incorporating additional multilingual data and introducing more languages. Compared to the pre-training data used in Qwen2.5, the number of supported languages has been significantly increased from 29 to 119, enhancing the model's linguistic coverage and cross-lingual capabilities. +To further expand the pre-training data corpus, we first employ the Qwen2.5-VL model ([Bai et al. 2025](https://arxiv.org/html/2505.09388#bib.bib8)) to perform text recognition on a large volume of PDF-like documents. The recognized text is then refined using the Qwen2.5 model ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), which helps improve its quality. Through this two-step process, we are able to obtain an additional set of high-quality text tokens, amounting to trillions in total. Besides, we employ Qwen2.5 ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), Qwen2.5-Math ([Yang et al. 2024c](https://arxiv.org/html/2505.09388#bib.bib71)), and Qwen2.5-Coder ([Hui et al. 2024](https://arxiv.org/html/2505.09388#bib.bib29)) models to synthesize trillions of text tokens in different formats, including textbooks, question-answering, instructions, and code snippets, covering dozens of domains. Finally, we further expand the pre-training corpus by incorporating additional multilingual data and introducing more languages. Compared to the pre-training data used in Qwen2.5, the number of supported languages has been significantly increased from 29 to 119, enhancing the model’s linguistic coverage and cross-lingual capabilities. -We have developed a multilingual data annotation system designed to enhance both the quality and diversity of training data. This system has been applied to our large-scale pre-training datasets, annotating over 30 trillion tokens across multiple dimensions such as educational value, fields, domains, and safety. These detailed annotations support more effective data filtering and combination. Unlike previous studies (doremi; doge; regmix) that optimize the data mixture at the data source or domain level, our method optimizes the data mixture at the instance-level through extensive ablation experiments on small proxy models with the fine-grained data labels. +We have developed a multilingual data annotation system designed to enhance both the quality and diversity of training data. This system has been applied to our large-scale pre-training datasets, annotating over 30 trillion tokens across multiple dimensions such as educational value, fields, domains, and safety. These detailed annotations support more effective data filtering and combination. Unlike previous studies ([Xie et al. 2023](https://arxiv.org/html/2505.09388#bib.bib66); [Fan et al. 2023](https://arxiv.org/html/2505.09388#bib.bib20); [Liu et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib38)) that optimize the data mixture at the data source or domain level, our method optimizes the data mixture at the instance-level through extensive ablation experiments on small proxy models with the fine-grained data labels. ### 3.2 Pre-training Stage @@ -76,29 +79,29 @@ General Stage (S1): At the first pre-training stage, all Qwen3 models are traine Reasoning Stage (S2): To further improve the reasoning ability, we optimize the pre-training corpus of this stage by increasing the proportion of STEM, coding, reasoning, and synthetic data. The models are further pre-trained with about 5T higher-quality tokens at a sequence length of 4,096 tokens. We also accelerate the learning rate decay during this stage. 3. (3) -Long Context Stage: In the final pre-training stage, we collect high-quality long context corpora to extend the context length of Qwen3 models. All models are pre-trained on hundreds of billions of tokens with a sequence length of 32,768 tokens. The long context corpus includes 75% of text between 16,384 to 32,768 tokens in length, and 25% of text between 4,096 to 16,384 in length. Following Qwen2.5 (qwen2.5), we increase the base frequency of RoPE from 10,000 to 1,000,000 using the ABF technique (ropeabf). Meanwhile, we introduce YARN (yarn) and Dual Chunk Attention (DCA, chunkllama) to achieve a four-fold increase in sequence length capacity during inference. +Long Context Stage: In the final pre-training stage, we collect high-quality long context corpora to extend the context length of Qwen3 models. All models are pre-trained on hundreds of billions of tokens with a sequence length of 32,768 tokens. The long context corpus includes 75% of text between 16,384 to 32,768 tokens in length, and 25% of text between 4,096 to 16,384 in length. Following Qwen2.5 ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), we increase the base frequency of RoPE from 10,000 to 1,000,000 using the ABF technique ([Xiong et al. 2023](https://arxiv.org/html/2505.09388#bib.bib67)). Meanwhile, we introduce YARN ([Peng et al. 2023](https://arxiv.org/html/2505.09388#bib.bib46)) and Dual Chunk Attention (DCA, [An et al. 2024](https://arxiv.org/html/2505.09388#bib.bib4)) to achieve a four-fold increase in sequence length capacity during inference. -Similar to Qwen2.5 (qwen2.5), we develop scaling laws for optimal hyper-parameters (e.g., learning rate scheduler, and batch size) predictions based on three pre-training stages mentioned above. Through extensive experiments, we systematically study the relationship between model architecture, training data, training stage, and optimal training hyper-parameters. Finally, we set the predicted optimal learning rate and batch size strategy for each dense or MoE model. +Similar to Qwen2.5 ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), we develop scaling laws for optimal hyper-parameters (e.g., learning rate scheduler, and batch size) predictions based on three pre-training stages mentioned above. Through extensive experiments, we systematically study the relationship between model architecture, training data, training stage, and optimal training hyper-parameters. Finally, we set the predicted optimal learning rate and batch size strategy for each dense or MoE model. ### 3.3 Pre-training Evaluation We conduct comprehensive evaluations of the base language models of the Qwen3 series. The evaluation of base models mainly focuses on their performance in general knowledge, reasoning, mathematics, scientific knowledge, coding, and multilingual capabilities. The evaluation datasets for pre-trained base models include 15 benchmarks: * • -General Tasks: MMLU (mmlu) (5-shot), MMLU-Pro (mmlupro) (5-shot, CoT), MMLU-redux (mmluredux) (5-shot), BBH (bbh) (3-shot, CoT), SuperGPQA (supergpqa)(5-shot, CoT). +General Tasks: MMLU ([Hendrycks et al. 2021a](https://arxiv.org/html/2505.09388#bib.bib25)) (5-shot), MMLU-Pro ([Wang et al. 2024](https://arxiv.org/html/2505.09388#bib.bib62)) (5-shot, CoT), MMLU-redux ([Gema et al. 2024](https://arxiv.org/html/2505.09388#bib.bib21)) (5-shot), BBH ([Suzgun et al. 2023](https://arxiv.org/html/2505.09388#bib.bib58)) (3-shot, CoT), SuperGPQA ([Du et al. 2025](https://arxiv.org/html/2505.09388#bib.bib18))(5-shot, CoT). * • -Math & STEM Tasks: GPQA (gpqa) (5-shot, CoT), GSM8K (gsm8k) (4-shot, CoT), MATH (math) (4-shot, CoT). +Math & STEM Tasks: GPQA ([Rein et al. 2023](https://arxiv.org/html/2505.09388#bib.bib51)) (5-shot, CoT), GSM8K ([Cobbe et al. 2021](https://arxiv.org/html/2505.09388#bib.bib13)) (4-shot, CoT), MATH ([Hendrycks et al. 2021b](https://arxiv.org/html/2505.09388#bib.bib26)) (4-shot, CoT). * • -Coding Tasks: EvalPlus (evalplus) (0-shot) (Average of HumanEval (humaneval), MBPP (mbpp), Humaneval+, MBPP+) (evalplus), MultiPL-E (multiple) (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript), MBPP-3shot (mbpp), CRUX-O of CRUXEval (1-shot) (gu2024cruxeval). +Coding Tasks: EvalPlus ([Liu et al. 2023a](https://arxiv.org/html/2505.09388#bib.bib37)) (0-shot) (Average of HumanEval ([Chen et al. 2021](https://arxiv.org/html/2505.09388#bib.bib12)), MBPP ([Austin et al. 2021](https://arxiv.org/html/2505.09388#bib.bib6)), Humaneval+, MBPP+) ([Liu et al. 2023a](https://arxiv.org/html/2505.09388#bib.bib37)), MultiPL-E ([Cassano et al. 2023](https://arxiv.org/html/2505.09388#bib.bib11)) (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript), MBPP-3shot ([Austin et al. 2021](https://arxiv.org/html/2505.09388#bib.bib6)), CRUX-O of CRUXEval (1-shot) ([Gu et al. 2024](https://arxiv.org/html/2505.09388#bib.bib22)). * • -Multilingual Tasks: MGSM (mgsm) (8-shot, CoT), MMMLU (mmmlu) (5-shot), INCLUDE (romanou2024includeevaluatingmultilinguallanguage) (5-shot). +Multilingual Tasks: MGSM ([Shi et al. 2023](https://arxiv.org/html/2505.09388#bib.bib55)) (8-shot, CoT), MMMLU ([OpenAI 2024](https://arxiv.org/html/2505.09388#bib.bib42)) (5-shot), INCLUDE ([Romanou et al. 2024](https://arxiv.org/html/2505.09388#bib.bib52)) (5-shot). -For the base model baselines, we compare the Qwen3 series base models with the Qwen2.5 base models (qwen2.5) and other leading open-source base models, including DeepSeek-V3 Base (deepseekv3), Gemma-3 (gemma3), Llama-3 (llama3), and Llama-4 (llama4) series base models, in terms of scale of parameters. All models are evaluated using the same evaluation pipeline and the widely-used evaluation settings to ensure fair comparison. +For the base model baselines, we compare the Qwen3 series base models with the Qwen2.5 base models ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)) and other leading open-source base models, including DeepSeek-V3 Base ([Liu et al. 2024a](https://arxiv.org/html/2505.09388#bib.bib36)), Gemma-3 ([Team et al. 2025](https://arxiv.org/html/2505.09388#bib.bib59)), Llama-3 ([Dubey et al. 2024](https://arxiv.org/html/2505.09388#bib.bib19)), and Llama-4 ([Meta-AI 2025](https://arxiv.org/html/2505.09388#bib.bib40)) series base models, in terms of scale of parameters. All models are evaluated using the same evaluation pipeline and the widely-used evaluation settings to ensure fair comparison. -#### Summary of Evaluation Results +##### Summary of Evaluation Results Based on the overall evaluation results, we highlight some key conclusions of Qwen3 base models. @@ -140,9 +143,9 @@ MGSM 82.40 82.21 79.69 82.68 83.53 MMMLU 84.40 83.49 83.09 85.88 86.70 INCLUDE 69.05 66.97 73.47 75.17 73.46 -#### Qwen3-235B-A22B-Base +##### Qwen3-235B-A22B-Base -We compare Qwen3-235B-A22B-Base to our previous similar-sized MoE Qwen2.5-Plus-Base (qwen2.5) and other leading open-source base models: Llama-4-Maverick (llama4), Qwen2.5-72B-Base (qwen2.5), DeepSeek-V3 Base (deepseekv3). From the results in Table [3](https://arxiv.org/html/2505.09388#S3.T3 "Table 3 ‣ Summary of Evaluation Results ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), the Qwen3-235B-A22B-Base model attains the highest performance scores across most of the evaluated benchmarks. We further compare Qwen3-235B-A22B-Base with other baselines separately for the detailed analysis. +We compare Qwen3-235B-A22B-Base to our previous similar-sized MoE Qwen2.5-Plus-Base ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)) and other leading open-source base models: Llama-4-Maverick ([Meta-AI 2025](https://arxiv.org/html/2505.09388#bib.bib40)), Qwen2.5-72B-Base ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), DeepSeek-V3 Base ([Liu et al. 2024a](https://arxiv.org/html/2505.09388#bib.bib36)). From the results in Table [3](https://arxiv.org/html/2505.09388#S3.T3 "Table 3 ‣ Summary of Evaluation Results ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), the Qwen3-235B-A22B-Base model attains the highest performance scores across most of the evaluated benchmarks. We further compare Qwen3-235B-A22B-Base with other baselines separately for the detailed analysis. 1. (1) Compared with the recently open-source model Llama-4-Maverick-Base, which has about twice the number of parameters, Qwen3-235B-A22B-Base still performs better on most benchmarks. @@ -268,9 +271,9 @@ MGSM 12.07 30.99 1.74 32.82 50.71 MMMLU 31.53 50.16 26.57 60.27 63.27 INCLUDE 24.74 34.26 25.62 39.55 45.57 -#### Qwen3-32B-Base +##### Qwen3-32B-Base -Qwen3-32B-Base is our largest dense model among the Qwen3 series. We compare it to the baselines of similar sizes, including Gemma-3-27B (gemma3) and Qwen2.5-32B (qwen2.5). In addition, we introduce two strong baselines: the recently open-source MoE model Llama-4-Scout, which has three times the parameters of Qwen3-32B-Base but half the activated parameters; and our previous flagship open-source dense model Qwen2.5-72B-Base, which has more than twice the number of parameters compared to Qwen3-32B-Base. The results are shown in Table [4](https://arxiv.org/html/2505.09388#S3.T4 "Table 4 ‣ Qwen3-235B-A22B-Base ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), which support three key conclusions: +Qwen3-32B-Base is our largest dense model among the Qwen3 series. We compare it to the baselines of similar sizes, including Gemma-3-27B ([Team et al. 2025](https://arxiv.org/html/2505.09388#bib.bib59)) and Qwen2.5-32B ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)). In addition, we introduce two strong baselines: the recently open-source MoE model Llama-4-Scout, which has three times the parameters of Qwen3-32B-Base but half the activated parameters; and our previous flagship open-source dense model Qwen2.5-72B-Base, which has more than twice the number of parameters compared to Qwen3-32B-Base. The results are shown in Table [4](https://arxiv.org/html/2505.09388#S3.T4 "Table 4 ‣ Qwen3-235B-A22B-Base ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), which support three key conclusions: 1. (1) Compared with the similar-sized models, Qwen3-32B-Base outperforms Qwen2.5-32B-Base and Gemma-3-27B Base on most benchmarks. Notably, Qwen3-32B-Base achieves 65.54 on MMLU-Pro and 39.78 on SuperGPQA, significantly outperforming its predecessor Qwen2.5-32B-Base. In addition, Qwen3-32B-Base achieves significantly higher encoding benchmark scores than all baseline models. @@ -281,9 +284,9 @@ Surprisingly, we find that Qwen3-32B-Base achieves competitive results compared 3. (3) Compared to Llama-4-Scout-Base, Qwen3-32B-Base significantly outperforms it on all 15 benchmarks, with only one-third of the number of parameters of Llama-4-Scout-Base, but twice the number of activated parameters. -#### Qwen3-14B-Base & Qwen3-30B-A3B-Base +##### Qwen3-14B-Base & Qwen3-30B-A3B-Base -The evaluation of the Qwen3-14B-Base and Qwen3-30B-A3B-Base is compared against baselines of similar sizes, including Gemma-3-12B Base, Qwen2.5-14B Base. Similarly, we also introduce two strong baselines: (1) Qwen2.5-Turbo (qwen2.5), which has 42B parameters and 6B activated parameters. Note that its activated parameters are twice those of Qwen3-30B-A3B-Base. (2) Qwen2.5-32B-Base, which has 11 times the activated parameters of Qwen3-30B-A3B and more than twice that of Qwen3-14B. The results are shown in Table [5](https://arxiv.org/html/2505.09388#S3.T5 "Table 5 ‣ Qwen3-235B-A22B-Base ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), where we can draw the following conclusions. +The evaluation of the Qwen3-14B-Base and Qwen3-30B-A3B-Base is compared against baselines of similar sizes, including Gemma-3-12B Base, Qwen2.5-14B Base. Similarly, we also introduce two strong baselines: (1) Qwen2.5-Turbo ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), which has 42B parameters and 6B activated parameters. Note that its activated parameters are twice those of Qwen3-30B-A3B-Base. (2) Qwen2.5-32B-Base, which has 11 times the activated parameters of Qwen3-30B-A3B and more than twice that of Qwen3-14B. The results are shown in Table [5](https://arxiv.org/html/2505.09388#S3.T5 "Table 5 ‣ Qwen3-235B-A22B-Base ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), where we can draw the following conclusions. 1. (1) Compared with the similar-sized models, Qwen3-14B-Base significantly performs better than Qwen2.5-14B-Base and Gemma-3-12B-Base on all 15 benchmarks. @@ -294,56 +297,468 @@ Similarly, Qwen3-14B-Base also achieves very competitive results compared to Qwe 3. (3) With only 1/5 activated non-embedding parameters, Qwen3-30B-A3B significantly outperforms Qwen2.5-14B-Base on all tasks, and achieves comparable performance to Qwen3-14B-Base and Qwen2.5-32B-Base, which brings us significant advantages in inference and training costs. -#### Qwen3-8B / 4B / 1.7B / 0.6B-Base +##### Qwen3-8B / 4B / 1.7B / 0.6B-Base For edge-side models, we take similar-sized Qwen2.5, Llama-3, and Gemma-3 base models as the baselines. The results can be seen in Table [6](https://arxiv.org/html/2505.09388#S3.T6 "Table 6 ‣ Qwen3-235B-A22B-Base ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), Table [7](https://arxiv.org/html/2505.09388#S3.T7 "Table 7 ‣ Qwen3-235B-A22B-Base ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"), and Table [8](https://arxiv.org/html/2505.09388#S3.T8 "Table 8 ‣ Qwen3-235B-A22B-Base ‣ 3.3 Pre-training Evaluation ‣ 3 Pre-training ‣ Qwen3 Technical Report"). All Qwen3 8B / 4B / 1.7B / 0.6B-Base models continue to maintain strong performance across nearly all benchmarks. Notably, Qwen3-8B / 4B / 1.7B-Base models even outperform larger size Qwen2.5-14B / 7B / 3B Base models on over half of the benchmarks, especially on STEM-related and coding benchmarks, reflecting the significant improvement of the Qwen3 models. ## 4 Post-training -![Image 1: Refer to caption](https://arxiv.org/html/2505.09388v1/x1.png) - Figure 1: Post-training pipeline of the Qwen3 series models. The post-training pipeline of Qwen3 is strategically designed with two core objectives: 1. (1) -Thinking Control: This involves the integration of two distinct modes, namely the ``non-thinking'' and ``thinking'' modes, providing users with the flexibility to choose whether the model should engage in reasoning or not, and to control the depth of thinking by specifying a token budget for the thinking process. +Thinking Control: This involves the integration of two distinct modes, namely the “non-thinking” and “thinking” modes, providing users with the flexibility to choose whether the model should engage in reasoning or not, and to control the depth of thinking by specifying a token budget for the thinking process. 2. (2) Strong-to-Weak Distillation: This aims to streamline and optimize the post-training process for lightweight models. By leveraging the knowledge from large-scale models, we substantially reduce both the computational costs and the development efforts required for building smaller-scale models. -As illustrated in Figure [1](https://arxiv.org/html/2505.09388#S4.F1 "Figure 1 ‣ 4 Post-training ‣ Qwen3 Technical Report"), the flagship models in the Qwen3 series follow a sophisticated four-stage training process. The first two stages focus on developing the models' ``thinking'' abilities. The next two stages aim to integrate strong ``non-thinking'' functionalities into the models. +As illustrated in Figure [1](https://arxiv.org/html/2505.09388#S4.F1 "Figure 1 ‣ 4 Post-training ‣ Qwen3 Technical Report"), the flagship models in the Qwen3 series follow a sophisticated four-stage training process. The first two stages focus on developing the models’ “thinking” abilities. The next two stages aim to integrate strong “non-thinking” functionalities into the models. -Preliminary experiments suggest that directly distilling the output logits from teacher models into lightweight student models can effectively enhance their performance while maintaining fine-grained control over their reasoning processes. This approach eliminates the necessity of performing an exhaustive four-stage training process individually for every small-scale model. It leads to better immediate performance, as indicated by higher Pass@1 scores, and also improves the model's ability of exploration, as reflected in improved Pass@64 results. In addition, it achieves these gains with much greater training efficiency, requiring only 1/10 of the GPU hours compared to the four-stage training method. +Preliminary experiments suggest that directly distilling the output logits from teacher models into lightweight student models can effectively enhance their performance while maintaining fine-grained control over their reasoning processes. This approach eliminates the necessity of performing an exhaustive four-stage training process individually for every small-scale model. It leads to better immediate performance, as indicated by higher Pass@1 scores, and also improves the model’s ability of exploration, as reflected in improved Pass@64 results. In addition, it achieves these gains with much greater training efficiency, requiring only 1/10 of the GPU hours compared to the four-stage training method. In the following sections, we present the four-stage training process and provide a detailed explanation of the Strong-to-Weak Distillation approach. ### 4.1 Long-CoT Cold Start -We begin by curating a comprehensive dataset that spans a wide range of categories, including math, code, logical reasoning, and general STEM problems. Each problem in the dataset is paired with verified reference answers or code-based test cases. This dataset serves as the foundation for the ``cold start'' phase of long Chain-of-Thought (long-CoT) training. +We begin by curating a comprehensive dataset that spans a wide range of categories, including math, code, logical reasoning, and general STEM problems. Each problem in the dataset is paired with verified reference answers or code-based test cases. This dataset serves as the foundation for the “cold start” phase of long Chain-of-Thought (long-CoT) training. -The dataset construction involves a rigorous two-phase filtering process: query filtering and response filtering. In the query filtering phase, we use Qwen2.5-72B-Instruct to identify and remove queries that are not easily verifiable. This includes queries containing multiple sub-questions or those asking for general text generation. Furthermore, we exclude queries that Qwen2.5-72B-Instruct can answer correctly without using CoT reasoning. This helps prevent the model from relying on superficial guessing and ensures that only complex problems requiring deeper reasoning are included. Additionally, we annotate each query's domain using Qwen2.5-72B-Instruct to maintain balanced domain representation across the dataset. +The dataset construction involves a rigorous two-phase filtering process: query filtering and response filtering. In the query filtering phase, we use Qwen2.5-72B-Instruct to identify and remove queries that are not easily verifiable. This includes queries containing multiple sub-questions or those asking for general text generation. Furthermore, we exclude queries that Qwen2.5-72B-Instruct can answer correctly without using CoT reasoning. This helps prevent the model from relying on superficial guessing and ensures that only complex problems requiring deeper reasoning are included. Additionally, we annotate each query’s domain using Qwen2.5-72B-Instruct to maintain balanced domain representation across the dataset. -After reserving a validation query set, we generate N candidate responses for each remaining query using QwQ-32B (qwq32b). When QwQ-32B consistently fails to generate correct solutions, human annotators manually assess the accuracy of the responses. For queries with positive Pass@N, further stringent filtering criteria are applied to remove responses that (1) yield incorrect final answers, (2) contain substantial repetition, (3) clearly indicate guesswork without adequate reasoning, (4) exhibit inconsistencies between the thinking and summary contents, (5) involve inappropriate language mixing or stylistic shifts, or (6) are suspected of being overly similar to potential validation set items. Subsequently, a carefully selected subset of the refined dataset is used for the initial cold-start training of the reasoning patterns. The objective at this stage is to instill foundational reasoning patterns in the model without overly emphasizing immediate reasoning performance. This approach ensures that the model's potential is not limited, allowing for greater flexibility and improvement during the subsequent reinforcement learning (RL) phase. To achieve this objective effectively, it is preferable to minimize both the number of training samples and the training steps during this preparatory phase. +After reserving a validation query set, we generate N candidate responses for each remaining query using QwQ-32B ([Qwen Team 2025](https://arxiv.org/html/2505.09388#bib.bib50)). When QwQ-32B consistently fails to generate correct solutions, human annotators manually assess the accuracy of the responses. For queries with positive Pass@N, further stringent filtering criteria are applied to remove responses that (1) yield incorrect final answers, (2) contain substantial repetition, (3) clearly indicate guesswork without adequate reasoning, (4) exhibit inconsistencies between the thinking and summary contents, (5) involve inappropriate language mixing or stylistic shifts, or (6) are suspected of being overly similar to potential validation set items. Subsequently, a carefully selected subset of the refined dataset is used for the initial cold-start training of the reasoning patterns. The objective at this stage is to instill foundational reasoning patterns in the model without overly emphasizing immediate reasoning performance. This approach ensures that the model’s potential is not limited, allowing for greater flexibility and improvement during the subsequent reinforcement learning (RL) phase. To achieve this objective effectively, it is preferable to minimize both the number of training samples and the training steps during this preparatory phase. ### 4.2 Reasoning RL -The query-verifier pairs used in the Reasoning RL stage must satisfy the following four criteria: (1) They were not used during the cold-start phase. (2) They are learnable for the cold-start model. (3) They are as challenging as possible. (4) They cover a broad range of sub-domains. We ultimately collect a total of 3,995 query-verifier pairs, and employed GRPO (deepseekmath) to update the model parameters. We observe that using a large batch size and a high number of rollouts per query, along with off-policy training to improve sample efficiency, is beneficial to the training process. We have also addressed how to balance exploration and exploitation by controlling the model’s entropy to increase steadily or remain stable, which is crucial for maintaining stable training. As a result, we achieve consistent improvements in both training reward and validation performance over the course of a single RL run, without any manual intervention on hyperparameters. For instance, the AIME'24 score of the Qwen3-235B-A22B model increases from 70.1 to 85.1 over a total of 170 RL training steps. +The query-verifier pairs used in the Reasoning RL stage must satisfy the following four criteria: (1) They were not used during the cold-start phase. (2) They are learnable for the cold-start model. (3) They are as challenging as possible. (4) They cover a broad range of sub-domains. We ultimately collect a total of 3,995 query-verifier pairs, and employed GRPO ([Shao et al. 2024](https://arxiv.org/html/2505.09388#bib.bib54)) to update the model parameters. We observe that using a large batch size and a high number of rollouts per query, along with off-policy training to improve sample efficiency, is beneficial to the training process. We have also addressed how to balance exploration and exploitation by controlling the model’s entropy to increase steadily or remain stable, which is crucial for maintaining stable training. As a result, we achieve consistent improvements in both training reward and validation performance over the course of a single RL run, without any manual intervention on hyperparameters. For instance, the AIME’24 score of the Qwen3-235B-A22B model increases from 70.1 to 85.1 over a total of 170 RL training steps. ### 4.3 Thinking Mode Fusion -The goal of the Thinking Mode Fusion stage is to integrate the ``non-thinking'' capabilities into the previously developed ``thinking'' model. This approach allows developers to manage and control reasoning behaviors, while also reducing the cost and complexity of deploying separate models for thinking and non-thinking tasks. To achieve this, we conduct continual supervised fine-tuning (SFT) on the Reasoning RL model and design a chat template to fuse the two modes. Moreover, we find that models capable of handling both modes proficiently perform consistently well under different thinking budgets. +The goal of the Thinking Mode Fusion stage is to integrate the “non-thinking” capabilities into the previously developed “thinking” model. This approach allows developers to manage and control reasoning behaviors, while also reducing the cost and complexity of deploying separate models for thinking and non-thinking tasks. To achieve this, we conduct continual supervised fine-tuning (SFT) on the Reasoning RL model and design a chat template to fuse the two modes. Moreover, we find that models capable of handling both modes proficiently perform consistently well under different thinking budgets. -#### Construction of SFT data. +##### Construction of SFT data. -The SFT dataset combines both the ``thinking'' and ``non-thinking'' data. To ensure that the performance of the Stage 2 model is not compromised by the additional SFT, the ``thinking'' data is generated via rejection sampling on Stage 1 queries using the Stage 2 model itself. The ``non-thinking'' data, on the other hand, is carefully curated to cover a diverse range of tasks, including coding, mathematics, instruction-following, multilingual tasks, creative writing, question answering, and role-playing. Additionally, we employ automatically generated checklists for assessing the response quality of ``non-thinking'' data. To enhance the performance on tasks with low-resource languages, we particularly increase the proportion of translation tasks. +The SFT dataset combines both the “thinking” and “non-thinking” data. To ensure that the performance of the Stage 2 model is not compromised by the additional SFT, the “thinking” data is generated via rejection sampling on Stage 1 queries using the Stage 2 model itself. The “non-thinking” data, on the other hand, is carefully curated to cover a diverse range of tasks, including coding, mathematics, instruction-following, multilingual tasks, creative writing, question answering, and role-playing. Additionally, we employ automatically generated checklists for assessing the response quality of “non-thinking” data. To enhance the performance on tasks with low-resource languages, we particularly increase the proportion of translation tasks. -#### Chat Template Design. +##### Chat Template Design. -To better integrate the two modes and enable users to dynamically switch the model's thinking process, we design chat templates for Qwen3, as shown in Table [4.3](https://arxiv.org/html/2505.09388#S4.SS3.SSS0.Px3 "Thinking Budget. ‣ 4.3 Thinking Mode Fusion ‣ 4 Post-training ‣ Qwen3 Technical Report"). Specifically, for samples in thinking mode and non-thinking mode, we introduce /think and /no_think flags in the user query or system message, respectively. This allows the model to follow the user's input and select the appropriate thinking mode accordingly. For non-thinking mode samples, we retain an empty thinking block in the assistant's response. This design ensures internal format consistency within the model and allows developers to prevent the model from engaging in thinking behavior by concatenating an empty think block in the chat template. By default, the model operates in thinking mode; therefore, we add some thinking mode training samples where the user queries do not include /think flags. For more complex multi-turn dialogs, we randomly insert multiple /think and /no_think flags into users' queries, with the model response adhering to the last flag encountered. +To better integrate the two modes and enable users to dynamically switch the model’s thinking process, we design chat templates for Qwen3, as shown in Table [9](https://arxiv.org/html/2505.09388#S4.T9 "Table 9 ‣ Thinking Budget. ‣ 4.3 Thinking Mode Fusion ‣ 4 Post-training ‣ Qwen3 Technical Report"). Specifically, for samples in thinking mode and non-thinking mode, we introduce /think and /no_think flags in the user query or system message, respectively. This allows the model to follow the user’s input and select the appropriate thinking mode accordingly. For non-thinking mode samples, we retain an empty thinking block in the assistant’s response. This design ensures internal format consistency within the model and allows developers to prevent the model from engaging in thinking behavior by concatenating an empty think block in the chat template. By default, the model operates in thinking mode; therefore, we add some thinking mode training samples where the user queries do not include /think flags. For more complex multi-turn dialogs, we randomly insert multiple /think and /no_think flags into users’ queries, with the model response adhering to the last flag encountered. -#### Thinking Budget. +##### Thinking Budget. -An additional advantage of Thinking Mode Fusion is that, once the model learns to respond in both non-thinking and thinking modes, it naturally develops the ability to handle intermediate cases—generating responses based on incomplete thinking. This capability lays the foundation for implementing budget control over the model's thinking process. Specifically, when the length of the model's thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction: ``Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n.\n\n''. After this instruction is inserted, the model proceeds to generate a final response based on its accumulated reasoning up to that point. It is worth noting that this ability is not explicitly trained but emerges naturally as a result of applying Thinking Mode Fusion. +An additional advantage of Thinking Mode Fusion is that, once the model learns to respond in both non-thinking and thinking modes, it naturally develops the ability to handle intermediate cases—generating responses based on incomplete thinking. This capability lays the foundation for implementing budget control over the model’s thinking process. Specifically, when the length of the model’s thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction: “Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n.\n\n”. After this instruction is inserted, the model proceeds to generate a final response based on its accumulated reasoning up to that point. It is worth noting that this ability is not explicitly trained but emerges naturally as a result of applying Thinking Mode Fusion. -Table 9: Examples of SFT data for thinking and non-thinking modes during the thinking mode fusion stage. For the thinking mode, the /think flag can be omitted since it represents the default behavior. This feature has been implemented in the chat template 2 2 2[https://huggingface.co/Qwen/Qwen3-32B/blob/main/tokenizer_config.json](https://huggingface.co/Qwen/Qwen3-32B/blob/main/tokenizer_config.json) supported by the Hugging Face's tokenizer, where the thinking mode can be disabled using an additional parameter enable_thinking=False. +Table 9: Examples of SFT data for thinking and non-thinking modes during the thinking mode fusion stage. For the thinking mode, the /think flag can be omitted since it represents the default behavior. This feature has been implemented in the chat template 2 2 2[https://huggingface.co/Qwen/Qwen3-32B/blob/main/tokenizer_config.json](https://huggingface.co/Qwen/Qwen3-32B/blob/main/tokenizer_config.json) supported by the Hugging Face’s tokenizer, where the thinking mode can be disabled using an additional parameter enable_thinking=False. + +Thinking Mode Non-Thinking Mode<|im_start|>user{query}/think<|im_end|><|im_start|>assistant{thinking_content}{response}<|im_end|><|im_start|>user{query}/no_think<|im_end|><|im_start|>assistant{response}<|im_end|> + +### 4.4 General RL + +The General RL stage aims to broadly enhance the models’ capabilities and stability across diverse scenarios. To facilitate this, we have established a sophisticated reward system covering over 20 distinct tasks, each with customized scoring criteria. These tasks specifically target enhancements in the following core capabilities: + +* • +Instruction Following: This capability ensures that models accurately interpret and follow user instructions, including requirements related to content, format, length, and the use of structured output, delivering responses that align with user expectations. + +* • +Format Following: In addition to explicit instructions, we expect the model to adhere to specific formatting conventions. For instance, it should respond appropriately to the /think and /no_think flags by switching between thinking and non-thinking modes, and consistently use designated tokens (e.g., and ) to separate the thinking and response parts in the final output. + +* • +Preference Alignment: For open-ended queries, preference alignment focuses on improving the model’s helpfulness, engagement, and style, ultimately delivering a more natural and satisfying user experience. + +* • +Agent Ability: This involves training the model to correctly invoke tools via designated interfaces. During the RL rollout, the model is allowed to perform complete multi-turn interaction cycles with real environment execution feedback, thereby improving its performance and stability in long-horizon decision-making tasks. + +* • +Abilities for Specialized Scenarios: In more specialized scenarios, we design tasks tailored to the specific context. For example, in Retrieval-Augmented Generation (RAG) tasks, we incorporate reward signals to guide the model toward generating accurate and contextually appropriate responses, thereby minimizing the risk of hallucination. + +To provide feedback for the aforementioned tasks, we utilized three distinct types of rewards: + +1. (1) +Rule-based Reward: The rule-based reward has been widely used in the reasoning RL stage, and is also useful for general tasks such as instruction following ([Lambert et al. 2024](https://arxiv.org/html/2505.09388#bib.bib32)) and format adherence. Well-designed rule-based rewards can assess the correctness of model outputs with high precision, preventing issues like reward hacking. + +2. (2) +Model-based Reward with Reference Answer: In this approach, we provide a reference answer for each query and prompt Qwen2.5-72B-Instruct to score the model’s response based on this reference. This method allows for more flexible handling of diverse tasks without requiring strict formatting, avoiding false negatives that can occur with purely rule-based rewards. + +3. (3) +Model-based Reward without Reference Answer: Leveraging human preference data, we train a reward model to assign scalar scores to model responses. This approach, which does not depend on a reference answer, can handle a broader range of queries while effectively enhancing the model’s engagement and helpfulness. + +### 4.5 Strong-to-Weak Distillation + +The Strong-to-Weak Distillation pipeline is specifically designed to optimize lightweight models, encompassing 5 dense models (Qwen3-0.6B, 1.7B, 4B, 8B, and 14B) and one MoE model (Qwen3-30B-A3B). This approach enhances model performance while effectively imparting robust mode-switching capabilities. The distillation process is divided into two primary phases: + +1. (1) +Off-policy Distillation: At this initial phase, we combine the outputs of teacher models generated with both /think and /no_think modes for response distillation. This helps lightweight student models develop basic reasoning skills and the ability to switch between different modes of thinking, laying a solid foundation for the next on-policy training phase. + +2. (2) +On-policy Distillation: In this phase, the student model generates on-policy sequences for fine-tuning. Specifically, prompts are sampled, and the student model produces responses in either /think or /no_think mode. The student model is then fine-tuned by aligning its logits with those of a teacher model (Qwen3-32B or Qwen3-235B-A22B) to minimize the KL divergence. + +### 4.6 Post-training Evaluation + +To comprehensively evaluate the quality of instruction-tuned models, we adopted automatic benchmarks to assess model performance under both thinking and non-thinking modes. These benchmarks are categorized into several dimensions: + +* • +General Tasks: We utilize benchmarks including MMLU-Redux ([Gema et al. 2024](https://arxiv.org/html/2505.09388#bib.bib21)), GPQA-Diamond ([Rein et al. 2023](https://arxiv.org/html/2505.09388#bib.bib51)), C-Eval ([Huang et al. 2023](https://arxiv.org/html/2505.09388#bib.bib28)), and LiveBench (2024-11-25) ([White et al. 2024](https://arxiv.org/html/2505.09388#bib.bib63)). For GPQA-Diamond, we sample 10 times for each query and report the averaged accuracy. + +* • +Alignment Tasks: To evaluate how well the model aligns with human preferences, we employ a suite of specialized benchmarks. For instruction-following performance, we report the strict-prompt accuracy of IFEval ([Zhou et al. 2023](https://arxiv.org/html/2505.09388#bib.bib73)). To assess alignment with human preferences on general topics, we utilize Arena-Hard ([Li et al. 2024](https://arxiv.org/html/2505.09388#bib.bib33)) and AlignBench v1.1 ([Liu et al. 2023b](https://arxiv.org/html/2505.09388#bib.bib39)). For writing tasks, we rely on Creative Writing V3 ([Paech 2024](https://arxiv.org/html/2505.09388#bib.bib45)) and WritingBench ([Wu et al. 2025](https://arxiv.org/html/2505.09388#bib.bib64)) to evaluate the model’s proficiency and creativity. + +* • +Math & Text Reasoning: For evaluating mathematical and logical reasoning skills, we employ high-level math benchmarks including MATH-500 ([Lightman et al. 2023](https://arxiv.org/html/2505.09388#bib.bib34)), AIME’24 and AIME’25 ([AIME 2025](https://arxiv.org/html/2505.09388#bib.bib2)), and text reasoning tasks including ZebraLogic ([Lin et al. 2025](https://arxiv.org/html/2505.09388#bib.bib35)) and AutoLogi ([Zhu et al. 2025](https://arxiv.org/html/2505.09388#bib.bib74)). For AIME problems, each year’s questions include Part I and Part II, totaling 30 questions. For each question, we sample 64 times and take the average accuracy as the final score. + +* • +Agent & Coding: To test the model’s proficiency in coding and agent-based tasks, we use BFCL v3 ([Yan et al. 2024](https://arxiv.org/html/2505.09388#bib.bib68)), LiveCodeBench (v5, 2024.10-2025.02) ([Jain et al. 2024](https://arxiv.org/html/2505.09388#bib.bib30)), and Codeforces Ratings from CodeElo ([Quan et al. 2025](https://arxiv.org/html/2505.09388#bib.bib48)). For BFCL, all Qwen3 models are evaluated using the FC format, and yarn was used to deploy the models to a context length of 64k for Multi-Turn evaluation. Some baselines are derived from the BFCL leaderboard, taking the higher scores between FC and Prompt formats. For models not reported on the leaderboard, the Prompt formats are evaluated. For LiveCodeBench, for the non-thinking mode, we use the officially recommended prompt, while for the thinking mode, we adjust the prompt template to allow the model to think more freely, by removing the restriction You will not return anything except for the program. To evaluate the performance gap between models and competitive programming experts, we use CodeForces to calculate Elo ratings. In our benchmark, each problem is solved by generating up to eight independent reasoning attempts. + +* • +Multilingual Tasks: For multilingual capabilities, we evaluate four kinds of tasks: instruction following, knowledge, mathematics, and logical reasoning. Instruction following is assessed using Multi-IF ([He et al. 2024](https://arxiv.org/html/2505.09388#bib.bib24)), which focuses on 8 key languages. Knowledge assessment consisted of two types: regional knowledge evaluated through INCLUDE ([Romanou et al. 2024](https://arxiv.org/html/2505.09388#bib.bib52)), covering 44 languages, and general knowledge assessed with MMMLU ([OpenAI 2024](https://arxiv.org/html/2505.09388#bib.bib42)) across 14 languages, excluding the unoptimized Yoruba language; for these two benchmarks, we sample only 10% of the original data to improve evaluation efficiency. The mathematics task employ MT-AIME2024 ([Son et al. 2025](https://arxiv.org/html/2505.09388#bib.bib56)), encompassing 55 languages, and PolyMath ([Wang et al. 2025](https://arxiv.org/html/2505.09388#bib.bib61)), which includes 18 languages. Logical reasoning is evaluated using MlogiQA, covering 10 languages, sourced from [Zhang et al. 2024](https://arxiv.org/html/2505.09388#bib.bib72). + +Table 10: Multilingual benchmarks and the included languages. The languages are identified in IETF language tags. + +Benchmark# Langs Languages +Multi-IF 8 en, es, fr, hi, it, pt, ru, zh +INCLUDE 44 ar, az, be, bg, bn, de, el, es, et, eu, fa, fi, fr, he, hi, hr, hu, hy, id, it, ja, ka,kk, ko, lt, mk, ml, ms, ne, nl, pl, pt, ru, sq, sr, ta, te, tl, tr, uk, ur, uz, vi, zh +MMMLU 14 ar, bn, de, en, es, fr, hi, id, it, ja, ko, pt, sw, zh +MT-AIME2024 55 af, ar, bg, bn, ca, cs, cy, da, de, el, en, es, et, fa, fi, fr, gu, he, hi, hr, hu, id,it, ja, kn, ko, lt, lv, mk, ml, mr, ne, nl, no, pa, pl, pt, ro, ru, sk, sl, so, sq, sv,sw, ta, te, th, tl, tr, uk, ur, vi, zh-Hans, zh-Hant +PolyMath 18 ar, bn, de, en, es, fr, id, it, ja, ko, ms, pt, ru, sw, te, th, vi, zh +MLogiQA 10 ar, en, es, fr, ja, ko, pt, th, vi, zh + +Table 11: Comparison among Qwen3-235B-A22B (Thinking) and other reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +OpenAI-o1 DeepSeek-R1 Grok-3-Beta(Think)Gemini2.5-Pro Qwen3-235B-A22B Architecture-MoE--MoE# Activated Params-37B--22B# Total Params-671B--235B General Tasks MMLU-Redux 92.8 92.9-93.7 92.7 GPQA-Diamond 78.0 71.5 80.2 84.0 71.1 C-Eval 85.5 91.8-82.9 89.6 LiveBench 2024-11-25 75.7 71.6-82.4 77.1 Alignment Tasks IFEval strict prompt 92.6 83.3-89.5 83.4 Arena-Hard 92.1 92.3-96.4 95.6 AlignBench v1.1 8.86 8.76-9.03 8.94 Creative Writing v3 81.7 85.5-86.0 84.6 WritingBench 7.69 7.71-8.09 8.03 Math & Text Reasoning MATH-500 96.4 97.3 98.8 98.0 AIME’24 74.3 79.8 83.9 92.0 85.7 AIME’25 79.2 70.0 77.3 86.7 81.5 ZebraLogic 81.0 78.7-87.4 80.3 AutoLogi 79.8 86.1-85.4 89.0 Agent &Coding BFCL v3 67.8 56.9-62.9 70.8 LiveCodeBench v5 63.9 64.3 70.6 70.4 70.7 CodeForces (Rating / Percentile)1891 / 96.7%2029 / 98.1%-2001 / 97.9%2056 / 98.2%Multilingual Tasks Multi-IF 48.8 67.7-77.8 71.9 INCLUDE 84.6 82.7-85.1 78.7 MMMLU 14 languages 88.4 86.4-86.9 84.3 MT-AIME2024 67.4 73.5-76.9 80.8 PolyMath 38.9 47.1-52.2 54.7 MLogiQA 75.5 73.8-75.6 77.1 + +Table 12: Comparison among Qwen3-235B-A22B (Non-thinking) and other non-reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +GPT-4o-2024-11-20 DeepSeek-V3 Qwen2.5-72B-Instruct LLaMA-4-Maverick Qwen3-235B-A22B Architecture-MoE Dense MoE MoE# Activated Params-37B 72B 17B 22B# Total Params-671B 72B 402B 235B General Tasks MMLU-Redux 87.0 89.1 86.8 91.8 89.2 GPQA-Diamond 46.0 59.1 49.0 69.8 62.9 C-Eval 75.5 86.5 84.7 83.5 86.1 LiveBench 2024-11-25 52.2 60.5 51.4 59.5 62.5 Alignment Tasks IFEval strict prompt 86.5 86.1 84.1 86.7 83.2 Arena-Hard 85.3 85.5 81.2 82.7 96.1 AlignBench v1.1 8.42 8.64 7.89 7.97 8.91 Creative Writing v3 81.1 74.0 61.8 61.3 80.4 WritingBench 7.11 6.49 7.06 5.46 7.70 Math & Text Reasoning MATH-500 77.2 90.2 83.6 90.6 91.2 AIME’24 11.1 39.2 18.9 38.5 40.1 AIME’25 7.6 28.8 15.0 15.9 24.7 ZebraLogic 27.4 42.1 26.6 40.0 37.7 AutoLogi 65.9 76.1 66.1 75.2 83.3 Agent &Coding BFCL v3 72.5 57.6 63.4 52.9 68.0 LiveCodeBench v5 32.7 33.1 30.7 37.2 35.3 CodeForces (Rating / Percentile)864 / 35.4%1134 / 54.1%859 / 35.0%712 / 24.3%1387 / 75.7%Multilingual Tasks Multi-IF 65.6 55.6 65.3 75.5 70.2 INCLUDE 78.8 76.7 69.6 80.9 75.6 MMMLU 14 languages 80.3 81.1 76.9 82.5 79.8 MT-AIME2024 9.2 20.9 12.7 27.0 32.4 PolyMath 13.7 20.4 16.9 26.1 27.0 MLogiQA 57.4 58.9 59.3 59.9 67.6 + +For all Qwen3 models in the thinking mode, we utilize a sampling temperature of 0.6, a top-p value of 0.95, and a top-k value of 20. Additionally, for Creative Writing v3 and WritingBench, we apply a presence penalty of 1.5 to encourage the generation of more diverse content. For Qwen3 models in the non-thinking mode, we configure the sampling hyperparameters with temperature = 0.7, top-p = 0.8, top-k = 20, and presence penalty = 1.5. For both the thinking and non-thinking modes, we set the max output length to 32,768 tokens, except AIME’24 and AIME’25 where we extend this length to 38,912 tokens to provide sufficient thinking space. + +##### Summary of Evaluation Results + +From the evaluation results, we summarize several key conclusions of the finalized Qwen3 models as follows: + +1. (1) +Our flagship model, Qwen3-235B-A22B, demonstrates the state-of-the-art overall performance among open-source models in both the thinking and non-thinking modes, surpassing strong baselines such as DeepSeek-R1 and DeepSeek-V3. Qwen3-235B-A22B is also highly competitive to closed-source leading models, such as OpenAI-o1, Gemini2.5-Pro, and GPT-4o, showcasing its profound reasoning capabilities and comprehensive general abilities. + +2. (2) +Our flagship dense model, Qwen3-32B, outperforms our previous strongest reasoning model, QwQ-32B, in most of the benchmarks, and performs comparably to the closed-source OpenAI-o3-mini, indicating its compelling reasoning capabilities. Qwen3-32B is also remarkably performant in the non-thinking mode and surpasses our previous flagship non-reasoning dense model, Qwen2.5-72B-Instruct. + +3. (3) +Our lightweight models, including Qwen3-30B-A3B, Qwen3-14B, and other smaller dense ones, possess consistently superior performance to the open-source models with a close or larger amount of parameters, proving the success of our Strong-to-Weak Distillation approach. + +The detailed results are as follows. + +##### Qwen3-235B-A22B + +For our flagship model Qwen3-235B-A22B, we compare it with the leading reasoning and non-reasoning models. For the thinking mode, we take OpenAI-o1 ([OpenAI 2024](https://arxiv.org/html/2505.09388#bib.bib43)), DeepSeek-R1 ([Guo et al. 2025](https://arxiv.org/html/2505.09388#bib.bib23)), Grok-3-Beta (Think) ([xAI 2025](https://arxiv.org/html/2505.09388#bib.bib65)), and Gemini2.5-Pro ([DeepMind 2025](https://arxiv.org/html/2505.09388#bib.bib16)) as the reasoning baselines. For the non-thinking mode, we take GPT-4o-2024-11-20 ([OpenAI 2024](https://arxiv.org/html/2505.09388#bib.bib41)), DeepSeek-V3 ([Liu et al. 2024a](https://arxiv.org/html/2505.09388#bib.bib36)), Qwen2.5-72B-Instruct ([Yang et al. 2024b](https://arxiv.org/html/2505.09388#bib.bib70)), and LLaMA-4-Maverick ([Meta-AI 2025](https://arxiv.org/html/2505.09388#bib.bib40)) as the non-reasoning baselines. We present the evaluation results in Table [11](https://arxiv.org/html/2505.09388#S4.T11 "Table 11 ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report") and [12](https://arxiv.org/html/2505.09388#S4.T12 "Table 12 ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"). + +1. (1) +From Table [11](https://arxiv.org/html/2505.09388#S4.T11 "Table 11 ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"), with only 60% activated and 35% total parameters, Qwen3-235B-A22B (Thinking) outperforms DeepSeek-R1 on 17/23 the benchmarks, particularly on the reasoning-demanded tasks (e.g., mathematics, agent, and coding), demonstrating the state-of-the-art reasoning capabilities of Qwen3-235B-A22B among open-source models. Moreover, Qwen3-235B-A22B (Thinking) is also highly competitive to the closed-source OpenAI-o1, Grok-3-Beta (Think), and Gemini2.5-Pro, substantially narrowing the gap in the reasoning capabilities between open-source and close-source models. + +2. (2) +From Table [12](https://arxiv.org/html/2505.09388#S4.T12 "Table 12 ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"), Qwen3-235B-A22B (Non-thinking) exceeds the other leading open-source models, including DeepSeek-V3, LLaMA-4-Maverick, and our previous flagship model Qwen2.5-72B-Instruct, and also surpasses the closed-source GPT-4o-2024-11-20 in 18/23 the benchmarks, indicating its inherent strong capabilities even when not enhanced with the deliberate thinking process. + +##### Qwen3-32B + +For our flagship dense model, Qwen3-32B, we take DeepSeek-R1-Distill-Llama-70B, OpenAI-o3-mini (medium), and our previous strongest reasoning model, QwQ-32B ([Qwen Team 2025](https://arxiv.org/html/2505.09388#bib.bib50)), as the baselines in the thinking mode. We also take GPT-4o-mini-2024-07-18, LLaMA-4-Scout, and our previous flagship model, Qwen2.5-72B-Instruct, as the baselines in the non-thinking mode. We present the evaluation results in Table [13](https://arxiv.org/html/2505.09388#S4.T13 "Table 13 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report") and [14](https://arxiv.org/html/2505.09388#S4.T14 "Table 14 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"). + +1. (1) +From Table [13](https://arxiv.org/html/2505.09388#S4.T13 "Table 13 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"), Qwen3-32B (Thinking) outperforms QwQ-32B on 17/23 the benchmarks, making it the new state-of-the-art reasoning model at the sweet size of 32B. Moreover, Qwen3-32B (Thinking) also competes with the closed-source OpenAI-o3-mini (medium) with better alignment and multilingual performance. + +2. (2) +From Table [14](https://arxiv.org/html/2505.09388#S4.T14 "Table 14 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"), Qwen3-32B (Non-thinking) exhibits superior performance to all the baselines on almost all the benchmarks. Particularly, Qwen3-32B (Non-thinking) performs on par with Qwen2.5-72B-Instruct on the general tasks with significant advantages on the alignment, multilingual, and reasoning-related tasks, again proving the fundamental improvements of Qwen3 over our previous Qwen2.5 series models. + +Table 13: Comparison among Qwen3-32B (Thinking) and other reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +DeepSeek-R1-Distill-Llama-70B QwQ-32B OpenAI-o3-mini(medium)Qwen3-32B Architecture Dense Dense-Dense# Activated Params 70B 32B-32B# Total Params 70B 32B-32B General Tasks MMLU-Redux 89.3 90.0 90.0 90.9 GPQA-Diamond 65.2 65.6 76.8 68.4 C-Eval 71.8 88.4 75.1 87.3 LiveBench 2024-11-25 54.5 72.0 70.0 74.9 Alignment Tasks IFEval strict prompt 79.3 83.9 91.5 85.0 Arena-Hard 60.6 89.5 89.0 93.8 AlignBench v1.1 6.74 8.70 8.38 8.72 Creative Writing v3 62.1 82.4 74.8 81.0 WritingBench 6.08 7.86 7.52 7.90 Math & Text Reasoning MATH-500 94.5 98.0 98.0 97.2 AIME’24 70.0 79.5 79.6 81.4 AIME’25 56.3 69.5 74.8 72.9 ZebraLogic 71.3 76.8 88.9 88.8 AutoLogi 83.5 88.1 86.3 87.3 Agent &Coding BFCL v3 49.3 66.4 64.6 70.3 LiveCodeBench v5 54.5 62.7 66.3 65.7 CodeForces (Rating / Percentile)1633 / 91.4%1982 / 97.7%2036 / 98.1%1977 / 97.7%Multilingual Tasks Multi-IF 57.6 68.3 48.4 73.0 INCLUDE 62.1 69.7 73.1 73.7 MMMLU 14 languages 69.6 80.9 79.3 80.6 MT-AIME2024 29.3 68.0 73.9 75.0 PolyMath 29.4 45.9 38.6 47.4 MLogiQA 60.3 75.5 71.1 76.3 + +Table 14: Comparison among Qwen3-32B (Non-thinking) and other non-reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +GPT-4o-mini-2024-07-18 LLaMA-4-Scout Qwen2.5-72B-Instruct Qwen3-32B Architecture-MoE Dense Dense# Activated Params-17B 72B 32B# Total Params-109B 72B 32B General Tasks MMLU-Redux 81.5 86.3 86.8 85.7 GPQA-Diamond 40.2 57.2 49.0 54.6 C-Eval 66.3 78.2 84.7 83.3 LiveBench 2024-11-25 41.3 47.6 51.4 59.8 Alignment Tasks IFEval strict prompt 80.4 84.7 84.1 83.2 Arena-Hard 74.9 70.5 81.2 92.8 AlignBench v1.1 7.81 7.49 7.89 8.58 Creative Writing v3 70.3 55.0 61.8 78.3 WritingBench 5.98 5.49 7.06 7.54 Math & Text Reasoning MATH-500 78.2 82.6 83.6 88.6 AIME’24 8.1 28.6 18.9 31.0 AIME’25 8.8 10.0 15.0 20.2 ZebraLogic 20.1 24.2 26.6 29.2 AutoLogi 52.6 56.8 66.1 78.5 Agent &Coding BFCL v3 64.0 45.4 63.4 63.0 LiveCodeBench v5 27.9 29.8 30.7 31.3 CodeForces (Rating / Percentile)1113 / 52.6%981 / 43.7%859 / 35.0%1353 / 71.0%Multilingual Tasks Multi-IF 62.4 64.2 65.3 70.7 INCLUDE 66.0 74.1 69.6 70.9 MMMLU 14 languages 72.1 77.5 76.9 76.5 MT-AIME2024 6.0 19.1 12.7 24.1 PolyMath 12.0 20.9 16.9 22.5 MLogiQA 42.6 53.9 59.3 62.9 + +Table 15: Comparison among Qwen3-30B-A3B / Qwen3-14B (Thinking) and other reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +DeepSeek-R1-Distill-Qwen-32B QwQ-32B Qwen3-14B Qwen3-30B-A3B Architecture Dense Dense Dense MoE# Activated Params 32B 32B 14B 3B# Total Params 32B 32B 14B 30B General Tasks MMLU-Redux 88.2 90.0 88.6 89.5 GPQA-Diamond 62.1 65.6 64.0 65.8 C-Eval 82.2 88.4 86.2 86.6 LiveBench 2024-11-25 45.6 72.0 71.3 74.3 Alignment Tasks IFEval strict prompt 72.5 83.9 85.4 86.5 Arena-Hard 60.8 89.5 91.7 91.0 AlignBench v1.1 7.25 8.70 8.56 8.70 Creative Writing v3 55.0 82.4 80.3 79.1 WritingBench 6.13 7.86 7.80 7.70 Math & Text Reasoning MATH-500 94.3 98.0 96.8 98.0 AIME’24 72.6 79.5 79.3 80.4 AIME’25 49.6 69.5 70.4 70.9 ZebraLogic 69.6 76.8 88.5 89.5 AutoLogi 74.6 88.1 89.2 88.7 Agent &Coding BFCL v3 53.5 66.4 70.4 69.1 LiveCodeBench v5 54.5 62.7 63.5 62.6 CodeForces (Rating / Percentile)1691 / 93.4%1982 / 97.7%1766 / 95.3%1974 / 97.7%Multilingual Tasks Multi-IF 31.3 68.3 74.8 72.2 INCLUDE 68.0 69.7 71.7 71.9 MMMLU 14 languages 78.6 80.9 77.9 78.4 MT-AIME2024 44.6 68.0 73.3 73.9 PolyMath 35.1 45.9 45.8 46.1 MLogiQA 63.3 75.5 71.1 70.1 + +Table 16: Comparison among Qwen3-30B-A3B / Qwen3-14B (Non-thinking) and other non-reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +Phi-4 Gemma-3-27B-IT Qwen2.5-32B-Instruct Qwen3-14B Qwen3-30B-A3B Architecture Dense Dense Dense Dense MoE# Activated Params 14B 27B 32B 14B 3B# Total Params 14B 27B 32B 14B 30B General Tasks MMLU-Redux 85.3 82.6 83.9 82.0 84.1 GPQA-Diamond 56.1 42.4 49.5 54.8 54.8 C-Eval 66.9 66.6 80.6 81.0 82.9 LiveBench 2024-11-25 41.6 49.2 50.0 59.6 59.4 Alignment Tasks IFEval strict prompt 62.1 80.6 79.5 84.8 83.7 Arena-Hard 75.4 86.8 74.5 86.3 88.0 AlignBench v1.1 7.61 7.80 7.71 8.52 8.55 Creative Writing v3 51.2 82.0 54.6 73.1 68.1 WritingBench 5.73 7.22 5.90 7.24 7.22 Math & Text Reasoning MATH-500 80.8 90.0 84.6 90.0 89.8 AIME’24 22.9 32.6 18.8 31.7 32.8 AIME’25 17.3 24.0 12.8 23.3 21.6 ZebraLogic 32.3 24.6 26.1 33.0 33.2 AutoLogi 66.2 64.2 65.5 82.0 81.5 Agent &Coding BFCL v3 47.0 59.1 62.8 61.5 58.6 LiveCodeBench v5 25.2 26.9 26.4 29.0 29.8 CodeForces (Rating / Percentile)1280 / 65.3%1063 / 49.3%903 / 38.2%1200 / 58.6%1267 / 64.1%Multilingual Tasks Multi-IF 49.5 69.8 63.2 72.9 70.8 INCLUDE 65.3 71.4 67.5 67.8 67.8 MMMLU 14 languages 74.7 76.1 74.2 72.6 73.8 MT-AIME2024 13.1 23.0 15.3 23.2 24.6 PolyMath 17.4 20.3 18.3 22.0 23.3 MLogiQA 53.1 58.5 58.0 58.9 53.3 + +Table 17: Comparison among Qwen3-8B / Qwen3-4B (Thinking) and other reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +DeepSeek-R1-Distill-Qwen-14B DeepSeek-R1-Distill-Qwen-32B Qwen3-4B Qwen3-8B Architecture Dense Dense Dense Dense# Activated Params 14B 32B 4B 8B# Total Params 14B 32B 4B 8B General Tasks MMLU-Redux 84.1 88.2 83.7 87.5 GPQA-Diamond 59.1 62.1 55.9 62.0 C-Eval 78.1 82.2 77.5 83.4 LiveBench 2024-11-25 52.3 45.6 63.6 67.1 Alignment Tasks IFEval strict prompt 72.6 72.5 81.9 85.0 Arena-Hard 48.0 60.8 76.6 85.8 AlignBench v1.1 7.43 7.25 8.30 8.46 Creative Writing v3 54.2 55.0 61.1 75.0 WritingBench 6.03 6.13 7.35 7.59 Math & Text Reasoning MATH-500 93.9 94.3 97.0 97.4 AIME’24 69.7 72.6 73.8 76.0 AIME’25 44.5 49.6 65.6 67.3 ZebraLogic 59.1 69.6 81.0 84.8 AutoLogi 78.6 74.6 87.9 89.1 Agent &Coding BFCL v3 49.5 53.5 65.9 68.1 LiveCodeBench v5 45.5 54.5 54.2 57.5 CodeForces (Rating / Percentile)1574 / 89.1%1691 / 93.4%1671 / 92.8%1785 / 95.6%Multilingual Tasks Multi-IF 29.8 31.3 66.3 71.2 INCLUDE 59.7 68.0 61.8 67.8 MMMLU 14 languages 73.8 78.6 69.8 74.4 MT-AIME2024 33.7 44.6 60.7 65.4 PolyMath 28.6 35.1 40.0 42.7 MLogiQA 53.6 63.3 65.9 69.0 + +Table 18: Comparison among Qwen3-8B / Qwen3-4B (Non-thinking) and other non-reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +LLaMA-3.1-8B-Instruct Gemma-3-12B-IT Qwen2.5-7B-Instruct Qwen2.5-14B-Instruct Qwen3-4B Qwen3-8B Architecture Dense Dense Dense Dense Dense Dense# Activated Params 8B 12B 7B 14B 4B 8B# Total Params 8B 12B 7B 14B 4B 8B General Tasks MMLU-Redux 61.7 77.8 75.4 80.0 77.3 79.5 GPQA-Diamond 32.8 40.9 36.4 45.5 41.7 39.3 C-Eval 52.0 61.1 76.2 78.0 72.2 77.9 LiveBench 2024-11-25 26.0 43.7 34.9 42.2 48.4 53.5 Alignment Tasks IFEval strict prompt 75.0 80.2 71.2 81.0 81.2 83.0 Arena-Hard 30.1 82.6 52.0 68.3 66.2 79.6 AlignBench v1.1 6.01 7.77 7.27 7.67 8.10 8.38 Creative Writing v3 52.8 79.9 49.8 55.8 53.6 64.5 WritingBench 4.57 7.05 5.82 5.93 6.85 7.15 Math & Text Reasoning MATH-500 54.8 85.6 77.6 83.4 84.8 87.4 AIME’24 6.3 22.4 9.1 15.2 25.0 29.1 AIME’25 2.7 18.8 12.1 13.6 19.1 20.9 ZebraLogic 12.8 17.8 12.0 19.7 35.2 26.7 AutoLogi 30.9 58.9 42.9 57.4 76.3 76.5 Agent &Coding BFCL v3 49.6 50.6 55.8 58.7 57.6 60.2 LiveCodeBench v5 10.8 25.7 14.4 21.9 21.3 22.8 CodeForces (Rating / Percentile)473 / 14.9%462 / 14.7%191 / 0.0%904 / 38.3%842 / 33.7%1110 / 52.4%Multilingual Tasks Multi-IF 52.1 65.6 47.7 55.5 61.3 69.2 INCLUDE 34.0 65.3 53.6 63.5 53.8 62.5 MMMLU 14 languages 44.4 70.0 61.4 70.3 61.7 66.9 MT-AIME2024 0.4 16.7 5.5 8.5 13.9 16.6 PolyMath 5.8 17.6 11.9 15.0 16.6 18.8 MLogiQA 41.9 54.5 49.5 51.3 49.9 51.4 + +Table 19: Comparison among Qwen3-1.7B / Qwen3-0.6B (Thinking) and other reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +DeepSeek-R1-Distill-Qwen-1.5B DeepSeek-R1-Distill-Llama-8B Qwen3-0.6B Qwen3-1.7B Architecture Dense Dense Dense Dense# Activated Params 1.5B 8B 0.6B 1.7B# Total Params 1.5B 8B 0.6B 1.7B General Tasks MMLU-Redux 45.4 66.4 55.6 73.9 GPQA-Diamond 33.8 49.0 27.9 40.1 C-Eval 27.1 50.4 50.4 68.1 LiveBench 2024-11-25 24.9 40.6 30.3 51.1 Alignment Tasks IFEval strict prompt 39.9 59.0 59.2 72.5 Arena-Hard 4.5 17.6 8.5 43.1 AlignBench v1.1 5.00 6.24 6.10 7.60 Creative Writing v3 16.4 51.1 30.6 48.0 WritingBench 4.03 5.42 5.61 7.02 Math & Text Reasoning MATH-500 83.9 89.1 77.6 93.4 AIME’24 28.9 50.4 10.7 48.3 AIME’25 22.8 27.8 15.1 36.8 ZebraLogic 4.9 37.1 30.3 63.2 AutoLogi 19.1 63.4 61.6 83.2 Agent &Coding BFCL v3 14.0 21.5 46.4 56.6 LiveCodeBench v5 13.2 42.5 12.3 33.2 Multilingual Tasks Multi-IF 13.3 27.0 36.1 51.2 INCLUDE 21.9 34.5 35.9 51.8 MMMLU 14 languages 27.3 40.1 43.1 59.1 MT-AIME2024 12.4 13.2 7.8 36.1 PolyMath 14.5 10.8 11.4 25.2 MLogiQA 29.0 32.8 40.9 56.0 + +Table 20: Comparison among Qwen3-1.7B / Qwen3-0.6B (Non-thinking) and other non-reasoning baselines. The highest and second-best scores are shown in bold and underlined, respectively. + +Gemma-3-1B-IT Phi-4-mini Qwen2.5-1.5B-Instruct Qwen2.5-3B-Instruct Qwen3-0.6B Qwen3-1.7B Architecture Dense Dense Dense Dense Dense Dense# Activated Params 1.0B 3.8B 1.5B 3.1B 0.6B 1.7B# Total Params 1.0B 3.8B 1.5B 3.1B 0.6B 1.7B General Tasks MMLU-Redux 33.3 67.9 50.7 64.4 44.6 64.4 GPQA-Diamond 19.2 25.2 29.8 30.3 22.9 28.6 C-Eval 28.5 40.0 53.3 68.2 42.6 61.0 LiveBench 2024-11-25 14.4 25.3 18.0 23.8 21.8 35.6 Alignment Tasks IFEval strict prompt 54.5 68.6 42.5 58.2 54.5 68.2 Arena-Hard 17.8 32.8 9.0 23.7 6.5 36.9 AlignBench v1.1 5.3 6.00 5.60 6.49 5.60 7.20 Creative Writing v3 52.8 10.3 31.5 42.8 28.4 43.6 WritingBench 5.18 4.05 4.67 5.55 5.13 6.54 Math & Text Reasoning MATH-500 46.4 67.6 55.0 67.2 55.2 73.0 AIME’24 0.9 8.1 0.9 6.7 3.4 13.4 AIME’25 0.8 5.3 0.4 4.2 2.6 9.8 ZebraLogic 1.9 2.7 3.4 4.8 4.2 12.8 AutoLogi 16.4 28.8 22.5 29.9 37.4 59.8 Agent &Coding BFCL v3 16.3 31.3 47.8 50.4 44.1 52.2 LiveCodeBench v5 1.8 10.4 5.3 9.2 3.6 11.6 Multilingual Tasks Multi-IF 32.8 40.5 20.2 32.3 33.3 44.7 INCLUDE 32.7 43.8 33.1 43.8 34.4 42.6 MMMLU 14 languages 32.5 51.4 40.4 51.8 37.1 48.3 MT-AIME2024 0.2 0.9 0.7 1.6 1.5 4.9 PolyMath 3.5 6.7 5.0 7.3 4.6 10.3 MLogiQA 31.8 39.5 40.9 39.5 37.3 41.1 + +##### Qwen3-30B-A3B & Qwen3-14B + +For Qwen3-30B-A3B and Qwen3-14B, we compare them with DeepSeek-R1-Distill-Qwen-32B and QwQ-32B in the thinking mode, and Phi-4 ([Abdin et al. 2024](https://arxiv.org/html/2505.09388#bib.bib1)), Gemma-3-27B-IT ([Team et al. 2025](https://arxiv.org/html/2505.09388#bib.bib59)), and Qwen2.5-32B-Instruct in the non-thinking mode, respectively. We present the evaluation results in Table [15](https://arxiv.org/html/2505.09388#S4.T15 "Table 15 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report") and [16](https://arxiv.org/html/2505.09388#S4.T16 "Table 16 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"). + +1. (1) +From Table [15](https://arxiv.org/html/2505.09388#S4.T15 "Table 15 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"), Qwen3-30B-A3B and Qwen3-14B (Thinking) are both highly competitive to QwQ-32B, especially on the reasoning-related benchmarks. It is noteworthy that Qwen3-30B-A3B achieves comparable performance to QwQ-32B with a smaller model size and less than 1/10 activated parameters, demonstrating the effectiveness of our Strong-to-Weak Distillation approach in endowing lightweight models with profound reasoning capabilities. + +2. (2) +From Table [16](https://arxiv.org/html/2505.09388#S4.T16 "Table 16 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"), Qwen3-30B-A3B and Qwen3-14B (Non-thinking) surpass the non-reasoning baselines in most of the benchmarks. They exceed our previous Qwen2.5-32B-Instruct model with significantly fewer activated and total parameters, allowing for more efficient and cost-effective performance. + +##### Qwen3-8B / 4B / 1.7B / 0.6B + +For Qwen3-8B and Qwen3-4B, we compare them with DeepSeek-R1-Distill-Qwen-14B and DeepSeek-R1-Distill-Qwen-32B in the thinking mode, and LLaMA-3.1-8B-Instruct ([Dubey et al. 2024](https://arxiv.org/html/2505.09388#bib.bib19)), Gemma-3-12B-IT ([Team et al. 2025](https://arxiv.org/html/2505.09388#bib.bib59)), Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct in the non-thinking mode, respectively. For Qwen3-1.7B and Qwen3-0.6B, we compare them with DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Llama-8B in the thinking mode, and Gemma-3-1B-IT, Phi-4-mini, Qwen2.5-1.5B-Instruct, and Qwen2.5-3B-Instruct in the non-thinking mode, respectively. We present the evaluation results of Qwen3-8B and Qwen3-4B in Table [17](https://arxiv.org/html/2505.09388#S4.T17 "Table 17 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report") and [18](https://arxiv.org/html/2505.09388#S4.T18 "Table 18 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report") and those of Qwen3-1.7B and Qwen3-0.6B in Table [19](https://arxiv.org/html/2505.09388#S4.T19 "Table 19 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report") and [20](https://arxiv.org/html/2505.09388#S4.T20 "Table 20 ‣ Qwen3-32B ‣ 4.6 Post-training Evaluation ‣ 4 Post-training ‣ Qwen3 Technical Report"), respectively. Overall, these edge-side models exhibit impressive performance and outperform baselines even with more parameters, including our previous Qwen2.5 models, in either the thinking or the non-thinking mode. These results, once again, demonstrate the efficacy of our Strong-to-Weak Distillation approach, making it possible for us to build the lightweight Qwen3 models with remarkably reduced costs and efforts. + +### 4.7 Discussion + +##### The Effectiveness of Thinking Budget + +To verify that Qwen3 can enhance its intelligence level by leveraging an increased thinking budget, we adjust the allocated thinking budget on four benchmarks across Mathematics, Coding, and STEM domains. The resulting scaling curves are presented in Figure [2](https://arxiv.org/html/2505.09388#S4.F2 "Figure 2 ‣ The Effectiveness of Thinking Budget ‣ 4.7 Discussion ‣ 4 Post-training ‣ Qwen3 Technical Report"), Qwen3 demonstrates scalable and smooth performance improvements correlated to the allocated thinking budget. Moreover, we observe that if we further extend the output length beyond 32K, the model’s performance is expected to improve further in the future. We leave this exploration as future work. + +Figure 2: Performance of Qwen3-235B-A22B with respect to the thinking budget. + +##### The Effectiveness and Efficiency of On-Policy Distillation + +We evaluate the effectiveness and efficiency of on-policy distillation by comparing the performance and computational cost—measured in GPU hours—after undergoing distillation versus direct reinforcement learning, both starting from the same off-policy distilled 8B checkpoint. For simplicity, we focus solely on math and code-related queries in this comparison. The results, summarized in Table [21](https://arxiv.org/html/2505.09388#S4.T21 "Table 21 ‣ The Effectiveness and Efficiency of On-Policy Distillation ‣ 4.7 Discussion ‣ 4 Post-training ‣ Qwen3 Technical Report"), show that distillation achieves significantly better performance than reinforcement learning while requiring approximately only 1/10 of the GPU hours. Furthermore, distillation from teacher logits enables the student model to expand its exploration space and enhance its reasoning potential, as evidenced by the improved pass@64 scores on the AIME’24 and AIME’25 benchmarks after distillation, compared to the initial checkpoint. In contrast, reinforcement learning does not lead to any improvement in pass@64 scores. These observations highlight the advantages of leveraging a stronger teacher model in guiding student model learning. + +Table 21: Comparison of reinforcement learning and on-policy distillation on Qwen3-8B. Numbers in parentheses indicate pass@64 scores. + +Method AIME’24 AIME’25 MATH500 LiveCodeBench v5 MMLU -Redux GPQA -Diamond GPU Hours +Off-policy Distillation 55.0 (90.0)42.8 (83.3)92.4 42.0 86.4 55.6- ++ Reinforcement Learning 67.6 (90.0)55.5 (83.3)94.8 52.9 86.9 61.3 17,920 ++ On-policy Distillation 74.4 (93.3)65.5 (86.7)97.0 60.3 88.3 63.3 1,800 + +##### The Effects of Thinking Mode Fusion and General RL + +To evaluate the effectiveness of Thinking Mode Fusion and General Reinforcement Learning (RL) during the post-training, we conduct evaluations on various stages of the Qwen-32B model. In addition to the datasets mentioned earlier, we introduce several in-house benchmarks to monitor other capabilities. These benchmarks include: + +* • +CounterFactQA: Contains counterfactual questions where the model needs to identify that the questions are not factual and avoid generating hallucinatory answers. + +* • +LengthCtrl: Includes creative writing tasks with length requirements; the final score is based on the difference between the generated content length and the target length. + +* • +ThinkFollow: Involves multi-turn dialogues with randomly inserted /think and /no_think flags to test whether the model can correctly switch thinking modes based on user queries. + +* • +ToolUse: Evaluates the stability of the model in single-turn, multi-turn, and multi-step tool calling processes. The score includes accuracy in intent recognition, format accuracy, and parameter accuracy during the tool calling process. + +Table 22: Performance of Qwen3-32B after Reasoning RL (Stage 2), Thinking Mode Fusion (Stage 3), and General RL (Stage 4). Benchmarks with * are in-house datasets. + +Stage 2 Reasoning RL Stage 3 Thinking Mode Fusion Stage 4 General RL +Benchmark Thinking Thinking Non-Thinking Thinking Non-Thinking +General Tasks LiveBench 2024-11-25 68.6 70.9+2.3 57.1 74.9+4.0 59.8+2.8 +Arena-Hard 86.8 89.4+2.6 88.5 93.8+4.4 92.8+4.3 +CounterFactQA*50.4 61.3+10.9 64.3 68.1+6.8 66.4+2.1 +Instruction& Format Following IFEval strict prompt 73.0 78.4+5.4 78.4 85.0+6.6 83.2+4.8 +Multi-IF 61.4 64.6+3.2 65.2 73.0+8.4 70.7+5.5 +LengthCtrl*62.6 70.6+8.0 84.9 73.5+2.9 87.3+2.4 +ThinkFollow*-88.7 98.9+10.2 +Agent BFCL v3 69.0 68.4-0.6 61.5 70.3+1.9 63.0+1.5 +ToolUse*63.3 70.4+7.1 73.2 85.5+15.1 86.5+13.3 +Knowledge &STEM MMLU-Redux 91.4 91.0-0.4 86.7 90.9-0.1 85.7-1.0 +GPQA-Diamond 68.8 69.0+0.2 50.4 68.4-0.6 54.6+4.3 +Math &Coding AIME’24 83.8 81.9-1.9 28.5 81.4-0.5 31.0+2.5 +LiveCodeBench v5 68.4 67.2-1.2 31.1 65.7-1.5 31.3+0.2 + +The results are shown in Table [22](https://arxiv.org/html/2505.09388#S4.T22 "Table 22 ‣ The Effects of Thinking Mode Fusion and General RL ‣ 4.7 Discussion ‣ 4 Post-training ‣ Qwen3 Technical Report"), where we can draw the following conclusions: + +1. (1) +Stage 3 integrates the non-thinking mode into the model, which already possesses thinking capabilities after the first two stages of training. The ThinkFollow benchmark score of 88.7 indicates that the model has developed an initial ability to switch between modes, though it still occasionally makes errors. Stage 3 also enhances the model’s general and instruction-following capabilities in thinking mode, with CounterFactQA improving by 10.9 points and LengthCtrl by 8.0 points. + +2. (2) +Stage 4 further strengthens the model’s general, instruction-following, and agent capabilities in both thinking and non-thinking modes. Notably, the ThinkFollow score improves to 98.9, ensuring accurate mode switching. + +3. (3) +For Knowledge, STEM, Math, and Coding tasks, Thinking Mode Fusion and General RL do not bring significant improvements. In contrast, for challenging tasks like AIME’24 and LiveCodeBench, the performance in thinking mode actually decreases after these two training stages. We conjecture this degradation is due to the model being trained on a broader range of general tasks, which may compromise its specialized capabilities in handling complex problems. During the development of Qwen3, we choose to accept this performance trade-off to enhance the model’s overall versatility. + +## 5 Conclusion + +In this technical report, we introduce Qwen3, the latest version of the Qwen series. Qwen3 features both thinking mode and non-thinking mode, allowing users to dynamically manage the number of tokens used for complex thinking tasks. The model was pre-trained on an extensive dataset containing 36 trillion tokens, enabling it to understand and generate text in 119 languages and dialects. Through a series of comprehensive evaluations, Qwen3 has shown strong performance across a range of standard benchmarks for both pre-trained and post-trained models, including tasks related to code generation, mathematics, reasoning, and agents. + +In the near future, our research will focus on several key areas. We will continue to scale up pretraining by using data that is both higher in quality and more diverse in content. At the same time, we will work on improving model architecture and training methods for the purposes of effective compression, scaling to extremely long contexts, etc. In addition, we plan to increase computational resources for reinforcement learning, with a particular emphasis on agent-based RL systems that learn from environmental feedback. This will allow us to build agents capable of tackling complex tasks that require inference time scaling. + +## 6 Authors + +Core Contributors: An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu + +Contributors: Bei Chen, Biao Sun, Bin Luo, Bin Zhang, Binghai Wang, Bowen Ping, Boyi Deng, Chang Si, Chaojie Yang, Chen Cheng, Chenfei Wu, Chengpeng Li, Chengyuan Li, Fan Hong, Guobin Zhao, Hang Zhang, Hangrui Hu, Hanyu Zhao, Hao Lin, Hao Xiang, Haoyan Huang, Hongkun Hao, Humen Zhong, Jialin Wang, Jiandong Jiang, Jianqiang Wan, Jianyuan Zeng, Jiawei Chen, Jie Zhang, Jin Xu, Jinkai Wang, Jinyang Zhang, Jinzheng He, Jun Tang, Kai Zhang, Ke Yi, Keming Lu, Keqin Chen, Langshi Chen, Le Jiang, Lei Zhang, Linjuan Wu, Man Yuan, Mingkun Yang, Minmin Sun, Mouxiang Chen, Na Ni, Nuo Chen, Peng Liu, Peng Wang, Peng Zhu, Pengcheng Zhang, Pengfei Wang, Qiaoyu Tang, Qing Fu, Qiuyue Wang, Rong Zhang, Rui Hu, Runji Lin, Shen Huang, Shuai Bai, Shutong Jiang, Sibo Song, Siqi Zhang, Song Chen, Tao He, Ting He, Tingfeng Hui, Wei Ding, Wei Liao, Wei Lin, Wei Zhang, Weijia Xu, Wenbin Ge, Wenmeng Zhou, Wenyuan Yu, Xianyan Jia, Xianzhong Shi, Xiaodong Deng, Xiaoming Huang, Xiaoyuan Li, Ximing Zhou, Xinyao Niu, Xipin Wei, Xuejing Liu, Yang Liu, Yang Yao, Yang Zhang, Yanpeng Li, Yantao Liu, Yidan Zhang, Yikai Zhu, Yiming Wang, Yiwen Hu, Yong Jiang, Yong Li, Yongan Yue, Yu Guan, Yuanzhi Zhu, Yunfei Chu, Yunlong Feng, Yuxin Zhou, Yuxuan Cai, Zeyao Ma, Zhaohai Li, Zheng Li, Zhengyang Tang, Zheren Fu, Zhi Li, Zhibo Yang, Zhifang Guo, Zhipeng Zhang, Zhiying Xu, Zhiyu Yin, Zhongshen Zeng, Zile Qiao, Ziye Meng, Zongmeng Zhang + +## Appendix A Appendix + +### A.1 Additional Evaluation Results + +#### A.1.1 Long-Context Ability + +Table 23: Performance of Qwen3 Models on the RULER benchmark. + +Model RULER +Avg.4K 8K 16K 32K 64K 128K +Qwen2.5-7B-Instruct 85.4 96.7 95.1 93.7 89.4 82.3 55.1 +Qwen2.5-14B-Instruct 91.4 97.7 96.8 95.9 93.4 86.7 78.1 +Qwen2.5-32B-Instruct 92.9 96.9 97.1 95.5 95.5 90.3 82.0 +Qwen2.5-72B-Instruct 95.1 97.7 97.2 97.7 96.5 93.0 88.4 +Non-thinking Mode Qwen3-4B 85.2 95.1 93.6 91.0 87.8 77.8 66.0 +Qwen3-8B 89.1 96.3 96.0 91.8 91.2 82.1 77.4 +Qwen3-14B 94.6 98.0 97.8 96.4 96.1 94.0 85.1 +Qwen3-32B 93.7 98.4 96.0 96.2 94.4 91.8 85.6 +Qwen3-30B-A3B 91.6 96.5 97.0 95.3 92.4 89.1 79.2 +Qwen3-235B-A22B 95.0 97.7 97.2 96.4 95.1 93.3 90.6 +Thinking Mode Qwen3-4B 83.5 92.7 88.7 86.5 83.2 83.0 67.2 +Qwen3-8B 84.4 94.7 94.4 86.1 80.8 78.3 72.0 +Qwen3-14B 90.1 95.4 93.6 89.8 91.9 90.6 79.0 +Qwen3-32B 91.0 94.7 93.7 91.6 92.5 90.0 83.5 +Qwen3-30B-A3B 86.6 94.1 92.7 89.0 86.6 82.1 75.0 +Qwen3-235B-A22B 92.2 95.1 94.8 93.0 92.3 92.0 86.0 + +For evaluating long-context processing capabilities, we report the results on the RULER benchmark ([Hsieh et al. 2024](https://arxiv.org/html/2505.09388#bib.bib27)) in Table [23](https://arxiv.org/html/2505.09388#A1.T23 "Table 23 ‣ A.1.1 Long-Context Ability ‣ A.1 Additional Evaluation Results ‣ Appendix A Appendix ‣ Qwen3 Technical Report"). To enable length extrapolation, we utilize YARN ([Peng et al. 2023](https://arxiv.org/html/2505.09388#bib.bib46)) with a scaling_factor=4. In thinking mode, we set the thinking budget to 8192 tokens to mitigate overly verbose reasoning on the extremely long inputs. + +The results show that: + +1. 1. +In non-thinking mode, Qwen3 outperforms Qwen2.5 models of a similar size in long-context processing tasks. + +2. 2. +In thinking mode, the model’s performance slightly degrades. We hypothesize that the thinking content does not provide significant benefits for these retrieval tasks, which do not rely on reasoning and may instead interfere with the retrieval process. We are committed to enhancing the long-context capability in the thinking mode in future versions. + +#### A.1.2 Multilingual Ability + +Table [24](https://arxiv.org/html/2505.09388#A1.T24 "Table 24 ‣ A.1.2 Multilingual Ability ‣ A.1 Additional Evaluation Results ‣ Appendix A Appendix ‣ Qwen3 Technical Report")-[35](https://arxiv.org/html/2505.09388#A1.T35 "Table 35 ‣ A.1.2 Multilingual Ability ‣ A.1 Additional Evaluation Results ‣ Appendix A Appendix ‣ Qwen3 Technical Report") presents the detailed benchmark scores across various languages, including Spanish, French, Portuguese, Italian, Arabic, Japanese, Korean, Indonesian, Russian, Vietnamese, German, and Thai. The results of these tables demonstrate that the Qwen3 series models achieve competitive performance across all evaluated benchmarks, showcasing their strong multilingual capabilities. + +To evaluate the performance of Qwen3 across a broader range of languages, we utilize Belebele ([Bandarkar et al. 2023](https://arxiv.org/html/2505.09388#bib.bib9)), a benchmark for natural language understanding. We conduct evaluations on 80 supported languages from the benchmark, excluding 42 unoptimized languages, as shown in Table [36](https://arxiv.org/html/2505.09388#A1.T36 "Table 36 ‣ A.1.2 Multilingual Ability ‣ A.1 Additional Evaluation Results ‣ Appendix A Appendix ‣ Qwen3 Technical Report") (organized by language family). The performance comparison between Qwen3 and other baseline models on the Belebele benchmark is presented in Table [37](https://arxiv.org/html/2505.09388#A1.T37 "Table 37 ‣ A.1.2 Multilingual Ability ‣ A.1 Additional Evaluation Results ‣ Appendix A Appendix ‣ Qwen3 Technical Report"). The results show that Qwen3 achieves comparable performance to similarly-sized Gemma models while outperforming Qwen2.5 significantly. + +Table 24: Benchmark scores for language: Spanish (es). The highest and second-best scores are shown in bold and underlined, respectively. + +Model Multi-IF MLogiQA INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 80.1 70.0 96.4 88.7 90.0 54.4 79.9 QwQ-32B 70.0 75.0 81.8 84.5 76.7 52.2 73.4 Qwen3-235B-A22B 74.2 76.2 89.1 86.7 86.7 57.3 78.4 Qwen3-32B 74.7 68.8 90.9 82.8 76.7 51.8 74.3 Qwen3-30B-A3B 74.9 71.2 80.0 81.9 76.7 48.5 72.2 Qwen3-14B 76.2 67.5 83.6 81.1 73.3 50.3 72.0 Qwen3-8B 74.1 70.0 78.2 79.2 70.0 43.7 69.2 Qwen3-4B 69.1 68.8 72.7 75.7 66.7 42.3 65.9 Qwen3-1.7B 56.0 55.0 72.7 64.5 46.7 30.2 54.2 Qwen3-0.6B 39.2 42.5 54.5 48.8 13.3 14.3 35.4 Non-thinking Mode GPT-4o-2024-1120 67.5 52.5 89.1 80.6 10.0 15.5 52.5 Gemma-3-27b-IT 73.5 57.5 89.1 77.7 30.0 22.4 58.4 Qwen2.5-72B-Instruct 66.7 61.3 80.0 80.1 20.0 18.8 54.5 Qwen3-235B-A22B 71.7 66.2 83.6 83.7 33.3 29.5 61.3 Qwen3-32B 72.1 65.0 83.6 80.4 26.7 24.7 58.8 Qwen3-30B-A3B 72.1 53.8 85.5 78.3 33.3 25.0 58.0 Qwen3-14B 76.2 63.7 78.2 77.4 40.0 25.0 60.1 Qwen3-8B 73.1 50.0 80.0 73.7 16.7 21.3 52.5 Qwen3-4B 65.8 50.0 60.0 68.3 13.3 17.3 45.8 Qwen3-1.7B 47.9 43.8 50.9 54.3 10.0 11.6 36.4 Qwen3-0.6B 35.5 37.5 43.6 39.5 3.3 5.8 27.5 + +Table 25: Benchmark scores for language: French (fr). The highest and second-best scores are shown in bold and underlined, respectively. + +Model Multi-IF MLogiQA INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 80.5 73.8 85.7 88.3 80.0 52.8 76.8 QwQ-32B 72.4 78.8 76.2 84.0 80.0 49.4 73.5 Qwen3-235B-A22B 77.3 78.8 85.7 86.6 86.7 57.4 78.8 Qwen3-32B 76.7 81.2 76.2 82.1 83.3 47.1 74.4 Qwen3-30B-A3B 75.2 67.5 83.3 81.0 76.7 46.9 71.8 Qwen3-14B 77.6 71.2 73.8 80.4 73.3 44.2 70.1 Qwen3-8B 73.8 66.2 85.7 77.9 70.0 45.3 69.8 Qwen3-4B 71.3 63.7 71.4 74.5 66.7 40.2 64.6 Qwen3-1.7B 52.6 56.2 54.8 64.8 60.0 28.7 52.8 Qwen3-0.6B 36.1 48.8 47.6 48.4 6.7 14.0 33.6 Non-thinking Mode GPT-4o-2024-1120 67.8 56.2 85.7 81.8 10.0 15.3 52.8 Gemma-3-27b-IT 73.9 57.5 73.8 78.3 23.3 21.5 54.7 Qwen2.5-72B-Instruct 72.1 55.0 81.0 80.2 26.7 15.7 55.1 Qwen3-235B-A22B 73.2 65.0 88.1 81.1 36.7 28.1 62.0 Qwen3-32B 75.8 60.0 73.8 79.5 30.0 23.0 57.0 Qwen3-30B-A3B 75.6 52.5 69.0 77.9 26.7 27.3 54.8 Qwen3-14B 78.4 63.7 73.8 75.1 33.3 24.4 58.1 Qwen3-8B 71.9 52.5 71.4 71.7 20.0 21.4 51.5 Qwen3-4B 64.2 47.5 61.9 67.6 20.0 19.2 46.7 Qwen3-1.7B 46.1 43.8 64.3 53.2 3.3 11.6 37.0 Qwen3-0.6B 32.8 35.0 38.1 39.4 6.7 4.6 26.1 + +Table 26: Benchmark scores for language: Portuguese (pt). The highest and second-best scores are shown in bold and underlined, respectively. + +Model Multi-IF MLogiQA INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 80.5 73.8 83.9 88.9 73.3 52.2 75.4 QwQ-32B 70.5 70.0 80.4 84.0 80.0 48.7 72.3 Qwen3-235B-A22B 73.6 78.8 78.6 86.2 86.7 58.3 77.0 Qwen3-32B 74.1 76.2 76.8 82.6 80.0 52.4 73.7 Qwen3-30B-A3B 76.1 71.2 71.4 81.0 76.7 49.3 71.0 Qwen3-14B 77.3 68.8 75.0 81.6 83.3 46.7 72.1 Qwen3-8B 73.9 67.5 75.0 78.6 56.7 44.8 66.1 Qwen3-4B 70.6 62.5 71.4 75.1 73.3 44.2 66.2 Qwen3-1.7B 55.6 60.0 53.6 64.6 46.7 28.2 51.4 Qwen3-0.6B 38.7 33.8 42.9 47.5 10.0 12.7 30.9 Non-thinking Mode GPT-4o-2024-1120 66.8 57.5 78.6 80.7 10.0 15.0 51.4 Gemma-3-27b-IT 72.9 55.0 75.0 77.1 33.3 20.9 55.7 Qwen2.5-72B-Instruct 68.8 55.0 71.4 82.2 23.3 11.3 52.0 Qwen3-235B-A22B 72.5 67.5 82.1 83.5 33.3 28.3 61.2 Qwen3-32B 71.1 61.3 73.2 80.6 30.0 23.9 56.7 Qwen3-30B-A3B 72.3 47.5 67.9 77.8 26.7 24.0 52.7 Qwen3-14B 75.5 58.8 75.0 76.5 26.7 25.8 56.4 Qwen3-8B 71.9 56.2 71.4 72.9 20.0 19.7 52.0 Qwen3-4B 66.1 50.0 73.2 66.7 10.0 18.1 47.4 Qwen3-1.7B 49.5 33.8 39.3 52.9 6.7 12.8 32.5 Qwen3-0.6B 36.6 37.5 42.9 37.5 3.3 5.7 27.2 + +Table 27: Benchmark scores for language: Italian (it). The highest and second-best scores are shown in bold and underlined, respectively. + +Model Multi-IF INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 80.9 100.0 87.2 90.0 54.1 82.4 QwQ-32B 71.2 96.4 84.9 76.7 49.3 75.7 Qwen3-235B-A22B 73.7 96.4 85.7 80.0 57.4 78.6 Qwen3-32B 76.6 90.9 81.6 80.0 49.7 75.8 Qwen3-30B-A3B 75.9 94.5 81.9 80.0 48.1 76.1 Qwen3-14B 79.0 94.5 80.2 70.0 47.0 74.1 Qwen3-8B 74.6 89.1 77.5 76.7 46.1 72.8 Qwen3-4B 69.8 83.6 74.4 76.7 44.5 69.8 Qwen3-1.7B 54.6 74.5 64.2 53.3 29.6 55.2 Qwen3-0.6B 37.8 45.5 45.9 6.7 13.3 29.8 Non-thinking Mode GPT-4o-2024-1120 67.6 98.2 80.7 13.3 15.2 55.0 Gemma-3-27b-IT 74.6 90.9 78.4 23.3 20.5 57.5 Qwen2.5-72B-Instruct 67.2 94.5 80.7 16.7 16.7 55.2 Qwen3-235B-A22B 72.9 92.7 82.6 33.3 28.6 62.0 Qwen3-32B 71.4 92.7 79.5 30.0 23.0 59.3 Qwen3-30B-A3B 73.9 87.3 77.7 33.3 24.8 59.4 Qwen3-14B 75.8 89.1 75.7 26.7 27.6 59.0 Qwen3-8B 72.1 85.5 72.9 13.3 23.8 53.5 Qwen3-4B 63.0 78.2 67.8 23.3 19.3 50.3 Qwen3-1.7B 46.1 70.9 53.4 6.7 11.9 37.8 Qwen3-0.6B 35.1 43.6 39.0 0.0 4.5 24.4 + +Table 28: Benchmark scores for language: Arabic (ar). The highest and second-best scores are shown in bold and underlined, respectively. + +Model MLogiQA INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 75.0 89.3 87.8 76.7 52.6 76.3 QwQ-32B 75.0 67.9 81.8 80.0 41.3 69.2 Qwen3-235B-A22B 80.0 71.4 83.6 76.7 53.7 73.1 Qwen3-32B 66.2 73.2 80.1 86.7 47.0 70.6 Qwen3-30B-A3B 66.2 66.1 77.2 83.3 47.3 68.0 Qwen3-14B 71.2 67.9 77.4 83.3 46.6 69.3 Qwen3-8B 65.0 67.9 74.4 76.7 44.9 65.8 Qwen3-4B 62.5 55.4 67.7 66.7 41.2 58.7 Qwen3-1.7B 55.0 44.6 53.2 36.7 25.8 43.1 Qwen3-0.6B 40.0 41.1 38.9 10.0 11.7 28.3 Non-thinking Mode GPT-4o-2024-1120 51.2 78.6 80.9 13.3 12.9 47.4 Gemma-3-27b-IT 56.2 62.5 74.4 26.7 22.8 48.5 Qwen2.5-72B-Instruct 56.2 66.1 77.2 6.7 14.7 44.2 Qwen3-235B-A22B 66.2 67.9 79.5 40.0 28.2 56.4 Qwen3-32B 55.0 69.6 75.7 23.3 25.4 49.8 Qwen3-30B-A3B 48.8 64.3 71.6 30.0 22.6 47.5 Qwen3-14B 52.5 60.7 69.5 23.3 23.5 45.9 Qwen3-8B 45.0 58.9 64.6 13.3 16.4 39.6 Qwen3-4B 52.5 42.9 56.7 13.3 15.3 36.1 Qwen3-1.7B 31.2 37.5 43.6 3.3 9.4 25.0 Qwen3-0.6B 40.0 39.3 35.4 0.0 3.8 23.7 + +Table 29: Benchmark scores for language: Japanese (ja). The highest and second-best scores are shown in bold and underlined, respectively. + +Model MLogiQA INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 72.5 74.5 83.8 83.3 55.4 73.9 QwQ-32B 73.8 86.3 82.3 53.3 39.9 67.1 Qwen3-235B-A22B 75.0 94.1 84.8 73.3 52.7 76.0 Qwen3-32B 70.0 90.2 80.2 76.7 47.7 73.0 Qwen3-30B-A3B 66.2 88.2 79.9 73.3 47.4 71.0 Qwen3-14B 68.8 88.2 79.4 66.7 45.7 69.8 Qwen3-8B 71.2 86.3 74.9 73.3 44.7 70.1 Qwen3-4B 63.7 80.4 72.5 53.3 40.7 62.1 Qwen3-1.7B 53.8 74.5 61.8 36.7 28.5 51.1 Qwen3-0.6B 47.5 47.1 45.1 13.3 14.5 33.5 Non-thinking Mode GPT-4o-2024-1120 60.0 92.2 81.9 10.0 12.5 51.3 Gemma-3-27b-IT 66.2 86.3 76.5 20.0 17.3 53.3 Qwen2.5-72B-Instruct 55.0 94.1 77.7 16.7 17.7 52.2 Qwen3-235B-A22B 67.5 92.2 80.9 26.7 26.9 58.8 Qwen3-32B 58.8 92.2 78.0 20.0 20.5 53.9 Qwen3-30B-A3B 51.2 82.4 74.9 30.0 20.6 51.8 Qwen3-14B 55.0 84.3 73.8 33.3 19.8 53.2 Qwen3-8B 47.5 82.4 69.9 20.0 18.5 47.7 Qwen3-4B 46.2 76.5 64.8 13.3 15.1 43.2 Qwen3-1.7B 40.0 68.6 46.3 3.3 11.6 34.0 Qwen3-0.6B 37.5 37.3 37.9 3.3 3.7 23.9 + +Table 30: Benchmark scores for language: Korean (ko). The highest and second-best scores are shown in bold and underlined, respectively. + +Model MLogiQA INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 75.0 88.0 85.9 76.7 50.0 75.1 QwQ-32B 76.2 72.0 81.8 60.0 40.0 66.0 Qwen3-235B-A22B 71.2 80.0 84.7 80.0 55.7 74.3 Qwen3-32B 71.2 74.0 79.2 80.0 48.5 70.6 Qwen3-30B-A3B 68.8 72.0 78.6 76.7 46.6 68.5 Qwen3-14B 67.5 74.0 79.6 76.7 46.0 68.8 Qwen3-8B 60.0 80.0 74.7 76.7 42.3 66.7 Qwen3-4B 66.2 74.0 68.8 70.0 40.6 63.9 Qwen3-1.7B 53.8 66.0 57.8 43.3 25.2 49.2 Qwen3-0.6B 33.8 52.0 41.5 13.3 11.8 30.5 Non-thinking Mode GPT-4o-2024-1120 63.7 80.0 80.5 13.3 12.9 50.1 Gemma-3-27b-IT 58.8 76.0 75.9 20.0 18.3 49.8 Qwen2.5-72B-Instruct 58.8 68.0 76.7 6.7 17.7 45.6 Qwen3-235B-A22B 63.7 76.0 79.8 33.3 27.9 56.1 Qwen3-32B 60.0 74.0 77.2 26.7 21.2 51.8 Qwen3-30B-A3B 52.5 72.0 72.5 16.7 20.7 46.9 Qwen3-14B 52.5 68.0 73.3 20.0 18.7 46.5 Qwen3-8B 52.5 76.0 66.5 23.3 16.3 46.9 Qwen3-4B 46.2 74.0 59.9 13.3 16.6 42.0 Qwen3-1.7B 48.8 58.0 46.0 6.7 9.0 33.7 Qwen3-0.6B 40.0 52.0 36.9 0.0 5.5 26.9 + +Table 31: Benchmark scores for language: Indonesian (id). The highest and second-best scores are shown in bold and underlined, respectively. + +Model INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 80.0 86.3 83.3 51.3 75.2 QwQ-32B 76.4 83.7 73.3 47.3 70.2 Qwen3-235B-A22B 80.0 87.2 80.0 53.5 75.2 Qwen3-32B 80.0 82.0 76.7 45.6 71.1 Qwen3-30B-A3B 81.8 80.4 80.0 44.9 71.8 Qwen3-14B 78.2 79.6 70.0 45.3 68.3 Qwen3-8B 72.7 77.7 70.0 43.8 66.0 Qwen3-4B 70.9 72.3 66.7 41.2 62.8 Qwen3-1.7B 63.6 61.2 36.7 26.8 47.1 Qwen3-0.6B 36.4 46.6 10.0 12.6 26.4 Non-thinking Mode GPT-4o-2024-1120 80.0 81.1 10.0 14.7 46.4 Gemma-3-27b-IT 76.4 75.9 13.3 22.6 47.0 Qwen2.5-72B-Instruct 74.5 78.8 10.0 16.6 45.0 Qwen3-235B-A22B 81.8 81.9 33.3 27.5 56.1 Qwen3-32B 81.8 77.2 23.3 24.3 51.6 Qwen3-30B-A3B 70.9 76.4 30.0 25.9 50.8 Qwen3-14B 70.9 74.1 26.7 24.6 49.1 Qwen3-8B 78.2 69.6 20.0 21.6 47.4 Qwen3-4B 67.3 66.5 13.3 19.0 41.5 Qwen3-1.7B 52.7 49.0 3.3 10.8 29.0 Qwen3-0.6B 52.7 40.0 3.3 5.1 25.3 + +Table 32: Benchmark scores for language: Russian (ru). The highest and second-best scores are shown in bold and underlined, respectively. + +Model Multi-IF INCLUDE MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 68.1 80.4 70.0 52.3 67.7 QwQ-32B 61.2 73.2 76.7 43.6 63.7 Qwen3-235B-A22B 62.2 80.4 80.0 53.1 68.9 Qwen3-32B 62.5 73.2 63.3 46.5 61.4 Qwen3-30B-A3B 60.7 76.8 73.3 45.4 64.0 Qwen3-14B 63.6 80.4 66.7 46.4 64.3 Qwen3-8B 62.9 69.6 63.3 37.7 58.4 Qwen3-4B 52.8 69.6 56.7 36.6 53.9 Qwen3-1.7B 37.8 46.4 20.0 22.8 31.8 Qwen3-0.6B 26.4 46.4 3.3 7.0 20.8 Non-thinking Mode GPT-4o-2024-1120 52.0 80.4 20.0 13.7 41.5 Gemma-3-27b-IT 57.3 71.4 23.3 21.6 43.4 Qwen2.5-72B-Instruct 54.1 67.9 20.0 13.3 38.8 Qwen3-235B-A22B 56.7 75.0 40.0 26.1 49.4 Qwen3-32B 58.6 71.4 30.0 23.3 45.8 Qwen3-30B-A3B 58.0 73.2 30.0 21.1 45.6 Qwen3-14B 60.3 71.4 26.7 24.2 45.6 Qwen3-8B 59.3 58.9 20.0 22.8 40.2 Qwen3-4B 46.1 58.9 13.3 17.8 34.0 Qwen3-1.7B 34.8 41.1 3.3 13.2 23.1 Qwen3-0.6B 25.5 46.4 0.0 5.8 19.4 + +Table 33: Benchmark scores for language: Vietnamese (vi). The highest and second-best scores are shown in bold and underlined, respectively. + +Model MLogiQA INCLUDE MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 72.5 89.1 70.0 52.1 70.9 QwQ-32B 71.2 69.1 70.0 49.2 64.9 Qwen3-235B-A22B 75.0 87.3 83.3 55.1 75.2 Qwen3-32B 67.5 81.8 83.3 44.0 69.2 Qwen3-30B-A3B 68.8 78.2 76.7 46.1 67.4 Qwen3-14B 72.5 72.7 73.3 45.8 66.1 Qwen3-8B 65.0 72.7 73.3 42.9 63.5 Qwen3-4B 68.8 63.6 60.0 42.2 58.6 Qwen3-1.7B 52.5 61.8 30.0 26.9 42.8 Qwen3-0.6B 33.8 38.2 6.7 9.8 22.1 Non-thinking Mode GPT-4o-2024-1120 57.5 81.8 10.0 13.0 40.6 Gemma-3-27b-IT 52.5 74.5 33.3 20.6 45.2 Qwen2.5-72B-Instruct 61.3 72.7 26.7 18.6 44.8 Qwen3-235B-A22B 70.0 83.6 36.7 27.1 54.4 Qwen3-32B 60.0 81.8 23.3 21.8 46.7 Qwen3-30B-A3B 52.5 81.8 20.0 24.7 44.8 Qwen3-14B 63.7 67.3 20.0 21.6 43.2 Qwen3-8B 48.8 65.5 20.0 19.1 38.4 Qwen3-4B 48.8 65.5 20.0 19.0 38.3 Qwen3-1.7B 36.2 60.0 3.3 10.9 27.6 Qwen3-0.6B 30.0 36.4 3.3 3.9 18.4 + +Table 34: Benchmark scores for language: German (de). The highest and second-best scores are shown in bold and underlined, respectively. + +Model INCLUDE MMMLU MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 50.0 85.6 86.7 53.8 69.0 QwQ-32B 57.1 83.8 76.7 51.0 67.2 Qwen3-235B-A22B 71.4 86.0 83.3 55.4 74.0 Qwen3-32B 64.3 81.9 86.7 48.1 70.2 Qwen3-30B-A3B 64.3 81.9 80.0 46.6 68.2 Qwen3-14B 57.1 80.9 70.0 48.1 64.0 Qwen3-8B 64.3 78.1 66.7 43.6 63.2 Qwen3-4B 57.1 74.0 73.3 43.1 61.9 Qwen3-1.7B 64.3 63.4 36.7 26.8 47.8 Qwen3-0.6B 57.1 47.6 10.0 13.7 32.1 Non-thinking Mode GPT-4o-2024-1120 57.1 80.4 10.0 13.5 40.2 Gemma-3-27b-IT 57.1 76.1 26.7 20.2 45.0 Qwen2.5-72B-Instruct 64.3 79.9 16.7 19.3 45.0 Qwen3-235B-A22B 71.4 81.7 40.0 25.9 54.8 Qwen3-32B 57.1 77.2 30.0 21.9 46.6 Qwen3-30B-A3B 57.1 77.7 23.3 25.2 45.8 Qwen3-14B 57.1 76.0 30.0 24.5 46.9 Qwen3-8B 64.3 70.8 20.0 19.9 43.8 Qwen3-4B 64.3 66.0 26.7 16.4 43.4 Qwen3-1.7B 42.9 53.2 10.0 10.6 29.2 Qwen3-0.6B 42.9 37.8 3.3 5.7 22.4 + +Table 35: Benchmark scores for language: Thai (th). The highest and second-best scores are shown in bold and underlined, respectively. + +Model MLogiQA MT-AIME24 PolyMath Average Thinking Mode Gemini2.5-Pro 73.8 80.0 50.7 68.2 QwQ-32B 75.0 60.0 41.3 58.8 Qwen3-235B-A22B 73.8 86.7 53.6 71.4 Qwen3-32B 73.8 76.7 46.9 65.8 Qwen3-30B-A3B 63.7 80.0 45.2 63.0 Qwen3-14B 65.0 76.7 44.4 62.0 Qwen3-8B 68.8 70.0 41.3 60.0 Qwen3-4B 60.0 60.0 39.4 53.1 Qwen3-1.7B 48.8 33.3 23.7 35.3 Qwen3-0.6B 33.8 13.3 11.4 19.5 Non-thinking Mode GPT-4o-2024-1120 52.5 10.0 11.9 24.8 Gemma-3-27b-IT 50.0 16.7 19.0 28.6 Qwen2.5-72B-Instruct 58.8 6.7 17.4 27.6 Qwen3-235B-A22B 61.3 23.3 27.6 37.4 Qwen3-32B 61.3 13.3 22.2 32.3 Qwen3-30B-A3B 50.0 30.0 22.3 34.1 Qwen3-14B 47.5 23.3 22.1 31.0 Qwen3-8B 42.5 10.0 17.2 23.2 Qwen3-4B 43.8 13.3 16.1 24.4 Qwen3-1.7B 42.5 6.7 9.5 19.6 Qwen3-0.6B 37.5 0.0 3.6 13.7 + +Table 36: Language families and language codes supported by Qwen3 in Belebele Benchmark + +Language family# Langs Language code (ISO 639-3_ISO 15924)Indo-European 40 por_Latn, deu_Latn, tgk_Cyrl, ces_Latn, nob_Latn, dan_Latn, snd_Arab, spa_Latn,isl_Latn, slv_Latn, eng_Latn, ory_Orya, hrv_Latn, ell_Grek, ukr_Cyrl, pan_Guru,srp_Cyrl, npi_Deva, mkd_Cyrl, guj_Gujr, nld_Latn, swe_Latn, hin_Deva, rus_Cyrl,asm_Beng, cat_Latn, als_Latn, sin_Sinh, urd_Arab, mar_Deva, lit_Latn, slk_Latn,ita_Latn, pol_Latn, bul_Cyrl, afr_Latn, ron_Latn, fra_Latn, ben_Beng, hye_Armn Sino-Tibetan 3 zho_Hans, mya_Mymr, zho_Hant Afro-Asiatic 8 heb_Hebr, apc_Arab, acm_Arab, ary_Arab, ars_Arab, arb_Arab, mlt_Latn, erz_Arab Austronesian 7 ilo_Latn, ceb_Latn, tgl_Latn, sun_Latn, jav_Latn, war_Latn, ind_Latn Dravidian 4 mal_Mlym, kan_Knda, tel_Telu, tam_Taml Turkic 4 kaz_Cyrl, azj_Latn, tur_Latn, uzn_Latn Tai-Kadai 2 tha_Thai, lao_Laoo Uralic 3 fin_Latn, hun_Latn, est_Latn Austroasiatic 2 vie_Latn, khm_Khmr Other 7 eus_Latn, kor_Hang, hat_Latn, swh_Latn, kea_Latn, jpn_Jpan, kat_Geor + +Table 37: Comparison of Belebele Benchmark performance between Qwen3 and other baseline models. Scores are highlighted with the highest in bold and the second-best underlined. + +Model Indo-European Sino-Tibetan Afro-Asiatic Austronesian Dravidian Turkic Tai-Kadai Uralic Austroasiatic Other Gemma-3-27B-IT 8 9.2 86.3 85.9 8 4.1 83.5 8 6.8 81.0 9 1.0 86.5 87.0 Qwen2.5-32B-Instruct 85.5 82.3 80.4 70.6 67.8 80.8 74.5 87.0 79.0 72.6 QwQ-32B 86.1 83.7 81.9 71.3 69.3 80.3 77.0 88.0 83.0 74.0 Qwen3-32B (Thinking)90.7 89.7 8 4.8 86.7 84.5 89.3 8 3.5 91.3 88.0 8 3.1 Qwen3-32B (Non-thinking)89.1 8 8.0 82.3 83.7 8 4.0 85.0 85.0 88.7 88.0 81.3 Gemma-3-12B-IT 85.8 8 3.3 83.4 79.3 7 9.0 8 2.8 77.5 8 9.0 83.0 8 1.6 Qwen2.5-14B-Instruct 82.7 78.9 80.4 69.1 66.2 74.2 72.2 83.9 77.9 70.4 Qwen3-14B (Thinking)88.6 87.3 8 2.4 82.4 81.0 83.8 83.5 91.0 8 2.5 81.7 Qwen3-14B (Non-thinking)8 7.4 82.7 80.1 8 0.7 78.0 81.8 8 0.5 87.7 81.5 77.0 Gemma-3-4B-IT 71.8 72.0 63.5 61.7 64.8 6 4.0 6 1.5 70.7 71.0 6 2.6 Qwen2.5-3B-Instruct 58.0 62.3 57.2 47.9 36.9 45.1 49.8 50.6 56.8 48.4 Qwen3-4B (Thinking)82.2 77.7 74.1 73.0 74.3 76.3 68.5 83.0 74.5 67.9 Qwen3-4B (Non-thinking)7 6.0 7 7.0 6 5.6 6 5.6 6 5.5 6 4.0 60.5 7 4.0 7 4.0 61.0 Gemma-3-1B-IT 36.5 36.0 30.0 29.1 28.8 27.3 28.0 32.7 33.0 30.9 Qwen2.5-1.5B-Instruct 41.5 43.0 39.6 34.8 28.6 29.7 39.4 33.8 42.0 36.0 Qwen3-1.7B (Thinking)69.7 66.0 59.4 58.6 52.8 57.8 53.5 70.3 63.5 53.4 Qwen3-1.7B (Non-thinking)5 8.8 6 2.7 5 0.8 5 3.0 4 3.3 4 8.0 4 6.0 5 4.3 5 4.0 4 3.9 + +## References + +* Abdin et al. (2024) Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. _arXiv preprint arXiv:2412.08905_, 2024. +* AIME (2025) AIME. AIME problems and solutions, 2025. URL [https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions). +* Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query Transformer models from multi-head checkpoints. In _EMNLP_, pp. 4895–4901. Association for Computational Linguistics, 2023. +* An et al. (2024) Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models. _CoRR_, abs/2402.17463, 2024. +* Anthropic (2025) Anthropic. Claude 3.7 Sonnet, 2025. URL [https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet). +* Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. _CoRR_, abs/2108.07732, 2021. +* Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. _CoRR_, abs/2309.16609, 2023. +* Bai et al. (2025) Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. _arXiv preprint arXiv:2502.13923_, 2025. +* Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The Belebele benchmark: A parallel reading comprehension dataset in 122 language variants. _CoRR_, abs/2308.16884, 2023. +* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In _NeurIPS_, 2020. +* Cassano et al. (2023) Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation. _IEEE Trans. Software Eng._, 49(7):3675–3691, 2023. +* Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. _CoRR_, abs/2107.03374, 2021. +* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. _CoRR_, abs/2110.14168, 2021. +* Dai et al. (2024) Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. _CoRR_, abs/2401.06066, 2024. +* Dauphin et al. (2017) Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In _ICML_, volume 70 of _Proceedings of Machine Learning Research_, pp. 933–941. PMLR, 2017. +* DeepMind (2025) Google DeepMind. Gemini 2.5, 2025. URL [https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/). +* Dehghani et al. (2023) Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In _ICML_, volume 202 of _Proceedings of Machine Learning Research_, pp. 7480–7512. PMLR, 2023. +* Du et al. (2025) Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines. _arXiv preprint arXiv:2502.14739_, 2025. +* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Llama 3 herd of models. _CoRR_, abs/2407.21783, 2024. +* Fan et al. (2023) Simin Fan, Matteo Pagliardini, and Martin Jaggi. DoGE: Domain reweighting with generalization estimation. _arXiv preprint arXiv:2310.15393_, 2023. +* Gema et al. (2024) Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with MMLU? _CoRR_, abs/2406.04127, 2024. +* Gu et al. (2024) Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. _arXiv preprint arXiv:2401.03065_, 2024. +* Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. _arXiv preprint arXiv:2501.12948_, 2025. +* He et al. (2024) Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-IF: Benchmarking LLMs on multi-turn and multilingual instructions following. _arXiv preprint arXiv:2410.15553_, 2024. +* Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In _ICLR_. OpenReview.net, 2021a. +* Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In _NeurIPS Datasets and Benchmarks_, 2021b. +* Hsieh et al. (2024) Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? _CoRR_, abs/2404.06654, 2024. +* Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In _NeurIPS_, 2023. +* Hui et al. (2024) Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-Coder technical report. _CoRR_, abs/2409.12186, 2024. +* Jain et al. (2024) Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. _CoRR_, abs/2403.07974, 2024. +* Jiang et al. (2023) Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Z. Pan. Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and efficient pre-LN Transformers. _CoRR_, abs/2305.14858, 2023. +* Lambert et al. (2024) Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. _CoRR_, abs/2411.15124, 2024. +* Li et al. (2024) Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. _CoRR_, abs/2406.11939, 2024. +* Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. _CoRR_, abs/2305.20050, 2023. +* Lin et al. (2025) Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. ZebraLogic: On the scaling limits of LLMs for logical reasoning. _CoRR_, abs/2502.01100, 2025. +* Liu et al. (2024a) Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. _arXiv preprint arXiv:2412.19437_, 2024a. +* Liu et al. (2023a) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In _NeurIPS_, 2023a. +* Liu et al. (2024b) Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. RegMix: Data mixture as regression for language model pre-training. _arXiv preprint arXiv:2407.01492_, 2024b. +* Liu et al. (2023b) Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. AlignBench: Benchmarking Chinese alignment of large language models. _CoRR_, abs/2311.18743, 2023b. +* Meta-AI (2025) Meta-AI. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, 2025. URL [https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/). +* OpenAI (2024) OpenAI. Hello GPT-4o, 2024. URL [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). +* OpenAI (2024) OpenAI. Multilingual massive multitask language understanding, 2024. URL [https://huggingface.co/datasets/openai/MMMLU](https://huggingface.co/datasets/openai/MMMLU). +* OpenAI (2024) OpenAI. Learning to reason with LLMs, 2024. URL [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/). +* OpenAI (2025) OpenAI. Introducing openai o3 and o4-mini, 2025. URL [https://openai.com/index/introducing-o3-and-o4-mini/](https://openai.com/index/introducing-o3-and-o4-mini/). +* Paech (2024) Samuel J. Paech. Creative writing v3, 2024. URL [https://eqbench.com/creative_writing.html](https://eqbench.com/creative_writing.html). +* Peng et al. (2023) Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. _CoRR_, abs/2309.00071, 2023. +* Qiu et al. (2025) Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-expert models. _CoRR_, abs/2501.11873, 2025. +* Quan et al. (2025) Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking competition-level code generation of LLMs with human-comparable Elo ratings. _CoRR_, abs/2501.01257, 2025. +* Qwen Team (2024) Qwen Team. QwQ: Reflect deeply on the boundaries of the unknown, November 2024. URL [https://qwenlm.github.io/blog/qwq-32b-preview/](https://qwenlm.github.io/blog/qwq-32b-preview/). +* Qwen Team (2025) Qwen Team. QwQ-32B: Embracing the power of reinforcement learning, March 2025. URL [https://qwenlm.github.io/blog/qwq-32b/](https://qwenlm.github.io/blog/qwq-32b/). +* Rein et al. (2023) David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. _CoRR_, abs/2311.12022, 2023. +* Romanou et al. (2024) Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzeminski, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, and Antoine Bosselut. INCLUDE: evaluating multilingual language understanding with regional knowledge. _CoRR_, abs/2411.19799, 2024. +* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In _ACL (1)_. The Association for Computer Linguistics, 2016. +* Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. _CoRR_, abs/2402.03300, 2024. +* Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In _ICLR_. OpenReview.net, 2023. +* Son et al. (2025) Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. Linguistic generalizability of test-time scaling in mathematical reasoning. _CoRR_, abs/2502.17407, 2025. +* Su et al. (2024) Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced Transformer with rotary position embedding. _Neurocomputing_, 568:127063, 2024. +* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In _ACL (Findings)_, pp. 13003–13051. Association for Computational Linguistics, 2023. +* Team et al. (2025) Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. _arXiv preprint arXiv:2503.19786_, 2025. +* Wang et al. (2020) Changhan Wang, Kyunghyun Cho, and Jiatao Gu. Neural machine translation with byte-level subwords. In _AAAI_, pp. 9154–9160. AAAI Press, 2020. +* Wang et al. (2025) Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. PolyMath: Evaluating mathematical reasoning in multilingual contexts, 2025. +* Wang et al. (2024) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. _CoRR_, abs/2406.01574, 2024. +* White et al. (2024) Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-free LLM benchmark. _CoRR_, abs/2406.19314, 2024. +* Wu et al. (2025) Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. WritingBench: A comprehensive benchmark for generative writing. _CoRR_, abs/2503.05244, 2025. +* xAI (2025) xAI. Grok 3 beta — the age of reasoning agents, 2025. URL [https://x.ai/news/grok-3](https://x.ai/news/grok-3). +* Xie et al. (2023) Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. _Advances in Neural Information Processing Systems_, 36:69798–69818, 2023. +* Xiong et al. (2023) Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. _CoRR_, abs/2309.16039, 2023. +* Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. [https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html), 2024. +* Yang et al. (2024a) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. _CoRR_, abs/2407.10671, 2024a. +* Yang et al. (2024b) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. _arXiv preprint arXiv:2412.15115_, 2024b. +* Yang et al. (2024c) An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. _CoRR_, abs/2409.12122, 2024c. +* Zhang et al. (2024) Yidan Zhang, Boyi Deng, Yu Wan, Baosong Yang, Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, and Jingren Zhou. P-MMEval: A parallel multilingual multitask benchmark for consistent evaluation of LLMs. _CoRR_, abs/2411.09116, 2024. +* Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. _CoRR_, abs/2311.07911, 2023. +* Zhu et al. (2025) Qin Zhu, Fei Huang, Runyu Peng, Keming Lu, Bowen Yu, Qinyuan Cheng, Xipeng Qiu, Xuanjing Huang, and Junyang Lin. AutoLogi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models. _CoRR_, abs/2502.16906, 2025. diff --git a/docs/evidence/spinningup_research_source_graph.md b/docs/evidence/spinningup_research_source_graph.md index 66bd1f3..dd673e0 100644 --- a/docs/evidence/spinningup_research_source_graph.md +++ b/docs/evidence/spinningup_research_source_graph.md @@ -4,7 +4,7 @@ Primary source: https://spinningup.openai.com/en/latest/spinningup/spinningup.ht Author: Joshua Achiam, OpenAI Date: October 13th, 2018 Related local cache: docs/evidence/spinningup_researcher.md -Fetch-status: excerpted from Spinning Up HTML via browser; source graph cross-checked against existing local evidence files where present. +Fetch-status: index only. The full page text now lives in docs/evidence/spinningup_researcher.md (fetched 2026-08-15); this file keeps the source graph so the two caches do not hold the same 3.3k words twice (CLAUDE agent) Use: RL research-process evidence, especially for source graph, fair comparisons, seeds, preregistration, and ablations. ## Why this matters for agents @@ -13,35 +13,11 @@ Spinning Up is not just an RL textbook page. Its researcher page is a compact re ## Quotes -> If you’re an aspiring deep RL researcher, you’ve probably heard all kinds of things about deep RL by this point. You know that it’s hard and it doesn’t always work. That even when you’re following a recipe, reproducibility is a challenge. And that if you’re starting from scratch, the learning curve is incredibly steep. - -> In particular, this will outline a useful curriculum for increasing raw knowledge, while interleaving it with the odds and ends that lead to better research. - -> Write your own implementations. You should implement as many of the core deep RL algorithms from scratch as you can, with the aim of writing the shortest correct implementation of each. - -> Simplicity is critical. You should organize your efforts so that you implement the simplest algorithms first, and only gradually introduce complexity. - -> Don’t overfit to existing implementations either. Study existing implementations for inspiration, but be careful not to overfit to the engineering details of those implementations. - -> Iterate fast in simple environments. To debug your implementations, try them with simple environments where learning should happen quickly. - -> Your ideal experiment turnaround-time at the debug stage is <5 minutes (on your local machine) or slightly longer but not much. - -> Start by exploring the literature to become aware of topics in the field. - -> Use the related work section and citations to find closely-related papers and do a deep dive in the literature. You’ll start to figure out where the unsolved problems are and where you can make an impact. - -> There are a many different ways to start thinking about ideas for projects, and the frame you choose influences how the project might evolve and what risks it will face. - -> Avoid reinventing the wheel. When you come up with a good idea that you want to start testing, that’s great! But while you’re still in the early stages with it, do the most thorough check you can to make sure it hasn’t already been done. - -> Under no circumstances handicap the baseline! - -> Beware of random seeds making things look stronger or weaker than they really are, so run everything for many random seeds (at least 3, but if you want to be thorough, do 10 or more). - -> This is to enforce a weak form of preregistration: you use the tuning stage to come up with your hypotheses, and you use the final runs to come up with your conclusions. - -> Check each claim separately. Another critical aspect of doing research is to run an ablation analysis. +The quotes that used to sit here (simplicity is critical, iterate fast in simple +environments, avoid reinventing the wheel, handicap the baseline, seeds, +preregistration, ablations) are all in the full page text at +[spinningup_researcher.md](spinningup_researcher.md). Read that file for the +wording; this one only carries the source graph. (CLAUDE agent, 2026-08-15) ## Source graph diff --git a/docs/evidence/spinningup_researcher.md b/docs/evidence/spinningup_researcher.md index 488cb3c..9fff29b 100644 --- a/docs/evidence/spinningup_researcher.md +++ b/docs/evidence/spinningup_researcher.md @@ -1,31 +1,196 @@ -# Spinning Up as a Deep RL Researcher — Joshua Achiam (OpenAI, 2018-10-13) +Source: https://spinningup.openai.com/en/latest/spinningup/spinningup.html +Title: "Spinning Up as a Deep RL Researcher" - Joshua Achiam (OpenAI), October 13th, 2018 +Fetched-via: curl https://r.jina.ai/, 2026-08-15 (CLAUDE agent) +Fetch-status: verbatim, full page including the reference list. Inline markdown links kept (they are the source graph). Replaces the earlier excerpts (CLAUDE agent) -Source: https://spinningup.openai.com/en/latest/spinningup/spinningup.html . Verbatim excerpts (the debugging/rigour passages) cached for the ML-debugging skill. +Title: Spinning Up as a Deep RL Researcher — Spinning Up documentation ---- +URL Source: https://spinningup.openai.com/en/latest/spinningup/spinningup.html -## Learn by Doing +Markdown Content: +[Spinning Up](https://spinningup.openai.com/en/latest/index.html) -**Simplicity is critical.** You should organize your efforts so that you implement the simplest algorithms first, and only gradually introduce complexity. If you start off trying to build something with too many moving parts, odds are good that it will break and you'll lose weeks trying to debug it. +By Joshua Achiam, October 13th, 2018 -**Focus on understanding.** Writing working RL code requires clear, detail-oriented understanding of the algorithms. This is because **broken RL code almost always fails silently,** where the code appears to run fine except that the agent never learns how to solve the task. Usually the problem is that something is being calculated with the wrong equation, or on the wrong distribution, or data is being piped into the wrong place. Sometimes the only way to find these bugs is to read the code with a critical eye, know exactly what it should be doing, and find where it deviates from the correct behavior. +Table of Contents -**But don't overfit to paper details.** Sometimes, the paper prescribes the use of more tricks than are strictly necessary, so be a bit wary of this, and try out simplifications where possible. For example, the original DDPG paper suggests a complex neural network architecture and initialization scheme, as well as batch normalization. These aren't strictly necessary, and some of the best-reported results for DDPG use simpler networks. As another example, the original A3C paper uses asynchronous updates from the various actor-learners, but it turns out that synchronous updates work about as well. +* [Spinning Up as a Deep RL Researcher](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#spinning-up-as-a-deep-rl-researcher) + * [The Right Background](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#the-right-background) + * [Learn by Doing](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#learn-by-doing) + * [Developing a Research Project](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#developing-a-research-project) + * [Doing Rigorous Research in RL](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#doing-rigorous-research-in-rl) + * [Closing Thoughts](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#closing-thoughts) + * [PS: Other Resources](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#ps-other-resources) + * [References](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#references) -**Don't overfit to existing implementations either.** Study existing implementations for inspiration, but be careful not to overfit to the engineering details of those implementations. RL libraries frequently make choices for abstraction that are good for code reuse between algorithms, but which are unnecessary if you're only writing a single algorithm or supporting a single use case. +If you’re an aspiring deep RL researcher, you’ve probably heard all kinds of things about deep RL by this point. You know that [it’s hard and it doesn’t always work](https://www.alexirpan.com/2018/02/14/rl-hard.html). That even when you’re following a recipe, [reproducibility](https://arxiv.org/abs/1708.04133)[is a challenge](https://arxiv.org/abs/1709.06560). And that if you’re starting from scratch, [the learning curve is incredibly steep](http://amid.fish/reproducing-deep-rl). It’s also the case that there are a lot of [great](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html)[resources](http://rll.berkeley.edu/deeprlcourse/)[out](https://sites.google.com/view/deep-rl-bootcamp/lectures)[there](http://joschu.net/docs/nuts-and-bolts.pdf), but the material is new enough that there’s not a clear, well-charted path to mastery. The goal of this column is to help you get past the initial hurdle, and give you a clear sense of how to spin up as a deep RL researcher. In particular, this will outline a useful curriculum for increasing raw knowledge, while interleaving it with the odds and ends that lead to better research. -**Iterate fast in simple environments.** To debug your implementations, try them with simple environments where learning should happen quickly [...]. Don't try to run an algorithm in Atari or a complex Humanoid environment if you haven't first verified that it works on the simplest possible toy task. Your ideal experiment turnaround-time at the debug stage is <5 minutes (on your local machine) or slightly longer but not much. +## [The Right Background](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#id50)[¶](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#the-right-background "Permalink to this headline") -**If it doesn't work, assume there's a bug.** Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it's a bug. Bad hyperparameters can significantly degrade RL performance, but if you're using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue. Also worth keeping in mind: sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment once your results look promising. +**Build up a solid mathematical background.** From probability and statistics, feel comfortable with random variables, Bayes’ theorem, chain rule of probability, expected values, standard deviations, and importance sampling. From multivariate calculus, understand gradients and (optionally, but it’ll help) Taylor series expansions. -**Measure everything.** Do a lot of instrumenting to see what's going on under-the-hood. The more stats about the learning process you read out at each iteration, the easier it is to debug—after all, you can't tell it's broken if you can't see that it's breaking. I personally like to look at the mean/std/min/max for cumulative rewards, episode lengths, and value function estimates, along with the losses for the objectives, and the details of any exploration parameters [...]. Also, watch videos of your agent's performance every now and then; this will give you some insights you wouldn't get otherwise. +**Build up a general knowledge of deep learning.** You don’t need to know every single special trick and architecture, but the basics help. Know about standard architectures ([MLP](http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/), [vanilla RNN](http://karpathy.github.io/2015/05/21/rnn-effectiveness/), [LSTM](https://arxiv.org/abs/1503.04069) ([also see this blog](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)), [GRU](https://arxiv.org/abs/1412.3555v1), [conv](http://colah.github.io/posts/2014-07-Conv-Nets-Modular/)[layers](https://cs231n.github.io/convolutional-networks/), [resnets](https://arxiv.org/abs/1512.03385), [attention](https://arxiv.org/abs/1409.0473)[mechanisms](https://arxiv.org/abs/1706.03762)), common regularizers ([weight decay](https://papers.nips.cc/paper/563-a-simple-weight-decay-can-improve-generalization.pdf), [dropout](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf)), normalization ([batch norm](https://arxiv.org/abs/1502.03167), [layer norm](https://arxiv.org/abs/1607.06450), [weight norm](https://arxiv.org/abs/1602.07868)), and optimizers ([SGD, momentum SGD](http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/), [Adam](https://arxiv.org/abs/1412.6980), [others](https://arxiv.org/abs/1609.04747)). Know what the [reparameterization trick](https://arxiv.org/abs/1312.6114) is. -## Doing Rigorous Research in RL +**Become familiar with at least one deep learning library.**[Tensorflow](https://www.tensorflow.org/) or [PyTorch](http://pytorch.org/) would be a good place to start. You don’t need to know how to do everything, but you should feel pretty confident in implementing a simple program to do supervised learning. -**Set up fair comparisons.** If you implement your baseline from scratch [...] it's important to spend as much time tuning your baseline as you spend tuning your own algorithm. This will make sure that comparisons are fair. Also, do your best to hold "all else equal" [...]. Under no circumstances handicap the baseline! +**Get comfortable with the main concepts and terminology in RL.** Know what states, actions, trajectories, policies, rewards, value functions, and action-value functions are. If you’re unfamiliar, Spinning Up ships with [an introduction](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html) to this material; it’s also worth checking out the [RL-Intro](https://github.com/jachiam/rl-intro/blob/master/Presentation/rl_intro.pdf) from the OpenAI Hackathon, or the exceptional and thorough [overview by Lilian Weng](https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-into-reinforcement-learning.html). Optionally, if you’re the sort of person who enjoys mathematical theory, study up on the math of [monotonic improvement theory](http://joschu.net/docs/thesis.pdf) (which forms the basis for advanced policy gradient algorithms), or [classical RL algorithms](https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf) (which despite being superseded by deep RL algorithms, contain valuable insights that sometimes drive new research). -**Remove stochasticity as a confounder.** Beware of random seeds making things look stronger or weaker than they really are, so run everything for many random seeds (at least 3, but if you want to be thorough, do 10 or more). [...] There's potentially enough variance that two different groups of random seeds can yield learning curves with differences so significant that they look like they don't come from the same distribution at all. +## [Learn by Doing](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#id51)[¶](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#learn-by-doing "Permalink to this headline") -**Run high-integrity experiments.** Don't just take the results from the best or most interesting runs to use in your paper. Instead, launch new, final experiments [...] and precommit to report on whatever comes out of that. This is to enforce a weak form of preregistration: you use the tuning stage to come up with your hypotheses, and you use the final runs to come up with your conclusions. +**Write your own implementations.** You should implement as many of the core deep RL algorithms from scratch as you can, with the aim of writing the shortest correct implementation of each. This is by far the best way to develop an understanding of how they work, as well as intuitions for their specific performance characteristics. -**Check each claim separately.** [...] run an ablation analysis. Any method you propose is likely to have several key design decisions [...] By systematically evaluating what would happen if you were to swap them out with alternate design choices, or remove them entirely, you can figure out how to correctly attribute credit for the benefits your method confers. +**Simplicity is critical.** You should organize your efforts so that you implement the simplest algorithms first, and only gradually introduce complexity. If you start off trying to build something with too many moving parts, odds are good that it will break and you’ll lose weeks trying to debug it. This is a common failure mode for people who are new to deep RL, and if you find yourself stuck in it, don’t be discouraged—but do try to change tack and work on a simpler algorithm instead, before returning to the more complex thing later. + +**Which algorithms?** You should probably start with vanilla policy gradient (also called [REINFORCE](https://arxiv.org/abs/1604.06778)), [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf), [A2C](https://blog.openai.com/baselines-acktr-a2c/) (the synchronous version of [A3C](https://arxiv.org/abs/1602.01783)), [PPO](https://arxiv.org/abs/1707.06347) (the variant with the clipped objective), and [DDPG](https://arxiv.org/abs/1509.02971), approximately in that order. The simplest versions of all of these can be written in just a few hundred lines of code (ballpark 250-300), and some of them even less (for example, [a no-frills version of VPG](https://github.com/jachiam/rl-intro/blob/master/pg_cartpole.py) can be written in about 80 lines). Write single-threaded code before you try writing parallelized versions of these algorithms. (Do try to parallelize at least one.) + +**Focus on understanding.** Writing working RL code requires clear, detail-oriented understanding of the algorithms. This is because **broken RL code almost always fails silently,** where the code appears to run fine except that the agent never learns how to solve the task. Usually the problem is that something is being calculated with the wrong equation, or on the wrong distribution, or data is being piped into the wrong place. Sometimes the only way to find these bugs is to read the code with a critical eye, know exactly what it should be doing, and find where it deviates from the correct behavior. Developing that knowledge requires you to engage with both academic literature and other existing implementations (when possible), so a good amount of your time should be spent on that reading. + +**What to look for in papers:** When implementing an algorithm based on a paper, scour that paper, especially the ablation analyses and supplementary material (where available). The ablations will give you an intuition for what parameters or subroutines have the biggest impact on getting things to work, which will help you diagnose bugs. Supplementary material will often give information about specific details like network architecture and optimization hyperparameters, and you should try to align your implementation to these details to improve your chances of getting it working. + +**But don’t overfit to paper details.** Sometimes, the paper prescribes the use of more tricks than are strictly necessary, so be a bit wary of this, and try out simplifications where possible. For example, the original DDPG paper suggests a complex neural network architecture and initialization scheme, as well as batch normalization. These aren’t strictly necessary, and some of the best-reported results for DDPG use simpler networks. As another example, the original A3C paper uses asynchronous updates from the various actor-learners, but it turns out that synchronous updates work about as well. + +**Don’t overfit to existing implementations either.** Study [existing](https://github.com/openai/baselines)[implementations](https://github.com/rll/rllab) for inspiration, but be careful not to overfit to the engineering details of those implementations. RL libraries frequently make choices for abstraction that are good for code reuse between algorithms, but which are unnecessary if you’re only writing a single algorithm or supporting a single use case. + +**Iterate fast in simple environments.** To debug your implementations, try them with simple environments where learning should happen quickly, like CartPole-v0, InvertedPendulum-v0, FrozenLake-v0, and HalfCheetah-v2 (with a short time horizon—only 100 or 250 steps instead of the full 1000) from the [OpenAI Gym](https://gym.openai.com/). Don’t try to run an algorithm in Atari or a complex Humanoid environment if you haven’t first verified that it works on the simplest possible toy task. Your ideal experiment turnaround-time at the debug stage is <5 minutes (on your local machine) or slightly longer but not much. These small-scale experiments don’t require any special hardware, and can be run without too much trouble on CPUs. + +**If it doesn’t work, assume there’s a bug.** Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually it’s a bug. Bad hyperparameters can significantly degrade RL performance, but if you’re using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue. Also worth keeping in mind: sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment once your results look promising. + +**Measure everything.** Do a lot of instrumenting to see what’s going on under-the-hood. The more stats about the learning process you read out at each iteration, the easier it is to debug—after all, you can’t tell it’s broken if you can’t see that it’s breaking. I personally like to look at the mean/std/min/max for cumulative rewards, episode lengths, and value function estimates, along with the losses for the objectives, and the details of any exploration parameters (like mean entropy for stochastic policy optimization, or current epsilon for epsilon-greedy as in DQN). Also, watch videos of your agent’s performance every now and then; this will give you some insights you wouldn’t get otherwise. + +**Scale experiments when things work.** After you have an implementation of an RL algorithm that seems to work correctly in the simplest environments, test it out on harder environments. Experiments at this stage will take longer—on the order of somewhere between a few hours and a couple of days, depending. Specialized hardware—like a beefy GPU or a 32-core machine—might be useful at this point, and you should consider looking into cloud computing resources like AWS or GCE. + +**Keep these habits!** These habits are worth keeping beyond the stage where you’re just learning about deep RL—they will accelerate your research! + +## [Developing a Research Project](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#id52)[¶](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#developing-a-research-project "Permalink to this headline") + +Once you feel reasonably comfortable with the basics in deep RL, you should start pushing on the boundaries and doing research. To get there, you’ll need an idea for a project. + +**Start by exploring the literature to become aware of topics in the field.** There are a wide range of topics you might find interesting: sample efficiency, exploration, transfer learning, hierarchy, memory, model-based RL, meta learning, and multi-agent, to name a few. If you’re looking for inspiration, or just want to get a rough sense of what’s out there, check out Spinning Up’s [key papers](https://spinningup.openai.com/en/latest/spinningup/keypapers.html) list. Find a paper that you enjoy on one of these subjects—something that inspires you—and read it thoroughly. Use the related work section and citations to find closely-related papers and do a deep dive in the literature. You’ll start to figure out where the unsolved problems are and where you can make an impact. + +**Approaches to idea-generation:** There are a many different ways to start thinking about ideas for projects, and the frame you choose influences how the project might evolve and what risks it will face. Here are a few examples: + +**Frame 1: Improving on an Existing Approach.** This is the incrementalist angle, where you try to get performance gains in an established problem setting by tweaking an existing algorithm. Reimplementing prior work is super helpful here, because it exposes you to the ways that existing algorithms are brittle and could be improved. A novice will find this the most accessible frame, but it can also be worthwhile for researchers at any level of experience. While some researchers find incrementalism less exciting, some of the most impressive achievements in machine learning have come from work of this nature. + +Because projects like these are tied to existing methods, they are by nature narrowly scoped and can wrap up quickly (a few months), which may be desirable (especially when starting out as a researcher). But this also sets up the risks: it’s possible that the tweaks you have in mind for an algorithm may fail to improve it, in which case, unless you come up with more tweaks, the project is just over and you have no clear signal on what to do next. + +**Frame 2: Focusing on Unsolved Benchmarks.** Instead of thinking about how to improve an existing method, you aim to succeed on a task that no one has solved before. For example: achieving perfect generalization from training levels to test levels in the [Sonic domain](https://contest.openai.com/2018-1/) or [Gym Retro](https://blog.openai.com/gym-retro/). When you hammer away at an unsolved task, you might try a wide variety of methods, including prior approaches and new ones that you invent for the project. It is possible for a novice to approch this kind of problem, but there will be a steeper learning curve. + +Projects in this frame have a broad scope and can go on for a while (several months to a year-plus). The main risk is that the benchmark is unsolvable without a substantial breakthrough, meaning that it would be easy to spend a lot of time without making any progress on it. But even if a project like this fails, it often leads the researcher to many new insights that become fertile soil for the next project. + +**Frame 3: Create a New Problem Setting.** Instead of thinking about existing methods or current grand challenges, think of an entirely different conceptual problem that hasn’t been studied yet. Then, figure out how to make progress on it. For projects along these lines, a standard benchmark probably doesn’t exist yet, and you will have to design one. This can be a huge challenge, but it’s worth embracing—great benchmarks move the whole field forward. + +Problems in this frame come up when they come up—it’s hard to go looking for them. + +**Avoid reinventing the wheel.** When you come up with a good idea that you want to start testing, that’s great! But while you’re still in the early stages with it, do the most thorough check you can to make sure it hasn’t already been done. It can be pretty disheartening to get halfway through a project, and only then discover that there’s already a paper about your idea. It’s especially frustrating when the work is concurrent, which happens from time to time! But don’t let that deter you—and definitely don’t let it motivate you to plant flags with not-quite-finished research and over-claim the merits of the partial work. Do good research and finish out your projects with complete and thorough investigations, because that’s what counts, and by far what matters most in the long run. + +## [Doing Rigorous Research in RL](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#id53)[¶](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#doing-rigorous-research-in-rl "Permalink to this headline") + +Now you’ve come up with an idea, and you’re fairly certain it hasn’t been done. You use the skills you’ve developed to implement it and you start testing it out on standard domains. It looks like it works! But what does that mean, and how well does it have to work to be important? This is one of the hardest parts of research in deep RL. In order to validate that your proposal is a meaningful contribution, you have to rigorously prove that it actually gets a performance benefit over the strongest possible baseline algorithm—whatever currently achieves SOTA (state of the art) on your test domains. If you’ve invented a new test domain, so there’s no previous SOTA, you still need to try out whatever the most reliable algorithm in the literature is that could plausibly do well in the new test domain, and then you have to beat that. + +**Set up fair comparisons.** If you implement your baseline from scratch—as opposed to comparing against another paper’s numbers directly—it’s important to spend as much time tuning your baseline as you spend tuning your own algorithm. This will make sure that comparisons are fair. Also, do your best to hold “all else equal” even if there are substantial differences between your algorithm and the baseline. For example, if you’re investigating architecture variants, keep the number of model parameters approximately equal between your model and the baseline. Under no circumstances handicap the baseline! It turns out that the baselines in RL are pretty strong, and getting big, consistent wins over them can be tricky or require some good insight in algorithm design. + +**Remove stochasticity as a confounder.** Beware of random seeds making things look stronger or weaker than they really are, so run everything for many random seeds (at least 3, but if you want to be thorough, do 10 or more). This is really important and deserves a lot of emphasis: deep RL seems fairly brittle with respect to random seed in a lot of common use cases. There’s potentially enough variance that two different groups of random seeds can yield learning curves with differences so significant that they look like they don’t come from the same distribution at all (see [figure 10 here](https://arxiv.org/pdf/1708.04133.pdf)). + +**Run high-integrity experiments.** Don’t just take the results from the best or most interesting runs to use in your paper. Instead, launch new, final experiments—for all of the methods that you intend to compare (if you are comparing against your own baseline implementations)—and precommit to report on whatever comes out of that. This is to enforce a weak form of [preregistration](https://cos.io/prereg/): you use the tuning stage to come up with your hypotheses, and you use the final runs to come up with your conclusions. + +**Check each claim separately.** Another critical aspect of doing research is to run an ablation analysis. Any method you propose is likely to have several key design decisions—like architecture choices or regularization techniques, for instance—each of which could separately impact performance. The claim you’ll make in your work is that those design decisions collectively help, but this is really a bundle of several claims in disguise: one for each such design element. By systematically evaluating what would happen if you were to swap them out with alternate design choices, or remove them entirely, you can figure out how to correctly attribute credit for the benefits your method confers. This lets you make each separate claim with a measure of confidence, and increases the overall strength of your work. + +## [Closing Thoughts](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#id54)[¶](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#closing-thoughts "Permalink to this headline") + +Deep RL is an exciting, fast-moving field, and we need as many people as possible to go through the open problems and make progress on them. Hopefully, you feel a bit more prepared to be a part of it after reading this! And whenever you’re ready, [let us know](https://jobs.lever.co/openai). + +## [References](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#id56)[¶](https://spinningup.openai.com/en/latest/spinningup/spinningup.html#references "Permalink to this headline") + +[1][Deep Reinforcement Learning Doesn’t Work Yet](https://www.alexirpan.com/2018/02/14/rl-hard.html), Alex Irpan, 2018 + +[2][Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control](https://arxiv.org/abs/1708.04133), Islam et al, 2017 + +[3][Deep Reinforcement Learning that Matters](https://arxiv.org/abs/1709.06560), Henderson et al, 2017 + +[4][Lessons Learned Reproducing a Deep Reinforcement Learning Paper](http://amid.fish/reproducing-deep-rl), Matthew Rahtz, 2018 + +[5][UCL Course on RL](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html) + +[6][Berkeley Deep RL Course](http://rll.berkeley.edu/deeprlcourse/) + +[7][Deep RL Bootcamp](https://sites.google.com/view/deep-rl-bootcamp/lectures) + +[8][Nuts and Bolts of Deep RL](http://joschu.net/docs/nuts-and-bolts.pdf), John Schulman + +[9][Stanford Deep Learning Tutorial: Multi-Layer Neural Network](http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/) + +[10][The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/), Andrej Karpathy, 2015 + +[11][LSTM: A Search Space Odyssey](https://arxiv.org/abs/1503.04069), Greff et al, 2015 + +[12][Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/), Chris Olah, 2015 + +[13][Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://arxiv.org/abs/1412.3555v1), Chung et al, 2014 (GRU paper) + +[14][Conv Nets: A Modular Perspective](http://colah.github.io/posts/2014-07-Conv-Nets-Modular/), Chris Olah, 2014 + +[15][Stanford CS231n, Convolutional Neural Networks for Visual Recognition](https://cs231n.github.io/convolutional-networks/) + +[16][Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385), He et al, 2015 (ResNets) + +[17][Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473), Bahdanau et al, 2014 (Attention mechanisms) + +[18][Attention Is All You Need](https://arxiv.org/abs/1706.03762), Vaswani et al, 2017 + +[19][A Simple Weight Decay Can Improve Generalization](https://papers.nips.cc/paper/563-a-simple-weight-decay-can-improve-generalization.pdf), Krogh and Hertz, 1992 + +[20][Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf), Srivastava et al, 2014 + +[21][Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167), Ioffe and Szegedy, 2015 + +[22][Layer Normalization](https://arxiv.org/abs/1607.06450), Ba et al, 2016 + +[23][Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://arxiv.org/abs/1602.07868), Salimans and Kingma, 2016 + +[24][Stanford Deep Learning Tutorial: Stochastic Gradient Descent](http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/) + +[25][Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980), Kingma and Ba, 2014 + +[26][An overview of gradient descent optimization algorithms](https://arxiv.org/abs/1609.04747), Sebastian Ruder, 2016 + +[27][Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114), Kingma and Welling, 2013 (Reparameterization trick) + +[28][Tensorflow](https://www.tensorflow.org/) + +[29][PyTorch](http://pytorch.org/) + +[30][Spinning Up in Deep RL: Introduction to RL, Part 1](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html) + +[31][RL-Intro](https://github.com/jachiam/rl-intro/blob/master/Presentation/rl_intro.pdf) Slides from OpenAI Hackathon, Josh Achiam, 2018 + +[32][A (Long) Peek into Reinforcement Learning](https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-into-reinforcement-learning.html), Lilian Weng, 2018 + +[33][Optimizing Expectations](http://joschu.net/docs/thesis.pdf), John Schulman, 2016 (Monotonic improvement theory) + +[34][Algorithms for Reinforcement Learning](https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf), Csaba Szepesvari, 2009 (Classic RL Algorithms) + +[35][Benchmarking Deep Reinforcement Learning for Continuous Control](https://arxiv.org/abs/1604.06778), Duan et al, 2016 + +[36][Playing Atari with Deep Reinforcement Learning](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf), Mnih et al, 2013 (DQN) + +[37][OpenAI Baselines: ACKTR & A2C](https://blog.openai.com/baselines-acktr-a2c/) + +[38][Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783), Mnih et al, 2016 (A3C) + +[39][Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347), Schulman et al, 2017 (PPO) + +[40][Continuous Control with Deep Reinforcement Learning](https://arxiv.org/abs/1509.02971), Lillicrap et al, 2015 (DDPG) + +[41][RL-Intro Policy Gradient Sample Code](https://github.com/jachiam/rl-intro/blob/master/pg_cartpole.py), Josh Achiam, 2018 + +[42][OpenAI Baselines](https://github.com/openai/baselines) + +[43][rllab](https://github.com/rll/rllab) + +[44][OpenAI Gym](https://gym.openai.com/) + +[45][OpenAI Retro Contest](https://contest.openai.com/2018-1/) + +[46][OpenAI Gym Retro](https://blog.openai.com/gym-retro/) + +[47][Center for Open Science](https://cos.io/prereg/), explaining what preregistration means in the context of scientific experiments.