wassname/pytorch-lightning
1
0 Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-10-02 12:40:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

Deployed b35229d with MkDocs version: 1.0.4

Browse Source
This commit is contained in:
William Falcon committed 2019-11-06 15:08:12 -05:00
1 parent 65095b6cac
commit 2cc732ca16
46 files changed
+995 -718
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Trainer/Checkpointing/index.html
+34 -20
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -344,14 +344,14 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#model-saving" class="md-nav__link">
<a href="#model-saving" title="Model saving" class="md-nav__link">
Model saving
</a>
</li>
<li class="md-nav__item">
<a href="#restoring-training-session" class="md-nav__link">
<a href="#restoring-training-session" title="Restoring training session" class="md-nav__link">
Restoring training session
</a>
@@ -524,14 +524,14 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#model-saving" class="md-nav__link">
<a href="#model-saving" title="Model saving" class="md-nav__link">
Model saving
</a>
</li>
<li class="md-nav__item">
<a href="#restoring-training-session" class="md-nav__link">
<a href="#restoring-training-session" title="Restoring training session" class="md-nav__link">
Restoring training session
</a>
@@ -601,19 +601,33 @@ To change the checkpoint path pass in :</p>
<p>You might want to not only load a model but also continue training it. Use this method to
restore the trainer state as well. This will continue from the epoch and global step you last left off.<br />
However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter). </p>
<p>Lightning will restore the session if you pass an experiment with the same version and there's a saved checkpoint. </p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5
6
7
8
9</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">test_tube</span> <span class="kn">import</span> <span class="n">Experiment</span>
<p>Lightning will restore the session if you pass a logger with the same version and there's a saved checkpoint. </p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">pytorch_lightning</span> <span class="kn">import</span> <span class="n">Trainer</span>
<span class="kn">from</span> <span class="nn">pytorch_lightning.logging</span> <span class="kn">import</span> <span class="n">TestTubeLogger</span>
<span class="n">exp</span> <span class="o">=</span> <span class="n">Experiment</span><span class="p">(</span><span class="n">version</span><span class="o">=</span><span class="n">a_previous_version_with_a_saved_checkpoint</span><span class="p">)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">experiment</span><span class="o">=</span><span class="n">exp</span><span class="p">)</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">TestTubeLogger</span><span class="p">(</span>
<span class="n">save_dir</span><span class="o">=</span><span class="s1">&#39;./savepath&#39;</span><span class="p">,</span>
<span class="n">version</span><span class="o">=</span><span class="mi">1</span> <span class="c1"># An existing version with a saved checkpoint</span>
<span class="p">)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span>
<span class="n">logger</span><span class="o">=</span><span class="n">logger</span><span class="p">,</span>
<span class="n">default_save_path</span><span class="o">=</span><span class="s1">&#39;./savepath&#39;</span>
<span class="p">)</span>
<span class="c1"># this fit call loads model weights and trainer state</span>
<span class="c1"># the trainer continues seamlessly from where you left off</span>
@@ -733,7 +747,7 @@ the system isn't different. If you add a layer, for instance, it might not work.
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/Distributed training/index.html
+51 -37
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -356,7 +356,7 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#choosing-a-backend" class="md-nav__link">
<a href="#choosing-a-backend" title="Choosing a backend" class="md-nav__link">
Choosing a backend
</a>
@@ -364,21 +364,21 @@
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#dataparallel-dp" class="md-nav__link">
<a href="#dataparallel-dp" title="DataParallel (dp)" class="md-nav__link">
DataParallel (dp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-ddp" class="md-nav__link">
<a href="#distributeddataparallel-ddp" title="DistributedDataParallel (ddp)" class="md-nav__link">
DistributedDataParallel (ddp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-2-ddp2" class="md-nav__link">
<a href="#distributeddataparallel-2-ddp2" title="DistributedDataParallel-2 (ddp2)" class="md-nav__link">
DistributedDataParallel-2 (ddp2)
</a>
@@ -390,56 +390,56 @@
</li>
<li class="md-nav__item">
<a href="#distributed-and-16-bit-precision" class="md-nav__link">
<a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
Distributed and 16-bit precision.
</a>
</li>
<li class="md-nav__item">
<a href="#cuda-flags" class="md-nav__link">
<a href="#cuda-flags" title="CUDA flags" class="md-nav__link">
CUDA flags
</a>
</li>
<li class="md-nav__item">
<a href="#16-bit-mixed-precision" class="md-nav__link">
<a href="#16-bit-mixed-precision" title="16-bit mixed precision" class="md-nav__link">
16-bit mixed precision
</a>
</li>
<li class="md-nav__item">
<a href="#single-gpu" class="md-nav__link">
<a href="#single-gpu" title="Single-gpu" class="md-nav__link">
Single-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-gpu" class="md-nav__link">
<a href="#multi-gpu" title="multi-gpu" class="md-nav__link">
multi-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-node" class="md-nav__link">
<a href="#multi-node" title="Multi-node" class="md-nav__link">
Multi-node
</a>
</li>
<li class="md-nav__item">
<a href="#auto-slurm-job-submission" class="md-nav__link">
<a href="#auto-slurm-job-submission" title="Auto-slurm-job-submission" class="md-nav__link">
Auto-slurm-job-submission
</a>
</li>
<li class="md-nav__item">
<a href="#self-balancing-architecture" class="md-nav__link">
<a href="#self-balancing-architecture" title="Self-balancing architecture" class="md-nav__link">
Self-balancing architecture
</a>
@@ -600,7 +600,7 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#choosing-a-backend" class="md-nav__link">
<a href="#choosing-a-backend" title="Choosing a backend" class="md-nav__link">
Choosing a backend
</a>
@@ -608,21 +608,21 @@
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#dataparallel-dp" class="md-nav__link">
<a href="#dataparallel-dp" title="DataParallel (dp)" class="md-nav__link">
DataParallel (dp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-ddp" class="md-nav__link">
<a href="#distributeddataparallel-ddp" title="DistributedDataParallel (ddp)" class="md-nav__link">
DistributedDataParallel (ddp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-2-ddp2" class="md-nav__link">
<a href="#distributeddataparallel-2-ddp2" title="DistributedDataParallel-2 (ddp2)" class="md-nav__link">
DistributedDataParallel-2 (ddp2)
</a>
@@ -634,56 +634,56 @@
</li>
<li class="md-nav__item">
<a href="#distributed-and-16-bit-precision" class="md-nav__link">
<a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
Distributed and 16-bit precision.
</a>
</li>
<li class="md-nav__item">
<a href="#cuda-flags" class="md-nav__link">
<a href="#cuda-flags" title="CUDA flags" class="md-nav__link">
CUDA flags
</a>
</li>
<li class="md-nav__item">
<a href="#16-bit-mixed-precision" class="md-nav__link">
<a href="#16-bit-mixed-precision" title="16-bit mixed precision" class="md-nav__link">
16-bit mixed precision
</a>
</li>
<li class="md-nav__item">
<a href="#single-gpu" class="md-nav__link">
<a href="#single-gpu" title="Single-gpu" class="md-nav__link">
Single-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-gpu" class="md-nav__link">
<a href="#multi-gpu" title="multi-gpu" class="md-nav__link">
multi-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-node" class="md-nav__link">
<a href="#multi-node" title="Multi-node" class="md-nav__link">
Multi-node
</a>
</li>
<li class="md-nav__item">
<a href="#auto-slurm-job-submission" class="md-nav__link">
<a href="#auto-slurm-job-submission" title="Auto-slurm-job-submission" class="md-nav__link">
Auto-slurm-job-submission
</a>
</li>
<li class="md-nav__item">
<a href="#self-balancing-architecture" class="md-nav__link">
<a href="#self-balancing-architecture" title="Self-balancing architecture" class="md-nav__link">
Self-balancing architecture
</a>
@@ -814,21 +814,35 @@ not allow 16-bit and DP training. We tried to get this to work, but it's an issu
</tbody>
</table>
<p>You also have the option of specifying which GPUs to use by passing a list: </p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5
6
7
8</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (int)</span>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (int) specifies how many GPUs to use.</span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="n">k</span><span class="p">)</span>
<span class="c1"># Above is equivalent to </span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">k</span><span class="p">)))</span>
<span class="c1"># You specify which GPUs (don&#39;t use if running on cluster) </span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>
<span class="c1"># can also be a string</span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="s1">&#39;0, 1&#39;</span><span class="p">)</span>
<span class="c1"># can also be -1 or &#39;-1&#39;, this uses all available GPUs</span>
<span class="c1"># this is equivalent to list(range(torch.cuda.available_devices()))</span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
</td></tr></table>
@@ -1025,7 +1039,7 @@ portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).
<p>Instead of manually building SLURM scripts, you can use the <a href="https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/">SlurmCluster object</a> to
do this for you. The SlurmCluster can also run a grid search if you pass in a <a href="https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/">HyperOptArgumentParser</a>.</p>
<p>Here is an example where you run a grid search of 9 combinations of hyperparams.
<a href="https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/new_project_templates/multi_node_examples">The full examples are here</a>.</p>
<a href="https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/new_project_templates/multi_node_examples">The full examples are here</a>.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
@@ -1184,7 +1198,7 @@ do this for you. The SlurmCluster can also run a grid search if you pass in a <a
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/Logging/index.html
+76 -28
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -368,14 +368,14 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#default_save_path" class="md-nav__link">
<a href="#default_save_path" title="default_save_path" class="md-nav__link">
default_save_path
</a>
</li>
<li class="md-nav__item">
<a href="#setting-up-logging" class="md-nav__link">
<a href="#setting-up-logging" title="Setting up logging" class="md-nav__link">
Setting up logging
</a>
@@ -383,70 +383,77 @@
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#test-tube" class="md-nav__link">
<a href="#test-tube" title="Test tube" class="md-nav__link">
Test tube
</a>
</li>
<li class="md-nav__item">
<a href="#mlflow" class="md-nav__link">
<a href="#mlflow" title="MLFlow" class="md-nav__link">
MLFlow
</a>
</li>
<li class="md-nav__item">
<a href="#custom-logger" class="md-nav__link">
<a href="#cometml" title="Comet.ml" class="md-nav__link">
Comet.ml
</a>
</li>
<li class="md-nav__item">
<a href="#custom-logger" title="Custom logger" class="md-nav__link">
Custom logger
</a>
</li>
<li class="md-nav__item">
<a href="#using-loggers" class="md-nav__link">
<a href="#using-loggers" title="Using loggers" class="md-nav__link">
Using loggers
</a>
</li>
<li class="md-nav__item">
<a href="#display-metrics-in-progress-bar" class="md-nav__link">
<a href="#display-metrics-in-progress-bar" title="Display metrics in progress bar" class="md-nav__link">
Display metrics in progress bar
</a>
</li>
<li class="md-nav__item">
<a href="#log-metric-row-every-k-batches" class="md-nav__link">
<a href="#log-metric-row-every-k-batches" title="Log metric row every k batches" class="md-nav__link">
Log metric row every k batches
</a>
</li>
<li class="md-nav__item">
<a href="#log-gpu-memory" class="md-nav__link">
<a href="#log-gpu-memory" title="Log GPU memory" class="md-nav__link">
Log GPU memory
</a>
</li>
<li class="md-nav__item">
<a href="#process-position" class="md-nav__link">
<a href="#process-position" title="Process position" class="md-nav__link">
Process position
</a>
</li>
<li class="md-nav__item">
<a href="#save-a-snapshot-of-all-hyperparameters" class="md-nav__link">
<a href="#save-a-snapshot-of-all-hyperparameters" title="Save a snapshot of all hyperparameters" class="md-nav__link">
Save a snapshot of all hyperparameters
</a>
</li>
<li class="md-nav__item">
<a href="#write-logs-file-to-csv-every-k-batches" class="md-nav__link">
<a href="#write-logs-file-to-csv-every-k-batches" title="Write logs file to csv every k batches" class="md-nav__link">
Write logs file to csv every k batches
</a>
@@ -600,14 +607,14 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#default_save_path" class="md-nav__link">
<a href="#default_save_path" title="default_save_path" class="md-nav__link">
default_save_path
</a>
</li>
<li class="md-nav__item">
<a href="#setting-up-logging" class="md-nav__link">
<a href="#setting-up-logging" title="Setting up logging" class="md-nav__link">
Setting up logging
</a>
@@ -615,70 +622,77 @@
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#test-tube" class="md-nav__link">
<a href="#test-tube" title="Test tube" class="md-nav__link">
Test tube
</a>
</li>
<li class="md-nav__item">
<a href="#mlflow" class="md-nav__link">
<a href="#mlflow" title="MLFlow" class="md-nav__link">
MLFlow
</a>
</li>
<li class="md-nav__item">
<a href="#custom-logger" class="md-nav__link">
<a href="#cometml" title="Comet.ml" class="md-nav__link">
Comet.ml
</a>
</li>
<li class="md-nav__item">
<a href="#custom-logger" title="Custom logger" class="md-nav__link">
Custom logger
</a>
</li>
<li class="md-nav__item">
<a href="#using-loggers" class="md-nav__link">
<a href="#using-loggers" title="Using loggers" class="md-nav__link">
Using loggers
</a>
</li>
<li class="md-nav__item">
<a href="#display-metrics-in-progress-bar" class="md-nav__link">
<a href="#display-metrics-in-progress-bar" title="Display metrics in progress bar" class="md-nav__link">
Display metrics in progress bar
</a>
</li>
<li class="md-nav__item">
<a href="#log-metric-row-every-k-batches" class="md-nav__link">
<a href="#log-metric-row-every-k-batches" title="Log metric row every k batches" class="md-nav__link">
Log metric row every k batches
</a>
</li>
<li class="md-nav__item">
<a href="#log-gpu-memory" class="md-nav__link">
<a href="#log-gpu-memory" title="Log GPU memory" class="md-nav__link">
Log GPU memory
</a>
</li>
<li class="md-nav__item">
<a href="#process-position" class="md-nav__link">
<a href="#process-position" title="Process position" class="md-nav__link">
Process position
</a>
</li>
<li class="md-nav__item">
<a href="#save-a-snapshot-of-all-hyperparameters" class="md-nav__link">
<a href="#save-a-snapshot-of-all-hyperparameters" title="Save a snapshot of all hyperparameters" class="md-nav__link">
Save a snapshot of all hyperparameters
</a>
</li>
<li class="md-nav__item">
<a href="#write-logs-file-to-csv-every-k-batches" class="md-nav__link">
<a href="#write-logs-file-to-csv-every-k-batches" title="Write logs file to csv every k batches" class="md-nav__link">
Write logs file to csv every k batches
</a>
@@ -806,6 +820,40 @@ to manage experiments and snapshots all hyperparameters you pass to a LightningM
</pre></div>
</td></tr></table>
<hr />
<h4 id="cometml">Comet.ml</h4>
<p>Log using <a href="https://www.comet.ml">comet</a></p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5
6
7</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">pytorch_lightning.logging</span> <span class="kn">import</span> <span class="n">CometLogger</span>
<span class="c1"># arguments made to CometLogger are passed on to the comet_ml.Experiment class</span>
<span class="n">comet_logger</span> <span class="o">=</span> <span class="n">CometLogger</span><span class="p">(</span>
<span class="n">api_key</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&quot;COMET_KEY&quot;</span><span class="p">],</span>
<span class="n">workspace</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&quot;COMET_KEY&quot;</span><span class="p">],</span>
<span class="p">)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">logger</span><span class="o">=</span><span class="n">comet_logger</span><span class="p">)</span>
</pre></div>
</td></tr></table>
<p>Use the logger anywhere in you LightningModule as follows:</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5
6</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">train_step</span><span class="p">(</span><span class="o">...</span><span class="p">):</span>
<span class="c1"># example</span>
<span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">experiment</span><span class="o">.</span><span class="n">whatever_comet_ml_supports</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">any_lightning_module_function_or_hook</span><span class="p">(</span><span class="o">...</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">experiment</span><span class="o">.</span><span class="n">whatever_comet_ml_supports</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="custom-logger">Custom logger</h4>
<p>You can implement your own logger by writing a class that inherits from
@@ -1039,7 +1087,7 @@ Lightning will stack progress bars according to this value. </p>
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/SLURM Managed Cluster/index.html
+8 -8
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -380,14 +380,14 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#running-grid-search-on-a-cluster" class="md-nav__link">
<a href="#running-grid-search-on-a-cluster" title="Running grid search on a cluster" class="md-nav__link">
Running grid search on a cluster
</a>
</li>
<li class="md-nav__item">
<a href="#walltime-auto-resubmit" class="md-nav__link">
<a href="#walltime-auto-resubmit" title="Walltime auto-resubmit" class="md-nav__link">
Walltime auto-resubmit
</a>
@@ -524,14 +524,14 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#running-grid-search-on-a-cluster" class="md-nav__link">
<a href="#running-grid-search-on-a-cluster" title="Running grid search on a cluster" class="md-nav__link">
Running grid search on a cluster
</a>
</li>
<li class="md-nav__item">
<a href="#walltime-auto-resubmit" class="md-nav__link">
<a href="#walltime-auto-resubmit" title="Walltime auto-resubmit" class="md-nav__link">
Walltime auto-resubmit
</a>
@@ -777,7 +777,7 @@ your SLURM script. </p>
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/Testing loop/index.html
+6 -6
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -392,7 +392,7 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#test" class="md-nav__link">
<a href="#test" title="test" class="md-nav__link">
test
</a>
@@ -517,7 +517,7 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#test" class="md-nav__link">
<a href="#test" title="test" class="md-nav__link">
test
</a>
@@ -658,7 +658,7 @@ First case is where you test right after a full training routine.</p>
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/Training Loop/index.html
+93 -22
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -404,54 +404,68 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#accumulated-gradients" class="md-nav__link">
<a href="#accumulated-gradients" title="Accumulated gradients" class="md-nav__link">
Accumulated gradients
</a>
</li>
<li class="md-nav__item">
<a href="#force-training-for-min-or-max-epochs" class="md-nav__link">
<a href="#force-training-for-min-or-max-epochs" title="Force training for min or max epochs" class="md-nav__link">
Force training for min or max epochs
</a>
</li>
<li class="md-nav__item">
<a href="#early-stopping" class="md-nav__link">
<a href="#early-stopping" title="Early stopping" class="md-nav__link">
Early stopping
</a>
</li>
<li class="md-nav__item">
<a href="#force-disable-early-stop" class="md-nav__link">
<a href="#force-disable-early-stop" title="Force disable early stop" class="md-nav__link">
Force disable early stop
</a>
</li>
<li class="md-nav__item">
<a href="#gradient-clipping" class="md-nav__link">
<a href="#gradient-clipping" title="Gradient Clipping" class="md-nav__link">
Gradient Clipping
</a>
</li>
<li class="md-nav__item">
<a href="#inspect-gradient-norms" class="md-nav__link">
<a href="#inspect-gradient-norms" title="Inspect gradient norms" class="md-nav__link">
Inspect gradient norms
</a>
</li>
<li class="md-nav__item">
<a href="#set-how-much-of-the-training-set-to-check" class="md-nav__link">
<a href="#set-how-much-of-the-training-set-to-check" title="Set how much of the training set to check" class="md-nav__link">
Set how much of the training set to check
</a>
</li>
<li class="md-nav__item">
<a href="#packed-sequences-as-inputs" title="Packed sequences as inputs" class="md-nav__link">
Packed sequences as inputs
</a>
</li>
<li class="md-nav__item">
<a href="#truncated-back-propagation-through-time" title="Truncated Back Propagation Through Time" class="md-nav__link">
Truncated Back Propagation Through Time
</a>
</li>
@@ -559,54 +573,68 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#accumulated-gradients" class="md-nav__link">
<a href="#accumulated-gradients" title="Accumulated gradients" class="md-nav__link">
Accumulated gradients
</a>
</li>
<li class="md-nav__item">
<a href="#force-training-for-min-or-max-epochs" class="md-nav__link">
<a href="#force-training-for-min-or-max-epochs" title="Force training for min or max epochs" class="md-nav__link">
Force training for min or max epochs
</a>
</li>
<li class="md-nav__item">
<a href="#early-stopping" class="md-nav__link">
<a href="#early-stopping" title="Early stopping" class="md-nav__link">
Early stopping
</a>
</li>
<li class="md-nav__item">
<a href="#force-disable-early-stop" class="md-nav__link">
<a href="#force-disable-early-stop" title="Force disable early stop" class="md-nav__link">
Force disable early stop
</a>
</li>
<li class="md-nav__item">
<a href="#gradient-clipping" class="md-nav__link">
<a href="#gradient-clipping" title="Gradient Clipping" class="md-nav__link">
Gradient Clipping
</a>
</li>
<li class="md-nav__item">
<a href="#inspect-gradient-norms" class="md-nav__link">
<a href="#inspect-gradient-norms" title="Inspect gradient norms" class="md-nav__link">
Inspect gradient norms
</a>
</li>
<li class="md-nav__item">
<a href="#set-how-much-of-the-training-set-to-check" class="md-nav__link">
<a href="#set-how-much-of-the-training-set-to-check" title="Set how much of the training set to check" class="md-nav__link">
Set how much of the training set to check
</a>
</li>
<li class="md-nav__item">
<a href="#packed-sequences-as-inputs" title="Packed sequences as inputs" class="md-nav__link">
Packed sequences as inputs
</a>
</li>
<li class="md-nav__item">
<a href="#truncated-back-propagation-through-time" title="Truncated Back Propagation Through Time" class="md-nav__link">
Truncated Back Propagation Through Time
</a>
</li>
@@ -632,7 +660,7 @@
<p>Below are all the things lightning automates for you in the training loop.</p>
<hr />
<h4 id="accumulated-gradients">Accumulated gradients</h4>
<p>Accumulated gradients runs K small batches of size N before doing a backwards pass. The effect is a large effective batch size of size KxN. </p>
<p>Accumulated gradients runs K small batches of size N before doing a backwards pass. The effect is a large effective batch size of size KxN.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (ie: no accumulated grads)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">accumulate_grad_batches</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
@@ -650,7 +678,7 @@
<hr />
<h4 id="early-stopping">Early stopping</h4>
<p>The trainer already sets up default early stopping for you.
<p>The trainer already sets up default early stopping for you.
To modify this behavior, pass in your own EarlyStopping callback.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
@@ -684,10 +712,10 @@ To modify this behavior, pass in your own EarlyStopping callback.</p>
<span class="c1"># without passing anything in, uses the default callback above</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">()</span>
<span class="c1"># pass in your own to override the default callback </span>
<span class="c1"># pass in your own to override the default callback</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">early_stop_callback</span><span class="o">=</span><span class="n">early_stop_callback</span><span class="p">)</span>
<span class="c1"># pass in None to disable it </span>
<span class="c1"># pass in None to disable it</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">early_stop_callback</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
</pre></div>
</td></tr></table>
@@ -747,6 +775,49 @@ Specifically, this will <a href="https://pytorch.org/docs/stable/nn.html#torch.n
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">train_percent_check</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="packed-sequences-as-inputs">Packed sequences as inputs</h4>
<p>When using PackedSequence, do 2 things:
1. return either a padded tensor in dataset or a list of variable length tensors in the dataloader collate_fn (example above shows the list implementation). <br />
2. Pack the sequence in forward or training and validation steps depending on use case.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># For use in dataloader</span>
<span class="k">def</span> <span class="nf">collate_fn</span><span class="p">(</span><span class="n">batch</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="p">[</span><span class="n">item</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">batch</span><span class="p">]</span>
<span class="n">y</span> <span class="o">=</span> <span class="p">[</span><span class="n">item</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">batch</span><span class="p">]</span>
<span class="k">return</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span>
<span class="c1"># In module</span>
<span class="k">def</span> <span class="nf">training_step</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">,</span> <span class="n">batch_nb</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">pack_sequence</span><span class="p">(</span><span class="n">batch</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">enforce_sorted</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">pack_sequence</span><span class="p">(</span><span class="n">batch</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">enforce_sorted</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="truncated-back-propagation-through-time">Truncated Back Propagation Through Time</h4>
<p>There are times when multiple backwards passes are needed for each batch. For example, it may save memory to use Truncated Back Propagation Through Time when training RNNs.</p>
<p>When this flag is enabled each batch is split into sequences of size truncated_bptt_steps and passed to training_step(...) separately. A default splitting function is provided, however, you can override it for more flexibility. See <a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks#tbptt_split_batch">tbptt_split_batch</a>.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (single backwards pass per batch)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">truncated_bptt_steps</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
<span class="c1"># (split batch into sequences of size 2)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">truncated_bptt_steps</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
</pre></div>
</td></tr></table>
@@ -816,7 +887,7 @@ Specifically, this will <a href="https://pytorch.org/docs/stable/nn.html#torch.n
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/Validation loop/index.html
+25 -16
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -416,35 +416,35 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#check-validation-every-n-epochs" class="md-nav__link">
<a href="#check-validation-every-n-epochs" title="Check validation every n epochs" class="md-nav__link">
Check validation every n epochs
</a>
</li>
<li class="md-nav__item">
<a href="#set-how-much-of-the-validation-set-to-check" class="md-nav__link">
<a href="#set-how-much-of-the-validation-set-to-check" title="Set how much of the validation set to check" class="md-nav__link">
Set how much of the validation set to check
</a>
</li>
<li class="md-nav__item">
<a href="#set-how-much-of-the-test-set-to-check" class="md-nav__link">
<a href="#set-how-much-of-the-test-set-to-check" title="Set how much of the test set to check" class="md-nav__link">
Set how much of the test set to check
</a>
</li>
<li class="md-nav__item">
<a href="#set-validation-check-frequency-within-1-training-epoch" class="md-nav__link">
<a href="#set-validation-check-frequency-within-1-training-epoch" title="Set validation check frequency within 1 training epoch" class="md-nav__link">
Set validation check frequency within 1 training epoch
</a>
</li>
<li class="md-nav__item">
<a href="#set-the-number-of-validation-sanity-steps" class="md-nav__link">
<a href="#set-the-number-of-validation-sanity-steps" title="Set the number of validation sanity steps" class="md-nav__link">
Set the number of validation sanity steps
</a>
@@ -545,35 +545,35 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#check-validation-every-n-epochs" class="md-nav__link">
<a href="#check-validation-every-n-epochs" title="Check validation every n epochs" class="md-nav__link">
Check validation every n epochs
</a>
</li>
<li class="md-nav__item">
<a href="#set-how-much-of-the-validation-set-to-check" class="md-nav__link">
<a href="#set-how-much-of-the-validation-set-to-check" title="Set how much of the validation set to check" class="md-nav__link">
Set how much of the validation set to check
</a>
</li>
<li class="md-nav__item">
<a href="#set-how-much-of-the-test-set-to-check" class="md-nav__link">
<a href="#set-how-much-of-the-test-set-to-check" title="Set how much of the test set to check" class="md-nav__link">
Set how much of the test set to check
</a>
</li>
<li class="md-nav__item">
<a href="#set-validation-check-frequency-within-1-training-epoch" class="md-nav__link">
<a href="#set-validation-check-frequency-within-1-training-epoch" title="Set validation check frequency within 1 training epoch" class="md-nav__link">
Set validation check frequency within 1 training epoch
</a>
</li>
<li class="md-nav__item">
<a href="#set-the-number-of-validation-sanity-steps" class="md-nav__link">
<a href="#set-the-number-of-validation-sanity-steps" title="Set the number of validation sanity steps" class="md-nav__link">
Set the number of validation sanity steps
</a>
@@ -647,16 +647,25 @@ Lightning will run 5 steps of validation in the beginning of training as a sanit
<hr />
<h4 id="set-validation-check-frequency-within-1-training-epoch">Set validation check frequency within 1 training epoch</h4>
<p>For large datasets it's often desirable to check validation multiple times within a training loop</p>
<p>For large datasets it's often desirable to check validation multiple times within a training loop.
Pass in a float to check that often within 1 training epoch.
Pass in an int k to check every k training batches. Must use an int if using
an IterableDataset.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT</span>
5
6
7
8</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">val_check_interval</span><span class="o">=</span><span class="mf">0.95</span><span class="p">)</span>
<span class="c1"># check every .25 of an epoch </span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">val_check_interval</span><span class="o">=</span><span class="mf">0.25</span><span class="p">)</span>
<span class="c1"># check every 100 train batches (ie: for IterableDatasets or fixed frequency)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">val_check_interval</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
</pre></div>
</td></tr></table>
@@ -739,7 +748,7 @@ Lightning will run 5 steps of validation in the beginning of training as a sanit
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/debugging/index.html
+16 -16
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -428,42 +428,42 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#fast-dev-run" class="md-nav__link">
<a href="#fast-dev-run" title="Fast dev run" class="md-nav__link">
Fast dev run
</a>
</li>
<li class="md-nav__item">
<a href="#inspect-gradient-norms" class="md-nav__link">
<a href="#inspect-gradient-norms" title="Inspect gradient norms" class="md-nav__link">
Inspect gradient norms
</a>
</li>
<li class="md-nav__item">
<a href="#make-model-overfit-on-subset-of-data" class="md-nav__link">
<a href="#make-model-overfit-on-subset-of-data" title="Make model overfit on subset of data" class="md-nav__link">
Make model overfit on subset of data
</a>
</li>
<li class="md-nav__item">
<a href="#print-the-parameter-count-by-layer" class="md-nav__link">
<a href="#print-the-parameter-count-by-layer" title="Print the parameter count by layer" class="md-nav__link">
Print the parameter count by layer
</a>
</li>
<li class="md-nav__item">
<a href="#print-which-gradients-are-nan" class="md-nav__link">
<a href="#print-which-gradients-are-nan" title="Print which gradients are nan" class="md-nav__link">
Print which gradients are nan
</a>
</li>
<li class="md-nav__item">
<a href="#log-gpu-usage" class="md-nav__link">
<a href="#log-gpu-usage" title="Log GPU usage" class="md-nav__link">
Log GPU usage
</a>
@@ -552,42 +552,42 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#fast-dev-run" class="md-nav__link">
<a href="#fast-dev-run" title="Fast dev run" class="md-nav__link">
Fast dev run
</a>
</li>
<li class="md-nav__item">
<a href="#inspect-gradient-norms" class="md-nav__link">
<a href="#inspect-gradient-norms" title="Inspect gradient norms" class="md-nav__link">
Inspect gradient norms
</a>
</li>
<li class="md-nav__item">
<a href="#make-model-overfit-on-subset-of-data" class="md-nav__link">
<a href="#make-model-overfit-on-subset-of-data" title="Make model overfit on subset of data" class="md-nav__link">
Make model overfit on subset of data
</a>
</li>
<li class="md-nav__item">
<a href="#print-the-parameter-count-by-layer" class="md-nav__link">
<a href="#print-the-parameter-count-by-layer" title="Print the parameter count by layer" class="md-nav__link">
Print the parameter count by layer
</a>
</li>
<li class="md-nav__item">
<a href="#print-which-gradients-are-nan" class="md-nav__link">
<a href="#print-which-gradients-are-nan" title="Print which gradients are nan" class="md-nav__link">
Print which gradients are nan
</a>
</li>
<li class="md-nav__item">
<a href="#log-gpu-usage" class="md-nav__link">
<a href="#log-gpu-usage" title="Log GPU usage" class="md-nav__link">
Log GPU usage
</a>
@@ -752,7 +752,7 @@ Use this to debug a full run of your program quickly</p>
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/hooks/index.html
+311 -22
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -444,68 +444,103 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#on_epoch_start" class="md-nav__link">
<a href="#on_epoch_start" title="on_epoch_start" class="md-nav__link">
on_epoch_start
</a>
</li>
<li class="md-nav__item">
<a href="#on_epoch_end" class="md-nav__link">
<a href="#on_epoch_end" title="on_epoch_end" class="md-nav__link">
on_epoch_end
</a>
</li>
<li class="md-nav__item">
<a href="#on_batch_start" class="md-nav__link">
<a href="#on_batch_start" title="on_batch_start" class="md-nav__link">
on_batch_start
</a>
</li>
<li class="md-nav__item">
<a href="#on_batch_end" class="md-nav__link">
<a href="#on_batch_end" title="on_batch_end" class="md-nav__link">
on_batch_end
</a>
</li>
<li class="md-nav__item">
<a href="#on_pre_performance_check" class="md-nav__link">
<a href="#on_pre_performance_check" title="on_pre_performance_check" class="md-nav__link">
on_pre_performance_check
</a>
</li>
<li class="md-nav__item">
<a href="#on_post_performance_check" class="md-nav__link">
<a href="#on_post_performance_check" title="on_post_performance_check" class="md-nav__link">
on_post_performance_check
</a>
</li>
<li class="md-nav__item">
<a href="#optimizer_step" class="md-nav__link">
<a href="#optimizer_step" title="optimizer_step" class="md-nav__link">
optimizer_step
</a>
</li>
<li class="md-nav__item">
<a href="#on_before_zero_grad" class="md-nav__link">
<a href="#on_before_zero_grad" title="on_before_zero_grad" class="md-nav__link">
on_before_zero_grad
</a>
</li>
<li class="md-nav__item">
<a href="#on_after_backward" class="md-nav__link">
<a href="#backward" title="backward" class="md-nav__link">
backward
</a>
</li>
<li class="md-nav__item">
<a href="#on_after_backward" title="on_after_backward" class="md-nav__link">
on_after_backward
</a>
</li>
<li class="md-nav__item">
<a href="#tbptt_split_batch" title="tbptt_split_batch" class="md-nav__link">
tbptt_split_batch
</a>
</li>
<li class="md-nav__item">
<a href="#configure_apex" title="configure_apex" class="md-nav__link">
configure_apex
</a>
</li>
<li class="md-nav__item">
<a href="#configure_ddp" title="configure_ddp" class="md-nav__link">
configure_ddp
</a>
</li>
<li class="md-nav__item">
<a href="#init_ddp_connection" title="init_ddp_connection" class="md-nav__link">
init_ddp_connection
</a>
</li>
@@ -579,68 +614,103 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#on_epoch_start" class="md-nav__link">
<a href="#on_epoch_start" title="on_epoch_start" class="md-nav__link">
on_epoch_start
</a>
</li>
<li class="md-nav__item">
<a href="#on_epoch_end" class="md-nav__link">
<a href="#on_epoch_end" title="on_epoch_end" class="md-nav__link">
on_epoch_end
</a>
</li>
<li class="md-nav__item">
<a href="#on_batch_start" class="md-nav__link">
<a href="#on_batch_start" title="on_batch_start" class="md-nav__link">
on_batch_start
</a>
</li>
<li class="md-nav__item">
<a href="#on_batch_end" class="md-nav__link">
<a href="#on_batch_end" title="on_batch_end" class="md-nav__link">
on_batch_end
</a>
</li>
<li class="md-nav__item">
<a href="#on_pre_performance_check" class="md-nav__link">
<a href="#on_pre_performance_check" title="on_pre_performance_check" class="md-nav__link">
on_pre_performance_check
</a>
</li>
<li class="md-nav__item">
<a href="#on_post_performance_check" class="md-nav__link">
<a href="#on_post_performance_check" title="on_post_performance_check" class="md-nav__link">
on_post_performance_check
</a>
</li>
<li class="md-nav__item">
<a href="#optimizer_step" class="md-nav__link">
<a href="#optimizer_step" title="optimizer_step" class="md-nav__link">
optimizer_step
</a>
</li>
<li class="md-nav__item">
<a href="#on_before_zero_grad" class="md-nav__link">
<a href="#on_before_zero_grad" title="on_before_zero_grad" class="md-nav__link">
on_before_zero_grad
</a>
</li>
<li class="md-nav__item">
<a href="#on_after_backward" class="md-nav__link">
<a href="#backward" title="backward" class="md-nav__link">
backward
</a>
</li>
<li class="md-nav__item">
<a href="#on_after_backward" title="on_after_backward" class="md-nav__link">
on_after_backward
</a>
</li>
<li class="md-nav__item">
<a href="#tbptt_split_batch" title="tbptt_split_batch" class="md-nav__link">
tbptt_split_batch
</a>
</li>
<li class="md-nav__item">
<a href="#configure_apex" title="configure_apex" class="md-nav__link">
configure_apex
</a>
</li>
<li class="md-nav__item">
<a href="#configure_ddp" title="configure_ddp" class="md-nav__link">
configure_ddp
</a>
</li>
<li class="md-nav__item">
<a href="#init_ddp_connection" title="init_ddp_connection" class="md-nav__link">
init_ddp_connection
</a>
</li>
@@ -807,6 +877,39 @@ Good place to inspect weight information with weights updated.</p>
</pre></div>
</td></tr></table>
<hr />
<h4 id="backward">backward</h4>
<p>Called to perform backward step.
Feel free to override as needed.</p>
<p>The loss passed in has already been scaled for accumulated gradients if requested.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">backward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">use_amp</span><span class="p">,</span> <span class="n">loss</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Override backward with your own implementation if you need to</span>
<span class="sd"> :param use_amp: Whether amp was requested or not</span>
<span class="sd"> :param loss: Loss is already scaled by accumulated grads</span>
<span class="sd"> :param optimizer: Current optimizer being used</span>
<span class="sd"> :return:</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="n">use_amp</span><span class="p">:</span>
<span class="k">with</span> <span class="n">amp</span><span class="o">.</span><span class="n">scale_loss</span><span class="p">(</span><span class="n">loss</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">)</span> <span class="k">as</span> <span class="n">scaled_loss</span><span class="p">:</span>
<span class="n">scaled_loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="on_after_backward">on_after_backward</h4>
<p>Called in the training loop after model.backward()
@@ -828,6 +931,192 @@ This is the ideal place to inspect or log gradient information </p>
<span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">experiment</span><span class="o">.</span><span class="n">add_histogram</span><span class="p">(</span><span class="n">tag</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">values</span><span class="o">=</span><span class="n">grads</span><span class="p">,</span> <span class="n">global_step</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">trainer</span><span class="o">.</span><span class="n">global_step</span><span class="p">)</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="tbptt_split_batch">tbptt_split_batch</h4>
<p>Called in the training loop after on_batch_start if <code>truncated_bptt_steps &gt; 0</code>. Each returned batch split is passed separately to training_step(...).</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">tbptt_split_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">,</span> <span class="n">split_size</span><span class="p">):</span>
<span class="n">splits</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">time_dims</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">split_size</span><span class="p">):</span>
<span class="n">batch_split</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">batch</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">):</span>
<span class="n">split_x</span> <span class="o">=</span> <span class="n">x</span><span class="p">[:,</span> <span class="n">t</span><span class="p">:</span><span class="n">t</span> <span class="o">+</span> <span class="n">split_size</span><span class="p">]</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="p">):</span>
<span class="n">split_x</span> <span class="o">=</span> <span class="p">[</span><span class="bp">None</span><span class="p">]</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="k">for</span> <span class="n">batch_idx</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">)):</span>
<span class="n">split_x</span><span class="p">[</span><span class="n">batch_idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="n">batch_idx</span><span class="p">][</span><span class="n">t</span><span class="p">:</span><span class="n">t</span> <span class="o">+</span> <span class="n">split_size</span><span class="p">]</span>
<span class="n">batch_split</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">split_x</span><span class="p">)</span>
<span class="n">splits</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">batch_split</span><span class="p">)</span>
<span class="k">return</span> <span class="n">splits</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="configure_apex">configure_apex</h4>
<p>Overwrite to define your own Apex implementation init.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">configure_apex</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">amp</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">optimizers</span><span class="p">,</span> <span class="n">amp_level</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Override to init AMP your own way</span>
<span class="sd"> Must return a model and list of optimizers</span>
<span class="sd"> :param amp:</span>
<span class="sd"> :param model:</span>
<span class="sd"> :param optimizers:</span>
<span class="sd"> :param amp_level:</span>
<span class="sd"> :return: Apex wrapped model and optimizers</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">model</span><span class="p">,</span> <span class="n">optimizers</span> <span class="o">=</span> <span class="n">amp</span><span class="o">.</span><span class="n">initialize</span><span class="p">(</span>
<span class="n">model</span><span class="p">,</span> <span class="n">optimizers</span><span class="p">,</span> <span class="n">opt_level</span><span class="o">=</span><span class="n">amp_level</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">model</span><span class="p">,</span> <span class="n">optimizers</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="configure_ddp">configure_ddp</h4>
<p>Overwrite to define your own DDP implementation init.
The only requirement is that:
1. On a validation batch the call goes to model.validation_step. <br />
2. On a training batch the call goes to model.training_step. <br />
3. On a testing batch, the call goes to model.test_step</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">configure_ddp</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device_ids</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Override to init DDP in a different way or use your own wrapper.</span>
<span class="sd"> Must return model.</span>
<span class="sd"> :param model:</span>
<span class="sd"> :param device_ids:</span>
<span class="sd"> :return: DDP wrapped model</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Lightning DDP simply routes to test_step, val_step, etc...</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">LightningDistributedDataParallel</span><span class="p">(</span>
<span class="n">model</span><span class="p">,</span>
<span class="n">device_ids</span><span class="o">=</span><span class="n">device_ids</span><span class="p">,</span>
<span class="n">find_unused_parameters</span><span class="o">=</span><span class="bp">True</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">model</span>
</pre></div>
</td></tr></table>
<hr />
<h4 id="init_ddp_connection">init_ddp_connection</h4>
<p>Override to init DDP in your own way. </p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">init_ddp_connection</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Connect all procs in the world using the env:// init</span>
<span class="sd"> Use the first node as the root address</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># use slurm job id for the port number</span>
<span class="c1"># guarantees unique ports across jobs from same grid search</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># use the last 4 numbers in the job id as the id</span>
<span class="n">default_port</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">&#39;SLURM_JOB_ID&#39;</span><span class="p">]</span>
<span class="n">default_port</span> <span class="o">=</span> <span class="n">default_port</span><span class="p">[</span><span class="o">-</span><span class="mi">4</span><span class="p">:]</span>
<span class="c1"># all ports should be in the 10k+ range</span>
<span class="n">default_port</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">default_port</span><span class="p">)</span> <span class="o">+</span> <span class="mi">15000</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">default_port</span> <span class="o">=</span> <span class="mi">12910</span>
<span class="c1"># if user gave a port number, use that one instead</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">default_port</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">&#39;MASTER_PORT&#39;</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">&#39;MASTER_PORT&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">default_port</span><span class="p">)</span>
<span class="c1"># figure out the root node addr</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">root_node</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">&#39;SLURM_NODELIST&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">&#39; &#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
<span class="n">root_node</span> <span class="o">=</span> <span class="s1">&#39;127.0.0.2&#39;</span>
<span class="n">root_node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">trainer</span><span class="o">.</span><span class="n">resolve_root_node_address</span><span class="p">(</span><span class="n">root_node</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">&#39;MASTER_ADDR&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">root_node</span>
<span class="n">dist</span><span class="o">.</span><span class="n">init_process_group</span><span class="p">(</span><span class="s1">&#39;nccl&#39;</span><span class="p">,</span> <span class="n">rank</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">proc_rank</span><span class="p">,</span> <span class="n">world_size</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">world_size</span><span class="p">)</span>
</pre></div>
</td></tr></table>
@@ -897,7 +1186,7 @@ This is the ideal place to inspect or log gradient information </p>
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>
Trainer/index.html
+7 -4
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -546,6 +546,7 @@
</ul>
<p><strong>Distributed training</strong> </p>
<ul>
<li><a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#init_ddp_connection">Implement Your Own Distributed (DDP) training</a></li>
<li><a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision">16-bit mixed precision</a></li>
<li><a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU">Multi-GPU</a></li>
<li><a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node">Multi-node</a></li>
@@ -574,6 +575,8 @@
<li><a href="https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers">Use multiple optimizers (like GANs)</a></li>
<li><a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check">Set how much of the training set to check (1-100%)</a></li>
<li><a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step">Step optimizers at arbitrary intervals</a></li>
<li><a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#packed-sequences-as-inputs">Packed sequences</a></li>
<li><a href="https://williamfalcon.github.io/pytorch-lightning//Training%20Loop/#truncated-back-propation-through-time">Truncated Back Propagation Through Time</a></li>
</ul>
<p><strong>Validation loop</strong> </p>
<ul>
@@ -657,7 +660,7 @@
</div>
<script src="../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:".."}})</script>