wassname/pytorch-lightning
1
0
Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-09-10 12:21:57 +08:00
Code Issues Packages Projects Releases Wiki Activity

Deployed 0eb6950 with MkDocs version: 1.0.4

Browse Source
This commit is contained in:
William Falcon
2019-10-05 16:10:29 -05:00
parent c5d8042012
commit 09017dfd67
23 changed files with 575 additions and 482 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Trainer/Distributed training/index.html
+103 -38
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.2">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
@@ -171,7 +171,7 @@
<main class="md-main">
<main class="md-main" role="main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -356,63 +356,90 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#choosing-a-backend" title="Choosing a backend" class="md-nav__link">
<a href="#choosing-a-backend" class="md-nav__link">
Choosing a backend
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#dataparallel-dp" class="md-nav__link">
DataParallel (dp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-ddp" class="md-nav__link">
DistributedDataParallel (ddp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-2-ddp2" class="md-nav__link">
DistributedDataParallel-2 (ddp2)
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
<a href="#distributed-and-16-bit-precision" class="md-nav__link">
Distributed and 16-bit precision.
</a>
</li>
<li class="md-nav__item">
<a href="#cuda-flags" title="CUDA flags" class="md-nav__link">
<a href="#cuda-flags" class="md-nav__link">
CUDA flags
</a>
</li>
<li class="md-nav__item">
<a href="#16-bit-mixed-precision" title="16-bit mixed precision" class="md-nav__link">
<a href="#16-bit-mixed-precision" class="md-nav__link">
16-bit mixed precision
</a>
</li>
<li class="md-nav__item">
<a href="#single-gpu" title="Single-gpu" class="md-nav__link">
<a href="#single-gpu" class="md-nav__link">
Single-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-gpu" title="multi-gpu" class="md-nav__link">
<a href="#multi-gpu" class="md-nav__link">
multi-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-node" title="Multi-node" class="md-nav__link">
<a href="#multi-node" class="md-nav__link">
Multi-node
</a>
</li>
<li class="md-nav__item">
<a href="#auto-slurm-job-submission" title="Auto-slurm-job-submission" class="md-nav__link">
<a href="#auto-slurm-job-submission" class="md-nav__link">
Auto-slurm-job-submission
</a>
</li>
<li class="md-nav__item">
<a href="#self-balancing-architecture" title="Self-balancing architecture" class="md-nav__link">
<a href="#self-balancing-architecture" class="md-nav__link">
Self-balancing architecture
</a>
@@ -573,63 +600,90 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#choosing-a-backend" title="Choosing a backend" class="md-nav__link">
<a href="#choosing-a-backend" class="md-nav__link">
Choosing a backend
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#dataparallel-dp" class="md-nav__link">
DataParallel (dp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-ddp" class="md-nav__link">
DistributedDataParallel (ddp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-2-ddp2" class="md-nav__link">
DistributedDataParallel-2 (ddp2)
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
<a href="#distributed-and-16-bit-precision" class="md-nav__link">
Distributed and 16-bit precision.
</a>
</li>
<li class="md-nav__item">
<a href="#cuda-flags" title="CUDA flags" class="md-nav__link">
<a href="#cuda-flags" class="md-nav__link">
CUDA flags
</a>
</li>
<li class="md-nav__item">
<a href="#16-bit-mixed-precision" title="16-bit mixed precision" class="md-nav__link">
<a href="#16-bit-mixed-precision" class="md-nav__link">
16-bit mixed precision
</a>
</li>
<li class="md-nav__item">
<a href="#single-gpu" title="Single-gpu" class="md-nav__link">
<a href="#single-gpu" class="md-nav__link">
Single-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-gpu" title="multi-gpu" class="md-nav__link">
<a href="#multi-gpu" class="md-nav__link">
multi-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-node" title="Multi-node" class="md-nav__link">
<a href="#multi-node" class="md-nav__link">
Multi-node
</a>
</li>
<li class="md-nav__item">
<a href="#auto-slurm-job-submission" title="Auto-slurm-job-submission" class="md-nav__link">
<a href="#auto-slurm-job-submission" class="md-nav__link">
Auto-slurm-job-submission
</a>
</li>
<li class="md-nav__item">
<a href="#self-balancing-architecture" title="Self-balancing architecture" class="md-nav__link">
<a href="#self-balancing-architecture" class="md-nav__link">
Self-balancing architecture
</a>
@@ -663,16 +717,26 @@ None of the flags below require changing anything about your lightningModel defi
<h4 id="choosing-a-backend">Choosing a backend</h4>
<p>Lightning supports two backends. DataParallel and DistributedDataParallel. Both can be used for single-node multi-GPU training.
For multi-node training you must use DistributedDataParallel. </p>
<p><strong>Warning: Your cluster must have NCCL installed and you must load it when submitting your SLURM script</strong></p>
<h5 id="dataparallel-dp">DataParallel (dp)</h5>
<p>Splits a batch across multiple GPUs on the same node. Cannot be used for multi-node training. </p>
<h5 id="distributeddataparallel-ddp">DistributedDataParallel (ddp)</h5>
<p>Trains a copy of the model on each GPU and only syncs gradients. If used with DistributedSampler, each GPU trains
on a subset of the full dataset. </p>
<h5 id="distributeddataparallel-2-ddp2">DistributedDataParallel-2 (ddp2)</h5>
<p>Works like DDP, except each node trains a single copy of the model using ALL GPUs on that node.
Very useful when dealing with negative samples, etc...</p>
<p>You can toggle between each mode by setting this flag.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5
6
7
8</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (when using single GPU or no GPUs)</span>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (when using single GPU or no GPUs)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">distributed_backend</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
<span class="c1"># Change to DataParallel (gpus &gt; 1)</span>
@@ -680,6 +744,9 @@ For multi-node training you must use DistributedDataParallel. </p>
<span class="c1"># change to distributed data parallel (gpus &gt; 1)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;ddp&#39;</span><span class="p">)</span>
<span class="c1"># change to distributed data parallel (gpus &gt; 1)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;ddp2&#39;</span><span class="p">)</span>
</pre></div>
</td></tr></table>
@@ -896,8 +963,7 @@ script for the above trainer configuration. </p>
30
31
32
33
34</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="ch">#!/bin/bash -l</span>
33</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="ch">#!/bin/bash -l</span>
<span class="c1"># SLURM SUBMIT SCRIPT</span>
<span class="c1">#SBATCH --nodes=12</span>
@@ -909,11 +975,6 @@ script for the above trainer configuration. </p>
<span class="c1"># activate conda env</span>
conda activate my_env
<span class="c1"># REQUIRED: Load the latest NCCL version</span>
<span class="c1"># the nccl version must match the cuda used to build your PyTorch distribution </span>
<span class="c1"># (ie: which instructions did you follow when installing PyTorch)</span>
<span class="c1"># module load NCCL/2.4.7-1-cuda.10.0</span>
<span class="c1"># -------------------------</span>
<span class="c1"># OPTIONAL</span>
<span class="c1"># -------------------------</span>
@@ -921,6 +982,10 @@ conda activate my_env
<span class="c1"># export NCCL_DEBUG=INFO</span>
<span class="c1"># export PYTHONFAULTHANDLER=1</span>
<span class="c1"># PyTorch comes with prebuilt NCCL support... but if you have issues with it</span>
<span class="c1"># you might need to load the latest version from your modules</span>
<span class="c1"># module load NCCL/2.4.7-1-cuda.10.0</span>
<span class="c1"># on your cluster you might need these:</span>
<span class="c1"># set the network interface</span>
<span class="c1"># export NCCL_SOCKET_IFNAME=^docker0,lo</span>
@@ -1119,7 +1184,7 @@ do this for you. The SlurmCluster can also run a grid search if you pass in a <a
</div>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script src="../../assets/javascripts/application.c648116f.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>