wassname/pytorch-lightning
1
0
Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-09-09 11:32:07 +08:00
Code Issues Packages Projects Releases Wiki Activity

Deployed b35229d with MkDocs version: 1.0.4

Browse Source
This commit is contained in:
William Falcon
2019-11-06 15:08:12 -05:00
parent 65095b6cac
commit 2cc732ca16
46 changed files with 995 additions and 718 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Trainer/Distributed training/index.html
+51 -37
View File
@@ -34,7 +34,7 @@
<meta name="lang:search.tokenizer" content="[\s\-]+">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.3">
<meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.4.0">
@@ -42,7 +42,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/application.30686662.css">
<link rel="stylesheet" href="../../assets/stylesheets/application.0284f74d.css">
@@ -171,7 +171,7 @@
<main class="md-main" role="main">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
@@ -356,7 +356,7 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#choosing-a-backend" class="md-nav__link">
<a href="#choosing-a-backend" title="Choosing a backend" class="md-nav__link">
Choosing a backend
</a>
@@ -364,21 +364,21 @@
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#dataparallel-dp" class="md-nav__link">
<a href="#dataparallel-dp" title="DataParallel (dp)" class="md-nav__link">
DataParallel (dp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-ddp" class="md-nav__link">
<a href="#distributeddataparallel-ddp" title="DistributedDataParallel (ddp)" class="md-nav__link">
DistributedDataParallel (ddp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-2-ddp2" class="md-nav__link">
<a href="#distributeddataparallel-2-ddp2" title="DistributedDataParallel-2 (ddp2)" class="md-nav__link">
DistributedDataParallel-2 (ddp2)
</a>
@@ -390,56 +390,56 @@
</li>
<li class="md-nav__item">
<a href="#distributed-and-16-bit-precision" class="md-nav__link">
<a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
Distributed and 16-bit precision.
</a>
</li>
<li class="md-nav__item">
<a href="#cuda-flags" class="md-nav__link">
<a href="#cuda-flags" title="CUDA flags" class="md-nav__link">
CUDA flags
</a>
</li>
<li class="md-nav__item">
<a href="#16-bit-mixed-precision" class="md-nav__link">
<a href="#16-bit-mixed-precision" title="16-bit mixed precision" class="md-nav__link">
16-bit mixed precision
</a>
</li>
<li class="md-nav__item">
<a href="#single-gpu" class="md-nav__link">
<a href="#single-gpu" title="Single-gpu" class="md-nav__link">
Single-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-gpu" class="md-nav__link">
<a href="#multi-gpu" title="multi-gpu" class="md-nav__link">
multi-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-node" class="md-nav__link">
<a href="#multi-node" title="Multi-node" class="md-nav__link">
Multi-node
</a>
</li>
<li class="md-nav__item">
<a href="#auto-slurm-job-submission" class="md-nav__link">
<a href="#auto-slurm-job-submission" title="Auto-slurm-job-submission" class="md-nav__link">
Auto-slurm-job-submission
</a>
</li>
<li class="md-nav__item">
<a href="#self-balancing-architecture" class="md-nav__link">
<a href="#self-balancing-architecture" title="Self-balancing architecture" class="md-nav__link">
Self-balancing architecture
</a>
@@ -600,7 +600,7 @@
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#choosing-a-backend" class="md-nav__link">
<a href="#choosing-a-backend" title="Choosing a backend" class="md-nav__link">
Choosing a backend
</a>
@@ -608,21 +608,21 @@
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#dataparallel-dp" class="md-nav__link">
<a href="#dataparallel-dp" title="DataParallel (dp)" class="md-nav__link">
DataParallel (dp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-ddp" class="md-nav__link">
<a href="#distributeddataparallel-ddp" title="DistributedDataParallel (ddp)" class="md-nav__link">
DistributedDataParallel (ddp)
</a>
</li>
<li class="md-nav__item">
<a href="#distributeddataparallel-2-ddp2" class="md-nav__link">
<a href="#distributeddataparallel-2-ddp2" title="DistributedDataParallel-2 (ddp2)" class="md-nav__link">
DistributedDataParallel-2 (ddp2)
</a>
@@ -634,56 +634,56 @@
</li>
<li class="md-nav__item">
<a href="#distributed-and-16-bit-precision" class="md-nav__link">
<a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
Distributed and 16-bit precision.
</a>
</li>
<li class="md-nav__item">
<a href="#cuda-flags" class="md-nav__link">
<a href="#cuda-flags" title="CUDA flags" class="md-nav__link">
CUDA flags
</a>
</li>
<li class="md-nav__item">
<a href="#16-bit-mixed-precision" class="md-nav__link">
<a href="#16-bit-mixed-precision" title="16-bit mixed precision" class="md-nav__link">
16-bit mixed precision
</a>
</li>
<li class="md-nav__item">
<a href="#single-gpu" class="md-nav__link">
<a href="#single-gpu" title="Single-gpu" class="md-nav__link">
Single-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-gpu" class="md-nav__link">
<a href="#multi-gpu" title="multi-gpu" class="md-nav__link">
multi-gpu
</a>
</li>
<li class="md-nav__item">
<a href="#multi-node" class="md-nav__link">
<a href="#multi-node" title="Multi-node" class="md-nav__link">
Multi-node
</a>
</li>
<li class="md-nav__item">
<a href="#auto-slurm-job-submission" class="md-nav__link">
<a href="#auto-slurm-job-submission" title="Auto-slurm-job-submission" class="md-nav__link">
Auto-slurm-job-submission
</a>
</li>
<li class="md-nav__item">
<a href="#self-balancing-architecture" class="md-nav__link">
<a href="#self-balancing-architecture" title="Self-balancing architecture" class="md-nav__link">
Self-balancing architecture
</a>
@@ -814,21 +814,35 @@ not allow 16-bit and DP training. We tried to get this to work, but it's an issu
</tbody>
</table>
<p>You also have the option of specifying which GPUs to use by passing a list: </p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5
6
7
8</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (int)</span>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (int) specifies how many GPUs to use.</span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="n">k</span><span class="p">)</span>
<span class="c1"># Above is equivalent to </span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">k</span><span class="p">)))</span>
<span class="c1"># You specify which GPUs (don&#39;t use if running on cluster) </span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>
<span class="c1"># can also be a string</span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="s1">&#39;0, 1&#39;</span><span class="p">)</span>
<span class="c1"># can also be -1 or &#39;-1&#39;, this uses all available GPUs</span>
<span class="c1"># this is equivalent to list(range(torch.cuda.available_devices()))</span>
<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
</td></tr></table>
@@ -1025,7 +1039,7 @@ portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).
<p>Instead of manually building SLURM scripts, you can use the <a href="https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/">SlurmCluster object</a> to
do this for you. The SlurmCluster can also run a grid search if you pass in a <a href="https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/">HyperOptArgumentParser</a>.</p>
<p>Here is an example where you run a grid search of 9 combinations of hyperparams.
<a href="https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/new_project_templates/multi_node_examples">The full examples are here</a>.</p>
<a href="https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/new_project_templates/multi_node_examples">The full examples are here</a>.</p>
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
@@ -1184,7 +1198,7 @@ do this for you. The SlurmCluster can also run a grid search if you pass in a <a
</div>
<script src="../../assets/javascripts/application.ac79c3b0.js"></script>
<script src="../../assets/javascripts/application.245445c6.js"></script>
<script>app.initialize({version:"1.0.4",url:{base:"../.."}})</script>