mirror of
https://github.com/wassname/pytorch-lightning.git
synced 2026-09-10 12:21:57 +08:00
Deployed acb4ebe with MkDocs version: 1.0.4
This commit is contained in:
William Falcon
@@ -402,6 +402,13 @@
|
||||
Multi-node
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#auto-slurm-job-submission" class="md-nav__link">
|
||||
Auto-slurm-job-submission
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -612,6 +619,13 @@
|
||||
Multi-node
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#auto-slurm-job-submission" class="md-nav__link">
|
||||
Auto-slurm-job-submission
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -649,6 +663,7 @@ None of the flags below require changing anything about your lightningModel defi
|
||||
<h4 id="choosing-a-backend">Choosing a backend</h4>
|
||||
<p>Lightning supports two backends. DataParallel and DistributedDataParallel. Both can be used for single-node multi-GPU training.
|
||||
For multi-node training you must use DistributedDataParallel. </p>
|
||||
<p><strong>Warning: Your cluster must have NCCL installed and you must load it when submitting your SLURM script</strong></p>
|
||||
<p>You can toggle between each mode by setting this flag.</p>
|
||||
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
|
||||
2
|
||||
@@ -834,14 +849,15 @@ In this setting, the model will run on all 8 GPUs at once using DataParallel und
|
||||
|
||||
<hr />
|
||||
<h4 id="multi-node">Multi-node</h4>
|
||||
<p>Multi-node training is easily done by specifying these flags.</p>
|
||||
<p>Multi-node training is easily done by specifying these flags. </p>
|
||||
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
|
||||
2</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># train on 12*8 GPUs</span>
|
||||
<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">nb_gpu_nodes</span><span class="o">=</span><span class="mi">12</span><span class="p">,</span> <span class="n">distributed_backend</span><span class="o">=</span><span class="s1">'ddp'</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</td></tr></table>
|
||||
|
||||
<p>In addition, make sure to set up your SLURM job correctly via the <a href="https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/">SlurmClusterObject</a>. In particular, specify the number of tasks per node correctly.</p>
|
||||
<p>You must configure your job submission script correctly for the trainer to work. Here is an example
|
||||
script for the above trainer configuration. </p>
|
||||
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
|
||||
2
|
||||
3
|
||||
@@ -865,30 +881,50 @@ In this setting, the model will run on all 8 GPUs at once using DataParallel und
|
||||
21
|
||||
22
|
||||
23
|
||||
24</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="n">cluster</span> <span class="o">=</span> <span class="n">SlurmCluster</span><span class="p">(</span>
|
||||
<span class="n">hyperparam_optimizer</span><span class="o">=</span><span class="n">test_tube</span><span class="o">.</span><span class="n">HyperOptArgumentParser</span><span class="p">(),</span>
|
||||
<span class="n">log_path</span><span class="o">=</span><span class="s1">'/some/path/to/save'</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
24
|
||||
25
|
||||
26
|
||||
27
|
||||
28
|
||||
29
|
||||
30
|
||||
31
|
||||
32
|
||||
33
|
||||
34</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="ch">#!/bin/bash -l</span>
|
||||
|
||||
<span class="c1"># OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT</span>
|
||||
<span class="c1"># which interface your nodes use for communication</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_command</span><span class="p">(</span><span class="s1">'export NCCL_SOCKET_IFNAME=^docker0,lo'</span><span class="p">)</span>
|
||||
<span class="c1"># SLURM SUBMIT SCRIPT</span>
|
||||
<span class="c1">#SBATCH --nodes=12</span>
|
||||
<span class="c1">#SBATCH --gres=gpu:8</span>
|
||||
<span class="c1">#SBATCH --ntasks-per-node=8</span>
|
||||
<span class="c1">#SBATCH --mem=0</span>
|
||||
<span class="c1">#SBATCH --time=0-02:00:00</span>
|
||||
|
||||
<span class="c1"># see output of the NCCL connection process</span>
|
||||
<span class="c1"># NCCL is how the nodes talk to each other</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_command</span><span class="p">(</span><span class="s1">'export NCCL_DEBUG=INFO'</span><span class="p">)</span>
|
||||
<span class="c1"># activate conda env</span>
|
||||
conda activate my_env
|
||||
|
||||
<span class="c1"># setting a master port here is a good idea.</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_command</span><span class="p">(</span><span class="s1">'export MASTER_PORT=</span><span class="si">%r</span><span class="s1">'</span> <span class="o">%</span> <span class="n">PORT</span><span class="p">)</span>
|
||||
<span class="c1"># REQUIRED: Load the latest NCCL version</span>
|
||||
<span class="c1"># the nccl version must match the cuda used to build your PyTorch distribution </span>
|
||||
<span class="c1"># (ie: which instructions did you follow when installing PyTorch)</span>
|
||||
<span class="c1"># module load NCCL/2.4.7-1-cuda.10.0</span>
|
||||
|
||||
<span class="c1"># good to load the latest NCCL version</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">load_modules</span><span class="p">([</span><span class="s1">'NCCL/2.4.7-1-cuda.10.0'</span><span class="p">])</span>
|
||||
<span class="c1"># -------------------------</span>
|
||||
<span class="c1"># OPTIONAL</span>
|
||||
<span class="c1"># -------------------------</span>
|
||||
<span class="c1"># debugging flags (optional)</span>
|
||||
<span class="c1"># export NCCL_DEBUG=INFO</span>
|
||||
<span class="c1"># export PYTHONFAULTHANDLER=1</span>
|
||||
|
||||
<span class="c1"># configure cluster</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">per_experiment_nb_nodes</span> <span class="o">=</span> <span class="mi">12</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">per_experiment_nb_gpus</span> <span class="o">=</span> <span class="mi">8</span>
|
||||
<span class="c1"># on your cluster you might need these:</span>
|
||||
<span class="c1"># set the network interface</span>
|
||||
<span class="c1"># export NCCL_SOCKET_IFNAME=^docker0,lo</span>
|
||||
<span class="c1"># -------------------------</span>
|
||||
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_slurm_cmd</span><span class="p">(</span><span class="n">cmd</span><span class="o">=</span><span class="s1">'ntasks-per-node'</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">comment</span><span class="o">=</span><span class="s1">'1 task per gpu'</span><span class="p">)</span>
|
||||
<span class="c1"># random port between 12k and 20k</span>
|
||||
<span class="nb">export</span> <span class="nv">MASTER_PORT</span><span class="o">=</span><span class="k">$((</span><span class="m">12000</span> <span class="o">+</span> RANDOM <span class="o">%</span> <span class="m">20000</span><span class="k">))</span>
|
||||
|
||||
<span class="c1"># run script from above</span>
|
||||
python my_main_file.py
|
||||
</pre></div>
|
||||
</td></tr></table>
|
||||
|
||||
@@ -914,6 +950,96 @@ portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).
|
||||
</pre></div>
|
||||
</td></tr></table>
|
||||
|
||||
<h4 id="auto-slurm-job-submission">Auto-slurm-job-submission</h4>
|
||||
<p>Instead of manually building SLURM scripts, you can use the <a href="https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/">SlurmCluster object</a> to
|
||||
do this for you. The SlurmCluster can also run a grid search if you pass in a <a href="https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/">HyperOptArgumentParser</a>.</p>
|
||||
<p>Here is an example where you run a grid search of 9 combinations of hyperparams.
|
||||
<a href="https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/new_project_templates/multi_node_examples">The full examples are here</a>.</p>
|
||||
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
12
|
||||
13
|
||||
14
|
||||
15
|
||||
16
|
||||
17
|
||||
18
|
||||
19
|
||||
20
|
||||
21
|
||||
22
|
||||
23
|
||||
24
|
||||
25
|
||||
26
|
||||
27
|
||||
28
|
||||
29
|
||||
30
|
||||
31
|
||||
32
|
||||
33
|
||||
34
|
||||
35
|
||||
36
|
||||
37
|
||||
38
|
||||
39
|
||||
40
|
||||
41</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># grid search 3 values of learning rate and 3 values of number of layers for your net</span>
|
||||
<span class="c1"># this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)</span>
|
||||
<span class="n">parser</span> <span class="o">=</span> <span class="n">HyperOptArgumentParser</span><span class="p">(</span><span class="n">strategy</span><span class="o">=</span><span class="s1">'grid_search'</span><span class="p">,</span> <span class="n">add_help</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
|
||||
<span class="n">parser</span><span class="o">.</span><span class="n">opt_list</span><span class="p">(</span><span class="s1">'--learning_rate'</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">options</span><span class="o">=</span><span class="p">[</span><span class="mf">1e-3</span><span class="p">,</span> <span class="mf">1e-2</span><span class="p">,</span> <span class="mf">1e-1</span><span class="p">],</span> <span class="n">tunable</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
|
||||
<span class="n">parser</span><span class="o">.</span><span class="n">opt_list</span><span class="p">(</span><span class="s1">'--layers'</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">options</span><span class="o">=</span><span class="p">[</span><span class="mi">16</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">64</span><span class="p">],</span> <span class="n">tunable</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
|
||||
<span class="n">hyperparams</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
|
||||
|
||||
<span class="c1"># Slurm cluster submits 9 jobs, each with a set of hyperparams</span>
|
||||
<span class="n">cluster</span> <span class="o">=</span> <span class="n">SlurmCluster</span><span class="p">(</span>
|
||||
<span class="n">hyperparam_optimizer</span><span class="o">=</span><span class="n">hyperparams</span><span class="p">,</span>
|
||||
<span class="n">log_path</span><span class="o">=</span><span class="s1">'/some/path/to/save'</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="c1"># OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT</span>
|
||||
<span class="c1"># which interface your nodes use for communication</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_command</span><span class="p">(</span><span class="s1">'export NCCL_SOCKET_IFNAME=^docker0,lo'</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># see output of the NCCL connection process</span>
|
||||
<span class="c1"># NCCL is how the nodes talk to each other</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_command</span><span class="p">(</span><span class="s1">'export NCCL_DEBUG=INFO'</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># setting a master port here is a good idea.</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_command</span><span class="p">(</span><span class="s1">'export MASTER_PORT=</span><span class="si">%r</span><span class="s1">'</span> <span class="o">%</span> <span class="n">PORT</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># ************** DON'T FORGET THIS ***************</span>
|
||||
<span class="c1"># MUST load the latest NCCL version</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">load_modules</span><span class="p">([</span><span class="s1">'NCCL/2.4.7-1-cuda.10.0'</span><span class="p">])</span>
|
||||
|
||||
<span class="c1"># configure cluster</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">per_experiment_nb_nodes</span> <span class="o">=</span> <span class="mi">12</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">per_experiment_nb_gpus</span> <span class="o">=</span> <span class="mi">8</span>
|
||||
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">add_slurm_cmd</span><span class="p">(</span><span class="n">cmd</span><span class="o">=</span><span class="s1">'ntasks-per-node'</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">comment</span><span class="o">=</span><span class="s1">'1 task per gpu'</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># submit a script with 9 combinations of hyper params</span>
|
||||
<span class="c1"># (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)</span>
|
||||
<span class="n">cluster</span><span class="o">.</span><span class="n">optimize_parallel_cluster_gpu</span><span class="p">(</span>
|
||||
<span class="n">main</span><span class="p">,</span>
|
||||
<span class="n">nb_trials</span><span class="o">=</span><span class="mi">9</span><span class="p">,</span> <span class="c1"># how many permutations of the grid search to run</span>
|
||||
<span class="n">job_name</span><span class="o">=</span><span class="s1">'name_for_squeue'</span>
|
||||
<span class="p">)</span>
|
||||
</pre></div>
|
||||
</td></tr></table>
|
||||
|
||||
<p>The other option is that you generate scripts on your own via a bash command or use another library...</p>
|
||||
<hr />
|
||||
<h4 id="self-balancing-architecture">Self-balancing architecture</h4>
|
||||
<p>Here lightning distributes parts of your module across available GPUs to optimize for speed and memory. </p>
|
||||
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
Reference in New Issue
Block a user