wassname/pytorch-lightning
1
0
Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-09-10 12:21:57 +08:00
Code Issues Packages Projects Releases Wiki Activity

updated docs

Browse Source
This commit is contained in:
William Falcon
2019-09-16 11:02:04 -04:00
parent b354988255
commit 286625a02f
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Trainer/Distributed training.md
+4 -1
View File
@@ -108,7 +108,7 @@ trainer = Trainer(gpus=8, distributed_backend='ddp')
Multi-node training is easily done by specifying these flags. Multi-node training is easily done by specifying these flags.
```python ```python
# train on 12*8 GPUs # train on 12*8 GPUs
trainer = Trainer(gpus=8, nb_gpu_nodes=12) trainer = Trainer(gpus=8, nb_gpu_nodes=12, distributed_backend='ddp')
``` ```
In addition, make sure to set up your SLURM job correctly via the [SlurmClusterObject](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/). In particular, specify the number of tasks per node correctly. In addition, make sure to set up your SLURM job correctly via the [SlurmClusterObject](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/). In particular, specify the number of tasks per node correctly.
@@ -140,6 +140,9 @@ cluster.per_experiment_nb_gpus = 8
cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu') cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
``` ```
**NOTE:** When running in DDP mode, any errors in your code will show up as an NCCL issue.
Set the ```NCCL_DEBUG=INFO``` flag to see the ACTUAL error.
Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a
portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes). portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).