From 286625a02f5a33c91f635ca10a61b16867fcd8e3 Mon Sep 17 00:00:00 2001 From: William Falcon Date: Mon, 16 Sep 2019 11:02:04 -0400 Subject: [PATCH] updated docs --- docs/Trainer/Distributed training.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/Trainer/Distributed training.md b/docs/Trainer/Distributed training.md index 0a68b761..c895e247 100644 --- a/docs/Trainer/Distributed training.md +++ b/docs/Trainer/Distributed training.md @@ -108,7 +108,7 @@ trainer = Trainer(gpus=8, distributed_backend='ddp') Multi-node training is easily done by specifying these flags. ```python # train on 12*8 GPUs -trainer = Trainer(gpus=8, nb_gpu_nodes=12) +trainer = Trainer(gpus=8, nb_gpu_nodes=12, distributed_backend='ddp') ``` In addition, make sure to set up your SLURM job correctly via the [SlurmClusterObject](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/). In particular, specify the number of tasks per node correctly. @@ -140,6 +140,9 @@ cluster.per_experiment_nb_gpus = 8 cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu') ``` +**NOTE:** When running in DDP mode, any errors in your code will show up as an NCCL issue. +Set the ```NCCL_DEBUG=INFO``` flag to see the ACTUAL error. + Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).