wassname/pytorch-lightning
1
0
Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-09-11 12:31:23 +08:00
Code Issues Packages Projects Releases Wiki Activity

doctest for .rst files (#1511)

Browse Source 
* add doctest to circleci

* Revert "add doctest to circleci"

This reverts commit c45b34ea911a81f87989f6c3a832b1e8d8c471c6.

* Revert "Revert "add doctest to circleci""

This reverts commit 41fca97fdcfe1cf4f6bdb3bbba75d25fa3b11f70.

* doctest docs rst files

* Revert "doctest docs rst files"

This reverts commit b4a2e83e3da5ed1909de500ec14b6b614527c07f.

* doctest only rst

* doctest debugging.rst

* doctest apex

* doctest callbacks

* doctest early stopping

* doctest for child modules

* doctest experiment reporting

* indentation

* doctest fast training

* doctest for hyperparams

* doctests for lr_finder

* doctests multi-gpu

* more doctest

* make doctest drone

* fix label build error

* update fast training

* update invalid imports

* fix problem with int device count

* rebase stuff

* wip

* wip

* wip

* intro guide

* add missing code block

* circleci

* logger import for doctest

* test if doctest runs on drone

* fix mnist download

* also run install deps for building docs

* install cmake

* try sudo

* hide output

* try pip stuff

* try to mock horovod

* Tranfer -> Transfer

* add torchvision to extras

* revert pip stuff

* mlflow file location

* do not mock torch

* torchvision

* drone extra req.

* try higher sphinx version

* Revert "try higher sphinx version"

This reverts commit 490ac28e46d6fd52352640dfdf0d765befa56988.

* try coverage command

* try coverage command

* try undoc flag

* newline

* undo drone

* report coverage

* review

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* remove torchvision from extras

* skip tests only if torchvision not available

* fix testoutput torchvision

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
This commit is contained in:
Adrian Wälchli
2020-05-04 22:16:54 -04:00
committed by GitHub
co-authored by Jirka Borovec
parent 48e808c20e
commit a6de1b8d75
25 changed files with 798 additions and 637 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/multi_gpu.rst
+32 -25
View File
@@ -1,3 +1,9 @@
.. testsetup:: *
import torch
from pytorch_lightning.trainer.trainer import Trainer
from pytorch_lightning.core.lightning import LightningModule
.. _multi-gpu-training:
Multi-GPU training
@@ -13,7 +19,7 @@ Delete .cuda() or .to() calls
Delete any calls to .cuda() or .to(device).
.. code-block:: python
.. testcode::
# before lightning
def forward(self, x):
@@ -30,7 +36,7 @@ Init using type_as
When you need to create a new tensor, use `type_as`.
This will make your code scale to any arbitrary number of GPUs or TPUs with Lightning
.. code-block:: python
.. testcode::
# before lightning
def forward(self, x):
@@ -47,7 +53,7 @@ Remove samplers
For multi-node or TPU training, in PyTorch we must use `torch.nn.DistributedSampler`. The
sampler makes sure each GPU sees the appropriate part of your data.
.. code-block:: python
.. testcode::
# without lightning
def train_dataloader(self):
@@ -62,7 +68,7 @@ sampler makes sure each GPU sees the appropriate part of your data.
With Lightning, you don't need to do this because it takes care of adding the correct samplers
when needed.
.. code-block:: python
.. testcode::
# with lightning
def train_dataloader(self):
@@ -131,10 +137,11 @@ each GPU will process 16 samples, after which the root node will aggregate the r
.. warning:: DP use is discouraged by PyTorch and Lightning. Use ddp which is more stable and at least 3x faster
.. code-block:: python
.. testcode::
:skipif: torch.cuda.device_count() < 2
# train on 1 GPU (using dp mode)
trainer = pl.Trainer(gpus=2, distributed_backend='dp')
# train on 2 GPUs (using dp mode)
trainer = Trainer(gpus=2, distributed_backend='dp')
Distributed Data Parallel
^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -157,10 +164,10 @@ Distributed Data Parallel
.. code-block:: python
# train on 8 GPUs (same machine (ie: node))
trainer = pl.Trainer(gpus=8, distributed_backend='ddp')
trainer = Trainer(gpus=8, distributed_backend='ddp')
# train on 32 GPUs (4 nodes)
trainer = pl.Trainer(gpus=8, distributed_backend='ddp', num_nodes=4)
trainer = Trainer(gpus=8, distributed_backend='ddp', num_nodes=4)
Distributed Data Parallel 2
^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -182,7 +189,7 @@ In this case, we can use ddp2 which behaves like dp in a machine and ddp across
.. code-block:: python
# train on 32 GPUs (4 nodes)
trainer = pl.Trainer(gpus=8, distributed_backend='ddp2', num_nodes=4)
trainer = Trainer(gpus=8, distributed_backend='ddp2', num_nodes=4)
Horovod
^^^^^^^
@@ -202,15 +209,15 @@ Horovod can be configured in the training script to run with any number of GPUs
.. code-block:: python
# train Horovod on GPU (number of GPUs / machines provided on command-line)
trainer = pl.Trainer(distributed_backend='horovod', gpus=1)
trainer = Trainer(distributed_backend='horovod', gpus=1)
# train Horovod on CPU (number of processes / machines provided on command-line)
trainer = pl.Trainer(distributed_backend='horovod')
trainer = Trainer(distributed_backend='horovod')
When starting the training job, the driver application will then be used to specify the total
number of worker processes:
.. code-block::
.. code-block:: bash
# run training with 4 GPUs on a single machine
horovodrun -np 4 python train.py
@@ -226,7 +233,7 @@ DP/DDP2 caveats
In DP and DDP2 each GPU within a machine sees a portion of a batch.
DP and ddp2 roughly do the following:
.. code-block:: python
.. testcode::
def distributed_forward(batch, model):
batch = torch.Tensor(32, 8)
@@ -245,7 +252,7 @@ DP and ddp2 roughly do the following:
So, when Lightning calls any of the `training_step`, `validation_step`, `test_step`
you will only be operating on one of those pieces.
.. code-block:: python
.. testcode::
# the batch here is a portion of the FULL batch
def training_step(self, batch, batch_idx):
@@ -255,7 +262,7 @@ For most metrics, this doesn't really matter. However, if you want
to add something to your computational graph (like softmax)
using all batch parts you can use the `training_step_end` step.
.. code-block:: python
.. testcode::
def training_step_end(self, outputs):
# only use when on dp
@@ -288,7 +295,7 @@ In pseudocode, the full sequence is:
to illustrate why this is needed, let's look at dataparallel
.. code-block:: python
.. testcode::
def training_step(self, batch, batch_idx):
x, y = batch
@@ -313,13 +320,13 @@ it will behave the same no matter the backend.
Validation and test step also have the same option when using dp
.. code-block:: python
.. testcode::
def validation_step_end(self, batch_parts_outputs):
...
def validation_step_end(self, batch_parts_outputs):
...
def test_step_end(self, batch_parts_outputs):
...
def test_step_end(self, batch_parts_outputs):
...
Implement Your Own Distributed (DDP) training
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -335,7 +342,7 @@ batch size.
Let's say you have a batch size of 7 in your dataloader.
.. code-block::
.. testcode::
class LitModel(LightningModule):
@@ -344,7 +351,7 @@ Let's say you have a batch size of 7 in your dataloader.
In (DDP, Horovod) your effective batch size will be 7 * gpus * num_nodes.
.. code-block::
.. code-block:: python
# effective batch size = 7 * 8
Trainer(gpus=8, distributed_backend='ddp|horovod')
@@ -356,7 +363,7 @@ In (DDP, Horovod) your effective batch size will be 7 * gpus * num_nodes.
In DDP2, your effective batch size will be 7 * num_nodes.
The reason is that the full batch is visible to all GPUs on the node when using DDP2.
.. code-block::
.. code-block:: python
# effective batch size = 7
Trainer(gpus=8, distributed_backend='ddp2')