wassname/pytorch-lightning
1
0
Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-09-10 12:21:57 +08:00
Code Issues Packages Projects Releases Wiki Activity

Docs (#813)

Browse Source 
* added outline of all features

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated common use cases doc

* updated docs
This commit is contained in:
William Falcon
2020-02-10 23:55:22 -05:00
committed by GitHub
parent af44583050
commit 4c6c3d04ce
28 changed files with 1163 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/apex.rst
+42
View File
@@ -0,0 +1,42 @@
16-bit training
=================
Lightning uses NVIDIA apex to handle 16-bit precision training.
To use 16-bit precision, do two things:
1. Install Apex
2. Set the amp trainer flag.
Install apex
----------------------------------------------
.. code-block:: bash
$ git clone https://github.com/NVIDIA/apex
$ cd apex
# ------------------------
# OPTIONAL: on your cluster you might need to load cuda 10 or 9
# depending on how you installed PyTorch
# see available modules
module avail
# load correct cuda before install
module load cuda-10.0
# ------------------------
# make sure you've loaded a cuda version > 4.0 and < 7.0
module load gcc-6.1.0
$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
Enable 16-bit
--------------
.. code-block:: python
# DEFAULT
trainer = Trainer(amp_level='O1', use_amp=False)
If you need to configure the apex init for your particular use case or want to use a different way of doing
16-bit training, override :meth:`pytorch_lightning.core.LightningModule.configure_apex`.
docs/source/checkpointing.rst
+80
View File
@@ -0,0 +1,80 @@
Checkpointing
==============
.. _model-saving:
Model saving
-------------------
To save a LightningModule, provide a :meth:`pytorch_lightning.callbacks.ModelCheckpoint` callback.
The Lightning checkpoint also saves the hparams (hyperparams) passed into the LightningModule init.
.. note:: hparams is a `Namespace <https://docs.python.org/2/library/argparse.html#argparse.Namespace>`_ or dictionary.
.. code-block:: python
:emphasize-lines: 8
from argparse import Namespace
# usually these come from command line args
args = Namespace(**{'learning_rate':0.001})
# define you module to have hparams as the first arg
# this means your checkpoint will have everything that went into making
# this model (in this case, learning rate)
class MyLightningModule(pl.LightningModule):
def __init__(self, hparams, ...):
self.hparams = hparams
my_model = MyLightningModule(args)
# auto-saves checkpoint
checkpoint_callback = ModelCheckpoint(filepath='my_path')
Trainer(checkpoint_callback=checkpoint_callback)
Model loading
-----------------------------------
To load a model, use :meth:`pytorch_lightning.core.LightningModule.load_from_checkpoint`
.. note:: If lightning created your checkpoint, your model will receive all the hyperparameters used
to create the checkpoint. (See: :ref:`model-saving`).
.. code-block:: python
# load weights without mapping
MyLightningModule.load_from_checkpoint('path/to/checkpoint.ckpt')
# load weights mapping all weights from GPU 1 to GPU 0
map_location = {'cuda:1':'cuda:0'}
MyLightningModule.load_from_checkpoint('path/to/checkpoint.ckpt', map_location=map_location)
Restoring training session
-----------------------------------
If you want to pick up training from where you left off, you have a few options.
1. Pass in a logger with the same experiment version to continue training.
.. code-block:: python
# train the first time and set the version number
logger = TensorboardLogger(version=10)
trainer = Trainer(logger=logger)
trainer.fit(model)
# when you init another logger with that same version, the model
# will continue where it left off
logger = TensorboardLogger(version=10)
trainer = Trainer(logger=logger)
trainer.fit(model)
2. A second option is to pass in a path to a checkpoint (see: :ref:`pytorch_lightning.trainer`).
.. code-block:: python
# train the first time and set the version number
trainer = Trainer(resume_from_checkpoint='some/path/to/my_checkpoint.ckpt')
trainer.fit(model)
docs/source/common-cases.rst
-27
View File
@@ -1,27 +0,0 @@
Multi-gpu (same node) training
==============================
Multi-node training
====================
16-bit precision
=================
gradient clipping
=================
modifying training via hooks
=============================
.. toctree::
:maxdepth: 3
pl_examples
profiling a training run
========================
.. toctree::
:maxdepth: 1
profiler
docs/source/debugging.rst
+75
View File
@@ -0,0 +1,75 @@
Debugging
==========
The following are flags that make debugging much easier.
Fast dev run
-------------------
This flag runs a "unit test" by running 1 training batch and 1 validation batch.
The point is to detect any bugs in the training/validation loop without having to wait for
a full epoch to crash.
.. code-block:: python
trainer = pl.Trainer(fast_dev_run=True)
Inspect gradient norms
-----------------------------------
Logs (to a logger), the norm of each weight matrix.
.. code-block:: python
# the 2-norm
trainer = pl.Trainer(track_grad_norm=2)
Log GPU usage
-----------------------------------
Logs (to a logger) the GPU usage for each GPU on the master machine.
(See: :ref:`trainer`)
.. code-block:: python
trainer = pl.Trainer(log_gpu_memory=True)
Make model overfit on subset of data
-----------------------------------
A good debugging technique is to take a tiny portion of your data (say 2 samples per class),
and try to get your model to overfit. If it can't, it's a sign it won't work with large datasets.
(See: :ref:`trainer`)
.. code-block:: python
trainer = pl.Trainer(overfit_pct=0.01)
Print the parameter count by layer
-----------------------------------
Whenever the .fit() function gets called, the Trainer will print the weights summary for the lightningModule.
To disable this behavior, turn off this flag:
(See: :ref:`trainer.weights_summary`)
.. code-block:: python
trainer = pl.Trainer(weights_summary=None)
Print which gradients are nan
------------------------------
Prints the tensors with nan gradients.
(See: :meth:`trainer.print_nan_grads`)
.. code-block:: python
trainer = pl.Trainer(print_nan_grads=False)
Set the number of validation sanity steps
-------------------------------------
Lightning runs a few steps of validation in the beginning of training.
This avoids crashing in the validation loop sometime deep into a lengthy training loop.
.. code-block:: python
# DEFAULT
trainer = Trainer(nb_sanity_val_steps=5)
docs/source/early_stopping.rst
+35
View File
@@ -0,0 +1,35 @@
Early stopping
==================
Enable Early Stopping
----------------------
There are two ways to enable early stopping.
.. note:: See: :ref:`trainer`
.. code-block:: python
# A) Looks for val_loss in validation_step return dict
trainer = Trainer(early_stop_callback=True)
# B) Or configure your own callback
early_stop_callback = EarlyStopping(
monitor='val_loss',
min_delta=0.00,
patience=3,
verbose=False,
mode='min'
)
trainer = Trainer(early_stop_callback=early_stop_callback)
Force disable early stop
-------------------------------------
To disable early stopping pass None to the early_stop_callback
.. note:: See: :ref:`trainer`
.. code-block:: python
# DEFAULT
trainer = Trainer(early_stop_callback=None)
docs/source/experiment_logging.rst
+139
View File
@@ -0,0 +1,139 @@
Experiment Logging
===================
Comet.ml
^^^^^^^^^^
`Comet.ml <https://www.comet.ml/site/>`_ is a third-party logger.
To use CometLogger as your logger do the following.
.. note:: See: :ref:`comet` docs.
.. code-block:: python
from pytorch_lightning.loggers import TestTubeLogger
comet_logger = CometLogger(
api_key=os.environ["COMET_KEY"],
workspace=os.environ["COMET_WORKSPACE"], # Optional
project_name="default_project", # Optional
rest_api_key=os.environ["COMET_REST_KEY"], # Optional
experiment_name="default" # Optional
)
trainer = Trainer(logger=comet_logger)
The CometLogger is available anywhere in your LightningModule
.. code-block:: python
class MyModule(pl.LightningModule):
def __init__(self, ...):
some_img = fake_image()
self.logger.experiment.add_image('generated_images', some_img, 0)
Neptune.ai
^^^^^^^^^^
`Neptune.ai <https://neptune.ai/>`_ is a third-party logger.
To use Neptune.ai as your logger do the following.
.. note:: See: :ref:`neptune` docs.
.. code-block:: python
from pytorch_lightning.loggers import NeptuneLogger
neptune_logger = NeptuneLogger(
project_name="USER_NAME/PROJECT_NAME",
experiment_name="default", # Optional,
params={"max_epochs": 10}, # Optional,
tags=["pytorch-lightning","mlp"] # Optional,
)
trainer = Trainer(logger=neptune_logger)
The Neptune.ai is available anywhere in your LightningModule
.. code-block:: python
class MyModule(pl.LightningModule):
def __init__(self, ...):
some_img = fake_image()
self.logger.experiment.add_image('generated_images', some_img, 0)
Tensorboard
^^^^^^^^^^^^^
To use `Tensorboard <https://pytorch.org/docs/stable/tensorboard.html>`_ as your logger do the following.
.. note:: See: TensorBoardLogger :ref:`tf-logger`
.. code-block:: python
from pytorch_lightning.loggers import TensorBoardLogger
logger = TensorBoardLogger("tb_logs", name="my_model")
trainer = Trainer(logger=logger)
The TensorBoardLogger is available anywhere in your LightningModule
.. code-block:: python
class MyModule(pl.LightningModule):
def __init__(self, ...):
some_img = fake_image()
self.logger.experiment.add_image('generated_images', some_img, 0)
Test Tube
^^^^^^^^^^^^^
`Test Tube <https://github.com/williamFalcon/test-tube>`_ is a tensorboard logger but with nicer file structure.
To use TestTube as your logger do the following.
.. note:: See: TestTube :ref:`testTube`
.. code-block:: python
from pytorch_lightning.loggers import TestTubeLogger
logger = TestTubeLogger("tb_logs", name="my_model")
trainer = Trainer(logger=logger)
The TestTubeLogger is available anywhere in your LightningModule
.. code-block:: python
class MyModule(pl.LightningModule):
def __init__(self, ...):
some_img = fake_image()
self.logger.experiment.add_image('generated_images', some_img, 0)
Wandb
^^^^^^^^^^^^^
`Wandb <https://www.wandb.com/>`_ is a third-party logger.
To use Wandb as your logger do the following.
.. note:: See: :ref:`wandb` docs
.. code-block:: python
from pytorch_lightning.loggers import WandbLogger
wandb_logger = WandbLogger()
trainer = Trainer(logger=wandb_logger)
The Wandb logger is available anywhere in your LightningModule
.. code-block:: python
class MyModule(pl.LightningModule):
def __init__(self, ...):
some_img = fake_image()
self.logger.experiment.add_image('generated_images', some_img, 0)
docs/source/experiment_reporting.rst
+133
View File
@@ -0,0 +1,133 @@
Experiment Reporting
=====================
Lightning supports many different experiment loggers. These loggers allow you to monitor losses, images, text, etc...
as training progresses. They usually provide a GUI to visualize and can sometimes even snapshot hyperparameters
used in each experiment.
Control logging frequency
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It may slow training down to log every single batch. Trainer has an option to log every k batches instead.
.. code-block:: python
# k = 10
Trainer(row_log_interval=10)
Control log writing frequency
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Writing to a logger can be expensive. In Lightning you can set the interval at which you
want to log using this trainer flag.
.. note:: See: :ref:`trainer`
.. code-block:: python
k = 100
Trainer(log_save_interval=k)
Log metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To plot metrics into whatever logger you passed in (tensorboard, comet, neptune, etc...)
1. Training_end, validation_end, test_end will all log anything in the "log" key of the return dict.
.. code-block:: python
def training_end(self, batch, batch_idx):
loss = some_loss()
...
logs = {'train_loss': loss}
results = {'log': logs}
return results
def validation_end(self, batch, batch_idx):
loss = some_loss()
...
logs = {'val_loss': loss}
results = {'log': logs}
return results
def test_end(self, batch, batch_idx):
loss = some_loss()
...
logs = {'test_loss': loss}
results = {'log': logs}
return results
2. Most of the time, you only need training_step and not training_end. You can also return logs from here:
.. code-block:: python
def training_step(self, batch, batch_idx):
loss = some_loss()
...
logs = {'train_loss': loss}
results = {'log': logs}
return results
3. In addition, you can also use any arbitrary functionality from a particular logger from within your LightningModule.
For instance, here we log images using tensorboard.
.. code-block:: python
def training_step(self, batch, batch_idx):
self.generated_imgs = self.decoder.generate()
sample_imgs = self.generated_imgs[:6]
grid = torchvision.utils.make_grid(sample_imgs)
self.logger.experiment.add_image('generated_images', grid, 0)
...
return results
Modify progress bar
^^^^^^^^^^^^^^^^^^^^^^
Each return dict from the training_end, validation_end, testing_end and training_step also has
a key called "progress_bar".
Here we show the validation loss in the progress bar
.. code-block:: python
def validation_end(self, batch, batch_idx):
loss = some_loss()
...
logs = {'val_loss': loss}
results = {'progress_bar': logs}
return results
Snapshot hyperparameters
^^^^^^^^^^^^^^^^^^^^^^^^^^
When training a model, it's useful to know what hyperparams went into that model.
When Lightning creates a checkpoint, it stores a key "hparams" with the hyperparams.
.. code-block:: python
lightning_checkpoint = torch.load(filepath, map_location=lambda storage, loc: storage)
hyperparams = lightning_checkpoint['hparams']
Some loggers also allow logging the hyperparams used in the experiment. For instance,
when using the TestTubeLogger or the TensorBoardLogger, all hyperparams will show
in the `hparams tab <https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_hparams>`_.
Snapshot code
^^^^^^^^^^^^^^^^^^^^^^^^^^
Loggers also allow you to snapshot a copy of the code used in this experiment.
For example, TestTubeLogger does this with a flag:
.. code-block:: python
from pytorch_lightning.loggers import TestTubeLogger
logger = TestTubeLogger(create_git_tag=True)
docs/source/fast_training.rst
+82
View File
@@ -0,0 +1,82 @@
Fast Training
================
There are multiple options to speed up different parts of the training by choosing to train
on a subset of data. This could be done for speed or debugging purposes.
Check validation every n epochs
-------------------------------------
If you have a small dataset you might want to check validation every n epochs
.. code-block:: python
# DEFAULT
trainer = Trainer(check_val_every_n_epoch=1)
Force training for min or max epochs
-------------------------------------
It can be useful to force training for a minimum number of epochs or limit to a max number.
.. note:: See: :ref:`trainer`
.. code-block:: python
# DEFAULT
trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000)
Set validation check frequency within 1 training epoch
-------------------------------------------------------
For large datasets it's often desirable to check validation multiple times within a training loop.
Pass in a float to check that often within 1 training epoch. Pass in an int k to check every k training batches.
Must use an int if using an IterableDataset.
.. code-block:: python
# DEFAULT
trainer = Trainer(val_check_interval=0.95)
# check every .25 of an epoch
trainer = Trainer(val_check_interval=0.25)
# check every 100 train batches (ie: for IterableDatasets or fixed frequency)
trainer = Trainer(val_check_interval=100)
Use training data subset
----------------------------------
If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
.. code-block:: python
# DEFAULT
trainer = Trainer(train_percent_check=1.0)
# check 10% only
trainer = Trainer(train_percent_check=0.1)
.. note:: train_percent_check will be overwritten by overfit_pct if overfit_pct > 0
Use test data subset
-------------------------------------
If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
test_percent_check will be overwritten by overfit_pct if overfit_pct > 0.
.. code-block:: python
# DEFAULT
trainer = Trainer(test_percent_check=1.0)
# check 10% only
trainer = Trainer(test_percent_check=0.1)
Use validation data subset
--------------------------------------------
If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
val_percent_check will be overwritten by overfit_pct if overfit_pct > 0
.. code-block:: python
# DEFAULT
trainer = Trainer(val_percent_check=1.0)
# check 10% only
trainer = Trainer(val_percent_check=0.1)
docs/source/hooks.rst
+53
View File
@@ -0,0 +1,53 @@
Hooks
=======
This is the order in which lightning calls the hooks. You can override each for custom behavior.
Training set-up
--------------------
- init_ddp_connection
- init_optimizers
- configure_apex
- configure_ddp
- get_train_dataloader
- get_test_dataloaders
- get_val_dataloaders
- summarize
- restore_weights
Training loop
--------------------
- on_epoch_start
- on_batch_start
- tbptt_split_batch
- training_step
- training_end (optional)
- backward
- on_after_backward
- optimizer.step()
- on_batch_end
- on_epoch_end
Validation loop
--------------------
- model.zero_grad()
- model.eval()
- torch.set_grad_enabled(False)
- validation_step
- validation_end
- model.train()
- torch.set_grad_enabled(True)
- on_post_performance_check
Test loop
------------
- model.zero_grad()
- model.eval()
- torch.set_grad_enabled(False)
- test_step
- test_end
- model.train()
- torch.set_grad_enabled(True)
- on_post_performance_check
docs/source/index.rst
+17 -2
View File
@@ -20,7 +20,7 @@ PyTorch-Lightning Documentation
callbacks
lightning-module
logging
loggers
trainer
.. toctree::
@@ -42,7 +42,22 @@ PyTorch-Lightning Documentation
:name: Common Use Cases
:caption: Common Use Cases
common-cases
apex
checkpointing
slurm
debugging
experiment_logging
experiment_reporting
early_stopping
fast_training
hooks
multi_gpu
single_gpu
sequences
training_tricks
test_set
optimizers
profiler
.. toctree::
:maxdepth: 1
docs/source/logging.rst → docs/source/loggers.rst
+2 -2
View File
@@ -1,9 +1,9 @@
.. role:: hidden
:class: hidden-section
Logging
Loggers
===========
.. automodule:: pytorch_lightning.logging
.. automodule:: pytorch_lightning.loggers
:exclude-members:
_abc_impl,
_save_model,
docs/source/modules.rst
+7
View File
@@ -0,0 +1,7 @@
pl_examples
===========
.. toctree::
:maxdepth: 4
pl_examples
docs/source/multi_gpu.rst
+69
View File
@@ -0,0 +1,69 @@
Multi-GPU training
=====================
Lightning supports multiple ways of doing distributed training.
Data Parallel (dp)
-------------------
`DataParallel <https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel>`_ splits a batch across k GPUs. That is, if you have a batch of 32 and use dp with 2 gpus,
each GPU will process 16 samples, after which the root node will aggregate the results.
.. code-block:: python
# train on 1 GPU (using dp mode)
trainer = pl.Trainer(gpus=2, distributed_backend='dp')
Distributed Data Parallel
---------------------------
`DistributedDataParallel <https://pytorch.org/docs/stable/nn.html#distributeddataparallel>`_ works as follows.
1. Each GPU across every node gets its own process.
2. Each GPU gets visibility into a subset of the overall dataset. It will only ever see that subset.
3. Each process inits the model.
.. note:: Make sure to set the random seed so that each model inits with the same weights
4. Each process performs a full forward and backward pass in parallel.
5. The gradients are synced and averaged across all processes.
6. Each process updates its optimizer.
.. code-block:: python
# train on 8 GPUs (same machine (ie: node))
trainer = pl.Trainer(gpus=8, distributed_backend='ddp')
# train on 32 GPUs (4 nodes)
trainer = pl.Trainer(gpus=8, distributed_backend='ddp', num_nodes=4)
Distributed Data Parallel 2
-----------------------------
In certain cases, it's advantageous to use all batches on the same machine instead of a subset.
For instance you might want to compute a NCE loss where it pays to have more negative samples.
In this case, we can use ddp2 which behaves like dp in a machine and ddp across nodes. DDP2 does the following:
1. Copies a subset of the data to each node.
2. Inits a model on each node.
3. Runs a forward and backward pass using DP.
4. Syncs gradients across nodes.
5. Applies the optimizer updates.
.. code-block:: python
# train on 32 GPUs (4 nodes)
trainer = pl.Trainer(gpus=8, distributed_backend='ddp2', num_nodes=4)
Implement Your Own Distributed (DDP) training
----------------------------------------------
If you need your own way to init PyTorch DDP you can override :meth:`pytorch_lightning.core.LightningModule.init_ddp_connection`.
If you also need to use your own DDP implementation, override: :meth:`pytorch_lightning.core.LightningModule.configure_ddp`.
docs/source/optimizers.rst
+99
View File
@@ -0,0 +1,99 @@
Optimization
===============
Learning rate scheduling
-------------------------------------
Every optimizer you use can be paired with any `LearningRateScheduler <https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate>`_.
.. code-block:: python
# no LR scheduler
def configure_optimizers(self):
return Adam(...)
# Adam + LR scheduler
def configure_optimizers(self):
return [Adam(...)], [ReduceLROnPlateau()]
# Two optimziers each with a scheduler
def configure_optimizers(self):
return [Adam(...), SGD(...)], [ReduceLROnPlateau(), LambdaLR()]
Use multiple optimizers (like GANs)
-------------------------------------
To use multiple optimizers return > 1 optimizers from :meth:`pytorch_lightning.core.LightningModule.configure_optimizers`
.. code-block:: python
# one optimizer
def configure_optimizers(self):
return Adam(...)
# two optimizers, no schedulers
def configure_optimizers(self):
return Adam(...), SGD(...)
# Two optimizers, one scheduler for adam only
def configure_optimizers(self):
return [Adam(...), SGD(...)], [ReduceLROnPlateau()]
Lightning will call each optimizer sequentially:
.. code-block:: python
for epoch in epochs:
for batch in data:
for opt in optimizers:
train_step(opt)
opt.step()
for scheduler in scheduler:
scheduler.step()
Step optimizers at arbitrary intervals
-------------------------------------
To do more interesting things with your optimizers such as learning rate warm-up or odd scheduling,
override the :meth:`optimizer_step' function.
For example, here step optimizer A every 2 batches and optimizer B every 4 batches
.. code-block:: python
def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
optimizer.step()
optimizer.zero_grad()
# Alternating schedule for optimizer steps (ie: GANs)
def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
# update generator opt every 2 steps
if optimizer_i == 0:
if batch_nb % 2 == 0 :
optimizer.step()
optimizer.zero_grad()
# update discriminator opt every 4 steps
if optimizer_i == 1:
if batch_nb % 4 == 0 :
optimizer.step()
optimizer.zero_grad()
# ...
# add as many optimizers as you want
Here we add a learning-rate warm up
.. code-block:: python
# learning rate warm-up
def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
# warm up lr
if self.trainer.global_step < 500:
lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
for pg in optimizer.param_groups:
pg['lr'] = lr_scale * self.hparams.learning_rate
# update params
optimizer.step()
optimizer.zero_grad()
docs/source/profiler.rst
+1 -1
View File
@@ -2,7 +2,7 @@
:class: hidden-section
Profiling performance during training
Performance and Bottleneck Profiler
===========
.. automodule:: pytorch_lightning.profiler
:exclude-members:
docs/source/sequences.rst
+45
View File
@@ -0,0 +1,45 @@
Sequential Data
================
Lightning has built in support for dealing with sequential data.
Packed sequences as inputs
----------------------------
When using PackedSequence, do 2 things:
1. return either a padded tensor in dataset or a list of variable length tensors in the dataloader collate_fn (example above shows the list implementation).
2. Pack the sequence in forward or training and validation steps depending on use case.
.. code-block:: python
# For use in dataloader
def collate_fn(batch):
x = [item[0] for item in batch]
y = [item[1] for item in batch]
return x, y
# In module
def training_step(self, batch, batch_nb):
x = rnn.pack_sequence(batch[0], enforce_sorted=False)
y = rnn.pack_sequence(batch[1], enforce_sorted=False)
Truncated Backpropagation Through Time
---------------------------------------
There are times when multiple backwards passes are needed for each batch.
For example, it may save memory to use Truncated Backpropagation Through Time when training RNNs.
Lightning can handle TBTT automatically via this flag.
.. code-block:: python
# DEFAULT (single backwards pass per batch)
trainer = Trainer(truncated_bptt_steps=None)
# (split batch into sequences of size 2)
trainer = Trainer(truncated_bptt_steps=2)
.. note:: If you need to modify how the batch is split,
override :meth:`pytorch_lightning.core.LightningModule.tbptt_split_batch`.
.. note:: Using this feature requires updating your LightningModule's :meth:`pytorch_lightning.core.LightningModule.training_step` to include
a `hiddens` arg.
docs/source/single_gpu.rst
+9
View File
@@ -0,0 +1,9 @@
Single GPU Training
====================
Make sure you are running on a machine that has at least one GPU. Lightning handles all the NVIDIA flags for you,
there's no need to set them yourself.
.. code-block:: python
# train on 1 GPU (using dp mode)
trainer = pl.Trainer(gpus=1)
docs/source/slurm.rst
+91
View File
@@ -0,0 +1,91 @@
Computing cluster (SLURM)
==========================
Lightning automates job the details behind training on a SLURM powered cluster.
.. _multi-node:
Multi-node training
--------------------
To train a model using multiple-nodes do the following:
1. Design your LightningModule.
2. Add `torch.DistributedSampler <https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler>`_
which enables access to a subset of your full dataset to each GPU.
3. Enable ddp in the trainer
.. code-block:: python
# train on 32 GPUs across 4 nodes
trainer = Trainer(gpus=8, num_nodes=4, distributed_backend='ddp')
4. It's a good idea to structure your train.py file like this:
.. code-block:: python
# train.py
def main(hparams):
model = LightningTemplateModel(hparams)
trainer = pl.Trainer(
gpus=8,
num_nodes=4,
distributed_backend='ddp'
)
trainer.fit(model)
if __name__ == '__main__':
root_dir = os.path.dirname(os.path.realpath(__file__))
parent_parser = ArgumentParser(add_help=False)
hyperparams = parser.parse_args()
# TRAIN
main(hyperparams)
4. Submit the appropriate SLURM job
.. code-block:: bash
#!/bin/bash -l
# SLURM SUBMIT SCRIPT
#SBATCH --nodes=4
#SBATCH --gres=gpu:8
#SBATCH --ntasks-per-node=8
#SBATCH --mem=0
#SBATCH --time=0-02:00:00
# activate conda env
source activate $1
# -------------------------
# debugging flags (optional)
export NCCL_DEBUG=INFO
export PYTHONFAULTHANDLER=1
# on your cluster you might need these:
# set the network interface
# export NCCL_SOCKET_IFNAME=^docker0,lo
# might need the latest cuda
# module load NCCL/2.4.7-1-cuda.10.0
# -------------------------
# run script from above
srun python3 train.py
Walltime auto-resubmit
-----------------------------------
When you use Lightning in a SLURM cluster, lightning automatically detects when it is about
to run into the walltime, and it does the following:
1. Saves a temporary checkpoint.
2. Requeues the job.
3. When the job starts, it loads the temporary checkpoint.
.. note:: To get this behavior you have to do nothing.
docs/source/test_set.rst
+39
View File
@@ -0,0 +1,39 @@
Test set
==========
Lightning forces the user to run the test set separately to make sure it isn't evaluated by mistake
Test after fit
----------------
To run the test set after training completes, use this method
.. code-block:: python
# run full training
trainer.fit(model)
# run test set
trainer.test()
Test pre-trained model
-----------------
To run the test set on a pretrained model, use this method.
.. code-block:: python
model = MyLightningModule.load_from_metrics(
weights_path='/path/to/pytorch_checkpoint.ckpt',
tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
on_gpu=True,
map_location=None
)
# init trainer with whatever options
trainer = Trainer(...)
# test (pass in the model)
trainer.test(model)
In this case, the options you pass to trainer will be used when
running the test set (ie: 16-bit, dp, ddp, etc...
docs/source/training_tricks.rst
+31
View File
@@ -0,0 +1,31 @@
Training Tricks
================
Lightning implements various tricks to help during training
Accumulate gradients
-------------------------------------
Accumulated gradients runs K small batches of size N before doing a backwards pass.
The effect is a large effective batch size of size KxN.
.. note:: See: :ref:`trainer`
.. code-block:: python
# DEFAULT (ie: no accumulated grads)
trainer = Trainer(accumulate_grad_batches=1)
Gradient Clipping
-------------------------------------
Gradient clipping may be enabled to avoid exploding gradients. Specifically, this will `clip the gradient
norm <https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_>`_ computed over all model parameters together.
.. note:: See: :ref:`trainer`
.. code-block:: python
# DEFAULT (ie: don't clip)
trainer = Trainer(gradient_clip_val=0)
# clip gradients with norm above 0.5
trainer = Trainer(gradient_clip_val=0.5)
pytorch_lightning/core/lightning.py
+10 -3
View File
@@ -110,10 +110,17 @@ class LightningModule(ABC, GradInformation, ModelIO, ModelHooks):
@abstractmethod
def training_step(self, *args, **kwargs):
"""return loss, dict with metrics for tqdm
r"""return loss, dict with metrics for tqdm
Args:
batch (torch.nn.Tensor | (Tensor, Tensor) | [Tensor, Tensor]): The output of your dataloader.
A tensor, tuple or list
batch_idx (int): Integer displaying index of this batch
optimizer_idx (int): If using multiple optimizers, this argument will also be present.
hiddens(:`Tensor <https://pytorch.org/docs/stable/tensors.html>`_): Passed in if truncated_bptt_steps > 0.
:param
:param batch: The output of your dataloader. A tensor, tuple or list
:param int batch_idx: Integer displaying which batch this is
:return: dict with loss key and optional log, progress keys
if implementing training_step, return whatever you need in that step:
pytorch_lightning/loggers/comet.py
+11 -5
View File
@@ -1,3 +1,11 @@
r"""
.. _comet:
CometLogger
-------------
"""
from logging import getLogger
try:
@@ -20,19 +28,19 @@ logger = getLogger(__name__)
class CometLogger(LightningLoggerBase):
r"""
Log using `comet.ml <https://www.comet.ml>`_.
"""
def __init__(self, api_key=None, save_dir=None, workspace=None,
rest_api_key=None, project_name=None, experiment_name=None, **kwargs):
r"""
Log using `comet <https://www.comet.ml>`_.
Requires either an API Key (online mode) or a local directory path (offline mode)
.. code-block:: python
# ONLINE MODE
from pytorch_lightning.loggers import CometLogger
# arguments made to CometLogger are passed on to the comet_ml.Experiment class
comet_logger = CometLogger(
api_key=os.environ["COMET_KEY"],
@@ -43,12 +51,10 @@ class CometLogger(LightningLoggerBase):
)
trainer = Trainer(logger=comet_logger)
.. code-block:: python
# OFFLINE MODE
from pytorch_lightning.loggers import CometLogger
# arguments made to CometLogger are passed on to the comet_ml.Experiment class
comet_logger = CometLogger(
save_dir=".",
pytorch_lightning/loggers/neptune.py
+25 -35
View File
@@ -1,41 +1,10 @@
"""
Log using `neptune <https://www.neptune.ml>`_
Neptune logger can be used in the online mode or offline (silent) mode.
To log experiment data in online mode, NeptuneLogger requries an API key:
.. code-block:: python
from pytorch_lightning.loggers import NeptuneLogger
# arguments made to NeptuneLogger are passed on to the neptune.experiments.Experiment class
neptune_logger = NeptuneLogger(
api_key=os.environ["NEPTUNE_API_TOKEN"],
project_name="USER_NAME/PROJECT_NAME",
experiment_name="default", # Optional,
params={"max_epochs": 10}, # Optional,
tags=["pytorch-lightning","mlp"] # Optional,
)
trainer = Trainer(max_epochs=10, logger=neptune_logger)
Use the logger anywhere in you LightningModule as follows:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.log_metric("acc_train", acc_train) # log metrics
self.logger.experiment.log_image("worse_predictions", prediction_image) # log images
self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint
self.logger.experiment.whatever_neptune_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.log_metric("acc_train", acc_train) # log metrics
self.logger.experiment.log_image("worse_predictions", prediction_image) # log images
self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint
self.logger.experiment.whatever_neptune_supports(...)
Log using `neptune-logger <https://www.neptune.ml>`_
.. _neptune:
NeptuneLogger
--------------
"""
from logging import getLogger
@@ -54,6 +23,10 @@ logger = getLogger(__name__)
class NeptuneLogger(LightningLoggerBase):
r"""
Neptune logger can be used in the online mode or offline (silent) mode.
To log experiment data in online mode, NeptuneLogger requries an API key:
"""
def __init__(self, api_key=None, project_name=None, offline_mode=False,
experiment_name=None, upload_source_files=None,
params=None, properties=None, tags=None, **kwargs):
@@ -92,6 +65,23 @@ class NeptuneLogger(LightningLoggerBase):
)
trainer = Trainer(max_epochs=10, logger=neptune_logger)
Use the logger anywhere in you LightningModule as follows:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.log_metric("acc_train", acc_train) # log metrics
self.logger.experiment.log_image("worse_predictions", prediction_image) # log images
self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint
self.logger.experiment.whatever_neptune_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.log_metric("acc_train", acc_train) # log metrics
self.logger.experiment.log_image("worse_predictions", prediction_image) # log images
self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint
self.logger.experiment.whatever_neptune_supports(...)
Args:
api_key (str | None): Required in online mode. Neputne API token, found on https://neptune.ml.
Read how to get your API key
pytorch_lightning/loggers/tensorboard.py
+3 -1
View File
@@ -18,8 +18,10 @@ class TensorBoardLogger(LightningLoggerBase):
Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to
`os.path.join(save_dir, name, version)`
.. _tf-logger:
Example
--------
------------------
.. code-block:: python
pytorch_lightning/loggers/test_tube.py
+35 -54
View File
@@ -1,34 +1,3 @@
"""
Log using `test tube <https://williamfalcon.github.io/test-tube>'_. Test tube logger is
a strict subclass of `PyTorch SummaryWriter <https://pytorch.org/docs/stable/tensorboard.html>`_, refer to their
documentation for all supported operations. The TestTubeLogger adds a nicer folder structure
to manage experiments and snapshots all hyperparameters you pass to a LightningModule.
.. code-block:: python
from pytorch_lightning.loggers import TestTubeLogger
tt_logger = TestTubeLogger(
save_dir=".",
name="default",
debug=False,
create_git_tag=False
)
trainer = Trainer(logger=tt_logger)
Use the logger anywhere in you LightningModule as follows:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.whatever_method_summary_writer_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.add_histogram(...)
"""
try:
from test_tube import Experiment
except ImportError:
@@ -39,30 +8,8 @@ from .base import LightningLoggerBase, rank_zero_only
class TestTubeLogger(LightningLoggerBase):
r"""
Log to local file system in TensorBoard format but using a nicer folder structure.
Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to
`os.path.join(save_dir, name, version)`
Example
-------
.. code-block:: python
logger = TestTubeLogger("tt_logs", name="my_exp_name")
trainer = Trainer(logger=logger)
trainer.train(model)
Args:
save_dir (str): Save directory
name (str): Experiment name. Defaults to "default".
description (str): A short snippet about this experiment
debug (bool): If True, it doesn't log anything
version (int): Experiment version. If version is not specified the logger inspects the save
directory for existing versions, then automatically assigns the next available version.
create_git_tag (bool): If True creates a git tag to save the code used in this experiment
(see `full docs <https://williamfalcon.github.io/test-tube>`_).
"""
__test__ = False
@@ -71,6 +18,40 @@ class TestTubeLogger(LightningLoggerBase):
self, save_dir, name="default", description=None, debug=False,
version=None, create_git_tag=False
):
r"""
.. _testTube:
Example
----------
.. code-block:: python
logger = TestTubeLogger("tt_logs", name="my_exp_name")
trainer = Trainer(logger=logger)
trainer.train(model)
Use the logger anywhere in you LightningModule as follows:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.whatever_method_summary_writer_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.add_histogram(...)
Args:
save_dir (str): Save directory
name (str): Experiment name. Defaults to "default".
description (str): A short snippet about this experiment
debug (bool): If True, it doesn't log anything
version (int): Experiment version. If version is not specified the logger inspects the save
directory for existing versions, then automatically assigns the next available version.
create_git_tag (bool): If True creates a git tag to save the code used in this experiment
"""
super().__init__()
self.save_dir = save_dir
self._name = name
pytorch_lightning/loggers/wandb.py
+9 -1
View File
@@ -1,3 +1,11 @@
r"""
.. _wandb:
WandbLogger
-------------
"""
import os
try:
@@ -10,7 +18,7 @@ from .base import LightningLoggerBase, rank_zero_only
class WandbLogger(LightningLoggerBase):
"""
Logger for W&B.
Logger for `W&B <https://www.wandb.com/>`_.
Args:
name (str): display name for the run.
pytorch_lightning/profiler/__init__.py
+14 -4
View File
@@ -1,6 +1,10 @@
"""
Profiling your training run can help you understand if there are any bottlenecks in your code.
Built-in checks
----------------
PyTorch Lightning supports profiling standard actions in the training loop out of the box, including:
- on_epoch_start
@@ -15,6 +19,9 @@ PyTorch Lightning supports profiling standard actions in the training loop out o
- training_end
- on_training_end
Enable simple profiling
-------------------------
If you only wish to profile the standard actions, you can set `profiler=True` when constructing
your `Trainer` object.
@@ -42,6 +49,9 @@ The profiler's results will be printed at the completion of a training `fit()`.
on_train_end | 5.449e-06 | 5.449e-06
Advanced Profiling
--------------------
If you want more information on the functions called during each event, you can use the `AdvancedProfiler`.
This option uses Python's cProfiler_ to provide a report of time spent on *each* function called within your code.
@@ -67,10 +77,10 @@ of logging it to the output in your terminal. The output below shows the profili
List reduced from 76 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
3752/1876 0.011 0.000 18.887 0.010 {built-in method builtins.next}
1876 0.008 0.000 18.877 0.010 dataloader.py:344(__next__)
1876 0.074 0.000 18.869 0.010 dataloader.py:383(_next_data)
1875 0.012 0.000 18.721 0.010 fetch.py:42(fetch)
1875 0.084 0.000 18.290 0.010 fetch.py:44(<listcomp>)
1876 0.008 0.000 18.877 0.010 dataloader.py:344(__next__)
1876 0.074 0.000 18.869 0.010 dataloader.py:383(_next_data)
1875 0.012 0.000 18.721 0.010 fetch.py:42(fetch)
1875 0.084 0.000 18.290 0.010 fetch.py:44(<listcomp>)
60000 1.759 0.000 18.206 0.000 mnist.py:80(__getitem__)
60000 0.267 0.000 13.022 0.000 transforms.py:68(__call__)
60000 0.182 0.000 7.020 0.000 transforms.py:93(__call__)
pytorch_lightning/trainer/trainer.py
+7 -2
View File
@@ -452,9 +452,14 @@ class Trainer(TrainerIOMixin,
# backprop every 5 steps in a batch
trainer = Trainer(truncated_bptt_steps=5)
Using this feature requires updating your LightningModule's `training_step()` to include
a `hiddens` arg.
Lightning takes care to split your batch along the time-dimension.
.. note:: If you need to modify how the batch is split,
override :meth:`pytorch_lightning.core.LightningModule.tbptt_split_batch`.
.. note:: Using this feature requires updating your LightningModule's
:meth:`pytorch_lightning.core.LightningModule.training_step` to include a `hiddens` arg.
resume_from_checkpoint (str): To resume training from a specific checkpoint pass in the path here.k
Example::