wassname/pytorch-lightning
1
0
Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-09-09 11:32:07 +08:00
Code Issues Packages Projects Releases Wiki Activity

Sphinx generated documentation (#521)

Browse Source 
* upgrade req.

* move MkDocs

* create Sphinx

* init Sphinx

* move md from MkDocs to Sphinx

* CI: build docs

* build Sphinx

formatting

move docs from MD to docstring in particular package/modules

formatting

add Sphinx ext.

rename root_module to core

drop implicit name "_logger"

drop duplicate name "overwrite"

fix imports

use pytorch theme

add sample link mapping

try fix RTD build

use forked template

fix some docs warnings

fix paths

add deprecation warnings

fix flake8

fix paths

revert refactor

revert MLFlowLogger

* revert example import

* update link

* Update lightning_module_template.py
This commit is contained in:
Jirka Borovec
2019-11-28 12:48:55 -05:00
committed by William Falcon
parent 47659daa5f
commit d71556e7a1
53 changed files with 2956 additions and 2725 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.circleci/config.yml
+18
View File
@@ -24,8 +24,25 @@ references:
py.test pytorch_lightning tests pl_examples -v --doctest-modules --junitxml=test-reports/pytest_junit.xml --flake8
no_output_timeout: 15m
make_docs: &make_docs
run:
name: Make Documentation
command: |
# sudo apt-get install pandoc
pip install -r requirements.txt --user
sudo pip install -r docs/requirements.txt
# sphinx-apidoc -o ./docs/source ./pytorch_lightning **/test_* --force --follow-links
cd docs; make clean ; make html
jobs:
Build-Docs:
docker:
- image: circleci/python:3.7
steps:
- checkout
- *make_docs
PyTorch:
docker:
- image: circleci/python:3.7
@@ -67,6 +84,7 @@ workflows:
version: 2
build:
jobs:
- Build-Docs
- PyTorch-v1.1
- PyTorch-v1.2
- PyTorch-v1.3
.readthedocs.yml
+8 -3
View File
@@ -5,9 +5,13 @@
# Required
version: 2
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/source/conf.py
# Build documentation with MkDocs
mkdocs:
configuration: mkdocs.yml
#mkdocs:
# configuration: mkdocs.yml
# Optionally build your docs in additional formats such as PDF and ePub
formats: all
@@ -16,4 +20,5 @@ formats: all
python:
version: 3.7
install:
- requirements: docs/requirements.txt
#- requirements: requirements.txt
- requirements: docs/requirements.txt
README.md
+5 -5
View File
@@ -1,6 +1,6 @@
<div align="center">
![Logo](./docs/source/_static/lightning_logo_small.png)
![Logo](docs/source/_static/images/lightning_logo_small.png)
# PyTorch Lightning
@@ -11,7 +11,7 @@
[![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
[![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
[![Build status](https://ci.appveyor.com/api/projects/status/NEW-PROJECT-ID?svg=true)](https://ci.appveyor.com/project/williamFalcon/pytorch-lightning)
[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
[![Coverage](docs/source/_static/images/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)
[![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
@@ -183,7 +183,7 @@ trainer.test()
Everything in gray!
You define the blue parts using the LightningModule interface:
![Overview](./docs/source/_static/overview_flat.jpg)
![Overview](docs/source/_static/images/overview_flat.jpg)
```python
# what to do in the training loop
@@ -266,11 +266,11 @@ def validation_end(self, outputs):
## Tensorboard
Lightning is fully integrated with tensorboard, MLFlow and supports any logging module.
![tensorboard-support](./docs/source/_static/tf_loss.png)
![tensorboard-support](docs/source/_static/images/tf_loss.png)
Lightning also adds a text column with all the hyperparameters for this experiment.
![tensorboard-support](./docs/source/_static/tf_tags.png)
![tensorboard-support](docs/source/_static/images/tf_tags.png)
## Lightning automates all of the following ([each is also configurable](https://williamfalcon.github.io/pytorch-lightning/Trainer/)):
docs/LightningModule/RequiredTrainerInterface.md
-801
View File
@@ -1,801 +0,0 @@
# Lightning Module interface
[[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/root_module.py)]
A lightning module is a strict superclass of nn.Module, it provides a standard interface for the trainer to interact with the model.
The easiest thing to do is copy the [minimal example](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) below and modify accordingly.
Otherwise, to Define a Lightning Module, implement the following methods:
**Required**:
- [training_step](RequiredTrainerInterface.md#training_step)
- [train_dataloader](RequiredTrainerInterface.md#train_dataloader)
- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)
**Optional**:
- [training_end](RequiredTrainerInterface.md#training_end)
- [validation_step](RequiredTrainerInterface.md#validation_step)
- [validation_end](RequiredTrainerInterface.md#validation_end)
- [test_step](RequiredTrainerInterface.md#test_step)
- [test_end](RequiredTrainerInterface.md#test_end)
- [val_dataloader](RequiredTrainerInterface.md#val_dataloader)
- [test_dataloader](RequiredTrainerInterface.md#test_dataloader)
- [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)
- [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)
- [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)
---
### Minimal example
```python
import os
import torch
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST
import torchvision.transforms as transforms
import pytorch_lightning as pl
class CoolModel(pl.LightningModule):
def __init__(self):
super(CoolModel, self).__init__()
# not the best model...
self.l1 = torch.nn.Linear(28 * 28, 10)
def forward(self, x):
return torch.relu(self.l1(x.view(x.size(0), -1)))
def training_step(self, batch, batch_nb):
# REQUIRED
x, y = batch
y_hat = self.forward(x)
return {'loss': F.cross_entropy(y_hat, y)}
def validation_step(self, batch, batch_nb):
# OPTIONAL
x, y = batch
y_hat = self.forward(x)
return {'val_loss': F.cross_entropy(y_hat, y)}
def validation_end(self, outputs):
# OPTIONAL
avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
return {'avg_val_loss': avg_loss}
def test_step(self, batch, batch_nb):
# OPTIONAL
x, y = batch
y_hat = self.forward(x)
return {'test_loss': F.cross_entropy(y_hat, y)}
def test_end(self, outputs):
# OPTIONAL
avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
return {'avg_test_loss': avg_loss}
def configure_optimizers(self):
# REQUIRED
return torch.optim.Adam(self.parameters(), lr=0.02)
@pl.data_loader
def train_dataloader(self):
return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
@pl.data_loader
def val_dataloader(self):
# OPTIONAL
# can also return a list of val dataloaders
return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
@pl.data_loader
def test_dataloader(self):
# OPTIONAL
# can also return a list of test dataloaders
return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
```
---
### How do these methods fit into the broader training?
The LightningModule interface is on the right. Each method corresponds to a part of a research project. Lightning automates everything not in blue.
<p align="center">
<a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg">
<img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg" height="900px">
</a>
</p>
## Required Methods
### training_step
``` {.python}
def training_step(self, batch, batch_nb)
```
In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
**Params**
| Param | description |
|---|---|
| batch | The output of your dataloader. A tensor, tuple or list |
| batch_nb | Integer displaying which batch this is |
**Return**
Dictionary or OrderedDict
| key | value | is required |
|---|---|---|
| loss | tensor scalar | Y |
| progress_bar | Dict for progress bar display. Must have only tensors | N |
| log | Dict of metrics to add to logger. Must have only tensors (no images, etc) | N |
**Example**
``` {.python}
def training_step(self, batch, batch_nb):
x, y, z = batch
# implement your own
out = self.forward(x)
loss = self.loss(out, x)
logger_logs = {'training_loss': loss} # optional (MUST ALL BE TENSORS)
# if using TestTubeLogger or TensorboardLogger you can nest scalars
logger_logs = {'losses': logger_logs} # optional (MUST ALL BE TENSORS)
output = {
'loss': loss, # required
'progress_bar': {'training_loss': loss}, # optional (MUST ALL BE TENSORS)
'log': logger_logs
}
# return a dict
return output
```
If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.
``` {.python}
# Multiple optimizers (ie: GANs)
def training_step(self, batch, batch_nb, optimizer_idx):
if optimizer_idx == 0:
# do training_step with encoder
if optimizer_idx == 1:
# do training_step with decoder
```
If you add truncated back propagation through time you will also get an additional argument with the hidden states of the previous step.
``` {.python}
# Truncated back-propagation through time
def training_step(self, batch, batch_nb, hiddens):
# hiddens are the hiddens from the previous truncated backprop step
```
You can also return a -1 instead of a dict to stop the current loop. This is useful if you want to
break out of the current training epoch early.
---
### training_end
``` {.python}
def training_end(self, train_step_outputs)
```
In certain cases (dp, ddp2), you might want to use all outputs of every process to do something.
For instance, if using negative samples, you could run a batch via dp and use ALL the outputs
for a single softmax across the full batch (ie: the denominator would use the full batch).
In this case you should define training_end to perform those calculations.
**Params**
| Param | description |
|---|---|
| outputs | What you return in training_step.
**Return**
Dictionary or OrderedDict
| key | value | is required |
|---|---|---|
| loss | tensor scalar | Y |
| progress_bar | Dict for progress bar display. Must have only tensors | N |
| log | Dict of metrics to add to logger. Must have only tensors (no images, etc) | N |
**Example**
``` {.python}
# WITHOUT training_end
# if used in DP or DDP2, this batch is 1/nb_gpus large
def training_step(self, batch, batch_nb):
# batch is 1/nb_gpus big
x, y = batch
out = self.forward(x)
loss = self.softmax(out)
loss = nce_loss(loss)
return {'loss': loss}
# --------------
# with training_end to do softmax over the full batch
def training_step(self, batch, batch_nb):
# batch is 1/nb_gpus big
x, y = batch
out = self.forward(x)
return {'out': out}
def training_end(self, outputs):
# this out is now the full size of the batch
out = outputs['out']
# this softmax now uses the full batch size
loss = self.softmax(out)
loss = nce_loss(loss)
return {'loss': loss}
```
If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.
``` {.python}
# Multiple optimizers (ie: GANs)
def training_step(self, batch, batch_nb, optimizer_idx):
if optimizer_idx == 0:
# do training_step with encoder
if optimizer_idx == 1:
# do training_step with decoder
```
If you add truncated back propagation through time you will also get an additional argument with the hidden states of the previous step.
``` {.python}
# Truncated back-propagation through time
def training_step(self, batch, batch_nb, hiddens):
# hiddens are the hiddens from the previous truncated backprop step
```
You can also return a -1 instead of a dict to stop the current loop. This is useful if you want to
break out of the current training epoch early.
---
### train_dataloader
``` {.python}
@pl.data_loader
def train_dataloader(self)
```
Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
If you want to change the data during every epoch DON'T use the data_loader decorator.
##### Return
PyTorch DataLoader
**Example**
``` {.python}
@pl.data_loader
def train_dataloader(self):
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
loader = torch.utils.data.DataLoader(
dataset=dataset,
batch_size=self.hparams.batch_size,
shuffle=True
)
return loader
```
---
### configure_optimizers
``` {.python}
def configure_optimizers(self)
```
Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple.
Lightning will call .backward() and .step() on each one in every epoch. If you use 16 bit precision it will also handle that.
**Note:** If you use multiple optimizers, training_step will have an additional ```optimizer_idx``` parameter.
**Note 2:** If you use LBFGS lightning handles the closure function automatically for you.
##### Return
Return any of these 3 options:
Single optimizer
List or Tuple - List of optimizers
Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers
**Example**
``` {.python}
# most cases
def configure_optimizers(self):
opt = Adam(self.parameters(), lr=0.01)
return opt
# multiple optimizer case (eg: GAN)
def configure_optimizers(self):
generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
return generator_opt, disriminator_opt
# example with learning_rate schedulers
def configure_optimizers(self):
generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
return [generator_opt, disriminator_opt], [discriminator_sched]
```
If you need to control how often those optimizers step or override the default .step() schedule, override
the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step) hook.
## Optional Methods
### validation_step
``` {.python}
# if you have one val dataloader:
def validation_step(self, batch, batch_nb)
# if you have multiple val dataloaders:
def validation_step(self, batch, batch_nb, dataloader_idxdx)
```
**OPTIONAL**
If you don't need to validate you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy.
When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
The dict you return here will be available in the `validation_end` method.
**Params**
| Param | description |
|---|---|
| batch | The output of your dataloader. A tensor, tuple or list |
| batch_nb | Integer displaying which batch this is |
| dataloader_idx | Integer displaying which dataloader this is (only if multiple val datasets used) |
**Return**
| Return | description | optional |
|---|---|---|
| dict | Dict or OrderedDict - passed to the validation_end step | N |
**Example**
``` {.python}
# CASE 1: A single validation dataset
def validation_step(self, batch, batch_nb):
x, y = batch
# implement your own
out = self.forward(x)
loss = self.loss(out, y)
# log 6 example images
# or generated text... or whatever
sample_imgs = x[:6]
grid = torchvision.utils.make_grid(sample_imgs)
self.logger.experiment.add_image('example_images', grid, 0)
# calculate acc
labels_hat = torch.argmax(out, dim=1)
val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
# all optional...
# return whatever you need for the collation function validation_end
output = OrderedDict({
'val_loss': loss_val,
'val_acc': torch.tensor(val_acc), # everything must be a tensor
})
# return an optional dict
return output
```
If you pass in multiple validation datasets, validation_step will have an additional argument.
```python
# CASE 2: multiple validation datasets
def validation_step(self, batch, batch_nb, dataset_idx):
# dataset_idx tells you which dataset this is.
```
The ```dataset_idx``` corresponds to the order of datasets returned in ```val_dataloader```.
---
### validation_end
``` {.python}
def validation_end(self, outputs)
```
If you didn't define a validation_step, this won't be called. Called at the end of the validation loop with the outputs of validation_step.
The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.
Any keys present in 'log', 'progress_bar' or the rest of the dictionary are available for callbacks to access.
**Params**
| Param | description |
|---|---|
| outputs | List of outputs you defined in validation_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
**Return**
Dictionary or OrderedDict
| key | value | is required |
|---|---|---|
| progress_bar | Dict for progress bar display. Must have only tensors | N |
| log | Dict of metrics to add to logger. Must have only tensors (no images, etc) | N |
**Example**
With a single dataloader
``` {.python}
def validation_end(self, outputs):
"""
Called at the end of validation to aggregate outputs
:param outputs: list of individual outputs of each validation step
:return:
"""
val_loss_mean = 0
val_acc_mean = 0
for output in outputs:
val_loss_mean += output['val_loss']
val_acc_mean += output['val_acc']
val_loss_mean /= len(outputs)
val_acc_mean /= len(outputs)
tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
# show val_loss and val_acc in progress bar but only log val_loss
results = {
'progress_bar': tqdm_dict,
'log': {'val_loss': val_loss_mean.item()}
}
return results
```
With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
one entry per dataloader, while the inner list contains the individual outputs of
each validation step for that dataloader.
``` {.python}
def validation_end(self, outputs):
"""
Called at the end of validation to aggregate outputs
:param outputs: list of list of individual outputs of each validation step
:return:
"""
val_loss_mean = 0
val_acc_mean = 0
i = 0
for dataloader_outputs in outputs:
for output in dataloader_outputs:
val_loss_mean += output['val_loss']
val_acc_mean += output['val_acc']
i += 1
val_loss_mean /= i
val_acc_mean /= i
tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
# show val_loss and val_acc in progress bar but only log val_loss
results = {
'progress_bar': tqdm_dict,
'log': {'val_loss': val_loss_mean.item()}
}
return results
```
### test_step
``` {.python}
# if you have one test dataloader:
def test_step(self, batch, batch_nb)
# if you have multiple test dataloaders:
def test_step(self, batch, batch_nb, dataloader_idxdx)
```
**OPTIONAL**
If you don't need to test you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy.
When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
The dict you return here will be available in the `test_end` method.
This function is used when you execute `trainer.test()`.
**Params**
| Param | description |
|---|---|
| batch | The output of your dataloader. A tensor, tuple or list |
| batch_nb | Integer displaying which batch this is |
| dataloader_idx | Integer displaying which dataloader this is (only if multiple test datasets used) |
**Return**
| Return | description | optional |
|---|---|---|
| dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
**Example**
``` {.python}
# CASE 1: A single test dataset
def test_step(self, batch, batch_nb):
x, y = batch
# implement your own
out = self.forward(x)
loss = self.loss(out, y)
# calculate acc
labels_hat = torch.argmax(out, dim=1)
test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
# all optional...
# return whatever you need for the collation function test_end
output = OrderedDict({
'test_loss': loss_test,
'test_acc': torch.tensor(test_acc), # everything must be a tensor
})
# return an optional dict
return output
```
If you pass in multiple test datasets, test_step will have an additional argument.
```python
# CASE 2: multiple test datasets
def test_step(self, batch, batch_nb, dataset_idx):
# dataset_idx tells you which dataset this is.
```
The ```dataset_idx``` corresponds to the order of datasets returned in ```test_dataloader```.
---
### test_end
``` {.python}
def test_end(self, outputs)
```
If you didn't define a test_step, this won't be called.
Called at the end of the test step with the output of each test_step.
The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.
**Params**
| Param | description |
|---|---|
| outputs | List of outputs you defined in test_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
**Return**
| Return | description | optional |
|---|---|---|
| dict | Dict of OrderedDict with metrics to display in progress bar | Y |
**Example**
``` {.python}
def test_end(self, outputs):
"""
Called at the end of test to aggregate outputs
:param outputs: list of individual outputs of each test step
:return:
"""
test_loss_mean = 0
test_acc_mean = 0
for output in outputs:
test_loss_mean += output['test_loss']
test_acc_mean += output['test_acc']
test_loss_mean /= len(outputs)
test_acc_mean /= len(outputs)
tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
# show test_loss and test_acc in progress bar but only log test_loss
results = {
'progress_bar': tqdm_dict,
'log': {'test_loss': val_loss_mean.item()}
}
return results
```
With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
one entry per dataloader, while the inner list contains the individual outputs of
each validation step for that dataloader.
``` {.python}
def test_end(self, outputs):
"""
Called at the end of test to aggregate outputs
:param outputs: list of individual outputs of each test step
:return:
"""
test_loss_mean = 0
test_acc_mean = 0
i = 0
for dataloader_outputs in outputs:
for output in dataloader_outputs:
test_loss_mean += output['test_loss']
test_acc_mean += output['test_acc']
i += 1
test_loss_mean /= i
test_acc_mean /= i
tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
# show test_loss and test_acc in progress bar but only log test_loss
results = {
'progress_bar': tqdm_dict,
'log': {'test_loss': val_loss_mean.item()}
}
return results
```
---
### on_save_checkpoint
``` {.python}
def on_save_checkpoint(self, checkpoint)
```
Called by lightning to checkpoint your model. Lightning saves the training state (current epoch, global_step, etc)
and also saves the model state_dict. If you want to save anything else, use this method to add your own
key-value pair.
##### Return
Nothing
**Example**
``` {.python}
def on_save_checkpoint(self, checkpoint):
# 99% of use cases you don't need to implement this method
checkpoint['something_cool_i_want_to_save'] = my_cool_pickable_object
```
---
### on_load_checkpoint
``` {.python}
def on_load_checkpoint(self, checkpoint)
```
Called by lightning to restore your model. Lighting auto-restores global step, epoch, etc...
It also restores the model state_dict.
If you saved something with **on_save_checkpoint** this is your chance to restore this.
##### Return
Nothing
**Example**
``` {.python}
def on_load_checkpoint(self, checkpoint):
# 99% of the time you don't need to implement this method
self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']
```
---
### val_dataloader
``` {.python}
@pl.data_loader
def val_dataloader(self)
```
**OPTIONAL**
If you don't need a validation dataset and a validation_step, you don't need to implement this method.
Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
If you want to change the data during every epoch DON'T use the data_loader decorator.
##### Return
PyTorch DataLoader or list of PyTorch Dataloaders.
**Example**
``` {.python}
@pl.data_loader
def val_dataloader(self):
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
loader = torch.utils.data.DataLoader(
dataset=dataset,
batch_size=self.hparams.batch_size,
shuffle=True
)
return loader
# can also return multiple dataloaders
@pl.data_loader
def val_dataloader(self):
return [loader_a, loader_b, ..., loader_n]
```
In the case where you return multiple val_dataloaders, the validation_step will have an arguement ```dataset_idx```
which matches the order here.
---
### test_dataloader
``` {.python}
@pl.data_loader
def test_dataloader(self)
```
**OPTIONAL**
If you don't need a test dataset and a test_step, you don't need to implement this method.
Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
If you want to change the data during every epoch DON'T use the data_loader decorator.
##### Return
PyTorch DataLoader
**Example**
``` {.python}
@pl.data_loader
def test_dataloader(self):
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
loader = torch.utils.data.DataLoader(
dataset=dataset,
batch_size=self.hparams.batch_size,
shuffle=True
)
return loader
```
---
### add_model_specific_args
``` {.python}
@staticmethod
def add_model_specific_args(parent_parser, root_dir)
```
Lightning has a list of default argparse commands.
This method is your chance to add or modify commands specific to your model.
The [hyperparameter argument parser](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/) is available anywhere in your model by calling self.hparams.
##### Return
An argument parser
**Example**
``` {.python}
@staticmethod
def add_model_specific_args(parent_parser, root_dir):
parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
# param overwrites
# parser.set_defaults(gradient_clip_val=5.0)
# network params
parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
parser.add_argument('--in_features', default=28*28)
parser.add_argument('--out_features', default=10)
parser.add_argument('--hidden_dim', default=50000) # use 500 for CPU, 50000 for GPU to see speed difference
# data
parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
# training params (opt)
parser.opt_list('--learning_rate', default=0.001, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
tunable=False)
parser.opt_list('--batch_size', default=256, type=int, options=[32, 64, 128, 256], tunable=False)
parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)
return parser
```
docs/LightningModule/methods.md
-67
View File
@@ -1,67 +0,0 @@
Lightning modules are strict superclasses of torch.nn.Module. A LightningModule offers the following in addition to that API.
---
### freeze
Freeze all params for inference
```{.python}
model = MyLightningModule(...)
model.freeze()
```
---
### load_from_checkpoint
This is the easiest/fastest way which loads hyperparameters and weights from a checkpoint,
such as the one saved by the `ModelCheckpoint` callback
```{.python}
pretrained_model = MyLightningModule.load_from_checkpoint(
checkpoint_path='/path/to/pytorch_checkpoint.ckpt'
)
# predict
pretrained_model.eval()
pretrained_model.freeze()
y_hat = pretrained_model(x)
```
---
### load_from_metrics
If you're using test tube, there is an alternate method which uses the meta_tags.csv
file from test-tube to rebuild the model. The meta_tags.csv file can be found in the
test-tube experiment save_dir.
```{.python}
pretrained_model = MyLightningModule.load_from_metrics(
weights_path='/path/to/pytorch_checkpoint.ckpt',
tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
on_gpu=True,
map_location=None
)
# predict
pretrained_model.eval()
pretrained_model.freeze()
y_hat = pretrained_model(x)
```
**Params**
| Param | description |
|---|---|
| weights_path | Path to a PyTorch checkpoint |
| tags_csv | Path to meta_tags.csv file generated by the test-tube Experiment |
| on_gpu | if True, puts model on GPU. Make sure to use transforms option if model devices have changed |
| map_location | A dictionary mapping saved weight GPU devices to new GPU devices |
**Returns**
LightningModule - The pretrained LightningModule
---
### unfreeze
Unfreeze all params for inference
```{.python}
model = MyLightningModule(...)
model.unfreeze()
```
docs/LightningModule/properties.md
-64
View File
@@ -1,64 +0,0 @@
A LightningModule has the following properties which you can access at any time
---
#### current_epoch
The current epoch
---
#### dtype
Current dtype
---
#### logger
A reference to the logger you passed into trainer.
Passing a logger is optional. If you don't pass one in, Lightning will create one for you automatically.
This logger saves logs to '''/os.getcwd()/lightning_logs'''
```python
Trainer(logger=your_logger)
```
Call it from anywhere in your LightningModule to add metrics, images, etc... whatever your logger supports.
Here is an example using the TestTubeLogger (which is a wrapper on [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html) with versioned folder structure).
```{.python}
# if logger is a tensorboard logger or TestTubeLogger
self.logger.experiment.add_embedding(...)
self.logger.experiment.log({'val_loss': 0.9})
self.logger.experiment.add_scalars(...)
```
---
#### global_step
Total training batches seen across all epochs
---
#### gradient_clip_val
The current gradient clip value
---
#### on_gpu
True if your model is currently running on GPUs. Useful to set flags around the LightningModule for different CPU vs GPU behavior.
---
#### trainer
Last resort access to any state the trainer has. Changing certain properties here could affect your training run.
```{.python}
self.trainer.optimizers
self.trainer.current_epoch
...
```
## Debugging
The LightningModule also offers these tricks to help debug.
---
#### example_input_array
In the LightningModule init, you can set a dummy tensor for this property
to get a print out of sizes coming into and out of every layer.
```python
def __init__(self):
# put the dimensions of the first input to your system
self.example_input_array = torch.rand(5, 28 * 28)
```
docs/Makefile
+19
View File
@@ -0,0 +1,19 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/Trainer/Checkpointing.md
-101
View File
@@ -1,101 +0,0 @@
Lightning can automate saving and loading checkpoints.
---
### Model saving
Checkpointing is enabled by default to the current working directory.
To change the checkpoint path pass in :
```python
Trainer(default_save_path='/your/path/to/save/checkpoints')
```
To modify the behavior of checkpointing pass in your own callback.
```{.python}
from pytorch_lightning.callbacks import ModelCheckpoint
# DEFAULTS used by the Trainer
checkpoint_callback = ModelCheckpoint(
filepath=os.getcwd(),
save_top_k=1,
verbose=True,
monitor='val_loss',
mode='min',
prefix=''
)
trainer = Trainer(checkpoint_callback=checkpoint_callback)
```
The `save_top_k` options works in the following ways:
| save_top_k | behavior |
| -------- | ----- |
| 0 | no models are saved |
| -1 | all models are saved |
| k >= 1 | the best k models are saved |
Also, if `save_top_k` >= 2 and the callback is called multiple
times inside an epoch, the name of the saved file will be
appended with a version count starting with `v0`.
---
### Restoring training session
You might want to not only load a model but also continue training it. Use this method to
restore the trainer state as well. This will continue from the epoch and global step you last left off.
However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).
Lightning will restore the session if you pass a logger with the same version and there's a saved checkpoint.
``` {.python}
from pytorch_lightning import Trainer
from pytorch_lightning.logging import TestTubeLogger
logger = TestTubeLogger(
save_dir='./savepath',
version=1 # An existing version with a saved checkpoint
)
trainer = Trainer(
logger=logger,
default_save_path='./savepath'
)
# this fit call loads model weights and trainer state
# the trainer continues seamlessly from where you left off
# without having to do anything else.
trainer.fit(model)
```
The trainer restores:
- global_step
- current_epoch
- All optimizers
- All lr_schedulers
- Model weights
You can even change the logic of your model as long as the weights and "architecture" of
the system isn't different. If you add a layer, for instance, it might not work.
At a rough level, here's [what happens inside Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/model_saving.py#L63):
```python
self.global_step = checkpoint['global_step']
self.current_epoch = checkpoint['epoch']
# restore the optimizers
optimizer_states = checkpoint['optimizer_states']
for optimizer, opt_state in zip(self.optimizers, optimizer_states):
optimizer.load_state_dict(opt_state)
# restore the lr schedulers
lr_schedulers = checkpoint['lr_schedulers']
for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
scheduler.load_state_dict(lrs_state)
# uses the model you passed into trainer
model.load_state_dict(checkpoint['state_dict'])
```
docs/Trainer/Distributed training.md
-262
View File
@@ -1,262 +0,0 @@
Lightning makes multi-gpu training and 16 bit training trivial.
*Note:*
None of the flags below require changing anything about your lightningModel definition.
---
#### Choosing a backend
Lightning supports two backends. DataParallel and DistributedDataParallel. Both can be used for single-node multi-GPU training.
For multi-node training you must use DistributedDataParallel.
##### DataParallel (dp)
Splits a batch across multiple GPUs on the same node. Cannot be used for multi-node training.
##### DistributedDataParallel (ddp)
Trains a copy of the model on each GPU and only syncs gradients. If used with DistributedSampler, each GPU trains
on a subset of the full dataset.
##### DistributedDataParallel-2 (ddp2)
Works like DDP, except each node trains a single copy of the model using ALL GPUs on that node.
Very useful when dealing with negative samples, etc...
You can toggle between each mode by setting this flag.
``` {.python}
# DEFAULT (when using single GPU or no GPUs)
trainer = Trainer(distributed_backend=None)
# Change to DataParallel (gpus > 1)
trainer = Trainer(distributed_backend='dp')
# change to distributed data parallel (gpus > 1)
trainer = Trainer(distributed_backend='ddp')
# change to distributed data parallel (gpus > 1)
trainer = Trainer(distributed_backend='ddp2')
```
If you request multiple nodes, the back-end will auto-switch to ddp.
We recommend you use DistributedDataparallel even for single-node multi-GPU training. It is MUCH faster than DP but *may*
have configuration issues depending on your cluster.
For a deeper understanding of what lightning is doing, feel free to read [this guide](https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565).
---
#### Distributed and 16-bit precision.
Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.
Below are the possible configurations we support.
| 1 GPU | 1+ GPUs | DP | DDP | 16-bit | command |
|---|---|---|---|---|---|
| Y | | | | | ```Trainer(gpus=1)``` |
| Y | | | | Y | ```Trainer(gpus=1, use_amp=True)``` |
| | Y | Y | | | ```Trainer(gpus=k, distributed_backend='dp')``` |
| | Y | | Y | | ```Trainer(gpus=k, distributed_backend='ddp')``` |
| | Y | | Y | Y | ```Trainer(gpus=k, distributed_backend='ddp', use_amp=True)``` |
You also have the option of specifying which GPUs to use by passing a list:
```python
# DEFAULT (int) specifies how many GPUs to use.
Trainer(gpus=k)
# Above is equivalent to
Trainer(gpus=list(range(k)))
# You specify which GPUs (don't use if running on cluster)
Trainer(gpus=[0, 1])
# can also be a string
Trainer(gpus='0, 1')
# can also be -1 or '-1', this uses all available GPUs
# this is equivalent to list(range(torch.cuda.available_devices()))
Trainer(gpus=-1)
```
---
#### CUDA flags
CUDA flags make certain GPUs visible to your script.
Lightning sets these for you automatically, there's NO NEED to do this yourself.
```python
# lightning will set according to what you give the trainer
# os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
# os.environ["CUDA_VISIBLE_DEVICES"] = "0"
```
However, when using a cluster, Lightning will NOT set these flags (and you should not either).
SLURM will set these for you.
---
#### 16-bit mixed precision
16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.
First, install apex (if install fails, look [here](https://github.com/NVIDIA/apex)):
```bash
$ git clone https://github.com/NVIDIA/apex
$ cd apex
# ------------------------
# OPTIONAL: on your cluster you might need to load cuda 10 or 9
# depending on how you installed PyTorch
# see available modules
module avail
# load correct cuda before install
module load cuda-10.0
# ------------------------
# make sure you've loaded a cuda version > 4.0 and < 7.0
module load gcc-6.1.0
$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```
then set this use_amp to True.
``` {.python}
# DEFAULT
trainer = Trainer(amp_level='O2', use_amp=False)
```
---
#### Single-gpu
Make sure you're on a GPU machine.
```python
# DEFAULT
trainer = Trainer(gpus=1)
```
---
#### multi-gpu
Make sure you're on a GPU machine. You can set as many GPUs as you want.
In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
```python
# to use DataParallel
trainer = Trainer(gpus=8, distributed_backend='dp')
# RECOMMENDED use DistributedDataParallel
trainer = Trainer(gpus=8, distributed_backend='ddp')
```
---
#### Multi-node
Multi-node training is easily done by specifying these flags.
```python
# train on 12*8 GPUs
trainer = Trainer(gpus=8, nb_gpu_nodes=12, distributed_backend='ddp')
```
You must configure your job submission script correctly for the trainer to work. Here is an example
script for the above trainer configuration.
```sh
#!/bin/bash -l
# SLURM SUBMIT SCRIPT
#SBATCH --nodes=12
#SBATCH --gres=gpu:8
#SBATCH --ntasks-per-node=8
#SBATCH --mem=0
#SBATCH --time=0-02:00:00
# activate conda env
conda activate my_env
# -------------------------
# OPTIONAL
# -------------------------
# debugging flags (optional)
# export NCCL_DEBUG=INFO
# export PYTHONFAULTHANDLER=1
# PyTorch comes with prebuilt NCCL support... but if you have issues with it
# you might need to load the latest version from your modules
# module load NCCL/2.4.7-1-cuda.10.0
# on your cluster you might need these:
# set the network interface
# export NCCL_SOCKET_IFNAME=^docker0,lo
# -------------------------
# random port between 12k and 20k
export MASTER_PORT=$((12000 + RANDOM % 20000))
# run script from above
python my_main_file.py
```
**NOTE:** When running in DDP mode, any errors in your code will show up as an NCCL issue.
Set the ```NCCL_DEBUG=INFO``` flag to see the ACTUAL error.
Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a
portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).
```python
# ie: this:
dataset = myDataset()
dataloader = Dataloader(dataset)
# becomes:
dataset = myDataset()
dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
dataloader = Dataloader(dataset, sampler=dist_sampler)
```
#### Auto-slurm-job-submission
Instead of manually building SLURM scripts, you can use the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) to
do this for you. The SlurmCluster can also run a grid search if you pass in a [HyperOptArgumentParser](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/).
Here is an example where you run a grid search of 9 combinations of hyperparams.
[The full examples are here](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/new_project_templates/multi_node_examples).
```python
# grid search 3 values of learning rate and 3 values of number of layers for your net
# this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
parser.opt_list('--learning_rate', default=0.001, type=float, options=[1e-3, 1e-2, 1e-1], tunable=True)
parser.opt_list('--layers', default=1, type=float, options=[16, 32, 64], tunable=True)
hyperparams = parser.parse_args()
# Slurm cluster submits 9 jobs, each with a set of hyperparams
cluster = SlurmCluster(
hyperparam_optimizer=hyperparams,
log_path='/some/path/to/save',
)
# OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
# which interface your nodes use for communication
cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
# see output of the NCCL connection process
# NCCL is how the nodes talk to each other
cluster.add_command('export NCCL_DEBUG=INFO')
# setting a master port here is a good idea.
cluster.add_command('export MASTER_PORT=%r' % PORT)
# ************** DON'T FORGET THIS ***************
# MUST load the latest NCCL version
cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
# configure cluster
cluster.per_experiment_nb_nodes = 12
cluster.per_experiment_nb_gpus = 8
cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
# submit a script with 9 combinations of hyper params
# (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
cluster.optimize_parallel_cluster_gpu(
main,
nb_trials=9, # how many permutations of the grid search to run
job_name='name_for_squeue'
)
```
The other option is that you generate scripts on your own via a bash command or use another library...
---
#### Self-balancing architecture
Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.
COMING SOON.
docs/Trainer/Logging.md
-242
View File
@@ -1,242 +0,0 @@
Lighting offers options for logging information about model, gpu usage, etc, via several different logging frameworks. It also offers printing options for training monitoring.
---
### default_save_path
Lightning sets a default TestTubeLogger and CheckpointCallback for you which log to
```os.getcwd()``` by default. To modify the logging path you can set:
```python
Trainer(default_save_path='/your/path/to/save/checkpoints')
```
If you need more custom behavior (different paths for both, different metrics, etc...)
from the logger and the checkpointCallback, pass in your own instances as explained below.
---
### Setting up logging
The trainer inits a default logger for you (TestTubeLogger). All logs will
go to the current working directory under a folder named ```os.getcwd()/lightning_logs``.
If you want to modify the default logging behavior even more, pass in a logger
(which should inherit from `LightningBaseLogger`).
```{.python}
my_logger = MyLightningLogger(...)
trainer = Trainer(logger=my_logger)
```
The path in this logger will overwrite default_save_path.
Lightning supports several common experiment tracking frameworks out of the box
---
#### Test tube
Log using [test tube](https://williamfalcon.github.io/test-tube/). Test tube logger is
a strict subclass of [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html), refer to their
documentation for all supported operations. The TestTubeLogger adds a nicer folder structure
to manage experiments and snapshots all hyperparameters you pass to a LightningModule.
```{.python}
from pytorch_lightning.logging import TestTubeLogger
tt_logger = TestTubeLogger(
save_dir=".",
name="default",
debug=False,
create_git_tag=False
)
trainer = Trainer(logger=tt_logger)
```
Use the logger anywhere in you LightningModule as follows:
```python
def train_step(...):
# example
self.logger.experiment.whatever_method_summary_writer_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.add_histogram(...)
```
---
#### MLFlow
Log using [mlflow](https://mlflow.org)
```{.python}
from pytorch_lightning.logging import MLFlowLogger
mlf_logger = MLFlowLogger(
experiment_name="default",
tracking_uri="file:/."
)
trainer = Trainer(logger=mlf_logger)
```
Use the logger anywhere in you LightningModule as follows:
```python
def train_step(...):
# example
self.logger.experiment.whatever_ml_flow_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.whatever_ml_flow_supports(...)
```
---
#### Comet.ml
Log using [comet](https://www.comet.ml)
Comet logger can be used in either online or offline mode.
To log in online mode, CometLogger requries an API key:
```{.python}
from pytorch_lightning.logging import CometLogger
# arguments made to CometLogger are passed on to the comet_ml.Experiment class
comet_logger = CometLogger(
api_key=os.environ["COMET_KEY"],
workspace=os.environ["COMET_WORKSPACE"], # Optional
project_name="default_project", # Optional
rest_api_key=os.environ["COMET_REST_KEY"], # Optional
experiment_name="default" # Optional
)
trainer = Trainer(logger=comet_logger)
```
To log in offline mode, CometLogger requires a path to a local directory:
```{.python}
from pytorch_lightning.logging import CometLogger
# arguments made to CometLogger are passed on to the comet_ml.Experiment class
comet_logger = CometLogger(
save_dir=".",
workspace=os.environ["COMET_WORKSPACE"], # Optional
project_name="default_project", # Optional
rest_api_key=os.environ["COMET_REST_KEY"], # Optional
experiment_name="default" # Optional
)
trainer = Trainer(logger=comet_logger)
```
Use the logger anywhere in you LightningModule as follows:
```python
def train_step(...):
# example
self.logger.experiment.whatever_comet_ml_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.whatever_comet_ml_supports(...)
```
---
#### Custom logger
You can implement your own logger by writing a class that inherits from
`LightningLoggerBase`. Use the `rank_zero_only` decorator to make sure that
only the first process in DDP training logs data.
```{.python}
from pytorch_lightning.logging import LightningLoggerBase, rank_zero_only
class MyLogger(LightningLoggerBase):
@rank_zero_only
def log_hyperparams(self, params):
# params is an argparse.Namespace
# your code to record hyperparameters goes here
pass
@rank_zero_only
def log_metrics(self, metrics, step_num):
# metrics is a dictionary of metric names and values
# your code to record metrics goes here
pass
def save(self):
# Optional. Any code necessary to save logger data goes here
pass
@rank_zero_only
def finalize(self, status):
# Optional. Any code that needs to be run after training
# finishes goes here
```
If you write a logger than may be useful to others, please send
a pull request to add it to Lighting!
---
#### Using loggers
You can call the logger anywhere from your LightningModule by doing:
```python
def train_step(...):
# example
self.logger.experiment.whatever_method_summary_writer_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.add_histogram(...)
```
#### Display metrics in progress bar
``` {.python}
# DEFAULT
trainer = Trainer(show_progress_bar=True)
```
---
#### Log metric row every k batches
Every k batches lightning will make an entry in the metrics log
``` {.python}
# DEFAULT (ie: save a .csv log file every 10 batches)
trainer = Trainer(row_log_interval=10)
```
---
#### Log GPU memory
Logs GPU memory when metrics are logged.
``` {.python}
# DEFAULT
trainer = Trainer(log_gpu_memory=None)
# log only the min/max utilization
trainer = Trainer(log_gpu_memory='min_max')
# log all the GPU memory (if on DDP, logs only that node)
trainer = Trainer(log_gpu_memory='all')
```
---
#### Process position
When running multiple models on the same machine we want to decide which progress bar to use.
Lightning will stack progress bars according to this value.
``` {.python}
# DEFAULT
trainer = Trainer(process_position=0)
# if this is the second model on the node, show the second progress bar below
trainer = Trainer(process_position=1)
```
---
#### Save a snapshot of all hyperparameters
Automatically log hyperparameters stored in the `hparams` attribute as an `argparse.Namespace`
``` {.python}
class MyModel(pl.Lightning):
def __init__(self, hparams):
self.hparams = hparams
...
args = parser.parse_args()
model = MyModel(args)
logger = TestTubeLogger(...)
t = Trainer(logger=logger)
trainer.fit(model)
```
---
#### Write logs file to csv every k batches
Every k batches, lightning will write the new logs to disk
``` {.python}
# DEFAULT (ie: save a .csv log file every 100 batches)
trainer = Trainer(log_save_interval=100)
```
docs/Trainer/SLURM Managed Cluster.md
-112
View File
@@ -1,112 +0,0 @@
Lightning supports model training on a cluster managed by SLURM in the following cases:
1. Training on a single cpu or single GPU.
2. Train on multiple GPUs on the same node using DataParallel or DistributedDataParallel
3. Training across multiple GPUs on multiple different nodes via DistributedDataParallel.
**Note: A node means a machine with multiple GPUs**
---
#### Running grid search on a cluster
To use lightning to run a hyperparameter search (grid-search or random-search) on a cluster do 4 things:
(1). Define the parameters for the grid search
```{.python}
from test_tube import HyperOptArgumentParser
# subclass of argparse
parser = HyperOptArgumentParser(strategy='random_search')
parser.add_argument('--learning_rate', default=0.002, type=float, help='the learning rate')
# let's enable optimizing over the number of layers in the network
parser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4, 8])
hparams = parser.parse_args()
```
**NOTE** You must set ```Tunable=True``` for that argument to be considered in the permutation set. Otherwise
test-tube will use the default value. This flag is useful when you don't want to search over an argument and
want to use the default instead.
(2). Define the cluster options in the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) (over 5 nodes and 8 gpus)
```{.python}
from test_tube.hpc import SlurmCluster
# hyperparameters is a test-tube hyper params object
# see https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/
hyperparams = args.parse()
# init cluster
cluster = SlurmCluster(
hyperparam_optimizer=hyperparams,
log_path='/path/to/log/results/to',
python_cmd='python3'
)
# let the cluster know where to email for a change in job status (ie: complete, fail, etc...)
cluster.notify_job_status(email='some@email.com', on_done=True, on_fail=True)
# set the job options. In this instance, we'll run 20 different models
# each with its own set of hyperparameters giving each one 1 GPU (ie: taking up 20 GPUs)
cluster.per_experiment_nb_gpus = 8
cluster.per_experiment_nb_nodes = 5
# we'll request 10GB of memory per node
cluster.memory_mb_per_node = 10000
# set a walltime of 10 minues
cluster.job_time = '10:00'
```
(3). Make a main function with your model and trainer. Each job will call this function with a particular
hparams configuration.
```{.python}
from pytorch_lightning import Trainer
def train_fx(trial_hparams, cluster_manager, _):
# hparams has a specific set of hyperparams
my_model = MyLightningModel()
# give the trainer the cluster object
trainer = Trainer()
trainer.fit(my_model)
```
(3). Start the grid/random search
```{.python}
# run the models on the cluster
cluster.optimize_parallel_cluster_gpu(
train_fx,
nb_trials=20,
job_name='my_grid_search_exp_name',
job_display_name='my_exp')
```
**NOTE** nb_trials specifies how many of the possible permutations to use. If using ```grid_search``` it will use
the depth first ordering. If using ```random_search``` it will use the first k shuffled options. FYI, random search
has been shown to be just as good as any Bayesian optimization method when using a reasonable number of samples (60),
[see this paper for more information](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf).
---
#### Walltime auto-resubmit
Lightning automatically resubmits jobs when they reach the walltime. Make sure to set the SIGUSR1 signal in
your SLURM script.
```bash
# 90 seconds before training ends
#SBATCH --signal=SIGUSR1@90
```
When lightning receives the SIGUSR1 signal it will:
1. save a checkpoint with 'hpc_ckpt' in the name.
2. resubmit the job using the SLURM_JOB_ID
When the script starts again, Lightning will:
1. search for a 'hpc_ckpt' checkpoint.
2. restore the model, optimizers, schedulers, epoch, etc...
docs/Trainer/Testing loop.md
-31
View File
@@ -1,31 +0,0 @@
To ensure you don't accidentally use test data to guide training decisions Lightning makes running the test set deliberate.
---
#### test
You have two options to run the test set.
First case is where you test right after a full training routine.
``` {.python}
# run full training
trainer.fit(model)
# run test set
trainer.test()
```
Second case is where you load a model and run the test set
```{.python}
model = MyLightningModule.load_from_metrics(
weights_path='/path/to/pytorch_checkpoint.ckpt',
tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
on_gpu=True,
map_location=None
)
# init trainer with whatever options
trainer = Trainer(...)
# test (pass in the model)
trainer.test(model)
```
In this second case, the options you pass to trainer will be used when running the test set (ie: 16-bit, dp, ddp, etc...)
docs/Trainer/Training Loop.md
-126
View File
@@ -1,126 +0,0 @@
The lightning training loop handles everything except the actual computations of your model. To decide what will happen in your training loop, define the [training_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#training_step).
Below are all the things lightning automates for you in the training loop.
---
#### Accumulated gradients
Accumulated gradients runs K small batches of size N before doing a backwards pass. The effect is a large effective batch size of size KxN.
``` {.python}
# DEFAULT (ie: no accumulated grads)
trainer = Trainer(accumulate_grad_batches=1)
```
---
#### Force training for min or max epochs
It can be useful to force training for a minimum number of epochs or limit to a max number
``` {.python}
# DEFAULT
trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000)
```
---
#### Early stopping
The trainer already sets up default early stopping for you.
To modify this behavior, pass in your own EarlyStopping callback.
``` {.python}
from pytorch_lightning.callbacks import EarlyStopping
# DEFAULTS used by Trainer
early_stop_callback = EarlyStopping(
monitor='val_loss',
min_delta=0.00,
patience=3,
verbose=False,
mode='min'
)
# without passing anything in, uses the default callback above
trainer = Trainer()
# pass in your own to override the default callback
trainer = Trainer(early_stop_callback=early_stop_callback)
# pass in None to disable it
trainer = Trainer(early_stop_callback=None)
```
---
#### Force disable early stop
To disable early stopping pass None to the early_stop_callback
``` {.python}
# DEFAULT
trainer = Trainer(early_stop_callback=None)
```
---
#### Gradient Clipping
Gradient clipping may be enabled to avoid exploding gradients.
Specifically, this will [clip the gradient norm computed over all model parameters *together*](https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_).
``` {.python}
# DEFAULT (ie: don't clip)
trainer = Trainer(gradient_clip_val=0)
# clip gradients with norm above 0.5
trainer = Trainer(gradient_clip_val=0.5)
```
---
#### Inspect gradient norms
Looking at grad norms can help you figure out where training might be going wrong.
``` {.python}
# DEFAULT (-1 doesn't track norms)
trainer = Trainer(track_grad_norm=-1)
# track the LP norm (P=2 here)
trainer = Trainer(track_grad_norm=2)
```
---
#### Set how much of the training set to check
If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
``` {.python}
# DEFAULT
trainer = Trainer(train_percent_check=1.0)
# check 10% only
trainer = Trainer(train_percent_check=0.1)
```
---
#### Packed sequences as inputs
When using PackedSequence, do 2 things:
1. return either a padded tensor in dataset or a list of variable length tensors in the dataloader collate_fn (example above shows the list implementation).
2. Pack the sequence in forward or training and validation steps depending on use case.
``` {.python}
# For use in dataloader
def collate_fn(batch):
x = [item[0] for item in batch]
y = [item[1] for item in batch]
return x, y
# In module
def training_step(self, batch, batch_nb):
x = rnn.pack_sequence(batch[0], enforce_sorted=False)
y = rnn.pack_sequence(batch[1], enforce_sorted=False)
```
---
#### Truncated Backpropagation Through Time
There are times when multiple backwards passes are needed for each batch. For example, it may save memory to use Truncated Backpropagation Through Time when training RNNs.
When this flag is enabled each batch is split into sequences of size truncated_bptt_steps and passed to training_step(...) separately. A default splitting function is provided, however, you can override it for more flexibility. See [tbptt_split_batch](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks#tbptt_split_batch).
``` {.python}
# DEFAULT (single backwards pass per batch)
trainer = Trainer(truncated_bptt_steps=None)
# (split batch into sequences of size 2)
trainer = Trainer(truncated_bptt_steps=2)
```
docs/Trainer/Validation loop.md
-70
View File
@@ -1,70 +0,0 @@
The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the [validation_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#validation_step).
Below are all the things lightning automates for you in the validation loop.
**Note**
Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don't have to wait until a full epoch to catch possible validation issues.
---
#### Check validation every n epochs
If you have a small dataset you might want to check validation every n epochs
``` {.python}
# DEFAULT
trainer = Trainer(check_val_every_n_epoch=1)
```
---
#### Set how much of the validation set to check
If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
``` {.python}
# DEFAULT
trainer = Trainer(val_percent_check=1.0)
# check 10% only
trainer = Trainer(val_percent_check=0.1)
```
---
#### Set how much of the test set to check
If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
``` {.python}
# DEFAULT
trainer = Trainer(test_percent_check=1.0)
# check 10% only
trainer = Trainer(test_percent_check=0.1)
```
---
#### Set validation check frequency within 1 training epoch
For large datasets it's often desirable to check validation multiple times within a training loop.
Pass in a float to check that often within 1 training epoch.
Pass in an int k to check every k training batches. Must use an int if using
an IterableDataset.
``` {.python}
# DEFAULT
trainer = Trainer(val_check_interval=0.95)
# check every .25 of an epoch
trainer = Trainer(val_check_interval=0.25)
# check every 100 train batches (ie: for IterableDatasets or fixed frequency)
trainer = Trainer(val_check_interval=100)
```
---
#### Set the number of validation sanity steps
Lightning runs a few steps of validation in the beginning of training. This avoids crashing in the validation loop sometime deep into a lengthy training loop.
``` {.python}
# DEFAULT
trainer = Trainer(nb_sanity_val_steps=5)
```
You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.
docs/Trainer/debugging.md
-59
View File
@@ -1,59 +0,0 @@
These flags are useful to help debug a model.
---
#### Fast dev run
This flag is meant for debugging a full train/val/test loop. It'll activate callbacks, everything but only with 1 training and 1 validation batch.
Use this to debug a full run of your program quickly
``` {.python}
# DEFAULT
trainer = Trainer(fast_dev_run=False)
```
---
#### Inspect gradient norms
Looking at grad norms can help you figure out where training might be going wrong.
``` {.python}
# DEFAULT (-1 doesn't track norms)
trainer = Trainer(track_grad_norm=-1)
# track the LP norm (P=2 here)
trainer = Trainer(track_grad_norm=2)
```
---
#### Make model overfit on subset of data
A useful debugging trick is to make your model overfit a tiny fraction of the data.
setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
``` {.python}
# DEFAULT don't overfit (ie: normal training)
trainer = Trainer(overfit_pct=0.0)
# overfit on 1% of data
trainer = Trainer(overfit_pct=0.01)
```
---
#### Print the parameter count by layer
By default lightning prints a list of parameters *and submodules* when it starts training.
``` {.python}
# DEFAULT print a full list of all submodules and their parameters.
trainer = Trainer(weights_summary='full')
# only print the top-level modules (i.e. the children of LightningModule).
trainer = Trainer(weights_summary='top')
```
---
#### Print which gradients are nan
This option prints a list of tensors with nan gradients.
``` {.python}
# DEFAULT
trainer = Trainer(print_nan_grads=False)
```
---
#### Log GPU usage
Lightning automatically logs gpu usage to the test tube logs. It'll only do it at the metric logging interval, so it doesn't slow down training.
docs/Trainer/hooks.md
-266
View File
@@ -1,266 +0,0 @@
# Hooks
[[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py)]
There are cases when you might want to do something different at different parts of the training/validation loop.
To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
**Contributing** If there's a hook you'd like to add, simply:
1. Fork PyTorchLightning.
2. Add the hook [here](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py).
3. Add the correct place in the [Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py) where it should be called.
---
#### on_epoch_start
Called in the training loop at the very beginning of the epoch.
```python
def on_epoch_start(self):
# do something when the epoch starts
```
---
#### on_epoch_end
Called in the training loop at the very end of the epoch.
```python
def on_epoch_end(self):
# do something when the epoch ends
```
---
#### on_batch_start
Called in the training loop before anything happens for that batch.
```python
def on_batch_start(self):
# do something when the batch starts
```
---
#### on_batch_end
Called in the training loop after the batch.
```python
def on_batch_end(self):
# do something when the batch ends
```
---
#### on_pre_performance_check
Called at the very beginning of the validation loop.
```python
def on_pre_performance_check(self):
# do something before validation starts
```
---
#### on_post_performance_check
Called at the very end of the validation loop.
```python
def on_post_performance_check(self):
# do something before validation end
```
---
#### optimizer_step
Calls .step() and .zero_grad for each optimizer.
You can override this method to adjust how you do the optimizer step for each optimizer
Called once per optimizer
```python
# DEFAULT
def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
optimizer.step()
optimizer.zero_grad()
# Alternating schedule for optimizer steps (ie: GANs)
def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
# update generator opt every 2 steps
if optimizer_i == 0:
if batch_nb % 2 == 0 :
optimizer.step()
optimizer.zero_grad()
# update discriminator opt every 4 steps
if optimizer_i == 1:
if batch_nb % 4 == 0 :
optimizer.step()
optimizer.zero_grad()
# ...
# add as many optimizers as you want
```
This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:
```python
# learning rate warm-up
def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
# warm up lr
if self.trainer.global_step < 500:
lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
for pg in optimizer.param_groups:
pg['lr'] = lr_scale * self.hparams.learning_rate
# update params
optimizer.step()
optimizer.zero_grad()
```
---
#### on_before_zero_grad
Called in the training loop after taking an optimizer step and before zeroing grads.
Good place to inspect weight information with weights updated.
Called once per optimizer
```python
def on_before_zero_grad(self, optimizer):
# do something with the optimizer or inspect it.
```
---
#### backward
Called to perform backward step.
Feel free to override as needed.
The loss passed in has already been scaled for accumulated gradients if requested.
```python
def backward(self, use_amp, loss, optimizer):
"""
Override backward with your own implementation if you need to
:param use_amp: Whether amp was requested or not
:param loss: Loss is already scaled by accumulated grads
:param optimizer: Current optimizer being used
:return:
"""
if use_amp:
with amp.scale_loss(loss, optimizer) as scaled_loss:
scaled_loss.backward()
else:
loss.backward()
```
---
#### on_after_backward
Called in the training loop after model.backward()
This is the ideal place to inspect or log gradient information
```python
def on_after_backward(self):
# example to inspect gradient information in tensorboard
if self.trainer.global_step % 25 == 0: # don't make the tf file huge
params = self.state_dict()
for k, v in params.items():
grads = v
name = k
self.logger.experiment.add_histogram(tag=name, values=grads, global_step=self.trainer.global_step)
```
---
#### tbptt_split_batch
Called in the training loop after on_batch_start if `truncated_bptt_steps > 0`. Each returned batch split is passed separately to training_step(...).
```python
def tbptt_split_batch(self, batch, split_size):
splits = []
for t in range(0, time_dims[0], split_size):
batch_split = []
for i, x in enumerate(batch):
if isinstance(x, torch.Tensor):
split_x = x[:, t:t + split_size]
elif isinstance(x, collections.Sequence):
split_x = [None] * len(x)
for batch_idx in range(len(x)):
split_x[batch_idx] = x[batch_idx][t:t + split_size]
batch_split.append(split_x)
splits.append(batch_split)
return splits
```
---
#### configure_apex
Overwrite to define your own Apex implementation init.
```python
def configure_apex(self, amp, model, optimizers, amp_level):
"""
Override to init AMP your own way
Must return a model and list of optimizers
:param amp:
:param model:
:param optimizers:
:param amp_level:
:return: Apex wrapped model and optimizers
"""
model, optimizers = amp.initialize(
model, optimizers, opt_level=amp_level,
)
return model, optimizers
```
---
#### configure_ddp
Overwrite to define your own DDP implementation init.
The only requirement is that:
1. On a validation batch the call goes to model.validation_step.
2. On a training batch the call goes to model.training_step.
3. On a testing batch, the call goes to model.test_step
```python
def configure_ddp(self, model, device_ids):
"""
Override to init DDP in a different way or use your own wrapper.
Must return model.
:param model:
:param device_ids:
:return: DDP wrapped model
"""
# Lightning DDP simply routes to test_step, val_step, etc...
model = LightningDistributedDataParallel(
model,
device_ids=device_ids,
find_unused_parameters=True
)
return model
```
---
#### init_ddp_connection
Override to init DDP in your own way.
```python
def init_ddp_connection(self):
"""
Connect all procs in the world using the env:// init
Use the first node as the root address
"""
# use slurm job id for the port number
# guarantees unique ports across jobs from same grid search
try:
# use the last 4 numbers in the job id as the id
default_port = os.environ['SLURM_JOB_ID']
default_port = default_port[-4:]
# all ports should be in the 10k+ range
default_port = int(default_port) + 15000
except Exception as e:
default_port = 12910
# if user gave a port number, use that one instead
try:
default_port = os.environ['MASTER_PORT']
except Exception:
os.environ['MASTER_PORT'] = str(default_port)
# figure out the root node addr
try:
root_node = os.environ['SLURM_NODELIST'].split(' ')[0]
except Exception:
root_node = '127.0.0.2'
root_node = self.trainer.resolve_root_node_address(root_node)
os.environ['MASTER_ADDR'] = root_node
dist.init_process_group('nccl', rank=self.proc_rank, world_size=self.world_size)
```
docs/Trainer/index.md
-90
View File
@@ -1,90 +0,0 @@
# Trainer
[[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py)]
The lightning trainer abstracts best practices for running a training, val, test routine. It calls parts of your model when it wants to hand over full control and otherwise makes training assumptions which are now standard practice in AI research.
This is the basic use of the trainer:
``` {.python}
from pytorch_lightning import Trainer
model = LightningTemplate()
trainer = Trainer()
trainer.fit(model)
```
But of course the fun is in all the advanced things it can do:
**Checkpointing**
- [Checkpoint callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
- [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
- [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics)
- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
**Computing cluster (SLURM)**
- [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster)
- [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)
**Debugging**
- [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
- [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
- [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
- [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
- [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
- [Print which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
**Distributed training**
- [Implement Your Own Distributed (DDP) training](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#init_ddp_connection)
- [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
- [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
- [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
- [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
- [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
**Experiment Logging**
- [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
- [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
- [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
- [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
- [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters)
- [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run)
- [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
**Training loop**
- [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
- [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
- [Early stopping callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)
- [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
- [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
- [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
- [Packed sequences](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#packed-sequences-as-inputs)
- [Truncated Backpropagation Through Time](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#truncated-backpropagtion-through-time)
**Validation loop**
- [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
- [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
- [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
- [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
- [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
**Testing loop**
- [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)
docs/__init__.py
View File
docs/examples/Examples.md
-140
View File
@@ -1,140 +0,0 @@
### Template model definition
In 99% of cases you want to just copy [one of the examples](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples) to start a new lightningModule and change the core of what your model is actually trying to do.
```bash
# get a copy of the module template
wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/pl_examples/new_project_templates/lightning_module_template.py
```
---
### Trainer Example
** \_\_main\_\_ function**
Normally, we want to let the \_\_main\_\_ function start the training.
Inside the main we parse training arguments with whatever hyperparameters we want. Your LightningModule will have a
chance to add hyperparameters.
```{.python}
from test_tube import HyperOptArgumentParser
if __name__ == '__main__':
# use default args given by lightning
root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
add_default_args(parent_parser, root_dir)
# allow model to overwrite or extend args
parser = ExampleModel.add_model_specific_args(parent_parser)
hyperparams = parser.parse_args()
# train model
main(hyperparams)
```
**Main Function**
The main function is your entry into the program. This is where you init your model, checkpoint directory, and launch the training.
The main function should have 3 arguments:
- hparams: a configuration of hyperparameters.
- slurm_manager: Slurm cluster manager object (can be None)
- dict: for you to return any values you want (useful in meta-learning, otherwise set to \_)
```python
def main(hparams, cluster, results_dict):
"""
Main training routine specific for this project
:param hparams:
:return:
"""
# build model
model = MyLightningModule(hparams)
# configure trainer
trainer = Trainer()
# train model
trainer.fit(model)
```
The __main__ function will start training on your **main** function. If you use the HyperParameterOptimizer
in hyper parameter optimization mode, this main function will get one set of hyperparameters. If you use it as a simple
argument parser you get the default arguments in the argument parser.
So, calling main(hyperparams) runs the model with the default argparse arguments.
```{.python}
main(hyperparams)
```
---
#### CPU hyperparameter search
```{.python}
# run a grid search over 20 hyperparameter combinations.
hyperparams.optimize_parallel_cpu(
main_local,
nb_trials=20,
nb_workers=1
)
```
---
#### Hyperparameter search on a single or multiple GPUs
```{.python}
# run a grid search over 20 hyperparameter combinations.
hyperparams.optimize_parallel_gpu(
main_local,
nb_trials=20,
nb_workers=1,
gpus=[0,1,2,3]
)
```
---
#### Hyperparameter search on a SLURM HPC cluster
```{.python}
def optimize_on_cluster(hyperparams):
# enable cluster training
cluster = SlurmCluster(
hyperparam_optimizer=hyperparams,
log_path=hyperparams.tt_save_path,
test_tube_exp_name=hyperparams.tt_name
)
# email for cluster coms
cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
# configure cluster
cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
cluster.job_time = '48:00:00'
cluster.gpu_type = '1080ti'
cluster.memory_mb_per_node = 48000
# any modules for code to run in env
cluster.add_command('source activate pytorch_lightning')
# name of exp
job_display_name = hyperparams.tt_name.split('_')[0]
job_display_name = job_display_name[0:3]
# run hopt
logging.info('submitting jobs...')
cluster.optimize_parallel_cluster_gpu(
main,
nb_trials=hyperparams.nb_hopt_trials,
job_name=job_display_name
)
# run cluster hyperparameter search
optimize_on_cluster(hyperparams)
```
docs/index.md
-143
View File
@@ -1,143 +0,0 @@
###### New project Quick Start
To start a new project define two files, a LightningModule and a Trainer file.
To illustrate Lightning power and simplicity, here's an example of a typical research flow.
###### Case 1: BERT
Let's say you're working on something like BERT but want to try different ways of training or even different networks.
You would define a single LightningModule and use flags to switch between your different ideas.
```python
class BERT(pl.LightningModule):
def __init__(self, model_name, task):
self.task = task
if model_name == 'transformer':
self.net = Transformer()
elif model_name == 'my_cool_version':
self.net = MyCoolVersion()
def training_step(self, batch, batch_nb):
if self.task == 'standard_bert':
# do standard bert training with self.net...
# return loss
if self.task == 'my_cool_task':
# do my own version with self.net
# return loss
```
###### Case 2: COOLER NOT BERT
But if you wanted to try something **completely** different, you'd define a new module for that.
```python
class CoolerNotBERT(pl.LightningModule):
def __init__(self):
self.net = ...
def training_step(self, batch, batch_nb):
# do some other cool task
# return loss
```
###### Rapid research flow
Then you could do rapid research by switching between these two and using the same trainer.
```python
if use_bert:
model = BERT()
else:
model = CoolerNotBERT()
trainer = Trainer(gpus=4, use_amp=True)
trainer.fit(model)
```
Notice a few things about this flow:
1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.
2. You get free GPU and 16-bit support without writing any of that code in your model.
3. You also get all of the capabilities below (without coding or testing yourself).
---
###### Templates
1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example)
2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
- [Basic CPU, GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/basic_examples)
- [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/multi_node_examples)
###### Docs shortcuts
- [LightningModule](LightningModule/RequiredTrainerInterface/)
- [Trainer](Trainer/)
###### Quick start examples
- [CPU example](examples/Examples/#cpu-hyperparameter-search)
- [Hyperparameter search on single GPU](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)
- [Hyperparameter search on multiple GPUs on same node](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)
- [Hyperparameter search on a SLURM HPC cluster](examples/Examples/#Hyperparameter search on a SLURM HPC cluster)
###### Checkpointing
- [Checkpoint callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
- [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
- [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics)
- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
###### Computing cluster (SLURM)
- [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster)
- [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)
###### Debugging
- [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
- [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
- [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
- [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
- [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
- [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
###### Distributed training
- [Implement Your Own Distributed (DDP) training](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#init_ddp_connection)
- [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
- [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
- [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
- [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
- [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
###### Experiment Logging
- [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
- [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
- [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
- [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
- [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters)
- [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run)
- [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
###### Training loop
- [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
- [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
- [Early stopping callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)
- [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
- [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
- [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
###### Validation loop
- [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
- [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
- [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
- [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
- [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
###### Testing loop
- [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)
docs/make.bat
+35
View File
@@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd
docs/requirements.txt
+9 -2
View File
@@ -1,2 +1,9 @@
mkdocs-material==4.4.0
mkdocs==1.0.4
sphinx>=1.8.3
recommonmark # fails with badges
m2r # fails with multi-line text
nbsphinx
pandoc
docutils
git+https://github.com/Borda/lightning_sphinx_theme.git
sphinxcontrib-fulltoc
sphinxcontrib-mockautodoc
docs/source/__init__.py
View File
docs/source/_static/coverage.svg → docs/source/_static/images/coverage.svg
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 901 B

After

Width:  |  Height:  |  Size: 901 B

docs/source/_static/lightning_logo.png → docs/source/_static/images/lightning_logo.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 11 KiB

After

Width:  |  Height:  |  Size: 11 KiB

docs/source/_static/lightning_logo_medium.png → docs/source/_static/images/lightning_logo_medium.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 8.3 KiB

After

Width:  |  Height:  |  Size: 8.3 KiB

docs/source/_static/lightning_logo_small.png → docs/source/_static/images/lightning_logo_small.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.6 KiB

After

Width:  |  Height:  |  Size: 2.6 KiB

docs/source/_static/overview_flat.jpg → docs/source/_static/images/overview_flat.jpg
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 410 KiB

After

Width:  |  Height:  |  Size: 410 KiB

docs/source/_static/tf_loss.png → docs/source/_static/images/tf_loss.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 219 KiB

After

Width:  |  Height:  |  Size: 219 KiB

docs/source/_static/tf_tags.png → docs/source/_static/images/tf_tags.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 214 KiB

After

Width:  |  Height:  |  Size: 214 KiB

docs/source/_templates/theme_variables.jinja
+17
View File
@@ -0,0 +1,17 @@
{%- set external_urls = {
'github': 'https://github.com/williamFalcon/pytorch-lightning',
'github_issues': 'https://github.com/williamFalcon/pytorch-lightning/issues',
'contributing': 'https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md',
'docs': 'https://williamfalcon.github.io/pytorch-lightning',
'twitter': 'https://twitter.com/PyTorchLightnin',
'discuss': 'https://discuss.pytorch.org',
'tutorials': 'https://williamfalcon.github.io/pytorch-lightning/',
'previous_pytorch_versions': 'https://williamfalcon.github.io/pytorch-lightning/',
'home': 'https://williamfalcon.github.io/pytorch-lightning/',
'get_started': 'https://williamfalcon.github.io/pytorch-lightning/',
'features': 'https://williamfalcon.github.io/pytorch-lightning/',
'blog': 'https://williamfalcon.github.io/pytorch-lightning/',
'resources': 'https://williamfalcon.github.io/pytorch-lightning/',
'support': 'https://williamfalcon.github.io/pytorch-lightning/',
}
-%}
docs/source/conf.py
+354
View File
@@ -0,0 +1,354 @@
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
import os
import sys
import glob
import shutil
import inspect
# import m2r
import builtins
import pt_lightning_sphinx_theme
PATH_HERE = os.path.abspath(os.path.dirname(__file__))
PATH_ROOT = os.path.join(PATH_HERE, '..', '..')
sys.path.insert(0, os.path.abspath(PATH_ROOT))
builtins.__LIGHTNING_SETUP__ = True
import pytorch_lightning # noqa: E402
# -- Project documents -------------------------------------------------------
# export the documentation
# with open('intro.rst', 'w') as fp:
# intro = pytorch_lightning.__doc__.replace(os.linesep + ' ', '')
# fp.write(m2r.convert(intro))
# # fp.write(pytorch_lightning.__doc__)
# export the READme
# with open(os.path.join(PATH_ROOT, 'README.md'), 'r') as fp:
# readme = fp.read()
# # replace all paths to relative
# for ndir in (os.path.basename(p) for p in glob.glob(os.path.join(PATH_ROOT, '*'))
# if os.path.isdir(p)):
# readme = readme.replace('](%s/' % ndir, '](%s/%s/' % (PATH_ROOT, ndir))
# with open('readme.md', 'w') as fp:
# fp.write(readme)
# -- Project information -----------------------------------------------------
project = 'PyTorch-Lightning'
copyright = pytorch_lightning.__copyright__
author = pytorch_lightning.__author__
# The short X.Y version
version = pytorch_lightning.__version__
# The full version, including alpha/beta/rc tags
release = pytorch_lightning.__version__
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
needs_sphinx = '1.4'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinxcontrib.mockautodoc',
# 'sphinxcontrib.fulltoc', # breaks pytorch-theme with unexpected kw argument 'titles_only'
'sphinx.ext.doctest',
'sphinx.ext.intersphinx',
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.linkcode',
'sphinx.ext.autosummary',
'sphinx.ext.napoleon',
'recommonmark',
# 'm2r',
'nbsphinx',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# https://berkeley-stat159-f17.github.io/stat159-f17/lectures/14-sphinx..html#conf.py-(cont.)
# https://stackoverflow.com/questions/38526888/embed-ipython-notebook-in-sphinx-document
# I execute the notebooks manually in advance. If notebooks test the code,
# they should be run at build time.
nbsphinx_execute = 'never'
nbsphinx_allow_errors = True
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
# source_suffix = ['.rst', '.md', '.ipynb']
source_suffix = {
'.rst': 'restructuredtext',
'.txt': 'markdown',
'.md': 'markdown',
'.ipynb': 'nbsphinx',
}
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['*.test_*']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
# http://www.sphinx-doc.org/en/master/usage/theming.html#builtin-themes
# html_theme = 'bizstyle'
# https://sphinx-themes.org
html_theme = 'pt_lightning_sphinx_theme'
html_theme_path = [pt_lightning_sphinx_theme.get_html_theme_path()]
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {
'pytorch_project': pytorch_lightning.__homepage__,
'canonical_url': pytorch_lightning.__homepage__,
'collapse_navigation': False,
'display_version': True,
'logo_only': False,
}
html_logo = '_static/images/lightning_logo_small.png'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = project + '-doc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
# 'preamble': '',
# Latex figure (float) alignment
'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, project + '.tex', project + ' Documentation', author, 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, project, project + ' Documentation', [author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, project, project + ' Documentation', author, project,
'One line description of project.', 'Miscellaneous'),
]
# -- Options for Epub output -------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = project
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''
# A unique identification for the text.
#
# epub_uid = ''
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
# -- Extension configuration -------------------------------------------------
# -- Options for intersphinx extension ---------------------------------------
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/': None}
# -- Options for todo extension ----------------------------------------------
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = True
# https://github.com/rtfd/readthedocs.org/issues/1139
# I use sphinx-apidoc to auto-generate API documentation for my project.
# Right now I have to commit these auto-generated files to my repository
# so that RTD can build them into HTML docs. It'd be cool if RTD could run
# sphinx-apidoc for me, since it's easy to forget to regen API docs
# and commit them to my repo after making changes to my code.
PACKAGES = [
pytorch_lightning.__name__,
'pl_examples',
]
def run_apidoc(_):
for pkg in PACKAGES:
argv = ['-e', '-o', PATH_HERE, os.path.join(PATH_HERE, PATH_ROOT, pkg),
'**/test_*', '--force', '--private', '--module-first']
try:
# Sphinx 1.7+
from sphinx.ext import apidoc
apidoc.main(argv)
except ImportError:
# Sphinx 1.6 (and earlier)
from sphinx import apidoc
argv.insert(0, apidoc.__file__)
apidoc.main(argv)
def setup(app):
app.connect('builder-inited', run_apidoc)
# copy all notebooks to local folder
path_nbs = os.path.join(PATH_HERE, 'notebooks')
if not os.path.isdir(path_nbs):
os.mkdir(path_nbs)
for path_ipynb in glob.glob(os.path.join(PATH_ROOT, 'notebooks', '*.ipynb')):
path_ipynb2 = os.path.join(path_nbs, os.path.basename(path_ipynb))
shutil.copy(path_ipynb, path_ipynb2)
# Ignoring Third-party packages
# https://stackoverflow.com/questions/15889621/sphinx-how-to-exclude-imports-in-automodule
MOCK_REQUIRE_PACKAGES = []
with open(os.path.join(PATH_ROOT, 'requirements.txt'), 'r') as fp:
for ln in fp.readlines():
found = [ln.index(ch) for ch in list(',=<>#') if ch in ln]
pkg = ln[:min(found)] if found else ln
if pkg.rstrip():
MOCK_REQUIRE_PACKAGES.append(pkg.rstrip())
# TODO: better parse from package since the import name and package name may differ
MOCK_MANUAL_PACKAGES = ['torch', 'torchvision', 'sklearn', 'test_tube', 'mlflow', 'comet_ml']
autodoc_mock_imports = MOCK_REQUIRE_PACKAGES + MOCK_MANUAL_PACKAGES
# for mod_name in MOCK_REQUIRE_PACKAGES:
# sys.modules[mod_name] = mock.Mock()
# Options for the linkcode extension
# ----------------------------------
github_user = 'williamFalcon'
github_repo = project
# Resolve function
# This function is used to populate the (source) links in the API
def linkcode_resolve(domain, info):
def find_source():
# try to find the file and line number, based on code from numpy:
# https://github.com/numpy/numpy/blob/master/doc/source/conf.py#L286
obj = sys.modules[info['module']]
for part in info['fullname'].split('.'):
obj = getattr(obj, part)
fname = inspect.getsourcefile(obj)
# https://github.com/rtfd/readthedocs.org/issues/5735
if any([s in fname for s in ('readthedocs', 'checkouts')]):
# /home/docs/checkouts/readthedocs.org/user_builds/pytorch_lightning/checkouts/
# devel/pytorch_lightning/utilities/cls_experiment.py#L26-L176
path_top = os.path.abspath(os.path.join('..', '..', '..'))
fname = os.path.relpath(fname, start=path_top)
else:
# Local build, imitate master
fname = 'master/' + os.path.relpath(fname, start=os.path.abspath('..'))
source, lineno = inspect.getsourcelines(obj)
return fname, lineno, lineno + len(source) - 1
if domain != 'py' or not info['module']:
return None
try:
filename = '%s#L%d-L%d' % find_source()
except Exception:
filename = info['module'].replace('.', '/') + '.py'
# import subprocess
# tag = subprocess.Popen(['git', 'rev-parse', 'HEAD'], stdout=subprocess.PIPE,
# universal_newlines=True).communicate()[0][:-1]
return "https://github.com/%s/%s/blob/%s" \
% (github_user, github_repo, filename)
autodoc_member_order = 'groupwise'
autoclass_content = 'both'
autodoc_default_flags = [
'members', 'undoc-members', 'show-inheritance', 'private-members',
# 'special-members', 'inherited-members'
]
docs/source/index.rst
+26
View File
@@ -0,0 +1,26 @@
.. PyTorch-Lightning documentation master file, created by
sphinx-quickstart on Fri Nov 15 07:48:22 2019.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to PyTorch-Lightning!
=============================
Table of content
----------------
.. toctree::
:maxdepth: 1
intro
pytorch_lightning
pl_examples
Indices and tables
------------------
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
docs/source/intro.md
+60
View File
@@ -0,0 +1,60 @@
## New project Quick Start
To start a new project define two files, a LightningModule and a Trainer file.
To illustrate Lightning power and simplicity, here's an example of a typical research flow.
### Case 1: BERT
Let's say you're working on something like BERT but want to try different ways of training or even different networks.
You would define a single LightningModule and use flags to switch between your different ideas.
```python
class BERT(pl.LightningModule):
def __init__(self, model_name, task):
self.task = task
if model_name == 'transformer':
self.net = Transformer()
elif model_name == 'my_cool_version':
self.net = MyCoolVersion()
def training_step(self, batch, batch_nb):
if self.task == 'standard_bert':
# do standard bert training with self.net...
# return loss
if self.task == 'my_cool_task':
# do my own version with self.net
# return loss
```
### Case 2: COOLER NOT BERT
But if you wanted to try something **completely** different, you'd define a new module for that.
```python
class CoolerNotBERT(pl.LightningModule):
def __init__(self):
self.net = ...
def training_step(self, batch, batch_nb):
# do some other cool task
# return loss
```
### Rapid research flow
Then you could do rapid research by switching between these two and using the same trainer.
```python
if use_bert:
model = BERT()
else:
model = CoolerNotBERT()
trainer = Trainer(gpus=4, use_amp=True)
trainer.fit(model)
```
Notice a few things about this flow:
1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.
2. You get free GPU and 16-bit support without writing any of that code in your model.
3. You also get all of the capabilities below (without coding or testing yourself).
mkdocs.yml
-16
View File
@@ -1,16 +0,0 @@
site_name: PyTorch lightning Documentation
theme:
name: 'material'
docs_dir: docs
repo_name: 'williamFalcon/pytorch-lightning'
repo_url: https://github.com/williamFalcon/pytorch-lightning
site_dir: 'site'
site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
dev_addr: '0.0.0.0:8000'
#google_analytics: ['UA-aasd', 'sitename']
markdown_extensions:
- codehilite:
guess_lang: false
linenums: true
pl_examples/__init__.py
+141
View File
@@ -1,3 +1,144 @@
"""
Template model definition
-------------------------
In 99% of cases you want to just copy `one of the examples
<https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples>`_
to start a new lightningModule and change the core of what your model is actually trying to do.
.. code-block:: bash
# get a copy of the module template
wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/pl_examples/new_project_templates/lightning_module_template.py # noqa: E501
Trainer Example
---------------
**`__main__` function**
Normally, we want to let the `__main__` function start the training.
Inside the main we parse training arguments with whatever hyperparameters we want.
Your LightningModule will have a chance to add hyperparameters.
.. code-block:: python
from test_tube import HyperOptArgumentParser
if __name__ == '__main__':
# use default args given by lightning
root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
add_default_args(parent_parser, root_dir)
# allow model to overwrite or extend args
parser = ExampleModel.add_model_specific_args(parent_parser)
hyperparams = parser.parse_args()
# train model
main(hyperparams)
**Main Function**
The main function is your entry into the program. This is where you init your model, checkpoint directory,
and launch the training. The main function should have 3 arguments:
- hparams: a configuration of hyperparameters.
- slurm_manager: Slurm cluster manager object (can be None)
- dict: for you to return any values you want (useful in meta-learning, otherwise set to)
.. code-block:: python
def main(hparams, cluster, results_dict):
# build model
model = MyLightningModule(hparams)
# configure trainer
trainer = Trainer()
# train model
trainer.fit(model)
The `__main__` function will start training on your **main** function.
If you use the HyperParameterOptimizer in hyper parameter optimization mode,
this main function will get one set of hyperparameters. If you use it as a simple
argument parser you get the default arguments in the argument parser.
So, calling main(hyperparams) runs the model with the default argparse arguments.::
main(hyperparams)
CPU hyperparameter search
-------------------------
.. code-block:: python
# run a grid search over 20 hyperparameter combinations.
hyperparams.optimize_parallel_cpu(
main_local,
nb_trials=20,
nb_workers=1
)
Hyperparameter search on a single or multiple GPUs
--------------------------------------------------
.. code-block:: python
# run a grid search over 20 hyperparameter combinations.
hyperparams.optimize_parallel_gpu(
main_local,
nb_trials=20,
nb_workers=1,
gpus=[0,1,2,3]
)
Hyperparameter search on a SLURM HPC cluster
--------------------------------------------
.. code-block:: python
def optimize_on_cluster(hyperparams):
# enable cluster training
cluster = SlurmCluster(
hyperparam_optimizer=hyperparams,
log_path=hyperparams.tt_save_path,
test_tube_exp_name=hyperparams.tt_name
)
# email for cluster coms
cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
# configure cluster
cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
cluster.job_time = '48:00:00'
cluster.gpu_type = '1080ti'
cluster.memory_mb_per_node = 48000
# any modules for code to run in env
cluster.add_command('source activate pytorch_lightning')
# name of exp
job_display_name = hyperparams.tt_name.split('_')[0]
job_display_name = job_display_name[0:3]
# run hopt
logging.info('submitting jobs...')
cluster.optimize_parallel_cluster_gpu(
main,
nb_trials=hyperparams.nb_hopt_trials,
job_name=job_display_name
)
# run cluster hyperparameter search
optimize_on_cluster(hyperparams)
"""
from .basic_examples.lightning_module_template import LightningTemplateModel
__all__ = [
pytorch_lightning/__init__.py
+2 -1
View File
@@ -1,9 +1,10 @@
"""Package info"""
__version__ = '0.5.3.2'
__author__ = ' William Falcon et al.'
__author__ = 'William Falcon et al.'
__author_email__ = 'waf2107@columbia.edu'
__license__ = 'Apache-2.0'
__copyright__ = 'Copyright (c) 2018-2019, %s.' % __author__
__homepage__ = 'https://github.com/williamFalcon/pytorch-lightning'
# this has to be simple string, see: https://github.com/pypa/twine/issues/522
__docs__ = "PyTorch Lightning is the lightweight PyTorch wrapper for ML researchers." \
pytorch_lightning/callbacks/pt_callbacks.py
+27 -19
View File
@@ -9,24 +9,25 @@ from pytorch_lightning.overrides.data_parallel import LightningDistributedDataPa
class Callback(object):
"""Abstract base class used to build new callbacks.
# Properties
params: dict. Training parameters
* params: dict. Training parameters
(eg. verbosity, batch size, number of epochs...).
Reference of the model being trained.
The `logs` dictionary that callback methods
take as argument will contain keys for quantities relevant to
the current batch or epoch.
Currently, the `.fit()` method of the `Sequential` model class
will include the following quantities in the `logs` that
it passes to its callbacks:
on_epoch_end: logs include `acc` and `loss`, and
The `logs` dictionary that callback methods take as argument will contain keys
for quantities relevant to the current batch or epoch.
Currently, the `.fit()` method of the `Sequential` model class will include the following
quantities in the `logs` that it passes to its callbacks:
* on_epoch_end: logs include `acc` and `loss`, and
optionally include `val_loss`
(if validation is enabled in `fit`), and `val_acc`
(if validation and accuracy monitoring are enabled).
on_batch_begin: logs include `size`,
* on_batch_begin: logs include `size`,
the number of samples in the current batch.
on_batch_end: logs include `loss`, and optionally `acc`
* on_batch_end: logs include `loss`, and optionally `acc`
(if accuracy monitoring is enabled).
"""
def __init__(self):
@@ -62,6 +63,7 @@ class Callback(object):
class EarlyStopping(Callback):
"""Stop training when a monitored quantity has stopped improving.
# Arguments
monitor: quantity to be monitored.
min_delta: minimum change in the monitored quantity
@@ -78,6 +80,7 @@ class EarlyStopping(Callback):
monitored has stopped increasing; in `auto`
mode, the direction is automatically inferred
from the name of the monitored quantity.
"""
def __init__(self, monitor='val_loss',
@@ -148,12 +151,14 @@ class EarlyStopping(Callback):
class ModelCheckpoint(Callback):
"""Save the model after every epoch.
`filepath` can contain named formatting options,
The `filepath` can contain named formatting options,
which will be filled the value of `epoch` and
keys in `logs` (passed in `on_epoch_end`).
For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`,
then the model checkpoints will be saved with the epoch number and
the validation loss in the filename.
# Arguments
filepath: string, path to save the model file.
monitor: quantity to monitor.
@@ -179,6 +184,7 @@ class ModelCheckpoint(Callback):
saved (`model.save_weights(filepath)`), else the full model
is saved (`model.save(filepath)`).
period: Interval (number of epochs) between checkpoints.
"""
def __init__(self, filepath, monitor='val_loss', verbose=0,
@@ -325,8 +331,10 @@ class ModelCheckpoint(Callback):
class GradientAccumulationScheduler(Callback):
"""Change gradient accumulation factor according to scheduling.
# Arguments
scheduling: dict, scheduling in format {epoch: accumulation_factor}
"""
def __init__(self, scheduling: dict):
@@ -355,11 +363,11 @@ class GradientAccumulationScheduler(Callback):
break
if __name__ == '__main__':
c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
for i, loss in enumerate(losses):
should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
logging.info(loss)
if should_stop:
break
# if __name__ == '__main__':
# c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
# losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
# for i, loss in enumerate(losses):
# should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
# logging.info(loss)
# if should_stop:
# break
pytorch_lightning/core/__init__.py
+150
View File
@@ -0,0 +1,150 @@
"""
Lightning Module interface
==========================
A lightning module is a strict superclass of nn.Module, it provides a standard interface
for the trainer to interact with the model.
The easiest thing to do is copy the minimal example below and modify accordingly.
Otherwise, to Define a Lightning Module, implement the following methods:
Minimal example
---------------
.. code-block:: python
import os
import torch
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST
import torchvision.transforms as transforms
import pytorch_lightning as pl
class CoolModel(pl.LightningModule):
def __init__(self):
super(CoolModel, self).__init__()
# not the best model...
self.l1 = torch.nn.Linear(28 * 28, 10)
def forward(self, x):
return torch.relu(self.l1(x.view(x.size(0), -1)))
def training_step(self, batch, batch_nb):
# REQUIRED
x, y = batch
y_hat = self.forward(x)
return {'loss': F.cross_entropy(y_hat, y)}
def validation_step(self, batch, batch_nb):
# OPTIONAL
x, y = batch
y_hat = self.forward(x)
return {'val_loss': F.cross_entropy(y_hat, y)}
def validation_end(self, outputs):
# OPTIONAL
avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
return {'avg_val_loss': avg_loss}
def test_step(self, batch, batch_nb):
# OPTIONAL
x, y = batch
y_hat = self.forward(x)
return {'test_loss': F.cross_entropy(y_hat, y)}
def test_end(self, outputs):
# OPTIONAL
avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
return {'avg_test_loss': avg_loss}
def configure_optimizers(self):
# REQUIRED
return torch.optim.Adam(self.parameters(), lr=0.02)
@pl.data_loader
def train_dataloader(self):
return DataLoader(MNIST(os.getcwd(), train=True, download=True,
transform=transforms.ToTensor()), batch_size=32)
@pl.data_loader
def val_dataloader(self):
# OPTIONAL
# can also return a list of val dataloaders
return DataLoader(MNIST(os.getcwd(), train=True, download=True,
transform=transforms.ToTensor()), batch_size=32)
@pl.data_loader
def test_dataloader(self):
# OPTIONAL
# can also return a list of test dataloaders
return DataLoader(MNIST(os.getcwd(), train=False, download=True,
transform=transforms.ToTensor()), batch_size=32)
How do these methods fit into the broader training?
---------------------------------------------------
The LightningModule interface is on the right. Each method corresponds
to a part of a research project. Lightning automates everything not in blue.
.. figure:: docs/source/_static/images/overview_flat.jpg
:align: center
Overview.
Optional Methods
----------------
**add_model_specific_args**
.. code-block:: python
@staticmethod
def add_model_specific_args(parent_parser, root_dir)
Lightning has a list of default argparse commands.
This method is your chance to add or modify commands specific to your model.
The `hyperparameter argument parser
<https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser>`_
is available anywhere in your model by calling self.hparams.
**Return**
An argument parser
**Example**
.. code-block:: python
@staticmethod
def add_model_specific_args(parent_parser, root_dir):
parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
# param overwrites
# parser.set_defaults(gradient_clip_val=5.0)
# network params
parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
parser.add_argument('--in_features', default=28*28)
parser.add_argument('--out_features', default=10)
# use 500 for CPU, 50000 for GPU to see speed difference
parser.add_argument('--hidden_dim', default=50000)
# data
parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
# training params (opt)
parser.opt_list('--learning_rate', default=0.001, type=float,
options=[0.0001, 0.0005, 0.001, 0.005], tunable=False)
parser.opt_list('--batch_size', default=256, type=int,
options=[32, 64, 128, 256], tunable=False)
parser.opt_list('--optimizer_name', default='adam', type=str,
options=['adam'], tunable=False)
return parser
"""
pytorch_lightning/core/hooks.py
+72 -7
View File
@@ -1,3 +1,17 @@
"""
# Hooks
There are cases when you might want to do something different at different parts of the training/validation loop.
To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
**Contributing** If there's a hook you'd like to add, simply:
1. Fork PyTorchLightning.
2. Add the hook :py:mod:`pytorch_lightning.base_module.hooks.py`.
3. Add the correct place in the :py:mod:`pytorch_lightning.models.trainer` where it should be called.
"""
import torch
@@ -19,28 +33,47 @@ class ModelHooks(torch.nn.Module):
pass
def on_batch_start(self, batch):
"""Called in the training loop before anything happens for that batch.
:param batch:
:return:
"""
# do something when the batch starts
pass
def on_batch_end(self):
"""Called in the training loop after the batch."""
# do something when the batch ends
pass
def on_epoch_start(self):
"""Called in the training loop at the very beginning of the epoch."""
# do something when the epoch starts
pass
def on_epoch_end(self):
"""Called in the training loop at the very end of the epoch."""
# do something when the epoch ends
pass
def on_pre_performance_check(self):
"""Called at the very beginning of the validation loop."""
# do something before validation starts
pass
def on_post_performance_check(self):
"""Called at the very end of the validation loop."""
# do something before validation end
pass
def on_before_zero_grad(self, optimizer):
"""
Called after optimizer.step() and before optimizer.zero_grad()
"""Called after optimizer.step() and before optimizer.zero_grad()
Called in the training loop after taking an optimizer step and before zeroing grads.
Good place to inspect weight information with weights updated.
for optimizer in optimizers::
for optimizer in optimizers:
optimizer.step()
model.on_before_zero_grad(optimizer) # < ---- called here
optimizer.zero_grad
@@ -48,22 +81,54 @@ class ModelHooks(torch.nn.Module):
:param optimizer:
:return:
"""
# do something with the optimizer or inspect it.
pass
def on_after_backward(self):
"""
Called after loss.backward() and before optimizers do anything
"""Called after loss.backward() and before optimizers do anything.
:return:
Called in the training loop after model.backward()
This is the ideal place to inspect or log gradient information
.. code-block:: python
def on_after_backward(self):
# example to inspect gradient information in tensorboard
if self.trainer.global_step % 25 == 0: # don't make the tf file huge
params = self.state_dict()
for k, v in params.items():
grads = v
name = k
self.logger.experiment.add_histogram(tag=name, values=grads,
global_step=self.trainer.global_step)
"""
pass
def backward(self, use_amp, loss, optimizer):
"""
Override backward with your own implementation if you need to
"""Override backward with your own implementation if you need to
:param use_amp: Whether amp was requested or not
:param loss: Loss is already scaled by accumulated grads
:param optimizer: Current optimizer being used
:return:
Called to perform backward step.
Feel free to override as needed.
The loss passed in has already been scaled for accumulated gradients if requested.
.. code-block:: python
def backward(self, use_amp, loss, optimizer):
if use_amp:
with amp.scale_loss(loss, optimizer) as scaled_loss:
scaled_loss.backward()
else:
loss.backward()
"""
if use_amp:
with amp.scale_loss(loss, optimizer) as scaled_loss:
pytorch_lightning/core/lightning.py
+822 -61
View File
File diff suppressed because it is too large Load Diff
pytorch_lightning/logging/__init__.py
+169
View File
@@ -1,3 +1,170 @@
"""
Lighting offers options for logging information about model, gpu usage, etc,
via several different logging frameworks. It also offers printing options for training monitoring.
**default_save_path**
Lightning sets a default TestTubeLogger and CheckpointCallback for you which log to
`os.getcwd()` by default. To modify the logging path you can set::
Trainer(default_save_path='/your/path/to/save/checkpoints')
If you need more custom behavior (different paths for both, different metrics, etc...)
from the logger and the checkpointCallback, pass in your own instances as explained below.
Setting up logging
------------------
The trainer inits a default logger for you (TestTubeLogger). All logs will
go to the current working directory under a folder named `os.getcwd()/lightning_logs`.
If you want to modify the default logging behavior even more, pass in a logger
(which should inherit from `LightningBaseLogger`).
.. code-block:: python
my_logger = MyLightningLogger(...)
trainer = Trainer(logger=my_logger)
The path in this logger will overwrite `default_save_path`.
Lightning supports several common experiment tracking frameworks out of the box
Custom logger
-------------
You can implement your own logger by writing a class that inherits from
`LightningLoggerBase`. Use the `rank_zero_only` decorator to make sure that
only the first process in DDP training logs data.
.. code-block:: python
from pytorch_lightning.logging import LightningLoggerBase, rank_zero_only
class MyLogger(LightningLoggerBase):
@rank_zero_only
def log_hyperparams(self, params):
# params is an argparse.Namespace
# your code to record hyperparameters goes here
pass
@rank_zero_only
def log_metrics(self, metrics, step_num):
# metrics is a dictionary of metric names and values
# your code to record metrics goes here
pass
def save(self):
# Optional. Any code necessary to save logger data goes here
pass
@rank_zero_only
def finalize(self, status):
# Optional. Any code that needs to be run after training
# finishes goes here
If you write a logger than may be useful to others, please send
a pull request to add it to Lighting!
Using loggers
-------------
You can call the logger anywhere from your LightningModule by doing:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.whatever_method_summary_writer_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.add_histogram(...)
Display metrics in progress bar
-------------------------------
.. code-block:: python
# DEFAULT
trainer = Trainer(show_progress_bar=True)
Log metric row every k batches
------------------------------
Every k batches lightning will make an entry in the metrics log
.. code-block:: python
# DEFAULT (ie: save a .csv log file every 10 batches)
trainer = Trainer(row_log_interval=10)
Log GPU memory
--------------
Logs GPU memory when metrics are logged.
.. code-block:: python
# DEFAULT
trainer = Trainer(log_gpu_memory=None)
# log only the min/max utilization
trainer = Trainer(log_gpu_memory='min_max')
# log all the GPU memory (if on DDP, logs only that node)
trainer = Trainer(log_gpu_memory='all')
Process position
----------------
When running multiple models on the same machine we want to decide which progress bar to use.
Lightning will stack progress bars according to this value.
.. code-block:: python
# DEFAULT
trainer = Trainer(process_position=0)
# if this is the second model on the node, show the second progress bar below
trainer = Trainer(process_position=1)
Save a snapshot of all hyperparameters
--------------------------------------
Automatically log hyperparameters stored in the `hparams` attribute as an `argparse.Namespace`
.. code-block:: python
class MyModel(pl.Lightning):
def __init__(self, hparams):
self.hparams = hparams
...
args = parser.parse_args()
model = MyModel(args)
logger = TestTubeLogger(...)
t = Trainer(logger=logger)
trainer.fit(model)
Write logs file to csv every k batches
--------------------------------------
Every k batches, lightning will write the new logs to disk
.. code-block:: python
# DEFAULT (ie: save a .csv log file every 100 batches)
trainer = Trainer(log_save_interval=100)
"""
from os import environ
from .base import LightningLoggerBase, rank_zero_only
@@ -5,10 +172,12 @@ try:
from .test_tube import TestTubeLogger
except ImportError:
pass
try:
from .mlflow import MLFlowLogger
except ImportError:
pass
try:
# needed to prevent ImportError and duplicated logs.
environ["COMET_DISABLE_AUTO_LOGGING"] = "1"
pytorch_lightning/logging/comet.py
+49
View File
@@ -1,3 +1,52 @@
"""
Log using `comet <https://www.comet.ml>`_
Comet logger can be used in either online or offline mode.
To log in online mode, CometLogger requries an API key:
.. code-block:: python
from pytorch_lightning.logging import CometLogger
# arguments made to CometLogger are passed on to the comet_ml.Experiment class
comet_logger = CometLogger(
api_key=os.environ["COMET_KEY"],
workspace=os.environ["COMET_WORKSPACE"], # Optional
project_name="default_project", # Optional
rest_api_key=os.environ["COMET_REST_KEY"], # Optional
experiment_name="default" # Optional
)
trainer = Trainer(logger=comet_logger)
To log in offline mode, CometLogger requires a path to a local directory:
.. code-block:: python
from pytorch_lightning.logging import CometLogger
# arguments made to CometLogger are passed on to the comet_ml.Experiment class
comet_logger = CometLogger(
save_dir=".",
workspace=os.environ["COMET_WORKSPACE"], # Optional
project_name="default_project", # Optional
rest_api_key=os.environ["COMET_REST_KEY"], # Optional
experiment_name="default" # Optional
)
trainer = Trainer(logger=comet_logger)
Use the logger anywhere in you LightningModule as follows:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.whatever_comet_ml_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.whatever_comet_ml_supports(...)
"""
from logging import getLogger
try:
pytorch_lightning/logging/mlflow.py
+26
View File
@@ -1,3 +1,29 @@
"""
Log using `mlflow <https://mlflow.org>'_
.. code-block:: python
from pytorch_lightning.logging import MLFlowLogger
mlf_logger = MLFlowLogger(
experiment_name="default",
tracking_uri="file:/."
)
trainer = Trainer(logger=mlf_logger)
Use the logger anywhere in you LightningModule as follows:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.whatever_ml_flow_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.whatever_ml_flow_supports(...)
"""
from logging import getLogger
from time import time
pytorch_lightning/logging/test_tube.py
+31
View File
@@ -1,3 +1,34 @@
"""
Log using `test tube <https://williamfalcon.github.io/test-tube>'_. Test tube logger is
a strict subclass of `PyTorch SummaryWriter <https://pytorch.org/docs/stable/tensorboard.html>`_, refer to their
documentation for all supported operations. The TestTubeLogger adds a nicer folder structure
to manage experiments and snapshots all hyperparameters you pass to a LightningModule.
.. code-block:: python
from pytorch_lightning.logging import TestTubeLogger
tt_logger = TestTubeLogger(
save_dir=".",
name="default",
debug=False,
create_git_tag=False
)
trainer = Trainer(logger=tt_logger)
Use the logger anywhere in you LightningModule as follows:
.. code-block:: python
def train_step(...):
# example
self.logger.experiment.whatever_method_summary_writer_supports(...)
def any_lightning_module_function_or_hook(...):
self.logger.experiment.add_histogram(...)
"""
try:
from test_tube import Experiment
except ImportError:
pytorch_lightning/trainer/__init__.py
+19
View File
@@ -0,0 +1,19 @@
"""
# Trainer
The lightning trainer abstracts best practices for running a training, val, test routine.
It calls parts of your model when it wants to hand over full control and otherwise makes
training assumptions which are now standard practice in AI research.
This is the basic use of the trainer:
.. code-block:: python
from pytorch_lightning import Trainer
model = LightningTemplate()
trainer = Trainer()
trainer.fit(model)
"""
pytorch_lightning/trainer/ddp_mixin.py
+115 -1
View File
@@ -1,6 +1,120 @@
"""
Lightning supports model training on a cluster managed by SLURM in the following cases:
1. Training on a single cpu or single GPU.
2. Train on multiple GPUs on the same node using DataParallel or DistributedDataParallel
3. Training across multiple GPUs on multiple different nodes via DistributedDataParallel.
.. note:: A node means a machine with multiple GPUs
Running grid search on a cluster
--------------------------------
To use lightning to run a hyperparameter search (grid-search or random-search) on a cluster do 4 things:
(1). Define the parameters for the grid search
.. code-block:: python
from test_tube import HyperOptArgumentParser
# subclass of argparse
parser = HyperOptArgumentParser(strategy='random_search')
parser.add_argument('--learning_rate', default=0.002, type=float, help='the learning rate')
# let's enable optimizing over the number of layers in the network
parser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4, 8])
hparams = parser.parse_args()
.. note:: You must set `Tunable=True` for that argument to be considered in the permutation set.
Otherwise test-tube will use the default value. This flag is useful when you don't want
to search over an argument and want to use the default instead.
(2). Define the cluster options in the
`SlurmCluster object <https://williamfalcon.github.io/test-tube/hpc/SlurmCluster>`_ (over 5 nodes and 8 gpus)
.. code-block:: python
from test_tube.hpc import SlurmCluster
# hyperparameters is a test-tube hyper params object
# see https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/
hyperparams = args.parse()
# init cluster
cluster = SlurmCluster(
hyperparam_optimizer=hyperparams,
log_path='/path/to/log/results/to',
python_cmd='python3'
)
# let the cluster know where to email for a change in job status (ie: complete, fail, etc...)
cluster.notify_job_status(email='some@email.com', on_done=True, on_fail=True)
# set the job options. In this instance, we'll run 20 different models
# each with its own set of hyperparameters giving each one 1 GPU (ie: taking up 20 GPUs)
cluster.per_experiment_nb_gpus = 8
cluster.per_experiment_nb_nodes = 5
# we'll request 10GB of memory per node
cluster.memory_mb_per_node = 10000
# set a walltime of 10 minues
cluster.job_time = '10:00'
(3). Make a main function with your model and trainer. Each job will call this function with a particular
hparams configuration.::
from pytorch_lightning import Trainer
def train_fx(trial_hparams, cluster_manager, _):
# hparams has a specific set of hyperparams
my_model = MyLightningModel()
# give the trainer the cluster object
trainer = Trainer()
trainer.fit(my_model)
`
(4). Start the grid/random search::
# run the models on the cluster
cluster.optimize_parallel_cluster_gpu(
train_fx,
nb_trials=20,
job_name='my_grid_search_exp_name',
job_display_name='my_exp')
.. note:: `nb_trials` specifies how many of the possible permutations to use. If using `grid_search` it will use
the depth first ordering. If using `random_search` it will use the first k shuffled options. FYI, random search
has been shown to be just as good as any Bayesian optimization method when using a reasonable number of samples (60),
see this `paper <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`_ for more information.
Walltime auto-resubmit
----------------------
Lightning automatically resubmits jobs when they reach the walltime. Make sure to set the SIGUSR1 signal in
your SLURM script.::
# 90 seconds before training ends
#SBATCH --signal=SIGUSR1@90
When lightning receives the SIGUSR1 signal it will:
1. save a checkpoint with 'hpc_ckpt' in the name.
2. resubmit the job using the SLURM_JOB_ID
When the script starts again, Lightning will:
1. search for a 'hpc_ckpt' checkpoint.
2. restore the model, optimizers, schedulers, epoch, etc...
"""
import os
import re
import warnings
import logging
import torch
pytorch_lightning/trainer/dp_mixin.py
+304
View File
@@ -1,3 +1,307 @@
"""
Lightning makes multi-gpu training and 16 bit training trivial.
.. note:: None of the flags below require changing anything about your lightningModel definition.
Choosing a backend
==================
Lightning supports two backends. DataParallel and DistributedDataParallel.
Both can be used for single-node multi-GPU training.
For multi-node training you must use DistributedDataParallel.
DataParallel (dp)
-----------------
Splits a batch across multiple GPUs on the same node. Cannot be used for multi-node training.
DistributedDataParallel (ddp)
-----------------------------
Trains a copy of the model on each GPU and only syncs gradients. If used with DistributedSampler, each GPU trains
on a subset of the full dataset.
DistributedDataParallel-2 (ddp2)
--------------------------------
Works like DDP, except each node trains a single copy of the model using ALL GPUs on that node.
Very useful when dealing with negative samples, etc...
You can toggle between each mode by setting this flag.
.. code-block:: python
# DEFAULT (when using single GPU or no GPUs)
trainer = Trainer(distributed_backend=None)
# Change to DataParallel (gpus > 1)
trainer = Trainer(distributed_backend='dp')
# change to distributed data parallel (gpus > 1)
trainer = Trainer(distributed_backend='ddp')
# change to distributed data parallel (gpus > 1)
trainer = Trainer(distributed_backend='ddp2')
If you request multiple nodes, the back-end will auto-switch to ddp.
We recommend you use DistributedDataparallel even for single-node multi-GPU training.
It is MUCH faster than DP but *may* have configuration issues depending on your cluster.
For a deeper understanding of what lightning is doing, feel free to read this
`guide <https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565>`_.
Distributed and 16-bit precision
--------------------------------
Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.
Below are the possible configurations we support.
+-------+---------+----+-----+---------+------------------------------------------------------------+
| 1 GPU | 1+ GPUs | DP | DDP | 16-bit | command |
+=======+=========+====+=====+=========+============================================================+
| Y | | | | | `Trainer(gpus=1)` |
+-------+---------+----+-----+---------+------------------------------------------------------------+
| Y | | | | Y | `Trainer(gpus=1, use_amp=True)` |
+-------+---------+----+-----+---------+------------------------------------------------------------+
| | Y | Y | | | `Trainer(gpus=k, distributed_backend='dp')` |
+-------+---------+----+-----+---------+------------------------------------------------------------+
| | Y | | Y | | `Trainer(gpus=k, distributed_backend='ddp')` |
+-------+---------+----+-----+---------+------------------------------------------------------------+
| | Y | | Y | Y | `Trainer(gpus=k, distributed_backend='ddp', use_amp=True)` |
+-------+---------+----+-----+---------+------------------------------------------------------------+
You also have the option of specifying which GPUs to use by passing a list:
.. code-block:: python
# DEFAULT (int) specifies how many GPUs to use.
Trainer(gpus=k)
# Above is equivalent to
Trainer(gpus=list(range(k)))
# You specify which GPUs (don't use if running on cluster)
Trainer(gpus=[0, 1])
# can also be a string
Trainer(gpus='0, 1')
# can also be -1 or '-1', this uses all available GPUs
# this is equivalent to list(range(torch.cuda.available_devices()))
Trainer(gpus=-1)
CUDA flags
----------
CUDA flags make certain GPUs visible to your script.
Lightning sets these for you automatically, there's NO NEED to do this yourself.
.. code-block:: python
# lightning will set according to what you give the trainer
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
However, when using a cluster, Lightning will NOT set these flags (and you should not either).
SLURM will set these for you.
16-bit mixed precision
----------------------
16 bit precision can cut your memory footprint by half. If using volta architecture GPUs
it can give a dramatic training speed-up as well.
First, install apex (if install fails, look `here <https://github.com/NVIDIA/apex>`_::
$ git clone https://github.com/NVIDIA/apex
$ cd apex
# ------------------------
# OPTIONAL: on your cluster you might need to load cuda 10 or 9
# depending on how you installed PyTorch
# see available modules
module avail
# load correct cuda before install
module load cuda-10.0
# ------------------------
# make sure you've loaded a cuda version > 4.0 and < 7.0
module load gcc-6.1.0
$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
then set this use_amp to True.::
# DEFAULT
trainer = Trainer(amp_level='O2', use_amp=False)
Single-gpu
----------
Make sure you're on a GPU machine.::
# DEFAULT
trainer = Trainer(gpus=1)
Multi-gpu
---------
Make sure you're on a GPU machine. You can set as many GPUs as you want.
In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
.. code-block:: python
# to use DataParallel
trainer = Trainer(gpus=8, distributed_backend='dp')
# RECOMMENDED use DistributedDataParallel
trainer = Trainer(gpus=8, distributed_backend='ddp')
Multi-node
----------
Multi-node training is easily done by specifying these flags.
.. code-block:: python
# train on 12*8 GPUs
trainer = Trainer(gpus=8, nb_gpu_nodes=12, distributed_backend='ddp')
You must configure your job submission script correctly for the trainer to work.
Here is an example script for the above trainer configuration.
.. code-block:: bash
#!/bin/bash -l
# SLURM SUBMIT SCRIPT
#SBATCH --nodes=12
#SBATCH --gres=gpu:8
#SBATCH --ntasks-per-node=8
#SBATCH --mem=0
#SBATCH --time=0-02:00:00
# activate conda env
conda activate my_env
# -------------------------
# OPTIONAL
# -------------------------
# debugging flags (optional)
# export NCCL_DEBUG=INFO
# export PYTHONFAULTHANDLER=1
# PyTorch comes with prebuilt NCCL support... but if you have issues with it
# you might need to load the latest version from your modules
# module load NCCL/2.4.7-1-cuda.10.0
# on your cluster you might need these:
# set the network interface
# export NCCL_SOCKET_IFNAME=^docker0,lo
# -------------------------
# random port between 12k and 20k
export MASTER_PORT=$((12000 + RANDOM % 20000))
# run script from above
python my_main_file.py
.. note:: When running in DDP mode, any errors in your code will show up as an NCCL issue.
Set the `NCCL_DEBUG=INFO` flag to see the ACTUAL error.
Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a
portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).
.. code-block:: python
# ie: this:
dataset = myDataset()
dataloader = Dataloader(dataset)
# becomes:
dataset = myDataset()
dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
dataloader = Dataloader(dataset, sampler=dist_sampler)
Auto-slurm-job-submission
-------------------------
Instead of manually building SLURM scripts, you can use the
`SlurmCluster object <https://williamfalcon.github.io/test-tube/hpc/SlurmCluster>`_
to do this for you. The SlurmCluster can also run a grid search if you pass
in a `HyperOptArgumentParser
<https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser>`_.
Here is an example where you run a grid search of 9 combinations of hyperparams.
The full examples are `here
<https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/new_project_templates/multi_node_examples>`_.
.. code-block:: python
# grid search 3 values of learning rate and 3 values of number of layers for your net
# this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32),
# (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
parser.opt_list('--learning_rate', default=0.001, type=float,
options=[1e-3, 1e-2, 1e-1], tunable=True)
parser.opt_list('--layers', default=1, type=float, options=[16, 32, 64], tunable=True)
hyperparams = parser.parse_args()
# Slurm cluster submits 9 jobs, each with a set of hyperparams
cluster = SlurmCluster(
hyperparam_optimizer=hyperparams,
log_path='/some/path/to/save',
)
# OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
# which interface your nodes use for communication
cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
# see output of the NCCL connection process
# NCCL is how the nodes talk to each other
cluster.add_command('export NCCL_DEBUG=INFO')
# setting a master port here is a good idea.
cluster.add_command('export MASTER_PORT=%r' % PORT)
# ************** DON'T FORGET THIS ***************
# MUST load the latest NCCL version
cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
# configure cluster
cluster.per_experiment_nb_nodes = 12
cluster.per_experiment_nb_gpus = 8
cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
# submit a script with 9 combinations of hyper params
# (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
cluster.optimize_parallel_cluster_gpu(
main,
nb_trials=9, # how many permutations of the grid search to run
job_name='name_for_squeue'
)
The other option is that you generate scripts on your own via a bash command or use another library...
Self-balancing architecture
---------------------------
Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.
"""
import torch
from pytorch_lightning.overrides.data_parallel import (
pytorch_lightning/trainer/evaluation_loop_mixin.py
+125
View File
@@ -1,3 +1,128 @@
"""
# Validation loop
The lightning validation loop handles everything except the actual computations of your model.
To decide what will happen in your validation loop, define the `validation_step` function.
Below are all the things lightning automates for you in the validation loop.
.. note:: Lightning will run 5 steps of validation in the beginning of training as a sanity
check so you don't have to wait until a full epoch to catch possible validation issues.
Check validation every n epochs
-------------------------------
If you have a small dataset you might want to check validation every n epochs
.. code-block:: python
# DEFAULT
trainer = Trainer(check_val_every_n_epoch=1)
Set how much of the validation set to check
-------------------------------------------
If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
.. code-block:: python
# DEFAULT
trainer = Trainer(val_percent_check=1.0)
# check 10% only
trainer = Trainer(val_percent_check=0.1)
Set how much of the test set to check
-------------------------------------
If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
.. code-block:: python
# DEFAULT
trainer = Trainer(test_percent_check=1.0)
# check 10% only
trainer = Trainer(test_percent_check=0.1)
Set validation check frequency within 1 training epoch
------------------------------------------------------
For large datasets it's often desirable to check validation multiple times within a training loop.
Pass in a float to check that often within 1 training epoch.
Pass in an int k to check every k training batches. Must use an int if using an IterableDataset.
.. code-block:: python
# DEFAULT
trainer = Trainer(val_check_interval=0.95)
# check every .25 of an epoch
trainer = Trainer(val_check_interval=0.25)
# check every 100 train batches (ie: for IterableDatasets or fixed frequency)
trainer = Trainer(val_check_interval=100)
Set the number of validation sanity steps
-----------------------------------------
Lightning runs a few steps of validation in the beginning of training.
This avoids crashing in the validation loop sometime deep into a lengthy training loop.
.. code-block:: python
# DEFAULT
trainer = Trainer(nb_sanity_val_steps=5)
You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.
# Testing loop
To ensure you don't accidentally use test data to guide training decisions Lightning
makes running the test set deliberate.
**test**
You have two options to run the test set.
First case is where you test right after a full training routine.
.. code-block:: python
# run full training
trainer.fit(model)
# run test set
trainer.test()
Second case is where you load a model and run the test set
.. code-block:: python
model = MyLightningModule.load_from_metrics(
weights_path='/path/to/pytorch_checkpoint.ckpt',
tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
on_gpu=True,
map_location=None
)
# init trainer with whatever options
trainer = Trainer(...)
# test (pass in the model)
trainer.test(model)
In this second case, the options you pass to trainer will be used when running
the test set (ie: 16-bit, dp, ddp, etc...)
"""
import torch
import sys
import tqdm
pytorch_lightning/trainer/train_loop_mixin.py
+151
View File
@@ -1,3 +1,154 @@
"""
The lightning training loop handles everything except the actual computations of your model.
To decide what will happen in your training loop, define the `training_step` function.
Below are all the things lightning automates for you in the training loop.
Accumulated gradients
---------------------
Accumulated gradients runs K small batches of size N before doing a backwards pass.
The effect is a large effective batch size of size KxN.
.. code-block:: python
# DEFAULT (ie: no accumulated grads)
trainer = Trainer(accumulate_grad_batches=1)
Force training for min or max epochs
------------------------------------
It can be useful to force training for a minimum number of epochs or limit to a max number
.. code-block:: python
# DEFAULT
trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000)
Early stopping
--------------
The trainer already sets up default early stopping for you.
To modify this behavior, pass in your own EarlyStopping callback.
.. code-block:: python
from pytorch_lightning.callbacks import EarlyStopping
# DEFAULTS used by Trainer
early_stop_callback = EarlyStopping(
monitor='val_loss',
min_delta=0.00,
patience=3,
verbose=False,
mode='min'
)
# without passing anything in, uses the default callback above
trainer = Trainer()
# pass in your own to override the default callback
trainer = Trainer(early_stop_callback=early_stop_callback)
# pass in None to disable it
trainer = Trainer(early_stop_callback=None)
Force disable early stop
------------------------
To disable early stopping pass None to the early_stop_callback
.. code-block:: python
# DEFAULT
trainer = Trainer(early_stop_callback=None)
Gradient Clipping
-----------------
Gradient clipping may be enabled to avoid exploding gradients.
Specifically, this will `clip the gradient norm computed over all model parameters
`together <https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_>`_.
.. code-block:: python
# DEFAULT (ie: don't clip)
trainer = Trainer(gradient_clip_val=0)
# clip gradients with norm above 0.5
trainer = Trainer(gradient_clip_val=0.5)
Inspect gradient norms
----------------------
Looking at grad norms can help you figure out where training might be going wrong.
.. code-block:: python
# DEFAULT (-1 doesn't track norms)
trainer = Trainer(track_grad_norm=-1)
# track the LP norm (P=2 here)
trainer = Trainer(track_grad_norm=2)
Set how much of the training set to check
-----------------------------------------
If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
.. code-block:: python
# DEFAULT
trainer = Trainer(train_percent_check=1.0)
# check 10% only
trainer = Trainer(train_percent_check=0.1)
Packed sequences as inputs
--------------------------
When using PackedSequence, do 2 things:
1. return either a padded tensor in dataset or a list of variable length tensors
in the dataloader collate_fn (example above shows the list implementation).
2. Pack the sequence in forward or training and validation steps depending on use case.
.. code-block:: python
# For use in dataloader
def collate_fn(batch):
x = [item[0] for item in batch]
y = [item[1] for item in batch]
return x, y
# In module
def training_step(self, batch, batch_nb):
x = rnn.pack_sequence(batch[0], enforce_sorted=False)
y = rnn.pack_sequence(batch[1], enforce_sorted=False)
Truncated Backpropagation Through Time
--------------------------------------
There are times when multiple backwards passes are needed for each batch.
For example, it may save memory to use Truncated Backpropagation Through Time when training RNNs.
When this flag is enabled each batch is split into sequences of size truncated_bptt_steps
and passed to training_step(...) separately. A default splitting function is provided,
however, you can override it for more flexibility. See `tbptt_split_batch`.
.. code-block:: python
# DEFAULT (single backwards pass per batch)
trainer = Trainer(truncated_bptt_steps=None)
# (split batch into sequences of size 2)
trainer = Trainer(truncated_bptt_steps=2)
"""
import numpy as np
import tqdm
pytorch_lightning/trainer/trainer.py
+35 -36
View File
@@ -89,45 +89,45 @@ class Trainer(TrainerIOMixin,
:param logger: Logger for experiment tracking
:param checkpoint_callback: Callback for checkpointing
:param early_stop_callback: Callback for early stopping
:param default_save_path: Default path for logs+weights if no logger/ckpt_callback passed
:param gradient_clip_val: int. 0 means don't clip.
:param gradient_clip: int. 0 means don't clip. Deprecated.
:param str default_save_path: Default path for logs+weights if no logger/ckpt_callback passed
:param int gradient_clip_val: 0 means don't clip.
:param int gradient_clip: 0 means don't clip. Deprecated.
:param process_position: shown in the tqdm bar
:param nb_gpu_nodes: number of GPU nodes
:param int nb_gpu_nodes: number of GPU nodes
:param gpus: int. (ie: 2 gpus) OR list to specify which GPUs [0, 1] OR '0,1'
OR '-1' / -1 to use all available gpus
:param log_gpu_memory: str. None, 'min_max', 'all'
:param show_progress_bar: Bool. If true shows tqdm bar
:param overfit_pct: float. uses this much of all datasets
:param track_grad_norm: int. -1 no tracking. Otherwise tracks that norm
:param check_val_every_n_epoch: int. check val every n train epochs
:param fast_dev_run: Bool. runs full iteration over everything to find bugs
:param accumulate_grad_batches: int. Accumulates grads every k batches
:param max_nb_epochs: int.
:param min_nb_epochs: int.
:param train_percent_check: int. How much of train set to check
:param val_percent_check: int. How much of val set to check
:param test_percent_check: int. How much of test set to check
:param val_check_interval: float/int. If float, % of tng epoch. If int, check every n batch
:param log_save_interval: int. Writes logs to disk this often
:param row_log_interval: int. How often to add logging rows
:param add_row_log_interval: int. How often to add logging rows. Deprecated.
:param distributed_backend: str. Options: 'dp', 'ddp', 'ddp2'.
:param use_amp: Bool. If true uses apex for 16bit precision
:param print_nan_grads: Bool. Prints nan gradients
:param weights_summary: str. Options: 'full', 'top', None to not print.
:param weights_save_path: Bool. Where to save weights if on cluster
:param amp_level: str. Check nvidia docs for level
:param nb_sanity_val_steps: int. How many val steps before a full train loop.
:param truncated_bptt_steps: int. Enables multiple backward passes for each batch.
:param str log_gpu_memory: None, 'min_max', 'all'
:param bool show_progress_bar: If true shows tqdm bar
:param float overfit_pct: uses this much of all datasets
:param int track_grad_norm: -1 no tracking. Otherwise tracks that norm
:param int check_val_every_n_epoch: check val every n train epochs
:param bool fast_dev_run: runs full iteration over everything to find bugs
:param int accumulate_grad_batches: Accumulates grads every k batches
:param int max_nb_epochs:
:param int min_nb_epochs:
:param int train_percent_check: How much of train set to check
:param int val_percent_check: How much of val set to check
:param int test_percent_check: How much of test set to check
:param float|int val_check_interval: If float, % of tng epoch. If int, check every n batch
:param int log_save_interval: Writes logs to disk this often
:param int row_log_interval: How often to add logging rows
:param int add_row_log_interval: How often to add logging rows. Deprecated.
:param str distributed_backend: Options: 'dp', 'ddp', 'ddp2'.
:param bool use_amp: If true uses apex for 16bit precision
:param bool print_nan_grads: Prints nan gradients
:param str weights_summary: Options: 'full', 'top', None to not print.
:param bool weights_save_path: Where to save weights if on cluster
:param str amp_level: Check nvidia docs for level
:param int nb_sanity_val_steps: How many val steps before a full train loop.
:param int truncated_bptt_steps: Enables multiple backward passes for each batch.
"""
# Transfer params
self.nb_gpu_nodes = nb_gpu_nodes
self.log_gpu_memory = log_gpu_memory
if not (gradient_clip is None):
# Backward compatibility
warnings.warn("gradient_clip has renamed to gradient_clip_val since v0.5.0",
DeprecationWarning)
warnings.warn("`gradient_clip` has renamed to `gradient_clip_val` since v0.5.0"
" and will be removed in v0.8.0", DeprecationWarning)
gradient_clip_val = gradient_clip
self.gradient_clip_val = gradient_clip_val
self.check_val_every_n_epoch = check_val_every_n_epoch
@@ -223,8 +223,8 @@ class Trainer(TrainerIOMixin,
self.val_check_interval = val_check_interval
if not (add_row_log_interval is None):
# backward compatibility
warnings.warn("gradient_clip has renamed to gradient_clip_val since v0.5.0",
DeprecationWarning)
warnings.warn("`gradient_clip` has renamed to `gradient_clip_val` since v0.5.0"
" and will be removed in v0.8.0", DeprecationWarning)
row_log_interval = add_row_log_interval
self.row_log_interval = row_log_interval
@@ -319,12 +319,11 @@ class Trainer(TrainerIOMixin,
@property
def tng_tqdm_dic(self):
"""
* Deprecated in v0.5.0. use training_tqdm_dict instead. *
"""*Deprecated in v0.5.0. use training_tqdm_dict instead.*
:return:
"""
warnings.warn("tng_tqdm_dict has renamed to training_tqdm_dict since v0.5.0",
DeprecationWarning)
warnings.warn("`tng_tqdm_dict` has renamed to `training_tqdm_dict` since v0.5.0"
" and will be removed in v0.8.0", DeprecationWarning)
return self.training_tqdm_dict
# -----------------------------
pytorch_lightning/trainer/trainer_io.py
+91
View File
@@ -1,3 +1,94 @@
"""
Lightning can automate saving and loading checkpoints
=====================================================
Checkpointing is enabled by default to the current working directory.
To change the checkpoint path pass in::
Trainer(default_save_path='/your/path/to/save/checkpoints')
To modify the behavior of checkpointing pass in your own callback.
.. code-block:: python
from pytorch_lightning.callbacks import ModelCheckpoint
# DEFAULTS used by the Trainer
checkpoint_callback = ModelCheckpoint(
filepath=os.getcwd(),
save_best_only=True,
verbose=True,
monitor='val_loss',
mode='min',
prefix=''
)
trainer = Trainer(checkpoint_callback=checkpoint_callback)
Restoring training session
--------------------------
You might want to not only load a model but also continue training it. Use this method to
restore the trainer state as well. This will continue from the epoch and global step you last left off.
However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).
Lightning will restore the session if you pass a logger with the same version and there's a saved checkpoint.
.. code-block:: python
from pytorch_lightning import Trainer
from pytorch_lightning.logging import TestTubeLogger
logger = TestTubeLogger(
save_dir='./savepath',
version=1 # An existing version with a saved checkpoint
)
trainer = Trainer(
logger=logger,
default_save_path='./savepath'
)
# this fit call loads model weights and trainer state
# the trainer continues seamlessly from where you left off
# without having to do anything else.
trainer.fit(model)
The trainer restores:
- global_step
- current_epoch
- All optimizers
- All lr_schedulers
- Model weights
You can even change the logic of your model as long as the weights and "architecture" of
the system isn't different. If you add a layer, for instance, it might not work.
At a rough level, here's what happens inside Trainer :py:mod:`pytorch_lightning.base_module.model_saving.py`:
.. code-block:: python
self.global_step = checkpoint['global_step']
self.current_epoch = checkpoint['epoch']
# restore the optimizers
optimizer_states = checkpoint['optimizer_states']
for optimizer, opt_state in zip(self.optimizers, optimizer_states):
optimizer.load_state_dict(opt_state)
# restore the lr schedulers
lr_schedulers = checkpoint['lr_schedulers']
for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
scheduler.load_state_dict(lrs_state)
# uses the model you passed into trainer
model.load_state_dict(checkpoint['state_dict'])
"""
import os
import re
import signal
pytorch_lightning/utilities/debugging.py
+76
View File
@@ -1,2 +1,78 @@
"""
These flags are useful to help debug a model.
Fast dev run
------------
This flag is meant for debugging a full train/val/test loop.
It'll activate callbacks, everything but only with 1 training and 1 validation batch.
Use this to debug a full run of your program quickly
.. code-block:: python
# DEFAULT
trainer = Trainer(fast_dev_run=False)
Inspect gradient norms
----------------------
Looking at grad norms can help you figure out where training might be going wrong.
.. code-block:: python
# DEFAULT (-1 doesn't track norms)
trainer = Trainer(track_grad_norm=-1)
# track the LP norm (P=2 here)
trainer = Trainer(track_grad_norm=2)
Make model overfit on subset of data
------------------------------------
A useful debugging trick is to make your model overfit a tiny fraction of the data.
setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
.. code-block:: python
# DEFAULT don't overfit (ie: normal training)
trainer = Trainer(overfit_pct=0.0)
# overfit on 1% of data
trainer = Trainer(overfit_pct=0.01)
Print the parameter count by layer
----------------------------------
By default lightning prints a list of parameters *and submodules* when it starts training.
.. code-block:: python
# DEFAULT print a full list of all submodules and their parameters.
trainer = Trainer(weights_summary='full')
# only print the top-level modules (i.e. the children of LightningModule).
trainer = Trainer(weights_summary='top')
Print which gradients are nan
-----------------------------
This option prints a list of tensors with nan gradients::
# DEFAULT
trainer = Trainer(print_nan_grads=False)
Log GPU usage
-------------
Lightning automatically logs gpu usage to the test tube logs.
It'll only do it at the metric logging interval, so it doesn't slow down training.
"""
class MisconfigurationException(Exception):
pass