Merge branch 'master' into tb

* fix dangling gradients

make sure only the gradients of the current optimizer's paramaters are calculated in the training step.

* add note about multiple optimizer gradient update

* Update training_loop.py

.circleci/config.yml

@@ -0,0 +1,90 @@
+# Python CircleCI 2.0 configuration file
+#
+# Check https://circleci.com/docs/2.0/language-python/ for more details
+#
+version: 2.0
+
+references:
+
+  install_deps: &install_deps
+    run:
+      name: Install Dependences
+      command: |
+        pip install "$TORCH_VERSION" --user
+        # this is temporal fix til test-tube is not merged and released
+        pip install -r requirements.txt --user
+        sudo pip install pytest pytest-cov pytest-flake8
+        pip install -r ./tests/requirements.txt --user
+
+  tests_format: &tests_format
+   run:
+     name: Tests and formating
+     command: |
+       python --version ; pip --version ; pip list
+       py.test pytorch_lightning tests pl_examples -v --doctest-modules --junitxml=test-reports/pytest_junit.xml --flake8
+     no_output_timeout: 15m
+
+  make_docs: &make_docs
+   run:
+     name: Make Documentation
+     command: |
+       # sudo apt-get install pandoc
+       pip install -r requirements.txt --user
+       sudo pip install -r docs/requirements.txt
+       # sphinx-apidoc -o ./docs/source ./pytorch_lightning **/test_* --force --follow-links
+       cd docs; make clean ; make html
+
+jobs:
+
+  Build-Docs:
+    docker:
+      - image: circleci/python:3.7
+    steps:
+      - checkout
+      - *make_docs
+
+  PyTorch:
+    docker:
+      - image: circleci/python:3.7
+    environment:
+      - TORCH_VERSION: "torch"
+    steps: &steps
+      - checkout
+
+      - *install_deps
+      - *tests_format
+
+      - store_test_results:
+          path: test-reports
+      - store_artifacts:
+          path: test-reports
+
+  PyTorch-v1.1:
+    docker:
+      - image: circleci/python:3.6
+    environment:
+      - TORCH_VERSION: "torch>=1.1, <1.2"
+    steps: *steps
+
+  PyTorch-v1.2:
+    docker:
+      - image: circleci/python:3.6
+    environment:
+      - TORCH_VERSION: "torch>=1.2, <1.3"
+    steps: *steps
+
+  PyTorch-v1.3:
+    docker:
+      - image: circleci/python:3.6
+    environment:
+      - TORCH_VERSION: "torch>=1.3, <1.4"
+    steps: *steps
+
+workflows:
+  version: 2
+  build:
+    jobs:
+      - Build-Docs
+      - PyTorch-v1.1
+      - PyTorch-v1.2
+      - PyTorch-v1.3

.github/ISSUE_TEMPLATE/bug_report.md

@@ -11,26 +11,52 @@ assignees: ''
 1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/williamFalcon/pytorch-lightning/issues/79).    
 2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
 
-**Describe the bug**
-A clear and concise description of what the bug is.
+## 🐛 Bug
+
+<!-- A clear and concise description of what the bug is. -->
+
+### To Reproduce
 
-**To Reproduce**
 Steps to reproduce the behavior:
+
 1. Go to '...'
-2. Click on '....'
+2. Run '....'
 3. Scroll down to '....'
 4. See error
 
-**Expected behavior**
-A clear and concise description of what you expected to happen.
+<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
 
-**Screenshots**
-If applicable, add screenshots to help explain your problem.
 
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
+#### Code sample
+<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
+Minimal means having the shortest code but still preserving the bug. -->
 
-**Additional context**
-Add any other context about the problem here.
+### Expected behavior
+
+<!-- A clear and concise description of what you expected to happen. -->
+
+### Environment
+
+Please copy and paste the output from our
+[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
+(or fill out the checklist below manually).
+
+You can get the script and run it with:
+```
+wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
+# For security purposes, please check the contents of collect_env.py before running it.
+python collect_env.py
+```
+
+ - PyTorch Version (e.g., 1.0):
+ - OS (e.g., Linux):
+ - How you installed PyTorch (`conda`, `pip`, source):
+ - Build command you used (if compiling from source):
+ - Python version:
+ - CUDA/cuDNN version:
+ - GPU models and configuration:
+ - Any other relevant information:
+
+### Additional context
+
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/documentation.md

@@ -7,11 +7,12 @@ assignees: ''
 
 ---
 
+## 📚 Documentation
+
 For typos and doc fixes, please go ahead and:
 
 1. Create an issue.
 2. Fix the typo.   
 3. Submit a PR.
 
-
 Thanks!

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,14 +7,21 @@ assignees: ''
 
 ---
 
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+## 🚀 Feature
+<!-- A clear and concise description of the feature proposal -->
 
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
+### Motivation
 
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
+<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
 
-**Additional context**
-Add any other context or screenshots about the feature request here.
+### Pitch
+
+<!-- A clear and concise description of what you want to happen. -->
+
+### Alternatives
+
+<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
+
+### Additional context
+
+<!-- Add any other context or screenshots about the feature request here. -->

.github/ISSUE_TEMPLATE/how-to-question.md

@@ -7,20 +7,24 @@ assignees: ''
 
 ---
 
+## ❓ Questions and Help
+
 ### Before asking:   
 1. search the issues.   
 2. search the docs.    
 
-If you still can't find what you need:     
-#### What is your question?    
+<!-- If you still can't find what you need: -->
 
-#### Code    
-Please paste a code snippet if your question requires it!   
+#### What is your question?
 
-#### What have you tried?    
+#### Code
 
-#### What's your environment?   
-- conda version (no venv)   
-- PyTorch version   
-- Lightning version   
-- Test-tube version
+<!-- Please paste a code snippet if your question requires it! -->   
+
+#### What have you tried?
+
+#### What's your environment?
+
+ - OS: [e.g. iOS, Linux, Win]
+ - Packaging [e.g. pip, conda]
+ - Version [e.g. 0.5.2.1]

.readthedocs.yml

@@ -5,9 +5,13 @@
 # Required
 version: 2
 
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/source/conf.py
+
 # Build documentation with MkDocs
-mkdocs:
-  configuration: mkdocs.yml
+#mkdocs:
+#  configuration: mkdocs.yml
 
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: all
@@ -16,4 +20,5 @@ formats: all
 python:
   version: 3.7
   install:
-    - requirements: docs/requirements.txt
+    #- requirements: requirements.txt
+    - requirements: docs/requirements.txt

.run_local_tests.sh

@@ -2,6 +2,7 @@
 rm -rf _ckpt_*
 rm -rf tests/save_dir*
 rm -rf tests/mlruns_*
+rm -rf tests/cometruns*
 rm -rf tests/tests/*
 rm -rf lightning_logs
 coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules

.travis.yml

@@ -16,9 +16,16 @@ language: python
 
 matrix:
   include:
-#    - dist: xenial  # Ubuntu 16.04
-#      python: 3.5
-#      env: TOXENV=py35
+    - dist: xenial  # Ubuntu 16.04
+      python: 3.6
+      env:
+        - TOXENV=py36
+        - MIN_REQUIREMENTS=1
+    - dist: xenial  # Ubuntu 16.04
+      python: 3.7
+      env:
+        - TOXENV=py37
+        - MIN_REQUIREMENTS=1
     - dist: bionic  # Ubuntu 18.04
       python: 3.6
       env: TOXENV=py36
@@ -26,13 +33,14 @@ matrix:
       python: 3.7
       env: TOXENV=py37
     - os: osx
-      osx_image: xcode9.4
+      # https://blog.travis-ci.com/2019-08-07-extensive-python-testing-on-travis-ci
+      osx_image: xcode10.3
       language: generic
-      env: TOXENV=py36
-      addons:
-        homebrew:
-          # update: true
-          packages: python3.6
+      env: TOXENV=py37
+      #addons:
+      #  homebrew:
+      #    # update: true
+      #    packages: python3.7
       before_install:
         - pip3 install virtualenv
         - virtualenv -p python3 ~/venv
@@ -51,11 +59,26 @@ install:
   - pip install future  # needed for `builtins`
   - sudo pip install tox
 
+before_script:
+  # rewrite all minimal requirements as strict
+  - if [[ "${MIN_REQUIREMENTS}" == "1" ]]; then
+      python -c "req = open('requirements.txt').read().replace('>', '=') ; open('requirements-ci.txt', 'w').write(req)" ;
+    else
+      cp requirements.txt requirements-ci.txt ;
+    fi
+  - pip install -r requirements-ci.txt -U
+
 script:
   # integration
   - tox --sitepackages
-  - pip install --editable .
+
   #- python setup.py install --dry-run --user
+  - virtualenv vEnv ;
+    source vEnv/bin/activate
+  - pip install --editable . ;
+    cd .. & python -c "import pytorch_lightning ; print(pytorch_lightning.__version__)"
+  - deactivate ;
+    rm -rf vEnv
 
 after_success:
   - coverage report

MANIFEST.in

@@ -37,6 +37,7 @@ exclude *.yml
 
 prune .git
 prune .github
+prune .circleci
 prune notebook*
 prune temp*
 prune test*

README.md

@@ -1,6 +1,6 @@
 <div align="center">
 
-![Logo](./docs/source/_static/lightning_logo_small.png)
+![Logo](docs/source/_static/images/lightning_logo_small.png)
 
 # PyTorch Lightning
 
@@ -11,11 +11,11 @@
 [![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
 [![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
 [![Build status](https://ci.appveyor.com/api/projects/status/NEW-PROJECT-ID?svg=true)](https://ci.appveyor.com/project/williamFalcon/pytorch-lightning)
-[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
+[![Coverage](docs/source/_static/images/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
 [![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
 
 [![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
-[![Gitter](https://badges.gitter.im/PyTorch-Lightning/community.svg)](https://gitter.im/PyTorch-Lightning/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+[![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/pytorch-lightning/shared_invite/enQtODU5ODIyNTUzODQwLTFkMDg5Mzc1MDBmNjEzMDgxOTVmYTdhYjA1MDdmODUyOTg2OGQ1ZWZkYTQzODhhNzdhZDA3YmNhMDhlMDY4YzQ)
 [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
 [![Next Release](https://img.shields.io/badge/Next%20Release-Dec%206-<COLOR>.svg)](https://shields.io/)
 
@@ -34,8 +34,26 @@ pip install pytorch-lightning
 ## Docs   
 **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**
 
+## Demo  
+[Copy and run this COLAB!](https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=HOk9c4_35FKg)
+
 ## What is it?  
-Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.
+Lightning is a very lightweight wrapper on PyTorch that decouples the science code from the engineering code. It's more of a style-guide than a framework. By refactoring your code, we can automate most of the non-research code.  
+
+To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format (the science) and Lightning will automate the rest (the engineering). Lightning guarantees tested, correct, modern best practices for the automated parts.
+
+- If you are a researcher, Lightning is infinitely flexible, you can modify everything down to the way .backward is called or distributed is set up. 
+- If you are a scientist or production team, lightning is very simple to use with best practice defaults.
+
+## What does lightning control for me?   
+
+Everything in Blue!   
+This is how lightning separates the science (red) from the engineering (blue).
+
+![Overview](docs/source/_static/images/pl.gif)
+
+## How much effort is it to convert?
+You're probably tired of switching frameworks at this point. But it is a very quick process to refactor into the Lightning format (ie: hours). [Check out this tutorial](https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538)
 
 ## Starting a new project?   
 [Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     
@@ -63,7 +81,7 @@ Lightning sets up all the boilerplate state-of-the-art training for you so you c
 ---
 
 ## How do I do use it?   
-Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/) which you fit using a Trainer.    
+Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/) which you fit using a Trainer.
 
 The LightningModule defines a *system* such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier such as the example below.     
 
@@ -91,7 +109,7 @@ To use lightning do 2 things:
         def forward(self, x):
             return torch.relu(self.l1(x.view(x.size(0), -1)))
     
-        def training_step(self, batch, batch_nb):
+        def training_step(self, batch, batch_idx):
             # REQUIRED
             x, y = batch
             y_hat = self.forward(x)
@@ -99,7 +117,7 @@ To use lightning do 2 things:
             tensorboard_logs = {'train_loss': loss}
             return {'loss': loss, 'log': tensorboard_logs}
     
-        def validation_step(self, batch, batch_nb):
+        def validation_step(self, batch, batch_idx):
             # OPTIONAL
             x, y = batch
             y_hat = self.forward(x)
@@ -149,16 +167,16 @@ use something other than tensorboard).
 Here are more advanced examples
 ```python   
 # train on cpu using only 10% of the data (for demo purposes)
-trainer = Trainer(max_nb_epochs=1, train_percent_check=0.1)
+trainer = Trainer(max_epochs=1, train_percent_check=0.1)
 
 # train on 4 gpus (lightning chooses GPUs for you)
-# trainer = Trainer(max_nb_epochs=1, gpus=4, distributed_backend='ddp')  
+# trainer = Trainer(max_epochs=1, gpus=4, distributed_backend='ddp')  
 
 # train on 4 gpus (you choose GPUs)
-# trainer = Trainer(max_nb_epochs=1, gpus=[0, 1, 3, 7], distributed_backend='ddp')   
+# trainer = Trainer(max_epochs=1, gpus=[0, 1, 3, 7], distributed_backend='ddp')   
 
 # train on 32 gpus across 4 nodes (make sure to submit appropriate SLURM job)
-# trainer = Trainer(max_nb_epochs=1, gpus=8, nb_gpu_nodes=4, distributed_backend='ddp')
+# trainer = Trainer(max_epochs=1, gpus=8, num_gpu_nodes=4, distributed_backend='ddp')
 
 # train (1 epoch only here for demo)
 trainer.fit(model)
@@ -173,34 +191,11 @@ When you're all done you can even run the test set separately.
 trainer.test()
 ```
 
-## What does lightning control for me?   
-
-Everything in gray!    
-You define the blue parts using the LightningModule interface:  
-
-![Overview](./docs/source/_static/overview_flat.jpg)
-
-```python
-# what to do in the training loop
-def training_step(self, batch, batch_nb):
-
-# what to do in the validation loop
-def validation_step(self, batch, batch_nb):
-
-# how to aggregate validation_step outputs
-def validation_end(self, outputs):
-
-# and your dataloaders
-def train_dataloader():
-def val_dataloader():
-def test_dataloader():
-```
-
 **Could be as complex as seq-2-seq + attention**    
 
 ```python
 # define what happens for training here
-def training_step(self, batch, batch_nb):
+def training_step(self, batch, batch_idx):
     x, y = batch
     
     # define your own forward and loss calculation
@@ -227,7 +222,7 @@ def training_step(self, batch, batch_nb):
 
 ```python
 # define what happens for validation here
-def validation_step(self, batch, batch_nb):    
+def validation_step(self, batch, batch_idx):    
     x, y = batch
     
     # or as basic as a CNN classification
@@ -261,11 +256,11 @@ def validation_end(self, outputs):
 ## Tensorboard    
 Lightning is fully integrated with tensorboard, MLFlow and supports any logging module.   
 
-![tensorboard-support](./docs/source/_static/tf_loss.png)
+![tensorboard-support](docs/source/_static/images/tf_loss.png)
 
 Lightning also adds a text column with all the hyperparameters for this experiment.      
 
-![tensorboard-support](./docs/source/_static/tf_tags.png)
+![tensorboard-support](docs/source/_static/images/tf_tags.png)
 
 ## Lightning automates all of the following ([each is also configurable](https://williamfalcon.github.io/pytorch-lightning/Trainer/)):
 

appveyor.yml

@@ -44,11 +44,13 @@ install:
   # purpose but it is problematic because it tends to cancel builds pushed
   # directly to master instead of just PR builds (or the converse).
   - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%path%
-  - pip install -U --user pip
-  - pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
-  - pip install -r ./tests/requirements.txt
+  #- pip install -U --user "pip<19.3"
+  - python -m pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
+  - python -m pip install -r ./tests/requirements.txt
+  - python -m pip install pytest-flake8
 
-# scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
+# scripts to run before tests (working directory and environment changes
+# are persisted from the previous steps such as "before_build")
 before_test:
   - python --version
   - pip --version
@@ -57,7 +59,9 @@ before_test:
 
 # to run your custom scripts instead of automatic tests
 test_script:
-  - tox --sitepackages --parallel auto
+  - coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules --flake8
+  #- python setup.py sdist
+  #- twine check dist/*
 
 on_success:
   - coverage report

docs/LightningModule/RequiredTrainerInterface.md

@@ -1,801 +0,0 @@
-# Lightning Module interface
-[[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/root_module.py)]
-
-A lightning module is a strict superclass of nn.Module, it provides a standard interface for the trainer to interact with the model.
-
-The easiest thing to do is copy the [minimal example](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) below and modify accordingly. 
-
-Otherwise, to Define a Lightning Module, implement the following methods:
-
-**Required**:  
-
-- [training_step](RequiredTrainerInterface.md#training_step)      
-- [train_dataloader](RequiredTrainerInterface.md#train_dataloader)    
-- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)    
-
-**Optional**:   
-
-- [training_end](RequiredTrainerInterface.md#training_end)      
-- [validation_step](RequiredTrainerInterface.md#validation_step)    
-- [validation_end](RequiredTrainerInterface.md#validation_end) 
-- [test_step](RequiredTrainerInterface.md#test_step)    
-- [test_end](RequiredTrainerInterface.md#test_end) 
-- [val_dataloader](RequiredTrainerInterface.md#val_dataloader)    
-- [test_dataloader](RequiredTrainerInterface.md#test_dataloader)    
-- [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)    
-- [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)    
-- [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)    
-
----
-### Minimal example   
-```python
-import os
-import torch
-from torch.nn import functional as F
-from torch.utils.data import DataLoader
-from torchvision.datasets import MNIST
-import torchvision.transforms as transforms
-
-import pytorch_lightning as pl
-
-class CoolModel(pl.LightningModule):
-
-    def __init__(self):
-        super(CoolModel, self).__init__()
-        # not the best model...
-        self.l1 = torch.nn.Linear(28 * 28, 10)
-
-    def forward(self, x):
-        return torch.relu(self.l1(x.view(x.size(0), -1)))
-
-    def training_step(self, batch, batch_nb):
-        # REQUIRED
-        x, y = batch
-        y_hat = self.forward(x)
-        return {'loss': F.cross_entropy(y_hat, y)}
-
-    def validation_step(self, batch, batch_nb):
-        # OPTIONAL
-        x, y = batch
-        y_hat = self.forward(x)
-        return {'val_loss': F.cross_entropy(y_hat, y)}
-
-    def validation_end(self, outputs):
-        # OPTIONAL
-        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
-        return {'avg_val_loss': avg_loss}
-
-    def test_step(self, batch, batch_nb):
-        # OPTIONAL
-        x, y = batch
-        y_hat = self.forward(x)
-        return {'test_loss': F.cross_entropy(y_hat, y)}
-
-    def test_end(self, outputs):
-        # OPTIONAL
-        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
-        return {'avg_test_loss': avg_loss}
-
-    def configure_optimizers(self):
-        # REQUIRED
-        return torch.optim.Adam(self.parameters(), lr=0.02)
-
-    @pl.data_loader
-    def train_dataloader(self):
-        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
-
-    @pl.data_loader
-    def val_dataloader(self):
-        # OPTIONAL
-        # can also return a list of val dataloaders
-        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
-
-    @pl.data_loader
-    def test_dataloader(self):
-        # OPTIONAL
-        # can also return a list of test dataloaders
-        return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
-```
----   
-### How do these methods fit into the broader training?     
-The LightningModule interface is on the right. Each method corresponds to a part of a research project. Lightning automates everything not in blue.    
-
-<p align="center">
-  <a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg">
-    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg" height="900px">
-  </a>
-</p>   
-
-## Required Methods    
-
-### training_step
-
-``` {.python}
-def training_step(self, batch, batch_nb)
-```
-
-In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
-
-**Params**    
-
-| Param  | description  |
-|---|---|
-|  batch | The output of your dataloader. A tensor, tuple or list  |
-|  batch_nb | Integer displaying which batch this is  |
-
-**Return**   
-
-Dictionary or OrderedDict   
-
-| key  | value  | is required |
-|---|---|---|
-|  loss | tensor scalar  | Y |
-|  progress_bar | Dict for progress bar display. Must have only tensors  | N |
-|  log | Dict of metrics to add to logger. Must have only tensors (no images, etc)  | N |
-
-
-**Example**
-
-``` {.python}
-def training_step(self, batch, batch_nb):
-    x, y, z = batch
-    
-    # implement your own
-    out = self.forward(x)
-    loss = self.loss(out, x)
-    
-    logger_logs = {'training_loss': loss} # optional (MUST ALL BE TENSORS)
-    
-    # if using TestTubeLogger or TensorboardLogger you can nest scalars
-    logger_logs = {'losses': logger_logs} # optional (MUST ALL BE TENSORS)
-    
-    output = {
-        'loss': loss, # required
-        'progress_bar': {'training_loss': loss}, # optional (MUST ALL BE TENSORS)
-        'log': logger_logs
-    }
-    
-    # return a dict
-    return output
-```    
-
-If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.    
-``` {.python}
-# Multiple optimizers (ie: GANs)     
-def training_step(self, batch, batch_nb, optimizer_idx):
-    if optimizer_idx == 0:
-        # do training_step with encoder
-    if optimizer_idx == 1:
-        # do training_step with decoder    
-```    
-
-If you add truncated back propagation through time you will also get an additional argument with the hidden states of the previous step.     
-``` {.python}
-# Truncated back-propagation through time   
-def training_step(self, batch, batch_nb, hiddens):
-    # hiddens are the hiddens from the previous truncated backprop step
-```    
-
-You can also return a -1 instead of a dict to stop the current loop. This is useful if you want to
-break out of the current training epoch early.
-
----   
-### training_end
-
-``` {.python}
-def training_end(self, train_step_outputs)
-```
-In certain cases (dp, ddp2), you might want to use all outputs of every process to do something.
-For instance, if using negative samples, you could run a batch via dp and use ALL the outputs 
-for a single softmax across the full batch (ie: the denominator would use the full batch).
-
-In this case you should define training_end to perform those calculations.
-
-
-**Params**    
-
-| Param  | description  |
-|---|---|
-|  outputs | What you return in training_step. 
-
-**Return**   
-
-Dictionary or OrderedDict   
-
-| key  | value  | is required |
-|---|---|---|
-|  loss | tensor scalar  | Y |
-|  progress_bar | Dict for progress bar display. Must have only tensors  | N |
-|  log | Dict of metrics to add to logger. Must have only tensors (no images, etc)  | N |
-
-
-**Example**
-
-``` {.python}
-# WITHOUT training_end
-# if used in DP or DDP2, this batch is 1/nb_gpus large
-def training_step(self, batch, batch_nb):
-    # batch is 1/nb_gpus big
-    x, y = batch
-    
-    out = self.forward(x)
-    loss = self.softmax(out)
-    loss = nce_loss(loss)
-    return {'loss': loss}
-
-# --------------
-# with training_end to do softmax over the full batch
-def training_step(self, batch, batch_nb):
-    # batch is 1/nb_gpus big
-    x, y = batch
-    
-    out = self.forward(x)
-    return {'out': out}
-
-def training_end(self, outputs):
-    # this out is now the full size of the batch
-    out = outputs['out']
-
-    # this softmax now uses the full batch size
-    loss = self.softmax(out)
-    loss = nce_loss(loss)
-    return {'loss': loss}
-```    
-
-If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.    
-``` {.python}
-# Multiple optimizers (ie: GANs)     
-def training_step(self, batch, batch_nb, optimizer_idx):
-    if optimizer_idx == 0:
-        # do training_step with encoder
-    if optimizer_idx == 1:
-        # do training_step with decoder    
-```    
-
-If you add truncated back propagation through time you will also get an additional argument with the hidden states of the previous step.     
-``` {.python}
-# Truncated back-propagation through time   
-def training_step(self, batch, batch_nb, hiddens):
-    # hiddens are the hiddens from the previous truncated backprop step
-```    
-
-You can also return a -1 instead of a dict to stop the current loop. This is useful if you want to
-break out of the current training epoch early.
-
---- 
-### train_dataloader 
-
-``` {.python}
-@pl.data_loader
-def train_dataloader(self)
-```
-Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.   
-If you want to change the data during every epoch DON'T use the data_loader decorator.
-
-##### Return
-PyTorch DataLoader
-
-**Example**
-
-``` {.python}
-@pl.data_loader
-def train_dataloader(self):
-    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
-    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
-    loader = torch.utils.data.DataLoader(
-        dataset=dataset,
-        batch_size=self.hparams.batch_size,
-        shuffle=True
-    )
-    return loader
-```
-
---- 
-### configure_optimizers 
-
-``` {.python}
-def configure_optimizers(self)
-```
-
-Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
-Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
-
-**Note:** If you use multiple optimizers, training_step will have an additional ```optimizer_idx``` parameter.    
-**Note 2:** If you use LBFGS lightning handles the closure function automatically for you.
-
-##### Return    
-Return any of these 3 options:   
-Single optimizer   
-List or Tuple - List of optimizers    
-Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers
-
-**Example**
-
-``` {.python}
-# most cases
-def configure_optimizers(self):
-    opt = Adam(self.parameters(), lr=0.01)
-    return opt
-
-# multiple optimizer case (eg: GAN)
-def configure_optimizers(self):
-    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
-    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
-    return generator_opt, disriminator_opt
-    
-# example with learning_rate schedulers  
-def configure_optimizers(self):
-    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
-    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
-    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
-    return [generator_opt, disriminator_opt], [discriminator_sched]
-```  
-
-If you need to control how often those optimizers step or override the default .step() schedule, override 
-the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step) hook.   
-
-## Optional Methods    
-
-### validation_step
-
-``` {.python}
-# if you have one val dataloader:
-def validation_step(self, batch, batch_nb)   
-
-# if you have multiple val dataloaders:  
-def validation_step(self, batch, batch_nb, dataloader_idxdx)
-```
-**OPTIONAL**    
-If you don't need to validate you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
-
-When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
-
-The dict you return here will be available in the `validation_end` method. 
-
-**Params**   
-
-| Param  | description  |
-|---|---|
-|  batch | The output of your dataloader. A tensor, tuple or list  |
-|  batch_nb | Integer displaying which batch this is  |
-|  dataloader_idx | Integer displaying which dataloader this is (only if multiple val datasets used)  |
-
-**Return**   
-
-| Return  | description  | optional |
-|---|---|---|   
-|  dict | Dict or OrderedDict - passed to the validation_end step | N |
-
-**Example**
-
-``` {.python}
-# CASE 1: A single validation dataset
-def validation_step(self, batch, batch_nb):
-    x, y = batch
-    
-    # implement your own
-    out = self.forward(x)
-    loss = self.loss(out, y)
-    
-    # log 6 example images
-    # or generated text... or whatever
-    sample_imgs = x[:6]
-    grid = torchvision.utils.make_grid(sample_imgs)
-    self.logger.experiment.add_image('example_images', grid, 0) 
-    
-    # calculate acc
-    labels_hat = torch.argmax(out, dim=1)
-    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-    
-    # all optional...
-    # return whatever you need for the collation function validation_end
-    output = OrderedDict({
-        'val_loss': loss_val,
-        'val_acc': torch.tensor(val_acc), # everything must be a tensor
-    })
-    
-    # return an optional dict
-    return output
-```   
-
-If you pass in multiple validation datasets, validation_step will have an additional argument.
-
-```python
-# CASE 2: multiple validation datasets
-def validation_step(self, batch, batch_nb, dataset_idx):
-    # dataset_idx tells you which dataset this is.   
-```   
-
-The ```dataset_idx``` corresponds to the order of datasets returned in ```val_dataloader```.    
-
---- 
-### validation_end
-
-``` {.python}
-def validation_end(self, outputs)
-```   
-If you didn't define a validation_step, this won't be called. Called at the end of the validation loop with the outputs of validation_step.
-
-The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
-Any keys present in 'log', 'progress_bar' or the rest of the dictionary are available for callbacks to access.
-**Params**    
-
-| Param  | description  |
-|---|---|
-|  outputs | List of outputs you defined in validation_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
-
-**Return**   
-
-Dictionary or OrderedDict   
-
-| key  | value  | is required |
-|---|---|---|
-|  progress_bar | Dict for progress bar display. Must have only tensors  | N |
-|  log | Dict of metrics to add to logger. Must have only tensors (no images, etc)  | N |
-
-**Example**
-
-With a single dataloader
-
-``` {.python}
-def validation_end(self, outputs):
-    """
-    Called at the end of validation to aggregate outputs
-    :param outputs: list of individual outputs of each validation step
-    :return:
-    """
-    val_loss_mean = 0
-    val_acc_mean = 0
-    for output in outputs:
-        val_loss_mean += output['val_loss']
-        val_acc_mean += output['val_acc']
-
-    val_loss_mean /= len(outputs)
-    val_acc_mean /= len(outputs)
-    tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-       
-    # show val_loss and val_acc in progress bar but only log val_loss
-    results = {
-        'progress_bar': tqdm_dict,
-        'log': {'val_loss': val_loss_mean.item()}
-    }
-    return results
-```
-
-With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
-one entry per dataloader, while the inner list contains the individual outputs of 
-each validation step for that dataloader.
-
-``` {.python}
-def validation_end(self, outputs):
-    """
-    Called at the end of validation to aggregate outputs
-    :param outputs: list of list of individual outputs of each validation step
-    :return:
-    """
-    val_loss_mean = 0
-    val_acc_mean = 0
-    i = 0
-    for dataloader_outputs in outputs:
-        for output in dataloader_outputs:
-            val_loss_mean += output['val_loss']
-            val_acc_mean += output['val_acc']
-            i += 1
-
-    val_loss_mean /= i
-    val_acc_mean /= i
-    tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-    
-    # show val_loss and val_acc in progress bar but only log val_loss
-    results = {
-        'progress_bar': tqdm_dict,
-        'log': {'val_loss': val_loss_mean.item()}
-    }
-    return results
-```
-
-### test_step
-
-``` {.python}
-# if you have one test dataloader:
-def test_step(self, batch, batch_nb)   
-
-# if you have multiple test dataloaders:  
-def test_step(self, batch, batch_nb, dataloader_idxdx)
-```
-**OPTIONAL**    
-If you don't need to test you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy.   
-
-When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
-
-The dict you return here will be available in the `test_end` method. 
-
-This function is used when you execute `trainer.test()`.
-
-**Params**   
-
-| Param  | description  |
-|---|---|
-|  batch | The output of your dataloader. A tensor, tuple or list  |
-|  batch_nb | Integer displaying which batch this is  |
-|  dataloader_idx | Integer displaying which dataloader this is (only if multiple test datasets used)  |
-
-**Return**   
-
-| Return  | description  | optional |
-|---|---|---|   
-|  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
-
-**Example**
-
-``` {.python}
-# CASE 1: A single test dataset
-def test_step(self, batch, batch_nb):
-    x, y = batch
-    
-    # implement your own
-    out = self.forward(x)
-    loss = self.loss(out, y)
-    
-    # calculate acc
-    labels_hat = torch.argmax(out, dim=1)
-    test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-    
-    # all optional...
-    # return whatever you need for the collation function test_end
-    output = OrderedDict({
-        'test_loss': loss_test,
-        'test_acc': torch.tensor(test_acc), # everything must be a tensor
-    })
-    
-    # return an optional dict
-    return output
-```   
-
-If you pass in multiple test datasets, test_step will have an additional argument.
-
-```python
-# CASE 2: multiple test datasets
-def test_step(self, batch, batch_nb, dataset_idx):
-    # dataset_idx tells you which dataset this is.   
-```   
-
-The ```dataset_idx``` corresponds to the order of datasets returned in ```test_dataloader```.    
-
---- 
-### test_end
-
-``` {.python}
-def test_end(self, outputs)
-```   
-If you didn't define a test_step, this won't be called.       
-
-Called at the end of the test step with the output of each test_step.
-
-The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
-
-**Params**    
-
-| Param  | description  |
-|---|---|
-|  outputs | List of outputs you defined in test_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
-
-**Return**   
-
-| Return  | description  | optional |
-|---|---|---|   
-|  dict | Dict of OrderedDict with metrics to display in progress bar | Y |
-
-**Example**
-
-``` {.python}
-def test_end(self, outputs):
-    """
-    Called at the end of test to aggregate outputs
-    :param outputs: list of individual outputs of each test step
-    :return:
-    """
-    test_loss_mean = 0
-    test_acc_mean = 0
-    for output in outputs:
-        test_loss_mean += output['test_loss']
-        test_acc_mean += output['test_acc']
-
-    test_loss_mean /= len(outputs)
-    test_acc_mean /= len(outputs)
-    tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
-    
-    # show test_loss and test_acc in progress bar but only log test_loss
-    results = {
-        'progress_bar': tqdm_dict,
-        'log': {'test_loss': val_loss_mean.item()}
-    }
-    return results
-```
-
-With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
-one entry per dataloader, while the inner list contains the individual outputs of 
-each validation step for that dataloader.
-
-``` {.python}
-def test_end(self, outputs):
-    """
-    Called at the end of test to aggregate outputs
-    :param outputs: list of individual outputs of each test step
-    :return:
-    """
-    test_loss_mean = 0
-    test_acc_mean = 0
-    i = 0
-    for dataloader_outputs in outputs:
-        for output in dataloader_outputs:
-            test_loss_mean += output['test_loss']
-            test_acc_mean += output['test_acc']
-            i += 1
-
-    test_loss_mean /= i 
-    test_acc_mean /= i
-    tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
-    
-    # show test_loss and test_acc in progress bar but only log test_loss
-    results = {
-        'progress_bar': tqdm_dict,
-        'log': {'test_loss': val_loss_mean.item()}
-    }
-    return results
-```
-
---- 
-### on_save_checkpoint 
-
-``` {.python}
-def on_save_checkpoint(self, checkpoint)
-```
-Called by lightning to checkpoint your model. Lightning saves the training state (current epoch, global_step, etc)
-and also saves the model state_dict. If you want to save anything else, use this method to add your own
-key-value pair.
-
-##### Return
-Nothing
-
-**Example**
-
-``` {.python}
-def on_save_checkpoint(self, checkpoint):
-    # 99% of use cases you don't need to implement this method 
-    checkpoint['something_cool_i_want_to_save'] = my_cool_pickable_object
-```
-
---- 
-### on_load_checkpoint 
-
-``` {.python}
-def on_load_checkpoint(self, checkpoint)
-```
-Called by lightning to restore your model. Lighting auto-restores global step, epoch, etc...
-It also restores the model state_dict.
-If you saved something with **on_save_checkpoint** this is your chance to restore this.
-
-##### Return
-Nothing  
-
-**Example**
-
-``` {.python}
-def on_load_checkpoint(self, checkpoint):
-    # 99% of the time you don't need to implement this method
-    self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']
-```
-
---- 
-### val_dataloader 
-
-``` {.python}
-@pl.data_loader
-def val_dataloader(self)
-```
-**OPTIONAL**    
-If you don't need a validation dataset and a validation_step, you don't need to implement this method.    
-
-Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.   
-If you want to change the data during every epoch DON'T use the data_loader decorator.   
-
-##### Return
-PyTorch DataLoader or list of PyTorch Dataloaders.    
-
-**Example**
-
-``` {.python}
-@pl.data_loader
-def val_dataloader(self):
-    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
-    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
-    loader = torch.utils.data.DataLoader(
-        dataset=dataset,
-        batch_size=self.hparams.batch_size,
-        shuffle=True
-    )
-    
-    return loader
-
-# can also return multiple dataloaders   
-@pl.data_loader
-def val_dataloader(self):
-    return [loader_a, loader_b, ..., loader_n]   
-```
-
-In the case where you return multiple val_dataloaders, the validation_step will have an arguement ```dataset_idx```
-which matches the order here.    
-
---- 
-### test_dataloader 
-
-``` {.python}
-@pl.data_loader
-def test_dataloader(self)
-```
-**OPTIONAL**    
-If you don't need a test dataset and a test_step, you don't need to implement this method.    
-
-Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
-If you want to change the data during every epoch DON'T use the data_loader decorator.   
-
-##### Return
-PyTorch DataLoader
-
-**Example**
-
-``` {.python}
-@pl.data_loader
-def test_dataloader(self):
-    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
-    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
-    loader = torch.utils.data.DataLoader(
-        dataset=dataset,
-        batch_size=self.hparams.batch_size,
-        shuffle=True
-    )
-    
-    return loader
-```
-
---- 
-### add_model_specific_args 
-
-``` {.python}
-@staticmethod
-def add_model_specific_args(parent_parser, root_dir)
-```
-Lightning has a list of default argparse commands.
-This method is your chance to add or modify commands specific to your model.
-The [hyperparameter argument parser](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/) is available anywhere in your model by calling self.hparams.
-
-##### Return
-An argument parser
-
-**Example**
-
-``` {.python}
-@staticmethod
-def add_model_specific_args(parent_parser, root_dir):
-    parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
-
-    # param overwrites
-    # parser.set_defaults(gradient_clip_val=5.0)
-
-    # network params
-    parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
-    parser.add_argument('--in_features', default=28*28)
-    parser.add_argument('--out_features', default=10)
-    parser.add_argument('--hidden_dim', default=50000) # use 500 for CPU, 50000 for GPU to see speed difference
-
-    # data
-    parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
-
-    # training params (opt)
-    parser.opt_list('--learning_rate', default=0.001, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
-                    tunable=False)
-    parser.opt_list('--batch_size', default=256, type=int, options=[32, 64, 128, 256], tunable=False)
-    parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)
-    return parser
-```

docs/LightningModule/methods.md

@@ -1,67 +0,0 @@
-Lightning modules are strict superclasses of torch.nn.Module. A LightningModule offers the following in addition to that API.
-
----    
-### freeze 
-Freeze all params for inference
-```{.python}
-model = MyLightningModule(...)
-model.freeze()
-```
-
----    
-### load_from_metrics
-This is the easiest/fastest way which loads hyperparameters and weights from a checkpoint,
-such as the one saved by the `ModelCheckpoint` callback
-
-```{.python}
-pretrained_model = MyLightningModule.load_from_checkpoint(
-    checkpoint_path='/path/to/pytorch_checkpoint.ckpt'
-)
-    
-# predict
-pretrained_model.eval()
-pretrained_model.freeze()
-y_hat = pretrained_model(x)
-```
-
----    
-### load_from_metrics
-If you're using test tube, there is an alternate method which uses the meta_tags.csv
-file from test-tube to rebuild the model. The meta_tags.csv file can be found in the
-test-tube experiment save_dir.       
-
-```{.python}
-pretrained_model = MyLightningModule.load_from_metrics(
-    weights_path='/path/to/pytorch_checkpoint.ckpt',
-    tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
-    on_gpu=True,
-    map_location=None
-)
-    
-# predict
-pretrained_model.eval()
-pretrained_model.freeze()
-y_hat = pretrained_model(x)
-```
-
-**Params**    
-
-| Param  | description  |
-|---|---|
-|  weights_path | Path to a PyTorch checkpoint  |
-|  tags_csv | Path to meta_tags.csv file generated by the test-tube Experiment  |
-|  on_gpu | if True, puts model on GPU. Make sure to use transforms option if model devices have changed  |
-|  map_location | A dictionary mapping saved weight GPU devices to new GPU devices |
-
-**Returns**    
-
-LightningModule - The pretrained LightningModule
-
----    
-### unfreeze 
-Unfreeze all params for inference
-```{.python}
-model = MyLightningModule(...)
-model.unfreeze()
-```
-

docs/LightningModule/properties.md

@@ -1,64 +0,0 @@
-A LightningModule has the following properties which you can access at any time
-
---- 
-#### current_epoch
-The current epoch   
-
----
-#### dtype    
-Current dtype    
-
----
-#### logger    
-A reference to the logger you passed into trainer. 
-Passing a logger is optional. If you don't pass one in, Lightning will create one for you automatically.
-This logger saves logs to '''/os.getcwd()/lightning_logs'''
-```python
-Trainer(logger=your_logger)
-```    
-
-Call it from anywhere in your LightningModule to add metrics, images, etc... whatever your logger supports.  
-
-Here is an example using the TestTubeLogger (which is a wrapper on [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html) with versioned folder structure).     
-```{.python}
-# if logger is a tensorboard logger or TestTubeLogger
-self.logger.experiment.add_embedding(...)   
-self.logger.experiment.log({'val_loss': 0.9})   
-self.logger.experiment.add_scalars(...)   
-```
-
---- 
-#### global_step   
-Total training batches seen across all epochs   
-
---- 
-#### gradient_clip_val   
-The current gradient clip value    
-
----
-#### on_gpu    
-True if your model is currently running on GPUs. Useful to set flags around the LightningModule for different CPU vs GPU behavior.    
-
----
-#### trainer    
-Last resort access to any state the trainer has. Changing certain properties here could affect your training run.
-```{.python}   
-self.trainer.optimizers
-self.trainer.current_epoch
-...   
-```   
-
-## Debugging   
-The LightningModule also offers these tricks to help debug.   
-
----   
-#### example_input_array    
-In the LightningModule init, you can set a dummy tensor for this property
-to get a print out of sizes coming into and out of every layer.   
-```python
-def __init__(self):
-    # put the dimensions of the first input to your system
-    self.example_input_array = torch.rand(5, 28 * 28)
-```    
-
-

docs/Makefile

@@ -0,0 +1,19 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/Trainer/Checkpointing.md

@@ -1,84 +0,0 @@
-Lightning can automate saving and loading checkpoints.
-
----
-### Model saving
-Checkpointing is enabled by default to the current working directory.
-To change the checkpoint path pass in :
-```python
-Trainer(default_save_path='/your/path/to/save/checkpoints')
-```
-
-To modify the behavior of checkpointing pass in your own callback.
-
-``` {.python}
-from pytorch_lightning.callbacks import ModelCheckpoint
-
-# DEFAULTS used by the Trainer
-checkpoint_callback = ModelCheckpoint(
-    filepath=os.getcwd(),
-    save_best_only=True,
-    verbose=True,
-    monitor='val_loss',
-    mode='min',
-    prefix=''
-)
-
-trainer = Trainer(checkpoint_callback=checkpoint_callback)
-```
-
----
-### Restoring training session 
-You might want to not only load a model but also continue training it. Use this method to
-restore the trainer state as well. This will continue from the epoch and global step you last left off.  
-However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).   
-
-Lightning will restore the session if you pass a logger with the same version and there's a saved checkpoint.   
-``` {.python}
-from pytorch_lightning import Trainer
-from pytorch_lightning.logging import TestTubeLogger
-
-logger = TestTubeLogger(
-    save_dir='./savepath',
-    version=1  # An existing version with a saved checkpoint
-)
-trainer = Trainer(
-    logger=logger,
-    default_save_path='./savepath'
-)
-
-# this fit call loads model weights and trainer state
-# the trainer continues seamlessly from where you left off
-# without having to do anything else.
-trainer.fit(model)
-```
-
-The trainer restores:  
-
-- global_step    
-- current_epoch    
-- All optimizers    
-- All lr_schedulers    
-- Model weights
-
-You can even change the logic of your model as long as the weights and "architecture" of 
-the system isn't different. If you add a layer, for instance, it might not work.   
-
-At a rough level, here's [what happens inside Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/model_saving.py#L63):   
-```python
-
-self.global_step = checkpoint['global_step']
-self.current_epoch = checkpoint['epoch']
-
-# restore the optimizers
-optimizer_states = checkpoint['optimizer_states']
-for optimizer, opt_state in zip(self.optimizers, optimizer_states):
-    optimizer.load_state_dict(opt_state)
-
-# restore the lr schedulers
-lr_schedulers = checkpoint['lr_schedulers']
-for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
-    scheduler.load_state_dict(lrs_state)
-
-# uses the model you passed into trainer        
-model.load_state_dict(checkpoint['state_dict'])
-```    

docs/Trainer/Distributed training.md

@@ -1,262 +0,0 @@
-Lightning makes multi-gpu training and 16 bit training trivial.
-
-*Note:*   
-None of the flags below require changing anything about your lightningModel definition. 
-
----
-#### Choosing a backend  
-Lightning supports two backends. DataParallel and DistributedDataParallel. Both can be used for single-node multi-GPU training.
-For multi-node training you must use DistributedDataParallel.   
-
-##### DataParallel (dp)   
-Splits a batch across multiple GPUs on the same node. Cannot be used for multi-node training.   
-
-##### DistributedDataParallel (ddp)   
-Trains a copy of the model on each GPU and only syncs gradients. If used with DistributedSampler, each GPU trains
-on a subset of the full dataset.  
-
-##### DistributedDataParallel-2 (ddp2)   
-Works like DDP, except each node trains a single copy of the model using ALL GPUs on that node. 
-Very useful when dealing with negative samples, etc...
-
-You can toggle between each mode by setting this flag.
-``` {.python}
-# DEFAULT (when using single GPU or no GPUs)
-trainer = Trainer(distributed_backend=None)
-
-# Change to DataParallel (gpus > 1)
-trainer = Trainer(distributed_backend='dp')
-
-# change to distributed data parallel (gpus > 1)
-trainer = Trainer(distributed_backend='ddp')
-
-# change to distributed data parallel (gpus > 1)
-trainer = Trainer(distributed_backend='ddp2')
-```
-
-If you request multiple nodes, the back-end will auto-switch to ddp.
-We recommend you use DistributedDataparallel even for single-node multi-GPU training. It is MUCH faster than DP but *may*
-have configuration issues depending on your cluster.
-
-For a deeper understanding of what lightning is doing, feel free to read [this guide](https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565).   
-
----
-#### Distributed and 16-bit precision.    
-Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
-not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.   
-
-Below are the possible configurations we support.    
-
-| 1 GPU  | 1+ GPUs  | DP  | DDP  | 16-bit  | command |
-|---|---|---|---|---|---|
-| Y  |   |   |   |  | ```Trainer(gpus=1)``` |
-| Y  |   |   |   | Y | ```Trainer(gpus=1, use_amp=True)``` |
-|   | Y | Y |   |   | ```Trainer(gpus=k, distributed_backend='dp')``` |
-|   | Y |  | Y  |  | ```Trainer(gpus=k, distributed_backend='ddp')``` |
-|   | Y |  | Y  | Y | ```Trainer(gpus=k, distributed_backend='ddp', use_amp=True)``` |
-
-You also have the option of specifying which GPUs to use by passing a list:   
-
-```python
-# DEFAULT (int) specifies how many GPUs to use.
-Trainer(gpus=k)  
-
-# Above is equivalent to 
-Trainer(gpus=list(range(k)))
-
-# You specify which GPUs (don't use if running on cluster)  
-Trainer(gpus=[0, 1])  
-
-# can also be a string
-Trainer(gpus='0, 1')
-
-# can also be -1 or '-1', this uses all available GPUs
-# this is equivalent to list(range(torch.cuda.available_devices()))
-Trainer(gpus=-1)
-```
-
----
-#### CUDA flags   
-CUDA flags make certain GPUs visible to your script. 
-Lightning sets these for you automatically, there's NO NEED to do this yourself.
-```python
-# lightning will set according to what you give the trainer
-# os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-# os.environ["CUDA_VISIBLE_DEVICES"] = "0"
-```
-
-However, when using a cluster, Lightning will NOT set these flags (and you should not either). 
-SLURM will set these for you.   
-
----
-#### 16-bit mixed precision
-16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.    
-First, install apex (if install fails, look [here](https://github.com/NVIDIA/apex)):
-```bash
-$ git clone https://github.com/NVIDIA/apex
-$ cd apex
-
-# ------------------------
-# OPTIONAL: on your cluster you might need to load cuda 10 or 9
-# depending on how you installed PyTorch
-
-# see available modules
-module avail
-
-# load correct cuda before install
-module load cuda-10.0
-# ------------------------
-
-# make sure you've loaded a cuda version > 4.0 and < 7.0
-module load gcc-6.1.0
-
-$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
-```
-
-then set this use_amp to True.
-``` {.python}
-# DEFAULT
-trainer = Trainer(amp_level='O2', use_amp=False)
-```
-
----
-#### Single-gpu
-Make sure you're on a GPU machine. 
-```python
-# DEFAULT
-trainer = Trainer(gpus=1)
-```
-
----
-#### multi-gpu 
-Make sure you're on a GPU machine. You can set as many GPUs as you want.
-In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
-```python
-# to use DataParallel
-trainer = Trainer(gpus=8, distributed_backend='dp')
-
-# RECOMMENDED use DistributedDataParallel
-trainer = Trainer(gpus=8, distributed_backend='ddp')
-```
-
----
-#### Multi-node
-Multi-node training is easily done by specifying these flags. 
-```python
-# train on 12*8 GPUs
-trainer = Trainer(gpus=8, nb_gpu_nodes=12, distributed_backend='ddp')
-```
-
-You must configure your job submission script correctly for the trainer to work. Here is an example
-script for the above trainer configuration.   
-
-```sh
-#!/bin/bash -l
-
-# SLURM SUBMIT SCRIPT
-#SBATCH --nodes=12
-#SBATCH --gres=gpu:8
-#SBATCH --ntasks-per-node=8
-#SBATCH --mem=0
-#SBATCH --time=0-02:00:00
-
-# activate conda env
-conda activate my_env
-
-# -------------------------
-# OPTIONAL
-# -------------------------
-# debugging flags (optional)
-# export NCCL_DEBUG=INFO
-# export PYTHONFAULTHANDLER=1
-
-# PyTorch comes with prebuilt NCCL support... but if you have issues with it
-# you might need to load the latest version from your  modules
-# module load NCCL/2.4.7-1-cuda.10.0
-
-# on your cluster you might need these:
-# set the network interface
-# export NCCL_SOCKET_IFNAME=^docker0,lo
-# -------------------------
-
-# random port between 12k and 20k
-export MASTER_PORT=$((12000 + RANDOM % 20000))
-
-# run script from above
-python my_main_file.py
-```
-
-**NOTE:** When running in DDP mode, any errors in your code will show up as an NCCL issue.
-Set the ```NCCL_DEBUG=INFO``` flag to see the ACTUAL error.
-
-Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a 
-portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).   
-
-```python
-# ie: this:
-dataset = myDataset()
-dataloader = Dataloader(dataset)
-
-# becomes:
-dataset = myDataset()
-dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-dataloader = Dataloader(dataset, sampler=dist_sampler)
-```
-
-#### Auto-slurm-job-submission
-Instead of manually building SLURM scripts, you can use the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) to
-do this for you. The SlurmCluster can also run a grid search if you pass in a [HyperOptArgumentParser](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/).
-
-Here is an example where you run a grid search of 9 combinations of hyperparams.
-[The full examples are here](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/new_project_templates/multi_node_examples).
-```python
-# grid search 3 values of learning rate and 3 values of number of layers for your net
-# this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
-parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
-parser.opt_list('--learning_rate', default=0.001, type=float, options=[1e-3, 1e-2, 1e-1], tunable=True)
-parser.opt_list('--layers', default=1, type=float, options=[16, 32, 64], tunable=True)
-hyperparams = parser.parse_args()
-
-# Slurm cluster submits 9 jobs, each with a set of hyperparams
-cluster = SlurmCluster(
-    hyperparam_optimizer=hyperparams,
-    log_path='/some/path/to/save',
-)
-
-# OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
-# which interface your nodes use for communication
-cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
-
-# see output of the NCCL connection process
-# NCCL is how the nodes talk to each other
-cluster.add_command('export NCCL_DEBUG=INFO')
-
-# setting a master port here is a good idea.
-cluster.add_command('export MASTER_PORT=%r' % PORT)
-
-# ************** DON'T FORGET THIS ***************
-# MUST load the latest NCCL version
-cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
-
-# configure cluster
-cluster.per_experiment_nb_nodes = 12 
-cluster.per_experiment_nb_gpus = 8
-
-cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')  
-
-# submit a script with 9 combinations of hyper params
-# (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
-cluster.optimize_parallel_cluster_gpu(
-    main,
-    nb_trials=9, # how many permutations of the grid search to run
-    job_name='name_for_squeue'
-)
-```
-
-The other option is that you generate scripts on your own via a bash command or use another library...
-
----
-#### Self-balancing architecture
-Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.   
-
-COMING SOON.

docs/Trainer/Logging.md

@@ -1,224 +0,0 @@
-Lighting offers options for logging information about model, gpu usage, etc, via several different logging frameworks. It also offers printing options for training monitoring.
-
----   
-### default_save_path   
-Lightning sets a default TestTubeLogger and CheckpointCallback for you which log to
-```os.getcwd()``` by default. To modify the logging path you can set:
-```python
-Trainer(default_save_path='/your/path/to/save/checkpoints')
-```
- 
-If you need more custom behavior (different paths for both, different metrics, etc...)
-from the logger and the checkpointCallback, pass in your own instances as explained below.
-
-
----
-### Setting up logging
-
-The trainer inits a default logger for you (TestTubeLogger). All logs will
-go to the current working directory under a folder named ```os.getcwd()/lightning_logs``. 
-
-If you want to modify the default logging behavior even more, pass in a logger
-(which should inherit from `LightningBaseLogger`).   
-
-```{.python}
-my_logger = MyLightningLogger(...)
-trainer = Trainer(logger=my_logger)
-```
-
-The path in this logger will overwrite default_save_path.
-
-Lightning supports several common experiment tracking frameworks out of the box
-
----
-#### Test tube
-
-Log using [test tube](https://williamfalcon.github.io/test-tube/). Test tube logger is
-a strict subclass of [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html), refer to their
-documentation for all supported operations. The TestTubeLogger adds a nicer folder structure
-to manage experiments and snapshots all hyperparameters you pass to a LightningModule.
-
-```{.python}
-from pytorch_lightning.logging import TestTubeLogger
-tt_logger = TestTubeLogger(
-    save_dir=".",
-    name="default",
-    debug=False,
-    create_git_tag=False
-)
-trainer = Trainer(logger=tt_logger)
-```
-
-Use the logger anywhere in you LightningModule as follows:
-```python
-def train_step(...):
-    # example
-    self.logger.experiment.whatever_method_summary_writer_supports(...)
-    
-def any_lightning_module_function_or_hook(...):
-    self.logger.experiment.add_histogram(...)
-```
-
----
-#### MLFlow
-
-Log using [mlflow](https://mlflow.org)
-
-```{.python}
-from pytorch_lightning.logging import MLFlowLogger
-mlf_logger = MLFlowLogger(
-    experiment_name="default",
-    tracking_uri="file:/."
-)
-trainer = Trainer(logger=mlf_logger)
-```
-Use the logger anywhere in you LightningModule as follows:
-```python
-def train_step(...):
-    # example
-    self.logger.experiment.whatever_ml_flow_supports(...)
-    
-def any_lightning_module_function_or_hook(...):
-    self.logger.experiment.whatever_ml_flow_supports(...)
-```
-
----
-#### Comet.ml
-
-Log using [comet](https://www.comet.ml)
-
-```{.python}
-from pytorch_lightning.logging import CometLogger
-# arguments made to CometLogger are passed on to the comet_ml.Experiment class
-comet_logger = CometLogger(
-    api_key=os.environ["COMET_KEY"],
-    workspace=os.environ["COMET_KEY"],
-)
-trainer = Trainer(logger=comet_logger)
-```
-Use the logger anywhere in you LightningModule as follows:
-```python
-def train_step(...):
-    # example
-    self.logger.experiment.whatever_comet_ml_supports(...)
-
-def any_lightning_module_function_or_hook(...):
-    self.logger.experiment.whatever_comet_ml_supports(...)
-```
-
----
-#### Custom logger
-
-You can implement your own logger by writing a class that inherits from
-`LightningLoggerBase`. Use the `rank_zero_only` decorator to make sure that
-only the first process in DDP training logs data.
-
-```{.python}
-from pytorch_lightning.logging import LightningLoggerBase, rank_zero_only
-
-class MyLogger(LightningLoggerBase):
-
-    @rank_zero_only
-    def log_hyperparams(self, params):
-        # params is an argparse.Namespace
-        # your code to record hyperparameters goes here
-        pass
-    
-    @rank_zero_only
-    def log_metrics(self, metrics, step_num):
-        # metrics is a dictionary of metric names and values
-        # your code to record metrics goes here
-        pass
-    
-    def save(self):
-        # Optional. Any code necessary to save logger data goes here
-        pass
-    
-    @rank_zero_only
-    def finalize(self, status):
-        # Optional. Any code that needs to be run after training
-        # finishes goes here
-```
-
-If you write a logger than may be useful to others, please send
-a pull request to add it to Lighting!
-
----
-#### Using loggers
-You can call the logger anywhere from your LightningModule by doing:
-```python
-def train_step(...):
-    # example
-    self.logger.experiment.whatever_method_summary_writer_supports(...)
-    
-def any_lightning_module_function_or_hook(...):
-    self.logger.experiment.add_histogram(...)
-```
-
-#### Display metrics in progress bar 
-``` {.python}
-# DEFAULT
-trainer = Trainer(show_progress_bar=True)
-```
-
----
-#### Log metric row every k batches 
-Every k batches lightning will make an entry in the metrics log
-``` {.python}
-# DEFAULT (ie: save a .csv log file every 10 batches)
-trainer = Trainer(row_log_interval=10)
-```   
-
----
-#### Log GPU memory
-Logs GPU memory when metrics are logged.   
-``` {.python}
-# DEFAULT
-trainer = Trainer(log_gpu_memory=None)
-
-# log only the min/max utilization
-trainer = Trainer(log_gpu_memory='min_max')
-
-# log all the GPU memory (if on DDP, logs only that node)
-trainer = Trainer(log_gpu_memory='all')
-```
-
----
-#### Process position
-When running multiple models on the same machine we want to decide which progress bar to use.
-Lightning will stack progress bars according to this value. 
-``` {.python}
-# DEFAULT
-trainer = Trainer(process_position=0)
-
-# if this is the second model on the node, show the second progress bar below
-trainer = Trainer(process_position=1)
-```
-
----
-#### Save a snapshot of all hyperparameters 
-Automatically log hyperparameters stored in the `hparams` attribute as an `argparse.Namespace` 
-``` {.python}
-
-class MyModel(pl.Lightning):
-    def __init__(self, hparams):
-        self.hparams = hparams
-
-    ...
-
-args = parser.parse_args()
-model = MyModel(args)
-
-logger = TestTubeLogger(...)
-t = Trainer(logger=logger)
-trainer.fit(model)
-```
-
----
-#### Write logs file to csv every k batches 
-Every k batches, lightning will write the new logs to disk
-``` {.python}
-# DEFAULT (ie: save a .csv log file every 100 batches)
-trainer = Trainer(log_save_interval=100)
-```
-

docs/Trainer/SLURM Managed Cluster.md

@@ -1,112 +0,0 @@
-Lightning supports model training on a cluster managed by SLURM in the following cases:    
-
-1. Training on a single cpu or single GPU.
-2. Train on multiple GPUs on the same node using DataParallel or DistributedDataParallel
-3. Training across multiple GPUs on multiple different nodes via DistributedDataParallel.
-
-**Note: A node means a machine with multiple GPUs**
-
----
-#### Running grid search on a cluster
-To use lightning to run a hyperparameter search (grid-search or random-search) on a cluster do 4 things:   
-
-(1). Define the parameters for the grid search    
-    
-```{.python}
-from test_tube import HyperOptArgumentParser
-
-# subclass of argparse
-parser = HyperOptArgumentParser(strategy='random_search')
-parser.add_argument('--learning_rate', default=0.002, type=float, help='the learning rate')
-
-# let's enable optimizing over the number of layers in the network
-parser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4, 8])
-
-hparams = parser.parse_args()    
-```    
-    
-**NOTE** You must set ```Tunable=True``` for that argument to be considered in the permutation set. Otherwise
-test-tube will use the default value. This flag is useful when you don't want to search over an argument and
-want to use the default instead.   
-     
-(2). Define the cluster options in the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) (over 5 nodes and 8 gpus)    
-
-```{.python}
-from test_tube.hpc import SlurmCluster
-
-# hyperparameters is a test-tube hyper params object
-# see https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/
-hyperparams = args.parse()
-
-# init cluster
-cluster = SlurmCluster(
-    hyperparam_optimizer=hyperparams,
-    log_path='/path/to/log/results/to',
-    python_cmd='python3'
-)
-
-# let the cluster know where to email for a change in job status (ie: complete, fail, etc...)
-cluster.notify_job_status(email='some@email.com', on_done=True, on_fail=True)
-
-# set the job options. In this instance, we'll run 20 different models
-# each with its own set of hyperparameters giving each one 1 GPU (ie: taking up 20 GPUs)
-cluster.per_experiment_nb_gpus = 8
-cluster.per_experiment_nb_nodes = 5
-
-# we'll request 10GB of memory per node
-cluster.memory_mb_per_node = 10000
-
-# set a walltime of 10 minues
-cluster.job_time = '10:00'
-```
-
-(3). Make a main function with your model and trainer. Each job will call this function with a particular
-hparams configuration.    
-```{.python}
-from pytorch_lightning import Trainer
-
-def train_fx(trial_hparams, cluster_manager, _):
-    # hparams has a specific set of hyperparams
-    
-    my_model = MyLightningModel()
-    
-    # give the trainer the cluster object
-    trainer = Trainer()
-    trainer.fit(my_model)
-
-```
-
-(3). Start the grid/random search     
-```{.python}
-# run the models on the cluster
-cluster.optimize_parallel_cluster_gpu(
-    train_fx, 
-    nb_trials=20, 
-    job_name='my_grid_search_exp_name', 
-    job_display_name='my_exp')
-```
-
-**NOTE** nb_trials specifies how many of the possible permutations to use. If using ```grid_search``` it will use
-the depth first ordering. If using ```random_search``` it will use the first k shuffled options. FYI, random search
-has been shown to be just as good as any Bayesian optimization method when using a reasonable number of samples (60),
-[see this paper for more information](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf).
-
----
-#### Walltime auto-resubmit
-Lightning automatically resubmits jobs when they reach the walltime. Make sure to set the SIGUSR1 signal in 
-your SLURM script.   
-
-```bash
-# 90 seconds before training ends
-#SBATCH --signal=SIGUSR1@90
-``` 
-
-When lightning receives the SIGUSR1 signal it will:
-1. save a checkpoint with 'hpc_ckpt' in the name.
-2. resubmit the job using the SLURM_JOB_ID  
-
-When the script starts again, Lightning will:
-1. search for a 'hpc_ckpt' checkpoint. 
-2. restore the model, optimizers, schedulers, epoch, etc...   
-
-

docs/Trainer/Testing loop.md

@@ -1,31 +0,0 @@
-To ensure you don't accidentally use test data to guide training decisions Lightning makes running the test set deliberate.   
-
----
-#### test
-You have two options to run the test set. 
-First case is where you test right after a full training routine.
-``` {.python}
-# run full training
-trainer.fit(model)
-
-# run test set
-trainer.test()
-```   
-
-Second case is where you load a model and run the test set   
-```{.python}
-model = MyLightningModule.load_from_metrics(
-    weights_path='/path/to/pytorch_checkpoint.ckpt',
-    tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
-    on_gpu=True,
-    map_location=None
-)
-
-# init trainer with whatever options
-trainer = Trainer(...)
-    
-# test (pass in the model)
-trainer.test(model)
-```
-In this second case, the options you pass to trainer will be used when running the test set (ie: 16-bit, dp, ddp, etc...)  
- 

docs/Trainer/Training Loop.md

@@ -1,126 +0,0 @@
-The lightning training loop handles everything except the actual computations of your model. To decide what will happen in your training loop, define the [training_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#training_step).
-
-Below are all the things lightning automates for you in the training loop.
-
----
-#### Accumulated gradients
-Accumulated gradients runs K small batches of size N before doing a backwards pass. The effect is a large effective batch size of size KxN.
-
-``` {.python}
-# DEFAULT (ie: no accumulated grads)
-trainer = Trainer(accumulate_grad_batches=1)
-```
-
----
-#### Force training for min or max epochs
-It can be useful to force training for a minimum number of epochs or limit to a max number
-``` {.python}
-# DEFAULT
-trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000)
-```
-
----
-#### Early stopping
-The trainer already sets up default early stopping for you.
-To modify this behavior, pass in your own EarlyStopping callback.
-``` {.python}
-from pytorch_lightning.callbacks import EarlyStopping
-
-# DEFAULTS used by Trainer
-early_stop_callback = EarlyStopping(
-    monitor='val_loss',
-    min_delta=0.00,
-    patience=3,
-    verbose=False,
-    mode='min'
-)
-
-# without passing anything in, uses the default callback above
-trainer = Trainer()
-
-# pass in your own to override the default callback
-trainer = Trainer(early_stop_callback=early_stop_callback)
-
-# pass in None to disable it
-trainer = Trainer(early_stop_callback=None)
-```
-
----
-#### Force disable early stop
-To disable early stopping pass None to the early_stop_callback
-``` {.python}
-# DEFAULT
-trainer = Trainer(early_stop_callback=None)
-```
-
----
-#### Gradient Clipping
-Gradient clipping may be enabled to avoid exploding gradients.
-Specifically, this will [clip the gradient norm computed over all model parameters *together*](https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_).
-
-``` {.python}
-# DEFAULT (ie: don't clip)
-trainer = Trainer(gradient_clip_val=0)
-
-# clip gradients with norm above 0.5
-trainer = Trainer(gradient_clip_val=0.5)
-```
-
----
-#### Inspect gradient norms
-Looking at grad norms can help you figure out where training might be going wrong.
-``` {.python}
-# DEFAULT (-1 doesn't track norms)
-trainer = Trainer(track_grad_norm=-1)
-
-# track the LP norm (P=2 here)
-trainer = Trainer(track_grad_norm=2)
-```
-
-
----
-#### Set how much of the training set to check
-If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
-
-train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
-
-``` {.python}
-# DEFAULT
-trainer = Trainer(train_percent_check=1.0)
-
-# check 10% only
-trainer = Trainer(train_percent_check=0.1)
-```
-
----
-#### Packed sequences as inputs
-When using PackedSequence, do 2 things:
-1. return either a padded tensor in dataset or a list of variable length tensors in the dataloader collate_fn (example above shows the list implementation).    
-2. Pack the sequence in forward or training and validation steps depending on use case.
-
-``` {.python}
-# For use in dataloader
-def collate_fn(batch):
-    x = [item[0] for item in batch]
-    y = [item[1] for item in batch]
-    return x, y
-
-# In module
-def training_step(self, batch, batch_nb):
-    x = rnn.pack_sequence(batch[0], enforce_sorted=False)
-    y = rnn.pack_sequence(batch[1], enforce_sorted=False)
-```
-
----
-#### Truncated Backpropagation Through Time
-There are times when multiple backwards passes are needed for each batch. For example, it may save memory to use Truncated Backpropagation Through Time when training RNNs.
-
-When this flag is enabled each batch is split into sequences of size truncated_bptt_steps and passed to training_step(...) separately. A default splitting function is provided, however, you can override it for more flexibility. See [tbptt_split_batch](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks#tbptt_split_batch).
-
-``` {.python}
-# DEFAULT (single backwards pass per batch)
-trainer = Trainer(truncated_bptt_steps=None)
-
-# (split batch into sequences of size 2)
-trainer = Trainer(truncated_bptt_steps=2)
-```

docs/Trainer/Validation loop.md

@@ -1,70 +0,0 @@
-The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the [validation_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#validation_step).
-Below are all the things lightning automates for you in the validation loop.
-
-**Note**   
-Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don't have to wait until a full epoch to catch possible validation issues.
-
-
----
-#### Check validation every n epochs
-If you have a small dataset you might want to check validation every n epochs
-``` {.python}
-# DEFAULT
-trainer = Trainer(check_val_every_n_epoch=1)
-```
-
----
-#### Set how much of the validation set to check 
-If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
-
-val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
-
-``` {.python}
-# DEFAULT
-trainer = Trainer(val_percent_check=1.0)
-
-# check 10% only
-trainer = Trainer(val_percent_check=0.1)
-```
-
----
-#### Set how much of the test set to check 
-If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
-
-test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
-
-``` {.python}
-# DEFAULT
-trainer = Trainer(test_percent_check=1.0)
-
-# check 10% only
-trainer = Trainer(test_percent_check=0.1)
-```
-
----
-####  Set validation check frequency within 1 training epoch
-For large datasets it's often desirable to check validation multiple times within a training loop.
-Pass in a float to check that often within 1 training epoch.
-Pass in an int k to check every k training batches. Must use an int if using 
-an IterableDataset.
- 
-``` {.python}
-# DEFAULT
-trainer = Trainer(val_check_interval=0.95)
-
-# check every .25 of an epoch 
-trainer = Trainer(val_check_interval=0.25)
-
-# check every 100 train batches (ie: for IterableDatasets or fixed frequency)
-trainer = Trainer(val_check_interval=100)
-```
-
----
-####  Set the number of validation sanity steps
-Lightning runs a few steps of validation in the beginning of training. This avoids crashing in the validation loop sometime deep into a lengthy training loop.
-``` {.python}
-# DEFAULT
-trainer = Trainer(nb_sanity_val_steps=5)
-```
-
-You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.

docs/Trainer/debugging.md

@@ -1,59 +0,0 @@
-These flags are useful to help debug a model.
-
----
-#### Fast dev run 
-This flag is meant for debugging a full train/val/test loop. It'll activate callbacks, everything but only with 1 training and 1 validation batch.
-Use this to debug a full run of your program quickly
-``` {.python}
-# DEFAULT
-trainer = Trainer(fast_dev_run=False)
-```
-
----
-#### Inspect gradient norms
-Looking at grad norms can help you figure out where training might be going wrong.
-``` {.python}
-# DEFAULT (-1 doesn't track norms)
-trainer = Trainer(track_grad_norm=-1)
-
-# track the LP norm (P=2 here)
-trainer = Trainer(track_grad_norm=2)
-```
-
----
-#### Make model overfit on subset of data
-A useful debugging trick is to make your model overfit a tiny fraction of the data.
-
-setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
-
-``` {.python}
-# DEFAULT don't overfit (ie: normal training)
-trainer = Trainer(overfit_pct=0.0)
-
-# overfit on 1% of data 
-trainer = Trainer(overfit_pct=0.01)
-```
-
----
-#### Print the parameter count by layer
-By default lightning prints a list of parameters *and submodules* when it starts training.
-
-``` {.python}
-# DEFAULT print a full list of all submodules and their parameters.
-trainer = Trainer(weights_summary='full')
-
-# only print the top-level modules (i.e. the children of LightningModule).
-trainer = Trainer(weights_summary='top')
-```
-
----
-#### Print which gradients are nan 
-This option prints a list of tensors with nan gradients.
-``` {.python}
-# DEFAULT
-trainer = Trainer(print_nan_grads=False)
-```
-
----
-#### Log GPU usage
-Lightning automatically logs gpu usage to the test tube logs. It'll only do it at the metric logging interval, so it doesn't slow down training.

docs/Trainer/hooks.md

@@ -1,266 +0,0 @@
-# Hooks
-[[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py)]   
-
-There are cases when you might want to do something different at different parts of the training/validation loop.
-To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
-
-**Contributing** If there's a hook you'd like to add, simply:    
-1. Fork PyTorchLightning.    
-2. Add the hook [here](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py).       
-3. Add the correct place in the [Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py) where it should be called.    
-
----
-#### on_epoch_start
-Called in the training loop at the very beginning of the epoch.   
-```python
-def on_epoch_start(self):
-    # do something when the epoch starts
-```
-
----
-#### on_epoch_end
-Called in the training loop at the very end of the epoch.   
-```python
-def on_epoch_end(self):
-    # do something when the epoch ends 
-```
-
----
-#### on_batch_start
-Called in the training loop before anything happens for that batch.   
-```python
-def on_batch_start(self):
-    # do something when the batch starts
-```
-
----
-#### on_batch_end
-Called in the training loop after the batch.   
-```python
-def on_batch_end(self):
-    # do something when the batch ends 
-```
-
----
-#### on_pre_performance_check
-Called at the very beginning of the validation loop.   
-```python
-def on_pre_performance_check(self):
-    # do something before validation starts 
-```
-
----
-#### on_post_performance_check
-Called at the very end of the validation loop.   
-```python
-def on_post_performance_check(self):
-    # do something before validation end
-```
-
----
-#### optimizer_step 
-Calls .step() and .zero_grad for each optimizer.  
-You can override this method to adjust how you do the optimizer step for each optimizer
-
-Called once per optimizer
-```python
-# DEFAULT
-def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
-    optimizer.step()   
-    optimizer.zero_grad()   
-    
-# Alternating schedule for optimizer steps (ie: GANs)    
-def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
-    # update generator opt every 2 steps
-    if optimizer_i == 0:
-        if batch_nb % 2 == 0 :
-            optimizer.step()
-            optimizer.zero_grad()
-   
-    # update discriminator opt every 4 steps
-    if optimizer_i == 1:
-        if batch_nb % 4 == 0 :
-            optimizer.step()
-            optimizer.zero_grad()    
-    
-    # ...
-    # add as many optimizers as you want 
-```
-
-This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:   
-
-```python
-# learning rate warm-up
-def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
-    # warm up lr
-    if self.trainer.global_step < 500:
-        lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
-        for pg in optimizer.param_groups:
-            pg['lr'] = lr_scale * self.hparams.learning_rate
-    
-    # update params
-    optimizer.step()
-    optimizer.zero_grad() 
-```
-
-
----
-#### on_before_zero_grad
-Called in the training loop after taking an optimizer step and before zeroing grads.
-Good place to inspect weight information with weights updated.
-
-Called once per optimizer
-```python
-def on_before_zero_grad(self, optimizer):
-    # do something with the optimizer or inspect it. 
-```
-
----
-#### backward
-Called to perform backward step.
-Feel free to override as needed.
-
-The loss passed in has already been scaled for accumulated gradients if requested.
-```python
-def backward(self, use_amp, loss, optimizer):
-    """
-    Override backward with your own implementation if you need to
-    :param use_amp: Whether amp was requested or not
-    :param loss: Loss is already scaled by accumulated grads
-    :param optimizer: Current optimizer being used
-    :return:
-    """
-    if use_amp:
-        with amp.scale_loss(loss, optimizer) as scaled_loss:
-            scaled_loss.backward()
-    else:
-        loss.backward()
-```
-
----
-#### on_after_backward
-Called in the training loop after model.backward()
-This is the ideal place to inspect or log gradient information 
-```python
-def on_after_backward(self):
-    # example to inspect gradient information in tensorboard
-    if self.trainer.global_step % 25 == 0:  # don't make the tf file huge
-        params = self.state_dict()
-        for k, v in params.items():
-            grads = v
-            name = k
-            self.logger.experiment.add_histogram(tag=name, values=grads, global_step=self.trainer.global_step)
-```
-
----
-#### tbptt_split_batch
-Called in the training loop after on_batch_start if `truncated_bptt_steps > 0`. Each returned batch split is passed separately to training_step(...).
-
-```python
-def tbptt_split_batch(self, batch, split_size):
-  splits = []
-  for t in range(0, time_dims[0], split_size):
-      batch_split = []
-      for i, x in enumerate(batch):
-          if isinstance(x, torch.Tensor):
-              split_x = x[:, t:t + split_size]
-          elif isinstance(x, collections.Sequence):
-              split_x = [None] * len(x)
-              for batch_idx in range(len(x)):
-                  split_x[batch_idx] = x[batch_idx][t:t + split_size]
-
-          batch_split.append(split_x)
-
-      splits.append(batch_split)
-
-  return splits
-```
-
----
-#### configure_apex
-Overwrite to define your own Apex implementation init.
-
-```python
-def configure_apex(self, amp, model, optimizers, amp_level):
-    """
-    Override to init AMP your own way
-    Must return a model and list of optimizers
-    :param amp:
-    :param model:
-    :param optimizers:
-    :param amp_level:
-    :return: Apex wrapped model and optimizers
-    """
-    model, optimizers = amp.initialize(
-        model, optimizers, opt_level=amp_level,
-    )
-
-    return model, optimizers
-```
-
----
-#### configure_ddp 
-Overwrite to define your own DDP implementation init.
-The only requirement is that:
-1. On a validation batch the call goes to model.validation_step.   
-2. On a training batch the call goes to model.training_step.   
-3. On a testing batch, the call goes to model.test_step
-
-```python
-def configure_ddp(self, model, device_ids):
-    """
-    Override to init DDP in a different way or use your own wrapper.
-    Must return model.
-    :param model:
-    :param device_ids:
-    :return: DDP wrapped model
-    """
-    # Lightning DDP simply routes to test_step, val_step, etc...
-    model = LightningDistributedDataParallel(
-        model,
-        device_ids=device_ids,
-        find_unused_parameters=True
-    )
-    return model
-```
-
----   
-#### init_ddp_connection   
-Override to init DDP in your own way.   
-
-```python
-def init_ddp_connection(self):
-    """
-    Connect all procs in the world using the env:// init
-    Use the first node as the root address
-    """
-
-    # use slurm job id for the port number
-    # guarantees unique ports across jobs from same grid search
-    try:
-        # use the last 4 numbers in the job id as the id
-        default_port = os.environ['SLURM_JOB_ID']
-        default_port = default_port[-4:]
-
-        # all ports should be in the 10k+ range
-        default_port = int(default_port) + 15000
-
-    except Exception as e:
-        default_port = 12910
-
-    # if user gave a port number, use that one instead
-    try:
-        default_port = os.environ['MASTER_PORT']
-    except Exception:
-        os.environ['MASTER_PORT'] = str(default_port)
-
-    # figure out the root node addr
-    try:
-        root_node = os.environ['SLURM_NODELIST'].split(' ')[0]
-    except Exception:
-        root_node = '127.0.0.2'
-
-    root_node = self.trainer.resolve_root_node_address(root_node)
-    os.environ['MASTER_ADDR'] = root_node
-    dist.init_process_group('nccl', rank=self.proc_rank, world_size=self.world_size)
-```

docs/Trainer/index.md

@@ -1,90 +0,0 @@
-# Trainer
-[[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py)]
-
-The lightning trainer abstracts best practices for running a training, val, test routine. It calls parts of your model when it wants to hand over full control and otherwise makes training assumptions which are now standard practice in AI research.
-
-This is the basic use of the trainer:
-
-``` {.python}
-from pytorch_lightning import Trainer
-
-model = LightningTemplate()
-
-trainer = Trainer()
-trainer.fit(model)
-```
-
-But of course the fun is in all the advanced things it can do:
-
-
-**Checkpointing**    
-
-- [Checkpoint callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)    
-- [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
-- [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
-- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
-
-**Computing cluster (SLURM)**    
-
-- [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
-- [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
-
-**Debugging**  
-
-- [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
-- [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
-- [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
-- [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
-- [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
-- [Print which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
-- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
-
-
-**Distributed training**    
-
-- [Implement Your Own Distributed (DDP) training](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#init_ddp_connection)
-- [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
-- [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
-- [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
-- [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
-- [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
-
-
-**Experiment Logging**   
-
-- [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
-- [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
-- [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
-- [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
-- [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
-- [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
-- [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
-
-**Training loop**    
-
-- [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)    
-- [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
-- [Early stopping callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)    
-- [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
-- [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
-- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
-- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
-- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
-- [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
-- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
-- [Packed sequences](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#packed-sequences-as-inputs)
-- [Truncated Backpropagation Through Time](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#truncated-backpropagtion-through-time)
-
-**Validation loop**    
-
-- [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
-- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
-- [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
-- [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
-- [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
-- [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
-
-
-**Testing loop**    
-
-- [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  

docs/examples/Examples.md

@@ -1,131 +0,0 @@
-### Template model definition
-In 99% of cases you want to just copy [one of the examples](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples) to start a new lightningModule and change the core of what your model is actually trying to do.
-
-```bash
-# get a copy of the module template
-wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/pl_examples/new_project_templates/lightning_module_template.py
-```
-
----    
-### Trainer Example 
-
-** \_\_main__ function**    
-
-Normally, we want to let the \_\_main__ function start the training.
-Inside the main we parse training arguments with whatever hyperparameters we want. Your LightningModule will have a 
-chance to add hyperparameters.   
-
-```{.python}
-from test_tube import HyperOptArgumentParser
-
-if __name__ == '__main__':
-
-    # use default args given by lightning
-    root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
-    parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
-    add_default_args(parent_parser, root_dir)
-
-    # allow model to overwrite or extend args
-    parser = ExampleModel.add_model_specific_args(parent_parser)
-    hyperparams = parser.parse_args()
-
-    # train model
-    main(hyperparams)
-```
-**Main Function**      
-
-The main function is your entry into the program. This is where you init your model, checkpoint directory, and launch the training.
-The main function should have 3 arguments:   
-- hparams: a configuration of hyperparameters.    
-- slurm_manager: Slurm cluster manager object (can be None)
-- dict: for you to return any values you want (useful in meta-learning, otherwise set to _)    
-
-```python
-def main(hparams, cluster, results_dict):
-    """
-    Main training routine specific for this project
-    :param hparams:
-    :return:
-    """
-    # build model
-    model = MyLightningModule(hparams)
-
-    # configure trainer
-    trainer = Trainer()
-
-    # train model
-    trainer.fit(model)
-```
-
-
-The __main__ function will start training on your **main** function. If you use the HyperParameterOptimizer
-in hyper parameter optimization mode, this main function will get one set of hyperparameters. If you use it as a simple
-argument parser you get the default arguments in the argument parser.
-
-So, calling main(hyperparams) runs the model with the default argparse arguments.       
-```{.python}
-main(hyperparams)
-```
-
----
-#### CPU hyperparameter search      
-
-```{.python}
-# run a grid search over 20 hyperparameter combinations.
-hyperparams.optimize_parallel_cpu(
-    main_local,
-    nb_trials=20,
-    nb_workers=1
-)
-```
-
----
-#### Hyperparameter search on a single or multiple GPUs       
-```{.python}
-# run a grid search over 20 hyperparameter combinations.
-hyperparams.optimize_parallel_gpu(
-    main_local,
-    nb_trials=20,
-    nb_workers=1,
-    gpus=[0,1,2,3]
-)
-```
-
----
-#### Hyperparameter search on a SLURM HPC cluster   
-```{.python}    
-def optimize_on_cluster(hyperparams):
-    # enable cluster training
-    cluster = SlurmCluster(
-        hyperparam_optimizer=hyperparams,
-        log_path=hyperparams.tt_save_path,
-        test_tube_exp_name=hyperparams.tt_name
-    )
-
-    # email for cluster coms
-    cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
-
-    # configure cluster
-    cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
-    cluster.job_time = '48:00:00'
-    cluster.gpu_type = '1080ti'
-    cluster.memory_mb_per_node = 48000
-
-    # any modules for code to run in env
-    cluster.add_command('source activate pytorch_lightning')
-
-    # name of exp
-    job_display_name = hyperparams.tt_name.split('_')[0]
-    job_display_name = job_display_name[0:3]
-
-    # run hopt
-    logging.info('submitting jobs...')
-    cluster.optimize_parallel_cluster_gpu(
-        main,
-        nb_trials=hyperparams.nb_hopt_trials,
-        job_name=job_display_name
-    )
-
-# run cluster hyperparameter search    
-optimize_on_cluster(hyperparams)
-```

docs/index.md

@@ -1,143 +0,0 @@
-###### New project Quick Start    
-To start a new project define two files, a LightningModule and a Trainer file.    
-To illustrate Lightning power and simplicity, here's an example of a typical research flow.    
-
-###### Case 1: BERT    
-Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
-You would define a single LightningModule and use flags to switch between your different ideas.   
-```python
-class BERT(pl.LightningModule):
-    def __init__(self, model_name, task):
-        self.task = task
-    
-        if model_name == 'transformer':
-            self.net = Transformer()
-        elif model_name == 'my_cool_version':
-            self.net = MyCoolVersion()
-            
-    def training_step(self, batch, batch_nb):
-        if self.task == 'standard_bert':
-            # do standard bert training with self.net...
-            # return loss
-            
-        if self.task == 'my_cool_task':
-            # do my own version with self.net
-            # return loss
-```   
-
-###### Case 2: COOLER NOT BERT    
-But if you wanted to try something **completely** different, you'd define a new module for that.    
-```python
-
-class CoolerNotBERT(pl.LightningModule):
-    def __init__(self):
-        self.net = ...
-        
-    def training_step(self, batch, batch_nb):
-        # do some other cool task
-        # return loss   
-```   
-
-###### Rapid research flow    
-Then you could do rapid research by switching between these two and using the same trainer.   
-```python
-
-if use_bert:
-    model = BERT()
-else:
-    model = CoolerNotBERT()
-    
-trainer = Trainer(gpus=4, use_amp=True)
-trainer.fit(model)
-```
-
-Notice a few things about this flow:   
-1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
-2. You get free GPU and 16-bit support without writing any of that code in your model.   
-3. You also get all of the capabilities below (without coding or testing yourself).     
-
----    
-###### Templates 
-1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
-2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
-    - [Basic CPU, GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/basic_examples)
-    - [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/multi_node_examples)
-
-###### Docs shortcuts
-- [LightningModule](LightningModule/RequiredTrainerInterface/)  
-- [Trainer](Trainer/)  
-
-###### Quick start examples 
-- [CPU example](examples/Examples/#cpu-hyperparameter-search)   
-- [Hyperparameter search on single GPU](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)    
-- [Hyperparameter search on multiple GPUs on same node](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)  
-- [Hyperparameter search on a SLURM HPC cluster](examples/Examples/#Hyperparameter search on a SLURM HPC cluster)      
-
-
-###### Checkpointing    
-
-- [Checkpoint callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)    
-- [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
-- [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
-- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
-
-###### Computing cluster (SLURM)    
-
-- [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
-- [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
-
-###### Debugging  
-
-- [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
-- [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
-- [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
-- [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
-- [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
-- [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
-- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
-
-
-###### Distributed training    
-
-- [Implement Your Own Distributed (DDP) training](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#init_ddp_connection)
-- [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
-- [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
-- [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
-- [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
-- [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
-
-
-###### Experiment Logging   
-
-- [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
-- [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
-- [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
-- [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
-- [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
-- [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
-- [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
-
-###### Training loop    
-
-- [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
-- [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
-- [Early stopping callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)    
-- [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
-- [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
-- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
-- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
-- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
-- [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
-- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
-
-###### Validation loop    
-
-- [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
-- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
-- [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
-- [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
-- [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
-- [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
-
-###### Testing loop  
-- [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+
+:end
+popd

docs/requirements.txt

@@ -1,2 +1,9 @@
-mkdocs-material==4.4.0
-mkdocs==1.0.4
+sphinx>=1.8.3
+recommonmark  # fails with badges
+m2r  # fails with multi-line text
+nbsphinx
+pandoc
+docutils
+git+https://github.com/Borda/lightning_sphinx_theme.git
+sphinxcontrib-fulltoc
+sphinxcontrib-mockautodoc

docs/source/_templates/theme_variables.jinja

@@ -0,0 +1,17 @@
+{%- set external_urls = {
+  'github': 'https://github.com/williamFalcon/pytorch-lightning',
+  'github_issues': 'https://github.com/williamFalcon/pytorch-lightning/issues',
+  'contributing': 'https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md',
+  'docs': 'https://williamfalcon.github.io/pytorch-lightning',
+  'twitter': 'https://twitter.com/PyTorchLightnin',
+  'discuss': 'https://discuss.pytorch.org',
+  'tutorials': 'https://williamfalcon.github.io/pytorch-lightning/',
+  'previous_pytorch_versions': 'https://williamfalcon.github.io/pytorch-lightning/',
+  'home': 'https://williamfalcon.github.io/pytorch-lightning/',
+  'get_started': 'https://williamfalcon.github.io/pytorch-lightning/',
+  'features': 'https://williamfalcon.github.io/pytorch-lightning/',
+  'blog': 'https://williamfalcon.github.io/pytorch-lightning/',
+  'resources': 'https://williamfalcon.github.io/pytorch-lightning/',
+  'support': 'https://williamfalcon.github.io/pytorch-lightning/',
+}
+-%}

docs/source/conf.py

@@ -0,0 +1,357 @@
+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+
+import os
+import sys
+import glob
+import shutil
+import inspect
+
+# import m2r
+import builtins
+import pt_lightning_sphinx_theme
+
+PATH_HERE = os.path.abspath(os.path.dirname(__file__))
+PATH_ROOT = os.path.join(PATH_HERE, '..', '..')
+sys.path.insert(0, os.path.abspath(PATH_ROOT))
+
+builtins.__LIGHTNING_SETUP__ = True
+
+import pytorch_lightning  # noqa: E402
+
+# -- Project documents -------------------------------------------------------
+
+# # export the documentation
+# with open('intro.rst', 'w') as fp:
+#     intro = pytorch_lightning.__doc__.replace(os.linesep + ' ', '')
+#     fp.write(m2r.convert(intro))
+#     # fp.write(pytorch_lightning.__doc__)
+
+# # export the READme
+# with open(os.path.join(PATH_ROOT, 'README.md'), 'r') as fp:
+#     readme = fp.read()
+# # replace all paths to relative
+# for ndir in (os.path.basename(p) for p in glob.glob(os.path.join(PATH_ROOT, '*'))
+#              if os.path.isdir(p)):
+#     readme = readme.replace('](%s/' % ndir, '](%s/%s/' % (PATH_ROOT, ndir))
+# with open('readme.md', 'w') as fp:
+#     fp.write(readme)
+
+for md in glob.glob(os.path.join(PATH_ROOT, '.github', '*.md')):
+    shutil.copy(md, os.path.join(PATH_HERE, os.path.basename(md)))
+
+# -- Project information -----------------------------------------------------
+
+project = 'PyTorch-Lightning'
+copyright = pytorch_lightning.__copyright__
+author = pytorch_lightning.__author__
+
+# The short X.Y version
+version = pytorch_lightning.__version__
+# The full version, including alpha/beta/rc tags
+release = pytorch_lightning.__version__
+
+
+# -- General configuration ---------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+
+needs_sphinx = '1.4'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinxcontrib.mockautodoc',
+    # 'sphinxcontrib.fulltoc',  # breaks pytorch-theme with unexpected kw argument 'titles_only'
+    'sphinx.ext.doctest',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.todo',
+    'sphinx.ext.coverage',
+    'sphinx.ext.linkcode',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.napoleon',
+    'recommonmark',
+    # 'm2r',
+    'nbsphinx',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# https://berkeley-stat159-f17.github.io/stat159-f17/lectures/14-sphinx..html#conf.py-(cont.)
+# https://stackoverflow.com/questions/38526888/embed-ipython-notebook-in-sphinx-document
+# I execute the notebooks manually in advance. If notebooks test the code,
+# they should be run at build time.
+nbsphinx_execute = 'never'
+nbsphinx_allow_errors = True
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+# source_suffix = ['.rst', '.md', '.ipynb']
+source_suffix = {
+    '.rst': 'restructuredtext',
+    '.txt': 'markdown',
+    '.md': 'markdown',
+    '.ipynb': 'nbsphinx',
+}
+
+# The master toctree document.
+master_doc = 'index'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['*.test_*']
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = None
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+# http://www.sphinx-doc.org/en/master/usage/theming.html#builtin-themes
+# html_theme = 'bizstyle'
+# https://sphinx-themes.org
+html_theme = 'pt_lightning_sphinx_theme'
+html_theme_path = [pt_lightning_sphinx_theme.get_html_theme_path()]
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+
+html_theme_options = {
+    'pytorch_project': pytorch_lightning.__homepage__,
+    'canonical_url': pytorch_lightning.__homepage__,
+    'collapse_navigation': False,
+    'display_version': True,
+    'logo_only': False,
+}
+
+html_logo = '_static/images/lightning_logo_small.png'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+#
+# html_sidebars = {}
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = project + '-doc'
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, project + '.tex', project + ' Documentation', author, 'manual'),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, project, project + ' Documentation', [author], 1)
+]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, project, project + ' Documentation', author, project,
+     'One line description of project.', 'Miscellaneous'),
+]
+
+
+# -- Options for Epub output -------------------------------------------------
+
+# Bibliographic Dublin Core info.
+epub_title = project
+
+# The unique identifier of the text. This can be a ISBN number
+# or the project homepage.
+#
+# epub_identifier = ''
+
+# A unique identification for the text.
+#
+# epub_uid = ''
+
+# A list of files that should not be packed into the epub file.
+epub_exclude_files = ['search.html']
+
+
+# -- Extension configuration -------------------------------------------------
+
+# -- Options for intersphinx extension ---------------------------------------
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {'https://docs.python.org/': None}
+
+# -- Options for todo extension ----------------------------------------------
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+
+# https://github.com/rtfd/readthedocs.org/issues/1139
+# I use sphinx-apidoc to auto-generate API documentation for my project.
+# Right now I have to commit these auto-generated files to my repository
+# so that RTD can build them into HTML docs. It'd be cool if RTD could run
+# sphinx-apidoc for me, since it's easy to forget to regen API docs
+# and commit them to my repo after making changes to my code.
+
+PACKAGES = [
+    pytorch_lightning.__name__,
+    'pl_examples',
+]
+
+
+def run_apidoc(_):
+    for pkg in PACKAGES:
+        argv = ['-e', '-o', PATH_HERE, os.path.join(PATH_HERE, PATH_ROOT, pkg),
+                '**/test_*', '--force', '--private', '--module-first']
+        try:
+            # Sphinx 1.7+
+            from sphinx.ext import apidoc
+            apidoc.main(argv)
+        except ImportError:
+            # Sphinx 1.6 (and earlier)
+            from sphinx import apidoc
+            argv.insert(0, apidoc.__file__)
+            apidoc.main(argv)
+
+
+def setup(app):
+    app.connect('builder-inited', run_apidoc)
+
+
+# copy all notebooks to local folder
+path_nbs = os.path.join(PATH_HERE, 'notebooks')
+if not os.path.isdir(path_nbs):
+    os.mkdir(path_nbs)
+for path_ipynb in glob.glob(os.path.join(PATH_ROOT, 'notebooks', '*.ipynb')):
+    path_ipynb2 = os.path.join(path_nbs, os.path.basename(path_ipynb))
+    shutil.copy(path_ipynb, path_ipynb2)
+
+# Ignoring Third-party packages
+# https://stackoverflow.com/questions/15889621/sphinx-how-to-exclude-imports-in-automodule
+
+MOCK_REQUIRE_PACKAGES = []
+with open(os.path.join(PATH_ROOT, 'requirements.txt'), 'r') as fp:
+    for ln in fp.readlines():
+        found = [ln.index(ch) for ch in list(',=<>#') if ch in ln]
+        pkg = ln[:min(found)] if found else ln
+        if pkg.rstrip():
+            MOCK_REQUIRE_PACKAGES.append(pkg.rstrip())
+
+# TODO: better parse from package since the import name and package name may differ
+MOCK_MANUAL_PACKAGES = ['torch', 'torchvision', 'sklearn', 'test_tube', 'mlflow', 'comet_ml']
+autodoc_mock_imports = MOCK_REQUIRE_PACKAGES + MOCK_MANUAL_PACKAGES
+# for mod_name in MOCK_REQUIRE_PACKAGES:
+#     sys.modules[mod_name] = mock.Mock()
+
+
+# Options for the linkcode extension
+# ----------------------------------
+github_user = 'williamFalcon'
+github_repo = project
+
+
+# Resolve function
+# This function is used to populate the (source) links in the API
+def linkcode_resolve(domain, info):
+    def find_source():
+        # try to find the file and line number, based on code from numpy:
+        # https://github.com/numpy/numpy/blob/master/doc/source/conf.py#L286
+        obj = sys.modules[info['module']]
+        for part in info['fullname'].split('.'):
+            obj = getattr(obj, part)
+        fname = inspect.getsourcefile(obj)
+        # https://github.com/rtfd/readthedocs.org/issues/5735
+        if any([s in fname for s in ('readthedocs', 'checkouts')]):
+            # /home/docs/checkouts/readthedocs.org/user_builds/pytorch_lightning/checkouts/
+            #  devel/pytorch_lightning/utilities/cls_experiment.py#L26-L176
+            path_top = os.path.abspath(os.path.join('..', '..', '..'))
+            fname = os.path.relpath(fname, start=path_top)
+        else:
+            # Local build, imitate master
+            fname = 'master/' + os.path.relpath(fname, start=os.path.abspath('..'))
+        source, lineno = inspect.getsourcelines(obj)
+        return fname, lineno, lineno + len(source) - 1
+
+    if domain != 'py' or not info['module']:
+        return None
+    try:
+        filename = '%s#L%d-L%d' % find_source()
+    except Exception:
+        filename = info['module'].replace('.', '/') + '.py'
+    # import subprocess
+    # tag = subprocess.Popen(['git', 'rev-parse', 'HEAD'], stdout=subprocess.PIPE,
+    #                        universal_newlines=True).communicate()[0][:-1]
+    return "https://github.com/%s/%s/blob/%s" \
+           % (github_user, github_repo, filename)
+
+
+autodoc_member_order = 'groupwise'
+autoclass_content = 'both'
+autodoc_default_flags = [
+    'members', 'undoc-members', 'show-inheritance', 'private-members',
+    # 'special-members', 'inherited-members'
+]

docs/source/documentation.rst

@@ -0,0 +1,8 @@
+Documentation
+=============
+
+
+.. toctree::
+   :maxdepth: 4
+
+   pytorch_lightning

docs/source/examples.rst

@@ -0,0 +1,8 @@
+Examples & Tutorials
+====================
+
+
+.. toctree::
+   :maxdepth: 3
+
+   pl_examples

docs/source/index.rst

@@ -0,0 +1,40 @@
+.. PyTorch-Lightning documentation master file, created by
+   sphinx-quickstart on Fri Nov 15 07:48:22 2019.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to PyTorch-Lightning!
+=============================
+
+.. toctree::
+   :maxdepth: 4
+   :name: start
+   :caption: Quick Start
+
+   new-project
+   examples
+
+.. toctree::
+   :maxdepth: 4
+   :name: docs
+   :caption: Docs
+
+   documentation
+
+.. toctree::
+   :maxdepth: 1
+   :name: community
+   :caption: Community
+
+   CODE_OF_CONDUCT.md
+   CONTRIBUTING.md
+   BECOMING_A_CORE_CONTRIBUTOR.md
+
+
+Indices and tables
+------------------
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+

docs/source/new-project.rst

@@ -0,0 +1,71 @@
+Quick Start
+===========
+To start a new project define two files, a LightningModule and a Trainer file.    
+To illustrate Lightning power and simplicity, here's an example of a typical research flow.    
+
+Case 1: BERT
+------------
+
+Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
+You would define a single LightningModule and use flags to switch between your different ideas.
+
+.. code-block:: python
+
+    class BERT(pl.LightningModule):
+        def __init__(self, model_name, task):
+            self.task = task
+
+            if model_name == 'transformer':
+                self.net = Transformer()
+            elif model_name == 'my_cool_version':
+                self.net = MyCoolVersion()
+
+        def training_step(self, batch, batch_idx):
+            if self.task == 'standard_bert':
+                # do standard bert training with self.net...
+                # return loss
+
+            if self.task == 'my_cool_task':
+                # do my own version with self.net
+                # return loss
+
+
+Case 2: COOLER NOT BERT
+-----------------------
+
+But if you wanted to try something **completely** different, you'd define a new module for that.
+
+
+.. code-block:: python
+
+    class CoolerNotBERT(pl.LightningModule):
+        def __init__(self):
+            self.net = ...
+
+        def training_step(self, batch, batch_idx):
+            # do some other cool task
+            # return loss
+
+
+Rapid research flow
+-------------------
+
+Then you could do rapid research by switching between these two and using the same trainer.
+
+
+.. code-block:: python
+
+    if use_bert:
+        model = BERT()
+    else:
+        model = CoolerNotBERT()
+
+    trainer = Trainer(gpus=4, use_amp=True)
+    trainer.fit(model)
+
+
+**Notice a few things about this flow:**
+
+1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
+2. You get free GPU and 16-bit support without writing any of that code in your model.   
+3. You also get all of the capabilities below (without coding or testing yourself).     

mkdocs.yml

@@ -1,16 +0,0 @@
-site_name: PyTorch lightning Documentation
-theme:
-  name: 'material'
-docs_dir: docs
-repo_name: 'williamFalcon/pytorch-lightning'
-repo_url: https://github.com/williamFalcon/pytorch-lightning
-site_dir: 'site'
-site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
-
-dev_addr: '0.0.0.0:8000'
-#google_analytics: ['UA-aasd', 'sitename']
-
-markdown_extensions:
-  - codehilite:
-      guess_lang: false
-      linenums: true

pl_examples/__init__.py

@@ -1,3 +1,144 @@
+"""
+Template model definition
+-------------------------
+
+In 99% of cases you want to just copy `one of the examples
+ <https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples>`_
+ to start a new lightningModule and change the core of what your model is actually trying to do.
+
+.. code-block:: bash
+
+    # get a copy of the module template
+    wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/pl_examples/new_project_templates/lightning_module_template.py  # noqa: E501
+
+
+Trainer Example
+---------------
+
+**`__main__` function**
+
+Normally, we want to let the `__main__` function start the training.
+ Inside the main we parse training arguments with whatever hyperparameters we want.
+ Your LightningModule will have a chance to add hyperparameters.
+
+.. code-block:: python
+
+    from test_tube import HyperOptArgumentParser
+
+    if __name__ == '__main__':
+
+        # use default args given by lightning
+        root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
+        parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
+        add_default_args(parent_parser, root_dir)
+
+        # allow model to overwrite or extend args
+        parser = ExampleModel.add_model_specific_args(parent_parser)
+        hyperparams = parser.parse_args()
+
+        # train model
+        main(hyperparams)
+
+**Main Function**
+
+The main function is your entry into the program. This is where you init your model, checkpoint directory,
+ and launch the training. The main function should have 3 arguments:
+- hparams: a configuration of hyperparameters.
+- slurm_manager: Slurm cluster manager object (can be None)
+- dict: for you to return any values you want (useful in meta-learning, otherwise set to)
+
+.. code-block:: python
+
+    def main(hparams, cluster, results_dict):
+        # build model
+        model = MyLightningModule(hparams)
+
+        # configure trainer
+        trainer = Trainer()
+
+        # train model
+        trainer.fit(model)
+
+
+The `__main__` function will start training on your **main** function.
+ If you use the HyperParameterOptimizer in hyper parameter optimization mode,
+ this main function will get one set of hyperparameters. If you use it as a simple
+ argument parser you get the default arguments in the argument parser.
+
+So, calling main(hyperparams) runs the model with the default argparse arguments.::
+
+    main(hyperparams)
+
+
+CPU hyperparameter search
+-------------------------
+
+.. code-block:: python
+
+    # run a grid search over 20 hyperparameter combinations.
+    hyperparams.optimize_parallel_cpu(
+        main_local,
+        nb_trials=20,
+        nb_workers=1
+    )
+
+
+Hyperparameter search on a single or multiple GPUs
+--------------------------------------------------
+
+.. code-block:: python
+
+    # run a grid search over 20 hyperparameter combinations.
+    hyperparams.optimize_parallel_gpu(
+        main_local,
+        nb_trials=20,
+        nb_workers=1,
+        gpus=[0,1,2,3]
+    )
+
+
+Hyperparameter search on a SLURM HPC cluster
+--------------------------------------------
+
+.. code-block:: python
+
+    def optimize_on_cluster(hyperparams):
+        # enable cluster training
+        cluster = SlurmCluster(
+            hyperparam_optimizer=hyperparams,
+            log_path=hyperparams.tt_save_path,
+            test_tube_exp_name=hyperparams.tt_name
+        )
+
+        # email for cluster coms
+        cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
+
+        # configure cluster
+        cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
+        cluster.job_time = '48:00:00'
+        cluster.gpu_type = '1080ti'
+        cluster.memory_mb_per_node = 48000
+
+        # any modules for code to run in env
+        cluster.add_command('source activate pytorch_lightning')
+
+        # name of exp
+        job_display_name = hyperparams.tt_name.split('_')[0]
+        job_display_name = job_display_name[0:3]
+
+        # run hopt
+        logging.info('submitting jobs...')
+        cluster.optimize_parallel_cluster_gpu(
+            main,
+            nb_trials=hyperparams.nb_hopt_trials,
+            job_name=job_display_name
+        )
+
+    # run cluster hyperparameter search
+    optimize_on_cluster(hyperparams)
+
+"""
+
 from .basic_examples.lightning_module_template import LightningTemplateModel
 
 __all__ = [

pl_examples/basic_examples/lightning_module_template.py

@@ -16,7 +16,7 @@ from torch.utils.data.distributed import DistributedSampler
 from torchvision.datasets import MNIST
 
 import pytorch_lightning as pl
-from pytorch_lightning.root_module.root_module import LightningModule
+from pytorch_lightning.core.lightning import LightningModule
 
 
 class LightningTemplateModel(LightningModule):

pl_examples/domain_templates/gan.py

@@ -90,12 +90,12 @@ class GAN(pl.LightningModule):
     def adversarial_loss(self, y_hat, y):
         return F.binary_cross_entropy(y_hat, y)
 
-    def training_step(self, batch, batch_nb, optimizer_i):
+    def training_step(self, batch, batch_idx, optimizer_idx):
         imgs, _ = batch
         self.last_imgs = imgs
 
         # train generator
-        if optimizer_i == 0:
+        if optimizer_idx == 0:
             # sample noise
             z = torch.randn(imgs.shape[0], self.hparams.latent_dim)
 
@@ -112,7 +112,10 @@ class GAN(pl.LightningModule):
             # self.logger.experiment.add_image('generated_images', grid, 0)
 
             # ground truth result (ie: all fake)
+            # put on GPU because we created this tensor inside training_loop
             valid = torch.ones(imgs.size(0), 1)
+            if self.on_gpu:
+                valid = valid.cuda(imgs.device.index)
 
             # adversarial loss is binary cross-entropy
             g_loss = self.adversarial_loss(self.discriminator(self.generated_imgs), valid)
@@ -125,15 +128,21 @@ class GAN(pl.LightningModule):
             return output
 
         # train discriminator
-        if optimizer_i == 1:
+        if optimizer_idx == 1:
             # Measure discriminator's ability to classify real from generated samples
 
             # how well can it label as real?
             valid = torch.ones(imgs.size(0), 1)
+            if self.on_gpu:
+                valid = valid.cuda(imgs.device.index)
+
             real_loss = self.adversarial_loss(self.discriminator(imgs), valid)
 
             # how well can it label as fake?
             fake = torch.zeros(imgs.size(0), 1)
+            if self.on_gpu:
+                fake = fake.cuda(imgs.device.index)
+
             fake_loss = self.adversarial_loss(
                 self.discriminator(self.generated_imgs.detach()), fake)
 

pl_examples/full_examples/imagenet/imagenet_example.py

@@ -234,7 +234,7 @@ def main(hparams):
     trainer = pl.Trainer(
         default_save_path=hparams.save_path,
         gpus=hparams.gpus,
-        max_nb_epochs=hparams.epochs,
+        max_epochs=hparams.epochs,
         distributed_backend=hparams.distributed_backend,
         use_amp=hparams.use_16bit
     )

pl_examples/multi_node_examples/multi_node_ddp2_demo.py

@@ -31,7 +31,7 @@ def main(hparams):
     # ------------------------
     trainer = Trainer(
         gpus=2,
-        nb_gpu_nodes=2,
+        num_nodes=2,
         distributed_backend='ddp2'
     )
 

pl_examples/multi_node_examples/multi_node_ddp_demo.py

@@ -31,7 +31,7 @@ def main(hparams):
     # ------------------------
     trainer = Trainer(
         gpus=2,
-        nb_gpu_nodes=2,
+        num_nodes=2,
         distributed_backend='ddp'
     )
 

pytorch_lightning/__init__.py

@@ -1,9 +1,10 @@
 """Package info"""
 
-__version__ = '0.5.3.2'
-__author__ = ' William Falcon et al.'
+__version__ = '0.6.0'
+__author__ = 'William Falcon et al.'
 __author_email__ = 'waf2107@columbia.edu'
 __license__ = 'Apache-2.0'
+__copyright__ = 'Copyright (c) 2018-2019, %s.' % __author__
 __homepage__ = 'https://github.com/williamFalcon/pytorch-lightning'
 # this has to be simple string, see: https://github.com/pypa/twine/issues/522
 __docs__ = "PyTorch Lightning is the lightweight PyTorch wrapper for ML researchers." \
@@ -25,11 +26,13 @@ if __LIGHTNING_SETUP__:
     # process, as it may not be compiled yet
 else:
     from .trainer.trainer import Trainer
-    from .root_module.root_module import LightningModule
-    from .root_module.decorators import data_loader
+    from .core.lightning import LightningModule
+    from .core.decorators import data_loader
+    import logging
 
     __all__ = [
         'Trainer',
         'LightningModule',
         'data_loader',
     ]
+    logging.basicConfig(level=logging.INFO)

pytorch_lightning/callbacks/pt_callbacks.py

@@ -4,29 +4,30 @@ import logging
 import warnings
 import numpy as np
 
-from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel
+from pytorch_lightning.overrides.data_parallel import LightningDistributedDataParallel
 
 
 class Callback(object):
     """Abstract base class used to build new callbacks.
+
     # Properties
-        params: dict. Training parameters
+        * params: dict. Training parameters
             (eg. verbosity, batch size, number of epochs...).
             Reference of the model being trained.
-    The `logs` dictionary that callback methods
-    take as argument will contain keys for quantities relevant to
-    the current batch or epoch.
-    Currently, the `.fit()` method of the `Sequential` model class
-    will include the following quantities in the `logs` that
-    it passes to its callbacks:
-        on_epoch_end: logs include `acc` and `loss`, and
+
+    The `logs` dictionary that callback methods take as argument will contain keys
+     for quantities relevant to the current batch or epoch.
+    Currently, the `.fit()` method of the `Sequential` model class will include the following
+     quantities in the `logs` that it passes to its callbacks:
+    * on_epoch_end: logs include `acc` and `loss`, and
             optionally include `val_loss`
             (if validation is enabled in `fit`), and `val_acc`
             (if validation and accuracy monitoring are enabled).
-        on_batch_begin: logs include `size`,
+    * on_batch_begin: logs include `size`,
             the number of samples in the current batch.
-        on_batch_end: logs include `loss`, and optionally `acc`
+    * on_batch_end: logs include `loss`, and optionally `acc`
             (if accuracy monitoring is enabled).
+
     """
 
     def __init__(self):
@@ -62,6 +63,7 @@ class Callback(object):
 
 class EarlyStopping(Callback):
     """Stop training when a monitored quantity has stopped improving.
+
     # Arguments
         monitor: quantity to be monitored.
         min_delta: minimum change in the monitored quantity
@@ -78,6 +80,7 @@ class EarlyStopping(Callback):
             monitored has stopped increasing; in `auto`
             mode, the direction is automatically inferred
             from the name of the monitored quantity.
+
     """
 
     def __init__(self, monitor='val_loss',
@@ -148,21 +151,29 @@ class EarlyStopping(Callback):
 
 class ModelCheckpoint(Callback):
     """Save the model after every epoch.
-    `filepath` can contain named formatting options,
+
+    The `filepath` can contain named formatting options,
     which will be filled the value of `epoch` and
     keys in `logs` (passed in `on_epoch_end`).
     For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`,
     then the model checkpoints will be saved with the epoch number and
     the validation loss in the filename.
+
     # Arguments
         filepath: string, path to save the model file.
         monitor: quantity to monitor.
         verbose: verbosity mode, 0 or 1.
-        save_best_only: if `save_best_only=True`,
-            the latest best model according to
-            the quantity monitored will not be overwritten.
+        save_top_k: if `save_top_k == k`,
+            the best k models according to
+            the quantity monitored will be saved.
+            if `save_top_k == 0`, no models are saved.
+            if `save_top_k == -1`, all models are saved.
+            Please note that the monitors are checked every `period` epochs.
+            if `save_top_k >= 2` and the callback is called multiple
+            times inside an epoch, the name of the saved file will be
+            appended with a version count starting with `v0`.
         mode: one of {auto, min, max}.
-            If `save_best_only=True`, the decision
+            If `save_top_k != 0`, the decision
             to overwrite the current save file is made
             based on either the maximization or the
             minimization of the monitored quantity. For `val_acc`,
@@ -173,30 +184,36 @@ class ModelCheckpoint(Callback):
             saved (`model.save_weights(filepath)`), else the full model
             is saved (`model.save(filepath)`).
         period: Interval (number of epochs) between checkpoints.
+
     """
 
     def __init__(self, filepath, monitor='val_loss', verbose=0,
-                 save_best_only=True, save_weights_only=False,
+                 save_top_k=1, save_weights_only=False,
                  mode='auto', period=1, prefix=''):
         super(ModelCheckpoint, self).__init__()
         if (
-            save_best_only and
+            save_top_k and
             os.path.isdir(filepath) and
             len(os.listdir(filepath)) > 0
         ):
             warnings.warn(
-                f"Checkpoint directory {filepath} exists and is not empty with save_best_only=True."
+                f"Checkpoint directory {filepath} exists and is not empty with save_top_k != 0."
                 "All files in this directory will be deleted when a checkpoint is saved!"
             )
 
         self.monitor = monitor
         self.verbose = verbose
         self.filepath = filepath
-        self.save_best_only = save_best_only
+        os.makedirs(filepath, exist_ok=True)
+        self.save_top_k = save_top_k
         self.save_weights_only = save_weights_only
         self.period = period
-        self.epochs_since_last_save = 0
+        self.epochs_since_last_check = 0
         self.prefix = prefix
+        self.best_k_models = {}
+        # {filename: monitor}
+        self.kth_best_model = ''
+        self.best = 0
 
         if mode not in ['auto', 'min', 'max']:
             warnings.warn(
@@ -206,72 +223,118 @@ class ModelCheckpoint(Callback):
 
         if mode == 'min':
             self.monitor_op = np.less
-            self.best = np.Inf
+            self.kth_value = np.Inf
+            self.mode = 'min'
         elif mode == 'max':
             self.monitor_op = np.greater
-            self.best = -np.Inf
+            self.kth_value = -np.Inf
+            self.mode = 'max'
         else:
             if 'acc' in self.monitor or self.monitor.startswith('fmeasure'):
                 self.monitor_op = np.greater
-                self.best = -np.Inf
+                self.kth_value = -np.Inf
+                self.mode = 'max'
             else:
                 self.monitor_op = np.less
-                self.best = np.Inf
+                self.kth_value = np.Inf
+                self.mode = 'min'
 
-    def save_model(self, filepath, overwrite):
-        dirpath = '/'.join(filepath.split('/')[:-1])
+    def _del_model(self, filepath):
+        dirpath = os.path.dirname(filepath)
 
         # make paths
-        os.makedirs(os.path.dirname(filepath), exist_ok=True)
+        os.makedirs(dirpath, exist_ok=True)
 
-        if overwrite:
-            for filename in os.listdir(dirpath):
-                if self.prefix in filename:
-                    path_to_delete = os.path.join(dirpath, filename)
-                    try:
-                        shutil.rmtree(path_to_delete)
-                    except OSError:
-                        os.remove(path_to_delete)
+        try:
+            shutil.rmtree(filepath)
+        except OSError:
+            os.remove(filepath)
+
+    def _save_model(self, filepath):
+        dirpath = os.path.dirname(filepath)
+
+        # make paths
+        os.makedirs(dirpath, exist_ok=True)
 
         # delegate the saving to the model
         self.save_function(filepath)
 
+    def check_monitor_top_k(self, current):
+        less_than_k_models = len(self.best_k_models.keys()) < self.save_top_k
+        if less_than_k_models:
+            return True
+        return self.monitor_op(current, self.best_k_models[self.kth_best_model])
+
     def on_epoch_end(self, epoch, logs=None):
         logs = logs or {}
-        self.epochs_since_last_save += 1
-        if self.epochs_since_last_save >= self.period:
-            self.epochs_since_last_save = 0
-            filepath = '{}/{}_ckpt_epoch_{}.ckpt'.format(self.filepath, self.prefix, epoch + 1)
-            if self.save_best_only:
+        self.epochs_since_last_check += 1
+
+        if self.save_top_k == 0:
+            # no models are saved
+            return
+        if self.epochs_since_last_check >= self.period:
+            self.epochs_since_last_check = 0
+            filepath = f'{self.filepath}/{self.prefix}_ckpt_epoch_{epoch}.ckpt'
+            version_cnt = 0
+            while os.path.isfile(filepath):
+                # this epoch called before
+                filepath = f'{self.filepath}/{self.prefix}_ckpt_epoch_{epoch}_v{version_cnt}.ckpt'
+                version_cnt += 1
+
+            if self.save_top_k != -1:
                 current = logs.get(self.monitor)
+
                 if current is None:
                     warnings.warn(
                         f'Can save best model only with {self.monitor} available,'
                         ' skipping.', RuntimeWarning)
                 else:
-                    if self.monitor_op(current, self.best):
+                    if self.check_monitor_top_k(current):
+
+                        # remove kth
+                        if len(self.best_k_models.keys()) == self.save_top_k:
+                            delpath = self.kth_best_model
+                            self.best_k_models.pop(self.kth_best_model)
+                            self._del_model(delpath)
+
+                        self.best_k_models[filepath] = current
+                        if len(self.best_k_models.keys()) == self.save_top_k:
+                            # monitor dict has reached k elements
+                            if self.mode == 'min':
+                                self.kth_best_model = max(self.best_k_models, key=self.best_k_models.get)
+                            else:
+                                self.kth_best_model = min(self.best_k_models, key=self.best_k_models.get)
+                            self.kth_value = self.best_k_models[self.kth_best_model]
+
+                        if self.mode == 'min':
+                            self.best = min(self.best_k_models.values())
+                        else:
+                            self.best = max(self.best_k_models.values())
                         if self.verbose > 0:
                             logging.info(
-                                f'\nEpoch {epoch + 1:05d}: {self.monitor} improved'
-                                f' from {self.best:0.5f} to {current:0.5f},'
-                                f' saving model to {filepath}')
-                        self.best = current
-                        self.save_model(filepath, overwrite=True)
+                                f'\nEpoch {epoch:05d}: {self.monitor} reached'
+                                f' {current:0.5f} (best {self.best:0.5f}), saving model to'
+                                f' {filepath} as top {self.save_top_k}')
+                        self._save_model(filepath)
 
                     else:
                         if self.verbose > 0:
                             logging.info(
-                                f'\nEpoch {epoch + 1:05d}: {self.monitor} did not improve')
+                                f'\nEpoch {epoch:05d}: {self.monitor}'
+                                f' was not in top {self.save_top_k}')
+
             else:
                 if self.verbose > 0:
-                    logging.info(f'\nEpoch {epoch + 1:05d}: saving model to {filepath}')
-                self.save_model(filepath, overwrite=False)
+                    logging.info(f'\nEpoch {epoch:05d}: saving model to {filepath}')
+                self._save_model(filepath)
 
 
 class GradientAccumulationScheduler(Callback):
     """Change gradient accumulation factor according to scheduling.
+
     # Arguments
         scheduling: dict, scheduling in format {epoch: accumulation_factor}
+
     """
 
     def __init__(self, scheduling: dict):
@@ -300,11 +363,11 @@ class GradientAccumulationScheduler(Callback):
                 break
 
 
-if __name__ == '__main__':
-    c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
-    losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
-    for i, loss in enumerate(losses):
-        should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
-        logging.info(loss)
-        if should_stop:
-            break
+# if __name__ == '__main__':
+#     c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
+#     losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
+#     for i, loss in enumerate(losses):
+#         should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
+#         logging.info(loss)
+#         if should_stop:
+#             break

pytorch_lightning/core/__init__.py

@@ -0,0 +1,150 @@
+"""
+Lightning Module interface
+==========================
+
+A lightning module is a strict superclass of nn.Module, it provides a standard interface
+ for the trainer to interact with the model.
+
+The easiest thing to do is copy the minimal example below and modify accordingly.
+
+Otherwise, to Define a Lightning Module, implement the following methods:
+
+
+Minimal example
+---------------
+
+.. code-block:: python
+
+    import os
+    import torch
+    from torch.nn import functional as F
+    from torch.utils.data import DataLoader
+    from torchvision.datasets import MNIST
+    import torchvision.transforms as transforms
+
+    import pytorch_lightning as pl
+
+    class CoolModel(pl.LightningModule):
+
+        def __init__(self):
+            super(CoolModel, self).__init__()
+            # not the best model...
+            self.l1 = torch.nn.Linear(28 * 28, 10)
+
+        def forward(self, x):
+            return torch.relu(self.l1(x.view(x.size(0), -1)))
+
+        def training_step(self, batch, batch_idx):
+            # REQUIRED
+            x, y = batch
+            y_hat = self.forward(x)
+            return {'loss': F.cross_entropy(y_hat, y)}
+
+        def validation_step(self, batch, batch_idx):
+            # OPTIONAL
+            x, y = batch
+            y_hat = self.forward(x)
+            return {'val_loss': F.cross_entropy(y_hat, y)}
+
+        def validation_end(self, outputs):
+            # OPTIONAL
+            val_loss_mean = torch.stack([x['val_loss'] for x in outputs]).mean()
+            return {'val_loss': val_loss_mean}
+
+        def test_step(self, batch, batch_idx):
+            # OPTIONAL
+            x, y = batch
+            y_hat = self.forward(x)
+            return {'test_loss': F.cross_entropy(y_hat, y)}
+
+        def test_end(self, outputs):
+            # OPTIONAL
+            test_loss_mean = torch.stack([x['test_loss'] for x in outputs]).mean()
+            return {'test_loss': test_loss_mean}
+
+        def configure_optimizers(self):
+            # REQUIRED
+            return torch.optim.Adam(self.parameters(), lr=0.02)
+
+        @pl.data_loader
+        def train_dataloader(self):
+            return DataLoader(MNIST(os.getcwd(), train=True, download=True,
+                              transform=transforms.ToTensor()), batch_size=32)
+
+        @pl.data_loader
+        def val_dataloader(self):
+            # OPTIONAL
+            # can also return a list of val dataloaders
+            return DataLoader(MNIST(os.getcwd(), train=True, download=True,
+                              transform=transforms.ToTensor()), batch_size=32)
+
+        @pl.data_loader
+        def test_dataloader(self):
+            # OPTIONAL
+            # can also return a list of test dataloaders
+            return DataLoader(MNIST(os.getcwd(), train=False, download=True,
+                              transform=transforms.ToTensor()), batch_size=32)
+
+
+How do these methods fit into the broader training?
+---------------------------------------------------
+
+The LightningModule interface is on the right. Each method corresponds
+ to a part of a research project. Lightning automates everything not in blue.
+
+.. figure::  docs/source/_static/images/overview_flat.jpg
+   :align:   center
+
+   Overview.
+
+
+Optional Methods
+----------------
+
+**add_model_specific_args**
+
+.. code-block:: python
+
+    @staticmethod
+    def add_model_specific_args(parent_parser, root_dir)
+
+Lightning has a list of default argparse commands.
+ This method is your chance to add or modify commands specific to your model.
+ The `hyperparameter argument parser
+  <https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser>`_
+ is available anywhere in your model by calling self.hparams.
+
+**Return**
+An argument parser
+
+**Example**
+
+.. code-block:: python
+
+    @staticmethod
+    def add_model_specific_args(parent_parser, root_dir):
+        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
+
+        # param overwrites
+        # parser.set_defaults(gradient_clip_val=5.0)
+
+        # network params
+        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
+        parser.add_argument('--in_features', default=28*28)
+        parser.add_argument('--out_features', default=10)
+        # use 500 for CPU, 50000 for GPU to see speed difference
+        parser.add_argument('--hidden_dim', default=50000)
+
+        # data
+        parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
+
+        # training params (opt)
+        parser.opt_list('--learning_rate', default=0.001, type=float,
+                        options=[0.0001, 0.0005, 0.001, 0.005], tunable=False)
+        parser.opt_list('--batch_size', default=256, type=int,
+                        options=[32, 64, 128, 256], tunable=False)
+        parser.opt_list('--optimizer_name', default='adam', type=str,
+                        options=['adam'], tunable=False)
+        return parser
+
+"""

pytorch_lightning/core/hooks.py

@@ -0,0 +1,153 @@
+"""
+# Hooks
+
+There are cases when you might want to do something different at different parts of the training/validation loop.
+ To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
+
+**Contributing** If there's a hook you'd like to add, simply:
+1. Fork PyTorchLightning.
+2. Add the hook :py:mod:`pytorch_lightning.base_module.hooks.py`.
+3. Add the correct place in the :py:mod:`pytorch_lightning.models.trainer` where it should be called.
+
+"""
+
+
+import torch
+
+
+try:
+    from apex import amp
+
+    APEX_AVAILABLE = True
+except ImportError:
+    APEX_AVAILABLE = False
+
+
+class ModelHooks(torch.nn.Module):
+
+    def on_sanity_check_start(self):
+        """
+        Called before starting evaluate
+        .. warning:: will be deprecated.
+        :return:
+        """
+        pass
+
+    def on_train_start(self):
+        """Called at the beginning of training before sanity check
+        :return:
+        """
+        # do something at the start of training
+        pass
+
+    def on_train_end(self):
+        """
+        Called at the end of training before logger experiment is closed
+        :return:
+        """
+        # do something at the end of training
+        pass
+
+    def on_batch_start(self, batch):
+        """Called in the training loop before anything happens for that batch.
+
+        :param batch:
+        :return:
+        """
+        # do something when the batch starts
+        pass
+
+    def on_batch_end(self):
+        """Called in the training loop after the batch."""
+        # do something when the batch ends
+        pass
+
+    def on_epoch_start(self):
+        """Called in the training loop at the very beginning of the epoch."""
+        # do something when the epoch starts
+        pass
+
+    def on_epoch_end(self):
+        """Called in the training loop at the very end of the epoch."""
+        # do something when the epoch ends
+        pass
+
+    def on_pre_performance_check(self):
+        """Called at the very beginning of the validation loop."""
+        # do something before validation starts
+        pass
+
+    def on_post_performance_check(self):
+        """Called at the very end of the validation loop."""
+        # do something before validation end
+        pass
+
+    def on_before_zero_grad(self, optimizer):
+        """Called after optimizer.step() and before optimizer.zero_grad()
+
+        Called in the training loop after taking an optimizer step and before zeroing grads.
+        Good place to inspect weight information with weights updated.
+
+        for optimizer in optimizers::
+
+            optimizer.step()
+            model.on_before_zero_grad(optimizer) # < ---- called here
+            optimizer.zero_grad
+
+        :param optimizer:
+        :return:
+        """
+        # do something with the optimizer or inspect it.
+        pass
+
+    def on_after_backward(self):
+        """Called after loss.backward() and before optimizers do anything.
+
+        :return:
+
+        Called in the training loop after model.backward()
+        This is the ideal place to inspect or log gradient information
+
+        .. code-block:: python
+
+            def on_after_backward(self):
+                # example to inspect gradient information in tensorboard
+                if self.trainer.global_step % 25 == 0:  # don't make the tf file huge
+                    params = self.state_dict()
+                    for k, v in params.items():
+                        grads = v
+                        name = k
+                        self.logger.experiment.add_histogram(tag=name, values=grads,
+                                                             global_step=self.trainer.global_step)
+
+        """
+        pass
+
+    def backward(self, use_amp, loss, optimizer):
+        """Override backward with your own implementation if you need to
+
+        :param use_amp: Whether amp was requested or not
+        :param loss: Loss is already scaled by accumulated grads
+        :param optimizer: Current optimizer being used
+        :return:
+
+        Called to perform backward step.
+        Feel free to override as needed.
+
+        The loss passed in has already been scaled for accumulated gradients if requested.
+
+        .. code-block:: python
+
+            def backward(self, use_amp, loss, optimizer):
+                if use_amp:
+                    with amp.scale_loss(loss, optimizer) as scaled_loss:
+                        scaled_loss.backward()
+                else:
+                    loss.backward()
+
+        """
+        if use_amp:
+            with amp.scale_loss(loss, optimizer) as scaled_loss:
+                scaled_loss.backward()
+        else:
+            loss.backward()

pytorch_lightning/core/memory.py

@@ -50,20 +50,31 @@ class ModelSummary(object):
         input_ = self.model.example_input_array
 
         if self.model.on_gpu:
-            input_ = input_.cuda(0)
+            device = next(self.model.parameters()).get_device()
+            # test if input is a list or a tuple
+            if isinstance(input_, (list, tuple)):
+                input_ = [input_i.cuda(device) if torch.is_tensor(input_i) else input_i
+                          for input_i in input_]
+            else:
+                input_ = input_.cuda(device)
 
         if self.model.trainer.use_amp:
-            input_ = input_.half()
+            # test if it is not a list or a tuple
+            if isinstance(input_, (list, tuple)):
+                input_ = [input_i.half() if torch.is_tensor(input_i) else input_i
+                          for input_i in input_]
+            else:
+                input_ = input_.half()
 
         with torch.no_grad():
 
             for _, m in mods:
-                if type(input_) is list or type(input_) is tuple:  # pragma: no cover
+                if isinstance(input_, (list, tuple)):  # pragma: no cover
                     out = m(*input_)
                 else:
                     out = m(input_)
 
-                if type(input_) is tuple or type(input_) is list:  # pragma: no cover
+                if isinstance(input_, (list, tuple)):  # pragma: no cover
                     in_size = []
                     for x in input_:
                         if type(x) is list:
@@ -75,7 +86,7 @@ class ModelSummary(object):
 
                 in_sizes.append(in_size)
 
-                if type(out) is tuple or type(out) is list:  # pragma: no cover
+                if isinstance(out, (list, tuple)):  # pragma: no cover
                     out_size = np.asarray([x.size() for x in out])
                 else:
                     out_size = np.array(out.size())
@@ -174,20 +185,20 @@ def print_mem_stack():  # pragma: no cover
 
 
 def count_mem_items():  # pragma: no cover
-    nb_params = 0
-    nb_tensors = 0
+    num_params = 0
+    num_tensors = 0
     for obj in gc.get_objects():
         try:
             if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                 obj_type = str(type(obj))
                 if 'parameter' in obj_type:
-                    nb_params += 1
+                    num_params += 1
                 else:
-                    nb_tensors += 1
+                    num_tensors += 1
         except Exception:
             pass
 
-    return nb_params, nb_tensors
+    return num_params, num_tensors
 
 
 def get_memory_profile(mode):

pytorch_lightning/core/model_saving.py

@@ -0,0 +1,10 @@
+"""
+.. warning:: `model_saving` module has been renamed to `saving` since v0.6.0 and will be removed in v0.8.0
+"""
+
+import warnings
+
+warnings.warn("`model_saving` module has been renamed to `saving` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
+
+from pytorch_lightning.core.saving import ModelIO  # noqa: E402

pytorch_lightning/core/root_module.py

@@ -0,0 +1,10 @@
+"""
+.. warning:: `root_module` module has been renamed to `lightning` since v0.6.0 and will be removed in v0.8.0
+"""
+
+import warnings
+
+warnings.warn("`root_module` module has been renamed to `lightning` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
+
+from pytorch_lightning.core.lightning import LightningModule  # noqa: E402

pytorch_lightning/logging/__init__.py

@@ -1,18 +1,189 @@
+"""
+Lighting offers options for logging information about model, gpu usage, etc,
+ via several different logging frameworks. It also offers printing options for training monitoring.
+
+**default_save_path**
+
+Lightning sets a default TestTubeLogger and CheckpointCallback for you which log to
+`os.getcwd()` by default. To modify the logging path you can set::
+
+    Trainer(default_save_path='/your/path/to/save/checkpoints')
+
+
+If you need more custom behavior (different paths for both, different metrics, etc...)
+ from the logger and the checkpointCallback, pass in your own instances as explained below.
+
+Setting up logging
+------------------
+
+The trainer inits a default logger for you (TestTubeLogger). All logs will
+go to the current working directory under a folder named `os.getcwd()/lightning_logs`.
+
+If you want to modify the default logging behavior even more, pass in a logger
+ (which should inherit from `LightningBaseLogger`).
+
+.. code-block:: python
+
+    my_logger = MyLightningLogger(...)
+    trainer = Trainer(logger=my_logger)
+
+
+The path in this logger will overwrite `default_save_path`.
+
+Lightning supports several common experiment tracking frameworks out of the box
+
+Custom logger
+-------------
+
+You can implement your own logger by writing a class that inherits from
+`LightningLoggerBase`. Use the `rank_zero_only` decorator to make sure that
+only the first process in DDP training logs data.
+
+.. code-block:: python
+
+    from pytorch_lightning.logging import LightningLoggerBase, rank_zero_only
+
+    class MyLogger(LightningLoggerBase):
+
+        @rank_zero_only
+        def log_hyperparams(self, params):
+            # params is an argparse.Namespace
+            # your code to record hyperparameters goes here
+            pass
+
+        @rank_zero_only
+        def log_metrics(self, metrics, step):
+            # metrics is a dictionary of metric names and values
+            # your code to record metrics goes here
+            pass
+
+        def save(self):
+            # Optional. Any code necessary to save logger data goes here
+            pass
+
+        @rank_zero_only
+        def finalize(self, status):
+            # Optional. Any code that needs to be run after training
+            # finishes goes here
+
+
+If you write a logger than may be useful to others, please send
+a pull request to add it to Lighting!
+
+Using loggers
+-------------
+
+You can call the logger anywhere from your LightningModule by doing:
+
+.. code-block:: python
+
+    def train_step(...):
+        # example
+        self.logger.experiment.whatever_method_summary_writer_supports(...)
+
+    def any_lightning_module_function_or_hook(...):
+        self.logger.experiment.add_histogram(...)
+
+Display metrics in progress bar
+-------------------------------
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(show_progress_bar=True)
+
+Log metric row every k batches
+------------------------------
+
+Every k batches lightning will make an entry in the metrics log
+
+.. code-block:: python
+
+    # DEFAULT (ie: save a .csv log file every 10 batches)
+    trainer = Trainer(row_log_interval=10)
+
+Log GPU memory
+--------------
+
+Logs GPU memory when metrics are logged.
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(log_gpu_memory=None)
+
+    # log only the min/max utilization
+    trainer = Trainer(log_gpu_memory='min_max')
+
+    # log all the GPU memory (if on DDP, logs only that node)
+    trainer = Trainer(log_gpu_memory='all')
+
+Process position
+----------------
+
+When running multiple models on the same machine we want to decide which progress bar to use.
+ Lightning will stack progress bars according to this value.
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(process_position=0)
+
+    # if this is the second model on the node, show the second progress bar below
+    trainer = Trainer(process_position=1)
+
+
+Save a snapshot of all hyperparameters
+--------------------------------------
+
+Automatically log hyperparameters stored in the `hparams` attribute as an `argparse.Namespace`
+
+.. code-block:: python
+
+    class MyModel(pl.Lightning):
+        def __init__(self, hparams):
+            self.hparams = hparams
+
+        ...
+
+    args = parser.parse_args()
+    model = MyModel(args)
+
+    logger = TestTubeLogger(...)
+    t = Trainer(logger=logger)
+    trainer.fit(model)
+
+Write logs file to csv every k batches
+--------------------------------------
+
+Every k batches, lightning will write the new logs to disk
+
+.. code-block:: python
+
+    # DEFAULT (ie: save a .csv log file every 100 batches)
+    trainer = Trainer(log_save_interval=100)
+
+"""
+
 from os import environ
 from .base import LightningLoggerBase, rank_zero_only
 
+from .tensorboard import TensorBoardLogger
+
 try:
-    from .test_tube_logger import TestTubeLogger
+    from .test_tube import TestTubeLogger
 except ImportError:
     pass
+
 try:
-    from .mlflow_logger import MLFlowLogger
+    from .mlflow import MLFlowLogger
 except ImportError:
     pass
+
 try:
     # needed to prevent ImportError and duplicated logs.
     environ["COMET_DISABLE_AUTO_LOGGING"] = "1"
 
-    from .comet_logger import CometLogger
+    from .comet import CometLogger
 except ImportError:
     del environ["COMET_DISABLE_AUTO_LOGGING"]

pytorch_lightning/logging/base.py

@@ -1,8 +1,9 @@
+from abc import ABC
 from functools import wraps
 
 
 def rank_zero_only(fn):
-    """Decorate a logger method to run it only on the process with rank 0
+    """Decorate a logger method to run it only on the process with rank 0.
 
     :param fn: Function to decorate
     """
@@ -15,62 +16,62 @@ def rank_zero_only(fn):
     return wrapped_fn
 
 
-class LightningLoggerBase(object):
-    """Base class for experiment loggers"""
+class LightningLoggerBase(ABC):
+    """Base class for experiment loggers."""
 
     def __init__(self):
         self._rank = 0
 
-    def log_metrics(self, metrics, step_num):
-        """Record metrics
+    @property
+    def experiment(self):
+        raise NotImplementedError()
 
-        :param metric: Dictionary with metric names as keys and measured
-        quanties as values
-        :param step_num: Step number at which the metrics should be recorded
+    def log_metrics(self, metrics, step):
+        """Record metrics.
+
+        :param float metric: Dictionary with metric names as keys and measured quanties as values
+        :param int|None step: Step number at which the metrics should be recorded
         """
         raise NotImplementedError()
 
     def log_hyperparams(self, params):
-        """Record hyperparameters
+        """Record hyperparameters.
 
         :param params: argparse.Namespace containing the hyperparameters
         """
         raise NotImplementedError()
 
     def save(self):
-        """Save log data"""
+        """Save log data."""
         pass
 
     def finalize(self, status):
-        """Do any processing that is necessary to finalize an experiment
+        """Do any processing that is necessary to finalize an experiment.
 
         :param status: Status that the experiment finished with (e.g. success, failed, aborted)
         """
         pass
 
     def close(self):
-        """Do any cleanup that is necessary to close an experiment"""
+        """Do any cleanup that is necessary to close an experiment."""
         pass
 
     @property
     def rank(self):
-        """
-        Process rank. In general, metrics should only be logged by the process
-        with rank 0
-        """
+        """Process rank. In general, metrics should only be logged by the process with rank 0."""
         return self._rank
 
     @rank.setter
     def rank(self, value):
-        """Set the process rank"""
+        """Set the process rank."""
         self._rank = value
 
     @property
     def name(self):
-        """Return the experiment name"""
+        """Return the experiment name."""
         raise NotImplementedError("Sub-classes must provide a name property")
 
     @property
     def version(self):
-        """Return the experiment version"""
+        """Return the experiment version."""
         raise NotImplementedError("Sub-classes must provide a version property")

pytorch_lightning/logging/comet.py

@@ -0,0 +1,174 @@
+"""
+Log using `comet <https://www.comet.ml>`_
+
+Comet logger can be used in either online or offline mode.
+To log in online mode, CometLogger requries an API key:
+
+.. code-block:: python
+
+    from pytorch_lightning.logging import CometLogger
+    # arguments made to CometLogger are passed on to the comet_ml.Experiment class
+    comet_logger = CometLogger(
+        api_key=os.environ["COMET_KEY"],
+        workspace=os.environ["COMET_WORKSPACE"], # Optional
+        project_name="default_project", # Optional
+        rest_api_key=os.environ["COMET_REST_KEY"], # Optional
+        experiment_name="default" # Optional
+    )
+    trainer = Trainer(logger=comet_logger)
+
+To log in offline mode, CometLogger requires a path to a local directory:
+
+.. code-block:: python
+
+    from pytorch_lightning.logging import CometLogger
+    # arguments made to CometLogger are passed on to the comet_ml.Experiment class
+    comet_logger = CometLogger(
+        save_dir=".",
+        workspace=os.environ["COMET_WORKSPACE"], # Optional
+        project_name="default_project", # Optional
+        rest_api_key=os.environ["COMET_REST_KEY"], # Optional
+        experiment_name="default" # Optional
+    )
+    trainer = Trainer(logger=comet_logger)
+
+
+Use the logger anywhere in you LightningModule as follows:
+
+.. code-block:: python
+
+    def train_step(...):
+        # example
+        self.logger.experiment.whatever_comet_ml_supports(...)
+
+    def any_lightning_module_function_or_hook(...):
+        self.logger.experiment.whatever_comet_ml_supports(...)
+
+
+"""
+
+from logging import getLogger
+
+try:
+    from comet_ml import Experiment as CometExperiment
+    from comet_ml import OfflineExperiment as CometOfflineExperiment
+    try:
+        from comet_ml.api import API
+    except ImportError:
+        # For more information, see: https://www.comet.ml/docs/python-sdk/releases/#release-300
+        from comet_ml.papi import API
+except ImportError:
+    raise ImportError('Missing comet_ml package.')
+
+from torch import is_tensor
+
+from .base import LightningLoggerBase, rank_zero_only
+from ..utilities.debugging import MisconfigurationException
+
+logger = getLogger(__name__)
+
+
+class CometLogger(LightningLoggerBase):
+    def __init__(self, api_key=None, save_dir=None, workspace=None,
+                 rest_api_key=None, project_name=None, experiment_name=None, **kwargs):
+        """Initialize a Comet.ml logger.
+        Requires either an API Key (online mode) or a local directory path (offline mode)
+
+        :param str api_key: Required in online mode. API key, found on Comet.ml
+        :param str save_dir: Required in offline mode. The path for the directory to save local comet logs
+        :param str workspace: Optional. Name of workspace for this user
+        :param str project_name: Optional. Send your experiment to a specific project.
+        Otherwise will be sent to Uncategorized Experiments.
+        If project name does not already exists Comet.ml will create a new project.
+        :param str rest_api_key: Optional. Rest API key found in Comet.ml settings.
+        This is used to determine version number
+        :param str experiment_name: Optional. String representing the name for this particular experiment on Comet.ml
+        """
+        super().__init__()
+        self._experiment = None
+
+        # Determine online or offline mode based on which arguments were passed to CometLogger
+        if save_dir is not None and api_key is not None:
+            # If arguments are passed for both save_dir and api_key, preference is given to online mode
+            self.mode = "online"
+            self.api_key = api_key
+        elif api_key is not None:
+            self.mode = "online"
+            self.api_key = api_key
+        elif save_dir is not None:
+            self.mode = "offline"
+            self.save_dir = save_dir
+        else:
+            # If neither api_key nor save_dir are passed as arguments, raise an exception
+            raise MisconfigurationException("CometLogger requires either api_key or save_dir during initialization.")
+
+        logger.info(f"CometLogger will be initialized in {self.mode} mode")
+
+        self.workspace = workspace
+        self.project_name = project_name
+        self._kwargs = kwargs
+
+        if rest_api_key is not None:
+            # Comet.ml rest API, used to determine version number
+            self.rest_api_key = rest_api_key
+            self.comet_api = API(self.rest_api_key)
+        else:
+            self.rest_api_key = None
+            self.comet_api = None
+
+        if experiment_name:
+            try:
+                self.name = experiment_name
+            except TypeError as e:
+                logger.exception("Failed to set experiment name for comet.ml logger")
+
+    @property
+    def experiment(self):
+        if self._experiment is not None:
+            return self._experiment
+
+        if self.mode == "online":
+            self._experiment = CometExperiment(
+                api_key=self.api_key,
+                workspace=self.workspace,
+                project_name=self.project_name,
+                **self._kwargs
+            )
+        else:
+            self._experiment = CometOfflineExperiment(
+                offline_directory=self.save_dir,
+                workspace=self.workspace,
+                project_name=self.project_name,
+                **self._kwargs
+            )
+
+        return self._experiment
+
+    @rank_zero_only
+    def log_hyperparams(self, params):
+        self.experiment.log_parameters(vars(params))
+
+    @rank_zero_only
+    def log_metrics(self, metrics, step=None):
+        # Comet.ml expects metrics to be a dictionary of detached tensors on CPU
+        for key, val in metrics.items():
+            if is_tensor(val):
+                metrics[key] = val.cpu().detach()
+
+        self.experiment.log_metrics(metrics, step=step)
+
+    @rank_zero_only
+    def finalize(self, status):
+        self.experiment.end()
+
+    @property
+    def name(self):
+        return self.experiment.project_name
+
+    @name.setter
+    def name(self, value):
+        self.experiment.set_name(value)
+
+    @property
+    def version(self):
+        return self.experiment.id

pytorch_lightning/logging/comet_logger.py

@@ -1,25 +1,10 @@
-try:
-    from comet_ml import Experiment as CometExperiment
-except ImportError:
-    raise ImportError('Missing comet_ml package.')
+"""
+.. warning:: `comet_logger` module has been renamed to `comet` since v0.6.0 and will be removed in v0.8.0
+"""
 
-from .base import LightningLoggerBase, rank_zero_only
+import warnings
 
+warnings.warn("`comet_logger` module has been renamed to `comet` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
 
-class CometLogger(LightningLoggerBase):
-    def __init__(self, *args, **kwargs):
-        super(CometLogger, self).__init__()
-        self.experiment = CometExperiment(*args, **kwargs)
-
-    @rank_zero_only
-    def log_hyperparams(self, params):
-        self.experiment.log_parameters(vars(params))
-
-    @rank_zero_only
-    def log_metrics(self, metrics, step_num):
-        # self.experiment.set_epoch(self, metrics.get('epoch', 0))
-        self.experiment.log_metrics(metrics)
-
-    @rank_zero_only
-    def finalize(self, status):
-        self.experiment.end()
+from pytorch_lightning.logging.comet import CometLogger  # noqa: E402

pytorch_lightning/logging/mlflow.py

@@ -0,0 +1,100 @@
+"""
+Log using `mlflow <https://mlflow.org>'_
+
+.. code-block:: python
+
+    from pytorch_lightning.logging import MLFlowLogger
+    mlf_logger = MLFlowLogger(
+        experiment_name="default",
+        tracking_uri="file:/."
+    )
+    trainer = Trainer(logger=mlf_logger)
+
+
+Use the logger anywhere in you LightningModule as follows:
+
+.. code-block:: python
+
+    def train_step(...):
+        # example
+        self.logger.experiment.whatever_ml_flow_supports(...)
+
+    def any_lightning_module_function_or_hook(...):
+        self.logger.experiment.whatever_ml_flow_supports(...)
+
+"""
+
+from logging import getLogger
+from time import time
+
+try:
+    import mlflow
+except ImportError:
+    raise ImportError('Missing mlflow package.')
+
+from .base import LightningLoggerBase, rank_zero_only
+
+logger = getLogger(__name__)
+
+
+class MLFlowLogger(LightningLoggerBase):
+    def __init__(self, experiment_name, tracking_uri=None, tags=None):
+        super().__init__()
+        self._mlflow_client = mlflow.tracking.MlflowClient(tracking_uri)
+        self.experiment_name = experiment_name
+        self._run_id = None
+        self.tags = tags
+
+    @property
+    def experiment(self):
+        return self._mlflow_client
+
+    @property
+    def run_id(self):
+        if self._run_id is not None:
+            return self._run_id
+
+        expt = self._mlflow_client.get_experiment_by_name(self.experiment_name)
+
+        if expt:
+            self._expt_id = expt.experiment_id
+        else:
+            logger.warning(f"Experiment with name {self.experiment_name} not found. Creating it.")
+            self._expt_id = self._mlflow_client.create_experiment(name=self.experiment_name)
+
+        run = self._mlflow_client.create_run(experiment_id=self._expt_id, tags=self.tags)
+        self._run_id = run.info.run_id
+        return self._run_id
+
+    @rank_zero_only
+    def log_hyperparams(self, params):
+        for k, v in vars(params).items():
+            self.experiment.log_param(self.run_id, k, v)
+
+    @rank_zero_only
+    def log_metrics(self, metrics, step=None):
+        timestamp_ms = int(time() * 1000)
+        for k, v in metrics.items():
+            if isinstance(v, str):
+                logger.warning(
+                    f"Discarding metric with string value {k}={v}"
+                )
+                continue
+            self.experiment.log_metric(self.run_id, k, v, timestamp_ms, step)
+
+    def save(self):
+        pass
+
+    @rank_zero_only
+    def finalize(self, status="FINISHED"):
+        if status == 'success':
+            status = 'FINISHED'
+        self.experiment.set_terminated(self.run_id, status)
+
+    @property
+    def name(self):
+        return self.experiment_name
+
+    @property
+    def version(self):
+        return self._run_id

pytorch_lightning/logging/mlflow_logger.py

@@ -1,70 +1,10 @@
-from logging import getLogger
-from time import time
+"""
+.. warning:: `mlflow_logger` module has been renamed to `mlflow` since v0.6.0 and will be removed in v0.8.0
+"""
 
-try:
-    import mlflow
-except ImportError:
-    raise ImportError('Missing mlflow package.')
+import warnings
 
-from .base import LightningLoggerBase, rank_zero_only
+warnings.warn("`mlflow_logger` module has been renamed to `mlflow` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
 
-logger = getLogger(__name__)
-
-
-class MLFlowLogger(LightningLoggerBase):
-    def __init__(self, experiment_name, tracking_uri=None, tags=None):
-        super().__init__()
-        self.experiment = mlflow.tracking.MlflowClient(tracking_uri)
-        self.experiment_name = experiment_name
-        self._run_id = None
-        self.tags = tags
-
-    @property
-    def run_id(self):
-        if self._run_id is not None:
-            return self._run_id
-
-        experiment = self.experiment.get_experiment_by_name(self.experiment_name)
-        if experiment is None:
-            logger.warning(
-                f"Experiment with name f{self.experiment_name} not found. Creating it."
-            )
-            self.experiment.create_experiment(self.experiment_name)
-            experiment = self.experiment.get_experiment_by_name(self.experiment_name)
-
-        run = self.experiment.create_run(experiment.experiment_id, tags=self.tags)
-        self._run_id = run.info.run_id
-        return self._run_id
-
-    @rank_zero_only
-    def log_hyperparams(self, params):
-        for k, v in vars(params).items():
-            self.experiment.log_param(self.run_id, k, v)
-
-    @rank_zero_only
-    def log_metrics(self, metrics, step_num=None):
-        timestamp_ms = int(time() * 1000)
-        for k, v in metrics.items():
-            if isinstance(v, str):
-                logger.warning(
-                    f"Discarding metric with string value {k}={v}"
-                )
-                continue
-            self.experiment.log_metric(self.run_id, k, v, timestamp_ms, step_num)
-
-    def save(self):
-        pass
-
-    @rank_zero_only
-    def finalize(self, status="FINISHED"):
-        if status == 'success':
-            status = 'FINISHED'
-        self.experiment.set_terminated(self.run_id, status)
-
-    @property
-    def name(self):
-        return self.experiment_name
-
-    @property
-    def version(self):
-        return self._run_id
+from pytorch_lightning.logging.mlflow import MLFlowLogger  # noqa: E402

pytorch_lightning/logging/tensorboard.py

@@ -0,0 +1,114 @@
+import os
+from warnings import warn
+
+import torch
+from pkg_resources import parse_version
+from torch.utils.tensorboard import SummaryWriter
+
+from .base import LightningLoggerBase, rank_zero_only
+
+
+class TensorBoardLogger(LightningLoggerBase):
+    r"""Log to local file system in TensorBoard format
+
+    Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to
+    `os.path.join(save_dir, name, version)`
+
+    :example:
+
+    .. code-block:: python
+
+        logger = TensorBoardLogger("tb_logs", name="my_model")
+        trainer = Trainer(logger=logger)
+        trainer.train(model)
+
+    :param str save_dir: Save directory
+    :param str name: Experiment name. Defaults to "default".
+    :param int version: Experiment version. If version is not specified the logger inspects the save
+        directory for existing versions, then automatically assigns the next available version.
+    :param \**kwargs: Other arguments are passed directly to the :class:`SummaryWriter` constructor.
+
+
+    """
+
+    def __init__(self, save_dir, name="default", version=None, **kwargs):
+        super().__init__()
+        self.save_dir = save_dir
+        self._name = name
+        self._version = version
+
+        self._experiment = None
+        self.kwargs = kwargs
+
+    @property
+    def experiment(self):
+        """The underlying :class:`torch.utils.tensorboard.SummaryWriter`.
+
+        :rtype: torch.utils.tensorboard.SummaryWriter
+        """
+        if self._experiment is not None:
+            return self._experiment
+
+        root_dir = os.path.join(self.save_dir, self.name)
+        os.makedirs(root_dir, exist_ok=True)
+        log_dir = os.path.join(root_dir, str(self.version))
+        self._experiment = SummaryWriter(log_dir=log_dir, **self.kwargs)
+        return self._experiment
+
+    @rank_zero_only
+    def log_hyperparams(self, params):
+        if parse_version(torch.__version__) < parse_version("1.3.0"):
+            warn(
+                f"Hyperparameter logging is not available for Torch version {torch.__version__}."
+                " Skipping log_hyperparams. Upgrade to Torch 1.3.0 or above to enable"
+                " hyperparameter logging."
+            )
+            # TODO: some alternative should be added
+            return
+        try:
+            # in case converting from namespace, todo: rather test if it is namespace
+            params = vars(params)
+        except TypeError:
+            pass
+        if params is not None:
+            # `add_hparams` requires both - hparams and metric
+            self.experiment.add_hparams(hparam_dict=dict(params), metric_dict={})
+
+    @rank_zero_only
+    def log_metrics(self, metrics, step=None):
+        for k, v in metrics.items():
+            if isinstance(v, torch.Tensor):
+                v = v.item()
+            self.experiment.add_scalar(k, v, step)
+
+    @rank_zero_only
+    def save(self):
+        try:
+            self.experiment.flush()
+        except AttributeError:
+            # you are using PT version (<v1.2) which does not have implemented flush
+            self.experiment._get_file_writer().flush()
+
+    @rank_zero_only
+    def finalize(self, status):
+        self.save()
+
+    @property
+    def name(self):
+        return self._name
+
+    @property
+    def version(self):
+        if self._version is None:
+            self._version = self._get_next_version()
+        return self._version
+
+    def _get_next_version(self):
+        root_dir = os.path.join(self.save_dir, self.name)
+        existing_versions = [
+            int(d) for d in os.listdir(root_dir) if os.path.isdir(os.path.join(root_dir, d)) and d.isdigit()
+        ]
+        if len(existing_versions) == 0:
+            return 0
+        else:
+            return max(existing_versions) + 1

pytorch_lightning/logging/test_tube.py

@@ -0,0 +1,140 @@
+"""
+Log using `test tube <https://williamfalcon.github.io/test-tube>'_. Test tube logger is
+a strict subclass of `PyTorch SummaryWriter <https://pytorch.org/docs/stable/tensorboard.html>`_, refer to their
+documentation for all supported operations. The TestTubeLogger adds a nicer folder structure
+to manage experiments and snapshots all hyperparameters you pass to a LightningModule.
+
+.. code-block:: python
+
+    from pytorch_lightning.logging import TestTubeLogger
+    tt_logger = TestTubeLogger(
+        save_dir=".",
+        name="default",
+        debug=False,
+        create_git_tag=False
+    )
+    trainer = Trainer(logger=tt_logger)
+
+
+Use the logger anywhere in you LightningModule as follows:
+
+.. code-block:: python
+
+    def train_step(...):
+        # example
+        self.logger.experiment.whatever_method_summary_writer_supports(...)
+
+    def any_lightning_module_function_or_hook(...):
+        self.logger.experiment.add_histogram(...)
+
+"""
+
+try:
+    from test_tube import Experiment
+except ImportError:
+    raise ImportError('Missing test-tube package.')
+
+from .base import LightningLoggerBase, rank_zero_only
+
+
+class TestTubeLogger(LightningLoggerBase):
+    __test__ = False
+
+    def __init__(
+            self, save_dir, name="default", description=None, debug=False,
+            version=None, create_git_tag=False
+    ):
+        super().__init__()
+        self.save_dir = save_dir
+        self._name = name
+        self.description = description
+        self.debug = debug
+        self._version = version
+        self.create_git_tag = create_git_tag
+        self._experiment = None
+
+    @property
+    def experiment(self):
+        if self._experiment is not None:
+            return self._experiment
+
+        self._experiment = Experiment(
+            save_dir=self.save_dir,
+            name=self._name,
+            debug=self.debug,
+            version=self.version,
+            description=self.description,
+            create_git_tag=self.create_git_tag,
+            rank=self.rank,
+        )
+        return self._experiment
+
+    @rank_zero_only
+    def log_hyperparams(self, params):
+        # TODO: HACK figure out where this is being set to true
+        self.experiment.debug = self.debug
+        self.experiment.argparse(params)
+
+    @rank_zero_only
+    def log_metrics(self, metrics, step=None):
+        # TODO: HACK figure out where this is being set to true
+        self.experiment.debug = self.debug
+        self.experiment.log(metrics, global_step=step)
+
+    @rank_zero_only
+    def save(self):
+        # TODO: HACK figure out where this is being set to true
+        self.experiment.debug = self.debug
+        self.experiment.save()
+
+    @rank_zero_only
+    def finalize(self, status):
+        # TODO: HACK figure out where this is being set to true
+        self.experiment.debug = self.debug
+        self.save()
+        self.close()
+
+    @rank_zero_only
+    def close(self):
+        # TODO: HACK figure out where this is being set to true
+        self.experiment.debug = self.debug
+        exp = self.experiment
+        exp.close()
+
+    @property
+    def rank(self):
+        return self._rank
+
+    @rank.setter
+    def rank(self, value):
+        self._rank = value
+        if self._experiment is not None:
+            self.experiment.rank = value
+
+    @property
+    def name(self):
+        if self._experiment is None:
+            return self._name
+        else:
+            return self.experiment.name
+
+    @property
+    def version(self):
+        if self._experiment is None:
+            return self._version
+        else:
+            return self.experiment.version
+
+    # Test tube experiments are not pickleable, so we need to override a few
+    # methods to get DDP working. See
+    # https://docs.python.org/3/library/pickle.html#handling-stateful-objects
+    # for more info.
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        state["_experiment"] = self.experiment.get_meta_copy()
+        return state
+
+    def __setstate__(self, state):
+        self._experiment = state["_experiment"].get_non_ddp_exp()
+        del state["_experiment"]
+        self.__dict__.update(state)

pytorch_lightning/logging/test_tube_logger.py

@@ -1,109 +1,10 @@
-try:
-    from test_tube import Experiment
-except ImportError:
-    raise ImportError('Missing test-tube package.')
+"""
+.. warning:: `test_tube_logger` module has been renamed to `test_tube` since v0.6.0 and will be removed in v0.8.0
+"""
 
-from .base import LightningLoggerBase, rank_zero_only
+import warnings
 
+warnings.warn("`test_tube_logger` module has been renamed to `test_tube` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
 
-class TestTubeLogger(LightningLoggerBase):
-    __test__ = False
-
-    def __init__(
-            self, save_dir, name="default", description=None, debug=False,
-            version=None, create_git_tag=False
-    ):
-        super().__init__()
-        self.save_dir = save_dir
-        self._name = name
-        self.description = description
-        self.debug = debug
-        self._version = version
-        self.create_git_tag = create_git_tag
-        self._experiment = None
-
-    @property
-    def experiment(self):
-        if self._experiment is not None:
-            return self._experiment
-
-        self._experiment = Experiment(
-            save_dir=self.save_dir,
-            name=self._name,
-            debug=self.debug,
-            version=self.version,
-            description=self.description,
-            create_git_tag=self.create_git_tag,
-            rank=self.rank,
-        )
-        return self._experiment
-
-    @rank_zero_only
-    def log_hyperparams(self, params):
-        # TODO: HACK figure out where this is being set to true
-        self.experiment.debug = self.debug
-        self.experiment.argparse(params)
-
-    @rank_zero_only
-    def log_metrics(self, metrics, step_num=None):
-        # TODO: HACK figure out where this is being set to true
-        self.experiment.debug = self.debug
-        self.experiment.log(metrics, global_step=step_num)
-
-    @rank_zero_only
-    def save(self):
-        # TODO: HACK figure out where this is being set to true
-        self.experiment.debug = self.debug
-        self.experiment.save()
-
-    @rank_zero_only
-    def finalize(self, status):
-        # TODO: HACK figure out where this is being set to true
-        self.experiment.debug = self.debug
-        self.save()
-        self.close()
-
-    @rank_zero_only
-    def close(self):
-        # TODO: HACK figure out where this is being set to true
-        self.experiment.debug = self.debug
-        exp = self.experiment
-        exp.close()
-
-    @property
-    def rank(self):
-        return self._rank
-
-    @rank.setter
-    def rank(self, value):
-        self._rank = value
-        if self._experiment is not None:
-            self.experiment.rank = value
-
-    @property
-    def name(self):
-        if self._experiment is None:
-            return self._name
-        else:
-            return self.experiment.name
-
-    @property
-    def version(self):
-        if self._experiment is None:
-            return self._version
-        else:
-            return self.experiment.version
-
-    # Test tube experiments are not pickleable, so we need to override a few
-    # methods to get DDP working. See
-    # https://docs.python.org/3/library/pickle.html#handling-stateful-objects
-    # for more info.
-    def __getstate__(self):
-        state = self.__dict__.copy()
-        state["_experiment"] = self.experiment.get_meta_copy()
-        return state
-
-    def __setstate__(self, state):
-        self._experiment = state["_experiment"].get_non_ddp_exp()
-        del state["_experiment"]
-        self.__dict__.update(state)
+from pytorch_lightning.logging.test_tube import TestTubeLogger  # noqa: E402

pytorch_lightning/overrides/override_data_parallel.py

@@ -0,0 +1,12 @@
+"""
+.. warning:: `override_data_parallel` module has been renamed to `data_parallel` since v0.6.0
+ and will be removed in v0.8.0
+"""
+
+import warnings
+
+warnings.warn("`override_data_parallel` module has been renamed to `data_parallel` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
+
+from pytorch_lightning.overrides.data_parallel import (  # noqa: E402
+    get_a_var, parallel_apply, LightningDataParallel, LightningDistributedDataParallel)

pytorch_lightning/pt_overrides/__init__.py

@@ -0,0 +1,10 @@
+"""
+.. warning:: `pt_overrides` package has been renamed to `overrides` since v0.6.0 and will be removed in v0.8.0
+"""
+
+import warnings
+
+warnings.warn("`pt_overrides` package has been renamed to `overrides` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
+
+from pytorch_lightning.overrides import override_data_parallel  # noqa: E402

pytorch_lightning/root_module/__init__.py

@@ -0,0 +1,11 @@
+"""
+.. warning:: `root_module` package has been renamed to `core` since v0.6.0 and will be removed in v0.8.0
+"""
+
+import warnings
+
+warnings.warn("`root_module` package has been renamed to `core` since v0.6.0"
+              " and will be removed in v0.8.0", DeprecationWarning)
+
+from pytorch_lightning.core import (  # noqa: E402
+    decorators, grads, hooks, root_module, memory, model_saving)

pytorch_lightning/root_module/hooks.py

@@ -1,72 +0,0 @@
-import torch
-
-
-try:
-    from apex import amp
-
-    APEX_AVAILABLE = True
-except ImportError:
-    APEX_AVAILABLE = False
-
-
-class ModelHooks(torch.nn.Module):
-
-    def on_sanity_check_start(self):
-        """
-        Called before starting evaluate
-        :return:
-        """
-        pass
-
-    def on_batch_start(self, batch):
-        pass
-
-    def on_batch_end(self):
-        pass
-
-    def on_epoch_start(self):
-        pass
-
-    def on_epoch_end(self):
-        pass
-
-    def on_pre_performance_check(self):
-        pass
-
-    def on_post_performance_check(self):
-        pass
-
-    def on_before_zero_grad(self, optimizer):
-        """
-        Called after optimizer.step() and before optimizer.zero_grad()
-
-        for optimizer in optimizers:
-            optimizer.step()
-            model.on_before_zero_grad(optimizer) # < ---- called here
-            optimizer.zero_grad
-
-        :param optimizer:
-        :return:
-        """
-        pass
-
-    def on_after_backward(self):
-        """
-        Called after loss.backward() and before optimizers do anything
-        :return:
-        """
-        pass
-
-    def backward(self, use_amp, loss, optimizer):
-        """
-        Override backward with your own implementation if you need to
-        :param use_amp: Whether amp was requested or not
-        :param loss: Loss is already scaled by accumulated grads
-        :param optimizer: Current optimizer being used
-        :return:
-        """
-        if use_amp:
-            with amp.scale_loss(loss, optimizer) as scaled_loss:
-                scaled_loss.backward()
-        else:
-            loss.backward()

pytorch_lightning/root_module/root_module.py

@@ -1,334 +0,0 @@
-import os
-import warnings
-import collections
-from argparse import Namespace
-
-import torch
-import torch.distributed as dist
-
-from pytorch_lightning.root_module.decorators import data_loader
-from pytorch_lightning.root_module.grads import GradInformation
-from pytorch_lightning.root_module.hooks import ModelHooks
-from pytorch_lightning.root_module.memory import ModelSummary
-from pytorch_lightning.root_module.model_saving import ModelIO
-from pytorch_lightning.trainer.trainer_io import load_hparams_from_tags_csv
-import logging
-from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel
-
-
-class LightningModule(GradInformation, ModelIO, ModelHooks):
-
-    def __init__(self, *args, **kwargs):
-        super(LightningModule, self).__init__(*args, **kwargs)
-
-        self.dtype = torch.FloatTensor
-        self.exp_save_path = None
-        self.current_epoch = 0
-        self.global_step = 0
-        self.loaded_optimizer_states_dict = {}
-        self.trainer = None
-        self.logger = None
-        self.example_input_array = None
-
-        # track if gpu was requested for checkpointing
-        self.on_gpu = False
-        self.use_dp = False
-        self.use_ddp = False
-        self.use_ddp2 = False
-        self.use_amp = False
-
-    def forward(self, *args, **kwargs):
-        """
-        Expand model in into whatever you need.
-        Also need to return the target
-        :param x:
-        :return:
-        """
-        raise NotImplementedError
-
-    def training_step(self, *args, **kwargs):
-        """
-        return loss, dict with metrics for tqdm
-        :param called with batch, batch_nb
-        additional: optimizer_i if multiple optimizers used
-        :return: dict with loss key and optional log, progress keys
-                if implementing training_step, return whatever you need in that step
-        """
-        raise NotImplementedError
-
-    def training_end(self, *args, **kwargs):
-        """
-        return loss, dict with metrics for tqdm
-        :param called with outputs of training_step
-        :return: dict with loss key and optional log, progress keys
-        """
-        pass
-
-    def validation_step(self, *args, **kwargs):
-        """
-        return whatever outputs will need to be aggregated in validation_end
-        OPTIONAL
-        :param called with batch, batch_nb
-        additional: dataset_i if multiple val datasets used
-        :return:
-        """
-        pass
-
-    def test_step(self, *args, **kwargs):
-        """
-        return whatever outputs will need to be aggregated in test_end
-        OPTIONAL
-        :param called with batch, batch_nb
-        additional: dataset_i if multiple val datasets used
-        :return:
-        """
-        pass
-
-    def validation_end(self, outputs):
-        """
-        Outputs has the appended output after each validation step
-        OPTIONAL
-        :param outputs:
-        :return: dic_with_metrics for tqdm
-        """
-        pass
-
-    def test_end(self, outputs):
-        """
-        Outputs has the appended output after each test step
-        OPTIONAL
-        :param outputs:
-        :return: dic_with_metrics for tqdm
-        """
-        pass
-
-    def configure_ddp(self, model, device_ids):
-        """
-        Override to init DDP in a different way or use your own wrapper.
-        Must return model.
-        :param model:
-        :param device_ids:
-        :return: DDP wrapped model
-        """
-        model = LightningDistributedDataParallel(
-            model,
-            device_ids=device_ids,
-            find_unused_parameters=True
-        )
-        return model
-
-    def init_ddp_connection(self, proc_rank, world_size):
-        """
-        Connect all procs in the world using the env:// init
-        Use the first node as the root address
-        """
-
-        # use slurm job id for the port number
-        # guarantees unique ports across jobs from same grid search
-        try:
-            # use the last 4 numbers in the job id as the id
-            default_port = os.environ['SLURM_JOB_ID']
-            default_port = default_port[-4:]
-
-            # all ports should be in the 10k+ range
-            default_port = int(default_port) + 15000
-
-        except Exception as e:
-            default_port = 12910
-
-        # if user gave a port number, use that one instead
-        try:
-            default_port = os.environ['MASTER_PORT']
-        except Exception:
-            os.environ['MASTER_PORT'] = str(default_port)
-
-        # figure out the root node addr
-        try:
-            root_node = os.environ['SLURM_NODELIST'].split(' ')[0]
-        except Exception:
-            root_node = '127.0.0.2'
-
-        root_node = self.trainer.resolve_root_node_address(root_node)
-        os.environ['MASTER_ADDR'] = root_node
-        dist.init_process_group('nccl', rank=proc_rank, world_size=world_size)
-
-    def configure_apex(self, amp, model, optimizers, amp_level):
-        """
-        Override to init AMP your own way
-        Must return a model and list of optimizers
-        :param amp:
-        :param model:
-        :param optimizers:
-        :param amp_level:
-        :return: Apex wrapped model and optimizers
-        """
-        model, optimizers = amp.initialize(
-            model, optimizers, opt_level=amp_level,
-        )
-
-        return model, optimizers
-
-    def configure_optimizers(self):
-        """
-        Return a list of optimizers and a list of schedulers (could be empty)
-        :return:
-        """
-        raise NotImplementedError
-
-    def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i, second_order_closure=None):
-        """
-        Do something instead of the standard optimizer behavior
-        :param epoch_nb:
-        :param batch_nb:
-        :param optimizer:
-        :param optimizer_i:
-        :param second_order_closure: closure for second order methods
-        :return:
-        """
-        if isinstance(optimizer, torch.optim.LBFGS):
-            optimizer.step(second_order_closure)
-        else:
-            optimizer.step()
-
-        # clear gradients
-        optimizer.zero_grad()
-
-    def tbptt_split_batch(self, batch, split_size):
-        """
-        Return list of batch splits. Each split will be passed to forward_step to enable truncated
-        back propagation through time. The default implementation splits root level Tensors and
-        Sequences at dim=1 (i.e. time dim). It assumes that each time dim is the same length.
-        :return:
-        """
-        time_dims = [len(x[0]) for x in batch if isinstance(
-            x, torch.Tensor) or isinstance(x, collections.Sequence)]
-        assert len(time_dims) >= 1, "Unable to determine batch time dimension"
-        assert all(x == time_dims[0] for x in time_dims), "Batch time dimension length is ambiguous"
-
-        splits = []
-        for t in range(0, time_dims[0], split_size):
-            batch_split = []
-            for i, x in enumerate(batch):
-                if isinstance(x, torch.Tensor):
-                    split_x = x[:, t:t + split_size]
-                elif isinstance(x, collections.Sequence):
-                    split_x = [None] * len(x)
-                    for batch_idx in range(len(x)):
-                        split_x[batch_idx] = x[batch_idx][t:t + split_size]
-
-                batch_split.append(split_x)
-
-            splits.append(batch_split)
-
-        return splits
-
-    @data_loader
-    def tng_dataloader(self):
-        """
-        Implement a PyTorch DataLoader
-        * Deprecated in v0.5.0. use train_dataloader instead. *
-        :return:
-        """
-        raise NotImplementedError
-
-    @data_loader
-    def train_dataloader(self):
-        """
-        Implement a PyTorch DataLoader
-        :return:
-        """
-        #
-        try:
-            output = self.tng_dataloader()
-            warnings.warn("tng_dataloader has been renamed to train_dataloader since v0.5.0",
-                          DeprecationWarning)
-            return output
-        except NotImplementedError:
-            raise NotImplementedError
-
-    @data_loader
-    def test_dataloader(self):
-        """
-        Implement a PyTorch DataLoader
-        :return:
-        """
-        return None
-
-    @data_loader
-    def val_dataloader(self):
-        """
-        Implement a PyTorch DataLoader
-        :return:
-        """
-        return None
-
-    @classmethod
-    def load_from_metrics(cls, weights_path, tags_csv):
-        """
-        Primary way of loading model from csv weights path
-        :param weights_path:
-        :param tags_csv:
-        :param map_location: dic for mapping storage {'cuda:1':'cuda:0'}
-        :return:
-        """
-        hparams = load_hparams_from_tags_csv(tags_csv)
-        hparams.__setattr__('on_gpu', False)
-
-        # load on CPU only to avoid OOM issues
-        # then its up to user to put back on GPUs
-        checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)
-
-        # load the state_dict on the model automatically
-        model = cls(hparams)
-        model.load_state_dict(checkpoint['state_dict'])
-
-        # give model a chance to load something
-        model.on_load_checkpoint(checkpoint)
-
-        return model
-
-    @classmethod
-    def load_from_checkpoint(cls, checkpoint_path):
-        """
-        Primary way of loading model from a checkpoint
-        :param checkpoint_path:
-        :param map_location: dic for mapping storage {'cuda:1':'cuda:0'}
-        :return:
-        """
-
-        # load on CPU only to avoid OOM issues
-        # then its up to user to put back on GPUs
-        checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
-        try:
-            ckpt_hparams = checkpoint['hparams']
-        except KeyError:
-            raise IOError(
-                "Checkpoint does not contain hyperparameters. Are your model hyperparameters stored"
-                "in self.hparams?"
-            )
-        hparams = Namespace(**ckpt_hparams)
-
-        # load the state_dict on the model automatically
-        model = cls(hparams)
-        model.load_state_dict(checkpoint['state_dict'])
-
-        # give model a chance to load something
-        model.on_load_checkpoint(checkpoint)
-
-        return model
-
-    def summarize(self, mode):
-        model_summary = ModelSummary(self, mode=mode)
-        logging.info(model_summary)
-
-    def freeze(self):
-        for param in self.parameters():
-            param.requires_grad = False
-
-        self.eval()
-
-    def unfreeze(self):
-        for param in self.parameters():
-            param.requires_grad = True
-
-        self.train()

pytorch_lightning/testing/__init__.py

@@ -1,6 +1,6 @@
-from .lm_test_module import LightningTestModel
-from .lm_test_module_base import LightningTestModelBase
-from .lm_test_module_mixins import (
+from .model import LightningTestModel
+from .model_base import LightningTestModelBase
+from .model_mixins import (
     LightningValidationStepMixin,
     LightningValidationMixin,
     LightningValidationStepMultipleDataloadersMixin,

pytorch_lightning/testing/model.py

@@ -1,7 +1,7 @@
 import torch
 
-from .lm_test_module_base import LightningTestModelBase
-from .lm_test_module_mixins import LightningValidationMixin, LightningTestMixin
+from .model_base import LightningTestModelBase
+from .model_mixins import LightningValidationMixin, LightningTestMixin
 
 
 class LightningTestModel(LightningValidationMixin, LightningTestMixin, LightningTestModelBase):

pytorch_lightning/testing/model_base.py

@@ -16,7 +16,23 @@ except ImportError:
     raise ImportError('Missing test-tube package.')
 
 from pytorch_lightning import data_loader
-from pytorch_lightning.root_module.root_module import LightningModule
+from pytorch_lightning.core.lightning import LightningModule
+
+
+class TestingMNIST(MNIST):
+
+    def __init__(self, root, train=True, transform=None, target_transform=None,
+                 download=False, num_samples=8000):
+        super(TestingMNIST, self).__init__(
+            root,
+            train=train,
+            transform=transform,
+            target_transform=target_transform,
+            download=download
+        )
+        # take just a subset of MNIST dataset
+        self.data = self.data[:num_samples]
+        self.targets = self.targets[:num_samples]
 
 
 class LightningTestModelBase(LightningModule):
@@ -105,7 +121,7 @@ class LightningTestModelBase(LightningModule):
             loss_val = loss_val.unsqueeze(0)
 
         # alternate possible outputs to test
-        if self.trainer.batch_nb % 1 == 0:
+        if self.trainer.batch_idx % 1 == 0:
             output = OrderedDict({
                 'loss': loss_val,
                 'progress_bar': {'some_val': loss_val * loss_val},
@@ -113,7 +129,7 @@ class LightningTestModelBase(LightningModule):
             })
 
             return output
-        if self.trainer.batch_nb % 2 == 0:
+        if self.trainer.batch_idx % 2 == 0:
             return loss_val
 
     # ---------------------
@@ -137,8 +153,8 @@ class LightningTestModelBase(LightningModule):
         # init data generators
         transform = transforms.Compose([transforms.ToTensor(),
                                         transforms.Normalize((0.5,), (1.0,))])
-        dataset = MNIST(root=self.hparams.data_root, train=train,
-                        transform=transform, download=True)
+        dataset = TestingMNIST(root=self.hparams.data_root, train=train,
+                               transform=transform, download=True, num_samples=2000)
 
         # when using multi-node we need to add the datasampler
         train_sampler = None

pytorch_lightning/trainer/__init__.py

@@ -0,0 +1,19 @@
+"""
+# Trainer
+
+The lightning trainer abstracts best practices for running a training, val, test routine.
+ It calls parts of your model when it wants to hand over full control and otherwise makes
+ training assumptions which are now standard practice in AI research.
+
+This is the basic use of the trainer:
+
+.. code-block:: python
+
+    from pytorch_lightning import Trainer
+
+    model = LightningTemplate()
+
+    trainer = Trainer()
+    trainer.fit(model)
+
+"""

pytorch_lightning/trainer/auto_mix_precision.py

@@ -1,3 +1,5 @@
+from abc import ABC
+
 try:
     from apex import amp
 
@@ -7,7 +9,7 @@ except ImportError:
 import logging
 
 
-class TrainerAMPMixin(object):
+class TrainerAMPMixin(ABC):
 
     def init_amp(self, use_amp):
         self.use_amp = use_amp and APEX_AVAILABLE
@@ -16,7 +18,7 @@ class TrainerAMPMixin(object):
 
         if use_amp and not APEX_AVAILABLE:  # pragma: no cover
             msg = """
-            You set use_amp=True but do not have apex installed.
+            You set `use_amp=True` but do not have apex installed.
             Install apex first using this guide and rerun with use_amp=True:
             https://github.com/NVIDIA/apex#linux
 

pytorch_lightning/trainer/callback_config.py

@@ -1,10 +1,19 @@
 import os
+from abc import ABC
 
 from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
-from pytorch_lightning.logging import TestTubeLogger
+from pytorch_lightning.logging import TensorboardLogger
 
 
-class TrainerCallbackConfigMixin(object):
+class TrainerCallbackConfigMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.default_save_path = None
+        self.save_checkpoint = None
+        self.slurm_job_id = None
+
     def configure_checkpoint_callback(self):
         """
         Weight path set in this priority:
@@ -60,7 +69,7 @@ class TrainerCallbackConfigMixin(object):
         # configure logger
         if logger is True:
             # default logger
-            self.logger = TestTubeLogger(
+            self.logger = TensorboardLogger(
                 save_dir=self.default_save_path,
                 version=self.slurm_job_id,
                 name='lightning_logs'

pytorch_lightning/trainer/data_loading.py

@@ -1,7 +1,18 @@
 import warnings
+from abc import ABC
 
 import torch.distributed as dist
-from torch.utils.data import IterableDataset
+try:
+    # loading for pyTorch 1.3
+    from torch.utils.data import IterableDataset
+except ImportError:
+    # loading for pyTorch 1.1
+    import torch
+    warnings.warn('Your version of pyTorch %s does not support `IterableDataset`,'
+                  ' please upgrade to 1.2+' % torch.__version__, ImportWarning)
+    EXIST_ITER_DATASET = False
+else:
+    EXIST_ITER_DATASET = True
 from torch.utils.data.distributed import DistributedSampler
 
 from pytorch_lightning.utilities.debugging import MisconfigurationException
@@ -14,7 +25,26 @@ except ImportError:
     APEX_AVAILABLE = False
 
 
-class TrainerDataLoadingMixin(object):
+class TrainerDataLoadingMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.proc_rank = None
+        self.use_ddp = None
+        self.use_ddp2 = None
+        self.shown_warnings = None
+        self.val_check_interval = None
+
+    def _percent_range_check(self, name):
+        value = getattr(self, name)
+        msg = f"`{name}` must lie in the range [0.0, 1.0], but got {value:.3f}."
+        if name == "val_check_interval":
+            msg += " If you want to disable validation set `val_percent_check` to 0.0 instead."
+
+        if not 0. <= value <= 1.:
+            raise ValueError(msg)
+
     def init_train_dataloader(self, model):
         """
         Dataloaders are provided by the model
@@ -24,19 +54,28 @@ class TrainerDataLoadingMixin(object):
         self.get_train_dataloader = model.train_dataloader
 
         # determine number of training batches
-        if isinstance(self.get_train_dataloader(), IterableDataset):
-            self.nb_training_batches = float('inf')
+        if EXIST_ITER_DATASET and isinstance(self.get_train_dataloader().dataset, IterableDataset):
+            self.num_training_batches = float('inf')
         else:
-            self.nb_training_batches = len(self.get_train_dataloader())
-            self.nb_training_batches = int(self.nb_training_batches * self.train_percent_check)
+            self._percent_range_check('train_percent_check')
+
+            self.num_training_batches = len(self.get_train_dataloader())
+            self.num_training_batches = int(self.num_training_batches * self.train_percent_check)
 
         # determine when to check validation
         # if int passed in, val checks that often
         # otherwise, it checks in [0, 1.0] % range of a training epoch
         if isinstance(self.val_check_interval, int):
             self.val_check_batch = self.val_check_interval
+            if self.val_check_batch > self.num_training_batches:
+                raise ValueError(
+                    f"`val_check_interval` ({self.val_check_interval}) must be less than or equal "
+                    f"to the number of the training batches ({self.num_training_batches}). "
+                    f"If you want to disable validation set `val_percent_check` to 0.0 instead.")
         else:
-            self.val_check_batch = int(self.nb_training_batches * self.val_check_interval)
+            self._percent_range_check('val_check_interval')
+
+            self.val_check_batch = int(self.num_training_batches * self.val_check_interval)
             self.val_check_batch = max(1, self.val_check_batch)
 
         on_ddp = self.use_ddp or self.use_ddp2
@@ -68,13 +107,15 @@ class TrainerDataLoadingMixin(object):
         :return:
         """
         self.get_val_dataloaders = model.val_dataloader
+        self.num_val_batches = 0
 
         # determine number of validation batches
         # val datasets could be none, 1 or 2+
         if self.get_val_dataloaders() is not None:
-            self.nb_val_batches = sum(len(dataloader) for dataloader in self.get_val_dataloaders())
-            self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
-            self.nb_val_batches = max(1, self.nb_val_batches)
+            self._percent_range_check('val_percent_check')
+
+            self.num_val_batches = sum(len(dataloader) for dataloader in self.get_val_dataloaders())
+            self.num_val_batches = int(self.num_val_batches * self.val_percent_check)
 
         on_ddp = self.use_ddp or self.use_ddp2
         if on_ddp and self.get_val_dataloaders() is not None:
@@ -104,40 +145,42 @@ class TrainerDataLoadingMixin(object):
                     break
 
     def init_test_dataloader(self, model):
-        """
-        Dataloaders are provided by the model
+        """Dataloaders are provided by the model.
+
         :param model:
-        :return:
         """
 
         self.get_test_dataloaders = model.test_dataloader
 
         # determine number of test batches
         if self.get_test_dataloaders() is not None:
+            self._percent_range_check('test_percent_check')
+
             len_sum = sum(len(dataloader) for dataloader in self.get_test_dataloaders())
-            self.nb_test_batches = len_sum
-            self.nb_test_batches = int(self.nb_test_batches * self.test_percent_check)
-            self.nb_test_batches = max(1, self.nb_test_batches)
+            self.num_test_batches = len_sum
+            self.num_test_batches = int(self.num_test_batches * self.test_percent_check)
 
         on_ddp = self.use_ddp or self.use_ddp2
         if on_ddp and self.get_test_dataloaders() is not None:
             for dataloader in self.get_test_dataloaders():
                 if not isinstance(dataloader.sampler, DistributedSampler):
                     msg = """
-                    Your test_dataloader(s) don't use DistributedSampler.
+                    Your `test_dataloader(s)` don't use DistributedSampler.
 
                     You're using multiple gpus and multiple nodes without using a
                     DistributedSampler to assign a subset of your data to each process.
                     To silence this warning, pass a DistributedSampler to your DataLoader.
 
-                    ie: this:
-                    dataset = myDataset()
-                    dataloader = Dataloader(dataset)
+                    ie: this::
 
-                    becomes:
-                    dataset = myDataset()
-                    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-                    dataloader = Dataloader(dataset, sampler=dist_sampler)
+                        dataset = myDataset()
+                        dataloader = Dataloader(dataset)
+
+                    becomes::
+
+                        dataset = myDataset()
+                        dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+                        dataloader = Dataloader(dataset, sampler=dist_sampler)
 
                     If you want each process to load the full dataset, ignore this warning.
                     """
@@ -167,11 +210,12 @@ class TrainerDataLoadingMixin(object):
             self.get_val_dataloaders()
 
         # support IterableDataset for train data
-        self.is_iterable_train_dataloader = isinstance(self.get_train_dataloader(), IterableDataset)
+        self.is_iterable_train_dataloader = (
+            EXIST_ITER_DATASET and isinstance(self.get_train_dataloader().dataset, IterableDataset))
         if self.is_iterable_train_dataloader and not isinstance(self.val_check_interval, int):
             m = '''
-            When using an iterableDataset for train_dataloader,
-            Trainer(val_check_interval) must be an int.
+            When using an iterableDataset for `train_dataloader`,
+            `Trainer(val_check_interval)` must be an int.
             An int k specifies checking validation every k training batches
             '''
             raise MisconfigurationException(m)
@@ -185,6 +229,10 @@ class TrainerDataLoadingMixin(object):
         self.val_percent_check = val_percent_check
         self.test_percent_check = test_percent_check
         if overfit_pct > 0:
+            if overfit_pct > 1:
+                raise ValueError(f"`overfit_pct` must be not greater than 1.0, but got "
+                                 f"{overfit_pct:.3f}.")
+
             self.train_percent_check = overfit_pct
             self.val_percent_check = overfit_pct
             self.test_percent_check = overfit_pct

pytorch_lightning/trainer/ddp_mixin.py

@@ -1,193 +0,0 @@
-import os
-import re
-import warnings
-import logging
-
-import torch
-
-from pytorch_lightning.utilities.debugging import MisconfigurationException
-
-try:
-    from apex import amp
-
-    APEX_AVAILABLE = True
-except ImportError:
-    APEX_AVAILABLE = False
-
-
-class TrainerDDPMixin(object):
-    def set_distributed_mode(self, distributed_backend, nb_gpu_nodes):
-        # skip for CPU
-        if self.num_gpus == 0:
-            return
-
-        # single GPU case
-        # in single gpu case we allow ddp so we can train on multiple
-        # nodes, 1 gpu per node
-        if self.num_gpus == 1:
-            self.single_gpu = True
-
-            if distributed_backend is not None:
-                self.use_dp = distributed_backend == 'dp'
-                self.use_ddp = distributed_backend == 'ddp'
-                self.use_ddp2 = distributed_backend == 'ddp2'
-
-                # disable single gpu when using ddp2
-                if self.use_ddp2:
-                    self.single_gpu = False
-
-        # multiple GPU case
-        elif self.num_gpus > 1:
-            if distributed_backend is not None:
-                # DP, DDP case
-                self.use_dp = distributed_backend == 'dp'
-                self.use_ddp = distributed_backend == 'ddp'
-                self.use_ddp2 = distributed_backend == 'ddp2'
-
-            elif distributed_backend is None:
-                m = 'When using multiple GPUs set ' \
-                    'Trainer(distributed_backend=dp) (or ddp)'
-                raise MisconfigurationException(m)
-
-        # throw error to force user ddp or ddp2 choice
-        if nb_gpu_nodes > 1 and not (self.use_ddp2 or self.use_ddp):  # pragma: no cover
-            w = 'DataParallel does not support nb_gpu_nodes > 1. ' \
-                'Switching to DistributedDataParallel for you. ' \
-                'To silence this warning set distributed_backend=ddp' \
-                'or distributed_backend=ddp2'
-            raise MisconfigurationException(w)
-
-        logging.info(f'gpu available: {torch.cuda.is_available()}, used: {self.on_gpu}')
-
-    def configure_slurm_ddp(self, nb_gpu_nodes):
-        self.is_slurm_managing_tasks = False
-
-        # extract SLURM flag vars
-        # whenever we have the correct number of tasks, we let slurm manage processes
-        # otherwise we launch the required number of processes
-        if self.use_ddp:
-            self.nb_requested_gpus = self.num_gpus * nb_gpu_nodes
-            self.nb_slurm_tasks = 0
-            try:
-                self.nb_slurm_tasks = int(os.environ['SLURM_NTASKS'])
-                self.is_slurm_managing_tasks = self.nb_slurm_tasks == self.nb_requested_gpus
-
-                # in interactive mode we don't manage tasks
-                job_name = os.environ['SLURM_JOB_NAME']
-                if job_name == 'bash':
-                    self.is_slurm_managing_tasks = False
-
-            except Exception:
-                # likely not on slurm, so set the slurm managed flag to false
-                self.is_slurm_managing_tasks = False
-
-        # used for tests only, set this flag to simulate slurm managing a task
-        try:
-            should_fake = int(os.environ['FAKE_SLURM_MANAGING_TASKS'])
-            if should_fake:
-                self.is_slurm_managing_tasks = True
-        except Exception as e:
-            pass
-
-    def set_nvidia_flags(self, is_slurm_managing_tasks, data_parallel_device_ids):
-        if data_parallel_device_ids is None:
-            return
-
-        # set the correct cuda visible devices (using pci order)
-        os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-
-        # when slurm is managing the task it sets the visible devices
-        if not is_slurm_managing_tasks:
-            if type(data_parallel_device_ids) is int:
-                id_str = ','.join(str(x) for x in list(range(data_parallel_device_ids)))
-                os.environ["CUDA_VISIBLE_DEVICES"] = id_str
-            else:
-                gpu_str = ','.join([str(x) for x in data_parallel_device_ids])
-                os.environ["CUDA_VISIBLE_DEVICES"] = gpu_str
-
-        logging.info(f'VISIBLE GPUS: {os.environ["CUDA_VISIBLE_DEVICES"]}')
-
-    def ddp_train(self, gpu_nb, model):
-        """
-        Entry point into a DP thread
-        :param gpu_nb:
-        :param model:
-        :param cluster_obj:
-        :return:
-        """
-        # node rank using relative slurm id
-        # otherwise default to node rank 0
-        try:
-            node_id = os.environ['SLURM_NODEID']
-            self.node_rank = int(node_id)
-        except Exception:
-            self.node_rank = 0
-
-        # show progressbar only on progress_rank 0
-        self.show_progress_bar = self.show_progress_bar and self.node_rank == 0 and gpu_nb == 0
-
-        # determine which process we are and world size
-        if self.use_ddp:
-            self.proc_rank = self.node_rank * self.num_gpus + gpu_nb
-            self.world_size = self.nb_gpu_nodes * self.num_gpus
-
-        elif self.use_ddp2:
-            self.proc_rank = self.node_rank
-            self.world_size = self.nb_gpu_nodes
-
-        # let the exp know the rank to avoid overwriting logs
-        if self.logger is not None:
-            self.logger.rank = self.proc_rank
-
-        # set up server using proc 0's ip address
-        # try to init for 20 times at max in case ports are taken
-        # where to store ip_table
-        model.trainer = self
-        model.init_ddp_connection(self.proc_rank, self.world_size)
-
-        # CHOOSE OPTIMIZER
-        # allow for lr schedulers as well
-        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
-
-        # MODEL
-        # copy model to each gpu
-        if self.distributed_backend == 'ddp':
-            torch.cuda.set_device(gpu_nb)
-        model.cuda(gpu_nb)
-
-        # set model properties before going into wrapper
-        self.copy_trainer_model_properties(model)
-
-        # override root GPU
-        self.root_gpu = gpu_nb
-
-        # AMP
-        # run through amp wrapper before going to distributed DP
-        if self.use_amp:
-            # An example
-            model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
-            self.optimizers = optimizers
-
-        # DDP2 uses all GPUs on the machine
-        if self.distributed_backend == 'ddp':
-            device_ids = [gpu_nb]
-        elif self.use_ddp2:
-            device_ids = self.data_parallel_device_ids
-
-        # allow user to configure ddp
-        model = model.configure_ddp(model, device_ids)
-
-        # continue training routine
-        self.run_pretrain_routine(model)
-
-    def resolve_root_node_address(self, root_node):
-        if '[' in root_node:
-            name = root_node.split('[')[0]
-            number = root_node.split(',')[0]
-            if '-' in number:
-                number = number.split('-')[0]
-
-            number = re.sub('[^0-9]', '', number)
-            root_node = name + number
-
-        return root_node

pytorch_lightning/trainer/distrib_data_parallel.py

@@ -0,0 +1,343 @@
+"""
+Lightning supports model training on a cluster managed by SLURM in the following cases:
+
+1. Training on a single cpu or single GPU.
+2. Train on multiple GPUs on the same node using DataParallel or DistributedDataParallel
+3. Training across multiple GPUs on multiple different nodes via DistributedDataParallel.
+
+.. note:: A node means a machine with multiple GPUs
+
+Running grid search on a cluster
+--------------------------------
+
+To use lightning to run a hyperparameter search (grid-search or random-search) on a cluster do 4 things:
+
+(1). Define the parameters for the grid search
+
+.. code-block:: python
+
+    from test_tube import HyperOptArgumentParser
+
+    # subclass of argparse
+    parser = HyperOptArgumentParser(strategy='random_search')
+    parser.add_argument('--learning_rate', default=0.002, type=float, help='the learning rate')
+
+    # let's enable optimizing over the number of layers in the network
+    parser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4, 8])
+
+    hparams = parser.parse_args()
+
+.. note:: You must set `Tunable=True` for that argument to be considered in the permutation set.
+ Otherwise test-tube will use the default value. This flag is useful when you don't want
+ to search over an argument and want to use the default instead.
+
+(2). Define the cluster options in the
+ `SlurmCluster object <https://williamfalcon.github.io/test-tube/hpc/SlurmCluster>`_ (over 5 nodes and 8 gpus)
+
+.. code-block:: python
+
+    from test_tube.hpc import SlurmCluster
+
+    # hyperparameters is a test-tube hyper params object
+    # see https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/
+    hyperparams = args.parse()
+
+    # init cluster
+    cluster = SlurmCluster(
+        hyperparam_optimizer=hyperparams,
+        log_path='/path/to/log/results/to',
+        python_cmd='python3'
+    )
+
+    # let the cluster know where to email for a change in job status (ie: complete, fail, etc...)
+    cluster.notify_job_status(email='some@email.com', on_done=True, on_fail=True)
+
+    # set the job options. In this instance, we'll run 20 different models
+    # each with its own set of hyperparameters giving each one 1 GPU (ie: taking up 20 GPUs)
+    cluster.per_experiment_nb_gpus = 8
+    cluster.per_experiment_nb_nodes = 5
+
+    # we'll request 10GB of memory per node
+    cluster.memory_mb_per_node = 10000
+
+    # set a walltime of 10 minues
+    cluster.job_time = '10:00'
+
+
+(3). Make a main function with your model and trainer. Each job will call this function with a particular
+hparams configuration.::
+
+    from pytorch_lightning import Trainer
+
+    def train_fx(trial_hparams, cluster_manager, _):
+        # hparams has a specific set of hyperparams
+
+        my_model = MyLightningModel()
+
+        # give the trainer the cluster object
+        trainer = Trainer()
+        trainer.fit(my_model)
+
+    `
+
+(4). Start the grid/random search::
+
+    # run the models on the cluster
+    cluster.optimize_parallel_cluster_gpu(
+        train_fx,
+        nb_trials=20,
+        job_name='my_grid_search_exp_name',
+        job_display_name='my_exp')
+
+.. note:: `nb_trials` specifies how many of the possible permutations to use. If using `grid_search` it will use
+ the depth first ordering. If using `random_search` it will use the first k shuffled options. FYI, random search
+ has been shown to be just as good as any Bayesian optimization method when using a reasonable number of samples (60),
+ see this `paper <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`_  for more information.
+
+Walltime auto-resubmit
+----------------------
+
+Lightning automatically resubmits jobs when they reach the walltime. Make sure to set the SIGUSR1 signal in
+your SLURM script.::
+
+    # 90 seconds before training ends
+    #SBATCH --signal=SIGUSR1@90
+
+When lightning receives the SIGUSR1 signal it will:
+1. save a checkpoint with 'hpc_ckpt' in the name.
+2. resubmit the job using the SLURM_JOB_ID
+
+When the script starts again, Lightning will:
+1. search for a 'hpc_ckpt' checkpoint.
+2. restore the model, optimizers, schedulers, epoch, etc...
+
+"""
+
+import os
+import re
+import logging
+import warnings
+from abc import ABC, abstractmethod
+
+import torch
+
+from pytorch_lightning.utilities.debugging import MisconfigurationException
+
+try:
+    from apex import amp
+
+    APEX_AVAILABLE = True
+except ImportError:
+    APEX_AVAILABLE = False
+
+
+class TrainerDDPMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.num_gpus = None
+        self.on_gpu = None
+        self.num_gpu_nodes = None
+        self.logger = None
+        self.data_parallel_device_ids = None
+        self.distributed_backend = None
+        self.use_amp = None
+        self.amp_level = None
+
+    @abstractmethod
+    def copy_trainer_model_properties(self, model):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def run_pretrain_routine(self, model):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def init_optimizers(self, optimizers):
+        # this is just empty shell for code from other class
+        pass
+
+    def set_distributed_mode(self, distributed_backend, num_gpu_nodes):
+        # skip for CPU
+        if self.num_gpus == 0:
+            return
+
+        # single GPU case
+        # in single gpu case we allow ddp so we can train on multiple
+        # nodes, 1 gpu per node
+        if self.num_gpus == 1:
+            self.single_gpu = True
+
+            if distributed_backend is not None:
+                self.use_dp = distributed_backend == 'dp'
+                self.use_ddp = distributed_backend == 'ddp'
+                self.use_ddp2 = distributed_backend == 'ddp2'
+
+                # disable single gpu when using ddp2
+                if self.use_ddp2:
+                    self.single_gpu = False
+
+        # multiple GPU case
+        elif self.num_gpus > 1:
+            if distributed_backend is not None:
+                # DP, DDP case
+                self.use_dp = distributed_backend == 'dp'
+                self.use_ddp = distributed_backend == 'ddp'
+                self.use_ddp2 = distributed_backend == 'ddp2'
+
+            elif distributed_backend is None:
+                m = 'You requested multiple GPUs but did not specify a backend' \
+                    'Trainer(distributed_backend=dp) (or ddp, ddp2)' \
+                    'Setting distributed_backend=dp for you'
+                warnings.warn(m)
+                self.use_dp = True
+                self.use_ddp = False
+                self.use_ddp2 = False
+
+        # throw error to force user ddp or ddp2 choice
+        if num_gpu_nodes > 1 and not (self.use_ddp2 or self.use_ddp):  # pragma: no cover
+            w = 'DataParallel does not support num_nodes > 1. ' \
+                'Switching to DistributedDataParallel for you. ' \
+                'To silence this warning set distributed_backend=ddp' \
+                'or distributed_backend=ddp2'
+            raise MisconfigurationException(w)
+
+        logging.info(f'gpu available: {torch.cuda.is_available()}, used: {self.on_gpu}')
+
+    def configure_slurm_ddp(self, num_gpu_nodes):
+        self.is_slurm_managing_tasks = False
+
+        # extract SLURM flag vars
+        # whenever we have the correct number of tasks, we let slurm manage processes
+        # otherwise we launch the required number of processes
+        if self.use_ddp:
+            self.num_requested_gpus = self.num_gpus * num_gpu_nodes
+            self.num_slurm_tasks = 0
+            try:
+                self.num_slurm_tasks = int(os.environ['SLURM_NTASKS'])
+                self.is_slurm_managing_tasks = self.num_slurm_tasks == self.num_requested_gpus
+
+                # in interactive mode we don't manage tasks
+                job_name = os.environ['SLURM_JOB_NAME']
+                if job_name == 'bash':
+                    self.is_slurm_managing_tasks = False
+
+            except Exception:
+                # likely not on slurm, so set the slurm managed flag to false
+                self.is_slurm_managing_tasks = False
+
+        # used for tests only, set this flag to simulate slurm managing a task
+        try:
+            should_fake = int(os.environ['FAKE_SLURM_MANAGING_TASKS'])
+            if should_fake:
+                self.is_slurm_managing_tasks = True
+        except Exception as e:
+            pass
+
+    def set_nvidia_flags(self, is_slurm_managing_tasks, data_parallel_device_ids):
+        if data_parallel_device_ids is None:
+            return
+
+        # set the correct cuda visible devices (using pci order)
+        os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+
+        # when slurm is managing the task it sets the visible devices
+        if not is_slurm_managing_tasks:
+            if type(data_parallel_device_ids) is int:
+                id_str = ','.join(str(x) for x in list(range(data_parallel_device_ids)))
+                os.environ["CUDA_VISIBLE_DEVICES"] = id_str
+            else:
+                gpu_str = ','.join([str(x) for x in data_parallel_device_ids])
+                os.environ["CUDA_VISIBLE_DEVICES"] = gpu_str
+
+        logging.info(f'VISIBLE GPUS: {os.environ["CUDA_VISIBLE_DEVICES"]}')
+
+    def ddp_train(self, gpu_idx, model):
+        """
+        Entry point into a DP thread
+        :param gpu_idx:
+        :param model:
+        :param cluster_obj:
+        :return:
+        """
+        # node rank using relative slurm id
+        # otherwise default to node rank 0
+        try:
+            node_id = os.environ['SLURM_NODEID']
+            self.node_rank = int(node_id)
+        except Exception:
+            self.node_rank = 0
+
+        # show progressbar only on progress_rank 0
+        self.show_progress_bar = self.show_progress_bar and self.node_rank == 0 and gpu_idx == 0
+
+        # determine which process we are and world size
+        if self.use_ddp:
+            self.proc_rank = self.node_rank * self.num_gpus + gpu_idx
+            self.world_size = self.num_gpu_nodes * self.num_gpus
+
+        elif self.use_ddp2:
+            self.proc_rank = self.node_rank
+            self.world_size = self.num_gpu_nodes
+
+        # let the exp know the rank to avoid overwriting logs
+        if self.logger is not None:
+            self.logger.rank = self.proc_rank
+
+        # set up server using proc 0's ip address
+        # try to init for 20 times at max in case ports are taken
+        # where to store ip_table
+        model.trainer = self
+        model.init_ddp_connection(self.proc_rank, self.world_size)
+
+        # CHOOSE OPTIMIZER
+        # allow for lr schedulers as well
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
+
+        # MODEL
+        # copy model to each gpu
+        if self.distributed_backend == 'ddp':
+            torch.cuda.set_device(gpu_idx)
+        model.cuda(gpu_idx)
+
+        # set model properties before going into wrapper
+        self.copy_trainer_model_properties(model)
+
+        # override root GPU
+        self.root_gpu = gpu_idx
+
+        # AMP
+        # run through amp wrapper before going to distributed DP
+        if self.use_amp:
+            # An example
+            model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
+            self.optimizers = optimizers
+
+        # DDP2 uses all GPUs on the machine
+        if self.distributed_backend == 'ddp':
+            device_ids = [gpu_idx]
+        elif self.use_ddp2:
+            device_ids = self.data_parallel_device_ids
+        else:
+            device_ids = None
+
+        # allow user to configure ddp
+        model = model.configure_ddp(model, device_ids)
+
+        # continue training routine
+        self.run_pretrain_routine(model)
+
+    def resolve_root_node_address(self, root_node):
+        if '[' in root_node:
+            name = root_node.split('[')[0]
+            number = root_node.split(',')[0]
+            if '-' in number:
+                number = number.split('-')[0]
+
+            number = re.sub('[^0-9]', '', number)
+            root_node = name + number
+
+        return root_node

pytorch_lightning/trainer/distrib_parts.py

@@ -0,0 +1,584 @@
+"""
+Lightning makes multi-gpu training and 16 bit training trivial.
+
+.. note:: None of the flags below require changing anything about your lightningModel definition.
+
+Choosing a backend
+==================
+
+Lightning supports two backends. DataParallel and DistributedDataParallel.
+ Both can be used for single-node multi-GPU training.
+ For multi-node training you must use DistributedDataParallel.
+
+DataParallel (dp)
+-----------------
+
+Splits a batch across multiple GPUs on the same node. Cannot be used for multi-node training.
+
+DistributedDataParallel (ddp)
+-----------------------------
+
+Trains a copy of the model on each GPU and only syncs gradients. If used with DistributedSampler, each GPU trains
+on a subset of the full dataset.
+
+DistributedDataParallel-2 (ddp2)
+--------------------------------
+
+Works like DDP, except each node trains a single copy of the model using ALL GPUs on that node.
+ Very useful when dealing with negative samples, etc...
+
+You can toggle between each mode by setting this flag.
+
+.. code-block:: python
+
+    # DEFAULT (when using single GPU or no GPUs)
+    trainer = Trainer(distributed_backend=None)
+
+    # Change to DataParallel (gpus > 1)
+    trainer = Trainer(distributed_backend='dp')
+
+    # change to distributed data parallel (gpus > 1)
+    trainer = Trainer(distributed_backend='ddp')
+
+    # change to distributed data parallel (gpus > 1)
+    trainer = Trainer(distributed_backend='ddp2')
+
+If you request multiple nodes, the back-end will auto-switch to ddp.
+ We recommend you use DistributedDataparallel even for single-node multi-GPU training.
+ It is MUCH faster than DP but *may* have configuration issues depending on your cluster.
+
+For a deeper understanding of what lightning is doing, feel free to read this
+ `guide <https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565>`_.
+
+Distributed and 16-bit precision
+--------------------------------
+
+Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
+ not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.
+
+Below are the possible configurations we support.
+
++-------+---------+----+-----+---------+------------------------------------------------------------+
+| 1 GPU | 1+ GPUs | DP | DDP | 16-bit  | command                                                    |
++=======+=========+====+=====+=========+============================================================+
+| Y     |         |    |     |         | `Trainer(gpus=1)`                                          |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+| Y     |         |    |     | Y       | `Trainer(gpus=1, use_amp=True)`                            |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+|       | Y       | Y  |     |         | `Trainer(gpus=k, distributed_backend='dp')`                |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+|       | Y       |    | Y   |         | `Trainer(gpus=k, distributed_backend='ddp')`               |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+|       | Y       |    | Y   | Y       | `Trainer(gpus=k, distributed_backend='ddp', use_amp=True)` |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+
+You also have the option of specifying which GPUs to use by passing a list:
+
+.. code-block:: python
+
+    # DEFAULT (int) specifies how many GPUs to use.
+    Trainer(gpus=k)
+
+    # Above is equivalent to
+    Trainer(gpus=list(range(k)))
+
+    # You specify which GPUs (don't use if running on cluster)
+    Trainer(gpus=[0, 1])
+
+    # can also be a string
+    Trainer(gpus='0, 1')
+
+    # can also be -1 or '-1', this uses all available GPUs
+    # this is equivalent to list(range(torch.cuda.available_devices()))
+    Trainer(gpus=-1)
+
+
+CUDA flags
+----------
+
+CUDA flags make certain GPUs visible to your script.
+ Lightning sets these for you automatically, there's NO NEED to do this yourself.
+
+.. code-block:: python
+
+    # lightning will set according to what you give the trainer
+    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+    os.environ["CUDA_VISIBLE_DEVICES"] = "0"
+
+
+However, when using a cluster, Lightning will NOT set these flags (and you should not either).
+ SLURM will set these for you.
+
+16-bit mixed precision
+----------------------
+
+16 bit precision can cut your memory footprint by half. If using volta architecture GPUs
+ it can give a dramatic training speed-up as well.
+ First, install apex (if install fails, look `here <https://github.com/NVIDIA/apex>`_::
+
+    $ git clone https://github.com/NVIDIA/apex
+    $ cd apex
+
+    # ------------------------
+    # OPTIONAL: on your cluster you might need to load cuda 10 or 9
+    # depending on how you installed PyTorch
+
+    # see available modules
+    module avail
+
+    # load correct cuda before install
+    module load cuda-10.0
+    # ------------------------
+
+    # make sure you've loaded a cuda version > 4.0 and < 7.0
+    module load gcc-6.1.0
+
+    $ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
+
+
+then set this use_amp to True.::
+
+    # DEFAULT
+    trainer = Trainer(amp_level='O2', use_amp=False)
+
+
+Single-gpu
+----------
+
+Make sure you're on a GPU machine.::
+
+    # DEFAULT
+    trainer = Trainer(gpus=1)
+
+Multi-gpu
+---------
+
+Make sure you're on a GPU machine. You can set as many GPUs as you want.
+ In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
+
+.. code-block:: python
+
+    # to use DataParallel
+    trainer = Trainer(gpus=8, distributed_backend='dp')
+
+    # RECOMMENDED use DistributedDataParallel
+    trainer = Trainer(gpus=8, distributed_backend='ddp')
+
+Custom device selection
+-----------------------
+
+The number of GPUs can also be selected with a list of indices or a string containing
+a comma separated list of GPU ids.
+The table below lists examples of possible input formats and how they are interpreted by Lightning.
+Note in particular the difference between `gpus=0`, `gpus=[0]` and `gpus="0"`.
+
++---------------+-----------+---------------------+---------------------------------+
+| `gpus`        | Type      | Parsed              | Meaning                         |
++===============+===========+=====================+=================================+
+| None          | NoneType  | None                | CPU                             |
++---------------+-----------+---------------------+---------------------------------+
+| 0             | int       | None                | CPU                             |
++---------------+-----------+---------------------+---------------------------------+
+| 3             | int       | [0, 1, 2]           | first 3 GPUs                    |
++---------------+-----------+---------------------+---------------------------------+
+| -1            | int       | [0, 1, 2, ...]      | all available GPUs              |
++---------------+-----------+---------------------+---------------------------------+
+| [0]           | list      | [0]                 | GPU 0                           |
++---------------+-----------+---------------------+---------------------------------+
+| [1, 3]        | list      | [1, 3]              | GPUs 1 and 3                    |
++---------------+-----------+---------------------+---------------------------------+
+| "0"           | str       | [0]                 | GPU 0                           |
++---------------+-----------+---------------------+---------------------------------+
+| "3"           | str       | [3]                 | GPU 3                           |
++---------------+-----------+---------------------+---------------------------------+
+| "1, 3"        | str       | [1, 3]              | GPUs 1 and 3                    |
++---------------+-----------+---------------------+---------------------------------+
+| "-1"          | str       | [0, 1, 2, ...]      | all available GPUs              |
++---------------+-----------+---------------------+---------------------------------+
+
+
+Multi-node
+----------
+
+Multi-node training is easily done by specifying these flags.
+
+.. code-block:: python
+
+    # train on 12*8 GPUs
+    trainer = Trainer(gpus=8, num_nodes=12, distributed_backend='ddp')
+
+
+You must configure your job submission script correctly for the trainer to work.
+ Here is an example script for the above trainer configuration.
+
+.. code-block:: bash
+
+    #!/bin/bash -l
+
+    # SLURM SUBMIT SCRIPT
+    #SBATCH --nodes=12
+    #SBATCH --gres=gpu:8
+    #SBATCH --ntasks-per-node=8
+    #SBATCH --mem=0
+    #SBATCH --time=0-02:00:00
+
+    # activate conda env
+    conda activate my_env
+
+    # -------------------------
+    # OPTIONAL
+    # -------------------------
+    # debugging flags (optional)
+    # export NCCL_DEBUG=INFO
+    # export PYTHONFAULTHANDLER=1
+
+    # PyTorch comes with prebuilt NCCL support... but if you have issues with it
+    # you might need to load the latest version from your  modules
+    # module load NCCL/2.4.7-1-cuda.10.0
+
+    # on your cluster you might need these:
+    # set the network interface
+    # export NCCL_SOCKET_IFNAME=^docker0,lo
+    # -------------------------
+
+    # random port between 12k and 20k
+    export MASTER_PORT=$((12000 + RANDOM % 20000))
+
+    # run script from above
+    python my_main_file.py
+
+.. note:: When running in DDP mode, any errors in your code will show up as an NCCL issue.
+ Set the `NCCL_DEBUG=INFO` flag to see the ACTUAL error.
+
+Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a
+ portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).
+
+.. code-block:: python
+
+    # ie: this:
+    dataset = myDataset()
+    dataloader = Dataloader(dataset)
+
+    # becomes:
+    dataset = myDataset()
+    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+    dataloader = Dataloader(dataset, sampler=dist_sampler)
+
+
+Auto-slurm-job-submission
+-------------------------
+
+Instead of manually building SLURM scripts, you can use the
+ `SlurmCluster object <https://williamfalcon.github.io/test-tube/hpc/SlurmCluster>`_
+ to do this for you. The SlurmCluster can also run a grid search if you pass
+ in a `HyperOptArgumentParser
+  <https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser>`_.
+
+Here is an example where you run a grid search of 9 combinations of hyperparams.
+ The full examples are `here
+ <https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/new_project_templates/multi_node_examples>`_.
+
+.. code-block:: python
+
+    # grid search 3 values of learning rate and 3 values of number of layers for your net
+    # this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32),
+    #  (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
+    parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
+    parser.opt_list('--learning_rate', default=0.001, type=float,
+                    options=[1e-3, 1e-2, 1e-1], tunable=True)
+    parser.opt_list('--layers', default=1, type=float, options=[16, 32, 64], tunable=True)
+    hyperparams = parser.parse_args()
+
+    # Slurm cluster submits 9 jobs, each with a set of hyperparams
+    cluster = SlurmCluster(
+        hyperparam_optimizer=hyperparams,
+        log_path='/some/path/to/save',
+    )
+
+    # OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
+    # which interface your nodes use for communication
+    cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
+
+    # see output of the NCCL connection process
+    # NCCL is how the nodes talk to each other
+    cluster.add_command('export NCCL_DEBUG=INFO')
+
+    # setting a master port here is a good idea.
+    cluster.add_command('export MASTER_PORT=%r' % PORT)
+
+    # ************** DON'T FORGET THIS ***************
+    # MUST load the latest NCCL version
+    cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
+
+    # configure cluster
+    cluster.per_experiment_nb_nodes = 12
+    cluster.per_experiment_nb_gpus = 8
+
+    cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
+
+    # submit a script with 9 combinations of hyper params
+    # (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
+    cluster.optimize_parallel_cluster_gpu(
+        main,
+        nb_trials=9, # how many permutations of the grid search to run
+        job_name='name_for_squeue'
+    )
+
+
+The other option is that you generate scripts on your own via a bash command or use another library...
+
+Self-balancing architecture
+---------------------------
+
+Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.
+
+"""
+
+from abc import ABC, abstractmethod
+
+import torch
+
+from pytorch_lightning.overrides.data_parallel import (
+    LightningDistributedDataParallel,
+    LightningDataParallel,
+)
+from pytorch_lightning.utilities.debugging import MisconfigurationException
+
+try:
+    from apex import amp
+
+    APEX_AVAILABLE = True
+except ImportError:
+    APEX_AVAILABLE = False
+
+
+class TrainerDPMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.on_gpu = None
+        self.use_dp = None
+        self.use_ddp2 = None
+        self.use_ddp = None
+        self.use_amp = None
+        self.testing = None
+        self.single_gpu = None
+        self.root_gpu = None
+        self.amp_level = None
+
+    @abstractmethod
+    def run_pretrain_routine(self, model):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def init_optimizers(self, optimizers):
+        # this is just empty shell for code from other class
+        pass
+
+    def copy_trainer_model_properties(self, model):
+        if isinstance(model, LightningDataParallel):
+            ref_model = model.module
+        elif isinstance(model, LightningDistributedDataParallel):
+            ref_model = model.module
+        else:
+            ref_model = model
+
+        for m in [model, ref_model]:
+            m.trainer = self
+            m.on_gpu = self.on_gpu
+            m.use_dp = self.use_dp
+            m.use_ddp2 = self.use_ddp2
+            m.use_ddp = self.use_ddp
+            m.use_amp = self.use_amp
+            m.testing = self.testing
+            m.single_gpu = self.single_gpu
+
+    def transfer_batch_to_gpu(self, batch, gpu_id):
+        # base case: object can be directly moved using `cuda` or `to`
+        if callable(getattr(batch, 'cuda', None)):
+            return batch.cuda(gpu_id)
+
+        elif callable(getattr(batch, 'to', None)):
+            return batch.to(torch.device('cuda', gpu_id))
+
+        # when list
+        elif isinstance(batch, list):
+            for i, x in enumerate(batch):
+                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
+            return batch
+
+        # when tuple
+        elif isinstance(batch, tuple):
+            batch = list(batch)
+            for i, x in enumerate(batch):
+                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
+            return tuple(batch)
+
+        # when dict
+        elif isinstance(batch, dict):
+            for k, v in batch.items():
+                batch[k] = self.transfer_batch_to_gpu(v, gpu_id)
+
+            return batch
+
+        # nothing matches, return the value as is without transform
+        return batch
+
+    def single_gpu_train(self, model):
+        model.cuda(self.root_gpu)
+
+        # CHOOSE OPTIMIZER
+        # allow for lr schedulers as well
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
+
+        if self.use_amp:
+            # An example
+            model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
+            self.optimizers = optimizers
+
+        self.run_pretrain_routine(model)
+
+    def dp_train(self, model):
+
+        # CHOOSE OPTIMIZER
+        # allow for lr schedulers as well
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
+
+        model.cuda(self.root_gpu)
+
+        # check for this bug (amp + dp + !01 doesn't work)
+        # https://github.com/NVIDIA/apex/issues/227
+        if self.use_dp and self.use_amp:
+            if self.amp_level == 'O2':
+                m = f"""
+                Amp level {self.amp_level} with DataParallel is not supported.
+                See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227.
+                We recommend you switch to ddp if you want to use amp
+                """
+                raise MisconfigurationException(m)
+            else:
+                model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
+
+        # create list of device ids
+        device_ids = self.data_parallel_device_ids
+        if type(device_ids) is int:
+            device_ids = list(range(device_ids))
+
+        model = LightningDataParallel(model, device_ids=device_ids)
+
+        self.run_pretrain_routine(model)
+
+
+def normalize_parse_gpu_string_input(s):
+    if type(s) is str:
+        if s == '-1':
+            return -1
+        else:
+            return [int(x.strip()) for x in s.split(',')]
+    else:
+        return s
+
+
+def get_all_available_gpus():
+    """
+    :return: a list of all available gpus
+    """
+    return list(range(torch.cuda.device_count()))
+
+
+def check_gpus_data_type(gpus):
+    """
+    :param gpus: gpus parameter as passed to the Trainer
+        Function checks that it is one of: None, Int, String or List
+        Throws otherwise
+    :return: return unmodified gpus variable
+    """
+
+    if (gpus is not None and
+        type(gpus) is not int and
+        type(gpus) is not str and
+        type(gpus) is not list):    # noqa E129
+        raise MisconfigurationException("GPUs must be int, string or list of ints or None.")
+
+
+def normalize_parse_gpu_input_to_list(gpus):
+    assert gpus is not None
+    if isinstance(gpus, list):
+        return gpus
+    else:  # must be an int
+        if not gpus:  # gpus==0
+            return None
+        elif gpus == -1:
+            return get_all_available_gpus()
+        else:
+            return list(range(gpus))
+
+
+def sanitize_gpu_ids(gpus):
+    """
+    :param gpus: list of ints corresponding to GPU indices
+        Checks that each of the GPUs in the list is actually available.
+        Throws if any of the GPUs is not available.
+    :return: unmodified gpus variable
+    """
+    all_available_gpus = get_all_available_gpus()
+    for gpu in gpus:
+        if gpu not in all_available_gpus:
+            message = f"""
+            You requested GPUs: {gpus}
+            But your machine only has: {all_available_gpus}
+            """
+            raise MisconfigurationException(message)
+    return gpus
+
+
+def parse_gpu_ids(gpus):
+    """
+    :param gpus: Int, string or list
+        An int -1 or string '-1' indicate that all available GPUs should be used.
+        A list of ints or a string containing list of comma separated integers
+        indicates specific GPUs to use
+        An int 0 means that no GPUs should be used
+        Any int N > 0 indicates that GPUs [0..N) should be used.
+    :return: List of gpus to be used
+
+        If no GPUs are available but the value of gpus variable indicates request for GPUs
+        then a misconfiguration exception is raised.
+    """
+
+    # Check that gpus param is None, Int, String or List
+    check_gpus_data_type(gpus)
+
+    # Handle the case when no gpus are requested
+    if gpus is None or type(gpus) is int and gpus == 0:
+        return None
+
+    # We know user requested GPUs therefore if some of the
+    # requested GPUs are not available an exception is thrown.
+
+    gpus = normalize_parse_gpu_string_input(gpus)
+    gpus = normalize_parse_gpu_input_to_list(gpus)
+    gpus = sanitize_gpu_ids(gpus)
+
+    if not gpus:
+        raise MisconfigurationException("GPUs requested but none are available.")
+    return gpus
+
+
+def determine_root_gpu_device(gpus):
+    """
+    :param gpus: non empty list of ints representing which gpus to use
+    :return: designated root GPU device
+    """
+    if gpus is None:
+        return None
+
+    assert isinstance(gpus, list), "gpus should be a list"
+    assert len(gpus), "gpus should be a non empty list"
+
+    # set root gpu
+    root_gpu = gpus[0]
+
+    return root_gpu

pytorch_lightning/trainer/dp_mixin.py

@@ -1,217 +0,0 @@
-import torch
-
-from pytorch_lightning.pt_overrides.override_data_parallel import (
-    LightningDistributedDataParallel, LightningDataParallel)
-from pytorch_lightning.utilities.debugging import MisconfigurationException
-
-try:
-    from apex import amp
-
-    APEX_AVAILABLE = True
-except ImportError:
-    APEX_AVAILABLE = False
-
-
-class TrainerDPMixin(object):
-    def copy_trainer_model_properties(self, model):
-        if isinstance(model, LightningDataParallel):
-            ref_model = model.module
-        elif isinstance(model, LightningDistributedDataParallel):
-            ref_model = model.module
-        else:
-            ref_model = model
-
-        for m in [model, ref_model]:
-            m.trainer = self
-            m.on_gpu = self.on_gpu
-            m.use_dp = self.use_dp
-            m.use_ddp2 = self.use_ddp2
-            m.use_ddp = self.use_ddp
-            m.use_amp = self.use_amp
-            m.testing = self.testing
-            m.single_gpu = self.single_gpu
-
-    def transfer_batch_to_gpu(self, batch, gpu_id):
-        # base case: object can be directly moved using `cuda` or `to`
-        if callable(getattr(batch, 'cuda', None)):
-            return batch.cuda(gpu_id)
-
-        elif callable(getattr(batch, 'to', None)):
-            return batch.to(torch.device('cuda', gpu_id))
-
-        # when list
-        elif isinstance(batch, list):
-            for i, x in enumerate(batch):
-                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
-            return batch
-
-        # when tuple
-        elif isinstance(batch, tuple):
-            batch = list(batch)
-            for i, x in enumerate(batch):
-                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
-            return tuple(batch)
-
-        # when dict
-        elif isinstance(batch, dict):
-            for k, v in batch.items():
-                batch[k] = self.transfer_batch_to_gpu(v, gpu_id)
-
-            return batch
-
-        # nothing matches, return the value as is without transform
-        return batch
-
-    def single_gpu_train(self, model):
-        # CHOOSE OPTIMIZER
-        # allow for lr schedulers as well
-        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
-
-        model.cuda(self.root_gpu)
-
-        if self.use_amp:
-            # An example
-            model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
-            self.optimizers = optimizers
-
-        self.run_pretrain_routine(model)
-
-    def dp_train(self, model):
-
-        # CHOOSE OPTIMIZER
-        # allow for lr schedulers as well
-        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
-
-        model.cuda(self.root_gpu)
-
-        # check for this bug (amp + dp + !01 doesn't work)
-        # https://github.com/NVIDIA/apex/issues/227
-        if self.use_dp and self.use_amp:
-            m = f"""
-            Amp level {self.amp_level} with DataParallel is not supported.
-            See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227.
-            We recommend you switch to ddp if you want to use amp
-            """
-            raise MisconfigurationException(m)
-
-        # create list of device ids
-        device_ids = self.data_parallel_device_ids
-        if type(device_ids) is int:
-            device_ids = list(range(device_ids))
-
-        model = LightningDataParallel(model, device_ids=device_ids)
-
-        self.run_pretrain_routine(model)
-
-
-def normalize_parse_gpu_string_input(s):
-    if type(s) is str:
-        if s == '-1':
-            return -1
-        else:
-            return [int(x.strip()) for x in s.split(',')]
-    else:
-        return s
-
-
-def get_all_available_gpus():
-    """
-    :return: a list of all available gpus
-    """
-    return list(range(torch.cuda.device_count()))
-
-
-def check_gpus_data_type(gpus):
-    """
-    :param gpus: gpus parameter as passed to the Trainer
-        Function checks that it is one of: None, Int, String or List
-        Throws otherwise
-    :return: return unmodified gpus variable
-    """
-
-    if (gpus is not None and
-        type(gpus) is not int and
-        type(gpus) is not str and
-        type(gpus) is not list):    # noqa E129
-        raise MisconfigurationException("GPUs must be int, string or list of ints or None.")
-
-
-def normalize_parse_gpu_input_to_list(gpus):
-    assert gpus is not None
-    if isinstance(gpus, list):
-        return gpus
-    else:  # must be an int
-        if not gpus:  # gpus==0
-            return None
-        elif gpus == -1:
-            return get_all_available_gpus()
-        else:
-            return list(range(gpus))
-
-
-def sanitize_gpu_ids(gpus):
-    """
-    :param gpus: list of ints corresponding to GPU indices
-        Checks that each of the GPUs in the list is actually available.
-        Throws if any of the GPUs is not available.
-    :return: unmodified gpus variable
-    """
-    all_available_gpus = get_all_available_gpus()
-    for gpu in gpus:
-        if gpu not in all_available_gpus:
-            message = f"""
-            Non-available gpu index {gpu} specified:
-            Available gpu indices are: {all_available_gpus}
-            """
-            raise MisconfigurationException(message)
-    return gpus
-
-
-def parse_gpu_ids(gpus):
-    """
-    :param gpus: Int, string or list
-        An int -1 or string '-1' indicate that all available GPUs should be used.
-        A list of ints or a string containing list of comma separated integers
-        indicates specific GPUs to use
-        An int 0 means that no GPUs should be used
-        Any int N > 0 indicates that GPUs [0..N) should be used.
-    :return: List of gpus to be used
-
-        If no GPUs are available but the value of gpus variable indicates request for GPUs
-        then a misconfiguration exception is raised.
-    """
-
-    # Check that gpus param is None, Int, String or List
-    check_gpus_data_type(gpus)
-
-    # Handle the case when no gpus are requested
-    if gpus is None or type(gpus) is int and gpus == 0:
-        return None
-
-    # We know user requested GPUs therefore if some of the
-    # requested GPUs are not available an exception is thrown.
-
-    gpus = normalize_parse_gpu_string_input(gpus)
-    gpus = normalize_parse_gpu_input_to_list(gpus)
-    gpus = sanitize_gpu_ids(gpus)
-
-    if not gpus:
-        raise MisconfigurationException("GPUs requested but non are available.")
-    return gpus
-
-
-def determine_root_gpu_device(gpus):
-    """
-    :param gpus: non empty list of ints representing which gpus to use
-    :return: designated root GPU device
-    """
-    if gpus is None:
-        return None
-
-    assert isinstance(gpus, list), "gpus should be a list"
-    assert len(gpus), "gpus should be a non empty list"
-
-    # set root gpu
-    root_gpu = gpus[0]
-
-    return root_gpu

pytorch_lightning/trainer/evaluation_loop.py

@@ -1,14 +1,197 @@
+"""
+# Validation loop
+
+The lightning validation loop handles everything except the actual computations of your model.
+To decide what will happen in your validation loop, define the `validation_step` function.
+Below are all the things lightning automates for you in the validation loop.
+
+.. note:: Lightning will run 5 steps of validation in the beginning of training as a sanity
+ check so you don't have to wait until a full epoch to catch possible validation issues.
+
+Check validation every n epochs
+-------------------------------
+
+If you have a small dataset you might want to check validation every n epochs
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(check_val_every_n_epoch=1)
+
+Set how much of the validation set to check
+-------------------------------------------
+
+If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
+
+val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(val_percent_check=1.0)
+
+    # check 10% only
+    trainer = Trainer(val_percent_check=0.1)
+
+Set how much of the test set to check
+-------------------------------------
+
+If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
+
+test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(test_percent_check=1.0)
+
+    # check 10% only
+    trainer = Trainer(test_percent_check=0.1)
+
+Set validation check frequency within 1 training epoch
+------------------------------------------------------
+
+For large datasets it's often desirable to check validation multiple times within a training loop.
+ Pass in a float to check that often within 1 training epoch.
+ Pass in an int k to check every k training batches. Must use an int if using an IterableDataset.
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(val_check_interval=0.95)
+
+    # check every .25 of an epoch
+    trainer = Trainer(val_check_interval=0.25)
+
+    # check every 100 train batches (ie: for IterableDatasets or fixed frequency)
+    trainer = Trainer(val_check_interval=100)
+
+
+Set the number of validation sanity steps
+-----------------------------------------
+
+Lightning runs a few steps of validation in the beginning of training.
+ This avoids crashing in the validation loop sometime deep into a lengthy training loop.
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(num_sanity_val_steps=5)
+
+
+You can use `Trainer(num_sanity_val_steps=0)` to skip the sanity check.
+
+# Testing loop
+
+To ensure you don't accidentally use test data to guide training decisions Lightning
+ makes running the test set deliberate.
+
+**test**
+
+You have two options to run the test set.
+First case is where you test right after a full training routine.
+
+.. code-block:: python
+
+    # run full training
+    trainer.fit(model)
+
+    # run test set
+    trainer.test()
+
+
+Second case is where you load a model and run the test set
+
+.. code-block:: python
+
+    model = MyLightningModule.load_from_metrics(
+        weights_path='/path/to/pytorch_checkpoint.ckpt',
+        tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
+        on_gpu=True,
+        map_location=None
+    )
+
+    # init trainer with whatever options
+    trainer = Trainer(...)
+
+    # test (pass in the model)
+    trainer.test(model)
+
+In this second case, the options you pass to trainer will be used when running
+ the test set (ie: 16-bit, dp, ddp, etc...)
+
+"""
+
+from abc import ABC, abstractmethod
+
 import torch
+import sys
 import tqdm
 
 from pytorch_lightning.utilities.debugging import MisconfigurationException
 
 
-class TrainerEvaluationLoopMixin(object):
+class TrainerEvaluationLoopMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.test_progress_bar = None
+        self.val_progress_bar = None
+        self.main_progress_bar = None
+        self.use_ddp = None
+        self.use_dp = None
+        self.use_ddp2 = None
+        self.single_gpu = None
+        self.data_parallel_device_ids = None
+        self.model = None
+        self.num_test_batches = None
+        self.num_val_batches = None
+        self.fast_dev_run = None
+        self.process_position = None
+        self.show_progress_bar = None
+        self.process_output = None
+        self.training_tqdm_dict = None
+        self.proc_rank = None
+        self.checkpoint_callback = None
+        self.current_epoch = None
+        self.callback_metrics = None
+        self.get_test_dataloaders = None
+        self.get_val_dataloaders = None
+
+    @abstractmethod
+    def copy_trainer_model_properties(self, model):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def get_model(self):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def is_overriden(self, m):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def transfer_batch_to_gpu(self, batch, gpu):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def add_tqdm_metrics(self, metrics):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def log_metrics(self, metrics, grad_norm_dic):
+        # this is just empty shell for code from other class
+        pass
 
     def evaluate(self, model, dataloaders, max_batches, test=False):
-        """
-        Run evaluation code
+        """Run evaluation code.
+
         :param model: PT model
         :param dataloaders: list of PT dataloaders
         :param max_batches: Scalar
@@ -104,11 +287,11 @@ class TrainerEvaluationLoopMixin(object):
             # select dataloaders
             if test:
                 dataloaders = self.get_test_dataloaders()
-                max_batches = self.nb_test_batches
+                max_batches = self.num_test_batches
             else:
                 # val
                 dataloaders = self.get_val_dataloaders()
-                max_batches = self.nb_val_batches
+                max_batches = self.num_val_batches
 
             # cap max batches to 1 when using fast_dev_run
             if self.fast_dev_run:
@@ -120,7 +303,7 @@ class TrainerEvaluationLoopMixin(object):
             desc = 'Testing' if test else 'Validating'
             pbar = tqdm.tqdm(desc=desc, total=max_batches, leave=test, position=position,
                              disable=not self.show_progress_bar, dynamic_ncols=True,
-                             unit='batch')
+                             unit='batch', file=sys.stdout)
             setattr(self, f'{"test" if test else "val"}_progress_bar', pbar)
 
             # run evaluation
@@ -138,7 +321,7 @@ class TrainerEvaluationLoopMixin(object):
             self.log_metrics(log_metrics, {})
 
             # track metrics for callbacks
-            self.callback_metrics = callback_metrics
+            self.callback_metrics.update(callback_metrics)
 
             # hook
             model.on_post_performance_check()
@@ -178,7 +361,7 @@ class TrainerEvaluationLoopMixin(object):
         if self.single_gpu:
             # for single GPU put inputs on gpu manually
             root_gpu = 0
-            if type(self.data_parallel_device_ids) is list:
+            if isinstance(self.data_parallel_device_ids, list):
                 root_gpu = self.data_parallel_device_ids[0]
             batch = self.transfer_batch_to_gpu(batch, root_gpu)
             args[0] = batch

pytorch_lightning/trainer/logging.py

@@ -1,16 +1,31 @@
+from abc import ABC
+
 import torch
 
-from pytorch_lightning.root_module import memory
+from pytorch_lightning.core import memory
 
 
-class TrainerLoggingMixin(object):
+class TrainerLoggingMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.current_epoch = None
+        self.on_gpu = None
+        self.log_gpu_memory = None
+        self.logger = None
+        self.tqdm_metrics = None
+        self.global_step = None
+        self.proc_rank = None
+        self.use_dp = None
+        self.use_ddp2 = None
+        self.num_gpus = None
+
+    def log_metrics(self, metrics, grad_norm_dic, step=None):
+        """Logs the metric dict passed in.
 
-    def log_metrics(self, metrics, grad_norm_dic):
-        """
-        Logs the metric dict passed in
         :param metrics:
         :param grad_norm_dic:
-        :return:
         """
         # added metrics by Lightning for convenience
         metrics['epoch'] = self.current_epoch
@@ -26,9 +41,10 @@ class TrainerLoggingMixin(object):
         # turn all tensors to scalars
         scalar_metrics = self.metrics_to_scalars(metrics)
 
+        step = step if step is not None else self.global_step
         # log actual metrics
         if self.proc_rank == 0 and self.logger is not None:
-            self.logger.log_metrics(scalar_metrics, step_num=self.global_step)
+            self.logger.log_metrics(scalar_metrics, step=step)
             self.logger.save()
 
     def add_tqdm_metrics(self, metrics):
@@ -52,8 +68,8 @@ class TrainerLoggingMixin(object):
         return new_metrics
 
     def process_output(self, output, train=False):
-        """
-        Reduces output according to the training mode.
+        """Reduces output according to the training mode.
+
         Separates loss from logging and tqdm metrics
         :param output:
         :return:
@@ -68,11 +84,12 @@ class TrainerLoggingMixin(object):
                 callback_metrics[k] = v
 
         if train and (self.use_dp or self.use_ddp2):
-            nb_gpus = self.num_gpus
-            callback_metrics = self.reduce_distributed_output(callback_metrics, nb_gpus)
+            num_gpus = self.num_gpus
+            callback_metrics = self.reduce_distributed_output(callback_metrics, num_gpus)
 
         for k, v in callback_metrics.items():
-            callback_metrics[k] = v.item()
+            if isinstance(v, torch.Tensor):
+                callback_metrics[k] = v.item()
 
         # ---------------
         # EXTRACT PROGRESS BAR KEYS
@@ -82,8 +99,8 @@ class TrainerLoggingMixin(object):
 
             # reduce progress metrics for tqdm when using dp
             if train and (self.use_dp or self.use_ddp2):
-                nb_gpus = self.num_gpus
-                progress_output = self.reduce_distributed_output(progress_output, nb_gpus)
+                num_gpus = self.num_gpus
+                progress_output = self.reduce_distributed_output(progress_output, num_gpus)
 
             progress_bar_metrics = progress_output
         except Exception:
@@ -98,8 +115,8 @@ class TrainerLoggingMixin(object):
 
             # reduce progress metrics for tqdm when using dp
             if train and (self.use_dp or self.use_ddp2):
-                nb_gpus = self.num_gpus
-                log_output = self.reduce_distributed_output(log_output, nb_gpus)
+                num_gpus = self.num_gpus
+                log_output = self.reduce_distributed_output(log_output, num_gpus)
 
             log_metrics = log_output
         except Exception:
@@ -142,8 +159,8 @@ class TrainerLoggingMixin(object):
 
         return loss, progress_bar_metrics, log_metrics, callback_metrics, hiddens
 
-    def reduce_distributed_output(self, output, nb_gpus):
-        if nb_gpus <= 1:
+    def reduce_distributed_output(self, output, num_gpus):
+        if num_gpus <= 1:
             return output
 
         # when using DP, we get one output per gpu
@@ -154,14 +171,14 @@ class TrainerLoggingMixin(object):
         for k, v in output.items():
             # recurse on nested dics
             if isinstance(output[k], dict):
-                output[k] = self.reduce_distributed_output(output[k], nb_gpus)
+                output[k] = self.reduce_distributed_output(output[k], num_gpus)
 
             # do nothing when there's a scalar
             elif isinstance(output[k], torch.Tensor) and output[k].dim() == 0:
                 pass
 
-            # reduce only metrics that have the same nb of gpus
-            elif output[k].size(0) == nb_gpus:
+            # reduce only metrics that have the same number of gpus
+            elif output[k].size(0) == num_gpus:
                 reduced = torch.mean(output[k])
                 output[k] = reduced
         return output

pytorch_lightning/trainer/model_hooks.py

@@ -1,7 +1,10 @@
-from pytorch_lightning.root_module.root_module import LightningModule
+import inspect
+from abc import ABC, abstractmethod
+
+from pytorch_lightning.core.lightning import LightningModule
 
 
-class TrainerModelHooksMixin(object):
+class TrainerModelHooksMixin(ABC):
 
     def is_function_implemented(self, f_name):
         model = self.get_model()
@@ -15,3 +18,13 @@ class TrainerModelHooksMixin(object):
         # when code pointers are different, it was overriden
         is_overriden = getattr(model, f_name).__code__ is not getattr(super_object, f_name).__code__
         return is_overriden
+
+    def has_arg(self, f_name, arg_name):
+        model = self.get_model()
+        f_op = getattr(model, f_name, None)
+        return arg_name in inspect.signature(f_op).parameters
+
+    @abstractmethod
+    def get_model(self):
+        # this is just empty shell for code from other class
+        pass

pytorch_lightning/trainer/train_loop_mixin.py

@@ -1,309 +0,0 @@
-import numpy as np
-import tqdm
-
-try:
-    from apex import amp
-
-    APEX_AVAILABLE = True
-except ImportError:
-    APEX_AVAILABLE = False
-
-
-class TrainerTrainLoopMixin(object):
-
-    def train(self):
-        # run all epochs
-        for epoch_nb in range(self.current_epoch, self.max_nb_epochs):
-            # set seed for distributed sampler (enables shuffling for each epoch)
-            if self.use_ddp and hasattr(self.get_train_dataloader().sampler, 'set_epoch'):
-                self.get_train_dataloader().sampler.set_epoch(epoch_nb)
-
-            # get model
-            model = self.get_model()
-
-            # update training progress in trainer and model
-            model.current_epoch = epoch_nb
-            self.current_epoch = epoch_nb
-
-            # val can be checked multiple times in epoch
-            is_val_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
-            val_checks_per_epoch = self.nb_training_batches // self.val_check_batch
-            val_checks_per_epoch = val_checks_per_epoch if is_val_epoch else 0
-
-            # total batches includes multiple val checks
-            self.total_batches = (self.nb_training_batches +
-                                  self.nb_val_batches * val_checks_per_epoch)
-            self.batch_loss_value = 0  # accumulated grads
-
-            if self.fast_dev_run:
-                # limit the number of batches to 2 (1 train and 1 val) in fast_dev_run
-                nb_iterations = 2
-            elif self.is_iterable_train_dataloader:
-                # for iterable train loader, the progress bar never ends
-                nb_iterations = None
-            else:
-                nb_iterations = self.total_batches
-
-            # reset progress bar
-            # .reset() doesn't work on disabled progress bar so we should check
-            if not self.main_progress_bar.disable:
-                self.main_progress_bar.reset(nb_iterations)
-            desc = f'Epoch {epoch_nb + 1}' if not self.is_iterable_train_dataloader else ''
-            self.main_progress_bar.set_description(desc)
-
-            # changing gradient according accumulation_scheduler
-            self.accumulation_scheduler.on_epoch_begin(epoch_nb, self)
-
-            # -----------------
-            # RUN TNG EPOCH
-            # -----------------
-            self.run_training_epoch()
-
-            # update LR schedulers
-            if self.lr_schedulers is not None:
-                for lr_scheduler in self.lr_schedulers:
-                    lr_scheduler.step(self.current_epoch)
-
-            # early stopping
-            met_min_epochs = epoch_nb > self.min_nb_epochs
-            if self.enable_early_stop and (met_min_epochs or self.fast_dev_run):
-                should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch_nb,
-                                                                    logs=self.callback_metrics)
-                # stop training
-                stop = should_stop and met_min_epochs
-                if stop:
-                    self.main_progress_bar.close()
-                    return
-
-        self.main_progress_bar.close()
-
-        if self.logger is not None:
-            self.logger.finalize("success")
-
-    def run_training_epoch(self):
-        # before epoch hook
-        if self.is_function_implemented('on_epoch_start'):
-            model = self.get_model()
-            model.on_epoch_start()
-
-        # run epoch
-        for batch_nb, batch in enumerate(self.get_train_dataloader()):
-            self.batch_nb = batch_nb
-
-            model = self.get_model()
-            model.global_step = self.global_step
-
-            # ---------------
-            # RUN TRAIN STEP
-            # ---------------
-            output = self.run_training_batch(batch, batch_nb)
-            batch_result, grad_norm_dic, batch_step_metrics = output
-
-            # when returning -1 from train_step, we end epoch early
-            early_stop_epoch = batch_result == -1
-
-            # ---------------
-            # RUN VAL STEP
-            # ---------------
-            is_val_check_batch = (batch_nb + 1) % self.val_check_batch == 0
-            can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
-            should_check_val = ((is_val_check_batch or early_stop_epoch) and can_check_epoch)
-
-            # fast_dev_run always forces val checking after train batch
-            if self.fast_dev_run or should_check_val:
-                self.run_evaluation(test=self.testing)
-
-            # when logs should be saved
-            should_save_log = (batch_nb + 1) % self.log_save_interval == 0 or early_stop_epoch
-            if should_save_log or self.fast_dev_run:
-                if self.proc_rank == 0 and self.logger is not None:
-                    self.logger.save()
-
-            # when metrics should be logged
-            should_log_metrics = batch_nb % self.row_log_interval == 0 or early_stop_epoch
-            if should_log_metrics or self.fast_dev_run:
-                # logs user requested information to logger
-                self.log_metrics(batch_step_metrics, grad_norm_dic)
-
-            self.global_step += 1
-            self.total_batch_nb += 1
-
-            # end epoch early
-            # stop when the flag is changed or we've gone past the amount
-            # requested in the batches
-            if early_stop_epoch or self.fast_dev_run:
-                break
-
-            # stop epoch if we limited nb batches
-            met_batch_limit = batch_nb >= self.nb_training_batches
-            if met_batch_limit:
-                break
-
-        # epoch end hook
-        if self.is_function_implemented('on_epoch_end'):
-            model = self.get_model()
-            model.on_epoch_end()
-
-    def run_training_batch(self, batch, batch_nb):
-        # track grad norms
-        grad_norm_dic = {}
-
-        # track all metrics for callbacks
-        all_callback_metrics = []
-
-        # track metrics to log
-        all_log_metrics = []
-
-        if batch is None:
-            return 0, grad_norm_dic
-
-        # hook
-        if self.is_function_implemented('on_batch_start'):
-            model_ref = self.get_model()
-            response = model_ref.on_batch_start(batch)
-
-            if response == -1:
-                return -1, grad_norm_dic
-
-        splits = [batch]
-        if self.truncated_bptt_steps is not None:
-            model_ref = self.get_model()
-            splits = model_ref.tbptt_split_batch(batch, self.truncated_bptt_steps)
-
-        self.hiddens = None
-        for split_nb, split_batch in enumerate(splits):
-            self.split_nb = split_nb
-
-            # call training_step once per optimizer
-            for opt_idx, optimizer in enumerate(self.optimizers):
-
-                # wrap the forward step in a closure so second order methods work
-                def optimizer_closure():
-                    # forward pass
-                    output = self.training_forward(
-                        split_batch, batch_nb, opt_idx, self.hiddens)
-
-                    closure_loss = output[0]
-                    progress_bar_metrics = output[1]
-                    log_metrics = output[2]
-                    callback_metrics = output[3]
-                    self.hiddens = output[4]
-
-                    # accumulate loss
-                    # (if accumulate_grad_batches = 1 no effect)
-                    closure_loss = closure_loss / self.accumulate_grad_batches
-
-                    # backward pass
-                    model_ref = self.get_model()
-                    model_ref.backward(self.use_amp, closure_loss, optimizer)
-
-                    # track metrics for callbacks
-                    all_callback_metrics.append(callback_metrics)
-
-                    # track progress bar metrics
-                    self.add_tqdm_metrics(progress_bar_metrics)
-                    all_log_metrics.append(log_metrics)
-
-                    # insert after step hook
-                    if self.is_function_implemented('on_after_backward'):
-                        model_ref = self.get_model()
-                        model_ref.on_after_backward()
-
-                    return closure_loss
-
-                # calculate loss
-                loss = optimizer_closure()
-
-                # nan grads
-                if self.print_nan_grads:
-                    self.print_nan_gradients()
-
-                # track total loss for logging (avoid mem leaks)
-                self.batch_loss_value += loss.item()
-
-                # gradient update with accumulated gradients
-                if (self.batch_nb + 1) % self.accumulate_grad_batches == 0:
-
-                    # track gradient norms when requested
-                    if batch_nb % self.row_log_interval == 0:
-                        if self.track_grad_norm > 0:
-                            model = self.get_model()
-                            grad_norm_dic = model.grad_norm(
-                                self.track_grad_norm)
-
-                    # clip gradients
-                    self.clip_gradients()
-
-                    # calls .step(), .zero_grad()
-                    # override function to modify this behavior
-                    model = self.get_model()
-                    model.optimizer_step(self.current_epoch, batch_nb,
-                                         optimizer, opt_idx, optimizer_closure)
-
-                    # calculate running loss for display
-                    self.running_loss.append(self.batch_loss_value)
-                    self.batch_loss_value = 0
-                    self.avg_loss = np.mean(self.running_loss[-100:])
-
-        # activate batch end hook
-        if self.is_function_implemented('on_batch_end'):
-            model = self.get_model()
-            model.on_batch_end()
-
-        # update progress bar
-        self.main_progress_bar.update(1)
-        self.main_progress_bar.set_postfix(**self.training_tqdm_dict)
-
-        # collapse all metrics into one dict
-        all_log_metrics = {k: v for d in all_log_metrics for k, v in d.items()}
-
-        # track all metrics for callbacks
-        self.callback_metrics = {k: v for d in all_callback_metrics for k, v in d.items()}
-
-        return 0, grad_norm_dic, all_log_metrics
-
-    def training_forward(self, batch, batch_nb, opt_idx, hiddens):
-        """
-        Handle forward for each training case (distributed, single gpu, etc...)
-        :param batch:
-        :param batch_nb:
-        :return:
-        """
-        # ---------------
-        # FORWARD
-        # ---------------
-        # enable not needing to add opt_idx to training_step
-        args = [batch, batch_nb]
-        if len(self.optimizers) > 1:
-            args.append(opt_idx)
-
-        # pass hiddens if using tbptt
-        if self.truncated_bptt_steps is not None:
-            args.append(hiddens)
-
-        # distributed forward
-        if self.use_ddp or self.use_ddp2 or self.use_dp:
-            output = self.model(*args)
-
-        # single GPU forward
-        elif self.single_gpu:
-            gpu_id = 0
-            if type(self.data_parallel_device_ids) is list:
-                gpu_id = self.data_parallel_device_ids[0]
-            batch = self.transfer_batch_to_gpu(batch, gpu_id)
-            args[0] = batch
-            output = self.model.training_step(*args)
-
-        # CPU forward
-        else:
-            output = self.model.training_step(*args)
-
-        # allow any mode to define training_end
-        if self.is_overriden('training_end'):
-            model_ref = self.get_model()
-            output = model_ref.training_end(output)
-
-        # format and reduce outputs accordingly
-        output = self.process_output(output, train=True)
-
-        return output

pytorch_lightning/trainer/trainer.py

@@ -3,6 +3,7 @@ The trainer handles all the logic for running a val loop, training loop, distrib
 """
 
 import os
+import sys
 import warnings
 import logging
 
@@ -12,21 +13,21 @@ import torch.multiprocessing as mp
 import tqdm
 from torch.optim.optimizer import Optimizer
 
-from pytorch_lightning.trainer.amp_mixin import TrainerAMPMixin
-from pytorch_lightning.trainer.callback_config_mixin import TrainerCallbackConfigMixin
-from pytorch_lightning.trainer.data_loading_mixin import TrainerDataLoadingMixin
-from pytorch_lightning.trainer.ddp_mixin import TrainerDDPMixin
-from pytorch_lightning.trainer.dp_mixin import TrainerDPMixin
-from pytorch_lightning.trainer.dp_mixin import (
+from pytorch_lightning.trainer.auto_mix_precision import TrainerAMPMixin
+from pytorch_lightning.trainer.callback_config import TrainerCallbackConfigMixin
+from pytorch_lightning.trainer.data_loading import TrainerDataLoadingMixin
+from pytorch_lightning.trainer.distrib_data_parallel import TrainerDDPMixin
+from pytorch_lightning.trainer.distrib_parts import (
+    TrainerDPMixin,
     parse_gpu_ids,
     determine_root_gpu_device
 )
-from pytorch_lightning.trainer.evaluation_loop_mixin import TrainerEvaluationLoopMixin
-from pytorch_lightning.trainer.logging_mixin import TrainerLoggingMixin
-from pytorch_lightning.trainer.model_hooks_mixin import TrainerModelHooksMixin
-from pytorch_lightning.trainer.train_loop_mixin import TrainerTrainLoopMixin
+from pytorch_lightning.trainer.evaluation_loop import TrainerEvaluationLoopMixin
+from pytorch_lightning.trainer.logging import TrainerLoggingMixin
+from pytorch_lightning.trainer.model_hooks import TrainerModelHooksMixin
+from pytorch_lightning.trainer.training_loop import TrainerTrainLoopMixin
 from pytorch_lightning.trainer.trainer_io import TrainerIOMixin
-from pytorch_lightning.trainer.training_tricks_mixin import TrainerTrainingTricksMixin
+from pytorch_lightning.trainer.training_tricks import TrainerTrainingTricksMixin
 from pytorch_lightning.utilities.debugging import MisconfigurationException
 
 try:
@@ -38,113 +39,163 @@ except ImportError:
 
 
 class Trainer(TrainerIOMixin,
-              TrainerDDPMixin,
               TrainerDPMixin,
+              TrainerDDPMixin,
+              TrainerLoggingMixin,
+              TrainerModelHooksMixin,
+              TrainerTrainingTricksMixin,
               TrainerDataLoadingMixin,
               TrainerAMPMixin,
               TrainerEvaluationLoopMixin,
               TrainerTrainLoopMixin,
-              TrainerLoggingMixin,
-              TrainerTrainingTricksMixin,
               TrainerCallbackConfigMixin,
-              TrainerModelHooksMixin):
+              ):
 
-    def __init__(self,
-                 logger=True,
-                 checkpoint_callback=True,
-                 early_stop_callback=True,
-                 default_save_path=None,
-                 gradient_clip_val=0,
-                 gradient_clip=None,  # backward compatible
-                 process_position=0,
-                 nb_gpu_nodes=1,
-                 gpus=None,
-                 log_gpu_memory=None,
-                 show_progress_bar=True,
-                 overfit_pct=0.0,
-                 track_grad_norm=-1,
-                 check_val_every_n_epoch=1,
-                 fast_dev_run=False,
-                 accumulate_grad_batches=1,
-                 max_nb_epochs=1000,
-                 min_nb_epochs=1,
-                 train_percent_check=1.0,
-                 val_percent_check=1.0,
-                 test_percent_check=1.0,
-                 val_check_interval=1.0,
-                 log_save_interval=100,
-                 row_log_interval=10,
-                 add_row_log_interval=None,  # backward compatible
-                 distributed_backend=None,
-                 use_amp=False,
-                 print_nan_grads=False,
-                 weights_summary='full',
-                 weights_save_path=None,
-                 amp_level='O1',
-                 nb_sanity_val_steps=5,
-                 truncated_bptt_steps=None):
+    def __init__(
+            self,
+            logger=True,
+            checkpoint_callback=True,
+            early_stop_callback=True,
+            default_save_path=None,
+            gradient_clip_val=0,
+            gradient_clip=None,  # backward compatible, todo: remove in v0.8.0
+            process_position=0,
+            nb_gpu_nodes=None,  # backward compatible, todo: remove in v0.8.0
+            num_nodes=1,
+            gpus=None,
+            log_gpu_memory=None,
+            show_progress_bar=True,
+            overfit_pct=0.0,
+            track_grad_norm=-1,
+            check_val_every_n_epoch=1,
+            fast_dev_run=False,
+            accumulate_grad_batches=1,
+            max_nb_epochs=None,  # backward compatible, todo: remove in v0.8.0
+            min_nb_epochs=None,  # backward compatible, todo: remove in v0.8.0
+            max_epochs=1000,
+            min_epochs=1,
+            train_percent_check=1.0,
+            val_percent_check=1.0,
+            test_percent_check=1.0,
+            val_check_interval=1.0,
+            log_save_interval=100,
+            row_log_interval=10,
+            add_row_log_interval=None,  # backward compatible, todo: remove in v0.8.0
+            distributed_backend=None,
+            use_amp=False,
+            print_nan_grads=False,
+            weights_summary='full',
+            weights_save_path=None,
+            amp_level='O1',
+            nb_sanity_val_steps=None,  # backward compatible, todo: remove in v0.8.0
+            num_sanity_val_steps=5,
+            truncated_bptt_steps=None,
+            resume_from_checkpoint=None,
+    ):
         """
 
         :param logger: Logger for experiment tracking
         :param checkpoint_callback: Callback for checkpointing
         :param early_stop_callback: Callback for early stopping
-        :param default_save_path: Default path for logs+weights if no logger/ckpt_callback passed
-        :param gradient_clip_val: int. 0 means don't clip.
-        :param gradient_clip: int. 0 means don't clip. Deprecated.
+        :param str default_save_path: Default path for logs+weights if no logger/ckpt_callback passed
+        :param int gradient_clip_val: 0 means don't clip.
+        :param int gradient_clip: 0 means don't clip. Deprecated.
         :param process_position: shown in the tqdm bar
-        :param nb_gpu_nodes: number of GPU nodes
-        :param gpus: int. (ie: 2 gpus) OR list to specify which GPUs [0, 1] OR '0,1'
+        :param int num_nodes: number of GPU nodes
+        :param list|str|int gpus: int. (ie: 2 gpus) OR list to specify which GPUs [0, 1] OR '0,1'
             OR '-1' / -1 to use all available gpus
-        :param log_gpu_memory: str. None, 'min_max', 'all'
-        :param show_progress_bar: Bool. If true shows tqdm bar
-        :param overfit_pct: float. uses this much of all datasets
-        :param track_grad_norm: int. -1 no tracking. Otherwise tracks that norm
-        :param check_val_every_n_epoch: int. check val every n train epochs
-        :param fast_dev_run: Bool. runs full iteration over everything to find bugs
-        :param accumulate_grad_batches: int. Accumulates grads every k batches
-        :param max_nb_epochs: int.
-        :param min_nb_epochs: int.
-        :param train_percent_check: int. How much of train set to check
-        :param val_percent_check: int. How much of val set to check
-        :param test_percent_check: int. How much of test set to check
-        :param val_check_interval: float/int. If float, % of tng epoch. If int, check every n batch
-        :param log_save_interval: int. Writes logs to disk this often
-        :param row_log_interval: int. How often to add logging rows
-        :param add_row_log_interval: int. How often to add logging rows. Deprecated.
-        :param distributed_backend: str. Options: 'dp', 'ddp', 'ddp2'.
-        :param use_amp: Bool. If true uses apex for 16bit precision
-        :param print_nan_grads: Bool. Prints nan gradients
-        :param weights_summary: str. Options: 'full', 'top', None to not print.
-        :param weights_save_path: Bool. Where to save weights if on cluster
-        :param amp_level: str. Check nvidia docs for level
-        :param nb_sanity_val_steps: int. How many val steps before a full train loop.
-        :param truncated_bptt_steps: int. Enables multiple backward passes for each batch.
+        :param str log_gpu_memory: None, 'min_max', 'all'
+        :param bool show_progress_bar: If true shows tqdm bar
+        :param float overfit_pct: uses this much of all datasets
+        :param int track_grad_norm: -1 no tracking. Otherwise tracks that norm
+        :param int check_val_every_n_epoch: check val every n train epochs
+        :param bool fast_dev_run: runs full iteration over everything to find bugs
+        :param int accumulate_grad_batches: Accumulates grads every k batches
+        :param int max_epochs:
+        :param int min_epochs:
+        :param int train_percent_check: How much of train set to check
+        :param int val_percent_check: How much of val set to check
+        :param int test_percent_check: How much of test set to check
+        :param float|int val_check_interval: If float, % of tng epoch. If int, check every n batch
+        :param int log_save_interval: Writes logs to disk this often
+        :param int row_log_interval: How often to add logging rows
+        :param int add_row_log_interval: How often to add logging rows. Deprecated.
+        :param str distributed_backend: Options: 'dp', 'ddp', 'ddp2'.
+        :param bool use_amp: If true uses apex for 16bit precision
+        :param bool print_nan_grads: Prints nan gradients
+        :param str weights_summary: Options: 'full', 'top', None to not print.
+        :param bool weights_save_path: Where to save weights if on cluster
+        :param str amp_level: Check nvidia docs for level
+        :param int num_sanity_val_steps: How many val steps before a full train loop.
+        :param int truncated_bptt_steps: Enables multiple backward passes for each batch.
+
+        .. warning:: Following arguments become deprecated and they will be removed in v0.8.0:
+            - `gradient_clip`,
+            - `nb_gpu_nodes`,
+            - `max_nb_epochs`,
+            - `min_nb_epochs`,
+            - `add_row_log_interval`,
+            - `nb_sanity_val_steps`
+
         """
         # Transfer params
-        self.nb_gpu_nodes = nb_gpu_nodes
+        # Backward compatibility
+        if nb_gpu_nodes is not None:
+            warnings.warn("`nb_gpu_nodes` has renamed to `num_nodes` since v0.5.0"
+                          " and will be removed in v0.8.0", DeprecationWarning)
+            if not num_nodes:  # in case you did not set the proper value
+                num_nodes = nb_gpu_nodes
+        self.num_gpu_nodes = num_nodes
+
         self.log_gpu_memory = log_gpu_memory
-        if not (gradient_clip is None):
-            # Backward compatibility
-            warnings.warn("gradient_clip has renamed to gradient_clip_val since v0.5.0",
-                          DeprecationWarning)
-            gradient_clip_val = gradient_clip
+
+        # Backward compatibility
+        if gradient_clip is not None:
+            warnings.warn("`gradient_clip` has renamed to `gradient_clip_val` since v0.5.0"
+                          " and will be removed in v0.8.0", DeprecationWarning)
+            if not gradient_clip_val:  # in case you did not set the proper value
+                gradient_clip_val = gradient_clip
         self.gradient_clip_val = gradient_clip_val
+
         self.check_val_every_n_epoch = check_val_every_n_epoch
         self.track_grad_norm = track_grad_norm
-        self.on_gpu = gpus is not None and torch.cuda.is_available()
+        self.on_gpu = True if (gpus and torch.cuda.is_available()) else False
         self.process_position = process_position
         self.weights_summary = weights_summary
-        self.max_nb_epochs = max_nb_epochs
-        self.min_nb_epochs = min_nb_epochs
-        self.nb_sanity_val_steps = nb_sanity_val_steps
+
+        # Backward compatibility
+        if max_nb_epochs is not None:
+            warnings.warn("`max_nb_epochs` has renamed to `max_epochs` since v0.5.0"
+                          " and will be removed in v0.8.0", DeprecationWarning)
+            if not max_epochs:  # in case you did not set the proper value
+                max_epochs = max_nb_epochs
+        self.max_epochs = max_epochs
+
+        # Backward compatibility
+        if min_nb_epochs is not None:
+            warnings.warn("`min_nb_epochs` has renamed to `min_epochs` since v0.5.0"
+                          " and will be removed in v0.8.0", DeprecationWarning)
+            if not min_epochs:  # in case you did not set the proper value
+                min_epochs = min_nb_epochs
+        self.min_epochs = min_epochs
+
+        # Backward compatibility
+        if nb_sanity_val_steps is not None:
+            warnings.warn("`nb_sanity_val_steps` has renamed to `num_sanity_val_steps` since v0.5.0"
+                          " and will be removed in v0.8.0", DeprecationWarning)
+            if not num_sanity_val_steps:  # in case you did not set the proper value
+                num_sanity_val_steps = nb_sanity_val_steps
+
+        self.num_sanity_val_steps = num_sanity_val_steps
         self.print_nan_grads = print_nan_grads
         self.truncated_bptt_steps = truncated_bptt_steps
+        self.resume_from_checkpoint = resume_from_checkpoint
         self.shown_warnings = set()
 
         self.fast_dev_run = fast_dev_run
         if self.fast_dev_run:
-            self.nb_sanity_val_steps = 1
-            self.max_nb_epochs = 1
+            self.num_sanity_val_steps = 1
+            self.max_epochs = 1
             m = '''
             Running in fast_dev_run mode: will run a full train,
             val loop using a single batch
@@ -157,15 +208,15 @@ class Trainer(TrainerIOMixin,
             self.default_save_path = os.getcwd()
 
         # training bookeeping
-        self.total_batch_nb = 0
+        self.total_batch_idx = 0
         self.running_loss = []
         self.avg_loss = 0
-        self.batch_nb = 0
+        self.batch_idx = 0
         self.tqdm_metrics = {}
         self.callback_metrics = {}
-        self.nb_val_batches = 0
-        self.nb_training_batches = 0
-        self.nb_test_batches = 0
+        self.num_val_batches = 0
+        self.num_training_batches = 0
+        self.num_test_batches = 0
         self.get_train_dataloader = None
         self.get_test_dataloaders = None
         self.get_val_dataloaders = None
@@ -185,6 +236,8 @@ class Trainer(TrainerIOMixin,
         self.early_stop_callback = None
         self.configure_early_stopping(early_stop_callback, logger)
 
+        self.reduce_lr_on_plateau_scheduler = None
+
         # configure checkpoint callback
         self.checkpoint_callback = checkpoint_callback
         self.weights_save_path = weights_save_path
@@ -202,13 +255,13 @@ class Trainer(TrainerIOMixin,
         self.use_dp = False
         self.single_gpu = False
         self.distributed_backend = distributed_backend
-        self.set_distributed_mode(distributed_backend, nb_gpu_nodes)
+        self.set_distributed_mode(distributed_backend, num_nodes)
 
         # init flags for SLURM+ddp to work
         self.proc_rank = 0
         self.world_size = 1
         self.node_rank = 0
-        self.configure_slurm_ddp(nb_gpu_nodes)
+        self.configure_slurm_ddp(num_nodes)
 
         # nvidia setup
         self.set_nvidia_flags(self.is_slurm_managing_tasks, self.data_parallel_device_ids)
@@ -220,11 +273,13 @@ class Trainer(TrainerIOMixin,
         # logging
         self.log_save_interval = log_save_interval
         self.val_check_interval = val_check_interval
-        if not (add_row_log_interval is None):
-            # backward compatibility
-            warnings.warn("gradient_clip has renamed to gradient_clip_val since v0.5.0",
-                          DeprecationWarning)
-            row_log_interval = add_row_log_interval
+
+        # backward compatibility
+        if add_row_log_interval is not None:
+            warnings.warn("`add_row_log_interval` has renamed to `row_log_interval` since v0.5.0"
+                          " and will be removed in v0.8.0", DeprecationWarning)
+            if not row_log_interval:  # in case you did not set the proper value
+                row_log_interval = add_row_log_interval
         self.row_log_interval = row_log_interval
 
         # how much of the data to use
@@ -235,37 +290,35 @@ class Trainer(TrainerIOMixin,
         self.amp_level = amp_level
         self.init_amp(use_amp)
 
-        # set logging options
-        logging.basicConfig(level=logging.INFO)
-
     @property
     def slurm_job_id(self):
         try:
             job_id = os.environ['SLURM_JOB_ID']
             job_id = int(job_id)
-        except Exception as e:
+        except Exception:
             job_id = None
         return job_id
 
     def __parse_gpu_ids(self, gpus):
-        """
-        :param gpus: Int, string or list of ids
-        :return:
+        """Parse GPUs id.
+
+        :param list|str|int gpus: input GPU ids
+        :return list(int):
         """
         # if gpus = -1 then use all available devices
         # otherwise, split the string using commas
         if gpus is not None:
-            if type(gpus) is list:
+            if isinstance(gpus, list):
                 gpus = gpus
-            elif type(gpus) is str:
+            elif isinstance(gpus, str):
                 if gpus == '-1':
                     gpus = list(range(0, torch.cuda.device_count()))
                 else:
                     gpus = [int(x.strip()) for x in gpus.split(',')]
-            elif type(gpus) is int:
+            elif isinstance(gpus, int):
                 gpus = gpus
             else:
-                raise Exception('gpus has to be a string, int or list of ints')
+                raise ValueError('`gpus` has to be a string, int or list of ints')
 
         return gpus
 
@@ -294,20 +347,19 @@ class Trainer(TrainerIOMixin,
 
     @property
     def training_tqdm_dict(self):
-        """
-        Read-only for tqdm metrics
+        """Read-only for tqdm metrics.
         :return:
         """
         tqdm_dict = {
             'loss': '{0:.3f}'.format(self.avg_loss),
-            'batch_nb': '{}'.format(self.batch_nb),
+            'batch_idx': '{}'.format(self.batch_idx),
         }
 
         if self.truncated_bptt_steps is not None:
-            tqdm_dict['split_nb'] = self.split_nb
+            tqdm_dict['split_idx'] = self.split_idx
 
         if self.logger is not None and self.logger.version is not None:
-            tqdm_dict['v_nb'] = self.logger.version
+            tqdm_dict['v_num'] = self.logger.version
 
         tqdm_dict.update(self.tqdm_metrics)
 
@@ -318,12 +370,13 @@ class Trainer(TrainerIOMixin,
 
     @property
     def tng_tqdm_dic(self):
-        """
-        * Deprecated in v0.5.0. use training_tqdm_dict instead. *
+        """Read-only for tqdm metrics.
+
+        .. warning:: Deprecated in v0.5.0. use training_tqdm_dict instead.
         :return:
         """
-        warnings.warn("tng_tqdm_dict has renamed to training_tqdm_dict since v0.5.0",
-                      DeprecationWarning)
+        warnings.warn("`tng_tqdm_dic` has renamed to `training_tqdm_dict` since v0.5.0"
+                      " and will be removed in v0.8.0", DeprecationWarning)
         return self.training_tqdm_dict
 
     # -----------------------------
@@ -354,8 +407,7 @@ class Trainer(TrainerIOMixin,
         else:
             # run through amp wrapper
             if self.use_amp:
-                raise MisconfigurationException('amp + cpu is not supported.'
-                                                ' Please use a GPU option')
+                raise MisconfigurationException('amp + cpu is not supported.  Please use a GPU option')
 
             # CHOOSE OPTIMIZER
             # allow for lr schedulers as well
@@ -376,17 +428,24 @@ class Trainer(TrainerIOMixin,
         # two lists
         elif len(optimizers) == 2 and isinstance(optimizers[0], list):
             optimizers, lr_schedulers = optimizers
+            lr_schedulers, self.reduce_lr_on_plateau_scheduler = self.configure_schedulers(lr_schedulers)
             return optimizers, lr_schedulers
 
         # single list or tuple
         elif isinstance(optimizers, list) or isinstance(optimizers, tuple):
             return optimizers, []
 
+    def configure_schedulers(self, schedulers):
+        for i, scheduler in enumerate(schedulers):
+            if isinstance(scheduler, torch.optim.lr_scheduler.ReduceLROnPlateau):
+                reduce_lr_on_plateau_scheduler = schedulers.pop(i)
+                return schedulers, reduce_lr_on_plateau_scheduler
+        return schedulers, None
+
     def run_pretrain_routine(self, model):
-        """
-        Sanity check a few things before starting actual training
+        """Sanity check a few things before starting actual training.
+
         :param model:
-        :return:
         """
         ref_model = model
         if self.data_parallel:
@@ -443,16 +502,18 @@ class Trainer(TrainerIOMixin,
         # run tiny validation (if validation defined)
         # to make sure program won't crash during val
         ref_model.on_sanity_check_start()
-        if self.get_val_dataloaders() is not None and self.nb_sanity_val_steps > 0:
+        ref_model.on_train_start()
+        if self.get_val_dataloaders() is not None and self.num_sanity_val_steps > 0:
             # init progress bars for validation sanity check
-            pbar = tqdm.tqdm(desc='Validation sanity check', total=self.nb_sanity_val_steps,
+            pbar = tqdm.tqdm(desc='Validation sanity check',
+                             total=self.num_sanity_val_steps * len(self.get_val_dataloaders()),
                              leave=False, position=2 * self.process_position,
                              disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch')
             self.main_progress_bar = pbar
             # dummy validation progress bar
             self.val_progress_bar = tqdm.tqdm(disable=True)
 
-            self.evaluate(model, self.get_val_dataloaders(), self.nb_sanity_val_steps, self.testing)
+            self.evaluate(model, self.get_val_dataloaders(), self.num_sanity_val_steps, self.testing)
 
             # close progress bars
             self.main_progress_bar.close()
@@ -460,7 +521,8 @@ class Trainer(TrainerIOMixin,
 
         # init progress bar
         pbar = tqdm.tqdm(leave=True, position=2 * self.process_position,
-                         disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch')
+                         disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch',
+                         file=sys.stdout)
         self.main_progress_bar = pbar
 
         # clear cache before training

pytorch_lightning/trainer/trainer_io.py

@@ -1,18 +1,129 @@
+"""
+Lightning can automate saving and loading checkpoints
+=====================================================
+
+Checkpointing is enabled by default to the current working directory.
+To change the checkpoint path pass in::
+
+    Trainer(default_save_path='/your/path/to/save/checkpoints')
+
+
+To modify the behavior of checkpointing pass in your own callback.
+
+.. code-block:: python
+
+    from pytorch_lightning.callbacks import ModelCheckpoint
+
+    # DEFAULTS used by the Trainer
+    checkpoint_callback = ModelCheckpoint(
+        filepath=os.getcwd(),
+        save_best_only=True,
+        verbose=True,
+        monitor='val_loss',
+        mode='min',
+        prefix=''
+    )
+
+    trainer = Trainer(checkpoint_callback=checkpoint_callback)
+
+
+Restoring training session
+--------------------------
+
+You might want to not only load a model but also continue training it. Use this method to
+restore the trainer state as well. This will continue from the epoch and global step you last left off.
+However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).
+
+Lightning will restore the session if you pass a logger with the same version and there's a saved checkpoint.
+
+.. code-block:: python
+
+    from pytorch_lightning import Trainer
+    from pytorch_lightning.logging import TestTubeLogger
+
+    logger = TestTubeLogger(
+        save_dir='./savepath',
+        version=1  # An existing version with a saved checkpoint
+    )
+    trainer = Trainer(
+        logger=logger,
+        default_save_path='./savepath'
+    )
+
+    # this fit call loads model weights and trainer state
+    # the trainer continues seamlessly from where you left off
+    # without having to do anything else.
+    trainer.fit(model)
+
+
+The trainer restores:
+
+- global_step
+- current_epoch
+- All optimizers
+- All lr_schedulers
+- Model weights
+
+You can even change the logic of your model as long as the weights and "architecture" of
+the system isn't different. If you add a layer, for instance, it might not work.
+
+At a rough level, here's what happens inside Trainer :py:mod:`pytorch_lightning.base_module.model_saving.py`:
+
+.. code-block:: python
+
+    self.global_step = checkpoint['global_step']
+    self.current_epoch = checkpoint['epoch']
+
+    # restore the optimizers
+    optimizer_states = checkpoint['optimizer_states']
+    for optimizer, opt_state in zip(self.optimizers, optimizer_states):
+        optimizer.load_state_dict(opt_state)
+
+    # restore the lr schedulers
+    lr_schedulers = checkpoint['lr_schedulers']
+    for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
+        scheduler.load_state_dict(lrs_state)
+
+    # uses the model you passed into trainer
+    model.load_state_dict(checkpoint['state_dict'])
+
+"""
+
 import os
 import re
 import signal
 import warnings
 from subprocess import call
 import logging
+from abc import ABC
 
 import torch
 import torch.distributed as dist
 
-from pytorch_lightning.pt_overrides.override_data_parallel import (
-    LightningDistributedDataParallel, LightningDataParallel)
+from pytorch_lightning.overrides.data_parallel import (
+    LightningDistributedDataParallel,
+    LightningDataParallel,
+)
 
 
-class TrainerIOMixin(object):
+class TrainerIOMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.model = None
+        self.on_gpu = None
+        self.root_gpu = None
+        self.resume_from_checkpoint = None
+        self.use_ddp = None
+        self.use_ddp2 = None
+        self.checkpoint_callback = None
+        self.proc_rank = None
+        self.weights_save_path = None
+        self.logger = None
+        self.early_stop_callback = None
+        self.lr_schedulers = None
+        self.optimizers = None
 
     def get_model(self):
         is_dp_module = isinstance(self.model, (LightningDistributedDataParallel,
@@ -45,8 +156,11 @@ class TrainerIOMixin(object):
             torch.cuda.empty_cache()
 
         if not did_restore_hpc_weights:
-            # restore weights if same exp version
-            self.restore_state_if_checkpoint_exists(model)
+            if self.resume_from_checkpoint is not None:
+                self.restore(self.resume_from_checkpoint, on_gpu=self.on_gpu)
+            else:
+                # restore weights if same exp version
+                self.restore_state_if_checkpoint_exists(model)
 
         # wait for all models to restore weights
         if self.use_ddp or self.use_ddp2:

pytorch_lightning/trainer/training_loop.py

@@ -0,0 +1,601 @@
+"""
+The lightning training loop handles everything except the actual computations of your model.
+ To decide what will happen in your training loop, define the `training_step` function.
+
+Below are all the things lightning automates for you in the training loop.
+
+Accumulated gradients
+---------------------
+
+Accumulated gradients runs K small batches of size N before doing a backwards pass.
+ The effect is a large effective batch size of size KxN.
+
+.. code-block:: python
+
+    # DEFAULT (ie: no accumulated grads)
+    trainer = Trainer(accumulate_grad_batches=1)
+
+Force training for min or max epochs
+------------------------------------
+
+It can be useful to force training for a minimum number of epochs or limit to a max number
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(min_epochs=1, max_epochs=1000)
+
+Early stopping
+--------------
+
+The trainer already sets up default early stopping for you.
+To modify this behavior, pass in your own EarlyStopping callback.
+
+.. code-block:: python
+
+    from pytorch_lightning.callbacks import EarlyStopping
+
+    # DEFAULTS used by Trainer
+    early_stop_callback = EarlyStopping(
+        monitor='val_loss',
+        min_delta=0.00,
+        patience=3,
+        verbose=False,
+        mode='min'
+    )
+
+    # without passing anything in, uses the default callback above
+    trainer = Trainer()
+
+    # pass in your own to override the default callback
+    trainer = Trainer(early_stop_callback=early_stop_callback)
+
+    # pass in min_epochs to enable the callback after min_epochs have run
+    trainer = Trainer(early_stop_callback=early_stop_callback, min_epochs=5)
+
+    # pass in None to disable it
+    trainer = Trainer(early_stop_callback=None)
+
+Force disable early stop
+------------------------
+
+To disable early stopping pass None to the early_stop_callback
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(early_stop_callback=None)
+
+Gradient Clipping
+-----------------
+
+Gradient clipping may be enabled to avoid exploding gradients.
+ Specifically, this will `clip the gradient norm computed over all model parameters
+ `together <https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_>`_.
+
+.. code-block:: python
+
+    # DEFAULT (ie: don't clip)
+    trainer = Trainer(gradient_clip_val=0)
+
+    # clip gradients with norm above 0.5
+    trainer = Trainer(gradient_clip_val=0.5)
+
+Inspect gradient norms
+----------------------
+
+Looking at grad norms can help you figure out where training might be going wrong.
+
+.. code-block:: python
+
+    # DEFAULT (-1 doesn't track norms)
+    trainer = Trainer(track_grad_norm=-1)
+
+    # track the LP norm (P=2 here)
+    trainer = Trainer(track_grad_norm=2)
+
+Set how much of the training set to check
+-----------------------------------------
+
+If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
+
+train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(train_percent_check=1.0)
+
+    # check 10% only
+    trainer = Trainer(train_percent_check=0.1)
+
+Packed sequences as inputs
+--------------------------
+
+When using PackedSequence, do 2 things:
+1. return either a padded tensor in dataset or a list of variable length tensors
+ in the dataloader collate_fn (example above shows the list implementation).
+2. Pack the sequence in forward or training and validation steps depending on use case.
+
+.. code-block:: python
+
+    # For use in dataloader
+    def collate_fn(batch):
+        x = [item[0] for item in batch]
+        y = [item[1] for item in batch]
+        return x, y
+
+    # In module
+    def training_step(self, batch, batch_idx):
+        x = rnn.pack_sequence(batch[0], enforce_sorted=False)
+        y = rnn.pack_sequence(batch[1], enforce_sorted=False)
+
+
+Truncated Backpropagation Through Time
+--------------------------------------
+
+There are times when multiple backwards passes are needed for each batch.
+ For example, it may save memory to use Truncated Backpropagation Through Time when training RNNs.
+
+When this flag is enabled each batch is split into sequences of size truncated_bptt_steps
+ and passed to training_step(...) separately. A default splitting function is provided,
+ however, you can override it for more flexibility. See `tbptt_split_batch`.
+
+.. code-block:: python
+
+    # DEFAULT (single backwards pass per batch)
+    trainer = Trainer(truncated_bptt_steps=None)
+
+    # (split batch into sequences of size 2)
+    trainer = Trainer(truncated_bptt_steps=2)
+
+
+"""
+
+import inspect
+from abc import ABC, abstractmethod
+import warnings
+
+import numpy as np
+
+from pytorch_lightning.utilities.debugging import MisconfigurationException
+
+try:
+    from apex import amp
+
+    APEX_AVAILABLE = True
+except ImportError:
+    APEX_AVAILABLE = False
+
+
+class TrainerTrainLoopMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.max_epochs = None
+        self.min_epochs = None
+        self.use_ddp = None
+        self.use_dp = None
+        self.use_ddp2 = None
+        self.single_gpu = None
+        self.data_parallel_device_ids = None
+        self.check_val_every_n_epoch = None
+        self.num_training_batches = None
+        self.val_check_batch = None
+        self.num_val_batches = None
+        self.fast_dev_run = None
+        self.is_iterable_train_dataloader = None
+        self.main_progress_bar = None
+        self.accumulation_scheduler = None
+        self.lr_schedulers = None
+        self.enable_early_stop = None
+        self.early_stop_callback = None
+        self.callback_metrics = None
+        self.logger = None
+        self.global_step = None
+        self.testing = None
+        self.log_save_interval = None
+        self.proc_rank = None
+        self.row_log_interval = None
+        self.total_batches = None
+        self.truncated_bptt_steps = None
+        self.optimizers = None
+        self.accumulate_grad_batches = None
+        self.use_amp = None
+        self.print_nan_grads = None
+        self.track_grad_norm = None
+        self.model = None
+        self.running_loss = None
+        self.training_tqdm_dict = None
+        self.get_train_dataloader = None
+        self.reduce_lr_on_plateau_scheduler = None
+
+    @property
+    def max_nb_epochs(self):
+        """
+        .. warning:: `max_nb_epochs` is deprecated and will be removed in v0.8.0, use `max_epochs` instead.
+        """
+        warnings.warn("`max_nb_epochs` is deprecated and will be removed in "
+                      "v0.8.0, use `max_epochs` instead.", DeprecationWarning)
+        return self.max_epochs
+
+    @property
+    def min_nb_epochs(self):
+        """
+        .. warning:: `min_nb_epochs` is deprecated and will be removed in v0.8.0, use `min_epochs` instead.
+        """
+        warnings.warn("`min_nb_epochs` is deprecated and will be removed in "
+                      "v0.8.0, use `min_epochs` instead.", DeprecationWarning)
+        return self.min_epochs
+
+    @abstractmethod
+    def get_model(self):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def is_function_implemented(self, m):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def run_evaluation(self, test):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def transfer_batch_to_gpu(self, batch, gpu):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def clip_gradients(self):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def print_nan_gradients(self):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def is_overriden(self, m):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def add_tqdm_metrics(self, metrics):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def log_metrics(self, metrics, grad_norm_dic):
+        # this is just empty shell for code from other class
+        pass
+
+    @abstractmethod
+    def process_output(self, output, train):
+        # this is just empty shell for code from other class
+        pass
+
+    def train(self):
+        model = self.get_model()
+        # run all epochs
+        for epoch in range(self.current_epoch, self.max_epochs):
+            # set seed for distributed sampler (enables shuffling for each epoch)
+            if self.use_ddp and hasattr(self.get_train_dataloader().sampler, 'set_epoch'):
+                self.get_train_dataloader().sampler.set_epoch(epoch)
+
+            # get model
+            model = self.get_model()
+
+            # update training progress in trainer and model
+            model.current_epoch = epoch
+            self.current_epoch = epoch
+
+            # val can be checked multiple times in epoch
+            is_val_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
+            val_checks_per_epoch = self.num_training_batches // self.val_check_batch
+            val_checks_per_epoch = val_checks_per_epoch if is_val_epoch else 0
+
+            # total batches includes multiple val checks
+            self.total_batches = (self.num_training_batches +
+                                  self.num_val_batches * val_checks_per_epoch)
+            self.batch_loss_value = 0  # accumulated grads
+
+            if self.fast_dev_run:
+                # limit the number of batches to 2 (1 train and 1 val) in fast_dev_run
+                num_iterations = 2
+            elif self.is_iterable_train_dataloader:
+                # for iterable train loader, the progress bar never ends
+                num_iterations = None
+            else:
+                num_iterations = self.total_batches
+
+            # reset progress bar
+            # .reset() doesn't work on disabled progress bar so we should check
+            if not self.main_progress_bar.disable:
+                self.main_progress_bar.reset(num_iterations)
+            desc = f'Epoch {epoch + 1}' if not self.is_iterable_train_dataloader else ''
+            self.main_progress_bar.set_description(desc)
+
+            # changing gradient according accumulation_scheduler
+            self.accumulation_scheduler.on_epoch_begin(epoch, self)
+
+            # -----------------
+            # RUN TNG EPOCH
+            # -----------------
+            self.run_training_epoch()
+
+            # update LR schedulers
+            if self.lr_schedulers is not None:
+                for lr_scheduler in self.lr_schedulers:
+                    lr_scheduler.step(epoch=self.current_epoch)
+            if self.reduce_lr_on_plateau_scheduler is not None:
+                val_loss = self.callback_metrics.get('val_loss')
+                if val_loss is None:
+                    avail_metrics = ','.join(list(self.callback_metrics.keys()))
+                    m = f'ReduceLROnPlateau conditioned on metric val_loss ' \
+                        f'which is not available. Available metrics are: {avail_metrics}'
+                    raise MisconfigurationException(m)
+                self.reduce_lr_on_plateau_scheduler.step(val_loss, epoch=self.current_epoch)
+
+            # early stopping
+            met_min_epochs = epoch >= self.min_epochs - 1
+            if self.enable_early_stop and (met_min_epochs or self.fast_dev_run):
+                should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch,
+                                                                    logs=self.callback_metrics)
+                # stop training
+                stop = should_stop and met_min_epochs
+                if stop:
+                    self.main_progress_bar.close()
+                    return
+
+        self.main_progress_bar.close()
+
+        model.on_train_end()
+
+        if self.logger is not None:
+            self.logger.finalize("success")
+
+    def run_training_epoch(self):
+        # before epoch hook
+        if self.is_function_implemented('on_epoch_start'):
+            model = self.get_model()
+            model.on_epoch_start()
+
+        # run epoch
+        for batch_idx, batch in enumerate(self.get_train_dataloader()):
+            # stop epoch if we limited the number of training batches
+            if batch_idx >= self.num_training_batches:
+                break
+
+            self.batch_idx = batch_idx
+
+            model = self.get_model()
+            model.global_step = self.global_step
+
+            # ---------------
+            # RUN TRAIN STEP
+            # ---------------
+            output = self.run_training_batch(batch, batch_idx)
+            batch_result, grad_norm_dic, batch_step_metrics = output
+
+            # when returning -1 from train_step, we end epoch early
+            early_stop_epoch = batch_result == -1
+
+            # ---------------
+            # RUN VAL STEP
+            # ---------------
+            is_val_check_batch = (batch_idx + 1) % self.val_check_batch == 0
+            can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
+            should_check_val = ((is_val_check_batch or early_stop_epoch) and can_check_epoch)
+
+            # fast_dev_run always forces val checking after train batch
+            if self.fast_dev_run or should_check_val:
+                self.run_evaluation(test=self.testing)
+
+            # when logs should be saved
+            should_save_log = (batch_idx + 1) % self.log_save_interval == 0 or early_stop_epoch
+            if should_save_log or self.fast_dev_run:
+                if self.proc_rank == 0 and self.logger is not None:
+                    self.logger.save()
+
+            # when metrics should be logged
+            should_log_metrics = batch_idx % self.row_log_interval == 0 or early_stop_epoch
+            if should_log_metrics or self.fast_dev_run:
+                # logs user requested information to logger
+                self.log_metrics(batch_step_metrics, grad_norm_dic)
+
+            self.global_step += 1
+            self.total_batch_idx += 1
+
+            # end epoch early
+            # stop when the flag is changed or we've gone past the amount
+            # requested in the batches
+            if early_stop_epoch or self.fast_dev_run:
+                break
+
+        # epoch end hook
+        if self.is_function_implemented('on_epoch_end'):
+            model = self.get_model()
+            model.on_epoch_end()
+
+    def run_training_batch(self, batch, batch_idx):
+        # track grad norms
+        grad_norm_dic = {}
+
+        # track all metrics for callbacks
+        all_callback_metrics = []
+
+        # track metrics to log
+        all_log_metrics = []
+
+        if batch is None:
+            return 0, grad_norm_dic, {}
+
+        # hook
+        if self.is_function_implemented('on_batch_start'):
+            model_ref = self.get_model()
+            response = model_ref.on_batch_start(batch)
+
+            if response == -1:
+                return -1, grad_norm_dic, {}
+
+        splits = [batch]
+        if self.truncated_bptt_steps is not None:
+            model_ref = self.get_model()
+            splits = model_ref.tbptt_split_batch(batch, self.truncated_bptt_steps)
+
+        self.hiddens = None
+        for split_idx, split_batch in enumerate(splits):
+            self.split_idx = split_idx
+
+            # call training_step once per optimizer
+            for opt_idx, optimizer in enumerate(self.optimizers):
+                # make sure only the gradients of the current optimizer's paramaters are calculated 
+                # in the training step to prevent dangling gradients in multiple-optimizer setup.
+                for param in self.get_model().parameters():
+                    param.requires_grad = False
+                for group in optimizer.param_groups:
+                    for param in group['params']:
+                        param.requires_grad = True
+
+                # wrap the forward step in a closure so second order methods work
+                def optimizer_closure():
+                    # forward pass
+                    output = self.training_forward(
+                        split_batch, batch_idx, opt_idx, self.hiddens)
+
+                    closure_loss = output[0]
+                    progress_bar_metrics = output[1]
+                    log_metrics = output[2]
+                    callback_metrics = output[3]
+                    self.hiddens = output[4]
+
+                    # accumulate loss
+                    # (if accumulate_grad_batches = 1 no effect)
+                    closure_loss = closure_loss / self.accumulate_grad_batches
+
+                    # backward pass
+                    model_ref = self.get_model()
+                    model_ref.backward(self.use_amp, closure_loss, optimizer)
+
+                    # track metrics for callbacks
+                    all_callback_metrics.append(callback_metrics)
+
+                    # track progress bar metrics
+                    self.add_tqdm_metrics(progress_bar_metrics)
+                    all_log_metrics.append(log_metrics)
+
+                    # insert after step hook
+                    if self.is_function_implemented('on_after_backward'):
+                        model_ref = self.get_model()
+                        model_ref.on_after_backward()
+
+                    return closure_loss
+
+                # calculate loss
+                loss = optimizer_closure()
+
+                # nan grads
+                if self.print_nan_grads:
+                    self.print_nan_gradients()
+
+                # track total loss for logging (avoid mem leaks)
+                self.batch_loss_value += loss.item()
+
+                # gradient update with accumulated gradients
+                if (self.batch_idx + 1) % self.accumulate_grad_batches == 0:
+
+                    # track gradient norms when requested
+                    if batch_idx % self.row_log_interval == 0:
+                        if self.track_grad_norm > 0:
+                            model = self.get_model()
+                            grad_norm_dic = model.grad_norm(
+                                self.track_grad_norm)
+
+                    # clip gradients
+                    self.clip_gradients()
+
+                    # calls .step(), .zero_grad()
+                    # override function to modify this behavior
+                    model = self.get_model()
+                    model.optimizer_step(self.current_epoch, batch_idx,
+                                         optimizer, opt_idx, optimizer_closure)
+
+                    # calculate running loss for display
+                    self.running_loss.append(self.batch_loss_value)
+                    self.batch_loss_value = 0
+                    self.avg_loss = np.mean(self.running_loss[-100:])
+
+        # activate batch end hook
+        if self.is_function_implemented('on_batch_end'):
+            model = self.get_model()
+            model.on_batch_end()
+
+        # update progress bar
+        self.main_progress_bar.update(1)
+        self.main_progress_bar.set_postfix(**self.training_tqdm_dict)
+
+        # collapse all metrics into one dict
+        all_log_metrics = {k: v for d in all_log_metrics for k, v in d.items()}
+
+        # track all metrics for callbacks
+        self.callback_metrics.update({k: v for d in all_callback_metrics for k, v in d.items()})
+
+        return 0, grad_norm_dic, all_log_metrics
+
+    def training_forward(self, batch, batch_idx, opt_idx, hiddens):
+        """
+        Handle forward for each training case (distributed, single gpu, etc...)
+        :param batch:
+        :param batch_idx:
+        :return:
+        """
+        # ---------------
+        # FORWARD
+        # ---------------
+        # enable not needing to add opt_idx to training_step
+        args = [batch, batch_idx]
+
+        if len(self.optimizers) > 1:
+            if self.has_arg('training_step', 'optimizer_idx'):
+                args.append(opt_idx)
+            else:
+                raise ValueError(
+                    f'Your LightningModule defines {len(self.optimizers)} optimizers but '
+                    f'training_step is missing the "optimizer_idx" argument.'
+                )
+
+        # pass hiddens if using tbptt
+        if self.truncated_bptt_steps is not None:
+            args.append(hiddens)
+
+        # distributed forward
+        if self.use_ddp or self.use_ddp2 or self.use_dp:
+            output = self.model(*args)
+
+        # single GPU forward
+        elif self.single_gpu:
+            gpu_id = 0
+            if isinstance(self.data_parallel_device_ids, list):
+                gpu_id = self.data_parallel_device_ids[0]
+            batch = self.transfer_batch_to_gpu(batch.copy(), gpu_id)
+            args[0] = batch
+            output = self.model.training_step(*args)
+
+        # CPU forward
+        else:
+            output = self.model.training_step(*args)
+
+        # allow any mode to define training_end
+        if self.is_overriden('training_end'):
+            model_ref = self.get_model()
+            output = model_ref.training_end(output)
+
+        # format and reduce outputs accordingly
+        output = self.process_output(output, train=True)
+
+        return output

pytorch_lightning/trainer/training_tricks.py

@@ -1,9 +1,21 @@
+from abc import ABC, abstractmethod
+
 import torch
 import logging
 from pytorch_lightning.callbacks import GradientAccumulationScheduler
 
 
-class TrainerTrainingTricksMixin(object):
+class TrainerTrainingTricksMixin(ABC):
+
+    def __init__(self):
+        # this is just a summary on variables used in this abstract class,
+        #  the proper values/initialisation should be done in child class
+        self.gradient_clip_val = None
+
+    @abstractmethod
+    def get_model(self):
+        # this is just empty shell for code from other class
+        pass
 
     def clip_gradients(self):
         if self.gradient_clip_val > 0:
@@ -13,7 +25,7 @@ class TrainerTrainingTricksMixin(object):
     def print_nan_gradients(self):
         model = self.get_model()
         for param in model.parameters():
-            if torch.isnan(param.grad.float()).any():
+            if (param.grad is not None) and torch.isnan(param.grad.float()).any():
                 logging.info(param, param.grad)
 
     def configure_accumulated_gradients(self, accumulate_grad_batches):

pytorch_lightning/utilities/arg_parse.py

@@ -15,8 +15,10 @@ def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None
     parser.opt_list('--accumulate_grad_batches', default=1, type=int, tunable=False,
                     help='accumulates gradients k times before applying update.'
                          ' Simulates huge batch size')
-    parser.add_argument('--max_nb_epochs', default=200, type=int, help='cap epochs')
-    parser.add_argument('--min_nb_epochs', default=2, type=int, help='min epochs')
+    parser.add_argument('--max_epochs', default=200, type=int,
+                        help='maximum number of epochs')
+    parser.add_argument('--min_epochs', default=2, type=int,
+                        help='minimum number of epochs')
     parser.add_argument('--train_percent_check', default=1.0, type=float,
                         help='how much of training set to check')
     parser.add_argument('--val_percent_check', default=1.0, type=float,
@@ -81,7 +83,7 @@ def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None
     parser.add_argument('--enable_tqdm', dest='enable_tqdm', default=False, action='store_true',
                         help='false removes the progress bar')
     parser.add_argument('--overfit', default=-1, type=float,
-                        help='% of dataset to use with this option. float, or -1 for none')
+                        help='%% of dataset to use with this option. float, or -1 for none')
 
     # debug args
     if rand_seed is not None:

pytorch_lightning/utilities/debugging.py

@@ -1,2 +1,78 @@
+"""
+These flags are useful to help debug a model.
+
+Fast dev run
+------------
+
+This flag is meant for debugging a full train/val/test loop.
+ It'll activate callbacks, everything but only with 1 training and 1 validation batch.
+ Use this to debug a full run of your program quickly
+
+.. code-block:: python
+
+    # DEFAULT
+    trainer = Trainer(fast_dev_run=False)
+
+
+Inspect gradient norms
+----------------------
+
+Looking at grad norms can help you figure out where training might be going wrong.
+
+.. code-block:: python
+
+    # DEFAULT (-1 doesn't track norms)
+    trainer = Trainer(track_grad_norm=-1)
+
+    # track the LP norm (P=2 here)
+    trainer = Trainer(track_grad_norm=2)
+
+
+Make model overfit on subset of data
+------------------------------------
+
+A useful debugging trick is to make your model overfit a tiny fraction of the data.
+
+setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
+
+.. code-block:: python
+
+    # DEFAULT don't overfit (ie: normal training)
+    trainer = Trainer(overfit_pct=0.0)
+
+    # overfit on 1% of data
+    trainer = Trainer(overfit_pct=0.01)
+
+
+Print the parameter count by layer
+----------------------------------
+
+By default lightning prints a list of parameters *and submodules* when it starts training.
+
+.. code-block:: python
+
+    # DEFAULT print a full list of all submodules and their parameters.
+    trainer = Trainer(weights_summary='full')
+
+    # only print the top-level modules (i.e. the children of LightningModule).
+    trainer = Trainer(weights_summary='top')
+
+Print which gradients are nan
+-----------------------------
+
+This option prints a list of tensors with nan gradients::
+
+    # DEFAULT
+    trainer = Trainer(print_nan_grads=False)
+
+Log GPU usage
+-------------
+
+Lightning automatically logs gpu usage to the test tube logs.
+ It'll only do it at the metric logging interval, so it doesn't slow down training.
+
+"""
+
+
 class MisconfigurationException(Exception):
     pass