release v

fix python syntax in code blocks to be consistent (#166 )
A couple code blocks used "{.python}" instead of just "python" for the syntax highlighting, which doesn't render properly in GitHub markdown.
2026-09-12 12:40:20 +08:00 · 2019-08-24 08:13:54 -04:00 · 2019-08-23 21:24:18 -04:00 · 2019-08-23 21:23:27 -04:00 · 2019-08-23 16:10:45 -04:00 · 2019-08-23 16:10:24 -04:00
38 changed files with 1868 additions and 518 deletions
@@ -1,4 +1,7 @@
-#see https://github.com/codecov/support/wiki/Codecov-Yaml
+# see https://docs.codecov.io/docs/codecov-yaml
+# Validation check:
+# $ curl --data-binary @.codecov.yml https://codecov.io/validate
+
 codecov:
  notify:
    require_ci_to_pass: yes
@@ -41,4 +44,3 @@ comment:
  behavior: default  # update if exists else create new
  # branches: *

-
@@ -7,6 +7,10 @@ assignees: ''

 ---

+### Common bugs:
+1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/williamFalcon/pytorch-lightning/issues/79).    
+2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
+
 **Describe the bug**
 A clear and concise description of what the bug is.

@@ -28,11 +32,5 @@ If applicable, add screenshots to help explain your problem.
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
-
 **Additional context**
 Add any other context about the problem here.
@@ -0,0 +1,26 @@
+---
+name: How to question
+about: Asking how-to questions
+title: ''
+labels: question
+assignees: ''
+
+---
+
+### Before asking:   
+1. search the issues.   
+2. search the docs.    
+
+If you still can't find what you need:     
+#### What is your question?    
+
+#### Code    
+Please paste a code snippet if your question requires it!   
+
+#### What have you tried?    
+
+#### What's your environment?   
+- conda version (no venv)   
+- PyTorch version   
+- Lightning version   
+- Test-tube version
@@ -11,6 +11,7 @@ pip-wheel-metadata/
 test_tube_exp/
 tests/tests_tt_dir/
 tests/save_dir
+default/

 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at waf2107@columbia.edu. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq
@@ -1,5 +1,10 @@
 # Contributing    
-Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!  
+Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!   
+
+## One less thing to remember    
+Simplify the API as much as possible from the user perspective. Any additions or improvements should minimize things the user needs to remember.   
+
+For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.   

 ## Lightning Design Principles   
 We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
@@ -10,17 +10,17 @@
 [![PyPI Status](https://badge.fury.io/py/pytorch-lightning.svg)](https://badge.fury.io/py/pytorch-lightning)
 [![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
 [![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
-<!-- 
-removed until windows install issues resolved.
-[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning) -->
+[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning)
+[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
+[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
+
+[![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
+[![Gitter](https://badges.gitter.im/PyTorch-Lightning/community.svg)](https://gitter.im/PyTorch-Lightning/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
 <!-- 
 removed until codecov badge isn't empy. likely a config error showing nothing on master.
 [![codecov](https://codecov.io/gh/Borda/pytorch-lightning/branch/master/graph/badge.svg)](https://codecov.io/gh/Borda/pytorch-lightning)
 -->
-[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
-[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)
-[![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)

 </div>

@@ -33,15 +33,15 @@ pip install pytorch-lightning
 **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**

 ## What is it?  
-Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. It defers core training and validation logic to you and automates the rest. It guarantees tested, correct, modern best practices for the automated parts.
+Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.

+## Starting a new project?   
+[Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     

 ## Why do I want to use lightning?
-When starting a new project the last thing you want to do is recode a training loop, multi-cluster training, 16-bit precision, early-stopping, model loading/saving, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.
+Every research project starts the same, a model, a training loop, validation loop, etc. As your research advances, you're likely to need distributed training, 16-bit precision, checkpointing, gradient accumulation, etc.   

-With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: The data and the training/validation loop logic. 
-
-Don't worry about training on multiple gpus or speeding up your code, lightning will do that for you!
+Lightning sets up all the boilerplate state-of-the-art training for you so you can focus on the research.   

 ---   
 ## README Table of Contents        
@@ -53,14 +53,18 @@ Don't worry about training on multiple gpus or speeding up your code, lightning
 - [Tutorials](https://github.com/williamFalcon/pytorch-lightning#tutorials)
 - [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md)    
 - [Bleeding edge install](https://github.com/williamFalcon/pytorch-lightning#bleeding-edge)   
- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)
+- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)   
+- [Asking for help](https://github.com/williamFalcon/pytorch-lightning#asking-for-help)
 - [FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    

 ---   
 ## How do I do use it?   
+Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule]((https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)) which you fit using a Trainer.    
+
+The LightningModule defines a *system* such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier such as the example below.     

 To use lightning do 2 things:  
-1. [Define a LightningModel](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
+1. [Define a LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
 ```python
 import os
 import torch
@@ -71,46 +75,51 @@ import torchvision.transforms as transforms

 import pytorch_lightning as pl

-class CoolModel(pl.LightningModule):
+class CoolSystem(pl.LightningModule):

    def __init__(self):
-        super(CoolModel, self).__init__()
+        super(CoolSystem, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

-    def my_loss(self, y_hat, y):
-        return F.cross_entropy(y_hat, y)
-
    def training_step(self, batch, batch_nb):
+        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
-        return {'loss': self.my_loss(y_hat, y)}
+        return {'loss': F.cross_entropy(y_hat, y)}

    def validation_step(self, batch, batch_nb):
+        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
-        return {'val_loss': self.my_loss(y_hat, y)}
+        return {'val_loss': F.cross_entropy(y_hat, y)}

    def validation_end(self, outputs):
+        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}

    def configure_optimizers(self):
-        return [torch.optim.Adam(self.parameters(), lr=0.02)]
+        # REQUIRED
+        # can return multiple optimizers and learning_rate schedulers
+        return torch.optim.Adam(self.parameters(), lr=0.02)

    @pl.data_loader
    def tng_dataloader(self):
+        # REQUIRED
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

    @pl.data_loader
    def val_dataloader(self):
+        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

    @pl.data_loader
    def test_dataloader(self):
+        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```

@@ -118,7 +127,7 @@ class CoolModel(pl.LightningModule):
 ```python
 from pytorch_lightning import Trainer

-model = CoolModel()
+model = CoolSystem()

 # most basic trainer, uses good defaults
 trainer = Trainer()    
@@ -133,7 +142,7 @@ from test_tube import Experiment
 exp = Experiment(save_dir=os.getcwd())

 # train on cpu using only 10% of the data (for demo purposes)
-# pass in experi
+# pass in experiment for automatic tensorboard logging.    
 trainer = Trainer(experiment=exp, max_nb_epochs=1, train_percent_check=0.1)

 # train on 4 gpus
@@ -157,7 +166,7 @@ You define the blue parts using the LightningModule interface:

 ![Ouverview](./docs/source/_static/overview_flat.jpg)

-```{.python}
+```python
 # what to do in the training loop
 def training_step(self, data_batch, batch_nb):

@@ -244,7 +253,7 @@ Lightning also adds a text column with all the hyperparameters for this experime
 ![tensorboard-support](./docs/source/_static/tf_tags.png)

 Simply note the path you set for the Experiment    
-``` {.python}   
+```python
 from test_tube import Experiment
 from pytorch-lightning import  Trainer

@@ -312,6 +321,7 @@ tensorboard --logdir /some/path
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)    
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
+- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)

 ###### Validation loop    

@@ -350,8 +360,24 @@ python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'
 - [9 key speed features in Pytorch-Lightning](https://towardsdatascience.com/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565)    
 - [SLURM, multi-node training with Lightning](https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd)     

+---    
+## Asking for help    
+Welcome to the Lightning community!   
+
+If you have any questions, feel free to:   
+1. [read the docs](https://williamfalcon.github.io/pytorch-lightning/).     
+2. [Search through the issues](https://github.com/williamFalcon/pytorch-lightning/issues?utf8=%E2%9C%93&q=my++question).      
+3. [Ask on stackoverflow](https://stackoverflow.com/questions/ask?guided=false) with the tag pytorch-lightning.   
+
+If no one replies to you quickly enough, feel free to post the stackoverflow link to our Gitter chat!   
+
+To chat with the rest of us visit our [gitter channel](https://gitter.im/PyTorch-Lightning/community?utm_source=share-link&utm_medium=link&utm_campaign=share-link)!     
+
 ---   
 ## FAQ    
+**How do I use Lightning for rapid research?**   
+[Here's a walk-through](https://williamfalcon.github.io/pytorch-lightning/)  
+
 **Why was Lightning created?**     
 Lightning has 3 goals in mind:
 1. Maximal flexibility while abstracting out the common boilerplate across research projects.   
@@ -370,6 +396,21 @@ Nope.
 **Are there plans to support virtualenv?**    
 Nope. Please use anaconda or miniconda.    

+**Which PyTorch versions do you support?**    
+##### PyTorch 1.1.0       
+```bash    
+# install pytorch 1.1.0 using the official instructions   
+
+# install test-tube 0.6.7.6 which supports 1.1.0   
+pip install test-tube==0.6.7.6   
+
+# install latest Lightning version without upgrading deps    
+pip install -U --no-deps pytorch-lightning
+```     
+
+##### PyTorch 1.2.0   
+Install via pip as normal   
+
 ## Bleeding edge
 If you can't wait for the next release, install the most up to date code with:  
 ```bash
@@ -45,9 +45,7 @@ install:
  # directly to master instead of just PR builds (or the converse).
  - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%path%
  - pip install -U --user pip
-  - pip install "https://download.pytorch.org/whl/cu90/torch-1.1.0-cp%PIP_PYVER%-cp%PIP_PYVER%m-win_amd%PYTHON_ARCH%.whl"
-    pip install "https://download.pytorch.org/whl/cu90/torchvision-0.3.0-cp%PIP_PYVER%-cp%PIP_PYVER%m-win_amd%PYTHON_ARCH%.whl"
-  - pip install -r requirements.txt
+  - pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
  - pip install -r ./tests/requirements.txt

 # scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
@@ -9,22 +9,20 @@ Otherwise, to Define a Lightning Module, implement the following methods:

 **Required**:  

- [training_step](RequiredTrainerInterface.md#training_step)   
- [validation_step](RequiredTrainerInterface.md#validation_step)
- [validation_end](RequiredTrainerInterface.md#validation_end)
-    
- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)
-
- [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)
- [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)
- [test_dataloader](RequiredTrainerInterface.md#test_dataloader)
+- [training_step](RequiredTrainerInterface.md#training_step)      
+- [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)    
+- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)    

 **Optional**:   

- [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)
- [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)
- [update_tng_log_metrics](RequiredTrainerInterface.md#update_tng_log_metrics)
- [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)
+- [validation_step](RequiredTrainerInterface.md#validation_step)    
+- [validation_end](RequiredTrainerInterface.md#validation_end)    
+- [val_dataloader](RequiredTrainerInterface.md#val_dataloader)    
+- [test_dataloader](RequiredTrainerInterface.md#test_dataloader)    
+- [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)    
+- [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)    
+- [update_tng_log_metrics](RequiredTrainerInterface.md#update_tng_log_metrics)    
+- [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)    

 ---
 ### Minimal example   
@@ -48,24 +46,25 @@ class CoolModel(pl.LightningModule):
    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

-    def my_loss(self, y_hat, y):
-        return F.cross_entropy(y_hat, y)
-
    def training_step(self, batch, batch_nb):
+        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
-        return {'loss': self.my_loss(y_hat, y)}
+        return {'loss': F.cross_entropy(y_hat, y)}

    def validation_step(self, batch, batch_nb):
+        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
-        return {'val_loss': self.my_loss(y_hat, y)}
+        return {'val_loss': F.cross_entropy(y_hat, y)}

    def validation_end(self, outputs):
+        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}

    def configure_optimizers(self):
+        # REQUIRED
        return [torch.optim.Adam(self.parameters(), lr=0.02)]

    @pl.data_loader
@@ -74,11 +73,14 @@ class CoolModel(pl.LightningModule):

    @pl.data_loader
    def val_dataloader(self):
+        # OPTIONAL
+        # can also return a list of val dataloaders
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

    @pl.data_loader
    def test_dataloader(self):
-        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
+        # OPTIONAL
+        return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
 ---   
 ### How do these methods fit into the broader training?     
@@ -90,7 +92,7 @@ The LightningModule interface is on the right. Each method corresponds to a part
  </a>
 </p>   

---
+## Required Methods    

 ### training_step

@@ -134,18 +136,106 @@ def training_step(self, data_batch, batch_nb):
    
    # return a dict
    return output
+```    
+
+If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.    
+``` {.python}
+# Multiple optimizers (ie: GANs)     
+def training_step(self, data_batch, batch_nb, optimizer_idx):
+    if optimizer_idx == 0:
+        # do training_step with encoder
+    if optimizer_idx == 1:
+        # do training_step with decoder    
+```    
+
+--- 
+### tng_dataloader 
+
+``` {.python}
+@pl.data_loader
+def tng_dataloader(self)
+```
+Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
+
+##### Return
+PyTorch DataLoader
+
+**Example**
+
+``` {.python}
+@pl.data_loader
+def tng_dataloader(self):
+    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
+    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
+    loader = torch.utils.data.DataLoader(
+        dataset=dataset,
+        batch_size=self.hparams.batch_size,
+        shuffle=True
+    )
+    return loader
 ```

---
+--- 
+### configure_optimizers 
+
+``` {.python}
+def configure_optimizers(self)
+```
+
+Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
+Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
+
+**Note:** If you use multiple optimizers, training_step will have an additional ```optimizer_idx``` parameter.    
+
+
+
+##### Return    
+Return any of these 3 options:   
+Single optimizer   
+List or Tuple - List of optimizers    
+Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers
+
+**Example**
+
+``` {.python}
+# most cases
+def configure_optimizers(self):
+    opt = Adam(self.parameters(), lr=0.01)
+    return opt
+
+# multiple optimizer case (eg: GAN)
+def configure_optimizers(self):
+    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
+    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
+    return generator_opt, disriminator_opt
+    
+# example with learning_rate schedulers  
+def configure_optimizers(self):
+    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
+    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
+    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
+    return [generator_opt, disriminator_opt], [discriminator_sched]
+```  
+
+If you need to control how often those optimizers step or override the default .step() schedule, override 
+the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step) hook.   
+
+## Optional Methods    

 ### validation_step

 ``` {.python}
-def validation_step(self, data_batch, batch_nb)
-```
+def validation_step(self, data_batch, batch_nb)   

-In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
-This is most likely the same as your training_step. But unlike training step, the outputs from here will go to validation_end for collation.
+# if have multiple val dataloaders:  
+def validation_step(self, data_batch, batch_nb, dataloader_idx)
+```
+**OPTIONAL**    
+If you don't need to validate you don't need to implement this method.    
+
+In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
+
+The dict you return here will be available in the validation_end method. 

 **Params**   

@@ -153,16 +243,18 @@ This is most likely the same as your training_step. But unlike training step, th
 |---|---|
 |  data_batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
+|  dataloader_i | Integer displaying which dataloader this is (only if multiple val datasets used)  |

 **Return**   

 | Return  | description  | optional |
 |---|---|---|   
-|  dict | Dict of OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
+|  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |

 **Example**

 ``` {.python}
+# CASE 1: A single validation dataset
 def validation_step(self, data_batch, batch_nb):
    x, y, z = data_batch
    
@@ -170,6 +262,12 @@ def validation_step(self, data_batch, batch_nb):
    out = self.forward(x)
    loss = self.loss(out, x)
    
+    # log 6 example images
+    # or generated text... or whatever
+    sample_imgs = x[:6]
+    grid = torchvision.utils.make_grid(sample_imgs)
+    self.experiment.add_image('example_images', grid, 0) 
+    
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
@@ -183,16 +281,29 @@ def validation_step(self, data_batch, batch_nb):
    
    # return an optional dict
    return output
-```
+```   
+
+If you pass in multiple validation datasets, validation_step will have an additional argument.
+
+```python
+# CASE 2: multiple validation datasets
+def validation_step(self, data_batch, batch_nb, dataset_idx):
+    # dataset_idx tells you which dataset this is.   
+```   
+
+The ```dataset_idx``` corresponds to the order of datasets returned in ```val_dataloader```.    

 --- 
 ### validation_end

 ``` {.python}
 def validation_end(self, outputs)
-```
+```   
+If you didn't define a validation_step, this won't be called.       

-Called at the end of the validation loop with the output of each validation_step.
+Called at the end of the validation loop with the output of each validation_step.  Called once per validation dataset.   
+
+The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    

 **Params**    

@@ -227,36 +338,6 @@ def validation_end(self, outputs):
    return tqdm_dic
 ```

--- 
-### configure_optimizers 
-
-``` {.python}
-def configure_optimizers(self)
-```
-
-Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
-Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
-
-
-##### Return
-List or Tuple - List of optimizers with an optional second list of learning-rate schedulers
-
-**Example**
-
-``` {.python}
-# most cases
-def configure_optimizers(self):
-    opt = Adam(self.parameters(), lr=0.01)
-    return [opt]
-    
-# gan example, with scheduler for discriminator
-def configure_optimizers(self):
-    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
-    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
-    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
-    return [generator_opt, disriminator_opt], [discriminator_sched]
-```
-
 --- 
 ### on_save_checkpoint 

@@ -299,44 +380,20 @@ def on_load_checkpoint(self, checkpoint):
    self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']
 ```

--- 
-### tng_dataloader 
-
-``` {.python}
-@pl.data_loader
-def tng_dataloader(self)
-```
-Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
-
-##### Return
-PyTorch DataLoader
-
-**Example**
-
-``` {.python}
-@pl.data_loader
-def tng_dataloader(self):
-    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
-    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
-    loader = torch.utils.data.DataLoader(
-        dataset=dataset,
-        batch_size=self.hparams.batch_size,
-        shuffle=True
-    )
-    return loader
-```
-
 --- 
 ### val_dataloader 

 ``` {.python}
@pl.data_loader
-def tng_dataloader(self)
+def val_dataloader(self)
 ```
-Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
+**OPTIONAL**    
+If you don't need a validation dataset and a validation_step, you don't need to implement this method.    
+
+Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.   

 ##### Return
-PyTorch DataLoader
+PyTorch DataLoader or list of PyTorch Dataloaders.    

 **Example**

@@ -352,8 +409,16 @@ def val_dataloader(self):
    )
    
    return loader
+
+# can also return multiple dataloaders   
+@pl.data_loader
+def val_dataloader(self):
+    return [loader_a, loader_b, ..., loader_n]   
 ```

+In the case where you return multiple val_dataloaders, the validation_step will have an arguement ```dataset_idx```
+which matches the order here.    
+
 --- 
 ### test_dataloader 

@@ -361,6 +426,9 @@ def val_dataloader(self):
@pl.data_loader
 def test_dataloader(self)
 ```
+**OPTIONAL**    
+If you don't need a test dataset and a test_step, you don't need to implement this method.    
+
 Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.

 ##### Return
@@ -389,7 +457,7 @@ def test_dataloader(self):
 def update_tng_log_metrics(self, logs)
 ```
 Called by lightning right before it logs metrics for this batch.
-This is a chance to ammend or add to the metrics about to be logged.
+This is a chance to amend or add to the metrics about to be logged.

 ##### Return
 Dict 
@@ -10,7 +10,7 @@ Current dtype

 ---
 #### experiment    
-An instance of test-tube Experiment which you can use to log anything for tensorboarX.   
+An instance of test-tube Experiment which you can use to log anything for tensorboard (subclass of [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html)).   
 ```{.python}
 self.experiment.add_embedding(...)   
 self.experiment.log({'val_loss': 0.9})   
@@ -5,7 +5,7 @@ Lightning can automate saving and loading checkpoints.
 To enable checkpointing, define the checkpoint callback and give it to the trainer.

 ``` {.python}
-from pytorch_lightning.utils.pt_callbacks import ModelCheckpoint
+from pytorch_lightning.callbacks import ModelCheckpoint

 checkpoint_callback = ModelCheckpoint(
    filepath='/path/to/store/weights.ckpt',
@@ -38,6 +38,7 @@ trainer.fit(model)
 ```

 The trainer restores:  
+
 - global_step    
 - current_epoch    
 - All optimizers    
@@ -65,4 +66,4 @@ for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):

 # uses the model you passed into trainer        
 model.load_state_dict(checkpoint['state_dict'])
-```    
+```    
@@ -5,7 +5,7 @@ Lighting offers a few options for logging information about model, gpu usage, et
 #### Display metrics in progress bar 
 ``` {.python}
 # DEFAULT
-trainer = Trainer(progress_bar=True)
+trainer = Trainer(show_progress_bar=True)
 ```

 ---
@@ -28,15 +28,18 @@ trainer = Trainer(enable_early_stop=True)
 ```

 ---
-#### Gradient Clipping 
-Use this to turn off early stopping and run training to the [max_epoch](#force-training-for-min-or-max-epochs)
+#### Gradient Clipping
+Gradient clipping may be enabled to avoid exploding gradients.
+Specifically, this will [clip the gradient norm computed over all model parameters *together*](https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_).
+
 ``` {.python}
 # DEFAULT (ie: don't clip)
 trainer = Trainer(gradient_clip=0)
+
+# clip gradients with norm above 0.5
+trainer = Trainer(gradient_clip=0.5)
 ```

-
-
 ---
 #### Inspect gradient norms
 Looking at grad norms can help you figure out where training might be going wrong.
@@ -51,7 +54,10 @@ trainer = Trainer(track_grad_norm=2)

 ---
 #### Set how much of the training set to check
-If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag
+If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
+
+train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
 ``` {.python}
 # DEFAULT
 trainer = Trainer(train_percent_check=1.0)
@@ -18,6 +18,9 @@ trainer = Trainer(check_val_every_n_epoch=1)
 ---
 #### Set how much of the validation set to check 
 If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
+
+val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_percent_check=1.0)
@@ -29,6 +32,9 @@ trainer = Trainer(val_percent_check=0.1)
 ---
 #### Set how much of the test set to check 
 If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
+
+test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
 ``` {.python}
 # DEFAULT
 trainer = Trainer(test_percent_check=1.0)
@@ -23,6 +23,9 @@ trainer = Trainer(track_grad_norm=2)
 ---
 #### Make model overfit on subset of data
 A useful debugging trick is to make your model overfit a tiny fraction of the data.
+
+setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
+
 ``` {.python}
 # DEFAULT don't overfit (ie: normal training)
 trainer = Trainer(overfit_pct=0.0)
@@ -67,6 +67,53 @@ def on_tng_metrics(self, metrics):
    # do something before validation end
 ```

+---   
+#### optimizer_step 
+Calls .step() and .zero_grad for each optimizer.  
+You can override this method to adjust how you do the optimizer step for each optimizer
+
+Called once per optimizer
+```python
+# DEFAULT
+def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
+    optimizer.step()   
+    optimizer.zero_grad()   
+    
+# Alternating schedule for optimizer steps (ie: GANs)    
+def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
+    # update generator opt every 2 steps
+    if optimizer_i == 0:
+        if batch_nb % 2 == 0 :
+            optimizer.step()
+            optimizer.zero_grad()
+   
+    # update discriminator opt every 4 steps
+    if optimizer_i == 1:
+        if batch_nb % 4 == 0 :
+            optimizer.step()
+            optimizer.zero_grad()    
+    
+    # ...
+    # add as many optimizers as you want 
+```
+
+This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:   
+
+```python
+# learning rate warm-up
+def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
+    # warm up lr
+    if self.trainer.global_step < 500:
+        lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
+        for pg in optimizer.param_groups:
+            pg['lr'] = lr_scale * self.hparams.learning_rate
+    
+    # update params
+    optimizer.step()
+    optimizer.zero_grad() 
+```
+
+
 ---
 #### on_before_zero_grad
 Called in the training loop after taking an optimizer step and before zeroing grads.
@@ -68,6 +68,7 @@ But of course the fun is in all the advanced things it can do:
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
+- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)

 **Validation loop**    

@@ -3,7 +3,7 @@ In 99% of cases you want to just copy [this template](https://github.com/william

 ```bash
 # get a copy of the module template
-wget https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/lightning_module_template.py
+wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/examples/new_project_templates/lightning_module_template.py
 ```

 ---    
@@ -1,12 +1,63 @@
 ###### New project Quick Start    
-To start a new project you define two files, a LightningModule and a Trainer file.    
+To start a new project define two files, a LightningModule and a Trainer file.    
+To illustrate Lightning power and simplicity, here's an example of a typical research flow.    

-A separate trainer file allows to run many LightningModules. Each LightningModule has the core
-logic to a particular research project.    
+###### Case 1: BERT    
+Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
+You would define a single LightningModule and use flags to switch between your different ideas.   
+```python
+class BERT(pl.LightningModule):
+    def __init__(self, model_name, task):
+        self.task = task
+    
+        if model_name == 'transformer':
+            self.net = Transformer()
+        elif model_name == 'my_cool_version':
+            self.net = MyCoolVersion()
+            
+    def training_step(self, batch, batch_nb):
+        if self.task == 'standard_bert':
+            # do standard bert training with self.net...
+            # return loss
+            
+        if self.task == 'my_cool_task':
+            # do my own version with self.net
+            # return loss
+```   

-For example, one lightningModule could be an image classifier, the other
-one could be a seq-2-seq model, both (optionally) ran by the same trainer file.
+###### Case 2: COOLER NOT BERT    
+But if you wanted to try something **completely** different, you'd define a new module for that.    
+```python

+class CoolerNotBERT(pl.LightningModule):
+    def __init__(self):
+        self.net = ...
+        
+    def training_step(self, batch, batch_nb):
+        # do some other cool task
+        # return loss   
+```   
+
+###### Rapid research flow    
+Then you could do rapid research by switching between these two and using the same trainer.   
+```python
+
+if use_bert:
+    model = BERT()
+else:
+    model = CoolerNotBERT()
+    
+trainer = Trainer(gpus=[0, 1, 2, 3], use_amp=True)
+trainer.fit(model)
+```
+
+Notice a few things about this flow:   
+1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
+2. You get free GPU and 16-bit support without writing any of that code in your model.   
+3. You also get all of the capabilities below (without coding or testing yourself).     
+
+---    
+###### Templates 
 1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
 2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_cpu_template.py) 
@@ -75,6 +126,7 @@ one could be a seq-2-seq model, both (optionally) ran by the same trainer file.
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
+- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)

 ###### Validation loop    

@@ -161,9 +161,9 @@ class LightningTemplateModel(LightningModule):
            # reduce manually when using dp
            val_acc = output['val_acc']
            if self.trainer.use_dp:
-                val_acc_mean = torch.mean(val_acc)
+                val_acc = torch.mean(val_acc)

-            val_acc_mean += val_acc_mean
+            val_acc_mean += val_acc

        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
@@ -189,16 +189,13 @@ class LightningTemplateModel(LightningModule):
        dataset = MNIST(root=self.hparams.data_root, train=train,
                        transform=transform, download=True)

-        # when using multi-node we need to add the datasampler
+        # when using multi-node (ddp) we need to add the datasampler
        train_sampler = None
        batch_size = self.hparams.batch_size

-        try:
-            if self.on_gpu:
-                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
-                batch_size = batch_size // self.trainer.world_size  # scale batch size
-        except Exception:
-            pass
+        if self.use_ddp:
+            train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
+            batch_size = batch_size // self.trainer.world_size  # scale batch size

        should_shuffle = train_sampler is None
        loader = DataLoader(
@@ -110,5 +110,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -110,5 +110,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -109,5 +109,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -0,0 +1,178 @@
+"""
+To run this template just do:  
+python gan.py   
+
+After a few epochs, launch tensorboard to see the images being generated at every batch.   
+
+tensorboard --logdir default
+"""
+from argparse import ArgumentParser
+import os
+import numpy as np
+
+import torchvision
+import torchvision.transforms as transforms
+from torchvision.datasets import MNIST
+
+from torch.utils.data import DataLoader
+
+import torch.nn as nn
+import torch.nn.functional as F
+import torch
+
+import pytorch_lightning as pl
+from test_tube import Experiment
+
+
+class Generator(nn.Module):
+    def __init__(self, latent_dim, img_shape):
+        super(Generator, self).__init__()
+        self.img_shape = img_shape
+
+        def block(in_feat, out_feat, normalize=True):
+            layers = [nn.Linear(in_feat, out_feat)]
+            if normalize:
+                layers.append(nn.BatchNorm1d(out_feat, 0.8))
+            layers.append(nn.LeakyReLU(0.2, inplace=True))
+            return layers
+
+        self.model = nn.Sequential(
+            *block(latent_dim, 128, normalize=False),
+            *block(128, 256),
+            *block(256, 512),
+            *block(512, 1024),
+            nn.Linear(1024, int(np.prod(img_shape))),
+            nn.Tanh()
+        )
+
+    def forward(self, z):
+        img = self.model(z)
+        img = img.view(img.size(0), *self.img_shape)
+        return img
+
+
+class Discriminator(nn.Module):
+    def __init__(self, img_shape):
+        super(Discriminator, self).__init__()
+
+        self.model = nn.Sequential(
+            nn.Linear(int(np.prod(img_shape)), 512),
+            nn.LeakyReLU(0.2, inplace=True),
+            nn.Linear(512, 256),
+            nn.LeakyReLU(0.2, inplace=True),
+            nn.Linear(256, 1),
+            nn.Sigmoid(),
+        )
+
+    def forward(self, img):
+        img_flat = img.view(img.size(0), -1)
+        validity = self.model(img_flat)
+
+        return validity
+
+
+class GAN(pl.LightningModule):
+
+    def __init__(self, hparams):
+        super(GAN, self).__init__()
+        self.hparams = hparams
+
+        # networks
+        mnist_shape = (1, 28, 28)
+        self.generator = Generator(latent_dim=hparams.latent_dim, img_shape=mnist_shape)
+        self.discriminator = Discriminator(img_shape=mnist_shape)
+
+        # cache for generated images
+        self.generated_imgs = None
+
+    def forward(self, z):
+        return self.generator(z)
+
+    def adversarial_loss(self, y_hat, y):
+        return F.binary_cross_entropy(y_hat, y)
+
+    def training_step(self, batch, batch_nb, optimizer_i):
+        imgs, _ = batch
+
+        # train generator
+        if optimizer_i == 0:
+            # sample noise
+            z = torch.randn(imgs.shape[0], self.hparams.latent_dim)
+
+            # match gpu device (or keep as cpu)
+            if self.on_gpu:
+                z = z.cuda(imgs.device.index)
+
+            # generate images
+            self.generated_imgs = self.forward(z)
+
+            # log sampled images
+            sample_imgs = self.generated_imgs[:6]
+            grid = torchvision.utils.make_grid(sample_imgs)
+            self.experiment.add_image('generated_images', grid, 0)
+
+            # ground truth result (ie: all fake)
+            valid = torch.ones(imgs.size(0), 1)
+
+            # adversarial loss is binary cross-entropy
+            g_loss = self.adversarial_loss(self.discriminator(self.generated_imgs), valid)
+
+            return g_loss
+
+        # train discriminator
+        if optimizer_i == 1:
+            # Measure discriminator's ability to classify real from generated samples
+
+            # how well can it label as real?
+            valid = torch.ones(imgs.size(0), 1)
+            real_loss = self.adversarial_loss(self.discriminator(imgs), valid)
+
+            # how well can it label as fake?
+            fake = torch.zeros(imgs.size(0), 1)
+            fake_loss = self.adversarial_loss(self.discriminator(self.generated_imgs.detach()), fake)
+
+            # discriminator loss is the average of these
+            d_loss = (real_loss + fake_loss) / 2
+
+            return d_loss
+
+    def configure_optimizers(self):
+        lr = self.hparams.lr
+        b1 = self.hparams.b1
+        b2 = self.hparams.b2
+
+        opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
+        opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))
+        return [opt_g, opt_d], []
+
+    @pl.data_loader
+    def tng_dataloader(self):
+        transform = transforms.Compose([transforms.ToTensor(),
+                                        transforms.Normalize([0.5], [0.5])])
+        dataset = MNIST(os.getcwd(), train=True, download=True, transform=transform)
+        return DataLoader(dataset, batch_size=self.hparams.batch_size)
+
+
+def main(hparams):
+    # save tensorboard logs
+    exp = Experiment(save_dir=os.getcwd())
+
+    # init model
+    model = GAN(hparams)
+
+    # fit trainer on CPU
+    trainer = pl.Trainer(experiment=exp, max_nb_epochs=200)
+    trainer.fit(model)
+
+
+if __name__ == '__main__':
+    parser = ArgumentParser()
+    parser.add_argument("--batch_size", type=int, default=64, help="size of the batches")
+    parser.add_argument("--lr", type=float, default=0.0002, help="adam: learning rate")
+    parser.add_argument("--b1", type=float, default=0.5, help="adam: decay of first order momentum of gradient")
+    parser.add_argument("--b2", type=float, default=0.999, help="adam: decay of first order momentum of gradient")
+    parser.add_argument("--latent_dim", type=int, default=100, help="dimensionality of the latent space")
+
+    hparams = parser.parse_args()
+
+    main(hparams)
@@ -8,3 +8,8 @@ site_description: 'Documentation for PyTorch LightningModule, the researcher ver

 dev_addr: '0.0.0.0:8000'
 #google_analytics: ['UA-aasd', 'sitename']
+
+markdown_extensions:
+  - codehilite:
+      guess_lang: false
+      linenums: true
@@ -12,6 +12,7 @@ import torch
 from torch.utils.data.distributed import DistributedSampler
 import torch.multiprocessing as mp
 import torch.distributed as dist
+from torch.optim.optimizer import Optimizer

 from pytorch_lightning.root_module.memory import get_gpu_memory_map
 from pytorch_lightning.root_module.model_saving import TrainerIO
@@ -59,7 +60,7 @@ class Trainer(TrainerIO):
                 current_gpu_name=0,
                 nb_gpu_nodes=1,
                 gpus=None,
-                 progress_bar=True,
+                 show_progress_bar=True,
                 overfit_pct=0.0,
                 track_grad_norm=-1,
                 check_val_every_n_epoch=1,
@@ -70,7 +71,7 @@ class Trainer(TrainerIO):
                 train_percent_check=1.0,
                 val_percent_check=1.0,
                 test_percent_check=1.0,
-                 val_check_interval=0.95,
+                 val_check_interval=1.0,
                 log_save_interval=100,
                 add_log_row_interval=10,
                 distributed_backend='dp',
@@ -90,7 +91,7 @@ class Trainer(TrainerIO):
        :param current_gpu_name:
        :param nb_gpu_nodes:
        :param gpus:
-        :param progress_bar:
+        :param show_progress_bar:
        :param overfit_pct:
        :param track_grad_norm:
        :param check_val_every_n_epoch:
@@ -120,7 +121,6 @@ class Trainer(TrainerIO):
        self.track_grad_norm = track_grad_norm
        self.fast_dev_run = fast_dev_run
        self.on_gpu = gpus is not None and torch.cuda.is_available()
-        self.progress_bar = progress_bar
        self.experiment = experiment
        self.exp_save_path = None
        if self.experiment is not None:
@@ -157,9 +157,9 @@ class Trainer(TrainerIO):
        self.avg_loss = 0
        self.batch_nb = 0
        self.tqdm_metrics = {}
-        self.nb_val_batches = None
-        self.nb_tng_batches = None
-        self.nb_test_batches = None
+        self.nb_val_batches = 0
+        self.nb_tng_batches = 0
+        self.nb_test_batches = 0

        # gpus come in as a string.
        # if gpus = -1 then use all available devices
@@ -221,11 +221,14 @@ class Trainer(TrainerIO):

        # training state
        self.optimizers = None
-        self.prog_bar = None
        self.global_step = 0
        self.current_epoch = 0
        self.total_batches = 0

+        # can't init progress bar here because starting a new process
+        # means the prog_bar won't survive pickling
+        self.show_progress_bar = show_progress_bar
+
        # logging
        self.log_save_interval = log_save_interval
        self.val_check_interval = val_check_interval
@@ -312,10 +315,18 @@ class Trainer(TrainerIO):
        f_op = getattr(model, f_name, None)
        return callable(f_op)

+    def __is_overriden(self, f_name):
+        model = self.__get_model()
+        super_object = super(model.__class__, model)
+
+        # when code pointers are different, it was overriden
+        is_overriden = getattr(model, f_name).__code__ is not getattr(super_object, f_name).__code__
+        return is_overriden
+
    @property
    def __tng_tqdm_dic(self):
        tqdm_dic = {
-            'tng_loss': '{0:.3f}'.format(self.avg_loss),
+            'loss': '{0:.3f}'.format(self.avg_loss),
            'epoch': '{}'.format(self.current_epoch),
            'batch_nb': '{}'.format(self.batch_nb),
        }
@@ -345,17 +356,19 @@ class Trainer(TrainerIO):
        self.nb_tng_batches = int(self.nb_tng_batches * self.train_percent_check)

        # determine number of validation batches
-        self.nb_val_batches = len(self.val_dataloader)
-        self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
-        self.nb_val_batches = max(1, self.nb_val_batches)
-        self.nb_val_batches = self.nb_val_batches
+        # val datasets could be none, 1 or 2+
+        if self.val_dataloader is not None:
+            self.nb_val_batches = sum(len(dataloader) for dataloader in self.val_dataloader)
+            self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
+            self.nb_val_batches = max(1, self.nb_val_batches)

        # determine number of test batches
-        self.nb_test_batches = len(self.test_dataloader)
+        self.nb_test_batches = len(self.test_dataloader) if self.test_dataloader is not None else 0
        self.nb_test_batches = int(self.nb_test_batches * self.test_percent_check)

        # determine when to check validation
        self.val_check_batch = int(self.nb_tng_batches * self.val_check_interval)
+        self.val_check_batch = max(1, self.val_check_batch)

    def __add_tqdm_metrics(self, metrics):
        for k, v in metrics.items():
@@ -364,7 +377,32 @@ class Trainer(TrainerIO):

            self.tqdm_metrics[k] = v

-    def validate(self, model, dataloader, max_batches):
+    def __validation_forward(self, model, data_batch, batch_i, dataloader_i):
+        # make dataloader_i arg in validation_step optional
+        args = [data_batch, batch_i]
+        if len(self.val_dataloader) > 1:
+            args.append(dataloader_i)
+
+        if self.use_ddp:
+            output = model(*args)
+        elif self.use_dp:
+            output = model(*args)
+        elif self.single_gpu:
+            # put inputs on gpu manually
+            gpu_id = self.data_parallel_device_ids[0]
+            data_batch = self.transfer_batch_to_gpu(data_batch, gpu_id)
+            args[0] = data_batch
+
+            # do non dp, ddp step
+            output = model.validation_step(*args)
+
+        else:
+            # CPU
+            output = model.validation_step(*args)
+
+        return output
+
+    def validate(self, model, dataloader, max_batches, dataloader_i):
        """
        Run validation code
        :param model: PT model
@@ -372,6 +410,7 @@ class Trainer(TrainerIO):
        :param max_batches: Scalar
        :return:
        """
+
        # enable eval mode
        model.zero_grad()
        model.eval()
@@ -388,41 +427,29 @@ class Trainer(TrainerIO):
            if data_batch is None:  # pragma: no cover
                continue

-            # stop short when on fast dev run
-            if max_batches is not None and batch_i >= max_batches:
+            # stop short when on fast_dev_run (sets max_batch=1)
+            if batch_i >= max_batches:
                break

            # -----------------
            # RUN VALIDATION STEP
            # -----------------
-            if self.use_ddp:
-                output = model(data_batch, batch_i)
-            elif self.use_dp:
-                output = model(data_batch, batch_i)
-            elif self.single_gpu:
-                # put inputs on gpu manually
-                gpu_id = self.data_parallel_device_ids[0]
-                for i, x in enumerate(data_batch):
-                    if isinstance(x, torch.Tensor):
-                        data_batch[i] = x.cuda(gpu_id)
-
-                # do non dp, ddp step
-                output = model.validation_step(data_batch, batch_i)
-
-            else:
-                output = model.validation_step(data_batch, batch_i)
+            output = self.__validation_forward(model, data_batch, batch_i, dataloader_i)

+            # track outputs for collation
            outputs.append(output)

            # batch done
-            if self.progress_bar and self.prog_bar is not None:
-                self.prog_bar.update(1)
+            if self.show_progress_bar:
+                self.progress_bar.update(1)

-        # give model a chance to do something with the outputs
-        if self.data_parallel:
-            val_results = model.module.validation_end(outputs)
-        else:
-            val_results = model.validation_end(outputs)
+        # give model a chance to do something with the outputs (and method defined)
+        val_results = {}
+        if self.__is_overriden('validation_end'):
+            if self.data_parallel:
+                val_results = model.module.validation_end(outputs)
+            else:
+                val_results = model.validation_end(outputs)

        # enable train mode again
        model.train()
@@ -438,25 +465,57 @@ class Trainer(TrainerIO):
        :param model:
        :return:
        """
+
        self.tng_dataloader = model.tng_dataloader
        self.test_dataloader = model.test_dataloader
        self.val_dataloader = model.val_dataloader

+        # handle returning an actual dataloader instead of a list of loaders
+        have_val_loaders = self.val_dataloader is not None
+        if have_val_loaders and not isinstance(self.val_dataloader, list):
+            self.val_dataloader = [self.val_dataloader]
+
        if self.use_ddp and not isinstance(self.tng_dataloader.sampler, DistributedSampler):
            msg = """
-when using multiple gpus and multiple nodes you must pass
- a DistributedSampler to DataLoader(sampler).
+            You're using multiple gpus and multiple nodes without using a DistributedSampler
+            to assign a subset of your data to each process. To silence this warning, pass a
+            DistributedSampler to your DataLoader.

-ie: this:
-dataset = myDataset()
-dataloader = Dataloader(dataset)
+            ie: this:
+            dataset = myDataset()
+            dataloader = Dataloader(dataset)

-becomes:
-dataset = myDataset()
-dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-dataloader = Dataloader(dataset, sampler=dist_sampler)
-"""
-            raise MisconfigurationException(msg)
+            becomes:
+            dataset = myDataset()
+            dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+            dataloader = Dataloader(dataset, sampler=dist_sampler)
+
+            If you want each process to load the full dataset, ignore this warning.
+            """
+            warnings.warn(msg)
+
+        if self.use_ddp and self.val_dataloader is not None:
+            for dataloader in self.val_dataloader:
+                if not isinstance(dataloader, DistributedSampler):
+                    msg = """
+                    Your val_dataloader(s) are not all DistributedSamplers.
+                    You're using multiple gpus and multiple nodes without using a DistributedSampler
+                    to assign a subset of your data to each process. To silence this warning, pass a
+                    DistributedSampler to your DataLoader.
+
+                    ie: this:
+                    dataset = myDataset()
+                    dataloader = Dataloader(dataset)
+
+                    becomes:
+                    dataset = myDataset()
+                    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+                    dataloader = Dataloader(dataset, sampler=dist_sampler)
+
+                    If you want each process to load the full dataset, ignore this warning.
+                    """
+                    warnings.warn(msg)
+                    break

    # -----------------------------
    # MODEL TRAINING
@@ -474,12 +533,15 @@ dataloader = Dataloader(dataset, sampler=dist_sampler)
                task = int(os.environ['SLURM_LOCALID'])
                self.ddp_train(task, model)
            else:
-                msg = """
-You requested %(nb_gpus)s GPUs but launched %(nb_tasks)s slurm tasks.
-We will launch %(nb_gpus)s processes for you.
-We recommend you let slurm manage the processes by setting: --ntasks-per-node=%(nb_gpus)s
-If you're not using SLURM, ignore this message!
-""" % {'nb_gpus': self.nb_requested_gpus, 'nb_tasks': self.nb_slurm_tasks}
+                nb_gpus = self.nb_requested_gpus
+                nb_tasks = self.nb_slurm_tasks
+                msg = f"""
+                You requested {nb_gpus}s GPUs but launched {nb_tasks}s slurm tasks.
+                We will launch {nb_gpus}s processes for you.
+                We recommend you let slurm manage the processes by setting:
+                --ntasks-per-node={nb_gpus}s
+                If you're not using SLURM, ignore this message!
+                """
                warnings.warn(msg)
                mp.spawn(self.ddp_train, nprocs=len(self.data_parallel_device_ids), args=(model, ))

@@ -500,9 +562,7 @@ If you're not using SLURM, ignore this message!

            # CHOOSE OPTIMIZER
            # allow for lr schedulers as well
-            self.optimizers = model.configure_optimizers()
-            if len(self.optimizers) == 2:
-                self.optimizers, self.lr_schedulers = self.optimizers
+            self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())

            self.__run_pretrain_routine(model)

@@ -510,12 +570,25 @@ If you're not using SLURM, ignore this message!
        # used for testing or when we need to know that training succeeded
        return 1

+    def init_optimizers(self, optimizers):
+
+        # single optimizer
+        if isinstance(optimizers, Optimizer):
+            return [optimizers], []
+
+        # two lists
+        elif len(optimizers) == 2 and isinstance(optimizers[0], list):
+            optimizers, lr_schedulers = optimizers
+            return optimizers, lr_schedulers
+
+        # single list or tuple
+        elif isinstance(optimizers, list) or isinstance(optimizers, tuple):
+            return optimizers, []
+
    def __single_gpu_train(self, model):
        # CHOOSE OPTIMIZER
        # allow for lr schedulers as well
-        self.optimizers = model.configure_optimizers()
-        if len(self.optimizers) == 2:
-            self.optimizers, self.lr_schedulers = self.optimizers
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())

        model.cuda(self.data_parallel_device_ids[0])

@@ -532,20 +605,18 @@ If you're not using SLURM, ignore this message!

        # CHOOSE OPTIMIZER
        # allow for lr schedulers as well
-        self.optimizers = model.configure_optimizers()
-        if len(self.optimizers) == 2:
-            self.optimizers, self.lr_schedulers = self.optimizers
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())

        model.cuda(self.data_parallel_device_ids[0])

        # check for this bug (amp + dp + !01 doesn't work)
        # https://github.com/NVIDIA/apex/issues/227
        if self.use_dp and self.use_amp:
-            m = """
-Amp level %r with DataParallel is not supported.
-See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227.
-We recommend you switch to ddp if you want to use amp
-""" % self.amp_level
+            m = f"""
+            Amp level {self.amp_level} with DataParallel is not supported.
+            See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227.
+            We recommend you switch to ddp if you want to use amp
+            """
            raise MisconfigurationException(m)

        model = LightningDataParallel(model, device_ids=self.data_parallel_device_ids)
@@ -575,7 +646,7 @@ We recommend you switch to ddp if you want to use amp
            self.experiment = self.experiment.get_non_ddp_exp()

        # show progbar only on prog_rank 0
-        self.prog_bar = self.prog_bar and self.node_rank == 0 and gpu_nb == 0
+        self.show_progress_bar = self.show_progress_bar and self.node_rank == 0 and gpu_nb == 0

        # determine which process we are and world size
        self.proc_rank = self.node_rank * len(self.data_parallel_device_ids) + gpu_nb
@@ -592,9 +663,7 @@ We recommend you switch to ddp if you want to use amp

        # CHOOSE OPTIMIZER
        # allow for lr schedulers as well
-        self.optimizers = model.configure_optimizers()
-        if len(self.optimizers) == 2:
-            self.optimizers, self.lr_schedulers = self.optimizers
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())

        # MODEL
        # copy model to each gpu
@@ -668,6 +737,9 @@ We recommend you switch to ddp if you want to use amp

        # set local properties on the model
        ref_model.on_gpu = self.on_gpu
+        ref_model.use_dp = self.use_dp
+        ref_model.use_ddp = self.use_ddp
+        ref_model.use_amp = self.use_amp

        # transfer data loaders from model
        self.get_dataloaders(ref_model)
@@ -702,127 +774,139 @@ We recommend you switch to ddp if you want to use amp
        if self.cluster is not None:  # pragma: no cover
            self.enable_auto_hpc_walltime_manager()

-        # run tiny validation to make sure program won't crash during val
+        # progress bar init
+        if self.show_progress_bar:
+            self.progress_bar = tqdm.tqdm(0, position=self.process_position)
+
+        # run tiny validation (if validation defined) to make sure program won't crash during val
        ref_model.on_sanity_check_start()
-        _ = self.validate(model, self.val_dataloader, max_batches=self.nb_sanity_val_steps)
+        if self.val_dataloader is not None:
+            for ds_i, dataloader in enumerate(self.val_dataloader):
+
+                # reset progress_bar limit for sanity check
+                if self.show_progress_bar:
+                    self.progress_bar.reset(self.nb_sanity_val_steps)
+
+                self.validate(model, dataloader, self.nb_sanity_val_steps, ds_i)

        # ---------------------------
        # CORE TRAINING LOOP
        # ---------------------------
-
        self.__train()

    def __train(self):
        # run all epochs
        for epoch_nb in range(self.current_epoch, self.max_nb_epochs):
-            # update the lr scheduler
-            if self.lr_schedulers is not None:
-                for lr_scheduler in self.lr_schedulers:
-                    lr_scheduler.step()
-
+            # get model
            model = self.__get_model()
+
+            # update training progress in trainer and model
            model.current_epoch = epoch_nb
-
-            # hook
-            if self.__is_function_implemented('on_epoch_start'):
-                model = self.__get_model()
-                model.on_epoch_start()
-
            self.current_epoch = epoch_nb
            self.total_batches = self.nb_tng_batches + self.nb_val_batches
            self.batch_loss_value = 0  # accumulated grads

-            # init progbar when requested
-            if self.progress_bar:
-                self.prog_bar = tqdm.tqdm(range(self.total_batches),
-                                          position=self.process_position)
+            # init progress_bar when requested
+            if self.show_progress_bar:
+                self.progress_bar.reset(self.total_batches)

-            for batch_nb, data_batch in enumerate(self.tng_dataloader):
-                self.batch_nb = batch_nb
-                self.global_step += 1
+            # -----------------
+            # RUN TNG EPOCH
+            # -----------------
+            self.run_tng_epoch()

-                model = self.__get_model()
-                model.global_step = self.global_step
-
-                # stop when the flag is changed or we've gone past the amount
-                #  requested in the batches
-                self.total_batch_nb += 1
-                met_batch_limit = batch_nb > self.nb_tng_batches
-                if met_batch_limit:
-                    break
-
-                # ---------------
-                # RUN TRAIN STEP
-                # ---------------
-                batch_result = self.__run_tng_batch(data_batch, batch_nb)
-                early_stop_epoch = batch_result == -1
-
-                # ---------------
-                # RUN VAL STEP
-                # ---------------
-                is_val_check_batch = (batch_nb + 1) % self.val_check_batch == 0
-                if self.fast_dev_run or is_val_check_batch or early_stop_epoch:
-                    self.__run_validation()
-
-                # when batch should be saved
-                if (batch_nb + 1) % self.log_save_interval == 0 or early_stop_epoch:
-                    if self.proc_rank == 0 and self.experiment is not None:
-                        self.experiment.save()
-
-                # when metrics should be logged
-                if batch_nb % self.add_log_row_interval == 0 or early_stop_epoch:
-                    # count items in memory
-                    # nb_params, nb_tensors = count_mem_items()
-
-                    model = self.__get_model()
-                    metrics = self.__tng_tqdm_dic
-
-                    # add gpu memory
-                    if self.on_gpu:
-                        mem_map = get_gpu_memory_map()
-                        metrics.update(mem_map)
-
-                    # add norms
-                    if self.track_grad_norm > 0:
-                        model = self.__get_model()
-                        grad_norm_dic = model.grad_norm(self.track_grad_norm)
-                        metrics.update(grad_norm_dic)
-
-                    if self.__is_function_implemented('on_tng_metrics'):
-                        model.on_tng_metrics(metrics)
-
-                    # log metrics
-                    scalar_metrics = self.__metrics_to_scalars(
-                        metrics, blacklist=self.__log_vals_blacklist())
-                    if self.proc_rank == 0 and self.experiment is not None:
-                        self.experiment.log(scalar_metrics, global_step=self.global_step)
-                        self.experiment.save()
-
-                # hook
-                if self.__is_function_implemented('on_batch_end'):
-                    model = self.__get_model()
-                    model.on_batch_end()
-
-                # end epoch early
-                if early_stop_epoch:
-                    break
-
-            # hook
-            if self.__is_function_implemented('on_epoch_end'):
-                model = self.__get_model()
-                model.on_epoch_end()
+            # update LR schedulers
+            if self.lr_schedulers is not None:
+                for lr_scheduler in self.lr_schedulers:
+                    lr_scheduler.step()

            # early stopping
            met_min_epochs = epoch_nb > self.min_nb_epochs
            if self.enable_early_stop and met_min_epochs:
                should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch_nb,
                                                                    logs=self.__tng_tqdm_dic)
-
                # stop training
                stop = should_stop and met_min_epochs
                if stop:
                    return

+    def run_tng_epoch(self):
+        # before epoch hook
+        if self.__is_function_implemented('on_epoch_start'):
+            model = self.__get_model()
+            model.on_epoch_start()
+
+        # run epoch
+        for batch_nb, data_batch in enumerate(self.tng_dataloader):
+            self.batch_nb = batch_nb
+            self.global_step += 1
+
+            model = self.__get_model()
+            model.global_step = self.global_step
+
+            # stop when the flag is changed or we've gone past the amount
+            #  requested in the batches
+            self.total_batch_nb += 1
+            met_batch_limit = batch_nb > self.nb_tng_batches
+            if met_batch_limit:
+                break
+
+            # ---------------
+            # RUN TRAIN STEP
+            # ---------------
+            batch_result = self.__run_tng_batch(data_batch, batch_nb)
+            early_stop_epoch = batch_result == -1
+
+            # ---------------
+            # RUN VAL STEP
+            # ---------------
+            is_val_check_batch = (batch_nb + 1) % self.val_check_batch == 0
+            if self.fast_dev_run or is_val_check_batch or early_stop_epoch:
+                self.__run_validation()
+
+            # when batch should be saved
+            if (batch_nb + 1) % self.log_save_interval == 0 or early_stop_epoch:
+                if self.proc_rank == 0 and self.experiment is not None:
+                    self.experiment.save()
+
+            # when metrics should be logged
+            if batch_nb % self.add_log_row_interval == 0 or early_stop_epoch:
+                # count items in memory
+                # nb_params, nb_tensors = count_mem_items()
+
+                model = self.__get_model()
+                metrics = self.__tng_tqdm_dic
+
+                # add gpu memory
+                if self.on_gpu:
+                    mem_map = get_gpu_memory_map()
+                    metrics.update(mem_map)
+
+                # add norms
+                if self.track_grad_norm > 0:
+                    model = self.__get_model()
+                    grad_norm_dic = model.grad_norm(self.track_grad_norm)
+                    metrics.update(grad_norm_dic)
+
+                if self.__is_function_implemented('on_tng_metrics'):
+                    model.on_tng_metrics(metrics)
+
+                # log metrics
+                scalar_metrics = self.__metrics_to_scalars(
+                    metrics, blacklist=self.__log_vals_blacklist())
+                if self.proc_rank == 0 and self.experiment is not None:
+                    self.experiment.log(scalar_metrics, global_step=self.global_step)
+                    self.experiment.save()
+
+            # end epoch early
+            if early_stop_epoch:
+                break
+
+        # epoch end hook
+        if self.__is_function_implemented('on_epoch_end'):
+            model = self.__get_model()
+            model.on_epoch_end()
+
    def __metrics_to_scalars(self, metrics, blacklist=set()):
        new_metrics = {}
        for k, v in metrics.items():
@@ -842,37 +926,65 @@ We recommend you switch to ddp if you want to use amp
        blacklist = {'batch_nb', 'v_nb', 'gpu'}
        return blacklist

-    def __run_tng_batch(self, data_batch, batch_nb):
-        if data_batch is None:
-            return 0
+    def transfer_batch_to_gpu(self, batch, gpu_id):
+        # base case
+        if isinstance(batch, torch.Tensor):
+            return batch.cuda(gpu_id)

-        # hook
-        if self.__is_function_implemented('on_batch_start'):
-            model_ref = self.__get_model()
-            response = model_ref.on_batch_start(data_batch)
+        # when list
+        elif isinstance(batch, list):
+            for i, x in enumerate(batch):
+                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
+            return batch

-            if response == -1:
-                return -1
+        # when tuple
+        elif isinstance(batch, tuple):
+            batch = list(batch)
+            for i, x in enumerate(batch):
+                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
+            return tuple(batch)

-        if self.progress_bar:
-            self.prog_bar.update(1)
+        # when dict
+        elif isinstance(batch, dict):
+            for k, v in batch.items():
+                batch[k] = self.transfer_batch_to_gpu(v, gpu_id)
+
+            return batch
+
+        # nothing matches, return the value as is without transform
+        return batch
+
+    def __tng_forward(self, data_batch, batch_nb, opt_idx):
+        """
+        Handle forward for each training case (distributed, single gpu, etc...)
+        :param data_batch:
+        :param batch_nb:
+        :return:
+        """
+        # ---------------
+        # FORWARD
+        # ---------------
+        # enable not needing to add opt_idx to training_step
+        args = [data_batch, batch_nb]
+        if len(self.optimizers) > 1:
+            args.append(opt_idx)

-        # forward pass
-        # return a scalar value and a dic with tqdm metrics
        if self.use_ddp:
-            output = self.model(data_batch, batch_nb)
+            output = self.model(*args)
        elif self.use_dp:
-            output = self.model(data_batch, batch_nb)
+            output = self.model(*args)
        elif self.single_gpu:
            gpu_id = self.data_parallel_device_ids[0]
-            for i, x in enumerate(data_batch):
-                if isinstance(x, torch.Tensor):
-                    data_batch[i] = x.cuda(gpu_id)
-            output = self.model.training_step(data_batch, batch_nb)
+            data_batch = self.transfer_batch_to_gpu(data_batch, gpu_id)
+            args[0] = data_batch
+            output = self.model.training_step(*args)

        else:
-            output = self.model.training_step(data_batch, batch_nb)
+            output = self.model.training_step(*args)

+        # ---------------
+        # TQDM metrics
+        # ---------------
        try:
            prog_output = output['prog']

@@ -885,6 +997,9 @@ We recommend you switch to ddp if you want to use amp
        except Exception:
            model_specific_tqdm_metrics_dic = {}

+        # ---------------
+        # EXTRACT LOSS
+        # ---------------
        # if output dict doesn't have the keyword loss
        # then assume the output=loss if scalar
        try:
@@ -897,64 +1012,85 @@ We recommend you switch to ddp if you want to use amp
        if self.use_dp:
            loss = reduce_distributed_output(loss, len(self.data_parallel_device_ids))

-        self.__add_tqdm_metrics(model_specific_tqdm_metrics_dic)
+        return loss, model_specific_tqdm_metrics_dic

-        # backward pass
-        if self.use_amp:
-            # scale loss when using amp
-            for optimizer in self.optimizers:
-                with amp.scale_loss(loss, optimizer) as scaled_loss:
-                    scaled_loss.backward()
-        else:
-            loss.backward()
-
-        # insert after step hook
-        if self.__is_function_implemented('on_after_backward'):
-            model_ref = self.__get_model()
-            response = model_ref.on_after_backward()
+    def __clip_gradients(self):
+        if self.gradient_clip > 0:
+            model = self.__get_model()
+            torch.nn.utils.clip_grad_norm_(model.parameters(), self.gradient_clip)

+    def __print_nan_grads(self):
        if self.print_nan_grads:
            model = self.__get_model()
            for param in model.parameters():
                print(param.grad.float().sum())

-        # avoid memory leaks
-        self.batch_loss_value += loss.item()
+    def __run_tng_batch(self, data_batch, batch_nb):
+        if data_batch is None:
+            return 0

-        # gradient update with accumulated gradients
-        if (self.batch_nb + 1) % self.accumulate_grad_batches == 0:
+        # hook
+        if self.__is_function_implemented('on_batch_start'):
+            model_ref = self.__get_model()
+            response = model_ref.on_batch_start(data_batch)

-            # clip gradients
-            if self.gradient_clip > 0:
+            if response == -1:
+                return -1
+
+        if self.show_progress_bar:
+            self.progress_bar.update(1)
+
+        # call training_step once per optimizer
+        for opt_idx, optimizer in enumerate(self.optimizers):
+
+            # forward pass
+            loss, model_specific_tqdm_metrics = self.__tng_forward(data_batch, batch_nb, opt_idx)
+
+            # track metrics
+            self.__add_tqdm_metrics(model_specific_tqdm_metrics)
+
+            # accumulate loss
+            # (if accumulate_grad_batches = 1 no effect)
+            loss = loss / self.accumulate_grad_batches
+
+            # backward pass
+            if self.use_amp:
+                with amp.scale_loss(loss, optimizer) as scaled_loss:
+                    scaled_loss.backward()
+            else:
+                loss.backward()
+
+            # insert after step hook
+            if self.__is_function_implemented('on_after_backward'):
+                model_ref = self.__get_model()
+                model_ref.on_after_backward()
+
+            # nan grads
+            self.__print_nan_grads()
+
+            # track total loss for logging (avoid mem leaks)
+            self.batch_loss_value += loss.item()
+
+            # gradient update with accumulated gradients
+            if (self.batch_nb + 1) % self.accumulate_grad_batches == 0:
+                # clip gradients
+                self.__clip_gradients()
+
+                # calls .step(), .zero_grad()
+                # override function to modify this behavior
                model = self.__get_model()
-                torch.nn.utils.clip_grad_norm_(model.parameters(), self.gradient_clip)
+                model.optimizer_step(self.current_epoch, batch_nb, optimizer, opt_idx)

-            # update gradients across all optimizers
-            for optimizer in self.optimizers:
-                optimizer.step()
+                # calculate running loss for display
+                self.running_loss.append(self.batch_loss_value)
+                self.batch_loss_value = 0
+                self.avg_loss = np.mean(self.running_loss[-100:])

-                # insert after step hook
-                if self.__is_function_implemented('on_before_zero_grad'):
-                    model_ref = self.__get_model()
-                    response = model_ref.on_before_zero_grad(optimizer)
-
-                # clear gradients
-                optimizer.zero_grad()
-
-            # queuing loss across batches blows it up proportionally...
-            #  divide out the number accumulated
-            self.batch_loss_value = self.batch_loss_value / self.accumulate_grad_batches
-
-            # track loss
-            self.running_loss.append(self.batch_loss_value)
-            self.batch_loss_value = 0
-            self.avg_loss = np.mean(self.running_loss[-100:])
-
-            # update progbar
-            if self.progress_bar:
-                # add model specific metrics
-                tqdm_metrics = self.__tng_tqdm_dic
-                self.prog_bar.set_postfix(**tqdm_metrics)
+                # update progbar
+                if self.show_progress_bar:
+                    # add model specific metrics
+                    tqdm_metrics = self.__tng_tqdm_dic
+                    self.progress_bar.set_postfix(**tqdm_metrics)

        # activate batch end hook
        if self.__is_function_implemented('on_batch_end'):
@@ -971,30 +1107,30 @@ We recommend you switch to ddp if you want to use amp
        elif not can_check_epoch:
            return

-        # hook
-        if self.__is_function_implemented('on_pre_performance_check'):
-            model = self.__get_model()
-            model.on_pre_performance_check()
+        # validate only if model has validation_step defined
+        if self.__is_overriden('validation_step'):

-        # use full val set on end of epoch
-        # use a small portion otherwise
-        max_batches = None if not self.fast_dev_run else 1
-        validation_results = self.validate(
-            self.model,
-            self.val_dataloader,
-            max_batches
-        )
-        self.__add_tqdm_metrics(validation_results)
+            # hook
+            if self.__is_function_implemented('on_pre_performance_check'):
+                model = self.__get_model()
+                model.on_pre_performance_check()

-        # hook
-        if self.__is_function_implemented('on_post_performance_check'):
-            model = self.__get_model()
-            model.on_post_performance_check()
+            # use val_percent_check set on end of epoch
+            # use a small portion otherwise
+            max_batches = self.nb_val_batches if not self.fast_dev_run else 1
+            for ds_i, dataloader in enumerate(self.val_dataloader):
+                val_out_metrics = self.validate(self.model, dataloader, max_batches, ds_i)
+                self.__add_tqdm_metrics(val_out_metrics)

-        if self.progress_bar:
-            # add model specific metrics
-            tqdm_metrics = self.__tng_tqdm_dic
-            self.prog_bar.set_postfix(**tqdm_metrics)
+            # hook
+            if self.__is_function_implemented('on_post_performance_check'):
+                model = self.__get_model()
+                model.on_post_performance_check()
+
+            if self.show_progress_bar:
+                # add model specific metrics
+                tqdm_metrics = self.__tng_tqdm_dic
+                self.progress_bar.set_postfix(**tqdm_metrics)

        # model checkpointing
        if self.proc_rank == 0 and self.checkpoint_callback is not None:
@@ -1,3 +1,5 @@
+import traceback
+

 def data_loader(fn):
    """
@@ -10,8 +12,17 @@ def data_loader(fn):

    @property
    def _data_loader(self):
-        if not hasattr(self, attr_name):
-            setattr(self, attr_name, fn(self))
-        return getattr(self, attr_name)
+        try:
+            value = getattr(self, attr_name)
+        except AttributeError:
+            try:
+                value = fn(self)  # Lazy evaluation, done only once.
+            except AttributeError as e:
+                # Guard against AttributeError suppression. (Issue #142)
+                traceback.print_exc()
+                error = f'{fn.__name__}: An AttributeError was encountered: ' + str(e)
+                raise RuntimeError(error) from e
+            setattr(self, attr_name, value)  # Memoize evaluation.
+        return value

    return _data_loader
@@ -23,6 +23,9 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):

        # track if gpu was requested for checkpointing
        self.on_gpu = False
+        self.use_dp = False
+        self.use_ddp = False
+        self.use_amp = False

    def forward(self, *args, **kwargs):
        """
@@ -33,29 +36,33 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        raise NotImplementedError

-    def validation_step(self, data_batch, batch_nb):
+    def training_step(self, *args, **kwargs):
        """
-        return whatever outputs will need to be aggregated in validation_end
-        :param data_batch:
+        return loss, dict with metrics for tqdm
+        :param called with batch, batch_nb
+        additional: optimizer_i if multiple optimizers used
        :return:
        """
        raise NotImplementedError

+    def validation_step(self, *args, **kwargs):
+        """
+        return whatever outputs will need to be aggregated in validation_end
+        OPTIONAL
+        :param called with batch, batch_nb
+        additional: dataset_i if multiple val datasets used
+        :return:
+        """
+        pass
+
    def validation_end(self, outputs):
        """
        Outputs has the appended output after each validation step
+        OPTIONAL
        :param outputs:
        :return: dic_with_metrics for tqdm
        """
-        raise NotImplementedError
-
-    def training_step(self, data_batch, batch_nb):
-        """
-        return loss, dict with metrics for tqdm
-        :param data_batch:
-        :return:
-        """
-        raise NotImplementedError
+        pass

    def configure_optimizers(self):
        """
@@ -64,10 +71,24 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        raise NotImplementedError

+    def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i):
+        """
+        Do something instead of the standard optimizer behavior
+        :param epoch_nb:
+        :param batch_nb:
+        :param optimizer:
+        :param optimizer_i:
+        :return:
+        """
+        optimizer.step()
+
+        # clear gradients
+        optimizer.zero_grad()
+
    @data_loader
    def tng_dataloader(self):
        """
-        Implement a function to load an h5py of this data
+        Implement a PyTorch DataLoader
        :return:
        """
        raise NotImplementedError
@@ -75,18 +96,18 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
    @data_loader
    def test_dataloader(self):
        """
-        Implement a function to load an h5py of this data
+        Implement a PyTorch DataLoader
        :return:
        """
-        raise NotImplementedError
+        return None

    @data_loader
    def val_dataloader(self):
        """
-        Implement a function to load an h5py of this data
+        Implement a PyTorch DataLoader
        :return:
        """
-        raise NotImplementedError
+        return None

    @classmethod
    def load_from_metrics(cls, weights_path, tags_csv, on_gpu, map_location=None):
@@ -0,0 +1,3 @@
+from .lm_test_module import LightningTestModel
+from .no_val_end_module import NoValEndTestModel
+from .no_val_module import NoValModel
@@ -109,7 +109,7 @@ class LightningTestModel(LightningModule):
        if self.trainer.batch_nb % 2 == 0:
            return loss_val

-    def validation_step(self, data_batch, batch_i):
+    def validation_step(self, data_batch, batch_i, dataloader_i):
        """
        Lightning calls this inside the validation loop
        :param data_batch:
@@ -151,6 +151,12 @@ class LightningTestModel(LightningModule):
                'test_dic': {'val_loss_a': loss_val}
            })
            return output
+        if batch_i % 5 == 0:
+            output = OrderedDict({
+                f'val_loss_{dataloader_i}': loss_val,
+                f'val_acc_{dataloader_i}': val_acc,
+            })
+            return output

    def validation_end(self, outputs):
        """
@@ -189,7 +195,7 @@ class LightningTestModel(LightningModule):
        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)

        # test returning only 1 list instead of 2
-        return [optimizer]
+        return optimizer

    def __dataloader(self, train):
        # init data generators
@@ -203,7 +209,7 @@ class LightningTestModel(LightningModule):
        batch_size = self.hparams.batch_size

        try:
-            if self.on_gpu and not self.force_remove_distributed_sampler:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
@@ -225,7 +231,7 @@ class LightningTestModel(LightningModule):

    @data_loader
    def val_dataloader(self):
-        return self.__dataloader(train=False)
+        return [self.__dataloader(train=False), self.__dataloader(train=False)]

    @data_loader
    def test_dataloader(self):
@@ -0,0 +1,247 @@
+import os
+from collections import OrderedDict
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch import optim
+from torch.utils.data import DataLoader
+from torch.utils.data.distributed import DistributedSampler
+from torchvision.datasets import MNIST
+from torchvision import transforms
+from test_tube import HyperOptArgumentParser
+
+from pytorch_lightning.root_module.root_module import LightningModule
+from pytorch_lightning import data_loader
+
+
+class NoValEndTestModel(LightningModule):
+    """
+    Sample model to show how to define a template
+    """
+
+    def __init__(self, hparams, force_remove_distributed_sampler=False):
+        """
+        Pass in parsed HyperOptArgumentParser to the model
+        :param hparams:
+        """
+        # init superclass
+        super(NoValEndTestModel, self).__init__()
+        self.hparams = hparams
+
+        self.batch_size = hparams.batch_size
+
+        # if you specify an example input, the summary will show input/output for each layer
+        self.example_input_array = torch.rand(5, 28 * 28)
+
+        # remove to test warning for dist sampler
+        self.force_remove_distributed_sampler = force_remove_distributed_sampler
+
+        # build model
+        self.__build_model()
+
+    # ---------------------
+    # MODEL SETUP
+    # ---------------------
+    def __build_model(self):
+        """
+        Layout model
+        :return:
+        """
+        self.c_d1 = nn.Linear(in_features=self.hparams.in_features,
+                              out_features=self.hparams.hidden_dim)
+        self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
+        self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)
+
+        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim,
+                              out_features=self.hparams.out_features)
+
+    # ---------------------
+    # TRAINING
+    # ---------------------
+    def forward(self, x):
+        """
+        No special modification required for lightning, define as you normally would
+        :param x:
+        :return:
+        """
+
+        x = self.c_d1(x)
+        x = torch.tanh(x)
+        x = self.c_d1_bn(x)
+        x = self.c_d1_drop(x)
+
+        x = self.c_d2(x)
+        logits = F.log_softmax(x, dim=1)
+
+        return logits
+
+    def loss(self, labels, logits):
+        nll = F.nll_loss(logits, labels)
+        return nll
+
+    def training_step(self, data_batch, batch_i):
+        """
+        Lightning calls this inside the training loop
+        :param data_batch:
+        :return:
+        """
+        # forward pass
+        x, y = data_batch
+        x = x.view(x.size(0), -1)
+
+        y_hat = self.forward(x)
+
+        # calculate loss
+        loss_val = self.loss(y, y_hat)
+
+        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
+        if self.trainer.use_dp:
+            loss_val = loss_val.unsqueeze(0)
+
+        # alternate possible outputs to test
+        if self.trainer.batch_nb % 1 == 0:
+            output = OrderedDict({
+                'loss': loss_val,
+                'prog': {'some_val': loss_val * loss_val}
+            })
+            return output
+        if self.trainer.batch_nb % 2 == 0:
+            return loss_val
+
+    def validation_step(self, data_batch, batch_nb):
+        """
+        Lightning calls this inside the validation loop
+        :param data_batch:
+        :return:
+        """
+        x, y = data_batch
+        x = x.view(x.size(0), -1)
+        y_hat = self.forward(x)
+
+        loss_val = self.loss(y, y_hat)
+
+        # acc
+        labels_hat = torch.argmax(y_hat, dim=1)
+        val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
+        val_acc = torch.tensor(val_acc)
+
+        if self.on_gpu:
+            val_acc = val_acc.cuda(loss_val.device.index)
+
+        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
+        if self.trainer.use_dp:
+            loss_val = loss_val.unsqueeze(0)
+            val_acc = val_acc.unsqueeze(0)
+
+        # alternate possible outputs to test
+        if batch_nb % 1 == 0:
+            output = OrderedDict({
+                'val_loss': loss_val,
+                'val_acc': val_acc,
+            })
+            return output
+        if batch_nb % 2 == 0:
+            return val_acc
+
+        if batch_nb % 3 == 0:
+            output = OrderedDict({
+                'val_loss': loss_val,
+                'val_acc': val_acc,
+                'test_dic': {'val_loss_a': loss_val}
+            })
+            return output
+
+    def on_tng_metrics(self, logs):
+        logs['some_tensor_to_test'] = torch.rand(1)
+
+    # ---------------------
+    # TRAINING SETUP
+    # ---------------------
+    def configure_optimizers(self):
+        """
+        return whatever optimizers we want here
+        :return: list of optimizers
+        """
+        # try no scheduler for this model (testing purposes)
+        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+
+        # test returning only 1 list instead of 2
+        return [optimizer]
+
+    def __dataloader(self, train):
+        # init data generators
+        transform = transforms.Compose([transforms.ToTensor(),
+                                        transforms.Normalize((0.5,), (1.0,))])
+        dataset = MNIST(root=self.hparams.data_root, train=train,
+                        transform=transform, download=True)
+
+        # when using multi-node we need to add the datasampler
+        train_sampler = None
+        batch_size = self.hparams.batch_size
+
+        try:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
+                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
+                batch_size = batch_size // self.trainer.world_size  # scale batch size
+        except Exception:
+            pass
+
+        should_shuffle = train_sampler is None
+        loader = DataLoader(
+            dataset=dataset,
+            batch_size=batch_size,
+            shuffle=should_shuffle,
+            sampler=train_sampler
+        )
+
+        return loader
+
+    @data_loader
+    def tng_dataloader(self):
+        return self.__dataloader(train=True)
+
+    @data_loader
+    def val_dataloader(self):
+        return self.__dataloader(train=False)
+
+    @data_loader
+    def test_dataloader(self):
+        return self.__dataloader(train=False)
+
+    @staticmethod
+    def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
+        """
+        Parameters you define here will be available to your model through self.hparams
+        :param parent_parser:
+        :param root_dir:
+        :return:
+        """
+        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
+
+        # param overwrites
+        # parser.set_defaults(gradient_clip=5.0)
+
+        # network params
+        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
+        parser.add_argument('--in_features', default=28 * 28, type=int)
+        parser.add_argument('--out_features', default=10, type=int)
+        # use 500 for CPU, 50000 for GPU to see speed difference
+        parser.add_argument('--hidden_dim', default=50000, type=int)
+
+        # data
+        parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
+
+        # training params (opt)
+        parser.opt_list('--learning_rate', default=0.001 * 8, type=float,
+                        options=[0.0001, 0.0005, 0.001, 0.005],
+                        tunable=False)
+        parser.opt_list('--optimizer_name', default='adam', type=str,
+                        options=['adam'], tunable=False)
+
+        # if using 2 nodes with 4 gpus each the batch size here
+        #  (256) will be 256 / (2*8) = 16 per gpu
+        parser.opt_list('--batch_size', default=256 * 8, type=int,
+                        options=[32, 64, 128, 256], tunable=False,
+                        help='batch size will be divided over all gpus being used across all nodes')
+        return parser
@@ -0,0 +1,196 @@
+import os
+from collections import OrderedDict
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch import optim
+from torch.utils.data import DataLoader
+from torch.utils.data.distributed import DistributedSampler
+from torchvision.datasets import MNIST
+from torchvision import transforms
+from test_tube import HyperOptArgumentParser
+
+from pytorch_lightning.root_module.root_module import LightningModule
+from pytorch_lightning import data_loader
+
+
+class NoValModel(LightningModule):
+    """
+    Sample model to show how to define a template
+    """
+
+    def __init__(self, hparams, force_remove_distributed_sampler=False):
+        """
+        Pass in parsed HyperOptArgumentParser to the model
+        :param hparams:
+        """
+        # init superclass
+        super(NoValModel, self).__init__()
+        self.hparams = hparams
+
+        self.batch_size = hparams.batch_size
+
+        # if you specify an example input, the summary will show input/output for each layer
+        self.example_input_array = torch.rand(5, 28 * 28)
+
+        # remove to test warning for dist sampler
+        self.force_remove_distributed_sampler = force_remove_distributed_sampler
+
+        # build model
+        self.__build_model()
+
+    # ---------------------
+    # MODEL SETUP
+    # ---------------------
+    def __build_model(self):
+        """
+        Layout model
+        :return:
+        """
+        self.c_d1 = nn.Linear(in_features=self.hparams.in_features,
+                              out_features=self.hparams.hidden_dim)
+        self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
+        self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)
+
+        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim,
+                              out_features=self.hparams.out_features)
+
+    # ---------------------
+    # TRAINING
+    # ---------------------
+    def forward(self, x):
+        """
+        No special modification required for lightning, define as you normally would
+        :param x:
+        :return:
+        """
+
+        x = self.c_d1(x)
+        x = torch.tanh(x)
+        x = self.c_d1_bn(x)
+        x = self.c_d1_drop(x)
+
+        x = self.c_d2(x)
+        logits = F.log_softmax(x, dim=1)
+
+        return logits
+
+    def loss(self, labels, logits):
+        nll = F.nll_loss(logits, labels)
+        return nll
+
+    def training_step(self, data_batch, batch_i):
+        """
+        Lightning calls this inside the training loop
+        :param data_batch:
+        :return:
+        """
+        # forward pass
+        x, y = data_batch
+        x = x.view(x.size(0), -1)
+
+        y_hat = self.forward(x)
+
+        # calculate loss
+        loss_val = self.loss(y, y_hat)
+
+        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
+        if self.trainer.use_dp:
+            loss_val = loss_val.unsqueeze(0)
+
+        # alternate possible outputs to test
+        if self.trainer.batch_nb % 1 == 0:
+            output = OrderedDict({
+                'loss': loss_val,
+                'prog': {'some_val': loss_val * loss_val}
+            })
+            return output
+        if self.trainer.batch_nb % 2 == 0:
+            return loss_val
+
+    def on_tng_metrics(self, logs):
+        logs['some_tensor_to_test'] = torch.rand(1)
+
+    # ---------------------
+    # TRAINING SETUP
+    # ---------------------
+    def configure_optimizers(self):
+        """
+        return whatever optimizers we want here
+        :return: list of optimizers
+        """
+        # try no scheduler for this model (testing purposes)
+        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+
+        # test returning only 1 list instead of 2
+        return [optimizer]
+
+    def __dataloader(self, train):
+        # init data generators
+        transform = transforms.Compose([transforms.ToTensor(),
+                                        transforms.Normalize((0.5,), (1.0,))])
+        dataset = MNIST(root=self.hparams.data_root, train=train,
+                        transform=transform, download=True)
+
+        # when using multi-node we need to add the datasampler
+        train_sampler = None
+        batch_size = self.hparams.batch_size
+
+        try:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
+                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
+                batch_size = batch_size // self.trainer.world_size  # scale batch size
+        except Exception:
+            pass
+
+        should_shuffle = train_sampler is None
+        loader = DataLoader(
+            dataset=dataset,
+            batch_size=batch_size,
+            shuffle=should_shuffle,
+            sampler=train_sampler
+        )
+
+        return loader
+
+    @data_loader
+    def tng_dataloader(self):
+        return self.__dataloader(train=True)
+
+    @staticmethod
+    def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
+        """
+        Parameters you define here will be available to your model through self.hparams
+        :param parent_parser:
+        :param root_dir:
+        :return:
+        """
+        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
+
+        # param overwrites
+        # parser.set_defaults(gradient_clip=5.0)
+
+        # network params
+        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
+        parser.add_argument('--in_features', default=28 * 28, type=int)
+        parser.add_argument('--out_features', default=10, type=int)
+        # use 500 for CPU, 50000 for GPU to see speed difference
+        parser.add_argument('--hidden_dim', default=50000, type=int)
+
+        # data
+        parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
+
+        # training params (opt)
+        parser.opt_list('--learning_rate', default=0.001 * 8, type=float,
+                        options=[0.0001, 0.0005, 0.001, 0.005],
+                        tunable=False)
+        parser.opt_list('--optimizer_name', default='adam', type=str,
+                        options=['adam'], tunable=False)
+
+        # if using 2 nodes with 4 gpus each the batch size here
+        #  (256) will be 256 / (2*8) = 16 per gpu
+        parser.opt_list('--batch_size', default=256 * 8, type=int,
+                        options=[32, 64, 128, 256], tunable=False,
+                        help='batch size will be divided over all gpus being used across all nodes')
+        return parser
@@ -45,6 +45,7 @@ omit =
    tests/test_models.py
    pytorch_lightning/testing_models/lm_test_module.py
    pytorch_lightning/utilities/arg_parse.py
+    examples/templates

 [flake8]
 ignore = E731,W504,F401,F841
@@ -9,12 +9,12 @@ from setuptools import setup, find_packages

 # https://packaging.python.org/discussions/install-requires-vs-requirements /
 # keep the meta-data here for simplicity in reading this file... it's not obvious
-# what happens and to non-engineers they won't know to look in init...
+# what happens and to non-engineers they won't know to look in init ...
 # the goal of the project is simplicity for researchers, don't want to add too much
 # engineer specific practices
 setup(
    name='pytorch-lightning',
-    version='0.4.0',
+    version='0.4.7',
    description='The Keras for ML researchers using PyTorch',
    author='William Falcon',
    author_email='waf2107@columbia.edu',
@@ -29,9 +29,9 @@ setup(
    keywords=['deep learning', 'pytorch', 'AI'],
    python_requires='>=3.6',
    install_requires=[
-        'torch==1.1.0',
+        'torch==1.2.0',
        'tqdm',
-        'test-tube>=0.6.7.6',
+        'test-tube>=0.6.9',
        'pandas>=0.20.3',
    ],
    classifiers=[
@@ -11,6 +11,7 @@ import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
+import numpy as np


 class CoolModel(pl.LightningModule):
@@ -136,41 +137,60 @@ def run_prediction(dataloader, trained_model):
    assert val_acc > 0.70, 'this model is expected to get > 0.7 in test set (it got %f)' % val_acc


-def main():
-
+def run_gpu_model_test(trainer_options, model, hparams, on_gpu=True):
    save_dir = init_save_dir()

    # exp file to get meta
    exp = get_exp(False)
+    exp.argparse(hparams)
    exp.save()

    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)

-    trainer = Trainer(
-        experiment=exp,
-        checkpoint_callback=checkpoint,
-        progress_bar=True,
-        max_nb_epochs=1,
-        gpus=[0, 1],
-        distributed_backend='dp',
-    )
-
-    model = CoolModel()
+    # add these to the trainer options
+    trainer_options['checkpoint_callback'] = checkpoint
+    trainer_options['experiment'] = exp

+    # fit model
+    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)

    # correct result and ok accuracy
    assert result == 1, 'amp + ddp model failed to complete'

    # test model loading
-    pretrained_model = load_model(exp, save_dir)
+    pretrained_model = load_model(exp, save_dir, on_gpu)

    # test model preds
    run_prediction(model.test_dataloader, pretrained_model)

+    if trainer.use_ddp:
+        # on hpc this would work fine... but need to hack it for the purpose of the test
+        trainer.model = pretrained_model
+        trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers()
+
+    # test HPC loading / saving
+    trainer.hpc_save(save_dir, exp)
+    trainer.hpc_load(save_dir, on_gpu=on_gpu)
+
    clear_save_dir()


+def main():
+
+    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
+    model, hparams = get_model()
+    trainer_options = dict(
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        gpus=[0, 1],
+        distributed_backend='ddp'
+    )
+
+    run_gpu_model_test(trainer_options, model, hparams)
+
+
 if __name__ == '__main__':
    main()
@@ -10,7 +10,7 @@ from test_tube import Experiment, SlurmCluster

 # sys.path += [os.path.abspath('..'), os.path.abspath('../..')]
 from pytorch_lightning import Trainer
-from pytorch_lightning.testing.lm_test_module import LightningTestModel
+from pytorch_lightning.testing import LightningTestModel, NoValEndTestModel, NoValModel
 from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
 from pytorch_lightning.utilities.debugging import MisconfigurationException
 from pytorch_lightning.root_module import memory
@@ -26,6 +26,227 @@ np.random.seed(SEED)
 # ------------------------------------------------------------------------
 # TESTS
 # ------------------------------------------------------------------------
+def test_multi_gpu_model_ddp():
+    """
+    Make sure DDP works
+    :return:
+    """
+    if not torch.cuda.is_available():
+        warnings.warn('test_multi_gpu_model_ddp cannot run.'
+                      ' Rerun on a GPU node to run this test')
+        return
+    if not torch.cuda.device_count() > 1:
+        warnings.warn('test_multi_gpu_model_ddp cannot run.'
+                      ' Rerun on a node with 2+ GPUs to run this test')
+        return
+
+    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
+    model, hparams = get_model()
+    trainer_options = dict(
+        show_progress_bar=False,
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        gpus=[0, 1],
+        distributed_backend='ddp'
+    )
+
+    run_gpu_model_test(trainer_options, model, hparams)
+
+
+def test_optimizer_return_options():
+
+    trainer = Trainer()
+    model, hparams = get_model()
+
+    # single optimizer
+    opt_a = torch.optim.Adam(model.parameters(), lr=0.002)
+    opt_b = torch.optim.SGD(model.parameters(), lr=0.002)
+    optim, lr_sched = trainer.init_optimizers(opt_a)
+    assert len(optim) == 1 and len(lr_sched) == 0
+
+    # opt tuple
+    opts = (opt_a, opt_b)
+    optim, lr_sched = trainer.init_optimizers(opts)
+    assert len(optim) == 2 and optim[0] == opts[0] and optim[1] == opts[1]
+    assert len(lr_sched) == 0
+
+    # opt list
+    opts = [opt_a, opt_b]
+    optim, lr_sched = trainer.init_optimizers(opts)
+    assert len(optim) == 2 and optim[0] == opts[0] and optim[1] == opts[1]
+    assert len(lr_sched) == 0
+
+    # opt tuple of lists
+    opts = ([opt_a], ['lr_scheduler'])
+    optim, lr_sched = trainer.init_optimizers(opts)
+    assert len(optim) == 1 and len(lr_sched) == 1
+    assert optim[0] == opts[0][0] and lr_sched[0] == 'lr_scheduler'
+
+
+def test_single_gpu_batch_parse():
+    if not torch.cuda.is_available():
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      'Rerun on a GPU node to run this test')
+        return
+    if not torch.cuda.device_count() > 1:
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      'Rerun on a node with 2+ GPUs to run this test')
+        return
+
+    trainer = Trainer()
+
+    # batch is just a tensor
+    batch = torch.rand(2, 3)
+    batch = trainer.transfer_batch_to_gpu(batch, 0)
+    assert batch.device.index == 0 and batch.type() == 'torch.cuda.FloatTensor'
+
+    # tensor list
+    batch = [torch.rand(2, 3), torch.rand(2, 3)]
+    batch = trainer.transfer_batch_to_gpu(batch, 0)
+    assert batch[0].device.index == 0 and batch[0].type() == 'torch.cuda.FloatTensor'
+    assert batch[1].device.index == 0 and batch[1].type() == 'torch.cuda.FloatTensor'
+
+    # tensor list of lists
+    batch = [[torch.rand(2, 3), torch.rand(2, 3)]]
+    batch = trainer.transfer_batch_to_gpu(batch, 0)
+    assert batch[0][0].device.index == 0 and batch[0][0].type() == 'torch.cuda.FloatTensor'
+    assert batch[0][1].device.index == 0 and batch[0][1].type() == 'torch.cuda.FloatTensor'
+
+    # tensor dict
+    batch = [{'a': torch.rand(2, 3), 'b': torch.rand(2, 3)}]
+    batch = trainer.transfer_batch_to_gpu(batch, 0)
+    assert batch[0]['a'].device.index == 0 and batch[0]['a'].type() == 'torch.cuda.FloatTensor'
+    assert batch[0]['b'].device.index == 0 and batch[0]['b'].type() == 'torch.cuda.FloatTensor'
+
+    # tuple of tensor list and list of tensor dict
+    batch = ([torch.rand(2, 3) for _ in range(2)],
+             [{'a': torch.rand(2, 3), 'b': torch.rand(2, 3)} for _ in range(2)])
+    batch = trainer.transfer_batch_to_gpu(batch, 0)
+    assert batch[0][0].device.index == 0 and batch[0][0].type() == 'torch.cuda.FloatTensor'
+
+    assert batch[1][0]['a'].device.index == 0
+    assert batch[1][0]['a'].type() == 'torch.cuda.FloatTensor'
+
+    assert batch[1][0]['b'].device.index == 0
+    assert batch[1][0]['b'].type() == 'torch.cuda.FloatTensor'
+
+
+def test_early_stopping_cpu_model():
+    """
+    Test each of the trainer options
+    :return:
+    """
+
+    stopping = EarlyStopping(monitor='val_loss')
+    trainer_options = dict(
+        early_stop_callback=stopping,
+        gradient_clip=1.0,
+        overfit_pct=0.20,
+        track_grad_norm=2,
+        print_nan_grads=True,
+        show_progress_bar=False,
+        experiment=get_exp(),
+        train_percent_check=0.1,
+        val_percent_check=0.1
+    )
+
+    model, hparams = get_model()
+    run_gpu_model_test(trainer_options, model, hparams, on_gpu=False)
+
+    # test freeze on cpu
+    model.freeze()
+    model.unfreeze()
+
+
+def test_no_val_module():
+    """
+    Tests use case where trainer saves the model, and user loads it from tags independently
+    :return:
+    """
+    hparams = get_hparams()
+    model = NoValModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    exp = get_exp(False)
+    exp.argparse(hparams)
+    exp.save()
+
+    trainer_options = dict(
+        max_nb_epochs=1,
+        cluster=SlurmCluster(),
+        experiment=exp,
+        checkpoint_callback=ModelCheckpoint(save_dir)
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # traning complete
+    assert result == 1, 'amp + ddp model failed to complete'
+
+    # save model
+    new_weights_path = os.path.join(save_dir, 'save_test.ckpt')
+    trainer.save_checkpoint(new_weights_path)
+
+    # load new model
+    tags_path = exp.get_data_path(exp.name, exp.version)
+    tags_path = os.path.join(tags_path, 'meta_tags.csv')
+    model_2 = LightningTestModel.load_from_metrics(weights_path=new_weights_path,
+                                                   tags_csv=tags_path, on_gpu=False)
+    model_2.eval()
+
+    # make prediction
+    clear_save_dir()
+
+
+def test_no_val_end_module():
+    """
+    Tests use case where trainer saves the model, and user loads it from tags independently
+    :return:
+    """
+    hparams = get_hparams()
+    model = NoValEndTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    exp = get_exp(False)
+    exp.argparse(hparams)
+    exp.save()
+
+    trainer_options = dict(
+        max_nb_epochs=1,
+        cluster=SlurmCluster(),
+        experiment=exp,
+        checkpoint_callback=ModelCheckpoint(save_dir)
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # traning complete
+    assert result == 1, 'amp + ddp model failed to complete'
+
+    # save model
+    new_weights_path = os.path.join(save_dir, 'save_test.ckpt')
+    trainer.save_checkpoint(new_weights_path)
+
+    # load new model
+    tags_path = exp.get_data_path(exp.name, exp.version)
+    tags_path = os.path.join(tags_path, 'meta_tags.csv')
+    model_2 = LightningTestModel.load_from_metrics(weights_path=new_weights_path,
+                                                   tags_csv=tags_path, on_gpu=False)
+    model_2.eval()
+
+    # make prediction
+    clear_save_dir()
+
+
 def test_simple_cpu():
    """
    Verify continue training session on CPU
@@ -71,7 +292,7 @@ def test_amp_single_gpu():
    model = LightningTestModel(hparams)

    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0],
        distributed_backend='dp',
@@ -136,7 +357,7 @@ def test_cpu_restore_training():
        # if model and state loaded correctly, predictions will be good even though we
        # haven't trained with the new loaded model
        trainer.model.eval()
-        run_prediction(trainer.val_dataloader, trainer.model)
+        _ = [run_prediction(dataloader, trainer.model) for dataloader in trainer.val_dataloader]

    model.on_sanity_check_start = assert_good_acc

@@ -167,7 +388,7 @@ def test_amp_gpu_ddp():
    model = LightningTestModel(hparams)

    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0, 1],
        distributed_backend='ddp',
@@ -387,7 +608,7 @@ def test_amp_gpu_ddp_slurm_managed():
    model = LightningTestModel(hparams)

    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0],
        distributed_backend='ddp',
@@ -445,33 +666,6 @@ def test_amp_gpu_ddp_slurm_managed():
    clear_save_dir()


-def test_early_stopping_cpu_model():
-    """
-    Test each of the trainer options
-    :return:
-    """
-
-    stopping = EarlyStopping()
-    trainer_options = dict(
-        early_stop_callback=stopping,
-        gradient_clip=1.0,
-        overfit_pct=0.20,
-        track_grad_norm=2,
-        print_nan_grads=True,
-        progress_bar=False,
-        experiment=get_exp(),
-        train_percent_check=0.1,
-        val_percent_check=0.1
-    )
-
-    model, hparams = get_model()
-    run_gpu_model_test(trainer_options, model, hparams, on_gpu=False)
-
-    # test freeze on cpu
-    model.freeze()
-    model.unfreeze()
-
-
 def test_cpu_model_with_amp():
    """
    Make sure model trains on CPU
@@ -479,7 +673,7 @@ def test_cpu_model_with_amp():
    """

    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        max_nb_epochs=1,
        train_percent_check=0.4,
@@ -500,7 +694,7 @@ def test_cpu_model():
    """

    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        max_nb_epochs=1,
        train_percent_check=0.4,
@@ -523,8 +717,9 @@ def test_all_features_cpu_model():
        overfit_pct=0.20,
        track_grad_norm=2,
        print_nan_grads=True,
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
+        accumulate_grad_batches=2,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.4
@@ -546,7 +741,7 @@ def test_single_gpu_model():
    model, hparams = get_model()

    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.1,
        val_percent_check=0.1,
@@ -571,7 +766,7 @@ def test_multi_gpu_model_dp():
        return
    model, hparams = get_model()
    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.1,
        val_percent_check=0.1,
@@ -608,34 +803,6 @@ def test_amp_gpu_dp():
        run_gpu_model_test(trainer_options, model, hparams)


-def test_multi_gpu_model_ddp():
-    """
-    Make sure DDP works
-    :return:
-    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_multi_gpu_model_ddp cannot run.'
-                      ' Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_multi_gpu_model_ddp cannot run.'
-                      ' Rerun on a node with 2+ GPUs to run this test')
-        return
-
-    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
-    model, hparams = get_model()
-    trainer_options = dict(
-        progress_bar=False,
-        max_nb_epochs=1,
-        train_percent_check=0.4,
-        val_percent_check=0.2,
-        gpus=[0, 1],
-        distributed_backend='ddp'
-    )
-
-    run_gpu_model_test(trainer_options, model, hparams)
-
-
 def test_ddp_sampler_error():
    """
    Make sure DDP + AMP work
@@ -658,19 +825,50 @@ def test_ddp_sampler_error():

    trainer = Trainer(
        experiment=exp,
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        gpus=[0, 1],
        distributed_backend='ddp',
        use_amp=True
    )

-    with pytest.raises(MisconfigurationException):
+    with pytest.warns(UserWarning):
        trainer.get_dataloaders(model)

    clear_save_dir()


+def test_multiple_val_dataloader():
+    """
+    Verify multiple val_dataloader
+    :return:
+    """
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    trainer_options = dict(
+        max_nb_epochs=1,
+        val_percent_check=0.1,
+        train_percent_check=0.1,
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # verify tng completed
+    assert result == 1
+
+    # verify there are 2 val loaders
+    assert len(trainer.val_dataloader) == 2, 'Multiple val_dataloaders not initiated properly'
+
+    # make sure predictions are good for each val set
+    [run_prediction(dataloader, trainer.model) for dataloader in trainer.val_dataloader]
+
+
 # ------------------------------------------------------------------------
 # UTILS
 # ------------------------------------------------------------------------
@@ -38,7 +38,7 @@ commands =
    flake8 .

 [flake8]
-exclude = .tox,*.egg,build,temp
+exclude = .tox,*.egg,build,temp,examples/templates
 select = E,W,F
 doctests = True
 verbose = 2
Author	SHA1	Message	Date
William Falcon	c24599f5e5	release v	2019-08-24 08:13:54 -04:00
Ryan McCormick	b22e5918a9	fix python syntax in code blocks to be consistent (#166 ) A couple code blocks used "{.python}" instead of just "python" for the syntax highlighting, which doesn't render properly in GitHub markdown.	2019-08-23 21:24:18 -04:00
William Falcon	4104a0fc47	cleaned up progbar (#165 ) * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * flake 8	2019-08-23 21:23:27 -04:00
William Falcon	2ad9a9708b	Update README.md	2019-08-23 16:10:45 -04:00
William Falcon	ecce22f4de	Update README.md	2019-08-23 16:10:24 -04:00
Sebastian Præsius	b31539f62e	Guard against AttributeError in dataloaders. (#161 ) A solution for https://github.com/williamFalcon/pytorch-lightning/issues/142. Since hasattr "calls getattr(object, name) and to see whether it raises an AttributeError or not", I replaced it with a single call to getattr. See also https://stackoverflow.com/questions/24971061/python-hasattr-vs-getattr	2019-08-23 08:21:39 -04:00
William Falcon	cbb9821d9b	Cleaned up val/tng/test nb batches (#163 ) Set all to be 0 instead of None. Cleaned up val batch	2019-08-23 07:42:17 -04:00
William Falcon	a1490e993a	Update README.md	2019-08-23 03:46:27 -04:00
William Falcon	77d085134b	Update README.md	2019-08-23 03:45:14 -04:00
William Falcon	c30f69f60d	Update lightning_module_template.py	2019-08-23 02:42:40 -04:00
William Falcon	d5d47eab0d	Update lightning_module_template.py	2019-08-23 02:39:05 -04:00
Sebastian Præsius	9fc66026f1	train = False in test_dataloader (#162 ) A small change to the CoolModel example. Now test_dataloader returns the MNIST test dataset.	2019-08-22 17:44:06 -04:00
eqs	4a0b56755c	bug fix for #157 (#158 ) * Separate condition list/tuple case into separated cases * Add test for tuple of tensor list and list of tensor dict * Update test_models.py	2019-08-21 10:22:51 -04:00
William Falcon	55a804b7cf	fixes #154 (#155 ) * fixes #154 * Update trainer.py * Update trainer.py	2019-08-20 16:59:26 -04:00
William Falcon	7119ec1693	Update CONTRIBUTING.md	2019-08-20 09:51:11 -04:00
Ananya Harsh Jha	5b694c7e0e	bug fix for #138 (#143 ) * bug fix for #138 * split if for readability	2019-08-19 15:03:04 -04:00
sebftw	4bdb976284	Set val_check_interval default to 1.0. (#145 ) See discussion in https://github.com/williamFalcon/pytorch-lightning/issues/139.	2019-08-19 10:42:08 -04:00
William Falcon	4ad4588122	Update README.md	2019-08-19 07:22:02 -04:00
William Falcon	ac8186cb3c	Update README.md	2019-08-19 07:20:10 -04:00
Jirka Borovec	dbbbba35c9	add Codecov info (#144 )	2019-08-19 06:35:09 -04:00
William Falcon	f2a02881e3	Update README.md	2019-08-18 19:17:25 -04:00
William Falcon	e8c423a3b0	Update README.md	2019-08-18 19:16:57 -04:00
William Falcon	73b70584e7	Update README.md	2019-08-18 19:16:25 -04:00
William Falcon	736cf9b162	Update README.md	2019-08-18 19:16:09 -04:00
William Falcon	5771583c9d	Update README.md	2019-08-18 19:15:41 -04:00
William Falcon	64503f0d5e	Update README.md	2019-08-18 19:15:09 -04:00
William Falcon	504418d157	Update README.md	2019-08-18 19:05:13 -04:00
William Falcon	ad61b03fe9	Update README.md	2019-08-18 18:51:47 -04:00
sebftw	a7a14dadb6	F.cross_entropy(y_hat, y)(y_hat, y) typo. (#137 ) This seems to be a typo. Throws TypeError: 'Tensor' object is not callable.	2019-08-18 18:17:43 -04:00
sebftw	b2a49197e4	tensorboarX to tensorboardX (#136 ) * tensorboarX to tensorboardX * Update properties.md	2019-08-18 18:17:05 -04:00
sebftw	23a4421595	Removed redundant line. (#140 )	2019-08-18 18:16:30 -04:00
sebftw	26d3f0dbea	Error if dataset size = 1 batch. (#141 ) Fix for the bug mentioned in https://github.com/williamFalcon/pytorch-lightning/issues/139	2019-08-18 18:15:58 -04:00
Maxim Andreev	e646d745da	use val_percent_check in validation step (#135 )	2019-08-18 11:02:28 -04:00
William Falcon	9aa9a1a796	Update lightning_module_template.py	2019-08-17 11:11:07 -04:00
Ir1dXD	48de39ed50	elaborate on the correlation between overfit_pct and xxx_percent_check (#132 ) * Update Training Loop.md * update docs and elaborate on the correlation	2019-08-17 10:23:25 -04:00
William Falcon	1a31782272	fixed str crash err	2019-08-17 10:20:58 -04:00
Ir1dXD	24a97956e4	fix typo in docs (#129 ) * fix typo * fix typo * fix typo * fix list	2019-08-17 07:48:33 -04:00
William Falcon	1b7d66d089	Merge branch 'master' of https://github.com/williamFalcon/pytorch-lightning	2019-08-16 17:14:40 -04:00
William Falcon	e60e002f17	updated docs	2019-08-16 17:14:31 -04:00
William Falcon	308239cef0	allow loss to be used for early stopping (#127 )	2019-08-16 11:58:44 -04:00
William Falcon	bdd86087e6	updated docs	2019-08-16 10:07:56 -04:00
William Falcon	50f0de094f	updated docs	2019-08-16 10:07:44 -04:00
William Falcon	bc401d0f59	updated docs	2019-08-16 10:02:28 -04:00
William Falcon	4b97319c2e	updated docs	2019-08-15 21:29:25 -04:00
William Falcon	90f01c05bc	updated docs	2019-08-15 21:21:26 -04:00
William Falcon	0e92a9d7af	updated docs	2019-08-15 21:19:29 -04:00
William Falcon	81837221a4	updated docs	2019-08-15 13:59:54 -04:00
William Falcon	44da88fd15	updated docs	2019-08-15 13:59:27 -04:00
William Falcon	3f1feb014f	release v0.4.6	2019-08-15 13:55:58 -04:00
Jirka Borovec	6f1d2c45fe	update Win CI req. (#123 )	2019-08-15 11:45:03 -04:00
William Falcon	a27fb5d54c	enhanced optimizer return options (#120 ) * added smarter optimizer options * added smarter optimizer options * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive	2019-08-15 11:31:56 -04:00
Jirka Borovec	83b1646e45	fix appveyor (#69 ) * fix appveyor * fix appveyor	2019-08-15 09:54:29 -04:00
William Falcon	db9254acbe	enable recursive parsing for single gpu inputs (#121 ) * added tests * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive	2019-08-15 09:39:09 -04:00
William Falcon	0f287ce5ea	Update tox.ini	2019-08-14 10:22:41 -04:00
William Falcon	590282f2b0	Update gan.py	2019-08-14 09:29:02 -04:00
William Falcon	2f984c9971	enable returning only opt list (#114 )	2019-08-14 09:02:11 -04:00
William Falcon	b64e94bae3	Merge branch 'master' of https://github.com/williamFalcon/pytorch-lightning	2019-08-14 08:59:15 -04:00
William Falcon	e000f052ac	ommit templates folder	2019-08-14 08:59:05 -04:00
William Falcon	c9117f74b2	Update gan.py	2019-08-14 08:43:50 -04:00
William Falcon	13f2d1ab1c	Update gan.py	2019-08-14 08:41:32 -04:00
William Falcon	0d5da5f29b	added gan template (#115 ) * added gan template * ommit templates folder	2019-08-14 08:38:49 -04:00
William Falcon	4795130538	Update README.md	2019-08-14 07:21:45 -04:00
William Falcon	5a834c794b	Update README.md	2019-08-14 07:19:58 -04:00
Ir1dXD	f0af138675	docs: enable syntax highlight (#109 )	2019-08-13 16:19:58 -04:00
William Falcon	3dea127edb	updated docs	2019-08-13 13:05:47 -04:00
William Falcon	d4b1ac94a0	updated docs	2019-08-13 13:03:39 -04:00
William Falcon	087be2f1c4	Merge branch 'master' of https://github.com/williamFalcon/pytorch-lightning	2019-08-13 13:02:21 -04:00
William Falcon	b89b7f0a8c	updated docs	2019-08-13 13:02:17 -04:00
William Falcon	bb75cec076	Update README.md	2019-08-13 12:56:12 -04:00
William Falcon	1cd5dde164	Update README.md	2019-08-13 12:11:24 -04:00
William Falcon	b02f4a4ccf	Update README.md	2019-08-13 12:10:08 -04:00
William Falcon	699fbabda7	updated optimizer_step docs	2019-08-13 11:59:33 -04:00
William Falcon	fd845d41c0	updated optimizer_step docs	2019-08-13 11:57:02 -04:00
William Falcon	d7660d3c64	updated optimizer_step docs	2019-08-13 11:55:10 -04:00
William Falcon	7e38f1f246	updated optimizer_step docs	2019-08-13 11:54:19 -04:00
William Falcon	7898d0c02a	updated optimizer_step docs	2019-08-13 11:51:31 -04:00
William Falcon	89c4c260ad	release v0.4.5	2019-08-13 11:48:17 -04:00
William Falcon	53ec3bc5bc	updated optimizer_step docs	2019-08-13 11:47:35 -04:00
William Falcon	acc16565c5	updated multiple val dataset docs	2019-08-13 11:43:21 -04:00
William Falcon	0d31b9a229	updated readme	2019-08-13 11:38:35 -04:00
William Falcon	7f53e7bfb3	Val idx optional in validation_step (#108 ) * made dataset_i only available with multiple datasets * updated interface signature * updated tests	2019-08-13 11:37:37 -04:00
William Falcon	905a2e5a12	allow user to control optimizer step for every optimizer * added custom hook for user defined optimizer step * refactored to allow multiple optimizers different training_step * refactored to allow multiple optimizers different training_step * refactored to allow multiple optimizers different training_step * refactored to allow multiple optimizers different training_step * refactored to allow multiple optimizers different training_step * pep8	2019-08-13 09:32:45 -04:00
William Falcon	1c08882e6c	Update issue templates	2019-08-13 07:06:17 -04:00
William Falcon	6f3152bcd6	Update README.md	2019-08-13 06:42:25 -04:00
William Falcon	190a3a9260	Update README.md	2019-08-13 06:39:33 -04:00
William Falcon	ea76ad2b28	Update RequiredTrainerInterface.md	2019-08-13 06:39:10 -04:00
William Falcon	4f0cf1e970	Update README.md	2019-08-13 06:37:30 -04:00
William Falcon	b1bf0a8d9b	Update README.md	2019-08-12 16:15:53 -04:00
William Falcon	a78ee48d3c	release v0.4.4	2019-08-12 16:09:03 -04:00
William Falcon	5d5968033f	LR scheduler + train refactor (#103 ) * split __train up for clarity * split __train up for clarity * added lr scheduler after epoch completes	2019-08-12 16:07:42 -04:00
William Falcon	309e45e4f8	Update setup.py	2019-08-12 16:02:56 -04:00
Sidhanth Holalkere	511f7ecb9a	Support for multiple val_dataloaders (#97 ) * Added support for multiple validation dataloaders * Fix typo in README.md * Update trainer.py * Add support for multiple dataloaders * Rename dataloader_index to dataloader_i * Added warning to check val_dataloaders Added a warning to ensure that all val_dataloaders were DistributedSamplers if ddp is enabled * Updated DistributedSampler warning * Fixed typo * Added multiple val_dataloaders * Multiple val_dataloader test * Update lightning_module_template.py Added dataloader_i to validation_step parameters * Update trainer.py * Reverted template changes * Create multi_val_module.py * Update no_val_end_module.py * New MultiValModel * Rename MultiValModel to MultiValTestModel * Revert to LightningTestModel * Update test_models.py * Update trainer.py * Update test_models.py * multiple val_dataloaders in test template * Fixed flake8 warnings * Update trainer.py * Fix flake errors * Fixed Flake8 errors * Update lm_test_module.py keep this test model with a single dataset for val * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update test_models.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update RequiredTrainerInterface.md * Update RequiredTrainerInterface.md * Update test_models.py * Update trainer.py dont need the else clause, val_dataloader is either a list or none because of get_dataloaders() * Update trainer.py fixed flake errors * Update trainer.py	2019-08-12 15:23:11 -04:00
William Falcon	46e27e38aa	Create CODE_OF_CONDUCT.md (#96 )	2019-08-11 10:03:49 -04:00
William Falcon	e5805bf8ff	val and test are optional now (#95 ) * made validation step optional * added no val model * val_step can be implemented but not validation_end * added no val end model * added tests * added tests * remove class * remove class * remove class * remove class * remove class * remove class * remove class * remove class * remove class * remove class * remove class * updated docs * updated docs * updated test * updated test * updated test * updated test * updated test * updated test * updated test * updated test * updated test * fix pep8	2019-08-11 10:01:57 -04:00
Nic Eggert	996b1f9a6d	When running DDP without DistributedSampler, throw warning instead of exception (#91 )	2019-08-10 15:58:12 -04:00
Coda Phillips	c1434f0a3e	Update github url for new project template (#90 ) Previous url requested html	2019-08-10 13:35:17 -04:00
William Falcon	b3a846c6cf	release v0.4.3	2019-08-10 10:04:00 -04:00
William Falcon	73d08557ba	fix accumulated grad norm fixes #87 (#88 ) * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py	2019-08-10 08:32:45 -04:00
Lorenzo Fabbri	09d4475cc7	Update Checkpointing.md (#83 ) * Update Checkpointing.md Modified import for ModelCheckpoint. * Update Checkpointing.md	2019-08-09 15:02:36 -04:00
Rich Lewis	dd0db4aba2	docs(trainer): fix gradient clipping entry (#85 ) - replace copy and paste error - write brief description - add link to pytorch docs for specific clipping implementation - add example configuration	2019-08-09 15:02:14 -04:00
William Falcon	018b8da50e	Update issue templates	2019-08-08 14:30:55 -04:00
William Falcon	66c8ed0091	Update README.md	2019-08-08 14:27:47 -04:00
William Falcon	48f0cd0f63	release v0.4.2	2019-08-08 14:23:50 -04:00
William Falcon	0c584a6a13	updated support for 1.2.0 (#80 )	2019-08-08 14:05:03 -04:00
William Falcon	44686f74d8	release v0.4.1	2019-08-08 13:08:25 -04:00
William Falcon	011a2f3dd7	release v0.4.1	2019-08-08 13:08:02 -04:00