release v0.4.8

Expectopatronum implement #89 (#182 )
* rename validate -> evaluate; implement test logic; allow multiple test_loaders * add test_step and test_end to LightningModule * add in_test_mode to pretraining to implement case 2 (test pretrained model) * fix code style issues * LightningTestModel: add optional second test set, implement test_step and test_end * implemented test for multiple test_dataloaders; fixed typo * add two test cases for #89 * add documentation for test_step, test_end; fix computation of loss in validation_step example * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Added proper dp ddp routing calls for test mode * Update trainer.py * Update test_models.py * Update trainer.py * Update trainer.py * Update override_data_parallel.py * Update test_models.py * Update test_models.py * Update trainer.py * Update trainer.py * Update trainer.py * Update test_models.py * Update test_models.py * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * Update trainer.py * Update override_data_parallel.py * Update debug.py * Update lm_test_module.py * Update test_models.py
2026-09-12 12:40:20 +08:00 · 2019-08-31 03:05:57 -04:00 · 2019-08-30 18:56:09 -04:00 · 2019-08-30 10:56:14 -04:00 · 2019-08-28 06:41:31 -04:00 · 2019-08-27 18:07:33 -04:00
39 changed files with 1540 additions and 361 deletions
@@ -1,4 +1,7 @@
-#see https://github.com/codecov/support/wiki/Codecov-Yaml
+# see https://docs.codecov.io/docs/codecov-yaml
 # Validation check:
 # $ curl --data-binary @.codecov.yml https://codecov.io/validate
 codecov:
  notify:
    require_ci_to_pass: yes
@@ -41,4 +44,3 @@ comment:
  behavior: default  # update if exists else create new
  # branches: *
@@ -1,5 +1,10 @@
 # Contributing    
-Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!  
+Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!   
 ## One less thing to remember    
 Simplify the API as much as possible from the user perspective. Any additions or improvements should minimize things the user needs to remember.   
 For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.   
 ## Lightning Design Principles   
 We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
@@ -11,6 +11,7 @@ pip-wheel-metadata/
 test_tube_exp/
 tests/tests_tt_dir/
 tests/save_dir
 default/
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -10,17 +10,17 @@
 [![PyPI Status](https://badge.fury.io/py/pytorch-lightning.svg)](https://badge.fury.io/py/pytorch-lightning)
 [![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
 [![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
-<!-- 
+[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning)
-removed until windows install issues resolved.
+[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
-[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning) -->
+[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
 [![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
 [![Gitter](https://badges.gitter.im/PyTorch-Lightning/community.svg)](https://gitter.im/PyTorch-Lightning/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
 <!-- 
 removed until codecov badge isn't empy. likely a config error showing nothing on master.
 [![codecov](https://codecov.io/gh/Borda/pytorch-lightning/branch/master/graph/badge.svg)](https://codecov.io/gh/Borda/pytorch-lightning)
 -->
 [![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
 [![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)
 [![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
 [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
 </div>
@@ -33,15 +33,15 @@ pip install pytorch-lightning
 **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**
 ## What is it?  
-Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. It defers core training and validation logic to you and automates the rest. It guarantees tested, correct, modern best practices for the automated parts.
+Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.
 ## Starting a new project?   
 [Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     
 ## Why do I want to use lightning?
-When starting a new project the last thing you want to do is recode a training loop, multi-cluster training, 16-bit precision, early-stopping, model loading/saving, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.
+Every research project starts the same, a model, a training loop, validation loop, etc. As your research advances, you're likely to need distributed training, 16-bit precision, checkpointing, gradient accumulation, etc.   
-With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: The data and the training/validation loop logic. 
+Lightning sets up all the boilerplate state-of-the-art training for you so you can focus on the research.   
 Don't worry about training on multiple gpus or speeding up your code, lightning will do that for you!
 ---   
 ## README Table of Contents        
@@ -53,14 +53,18 @@ Don't worry about training on multiple gpus or speeding up your code, lightning
 - [Tutorials](https://github.com/williamFalcon/pytorch-lightning#tutorials)
 - [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md)    
 - [Bleeding edge install](https://github.com/williamFalcon/pytorch-lightning#bleeding-edge)   
- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)
+- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)   
 - [Asking for help](https://github.com/williamFalcon/pytorch-lightning#asking-for-help)
 - [FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
 ---   
 ## How do I do use it?   
 Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule]((https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)) which you fit using a Trainer.    
 The LightningModule defines a *system* such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier such as the example below.     
 To use lightning do 2 things:  
-1. [Define a LightningModel](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
+1. [Define a LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
 ```python
 import os
 import torch
@@ -71,10 +75,10 @@ import torchvision.transforms as transforms
 import pytorch_lightning as pl
-class CoolModel(pl.LightningModule):
+class CoolSystem(pl.LightningModule):
    def __init__(self):
-        super(CoolModel, self).__init__()
+        super(CoolSystem, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)
@@ -100,7 +104,8 @@ class CoolModel(pl.LightningModule):
    def configure_optimizers(self):
        # REQUIRED
-        return [torch.optim.Adam(self.parameters(), lr=0.02)]
+        # can return multiple optimizers and learning_rate schedulers
        return torch.optim.Adam(self.parameters(), lr=0.02)
    @pl.data_loader
    def tng_dataloader(self):
@@ -122,7 +127,7 @@ class CoolModel(pl.LightningModule):
 ```python
 from pytorch_lightning import Trainer
-model = CoolModel()
+model = CoolSystem()
 # most basic trainer, uses good defaults
 trainer = Trainer()    
@@ -161,7 +166,7 @@ You define the blue parts using the LightningModule interface:
 ![Ouverview](./docs/source/_static/overview_flat.jpg)
-```{.python}
+```python
 # what to do in the training loop
 def training_step(self, data_batch, batch_nb):
@@ -248,7 +253,7 @@ Lightning also adds a text column with all the hyperparameters for this experime
 ![tensorboard-support](./docs/source/_static/tf_tags.png)
 Simply note the path you set for the Experiment    
-``` {.python}   
+```python
 from test_tube import Experiment
 from pytorch-lightning import  Trainer
@@ -316,6 +321,7 @@ tensorboard --logdir /some/path
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)    
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 ###### Validation loop    
@@ -354,8 +360,24 @@ python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'
 - [9 key speed features in Pytorch-Lightning](https://towardsdatascience.com/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565)    
 - [SLURM, multi-node training with Lightning](https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd)     
 ---    
 ## Asking for help    
 Welcome to the Lightning community!   
 If you have any questions, feel free to:   
 1. [read the docs](https://williamfalcon.github.io/pytorch-lightning/).     
 2. [Search through the issues](https://github.com/williamFalcon/pytorch-lightning/issues?utf8=%E2%9C%93&q=my++question).      
 3. [Ask on stackoverflow](https://stackoverflow.com/questions/ask?guided=false) with the tag pytorch-lightning.   
 If no one replies to you quickly enough, feel free to post the stackoverflow link to our Gitter chat!   
 To chat with the rest of us visit our [gitter channel](https://gitter.im/PyTorch-Lightning/community?utm_source=share-link&utm_medium=link&utm_campaign=share-link)!     
 ---   
 ## FAQ    
 **How do I use Lightning for rapid research?**   
 [Here's a walk-through](https://williamfalcon.github.io/pytorch-lightning/)  
 **Why was Lightning created?**     
 Lightning has 3 goals in mind:
 1. Maximal flexibility while abstracting out the common boilerplate across research projects.   
@@ -375,8 +397,19 @@ Nope.
 Nope. Please use anaconda or miniconda.    
 **Which PyTorch versions do you support?**    
-Lightning 0.4.2+ supports PyTorch 1.2.0.    
+##### PyTorch 1.1.0       
-For PyTorch 1.1.0 install Lightning 0.4.0 with test-tube=0.6.7.6.    
+```bash    
 # install pytorch 1.1.0 using the official instructions   
 # install test-tube 0.6.7.6 which supports 1.1.0   
 pip install test-tube==0.6.7.6   
 # install latest Lightning version without upgrading deps    
 pip install -U --no-deps pytorch-lightning
 ```     
 ##### PyTorch 1.2.0   
 Install via pip as normal   
 ## Bleeding edge
 If you can't wait for the next release, install the most up to date code with:  
@@ -45,9 +45,7 @@ install:
  # directly to master instead of just PR builds (or the converse).
  - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%path%
  - pip install -U --user pip
-  - pip install "https://download.pytorch.org/whl/cu90/torch-1.1.0-cp%PIP_PYVER%-cp%PIP_PYVER%m-win_amd%PYTHON_ARCH%.whl"
+  - pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
    pip install "https://download.pytorch.org/whl/cu90/torchvision-0.3.0-cp%PIP_PYVER%-cp%PIP_PYVER%m-win_amd%PYTHON_ARCH%.whl"
  - pip install -r requirements.txt
  - pip install -r ./tests/requirements.txt
 # scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
@@ -9,20 +9,22 @@ Otherwise, to Define a Lightning Module, implement the following methods:
 **Required**:  
- [training_step](RequiredTrainerInterface.md#training_step)   
+- [training_step](RequiredTrainerInterface.md#training_step)      
- [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)
+- [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)    
- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)
+- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)    
 **Optional**:   
 - [validation_step](RequiredTrainerInterface.md#validation_step)
 - [validation_end](RequiredTrainerInterface.md#validation_end)
 - [val_dataloader](RequiredTrainerInterface.md#val_dataloader)
 - [test_dataloader](RequiredTrainerInterface.md#test_dataloader)
- [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)
+- [validation_step](RequiredTrainerInterface.md#validation_step)    
- [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)
+- [validation_end](RequiredTrainerInterface.md#validation_end) 
- [update_tng_log_metrics](RequiredTrainerInterface.md#update_tng_log_metrics)
+- [test_step](RequiredTrainerInterface.md#test_step)    
- [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)
+- [test_end](RequiredTrainerInterface.md#test_end) 
 - [val_dataloader](RequiredTrainerInterface.md#val_dataloader)    
 - [test_dataloader](RequiredTrainerInterface.md#test_dataloader)    
 - [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)    
 - [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)    
 - [update_tng_log_metrics](RequiredTrainerInterface.md#update_tng_log_metrics)    
 - [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)    
 ---
 ### Minimal example   
@@ -50,19 +52,30 @@ class CoolModel(pl.LightningModule):
        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
-        return {'loss': F.cross_entropy(y_hat, y)(y_hat, y)}
+        return {'loss': F.cross_entropy(y_hat, y)}
    def validation_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
-        return {'val_loss': F.cross_entropy(y_hat, y)(y_hat, y)}
+        return {'val_loss': F.cross_entropy(y_hat, y)}
    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}
    def test_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
        return {'test_loss': F.cross_entropy(y_hat, y)}
    def test_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
        return {'avg_test_loss': avg_loss}
    def configure_optimizers(self):
        # REQUIRED
        return [torch.optim.Adam(self.parameters(), lr=0.02)]
@@ -80,7 +93,8 @@ class CoolModel(pl.LightningModule):
    @pl.data_loader
    def test_dataloader(self):
        # OPTIONAL
-        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
+        # can also return a list of test dataloaders
        return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
 ---   
 ### How do these methods fit into the broader training?     
@@ -136,7 +150,17 @@ def training_step(self, data_batch, batch_nb):
    # return a dict
    return output
-```
+```    
 If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.    
 ``` {.python}
 # Multiple optimizers (ie: GANs)     
 def training_step(self, data_batch, batch_nb, optimizer_idx):
    if optimizer_idx == 0:
        # do training_step with encoder
    if optimizer_idx == 1:
        # do training_step with decoder    
 ```    
 --- 
 ### tng_dataloader 
@@ -175,9 +199,15 @@ def configure_optimizers(self)
 Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
 Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
 **Note:** If you use multiple optimizers, training_step will have an additional ```optimizer_idx``` parameter.    
-##### Return
+
-List or Tuple - List of optimizers with an optional second list of learning-rate schedulers
+
 ##### Return    
 Return any of these 3 options:   
 Single optimizer   
 List or Tuple - List of optimizers    
 Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers
 **Example**
@@ -185,29 +215,42 @@ List or Tuple - List of optimizers with an optional second list of learning-rate
 # most cases
 def configure_optimizers(self):
    opt = Adam(self.parameters(), lr=0.01)
-    return [opt]
+    return opt
 # multiple optimizer case (eg: GAN)
 def configure_optimizers(self):
    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
    return generator_opt, disriminator_opt
-# gan example, with scheduler for discriminator
+# example with learning_rate schedulers  
 def configure_optimizers(self):
    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
    return [generator_opt, disriminator_opt], [discriminator_sched]
-```
+```  
 If you need to control how often those optimizers step or override the default .step() schedule, override 
 the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step) hook.   
 ## Optional Methods    
 ### validation_step
 ``` {.python}
-def validation_step(self, data_batch, batch_nb, dataloader_i)
+# if you have one val dataloader:
 def validation_step(self, data_batch, batch_nb)   
 # if you have multiple val dataloaders:  
 def validation_step(self, data_batch, batch_nb, dataloader_idx)
 ```
 **OPTIONAL**    
 If you don't need to validate you don't need to implement this method.    
-In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes, calculate accuracy, or save example outputs (using self.experiment or whatever you want). Really, anything you want.       
+In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
-This is most likely the same as your training_step. But unlike training step, the outputs from here will go to validation_end for collation.
+The dict you return here will be available in the `validation_end` method. 
 **Params**   
@@ -221,18 +264,24 @@ This is most likely the same as your training_step. But unlike training step, th
 | Return  | description  | optional |
 |---|---|---|   
-|  dict | Dict of OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
+|  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
 **Example**
 ``` {.python}
 # CASE 1: A single validation dataset
 def validation_step(self, data_batch, batch_nb):
-    x, y, z = data_batch
+    x, y = data_batch
    # implement your own
    out = self.forward(x)
-    loss = self.loss(out, x)
+    loss = self.loss(out, y)
    # log 6 example images
    # or generated text... or whatever
    sample_imgs = x[:6]
    grid = torchvision.utils.make_grid(sample_imgs)
    self.experiment.add_image('example_images', grid, 0) 
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
@@ -304,6 +353,119 @@ def validation_end(self, outputs):
    return tqdm_dic
 ```
 ### test_step
 ``` {.python}
 # if you have one test dataloader:
 def test_step(self, data_batch, batch_nb)   
 # if you have multiple test dataloaders:  
 def test_step(self, data_batch, batch_nb, dataloader_idx)
 ```
 **OPTIONAL**    
 If you don't need to test you don't need to implement this method.    
 In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
 The dict you return here will be available in the `test_end` method. 
 This function is used when you execute `trainer.test()`.
 **Params**   
 | Param  | description  |
 |---|---|
 |  data_batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
 |  dataloader_i | Integer displaying which dataloader this is (only if multiple test datasets used)  |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
 **Example**
 ``` {.python}
 # CASE 1: A single test dataset
 def test_step(self, data_batch, batch_nb):
    x, y = data_batch
    # implement your own
    out = self.forward(x)
    loss = self.loss(out, y)
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    # all optional...
    # return whatever you need for the collation function test_end
    output = OrderedDict({
        'test_loss': loss_test,
        'test_acc': torch.tensor(test_acc), # everything must be a tensor
    })
    # return an optional dict
    return output
 ```   
 If you pass in multiple test datasets, test_step will have an additional argument.
 ```python
 # CASE 2: multiple test datasets
 def test_step(self, data_batch, batch_nb, dataset_idx):
    # dataset_idx tells you which dataset this is.   
 ```   
 The ```dataset_idx``` corresponds to the order of datasets returned in ```test_dataloader```.    
 --- 
 ### test_end
 ``` {.python}
 def test_end(self, outputs)
 ```   
 If you didn't define a test_step, this won't be called.       
 Called at the end of the test step with the output of each test_step.  Called once per test dataset.   
 The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
 **Params**    
 | Param  | description  |
 |---|---|
 |  outputs | List of outputs you defined test_step |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict of OrderedDict with metrics to display in progress bar | Y |
 **Example**
 ``` {.python}
 def test_end(self, outputs):
    """
    Called at the end of test to aggregate outputs
    :param outputs: list of individual outputs of each test step
    :return:
    """
    test_loss_mean = 0
    test_acc_mean = 0
    for output in outputs:
        test_loss_mean += output['test_loss']
        test_acc_mean += output['test_acc']
    test_loss_mean /= len(outputs)
    test_acc_mean /= len(outputs)
    tqdm_dic = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
    return tqdm_dic
 ```
 --- 
 ### on_save_checkpoint 
@@ -351,7 +513,7 @@ def on_load_checkpoint(self, checkpoint):
 ``` {.python}
@pl.data_loader
-def tng_dataloader(self)
+def val_dataloader(self)
 ```
 **OPTIONAL**    
 If you don't need a validation dataset and a validation_step, you don't need to implement this method.    
@@ -423,7 +585,7 @@ def test_dataloader(self):
 def update_tng_log_metrics(self, logs)
 ```
 Called by lightning right before it logs metrics for this batch.
-This is a chance to ammend or add to the metrics about to be logged.
+This is a chance to amend or add to the metrics about to be logged.
 ##### Return
 Dict 
@@ -10,7 +10,7 @@ Current dtype
 ---
 #### experiment    
-An instance of test-tube Experiment which you can use to log anything for tensorboarX.   
+An instance of test-tube Experiment which you can use to log anything for tensorboard (subclass of [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html)).   
 ```{.python}
 self.experiment.add_embedding(...)   
 self.experiment.log({'val_loss': 0.9})   
@@ -1,4 +1,4 @@
-i Lightning can automate saving and loading checkpoints.
+Lightning can automate saving and loading checkpoints.
 ---
 ### Model saving
@@ -38,6 +38,7 @@ trainer.fit(model)
 ```
 The trainer restores:  
 - global_step    
 - current_epoch    
 - All optimizers    
@@ -5,7 +5,7 @@ Lighting offers a few options for logging information about model, gpu usage, et
 #### Display metrics in progress bar 
 ``` {.python}
 # DEFAULT
-trainer = Trainer(progress_bar=True)
+trainer = Trainer(show_progress_bar=True)
 ```
 ---
@@ -54,7 +54,10 @@ trainer = Trainer(track_grad_norm=2)
 ---
 #### Set how much of the training set to check
-If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag
+If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
 train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(train_percent_check=1.0)
@@ -5,8 +5,6 @@ Below are all the things lightning automates for you in the validation loop.
 Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don't have to wait until a full epoch to catch possible validation issues.
 ---
 #### Check validation every n epochs
 If you have a small dataset you might want to check validation every n epochs
@@ -18,6 +16,9 @@ trainer = Trainer(check_val_every_n_epoch=1)
 ---
 #### Set how much of the validation set to check 
 If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
 val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_percent_check=1.0)
@@ -29,6 +30,9 @@ trainer = Trainer(val_percent_check=0.1)
 ---
 #### Set how much of the test set to check 
 If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
 test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(test_percent_check=1.0)
@@ -54,4 +58,6 @@ Lightning runs a few steps of validation in the beginning of training. This avoi
 ``` {.python}
 # DEFAULT
 trainer = Trainer(nb_sanity_val_steps=5)
-```
+```
 You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.
@@ -23,6 +23,9 @@ trainer = Trainer(track_grad_norm=2)
 ---
 #### Make model overfit on subset of data
 A useful debugging trick is to make your model overfit a tiny fraction of the data.
 setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
 ``` {.python}
 # DEFAULT don't overfit (ie: normal training)
 trainer = Trainer(overfit_pct=0.0)
@@ -92,8 +92,28 @@ def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
        if batch_nb % 4 == 0 :
            optimizer.step()
            optimizer.zero_grad()    
    # ...
    # add as many optimizers as you want 
 ```
 This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:   
 ```python
 # learning rate warm-up
 def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
    # warm up lr
    if self.trainer.global_step < 500:
        lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
        for pg in optimizer.param_groups:
            pg['lr'] = lr_scale * self.hparams.learning_rate
    # update params
    optimizer.step()
    optimizer.zero_grad() 
 ```
 ---
 #### on_before_zero_grad
 Called in the training loop after taking an optimizer step and before zeroing grads.
@@ -68,6 +68,7 @@ But of course the fun is in all the advanced things it can do:
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 **Validation loop**    
@@ -40,7 +40,7 @@ The main function should have 3 arguments:
 - slurm_manager: Slurm cluster manager object (can be None)
 - dict: for you to return any values you want (useful in meta-learning, otherwise set to _)    
-```{}
+```python
 def main(hparams, cluster, results_dict):
    """
    Main training routine specific for this project
@@ -1,12 +1,63 @@
 ###### New project Quick Start    
-To start a new project you define two files, a LightningModule and a Trainer file.    
+To start a new project define two files, a LightningModule and a Trainer file.    
 To illustrate Lightning power and simplicity, here's an example of a typical research flow.    
-A separate trainer file allows to run many LightningModules. Each LightningModule has the core
+###### Case 1: BERT    
-logic to a particular research project.    
+Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
 You would define a single LightningModule and use flags to switch between your different ideas.   
 ```python
 class BERT(pl.LightningModule):
    def __init__(self, model_name, task):
        self.task = task
        if model_name == 'transformer':
            self.net = Transformer()
        elif model_name == 'my_cool_version':
            self.net = MyCoolVersion()
    def training_step(self, batch, batch_nb):
        if self.task == 'standard_bert':
            # do standard bert training with self.net...
            # return loss
        if self.task == 'my_cool_task':
            # do my own version with self.net
            # return loss
 ```   
-For example, one lightningModule could be an image classifier, the other
+###### Case 2: COOLER NOT BERT    
-one could be a seq-2-seq model, both (optionally) ran by the same trainer file.
+But if you wanted to try something **completely** different, you'd define a new module for that.    
 ```python
 class CoolerNotBERT(pl.LightningModule):
    def __init__(self):
        self.net = ...
    def training_step(self, batch, batch_nb):
        # do some other cool task
        # return loss   
 ```   
 ###### Rapid research flow    
 Then you could do rapid research by switching between these two and using the same trainer.   
 ```python
 if use_bert:
    model = BERT()
 else:
    model = CoolerNotBERT()
 trainer = Trainer(gpus=[0, 1, 2, 3], use_amp=True)
 trainer.fit(model)
 ```
 Notice a few things about this flow:   
 1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
 2. You get free GPU and 16-bit support without writing any of that code in your model.   
 3. You also get all of the capabilities below (without coding or testing yourself).     
 ---    
 ###### Templates 
 1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
 2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_cpu_template.py) 
@@ -75,6 +126,7 @@ one could be a seq-2-seq model, both (optionally) ran by the same trainer file.
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 ###### Validation loop    
@@ -161,9 +161,9 @@ class LightningTemplateModel(LightningModule):
            # reduce manually when using dp
            val_acc = output['val_acc']
            if self.trainer.use_dp:
-                val_acc_mean = torch.mean(val_acc)
+                val_acc = torch.mean(val_acc)
-            val_acc_mean += val_acc_mean
+            val_acc_mean += val_acc
        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
@@ -189,16 +189,13 @@ class LightningTemplateModel(LightningModule):
        dataset = MNIST(root=self.hparams.data_root, train=train,
                        transform=transform, download=True)
-        # when using multi-node we need to add the datasampler
+        # when using multi-node (ddp) we need to add the datasampler
        train_sampler = None
        batch_size = self.hparams.batch_size
-        try:
+        if self.use_ddp:
-            if self.on_gpu:
+            train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
-                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
+            batch_size = batch_size // self.trainer.world_size  # scale batch size
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
            pass
        should_shuffle = train_sampler is None
        loader = DataLoader(
@@ -110,5 +110,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -110,5 +110,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -109,5 +109,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -0,0 +1,178 @@
 """
 To run this template just do:  
 python gan.py   
 After a few epochs, launch tensorboard to see the images being generated at every batch.   
 tensorboard --logdir default
 """
 from argparse import ArgumentParser
 import os
 import numpy as np
 import torchvision
 import torchvision.transforms as transforms
 from torchvision.datasets import MNIST
 from torch.utils.data import DataLoader
 import torch.nn as nn
 import torch.nn.functional as F
 import torch
 import pytorch_lightning as pl
 from test_tube import Experiment
 class Generator(nn.Module):
    def __init__(self, latent_dim, img_shape):
        super(Generator, self).__init__()
        self.img_shape = img_shape
        def block(in_feat, out_feat, normalize=True):
            layers = [nn.Linear(in_feat, out_feat)]
            if normalize:
                layers.append(nn.BatchNorm1d(out_feat, 0.8))
            layers.append(nn.LeakyReLU(0.2, inplace=True))
            return layers
        self.model = nn.Sequential(
            *block(latent_dim, 128, normalize=False),
            *block(128, 256),
            *block(256, 512),
            *block(512, 1024),
            nn.Linear(1024, int(np.prod(img_shape))),
            nn.Tanh()
        )
    def forward(self, z):
        img = self.model(z)
        img = img.view(img.size(0), *self.img_shape)
        return img
 class Discriminator(nn.Module):
    def __init__(self, img_shape):
        super(Discriminator, self).__init__()
        self.model = nn.Sequential(
            nn.Linear(int(np.prod(img_shape)), 512),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(256, 1),
            nn.Sigmoid(),
        )
    def forward(self, img):
        img_flat = img.view(img.size(0), -1)
        validity = self.model(img_flat)
        return validity
 class GAN(pl.LightningModule):
    def __init__(self, hparams):
        super(GAN, self).__init__()
        self.hparams = hparams
        # networks
        mnist_shape = (1, 28, 28)
        self.generator = Generator(latent_dim=hparams.latent_dim, img_shape=mnist_shape)
        self.discriminator = Discriminator(img_shape=mnist_shape)
        # cache for generated images
        self.generated_imgs = None
    def forward(self, z):
        return self.generator(z)
    def adversarial_loss(self, y_hat, y):
        return F.binary_cross_entropy(y_hat, y)
    def training_step(self, batch, batch_nb, optimizer_i):
        imgs, _ = batch
        # train generator
        if optimizer_i == 0:
            # sample noise
            z = torch.randn(imgs.shape[0], self.hparams.latent_dim)
            # match gpu device (or keep as cpu)
            if self.on_gpu:
                z = z.cuda(imgs.device.index)
            # generate images
            self.generated_imgs = self.forward(z)
            # log sampled images
            sample_imgs = self.generated_imgs[:6]
            grid = torchvision.utils.make_grid(sample_imgs)
            self.experiment.add_image('generated_images', grid, 0)
            # ground truth result (ie: all fake)
            valid = torch.ones(imgs.size(0), 1)
            # adversarial loss is binary cross-entropy
            g_loss = self.adversarial_loss(self.discriminator(self.generated_imgs), valid)
            return g_loss
        # train discriminator
        if optimizer_i == 1:
            # Measure discriminator's ability to classify real from generated samples
            # how well can it label as real?
            valid = torch.ones(imgs.size(0), 1)
            real_loss = self.adversarial_loss(self.discriminator(imgs), valid)
            # how well can it label as fake?
            fake = torch.zeros(imgs.size(0), 1)
            fake_loss = self.adversarial_loss(self.discriminator(self.generated_imgs.detach()), fake)
            # discriminator loss is the average of these
            d_loss = (real_loss + fake_loss) / 2
            return d_loss
    def configure_optimizers(self):
        lr = self.hparams.lr
        b1 = self.hparams.b1
        b2 = self.hparams.b2
        opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
        opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))
        return [opt_g, opt_d], []
    @pl.data_loader
    def tng_dataloader(self):
        transform = transforms.Compose([transforms.ToTensor(),
                                        transforms.Normalize([0.5], [0.5])])
        dataset = MNIST(os.getcwd(), train=True, download=True, transform=transform)
        return DataLoader(dataset, batch_size=self.hparams.batch_size)
 def main(hparams):
    # save tensorboard logs
    exp = Experiment(save_dir=os.getcwd())
    # init model
    model = GAN(hparams)
    # fit trainer on CPU
    trainer = pl.Trainer(experiment=exp, max_nb_epochs=200)
    trainer.fit(model)
 if __name__ == '__main__':
    parser = ArgumentParser()
    parser.add_argument("--batch_size", type=int, default=64, help="size of the batches")
    parser.add_argument("--lr", type=float, default=0.0002, help="adam: learning rate")
    parser.add_argument("--b1", type=float, default=0.5, help="adam: decay of first order momentum of gradient")
    parser.add_argument("--b2", type=float, default=0.999, help="adam: decay of first order momentum of gradient")
    parser.add_argument("--latent_dim", type=int, default=100, help="dimensionality of the latent space")
    hparams = parser.parse_args()
    main(hparams)
@@ -2,9 +2,15 @@ site_name: PyTorch lightning Documentation
 theme:
  name: 'material'
 docs_dir: docs
 repo_name: 'williamFalcon/pytorch-lightning'
 repo_url: https://github.com/williamFalcon/pytorch-lightning
 site_dir: 'site'
 site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
 dev_addr: '0.0.0.0:8000'
 #google_analytics: ['UA-aasd', 'sitename']
 markdown_extensions:
  - codehilite:
      guess_lang: false
      linenums: true
@@ -1,6 +1,7 @@
-from .pt_callbacks import EarlyStopping, ModelCheckpoint
+from .pt_callbacks import EarlyStopping, ModelCheckpoint, GradientAccumulationScheduler
 __all__ = [
    'EarlyStopping',
    'ModelCheckpoint',
    'GradientAccumulationScheduler',
 ]
@@ -1,5 +1,6 @@
 import os
 import shutil
 import warnings
 import numpy as np
@@ -254,6 +255,37 @@ class ModelCheckpoint(Callback):
                self.save_model(filepath, overwrite=False)
 class GradientAccumulationScheduler(Callback):
    """Change gradient accumulation factor according to scheduling.
    # Arguments
        scheduling: dict, scheduling in format {epoch: accumulation_factor}
    """
    def __init__(self, scheduling: dict):
        if scheduling == {}:  # empty dict error
            raise TypeError("Empty dict cannot be interpreted correct")
        for key in scheduling.keys():
            if not isinstance(key, int) or not isinstance(scheduling[key], int):
                raise TypeError("All epoches and accumulation factor must be integers")
        minimal_epoch = min(scheduling.keys())
        if minimal_epoch < 1:
            msg = f"Epochs indexing from 1, epoch {minimal_epoch} cannot be interpreted correct"
            raise IndexError(msg)
        elif minimal_epoch != 1:  # if user didnt define first epoch accumulation factor
            scheduling.update({1: 1})
        self.scheduling = scheduling
        self.epochs = sorted(scheduling.keys())
    def on_epoch_begin(self, epoch, trainer):
        epoch += 1  # indexing epochs from 1
        for i in reversed(range(len(self.epochs))):
            if epoch >= self.epochs[i]:
                trainer.accumulate_grad_batches = self.scheduling.get(self.epochs[i])
                break
 if __name__ == '__main__':
    c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
    losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
@@ -12,13 +12,15 @@ import torch
 from torch.utils.data.distributed import DistributedSampler
 import torch.multiprocessing as mp
 import torch.distributed as dist
 from torch.optim.optimizer import Optimizer
 from pytorch_lightning.root_module.root_module import LightningModule
 from pytorch_lightning.root_module.memory import get_gpu_memory_map
 from pytorch_lightning.root_module.model_saving import TrainerIO
 from pytorch_lightning.pt_overrides.override_data_parallel import (
    LightningDistributedDataParallel, LightningDataParallel)
 from pytorch_lightning.callbacks import GradientAccumulationScheduler
 from pytorch_lightning.utilities.debugging import MisconfigurationException
 import pdb
 try:
    from apex import amp
@@ -60,7 +62,7 @@ class Trainer(TrainerIO):
                 current_gpu_name=0,
                 nb_gpu_nodes=1,
                 gpus=None,
-                 progress_bar=True,
+                 show_progress_bar=True,
                 overfit_pct=0.0,
                 track_grad_norm=-1,
                 check_val_every_n_epoch=1,
@@ -71,7 +73,7 @@ class Trainer(TrainerIO):
                 train_percent_check=1.0,
                 val_percent_check=1.0,
                 test_percent_check=1.0,
-                 val_check_interval=0.95,
+                 val_check_interval=1.0,
                 log_save_interval=100,
                 add_log_row_interval=10,
                 distributed_backend='dp',
@@ -91,7 +93,7 @@ class Trainer(TrainerIO):
        :param current_gpu_name:
        :param nb_gpu_nodes:
        :param gpus:
-        :param progress_bar:
+        :param show_progress_bar:
        :param overfit_pct:
        :param track_grad_norm:
        :param check_val_every_n_epoch:
@@ -121,7 +123,6 @@ class Trainer(TrainerIO):
        self.track_grad_norm = track_grad_norm
        self.fast_dev_run = fast_dev_run
        self.on_gpu = gpus is not None and torch.cuda.is_available()
        self.progress_bar = progress_bar
        self.experiment = experiment
        self.exp_save_path = None
        if self.experiment is not None:
@@ -138,7 +139,13 @@ class Trainer(TrainerIO):
        self.early_stop = early_stop_callback
        self.model = None
        self.max_nb_epochs = max_nb_epochs
-        self.accumulate_grad_batches = accumulate_grad_batches
+        if isinstance(accumulate_grad_batches, dict):
            self.accumulation_scheduler = GradientAccumulationScheduler(accumulate_grad_batches)
        elif isinstance(accumulate_grad_batches, int):
            schedule = {1: accumulate_grad_batches}
            self.accumulation_scheduler = GradientAccumulationScheduler(schedule)
        else:
            raise TypeError("Gradient accumulation supports only int and dict types")
        self.early_stop_callback = early_stop_callback
        self.min_nb_epochs = min_nb_epochs
        self.nb_sanity_val_steps = nb_sanity_val_steps
@@ -151,6 +158,7 @@ class Trainer(TrainerIO):
        self.use_ddp = False
        self.use_dp = False
        self.single_gpu = False
        self.testing = False
        # training bookeeping
        self.total_batch_nb = 0
@@ -158,9 +166,9 @@ class Trainer(TrainerIO):
        self.avg_loss = 0
        self.batch_nb = 0
        self.tqdm_metrics = {}
-        self.nb_val_batches = None
+        self.nb_val_batches = 0
-        self.nb_tng_batches = None
+        self.nb_tng_batches = 0
-        self.nb_test_batches = None
+        self.nb_test_batches = 0
        # gpus come in as a string.
        # if gpus = -1 then use all available devices
@@ -222,11 +230,14 @@ class Trainer(TrainerIO):
        # training state
        self.optimizers = None
        self.prog_bar = None
        self.global_step = 0
        self.current_epoch = 0
        self.total_batches = 0
        # can't init progress bar here because starting a new process
        # means the prog_bar won't survive pickling
        self.show_progress_bar = show_progress_bar
        # logging
        self.log_save_interval = log_save_interval
        self.val_check_interval = val_check_interval
@@ -324,7 +335,7 @@ class Trainer(TrainerIO):
    @property
    def __tng_tqdm_dic(self):
        tqdm_dic = {
-            'tng_loss': '{0:.3f}'.format(self.avg_loss),
+            'loss': '{0:.3f}'.format(self.avg_loss),
            'epoch': '{}'.format(self.current_epoch),
            'batch_nb': '{}'.format(self.batch_nb),
        }
@@ -355,20 +366,20 @@ class Trainer(TrainerIO):
        # determine number of validation batches
        # val datasets could be none, 1 or 2+
        self.nb_val_batches = 0
        if self.val_dataloader is not None:
            self.nb_val_batches = sum(len(dataloader) for dataloader in self.val_dataloader)
-
+            self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
-        self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
+            self.nb_val_batches = max(1, self.nb_val_batches)
        self.nb_val_batches = max(1, self.nb_val_batches)
        self.nb_val_batches = self.nb_val_batches
        # determine number of test batches
-        self.nb_test_batches = len(self.test_dataloader) if self.test_dataloader is not None else 0
+        if self.test_dataloader is not None:
-        self.nb_test_batches = int(self.nb_test_batches * self.test_percent_check)
+            self.nb_test_batches = sum(len(dataloader) for dataloader in self.test_dataloader)
            self.nb_test_batches = int(self.nb_test_batches * self.test_percent_check)
            self.nb_test_batches = max(1, self.nb_test_batches)
        # determine when to check validation
        self.val_check_batch = int(self.nb_tng_batches * self.val_check_interval)
        self.val_check_batch = max(1, self.val_check_batch)
    def __add_tqdm_metrics(self, metrics):
        for k, v in metrics.items():
@@ -377,41 +388,45 @@ class Trainer(TrainerIO):
            self.tqdm_metrics[k] = v
-    def __validation_forward(self, model, data_batch, batch_i, dataloader_i):
+    def __evaluation_forward(self, model, data_batch, batch_i, dataloader_i, test=False):
        # make dataloader_i arg in validation_step optional
        args = [data_batch, batch_i]
-        if len(self.val_dataloader) > 1:
+
        if test and len(self.test_dataloader) > 1:
            args.append(dataloader_i)
-        if self.use_ddp:
+        elif len(self.val_dataloader) > 1:
            args.append(dataloader_i)
        # handle DP, DDP forward
        if self.use_ddp or self.use_dp:
            output = model(*args)
-        elif self.use_dp:
+            return output
-            output = model(*args)
+
-        elif self.single_gpu:
+        # CPU, single GPU
-            # put inputs on gpu manually
+        if self.single_gpu:
            # for single GPU put inputs on gpu manually
            gpu_id = self.data_parallel_device_ids[0]
-            for i, x in enumerate(data_batch):
+            data_batch = self.transfer_batch_to_gpu(data_batch, gpu_id)
-                if isinstance(x, torch.Tensor):
+            args[0] = data_batch
                    data_batch[i] = x.cuda(gpu_id)
            # do non dp, ddp step
            output = model.validation_step(*args)
        if test:
            output = model.test_step(*args)
        else:
            # CPU
            output = model.validation_step(*args)
        return output
-    def validate(self, model, dataloader, max_batches, dataloader_i):
+    def evaluate(self, model, dataloader, max_batches, dataloader_i, test=False):
        """
-        Run validation code
+        Run evaluation code
        :param model: PT model
        :param dataloader: PT dataloader
        :param max_batches: Scalar
        :param dataloader_i:
        :param test: boolean
        :return:
        """
        # enable eval mode
        model.zero_grad()
        model.eval()
@@ -428,29 +443,31 @@ class Trainer(TrainerIO):
            if data_batch is None:  # pragma: no cover
                continue
-            # stop short when on fast dev run
+            # stop short when on fast_dev_run (sets max_batch=1)
-            if max_batches is not None and batch_i >= max_batches:
+            if batch_i >= max_batches:
                break
            # -----------------
-            # RUN VALIDATION STEP
+            # RUN EVALUATION STEP
            # -----------------
-            output = self.__validation_forward(model, data_batch, batch_i, dataloader_i)
+            output = self.__evaluation_forward(model, data_batch, batch_i, dataloader_i,
                                               test)
            # track outputs for collation
            outputs.append(output)
            # batch done
-            if self.progress_bar and self.prog_bar is not None:
+            if self.show_progress_bar:
-                self.prog_bar.update(1)
+                self.progress_bar.update(1)
        eval_results = {}
        # give model a chance to do something with the outputs (and method defined)
-        val_results = {}
+        model = self.__get_model()
-        if self.__is_overriden('validation_end'):
+        if test and self.__is_overriden('test_end'):
-            if self.data_parallel:
+            eval_results = model.test_end(outputs)
-                val_results = model.module.validation_end(outputs)
+        elif self.__is_overriden('validation_end'):
-            else:
+            eval_results = model.validation_end(outputs)
                val_results = model.validation_end(outputs)
        # enable train mode again
        model.train()
@@ -458,7 +475,7 @@ class Trainer(TrainerIO):
        # enable gradients to save memory
        torch.set_grad_enabled(True)
-        return val_results
+        return eval_results
    def get_dataloaders(self, model):
        """
@@ -466,62 +483,89 @@ class Trainer(TrainerIO):
        :param model:
        :return:
        """
        self.tng_dataloader = model.tng_dataloader
        self.tng_dataloader = model.tng_dataloader
        self.test_dataloader = model.test_dataloader
        self.val_dataloader = model.val_dataloader
        # handle returning an actual dataloader instead of a list of loaders
        have_test_loaders = self.test_dataloader is not None
        if have_test_loaders and not isinstance(self.test_dataloader, list):
            self.test_dataloader = [self.test_dataloader]
        have_val_loaders = self.val_dataloader is not None
        if have_val_loaders and not isinstance(self.val_dataloader, list):
            self.val_dataloader = [self.val_dataloader]
        if self.use_ddp and not isinstance(self.tng_dataloader.sampler, DistributedSampler):
            msg = """
-You're using multiple gpus and multiple nodes without using a DistributedSampler
+            You're using multiple gpus and multiple nodes without using a DistributedSampler
-to assign a subset of your data to each process. To silence this warning, pass a
+            to assign a subset of your data to each process. To silence this warning, pass a
-DistributedSampler to your DataLoader.
+            DistributedSampler to your DataLoader.
-ie: this:
+            ie: this:
-dataset = myDataset()
+            dataset = myDataset()
-dataloader = Dataloader(dataset)
+            dataloader = Dataloader(dataset)
-becomes:
+            becomes:
-dataset = myDataset()
+            dataset = myDataset()
-dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+            dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-dataloader = Dataloader(dataset, sampler=dist_sampler)
+            dataloader = Dataloader(dataset, sampler=dist_sampler)
-If you want each process to load the full dataset, ignore this warning.
+            If you want each process to load the full dataset, ignore this warning.
-"""
+            """
            warnings.warn(msg)
-        if self.use_ddp and\
+        if self.use_ddp and self.val_dataloader is not None:
-                not all(isinstance(dataloader, DistributedSampler)
+            for dataloader in self.val_dataloader:
-                        for dataloader in self.val_dataloader):
+                if not isinstance(dataloader, DistributedSampler):
-            msg = """
+                    msg = """
-You're val_dataloader(s) are not all DistributedSamplers.
+                    Your val_dataloader(s) are not all DistributedSamplers.
-You're using multiple gpus and multiple nodes without using a DistributedSampler
+                    You're using multiple gpus and multiple nodes without using a DistributedSampler
-to assign a subset of your data to each process. To silence this warning, pass a
+                    to assign a subset of your data to each process. To silence this warning, pass a
-DistributedSampler to your DataLoader.
+                    DistributedSampler to your DataLoader.
-ie: this:
+                    ie: this:
-dataset = myDataset()
+                    dataset = myDataset()
-dataloader = Dataloader(dataset)
+                    dataloader = Dataloader(dataset)
-becomes:
+                    becomes:
-dataset = myDataset()
+                    dataset = myDataset()
-dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+                    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-dataloader = Dataloader(dataset, sampler=dist_sampler)
+                    dataloader = Dataloader(dataset, sampler=dist_sampler)
-If you want each process to load the full dataset, ignore this warning.
+                    If you want each process to load the full dataset, ignore this warning.
-"""
+                    """
-            warnings.warn(msg)
+                    warnings.warn(msg)
                    break
        if self.use_ddp and self.test_dataloader is not None:
            for dataloader in self.test_dataloader:
                if not isinstance(dataloader, DistributedSampler):
                    msg = """
                    Your test_dataloader(s) are not all DistributedSamplers.
                    You're using multiple gpus and multiple nodes without using a DistributedSampler
                    to assign a subset of your data to each process. To silence this warning, pass a
                    DistributedSampler to your DataLoader.
                    ie: this:
                    dataset = myDataset()
                    dataloader = Dataloader(dataset)
                    becomes:
                    dataset = myDataset()
                    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
                    dataloader = Dataloader(dataset, sampler=dist_sampler)
                    If you want each process to load the full dataset, ignore this warning.
                    """
                    warnings.warn(msg)
                    break
    # -----------------------------
    # MODEL TRAINING
    # -----------------------------
    def fit(self, model):
        # when using multi-node or DDP within a node start each module in a separate process
        if self.use_ddp:
            # must copy only the meta of the exp so it survives pickle/unpickle
@@ -533,12 +577,15 @@ If you want each process to load the full dataset, ignore this warning.
                task = int(os.environ['SLURM_LOCALID'])
                self.ddp_train(task, model)
            else:
-                msg = """
+                nb_gpus = self.nb_requested_gpus
-You requested %(nb_gpus)s GPUs but launched %(nb_tasks)s slurm tasks.
+                nb_tasks = self.nb_slurm_tasks
-We will launch %(nb_gpus)s processes for you.
+                msg = f"""
-We recommend you let slurm manage the processes by setting: --ntasks-per-node=%(nb_gpus)s
+                You requested {nb_gpus}s GPUs but launched {nb_tasks}s slurm tasks.
-If you're not using SLURM, ignore this message!
+                We will launch {nb_gpus}s processes for you.
-""" % {'nb_gpus': self.nb_requested_gpus, 'nb_tasks': self.nb_slurm_tasks}
+                We recommend you let slurm manage the processes by setting:
                --ntasks-per-node={nb_gpus}s
                If you're not using SLURM, ignore this message!
                """
                warnings.warn(msg)
                mp.spawn(self.ddp_train, nprocs=len(self.data_parallel_device_ids), args=(model, ))
@@ -559,9 +606,7 @@ If you're not using SLURM, ignore this message!
            # CHOOSE OPTIMIZER
            # allow for lr schedulers as well
-            self.optimizers = model.configure_optimizers()
+            self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
            if len(self.optimizers) == 2:
                self.optimizers, self.lr_schedulers = self.optimizers
            self.__run_pretrain_routine(model)
@@ -569,12 +614,25 @@ If you're not using SLURM, ignore this message!
        # used for testing or when we need to know that training succeeded
        return 1
    def init_optimizers(self, optimizers):
        # single optimizer
        if isinstance(optimizers, Optimizer):
            return [optimizers], []
        # two lists
        elif len(optimizers) == 2 and isinstance(optimizers[0], list):
            optimizers, lr_schedulers = optimizers
            return optimizers, lr_schedulers
        # single list or tuple
        elif isinstance(optimizers, list) or isinstance(optimizers, tuple):
            return optimizers, []
    def __single_gpu_train(self, model):
        # CHOOSE OPTIMIZER
        # allow for lr schedulers as well
-        self.optimizers = model.configure_optimizers()
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
        if len(self.optimizers) == 2:
            self.optimizers, self.lr_schedulers = self.optimizers
        model.cuda(self.data_parallel_device_ids[0])
@@ -591,20 +649,18 @@ If you're not using SLURM, ignore this message!
        # CHOOSE OPTIMIZER
        # allow for lr schedulers as well
-        self.optimizers = model.configure_optimizers()
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
        if len(self.optimizers) == 2:
            self.optimizers, self.lr_schedulers = self.optimizers
        model.cuda(self.data_parallel_device_ids[0])
        # check for this bug (amp + dp + !01 doesn't work)
        # https://github.com/NVIDIA/apex/issues/227
        if self.use_dp and self.use_amp:
-            m = """
+            m = f"""
-Amp level %r with DataParallel is not supported.
+            Amp level {self.amp_level} with DataParallel is not supported.
-See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227.
+            See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227.
-We recommend you switch to ddp if you want to use amp
+            We recommend you switch to ddp if you want to use amp
-""" % self.amp_level
+            """
            raise MisconfigurationException(m)
        model = LightningDataParallel(model, device_ids=self.data_parallel_device_ids)
@@ -634,7 +690,7 @@ We recommend you switch to ddp if you want to use amp
            self.experiment = self.experiment.get_non_ddp_exp()
        # show progbar only on prog_rank 0
-        self.prog_bar = self.prog_bar and self.node_rank == 0 and gpu_nb == 0
+        self.show_progress_bar = self.show_progress_bar and self.node_rank == 0 and gpu_nb == 0
        # determine which process we are and world size
        self.proc_rank = self.node_rank * len(self.data_parallel_device_ids) + gpu_nb
@@ -651,9 +707,7 @@ We recommend you switch to ddp if you want to use amp
        # CHOOSE OPTIMIZER
        # allow for lr schedulers as well
-        self.optimizers = model.configure_optimizers()
+        self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
        if len(self.optimizers) == 2:
            self.optimizers, self.lr_schedulers = self.optimizers
        # MODEL
        # copy model to each gpu
@@ -723,10 +777,15 @@ We recommend you switch to ddp if you want to use amp
        if self.data_parallel:
            ref_model = model.module
        # give model convenience properties
        ref_model.trainer = self
        # set local properties on the model
        ref_model.on_gpu = self.on_gpu
        ref_model.use_dp = self.use_dp
        ref_model.use_ddp = self.use_ddp
        ref_model.use_amp = self.use_amp
        ref_model.testing = self.testing
        # transfer data loaders from model
        self.get_dataloaders(ref_model)
@@ -738,15 +797,13 @@ We recommend you switch to ddp if you want to use amp
        if self.proc_rank == 0 and self.print_weights_summary:
            ref_model.summarize()
-        # give model convenience properties
+        # link up experiment object
        ref_model.trainer = self
        if self.experiment is not None:
            ref_model.experiment = self.experiment
-        # save exp to get started
+            # save exp to get started
-        if self.proc_rank == 0 and self.experiment is not None:
+            if self.proc_rank == 0:
-            self.experiment.save()
+                self.experiment.save()
        # track model now.
        # if cluster resets state, the model will update with the saved weights
@@ -761,11 +818,26 @@ We recommend you switch to ddp if you want to use amp
        if self.cluster is not None:  # pragma: no cover
            self.enable_auto_hpc_walltime_manager()
-        # run tiny validation (if validation defined) to make sure program won't crash during val
+        # progress bar init
        if self.show_progress_bar:
            self.progress_bar = tqdm.tqdm(0, position=self.process_position)
        # when testing requested only run test and return
        if self.testing:
            self.__run_evaluation(test=True)
            return
        # run tiny validation (if validation defined)
        # to make sure program won't crash during val
        ref_model.on_sanity_check_start()
-        if self.val_dataloader is not None:
+        if self.val_dataloader is not None and self.nb_sanity_val_steps > 0:
            for ds_i, dataloader in enumerate(self.val_dataloader):
-                self.validate(model, dataloader, self.nb_sanity_val_steps, ds_i)
+
                # reset progress_bar limit for sanity check
                if self.show_progress_bar:
                    self.progress_bar.reset(self.nb_sanity_val_steps)
                self.evaluate(model, dataloader, self.nb_sanity_val_steps, ds_i, self.testing)
        # ---------------------------
        # CORE TRAINING LOOP
@@ -784,10 +856,12 @@ We recommend you switch to ddp if you want to use amp
            self.total_batches = self.nb_tng_batches + self.nb_val_batches
            self.batch_loss_value = 0  # accumulated grads
-            # init progbar when requested
+            # init progress_bar when requested
-            if self.progress_bar:
+            if self.show_progress_bar:
-                self.prog_bar = tqdm.tqdm(range(self.total_batches),
+                self.progress_bar.reset(self.total_batches)
-                                          position=self.process_position)
+
            # changing gradient according accumulation_scheduler
            self.accumulation_scheduler.on_epoch_begin(epoch_nb, self)
            # -----------------
            # RUN TNG EPOCH
@@ -840,8 +914,10 @@ We recommend you switch to ddp if you want to use amp
            # RUN VAL STEP
            # ---------------
            is_val_check_batch = (batch_nb + 1) % self.val_check_batch == 0
            can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
            if self.fast_dev_run or is_val_check_batch or early_stop_epoch:
-                self.__run_validation()
+                if can_check_epoch:
                    self.__run_evaluation(test=self.testing)
            # when batch should be saved
            if (batch_nb + 1) % self.log_save_interval == 0 or early_stop_epoch:
@@ -886,6 +962,13 @@ We recommend you switch to ddp if you want to use amp
            model = self.__get_model()
            model.on_epoch_end()
    def test(self, model=None):
        if model is not None:
            self.testing = True
            self.fit(model)
        else:
            self.__run_evaluation(test=True)
    def __metrics_to_scalars(self, metrics, blacklist=set()):
        new_metrics = {}
        for k, v in metrics.items():
@@ -905,6 +988,34 @@ We recommend you switch to ddp if you want to use amp
        blacklist = {'batch_nb', 'v_nb', 'gpu'}
        return blacklist
    def transfer_batch_to_gpu(self, batch, gpu_id):
        # base case
        if isinstance(batch, torch.Tensor):
            return batch.cuda(gpu_id)
        # when list
        elif isinstance(batch, list):
            for i, x in enumerate(batch):
                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
            return batch
        # when tuple
        elif isinstance(batch, tuple):
            batch = list(batch)
            for i, x in enumerate(batch):
                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
            return tuple(batch)
        # when dict
        elif isinstance(batch, dict):
            for k, v in batch.items():
                batch[k] = self.transfer_batch_to_gpu(v, gpu_id)
            return batch
        # nothing matches, return the value as is without transform
        return batch
    def __tng_forward(self, data_batch, batch_nb, opt_idx):
        """
        Handle forward for each training case (distributed, single gpu, etc...)
@@ -926,9 +1037,8 @@ We recommend you switch to ddp if you want to use amp
            output = self.model(*args)
        elif self.single_gpu:
            gpu_id = self.data_parallel_device_ids[0]
-            for i, x in enumerate(data_batch):
+            data_batch = self.transfer_batch_to_gpu(data_batch, gpu_id)
-                if isinstance(x, torch.Tensor):
+            args[0] = data_batch
                    data_batch[i] = x.cuda(gpu_id)
            output = self.model.training_step(*args)
        else:
@@ -989,8 +1099,8 @@ We recommend you switch to ddp if you want to use amp
            if response == -1:
                return -1
-        if self.progress_bar:
+        if self.show_progress_bar:
-            self.prog_bar.update(1)
+            self.progress_bar.update(1)
        # call training_step once per optimizer
        for opt_idx, optimizer in enumerate(self.optimizers):
@@ -1039,10 +1149,10 @@ We recommend you switch to ddp if you want to use amp
                self.avg_loss = np.mean(self.running_loss[-100:])
                # update progbar
-                if self.progress_bar:
+                if self.show_progress_bar:
                    # add model specific metrics
                    tqdm_metrics = self.__tng_tqdm_dic
-                    self.prog_bar.set_postfix(**tqdm_metrics)
+                    self.progress_bar.set_postfix(**tqdm_metrics)
        # activate batch end hook
        if self.__is_function_implemented('on_batch_end'):
@@ -1051,41 +1161,58 @@ We recommend you switch to ddp if you want to use amp
        return 0
-    def __run_validation(self):
+    def __run_evaluation(self, test=False):
-        # decide if can check epochs
+        # when testing make sure user defined a test step
-        can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
+        can_run_test_step = False
-        if self.fast_dev_run:
+        if test:
-            print('skipping to check performance bc of --fast_dev_run')
+            can_run_test_step = self.__is_overriden('test_step') and self.__is_overriden('test_end')
-        elif not can_check_epoch:
+            if not can_run_test_step:
-            return
+                m = '''You called .test() without defining a test step or test_end.
                Please define and try again'''
                raise MisconfigurationException(m)
        # validate only if model has validation_step defined
-        if self.__is_overriden('validation_step'):
+        # test only if test_step or validation_step are defined
        run_val_step = self.__is_overriden('validation_step')
        if run_val_step or can_run_test_step:
            # hook
-            if self.__is_function_implemented('on_pre_performance_check'):
+            model = self.__get_model()
-                model = self.__get_model()
+            model.on_pre_performance_check()
                model.on_pre_performance_check()
-            # use full val set on end of epoch
+            # select dataloaders
-            # use a small portion otherwise
+            dataloaders = self.val_dataloader
-            max_batches = None if not self.fast_dev_run else 1
+            max_batches = self.nb_val_batches
            for ds_i, dataloader in enumerate(self.val_dataloader):
                val_out_metrics = self.validate(self.model, dataloader, max_batches, ds_i)
                self.__add_tqdm_metrics(val_out_metrics)
-            # hook
+            # calculate max batches to use
-            if self.__is_function_implemented('on_post_performance_check'):
+            if test:
-                model = self.__get_model()
+                dataloaders = self.test_dataloader
                max_batches = self.nb_test_batches
            # cap max batches to 1 when using fast_dev_run
            if self.fast_dev_run:
                max_batches = 1
            for ds_i, dataloader in enumerate(dataloaders):
                eval_out_metrics = self.evaluate(self.model,
                                                 dataloader,
                                                 max_batches,
                                                 ds_i,
                                                 test)
                self.__add_tqdm_metrics(eval_out_metrics)
                # hook
                model.on_post_performance_check()
-            if self.progress_bar:
+            if self.show_progress_bar:
                # add model specific metrics
                tqdm_metrics = self.__tng_tqdm_dic
-                self.prog_bar.set_postfix(**tqdm_metrics)
+                self.progress_bar.set_postfix(**tqdm_metrics)
        # model checkpointing
-        if self.proc_rank == 0 and self.checkpoint_callback is not None:
+        if self.proc_rank == 0 and self.checkpoint_callback is not None and not test:
            print('save callback...')
            self.checkpoint_callback.on_epoch_end(epoch=self.current_epoch,
                                                  logs=self.__tng_tqdm_dic)
@@ -56,6 +56,8 @@ class LightningDataParallel(DataParallel):
            # lightning
            if self.module.training:
                return self.module.training_step(*inputs[0], **kwargs[0])
            elif self.module.testing:
                return self.module.test_step(*inputs[0], **kwargs[0])
            else:
                return self.module.validation_step(*inputs[0], **kwargs[0])
@@ -89,6 +91,8 @@ class LightningDistributedDataParallel(DistributedDataParallel):
                # lightning
                if self.module.training:
                    output = self.module.training_step(*inputs[0], **kwargs[0])
                elif self.module.testing:
                    output = self.module.test_step(*inputs[0], **kwargs[0])
                else:
                    output = self.module.validation_step(*inputs[0], **kwargs[0])
            else:
@@ -153,6 +157,10 @@ def parallel_apply(modules, inputs, kwargs_tup=None, devices=None):  # pragma: n
                # CHANGE
                if module.training:
                    output = module.training_step(*input, **kwargs)
                elif module.testing:
                    output = module.test_step(*input, **kwargs)
                else:
                    output = module.validation_step(*input, **kwargs)
                # ---------------
@@ -1,3 +1,5 @@
 import traceback
 def data_loader(fn):
    """
@@ -10,8 +12,17 @@ def data_loader(fn):
    @property
    def _data_loader(self):
-        if not hasattr(self, attr_name):
+        try:
-            setattr(self, attr_name, fn(self))
+            value = getattr(self, attr_name)
-        return getattr(self, attr_name)
+        except AttributeError:
            try:
                value = fn(self)  # Lazy evaluation, done only once.
            except AttributeError as e:
                # Guard against AttributeError suppression. (Issue #142)
                traceback.print_exc()
                error = f'{fn.__name__}: An AttributeError was encountered: ' + str(e)
                raise RuntimeError(error) from e
            setattr(self, attr_name, value)  # Memoize evaluation.
        return value
    return _data_loader
@@ -5,7 +5,7 @@ class ModelHooks(torch.nn.Module):
    def on_sanity_check_start(self):
        """
-        Called before starting validate
+        Called before starting evaluate
        :return:
        """
        pass
@@ -23,6 +23,9 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        # track if gpu was requested for checkpointing
        self.on_gpu = False
        self.use_dp = False
        self.use_ddp = False
        self.use_amp = False
    def forward(self, *args, **kwargs):
        """
@@ -52,6 +55,16 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        pass
    def test_step(self, *args, **kwargs):
        """
        return whatever outputs will need to be aggregated in test_end
        OPTIONAL
        :param called with batch, batch_nb
        additional: dataset_i if multiple val datasets used
        :return:
        """
        pass
    def validation_end(self, outputs):
        """
        Outputs has the appended output after each validation step
@@ -61,6 +74,15 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        pass
    def test_end(self, outputs):
        """
        Outputs has the appended output after each test step
        OPTIONAL
        :param outputs:
        :return: dic_with_metrics for tqdm
        """
        pass
    def configure_optimizers(self):
        """
        Return a list of optimizers and a list of schedulers (could be empty)
@@ -20,7 +20,7 @@ class LightningTestModel(LightningModule):
    Sample model to show how to define a template
    """
-    def __init__(self, hparams, force_remove_distributed_sampler=False):
+    def __init__(self, hparams, force_remove_distributed_sampler=False, use_two_test_sets=False):
        """
        Pass in parsed HyperOptArgumentParser to the model
        :param hparams:
@@ -28,6 +28,7 @@ class LightningTestModel(LightningModule):
        # init superclass
        super(LightningTestModel, self).__init__()
        self.hparams = hparams
        self.use_two_test_sets = use_two_test_sets  # for some tests regarding testing
        self.batch_size = hparams.batch_size
@@ -167,12 +168,22 @@ class LightningTestModel(LightningModule):
        # if returned a scalar from validation_step, outputs is a list of tensor scalars
        # we return just the average in this case (if we want)
        # return torch.stack(outputs).mean()
        val_loss_mean = 0
        val_acc_mean = 0
        for output in outputs:
-            val_loss_mean += output['val_loss']
+            val_loss = output['val_loss']
-            val_acc_mean += output['val_acc']
+
            # reduce manually when using dp
            if self.trainer.use_dp:
                val_loss = torch.mean(val_loss)
            val_loss_mean += val_loss
            # reduce manually when using dp
            val_acc = output['val_acc']
            if self.trainer.use_dp:
                val_acc = torch.mean(val_acc)
            val_acc_mean += val_acc
        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
@@ -180,6 +191,87 @@ class LightningTestModel(LightningModule):
        tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
        return tqdm_dic
    def test_step(self, data_batch, batch_i, dataloader_i):
        """
        Lightning calls this inside the validation loop
        :param data_batch:
        :return:
        """
        x, y = data_batch
        x = x.view(x.size(0), -1)
        y_hat = self.forward(x)
        loss_test = self.loss(y, y_hat)
        # acc
        labels_hat = torch.argmax(y_hat, dim=1)
        test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
        test_acc = torch.tensor(test_acc)
        if self.on_gpu:
            test_acc = test_acc.cuda(loss_test.device.index)
        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
        if self.trainer.use_dp:
            loss_test = loss_test.unsqueeze(0)
            test_acc = test_acc.unsqueeze(0)
        # alternate possible outputs to test
        if batch_i % 1 == 0:
            output = OrderedDict({
                'test_loss': loss_test,
                'test_acc': test_acc,
            })
            return output
        if batch_i % 2 == 0:
            return test_acc
        if batch_i % 3 == 0:
            output = OrderedDict({
                'test_loss': loss_test,
                'test_acc': test_acc,
                'test_dic': {'test_loss_a': loss_test}
            })
            return output
        if batch_i % 5 == 0:
            output = OrderedDict({
                f'test_loss_{dataloader_i}': loss_test,
                f'test_acc_{dataloader_i}': test_acc,
            })
            return output
    def test_end(self, outputs):
        """
        Called at the end of validation to aggregate outputs
        :param outputs: list of individual outputs of each validation step
        :return:
        """
        # if returned a scalar from test_step, outputs is a list of tensor scalars
        # we return just the average in this case (if we want)
        # return torch.stack(outputs).mean()
        test_loss_mean = 0
        test_acc_mean = 0
        for output in outputs:
            test_loss = output['test_loss']
            # reduce manually when using dp
            if self.trainer.use_dp:
                test_loss = torch.mean(test_loss)
            test_loss_mean += test_loss
            # reduce manually when using dp
            test_acc = output['test_acc']
            if self.trainer.use_dp:
                test_acc = torch.mean(test_acc)
            test_acc_mean += test_acc
        test_loss_mean /= len(outputs)
        test_acc_mean /= len(outputs)
        tqdm_dic = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
        return tqdm_dic
    def on_tng_metrics(self, logs):
        logs['some_tensor_to_test'] = torch.rand(1)
@@ -195,7 +287,7 @@ class LightningTestModel(LightningModule):
        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
        # test returning only 1 list instead of 2
-        return [optimizer]
+        return optimizer
    def __dataloader(self, train):
        # init data generators
@@ -209,7 +301,7 @@ class LightningTestModel(LightningModule):
        batch_size = self.hparams.batch_size
        try:
-            if self.on_gpu and not self.force_remove_distributed_sampler:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
@@ -235,6 +327,8 @@ class LightningTestModel(LightningModule):
    @data_loader
    def test_dataloader(self):
        if self.use_two_test_sets:
            return [self.__dataloader(train=False), self.__dataloader(train=False)]
        return self.__dataloader(train=False)
    @staticmethod
@@ -181,7 +181,7 @@ class NoValEndTestModel(LightningModule):
        batch_size = self.hparams.batch_size
        try:
-            if self.on_gpu and not self.force_remove_distributed_sampler:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
@@ -138,7 +138,7 @@ class NoValModel(LightningModule):
        batch_size = self.hparams.batch_size
        try:
-            if self.on_gpu and not self.force_remove_distributed_sampler:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
@@ -45,6 +45,7 @@ omit =
    tests/test_models.py
    pytorch_lightning/testing_models/lm_test_module.py
    pytorch_lightning/utilities/arg_parse.py
    examples/templates
 [flake8]
 ignore = E731,W504,F401,F841
@@ -14,7 +14,7 @@ from setuptools import setup, find_packages
 # engineer specific practices
 setup(
    name='pytorch-lightning',
-    version='0.4.5',
+    version='0.4.8',
    description='The Keras for ML researchers using PyTorch',
    author='William Falcon',
    author_email='waf2107@columbia.edu',
@@ -30,7 +30,7 @@ setup(
    python_requires='>=3.6',
    install_requires=[
        'torch==1.2.0',
-        'tqdm',
+        'tqdm>=4.35.0',
        'test-tube>=0.6.9',
        'pandas>=0.20.3',
    ],
@@ -1,5 +1,6 @@
 from pytorch_lightning import Trainer
 from examples import LightningTemplateModel
 from pytorch_lightning.testing import LightningTestModel, NoValEndTestModel, NoValModel
 from argparse import Namespace
 from test_tube import Experiment
 from pytorch_lightning.callbacks import ModelCheckpoint
@@ -11,6 +12,8 @@ import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 import numpy as np
 import pdb
 class CoolModel(pl.LightningModule):
@@ -72,10 +75,11 @@ def get_model():
    return model, hparams
-def get_exp(debug=True):
+def get_exp(debug=True, version=None):
    # set up exp object without actually saving logs
    root_dir = os.path.dirname(os.path.realpath(__file__))
-    exp = Experiment(debug=debug, save_dir=root_dir, name='tests_tt_dir')
+    save_dir = os.path.join(root_dir, 'save_dir')
    exp = Experiment(debug=debug, save_dir=save_dir, name='tests_tt_dir', version=version)
    return exp
@@ -98,7 +102,7 @@ def clear_save_dir():
        shutil.rmtree(save_dir)
-def load_model(exp, save_dir):
+def load_model(exp, save_dir, on_gpu, map_location=None, module_class=LightningTemplateModel):
    # load trained model
    tags_path = exp.get_data_path(exp.name, exp.version)
@@ -107,8 +111,10 @@ def load_model(exp, save_dir):
    checkpoints = [x for x in os.listdir(save_dir) if '.ckpt' in x]
    weights_dir = os.path.join(save_dir, checkpoints[0])
-    trained_model = LightningTemplateModel.load_from_metrics(weights_path=weights_dir,
+    trained_model = module_class.load_from_metrics(weights_path=weights_dir,
-                                                             tags_csv=tags_path, on_gpu=True)
+                                                   tags_csv=tags_path,
                                                   on_gpu=on_gpu,
                                                   map_location=map_location)
    assert trained_model is not None, 'loading model failed'
@@ -130,47 +136,124 @@ def run_prediction(dataloader, trained_model):
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    val_acc = torch.tensor(val_acc)
    val_acc = val_acc.item()
    print(val_acc)
    assert val_acc > 0.70, 'this model is expected to get > 0.7 in test set (it got %f)' % val_acc
-def main():
+# ------------------------------------------------------------------------
-
+def run_gpu_model_test(trainer_options, model, hparams, on_gpu=True):
    save_dir = init_save_dir()
    # exp file to get meta
    exp = get_exp(False)
    exp.argparse(hparams)
    exp.save()
    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)
-    trainer = Trainer(
+    # add these to the trainer options
-        experiment=exp,
+    trainer_options['checkpoint_callback'] = checkpoint
-        checkpoint_callback=checkpoint,
+    trainer_options['experiment'] = exp
        progress_bar=True,
        max_nb_epochs=1,
        gpus=[0, 1],
        distributed_backend='dp',
    )
    model = CoolModel()
    # fit model
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
    # correct result and ok accuracy
    assert result == 1, 'amp + ddp model failed to complete'
    # test model loading
-    pretrained_model = load_model(exp, save_dir)
+    pretrained_model = load_model(exp, save_dir, on_gpu)
-    # test model preds
+    # test new model accuracy
    run_prediction(model.test_dataloader, pretrained_model)
    if trainer.use_ddp:
        # on hpc this would work fine... but need to hack it for the purpose of the test
        trainer.model = pretrained_model
        trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers()
    # test HPC loading / saving
    trainer.hpc_save(save_dir, exp)
    trainer.hpc_load(save_dir, on_gpu=on_gpu)
    clear_save_dir()
 def assert_ok_val_acc(trainer):
    # this model should get 0.80+ acc
    acc = trainer.tng_tqdm_dic['val_acc']
    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
 def assert_ok_test_acc(trainer):
    # this model should get 0.80+ acc
    acc = trainer.tng_tqdm_dic['test_acc']
    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
 def get_hparams(continue_training=False, hpc_exp_number=0):
    root_dir = os.path.dirname(os.path.realpath(__file__))
    args = {
        'drop_prob': 0.2,
        'batch_size': 32,
        'in_features': 28 * 28,
        'learning_rate': 0.001 * 8,
        'optimizer_name': 'adam',
        'data_root': os.path.join(root_dir, 'mnist'),
        'out_features': 10,
        'hidden_dim': 1000}
    if continue_training:
        args['test_tube_do_checkpoint_load'] = True
        args['hpc_exp_number'] = hpc_exp_number
    hparams = Namespace(**args)
    return hparams
 def main():
    """Verify test() on fitted model"""
    hparams = get_hparams()
    model = LightningTestModel(hparams)
    save_dir = init_save_dir()
    # exp file to get meta
    exp = get_exp(False)
    exp.argparse(hparams)
    exp.save()
    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)
    trainer_options = dict(
        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.2,
        checkpoint_callback=checkpoint,
        experiment=exp,
        gpus=[0, 1],
        distributed_backend='ddp'
    )
    # fit model
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
    # correct result and ok accuracy
    assert result == 1, 'training failed to complete'
    pretrained_model = load_model(exp, save_dir, on_gpu=True, module_class=LightningTestModel)
    new_trainer = Trainer(**trainer_options)
    new_trainer.test(pretrained_model)
    # test we have good test accuracy
    assert_ok_test_acc(new_trainer)
    # clear_save_dir()
 if __name__ == '__main__':
    main()
@@ -11,7 +11,11 @@ from test_tube import Experiment, SlurmCluster
 # sys.path += [os.path.abspath('..'), os.path.abspath('../..')]
 from pytorch_lightning import Trainer
 from pytorch_lightning.testing import LightningTestModel, NoValEndTestModel, NoValModel
-from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
+from pytorch_lightning.callbacks import (
    ModelCheckpoint,
    EarlyStopping,
    GradientAccumulationScheduler,
 )
 from pytorch_lightning.utilities.debugging import MisconfigurationException
 from pytorch_lightning.root_module import memory
 from pytorch_lightning.models.trainer import reduce_distributed_output
@@ -26,6 +30,342 @@ np.random.seed(SEED)
 # ------------------------------------------------------------------------
 # TESTS
 # ------------------------------------------------------------------------
 def test_running_test_pretrained_model_ddp():
    """Verify test() on pretrained model"""
    if not can_run_gpu_test():
        return
    hparams = get_hparams()
    model = LightningTestModel(hparams)
    save_dir = init_save_dir()
    # exp file to get meta
    exp = get_exp(False)
    exp.argparse(hparams)
    exp.save()
    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)
    trainer_options = dict(
        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.2,
        checkpoint_callback=checkpoint,
        experiment=exp,
        gpus=[0, 1],
        distributed_backend='ddp'
    )
    # fit model
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
    # correct result and ok accuracy
    assert result == 1, 'training failed to complete'
    pretrained_model = load_model(exp, save_dir, on_gpu=True, module_class=LightningTestModel)
    # run test set
    new_trainer = Trainer(**trainer_options)
    new_trainer.test(pretrained_model)
    run_prediction(model.test_dataloader, pretrained_model)
    # test we have good test accuracy
    clear_save_dir()
 def test_running_test_after_fitting():
    """Verify test() on fitted model"""
    hparams = get_hparams()
    model = LightningTestModel(hparams)
    save_dir = init_save_dir()
    # exp file to get meta
    exp = get_exp(False)
    exp.argparse(hparams)
    exp.save()
    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)
    trainer_options = dict(
        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.2,
        test_percent_check=0.2,
        checkpoint_callback=checkpoint,
        experiment=exp
    )
    # fit model
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
    assert result == 1, 'training failed to complete'
    trainer.test()
    # test we have good test accuracy
    assert_ok_test_acc(trainer)
    clear_save_dir()
 def test_running_test_pretrained_model():
    """Verify test() on pretrained model"""
    hparams = get_hparams()
    model = LightningTestModel(hparams)
    save_dir = init_save_dir()
    # exp file to get meta
    exp = get_exp(False)
    exp.argparse(hparams)
    exp.save()
    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)
    trainer_options = dict(
        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.2,
        checkpoint_callback=checkpoint,
        experiment=exp
    )
    # fit model
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
    # correct result and ok accuracy
    assert result == 1, 'training failed to complete'
    pretrained_model = load_model(exp, save_dir, on_gpu=False, module_class=LightningTestModel)
    new_trainer = Trainer(**trainer_options)
    new_trainer.test(pretrained_model)
    # test we have good test accuracy
    assert_ok_test_acc(new_trainer)
    clear_save_dir()
 def test_running_test_pretrained_model_dp():
    """Verify test() on pretrained model"""
    if not can_run_gpu_test():
        return
    hparams = get_hparams()
    model = LightningTestModel(hparams)
    save_dir = init_save_dir()
    # exp file to get meta
    exp = get_exp(False)
    exp.argparse(hparams)
    exp.save()
    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)
    trainer_options = dict(
        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.2,
        checkpoint_callback=checkpoint,
        experiment=exp,
        gpus=[0, 1],
        distributed_backend='dp'
    )
    # fit model
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
    # correct result and ok accuracy
    assert result == 1, 'training failed to complete'
    pretrained_model = load_model(exp, save_dir, on_gpu=True, module_class=LightningTestModel)
    new_trainer = Trainer(**trainer_options)
    new_trainer.test(pretrained_model)
    # test we have good test accuracy
    assert_ok_test_acc(new_trainer)
    clear_save_dir()
 def test_gradient_accumulation_scheduling():
    """
    Test grad accumulation by the freq of optimizer updates
    """
    # test incorrect configs
    with pytest.raises(IndexError):
        assert Trainer(accumulate_grad_batches={0: 3, 1: 4, 4: 6})
        assert Trainer(accumulate_grad_batches={-2: 3})
    with pytest.raises(TypeError):
        assert Trainer(accumulate_grad_batches={})
        assert Trainer(accumulate_grad_batches=[[2, 3], [4, 6]])
        assert Trainer(accumulate_grad_batches={1: 2, 3.: 4})
        assert Trainer(accumulate_grad_batches={1: 2.5, 3: 5})
    # test optimizer call freq matches scheduler
    def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i):
        # only test the first 12 batches in epoch
        if batch_nb < 12:
            if epoch_nb == 0:
                # reset counter when starting epoch
                if batch_nb == 0:
                    self.prev_called_batch_nb = 0
                    # use this opportunity to test once
                    assert self.trainer.accumulate_grad_batches == 1
                assert batch_nb == self.prev_called_batch_nb
                self.prev_called_batch_nb += 1
            elif 1 <= epoch_nb <= 2:
                # reset counter when starting epoch
                if batch_nb == 1:
                    self.prev_called_batch_nb = 1
                    # use this opportunity to test once
                    assert self.trainer.accumulate_grad_batches == 2
                assert batch_nb == self.prev_called_batch_nb
                self.prev_called_batch_nb += 2
            else:
                if batch_nb == 3:
                    self.prev_called_batch_nb = 3
                    # use this opportunity to test once
                    assert self.trainer.accumulate_grad_batches == 4
                assert batch_nb == self.prev_called_batch_nb
                self.prev_called_batch_nb += 3
        optimizer.step()
        # clear gradients
        optimizer.zero_grad()
    hparams = get_hparams()
    model = LightningTestModel(hparams)
    schedule = {1: 2, 3: 4}
    trainer = Trainer(accumulate_grad_batches=schedule,
                      train_percent_check=0.1,
                      val_percent_check=0.1,
                      max_nb_epochs=4)
    # for the test
    trainer.optimizer_step = optimizer_step
    model.prev_called_batch_nb = 0
    trainer.fit(model)
 def test_multi_gpu_model_ddp():
    """
    Make sure DDP works
    :return:
    """
    if not can_run_gpu_test():
        return
    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
    model, hparams = get_model()
    trainer_options = dict(
        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.2,
        gpus=[0, 1],
        distributed_backend='ddp'
    )
    run_gpu_model_test(trainer_options, model, hparams)
 def test_optimizer_return_options():
    trainer = Trainer()
    model, hparams = get_model()
    # single optimizer
    opt_a = torch.optim.Adam(model.parameters(), lr=0.002)
    opt_b = torch.optim.SGD(model.parameters(), lr=0.002)
    optim, lr_sched = trainer.init_optimizers(opt_a)
    assert len(optim) == 1 and len(lr_sched) == 0
    # opt tuple
    opts = (opt_a, opt_b)
    optim, lr_sched = trainer.init_optimizers(opts)
    assert len(optim) == 2 and optim[0] == opts[0] and optim[1] == opts[1]
    assert len(lr_sched) == 0
    # opt list
    opts = [opt_a, opt_b]
    optim, lr_sched = trainer.init_optimizers(opts)
    assert len(optim) == 2 and optim[0] == opts[0] and optim[1] == opts[1]
    assert len(lr_sched) == 0
    # opt tuple of lists
    opts = ([opt_a], ['lr_scheduler'])
    optim, lr_sched = trainer.init_optimizers(opts)
    assert len(optim) == 1 and len(lr_sched) == 1
    assert optim[0] == opts[0][0] and lr_sched[0] == 'lr_scheduler'
 def test_single_gpu_batch_parse():
    if not can_run_gpu_test():
        return
    trainer = Trainer()
    # batch is just a tensor
    batch = torch.rand(2, 3)
    batch = trainer.transfer_batch_to_gpu(batch, 0)
    assert batch.device.index == 0 and batch.type() == 'torch.cuda.FloatTensor'
    # tensor list
    batch = [torch.rand(2, 3), torch.rand(2, 3)]
    batch = trainer.transfer_batch_to_gpu(batch, 0)
    assert batch[0].device.index == 0 and batch[0].type() == 'torch.cuda.FloatTensor'
    assert batch[1].device.index == 0 and batch[1].type() == 'torch.cuda.FloatTensor'
    # tensor list of lists
    batch = [[torch.rand(2, 3), torch.rand(2, 3)]]
    batch = trainer.transfer_batch_to_gpu(batch, 0)
    assert batch[0][0].device.index == 0 and batch[0][0].type() == 'torch.cuda.FloatTensor'
    assert batch[0][1].device.index == 0 and batch[0][1].type() == 'torch.cuda.FloatTensor'
    # tensor dict
    batch = [{'a': torch.rand(2, 3), 'b': torch.rand(2, 3)}]
    batch = trainer.transfer_batch_to_gpu(batch, 0)
    assert batch[0]['a'].device.index == 0 and batch[0]['a'].type() == 'torch.cuda.FloatTensor'
    assert batch[0]['b'].device.index == 0 and batch[0]['b'].type() == 'torch.cuda.FloatTensor'
    # tuple of tensor list and list of tensor dict
    batch = ([torch.rand(2, 3) for _ in range(2)],
             [{'a': torch.rand(2, 3), 'b': torch.rand(2, 3)} for _ in range(2)])
    batch = trainer.transfer_batch_to_gpu(batch, 0)
    assert batch[0][0].device.index == 0 and batch[0][0].type() == 'torch.cuda.FloatTensor'
    assert batch[1][0]['a'].device.index == 0
    assert batch[1][0]['a'].type() == 'torch.cuda.FloatTensor'
    assert batch[1][0]['b'].device.index == 0
    assert batch[1][0]['b'].type() == 'torch.cuda.FloatTensor'
 def test_early_stopping_cpu_model():
    """
@@ -40,7 +380,7 @@ def test_early_stopping_cpu_model():
        overfit_pct=0.20,
        track_grad_norm=2,
        print_nan_grads=True,
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        train_percent_check=0.1,
        val_percent_check=0.1
@@ -80,7 +420,7 @@ def test_no_val_module():
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
-    # traning complete
+    # training complete
    assert result == 1, 'amp + ddp model failed to complete'
    # save model
@@ -187,7 +527,7 @@ def test_amp_single_gpu():
    model = LightningTestModel(hparams)
    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0],
        distributed_backend='dp',
@@ -268,13 +608,7 @@ def test_amp_gpu_ddp():
    Make sure DDP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
+    if not can_run_gpu_test():
        warnings.warn('test_amp_gpu_ddp cannot run.'
                      'Rerun on a GPU node to run this test')
        return
    if not torch.cuda.device_count() > 1:
        warnings.warn('test_amp_gpu_ddp cannot run.'
                      'Rerun on a node with 2+ GPUs to run this test')
        return
    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
@@ -283,7 +617,7 @@ def test_amp_gpu_ddp():
    model = LightningTestModel(hparams)
    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0, 1],
        distributed_backend='ddp',
@@ -486,13 +820,7 @@ def test_amp_gpu_ddp_slurm_managed():
    Make sure DDP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
+    if not can_run_gpu_test():
        warnings.warn('test_amp_gpu_ddp cannot run.'
                      ' Rerun on a GPU node to run this test')
        return
    if not torch.cuda.device_count() > 1:
        warnings.warn('test_amp_gpu_ddp cannot run.'
                      ' Rerun on a node with 2+ GPUs to run this test')
        return
    # simulate setting slurm flags
@@ -503,7 +831,7 @@ def test_amp_gpu_ddp_slurm_managed():
    model = LightningTestModel(hparams)
    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0],
        distributed_backend='ddp',
@@ -568,7 +896,7 @@ def test_cpu_model_with_amp():
    """
    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        max_nb_epochs=1,
        train_percent_check=0.4,
@@ -589,7 +917,7 @@ def test_cpu_model():
    """
    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        max_nb_epochs=1,
        train_percent_check=0.4,
@@ -612,7 +940,7 @@ def test_all_features_cpu_model():
        overfit_pct=0.20,
        track_grad_norm=2,
        print_nan_grads=True,
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        accumulate_grad_batches=2,
        max_nb_epochs=1,
@@ -636,7 +964,7 @@ def test_single_gpu_model():
    model, hparams = get_model()
    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.1,
        val_percent_check=0.1,
@@ -651,17 +979,12 @@ def test_multi_gpu_model_dp():
    Make sure DP works
    :return:
    """
-    if not torch.cuda.is_available():
+    if not can_run_gpu_test():
        warnings.warn('test_multi_gpu_model_dp cannot run.'
                      ' Rerun on a GPU node to run this test')
        return
    if not torch.cuda.device_count() > 1:
        warnings.warn('test_multi_gpu_model_dp cannot run.'
                      ' Rerun on a node with 2+ GPUs to run this test')
        return
    model, hparams = get_model()
    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.1,
        val_percent_check=0.1,
@@ -679,14 +1002,9 @@ def test_amp_gpu_dp():
    Make sure DP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
+    if not can_run_gpu_test():
        warnings.warn('test_amp_gpu_dp cannot run.'
                      ' Rerun on a GPU node to run this test')
        return
    if not torch.cuda.device_count() > 1:
        warnings.warn('test_amp_gpu_dp cannot run.'
                      ' Rerun on a node with 2+ GPUs to run this test')
        return
    model, hparams = get_model()
    trainer_options = dict(
        max_nb_epochs=1,
@@ -698,44 +1016,12 @@ def test_amp_gpu_dp():
        run_gpu_model_test(trainer_options, model, hparams)
 def test_multi_gpu_model_ddp():
    """
    Make sure DDP works
    :return:
    """
    if not torch.cuda.is_available():
        warnings.warn('test_multi_gpu_model_ddp cannot run.'
                      ' Rerun on a GPU node to run this test')
        return
    if not torch.cuda.device_count() > 1:
        warnings.warn('test_multi_gpu_model_ddp cannot run.'
                      ' Rerun on a node with 2+ GPUs to run this test')
        return
    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
    model, hparams = get_model()
    trainer_options = dict(
        progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.4,
        val_percent_check=0.2,
        gpus=[0, 1],
        distributed_backend='ddp'
    )
    run_gpu_model_test(trainer_options, model, hparams)
 def test_ddp_sampler_error():
    """
    Make sure DDP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
+    if not can_run_gpu_test():
        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a GPU node to run this test')
        return
    if not torch.cuda.device_count() > 1:
        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a node with 2+ GPUs to run this test')
        return
    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
@@ -748,7 +1034,7 @@ def test_ddp_sampler_error():
    trainer = Trainer(
        experiment=exp,
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        gpus=[0, 1],
        distributed_backend='ddp',
@@ -769,7 +1055,34 @@ def test_multiple_val_dataloader():
    hparams = get_hparams()
    model = LightningTestModel(hparams)
-    save_dir = init_save_dir()
+    # exp file to get meta
    trainer_options = dict(
        max_nb_epochs=1,
        val_percent_check=0.1,
        train_percent_check=1.0,
    )
    # fit model
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)
    # verify tng completed
    assert result == 1
    # verify there are 2 val loaders
    assert len(trainer.val_dataloader) == 2, 'Multiple val_dataloaders not initiated properly'
    # make sure predictions are good for each val set
    [run_prediction(dataloader, trainer.model) for dataloader in trainer.val_dataloader]
 def test_multiple_test_dataloader():
    """
    Verify multiple test_dataloader
    :return:
    """
    hparams = get_hparams()
    model = LightningTestModel(hparams, use_two_test_sets=True)
    # exp file to get meta
    trainer_options = dict(
@@ -786,10 +1099,10 @@ def test_multiple_val_dataloader():
    assert result == 1
    # verify there are 2 val loaders
-    assert len(trainer.val_dataloader) == 2, 'Multiple val_dataloaders not initiated properly'
+    assert len(trainer.test_dataloader) == 2, 'Multiple test_dataloaders not initiated properly'
-    # make sure predictions are good for each val set
+    # make sure predictions are good for each test set
-    [run_prediction(dataloader, trainer.model) for dataloader in trainer.val_dataloader]
+    [run_prediction(dataloader, trainer.model) for dataloader in trainer.test_dataloader]
 # ------------------------------------------------------------------------
@@ -820,7 +1133,7 @@ def run_gpu_model_test(trainer_options, model, hparams, on_gpu=True):
    # test model loading
    pretrained_model = load_model(exp, save_dir, on_gpu)
-    # test model preds
+    # test new model accuracy
    run_prediction(model.test_dataloader, pretrained_model)
    if trainer.use_ddp:
@@ -871,7 +1184,8 @@ def get_model(use_test_model=False):
 def get_exp(debug=True, version=None):
    # set up exp object without actually saving logs
    root_dir = os.path.dirname(os.path.realpath(__file__))
-    exp = Experiment(debug=debug, save_dir=root_dir, name='tests_tt_dir', version=version)
+    save_dir = os.path.join(root_dir, 'save_dir')
    exp = Experiment(debug=debug, save_dir=save_dir, name='tests_tt_dir', version=version)
    return exp
@@ -880,7 +1194,8 @@ def init_save_dir():
    save_dir = os.path.join(root_dir, 'save_dir')
    if os.path.exists(save_dir):
-        shutil.rmtree(save_dir)
+        n = np.random.randint(0, 10000000, 1)[0]
        shutil.move(save_dir, save_dir + f'_{n}')
    os.makedirs(save_dir, exist_ok=True)
@@ -891,10 +1206,11 @@ def clear_save_dir():
    root_dir = os.path.dirname(os.path.realpath(__file__))
    save_dir = os.path.join(root_dir, 'save_dir')
    if os.path.exists(save_dir):
-        shutil.rmtree(save_dir)
+        n = np.random.randint(0, 10000000, 1)[0]
        shutil.move(save_dir, save_dir + f'_{n}')
-def load_model(exp, save_dir, on_gpu, map_location=None):
+def load_model(exp, save_dir, on_gpu, map_location=None, module_class=LightningTemplateModel):
    # load trained model
    tags_path = exp.get_data_path(exp.name, exp.version)
@@ -903,10 +1219,10 @@ def load_model(exp, save_dir, on_gpu, map_location=None):
    checkpoints = [x for x in os.listdir(save_dir) if '.ckpt' in x]
    weights_dir = os.path.join(save_dir, checkpoints[0])
-    trained_model = LightningTemplateModel.load_from_metrics(weights_path=weights_dir,
+    trained_model = module_class.load_from_metrics(weights_path=weights_dir,
-                                                             tags_csv=tags_path,
+                                                   tags_csv=tags_path,
-                                                             on_gpu=on_gpu,
+                                                   on_gpu=on_gpu,
-                                                             map_location=map_location)
+                                                   map_location=map_location)
    assert trained_model is not None, 'loading model failed'
@@ -925,19 +1241,35 @@ def run_prediction(dataloader, trained_model):
    # acc
    labels_hat = torch.argmax(y_hat, dim=1)
-    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
+    acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-    val_acc = torch.tensor(val_acc)
+    acc = torch.tensor(acc)
-    val_acc = val_acc.item()
+    acc = acc.item()
-    print(val_acc)
+    assert acc > 0.50, f'this model is expected to get > 0.50 in test set (it got {acc})'
    assert val_acc > 0.50, 'this model is expected to get > 0.50 in test set (it got %f)' % val_acc
-def assert_ok_acc(trainer):
+def assert_ok_val_acc(trainer):
    # this model should get 0.80+ acc
    acc = trainer.tng_tqdm_dic['val_acc']
-    assert acc > 0.50, 'model failed to get expected 0.50 validation accuracy. Got: %f' % acc
+    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
 def assert_ok_test_acc(trainer):
    # this model should get 0.80+ acc
    acc = trainer.tng_tqdm_dic['test_acc']
    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
 def can_run_gpu_test():
    if not torch.cuda.is_available():
        warnings.warn('test_multi_gpu_model_ddp cannot run.'
                      ' Rerun on a GPU node to run this test')
        return False
    if not torch.cuda.device_count() > 1:
        warnings.warn('test_multi_gpu_model_ddp cannot run.'
                      ' Rerun on a node with 2+ GPUs to run this test')
        return False
    return True
 if __name__ == '__main__':
@@ -38,7 +38,7 @@ commands =
    flake8 .
 [flake8]
-exclude = .tox,*.egg,build,temp
+exclude = .tox,*.egg,build,temp,examples/templates
 select = E,W,F
 doctests = True
 verbose = 2
Author	SHA1	Message	Date
William Falcon	60384eb61e	release v0.4.8	2019-08-31 03:05:57 -04:00
William Falcon	f51b45933b	Expectopatronum implement #89 (#182 ) * rename validate -> evaluate; implement test logic; allow multiple test_loaders * add test_step and test_end to LightningModule * add in_test_mode to pretraining to implement case 2 (test pretrained model) * fix code style issues * LightningTestModel: add optional second test set, implement test_step and test_end * implemented test for multiple test_dataloaders; fixed typo * add two test cases for #89 * add documentation for test_step, test_end; fix computation of loss in validation_step example * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Added proper dp ddp routing calls for test mode * Update trainer.py * Update test_models.py * Update trainer.py * Update trainer.py * Update override_data_parallel.py * Update test_models.py * Update test_models.py * Update trainer.py * Update trainer.py * Update trainer.py * Update test_models.py * Update test_models.py * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * Update trainer.py * Update override_data_parallel.py * Update debug.py * Update lm_test_module.py * Update test_models.py	2019-08-30 18:56:09 -04:00
Stanislav	73cf47112e	Gradient accumulation callback (#150 ) * Gradient accumulation callback * little test case * typo * import fix * method name fix * fix epochs indexing from 1 * better code style * code style fix v2 :/ * change interface * fix Trainre new api in tests * trainer api bug fix * new raising error, new update method * extentions tests * a little better tests * typo fix * flack8 better * using scheduler for int and dict * typo * firs epoch bug fix * test update * empty dict exception * floats check * codestyle fix * grad counting test * someday, i will install normal linter * add more checks * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py	2019-08-30 10:56:14 -04:00
Ir1dXD	c2247350bb	feat(val_sanity): enable skipping validation sanity (#176 ) * feat(val_sanity): enable skipping validation sanity when self.nb_sanity_val_steps is 0 * docs: elaborate on skipping	2019-08-28 06:41:31 -04:00
William Falcon	67c314272b	Update setup.py (#174 )	2019-08-27 18:07:33 -04:00
Ir1dXD	da4c1e3409	docs: add repo_name in the upright corner (#171 )	2019-08-27 16:46:18 -04:00
Jirka Borovec	cd89b4ef43	move GH docs (#168 )	2019-08-27 07:10:26 -04:00
Ir1dXD	6eb6daa278	enable highlight (#170 )	2019-08-27 07:09:46 -04:00
William Falcon	c24599f5e5	release v	2019-08-24 08:13:54 -04:00
Ryan McCormick	b22e5918a9	fix python syntax in code blocks to be consistent (#166 ) A couple code blocks used "{.python}" instead of just "python" for the syntax highlighting, which doesn't render properly in GitHub markdown.	2019-08-23 21:24:18 -04:00
William Falcon	4104a0fc47	cleaned up progbar (#165 ) * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * flake 8	2019-08-23 21:23:27 -04:00
William Falcon	2ad9a9708b	Update README.md	2019-08-23 16:10:45 -04:00
William Falcon	ecce22f4de	Update README.md	2019-08-23 16:10:24 -04:00
Sebastian Præsius	b31539f62e	Guard against AttributeError in dataloaders. (#161 ) A solution for https://github.com/williamFalcon/pytorch-lightning/issues/142. Since hasattr "calls getattr(object, name) and to see whether it raises an AttributeError or not", I replaced it with a single call to getattr. See also https://stackoverflow.com/questions/24971061/python-hasattr-vs-getattr	2019-08-23 08:21:39 -04:00
William Falcon	cbb9821d9b	Cleaned up val/tng/test nb batches (#163 ) Set all to be 0 instead of None. Cleaned up val batch	2019-08-23 07:42:17 -04:00
William Falcon	a1490e993a	Update README.md	2019-08-23 03:46:27 -04:00
William Falcon	77d085134b	Update README.md	2019-08-23 03:45:14 -04:00
William Falcon	c30f69f60d	Update lightning_module_template.py	2019-08-23 02:42:40 -04:00
William Falcon	d5d47eab0d	Update lightning_module_template.py	2019-08-23 02:39:05 -04:00
Sebastian Præsius	9fc66026f1	train = False in test_dataloader (#162 ) A small change to the CoolModel example. Now test_dataloader returns the MNIST test dataset.	2019-08-22 17:44:06 -04:00
eqs	4a0b56755c	bug fix for #157 (#158 ) * Separate condition list/tuple case into separated cases * Add test for tuple of tensor list and list of tensor dict * Update test_models.py	2019-08-21 10:22:51 -04:00
William Falcon	55a804b7cf	fixes #154 (#155 ) * fixes #154 * Update trainer.py * Update trainer.py	2019-08-20 16:59:26 -04:00
William Falcon	7119ec1693	Update CONTRIBUTING.md	2019-08-20 09:51:11 -04:00
Ananya Harsh Jha	5b694c7e0e	bug fix for #138 (#143 ) * bug fix for #138 * split if for readability	2019-08-19 15:03:04 -04:00
sebftw	4bdb976284	Set val_check_interval default to 1.0. (#145 ) See discussion in https://github.com/williamFalcon/pytorch-lightning/issues/139.	2019-08-19 10:42:08 -04:00
William Falcon	4ad4588122	Update README.md	2019-08-19 07:22:02 -04:00
William Falcon	ac8186cb3c	Update README.md	2019-08-19 07:20:10 -04:00
Jirka Borovec	dbbbba35c9	add Codecov info (#144 )	2019-08-19 06:35:09 -04:00
William Falcon	f2a02881e3	Update README.md	2019-08-18 19:17:25 -04:00
William Falcon	e8c423a3b0	Update README.md	2019-08-18 19:16:57 -04:00
William Falcon	73b70584e7	Update README.md	2019-08-18 19:16:25 -04:00
William Falcon	736cf9b162	Update README.md	2019-08-18 19:16:09 -04:00
William Falcon	5771583c9d	Update README.md	2019-08-18 19:15:41 -04:00
William Falcon	64503f0d5e	Update README.md	2019-08-18 19:15:09 -04:00
William Falcon	504418d157	Update README.md	2019-08-18 19:05:13 -04:00
William Falcon	ad61b03fe9	Update README.md	2019-08-18 18:51:47 -04:00
sebftw	a7a14dadb6	F.cross_entropy(y_hat, y)(y_hat, y) typo. (#137 ) This seems to be a typo. Throws TypeError: 'Tensor' object is not callable.	2019-08-18 18:17:43 -04:00
sebftw	b2a49197e4	tensorboarX to tensorboardX (#136 ) * tensorboarX to tensorboardX * Update properties.md	2019-08-18 18:17:05 -04:00
sebftw	23a4421595	Removed redundant line. (#140 )	2019-08-18 18:16:30 -04:00
sebftw	26d3f0dbea	Error if dataset size = 1 batch. (#141 ) Fix for the bug mentioned in https://github.com/williamFalcon/pytorch-lightning/issues/139	2019-08-18 18:15:58 -04:00
Maxim Andreev	e646d745da	use val_percent_check in validation step (#135 )	2019-08-18 11:02:28 -04:00
William Falcon	9aa9a1a796	Update lightning_module_template.py	2019-08-17 11:11:07 -04:00
Ir1dXD	48de39ed50	elaborate on the correlation between overfit_pct and xxx_percent_check (#132 ) * Update Training Loop.md * update docs and elaborate on the correlation	2019-08-17 10:23:25 -04:00
William Falcon	1a31782272	fixed str crash err	2019-08-17 10:20:58 -04:00
Ir1dXD	24a97956e4	fix typo in docs (#129 ) * fix typo * fix typo * fix typo * fix list	2019-08-17 07:48:33 -04:00
William Falcon	1b7d66d089	Merge branch 'master' of https://github.com/williamFalcon/pytorch-lightning	2019-08-16 17:14:40 -04:00
William Falcon	e60e002f17	updated docs	2019-08-16 17:14:31 -04:00
William Falcon	308239cef0	allow loss to be used for early stopping (#127 )	2019-08-16 11:58:44 -04:00
William Falcon	bdd86087e6	updated docs	2019-08-16 10:07:56 -04:00
William Falcon	50f0de094f	updated docs	2019-08-16 10:07:44 -04:00
William Falcon	bc401d0f59	updated docs	2019-08-16 10:02:28 -04:00
William Falcon	4b97319c2e	updated docs	2019-08-15 21:29:25 -04:00
William Falcon	90f01c05bc	updated docs	2019-08-15 21:21:26 -04:00
William Falcon	0e92a9d7af	updated docs	2019-08-15 21:19:29 -04:00
William Falcon	81837221a4	updated docs	2019-08-15 13:59:54 -04:00
William Falcon	44da88fd15	updated docs	2019-08-15 13:59:27 -04:00
William Falcon	3f1feb014f	release v0.4.6	2019-08-15 13:55:58 -04:00
Jirka Borovec	6f1d2c45fe	update Win CI req. (#123 )	2019-08-15 11:45:03 -04:00
William Falcon	a27fb5d54c	enhanced optimizer return options (#120 ) * added smarter optimizer options * added smarter optimizer options * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added smarter optimizer options tests * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive	2019-08-15 11:31:56 -04:00
Jirka Borovec	83b1646e45	fix appveyor (#69 ) * fix appveyor * fix appveyor	2019-08-15 09:54:29 -04:00
William Falcon	db9254acbe	enable recursive parsing for single gpu inputs (#121 ) * added tests * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive * added single gpu data transfer recursive	2019-08-15 09:39:09 -04:00
William Falcon	0f287ce5ea	Update tox.ini	2019-08-14 10:22:41 -04:00
William Falcon	590282f2b0	Update gan.py	2019-08-14 09:29:02 -04:00
William Falcon	2f984c9971	enable returning only opt list (#114 )	2019-08-14 09:02:11 -04:00
William Falcon	b64e94bae3	Merge branch 'master' of https://github.com/williamFalcon/pytorch-lightning	2019-08-14 08:59:15 -04:00
William Falcon	e000f052ac	ommit templates folder	2019-08-14 08:59:05 -04:00
William Falcon	c9117f74b2	Update gan.py	2019-08-14 08:43:50 -04:00
William Falcon	13f2d1ab1c	Update gan.py	2019-08-14 08:41:32 -04:00
William Falcon	0d5da5f29b	added gan template (#115 ) * added gan template * ommit templates folder	2019-08-14 08:38:49 -04:00
William Falcon	4795130538	Update README.md	2019-08-14 07:21:45 -04:00
William Falcon	5a834c794b	Update README.md	2019-08-14 07:19:58 -04:00
Ir1dXD	f0af138675	docs: enable syntax highlight (#109 )	2019-08-13 16:19:58 -04:00
William Falcon	3dea127edb	updated docs	2019-08-13 13:05:47 -04:00
William Falcon	d4b1ac94a0	updated docs	2019-08-13 13:03:39 -04:00
William Falcon	087be2f1c4	Merge branch 'master' of https://github.com/williamFalcon/pytorch-lightning	2019-08-13 13:02:21 -04:00
William Falcon	b89b7f0a8c	updated docs	2019-08-13 13:02:17 -04:00
William Falcon	bb75cec076	Update README.md	2019-08-13 12:56:12 -04:00
William Falcon	1cd5dde164	Update README.md	2019-08-13 12:11:24 -04:00
William Falcon	b02f4a4ccf	Update README.md	2019-08-13 12:10:08 -04:00
William Falcon	699fbabda7	updated optimizer_step docs	2019-08-13 11:59:33 -04:00
William Falcon	fd845d41c0	updated optimizer_step docs	2019-08-13 11:57:02 -04:00
William Falcon	d7660d3c64	updated optimizer_step docs	2019-08-13 11:55:10 -04:00
William Falcon	7e38f1f246	updated optimizer_step docs	2019-08-13 11:54:19 -04:00
William Falcon	7898d0c02a	updated optimizer_step docs	2019-08-13 11:51:31 -04:00