release v0.4.8

Expectopatronum implement #89 (#182 )
* rename validate -> evaluate; implement test logic; allow multiple test_loaders * add test_step and test_end to LightningModule * add in_test_mode to pretraining to implement case 2 (test pretrained model) * fix code style issues * LightningTestModel: add optional second test set, implement test_step and test_end * implemented test for multiple test_dataloaders; fixed typo * add two test cases for #89 * add documentation for test_step, test_end; fix computation of loss in validation_step example * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Added proper dp ddp routing calls for test mode * Update trainer.py * Update test_models.py * Update trainer.py * Update trainer.py * Update override_data_parallel.py * Update test_models.py * Update test_models.py * Update trainer.py * Update trainer.py * Update trainer.py * Update test_models.py * Update test_models.py * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * Update trainer.py * Update override_data_parallel.py * Update debug.py * Update lm_test_module.py * Update test_models.py
2026-09-14 11:33:33 +08:00 · 2019-08-31 03:05:57 -04:00 · 2019-08-30 18:56:09 -04:00 · 2019-08-30 10:56:14 -04:00 · 2019-08-28 06:41:31 -04:00 · 2019-08-27 18:07:33 -04:00
32 changed files with 1178 additions and 312 deletions
@@ -1,4 +1,7 @@
-#see https://github.com/codecov/support/wiki/Codecov-Yaml
+# see https://docs.codecov.io/docs/codecov-yaml
+# Validation check:
+# $ curl --data-binary @.codecov.yml https://codecov.io/validate
+
 codecov:
  notify:
    require_ci_to_pass: yes
@@ -41,4 +44,3 @@ comment:
  behavior: default  # update if exists else create new
  # branches: *

-
@@ -1,5 +1,10 @@
 # Contributing    
-Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!  
+Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!   
+
+## One less thing to remember    
+Simplify the API as much as possible from the user perspective. Any additions or improvements should minimize things the user needs to remember.   
+
+For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.   

 ## Lightning Design Principles   
 We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
@@ -12,8 +12,10 @@
 [![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
 [![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning)
 [![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
-[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)
+[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
+
 [![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
+[![Gitter](https://badges.gitter.im/PyTorch-Lightning/community.svg)](https://gitter.im/PyTorch-Lightning/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
 <!-- 
 removed until codecov badge isn't empy. likely a config error showing nothing on master.
@@ -31,15 +33,15 @@ pip install pytorch-lightning
 **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**

 ## What is it?  
-Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. It defers core training and validation logic to you and automates the rest. It guarantees tested, correct, modern best practices for the automated parts.
+Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.

+## Starting a new project?   
+[Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     

 ## Why do I want to use lightning?
-When starting a new project the last thing you want to do is recode a training loop, multi-cluster training, 16-bit precision, early-stopping, model loading/saving, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.
+Every research project starts the same, a model, a training loop, validation loop, etc. As your research advances, you're likely to need distributed training, 16-bit precision, checkpointing, gradient accumulation, etc.   

-With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: The data and the training/validation loop logic. 
-
-Don't worry about training on multiple gpus or speeding up your code, lightning will do that for you!
+Lightning sets up all the boilerplate state-of-the-art training for you so you can focus on the research.   

 ---   
 ## README Table of Contents        
@@ -51,14 +53,15 @@ Don't worry about training on multiple gpus or speeding up your code, lightning
 - [Tutorials](https://github.com/williamFalcon/pytorch-lightning#tutorials)
 - [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md)    
 - [Bleeding edge install](https://github.com/williamFalcon/pytorch-lightning#bleeding-edge)   
- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)
+- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)   
+- [Asking for help](https://github.com/williamFalcon/pytorch-lightning#asking-for-help)
 - [FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    

 ---   
 ## How do I do use it?   
-The research code goes into a [LightningModule]((https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)) which you fit using a Trainer.    
+Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule]((https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)) which you fit using a Trainer.    

-Think of the LightningModule as a *system* such as seq-2-seq, GAN, etc... However, the LightningModule can ALSO just be a simple classifier such as the example below.     
+The LightningModule defines a *system* such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier such as the example below.     

 To use lightning do 2 things:  
 1. [Define a LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
@@ -101,7 +104,8 @@ class CoolSystem(pl.LightningModule):

    def configure_optimizers(self):
        # REQUIRED
-        return [torch.optim.Adam(self.parameters(), lr=0.02)]
+        # can return multiple optimizers and learning_rate schedulers
+        return torch.optim.Adam(self.parameters(), lr=0.02)

    @pl.data_loader
    def tng_dataloader(self):
@@ -162,7 +166,7 @@ You define the blue parts using the LightningModule interface:

 ![Ouverview](./docs/source/_static/overview_flat.jpg)

-```{.python}
+```python
 # what to do in the training loop
 def training_step(self, data_batch, batch_nb):

@@ -249,7 +253,7 @@ Lightning also adds a text column with all the hyperparameters for this experime
 ![tensorboard-support](./docs/source/_static/tf_tags.png)

 Simply note the path you set for the Experiment    
-``` {.python}   
+```python
 from test_tube import Experiment
 from pytorch-lightning import  Trainer

@@ -356,8 +360,24 @@ python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'
 - [9 key speed features in Pytorch-Lightning](https://towardsdatascience.com/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565)    
 - [SLURM, multi-node training with Lightning](https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd)     

+---    
+## Asking for help    
+Welcome to the Lightning community!   
+
+If you have any questions, feel free to:   
+1. [read the docs](https://williamfalcon.github.io/pytorch-lightning/).     
+2. [Search through the issues](https://github.com/williamFalcon/pytorch-lightning/issues?utf8=%E2%9C%93&q=my++question).      
+3. [Ask on stackoverflow](https://stackoverflow.com/questions/ask?guided=false) with the tag pytorch-lightning.   
+
+If no one replies to you quickly enough, feel free to post the stackoverflow link to our Gitter chat!   
+
+To chat with the rest of us visit our [gitter channel](https://gitter.im/PyTorch-Lightning/community?utm_source=share-link&utm_medium=link&utm_campaign=share-link)!     
+
 ---   
 ## FAQ    
+**How do I use Lightning for rapid research?**   
+[Here's a walk-through](https://williamfalcon.github.io/pytorch-lightning/)  
+
 **Why was Lightning created?**     
 Lightning has 3 goals in mind:
 1. Maximal flexibility while abstracting out the common boilerplate across research projects.   
@@ -16,7 +16,9 @@ Otherwise, to Define a Lightning Module, implement the following methods:
 **Optional**:   

 - [validation_step](RequiredTrainerInterface.md#validation_step)    
- [validation_end](RequiredTrainerInterface.md#validation_end)    
+- [validation_end](RequiredTrainerInterface.md#validation_end) 
+- [test_step](RequiredTrainerInterface.md#test_step)    
+- [test_end](RequiredTrainerInterface.md#test_end) 
 - [val_dataloader](RequiredTrainerInterface.md#val_dataloader)    
 - [test_dataloader](RequiredTrainerInterface.md#test_dataloader)    
 - [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)    
@@ -50,19 +52,30 @@ class CoolModel(pl.LightningModule):
        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
-        return {'loss': F.cross_entropy(y_hat, y)(y_hat, y)}
+        return {'loss': F.cross_entropy(y_hat, y)}

    def validation_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
-        return {'val_loss': F.cross_entropy(y_hat, y)(y_hat, y)}
+        return {'val_loss': F.cross_entropy(y_hat, y)}

    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}

+    def test_step(self, batch, batch_nb):
+        # OPTIONAL
+        x, y = batch
+        y_hat = self.forward(x)
+        return {'test_loss': F.cross_entropy(y_hat, y)}
+
+    def test_end(self, outputs):
+        # OPTIONAL
+        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
+        return {'avg_test_loss': avg_loss}
+
    def configure_optimizers(self):
        # REQUIRED
        return [torch.optim.Adam(self.parameters(), lr=0.02)]
@@ -80,7 +93,8 @@ class CoolModel(pl.LightningModule):
    @pl.data_loader
    def test_dataloader(self):
        # OPTIONAL
-        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
+        # can also return a list of test dataloaders
+        return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
 ---   
 ### How do these methods fit into the broader training?     
@@ -189,8 +203,11 @@ Lightning will call .backward() and .step() on each one in every epoch.  If you



-##### Return
-List or Tuple - List of optimizers with an optional second list of learning-rate schedulers
+##### Return    
+Return any of these 3 options:   
+Single optimizer   
+List or Tuple - List of optimizers    
+Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers

 **Example**

@@ -198,9 +215,15 @@ List or Tuple - List of optimizers with an optional second list of learning-rate
 # most cases
 def configure_optimizers(self):
    opt = Adam(self.parameters(), lr=0.01)
-    return [opt]
+    return opt
+
+# multiple optimizer case (eg: GAN)
+def configure_optimizers(self):
+    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
+    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
+    return generator_opt, disriminator_opt
    
-# gan example, with scheduler for discriminator
+# example with learning_rate schedulers  
 def configure_optimizers(self):
    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
@@ -216,17 +239,18 @@ the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/h
 ### validation_step

 ``` {.python}
+# if you have one val dataloader:
 def validation_step(self, data_batch, batch_nb)   

-# if have multiple val dataloaders:  
+# if you have multiple val dataloaders:  
 def validation_step(self, data_batch, batch_nb, dataloader_idx)
 ```
 **OPTIONAL**    
 If you don't need to validate you don't need to implement this method.    

-In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes, calculate accuracy, or save example outputs (using self.experiment or whatever you want). Really, anything you want.       
+In this step you'd normally generate examples or calculate anything of interest such as accuracy. 

-This is most likely the same as your training_step. But unlike training step, the outputs from here will go to validation_end for collation.
+The dict you return here will be available in the `validation_end` method. 

 **Params**   

@@ -240,18 +264,24 @@ This is most likely the same as your training_step. But unlike training step, th

 | Return  | description  | optional |
 |---|---|---|   
-|  dict | Dict of OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
+|  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |

 **Example**

 ``` {.python}
 # CASE 1: A single validation dataset
 def validation_step(self, data_batch, batch_nb):
-    x, y, z = data_batch
+    x, y = data_batch
    
    # implement your own
    out = self.forward(x)
-    loss = self.loss(out, x)
+    loss = self.loss(out, y)
+    
+    # log 6 example images
+    # or generated text... or whatever
+    sample_imgs = x[:6]
+    grid = torchvision.utils.make_grid(sample_imgs)
+    self.experiment.add_image('example_images', grid, 0) 
    
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
@@ -323,6 +353,119 @@ def validation_end(self, outputs):
    return tqdm_dic
 ```

+### test_step
+
+``` {.python}
+# if you have one test dataloader:
+def test_step(self, data_batch, batch_nb)   
+
+# if you have multiple test dataloaders:  
+def test_step(self, data_batch, batch_nb, dataloader_idx)
+```
+**OPTIONAL**    
+If you don't need to test you don't need to implement this method.    
+
+In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
+
+The dict you return here will be available in the `test_end` method. 
+
+This function is used when you execute `trainer.test()`.
+
+**Params**   
+
+| Param  | description  |
+|---|---|
+|  data_batch | The output of your dataloader. A tensor, tuple or list  |
+|  batch_nb | Integer displaying which batch this is  |
+|  dataloader_i | Integer displaying which dataloader this is (only if multiple test datasets used)  |
+
+**Return**   
+
+| Return  | description  | optional |
+|---|---|---|   
+|  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
+
+**Example**
+
+``` {.python}
+# CASE 1: A single test dataset
+def test_step(self, data_batch, batch_nb):
+    x, y = data_batch
+    
+    # implement your own
+    out = self.forward(x)
+    loss = self.loss(out, y)
+    
+    # calculate acc
+    labels_hat = torch.argmax(out, dim=1)
+    test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
+    
+    # all optional...
+    # return whatever you need for the collation function test_end
+    output = OrderedDict({
+        'test_loss': loss_test,
+        'test_acc': torch.tensor(test_acc), # everything must be a tensor
+    })
+    
+    # return an optional dict
+    return output
+```   
+
+If you pass in multiple test datasets, test_step will have an additional argument.
+
+```python
+# CASE 2: multiple test datasets
+def test_step(self, data_batch, batch_nb, dataset_idx):
+    # dataset_idx tells you which dataset this is.   
+```   
+
+The ```dataset_idx``` corresponds to the order of datasets returned in ```test_dataloader```.    
+
+--- 
+### test_end
+
+``` {.python}
+def test_end(self, outputs)
+```   
+If you didn't define a test_step, this won't be called.       
+
+Called at the end of the test step with the output of each test_step.  Called once per test dataset.   
+
+The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
+
+**Params**    
+
+| Param  | description  |
+|---|---|
+|  outputs | List of outputs you defined test_step |
+
+**Return**   
+
+| Return  | description  | optional |
+|---|---|---|   
+|  dict | Dict of OrderedDict with metrics to display in progress bar | Y |
+
+**Example**
+
+``` {.python}
+def test_end(self, outputs):
+    """
+    Called at the end of test to aggregate outputs
+    :param outputs: list of individual outputs of each test step
+    :return:
+    """
+    test_loss_mean = 0
+    test_acc_mean = 0
+    for output in outputs:
+        test_loss_mean += output['test_loss']
+        test_acc_mean += output['test_acc']
+
+    test_loss_mean /= len(outputs)
+    test_acc_mean /= len(outputs)
+    tqdm_dic = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
+    return tqdm_dic
+```
+
 --- 
 ### on_save_checkpoint 

@@ -370,7 +513,7 @@ def on_load_checkpoint(self, checkpoint):

 ``` {.python}
@pl.data_loader
-def tng_dataloader(self)
+def val_dataloader(self)
 ```
 **OPTIONAL**    
 If you don't need a validation dataset and a validation_step, you don't need to implement this method.    
@@ -442,7 +585,7 @@ def test_dataloader(self):
 def update_tng_log_metrics(self, logs)
 ```
 Called by lightning right before it logs metrics for this batch.
-This is a chance to ammend or add to the metrics about to be logged.
+This is a chance to amend or add to the metrics about to be logged.

 ##### Return
 Dict 
@@ -10,7 +10,7 @@ Current dtype

 ---
 #### experiment    
-An instance of test-tube Experiment which you can use to log anything for tensorboarX.   
+An instance of test-tube Experiment which you can use to log anything for tensorboard (subclass of [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html)).   
 ```{.python}
 self.experiment.add_embedding(...)   
 self.experiment.log({'val_loss': 0.9})   
@@ -1,4 +1,4 @@
-i Lightning can automate saving and loading checkpoints.
+Lightning can automate saving and loading checkpoints.

 ---
 ### Model saving
@@ -38,6 +38,7 @@ trainer.fit(model)
 ```

 The trainer restores:  
+
 - global_step    
 - current_epoch    
 - All optimizers    
@@ -5,7 +5,7 @@ Lighting offers a few options for logging information about model, gpu usage, et
 #### Display metrics in progress bar 
 ``` {.python}
 # DEFAULT
-trainer = Trainer(progress_bar=True)
+trainer = Trainer(show_progress_bar=True)
 ```

 ---
@@ -54,7 +54,10 @@ trainer = Trainer(track_grad_norm=2)

 ---
 #### Set how much of the training set to check
-If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag
+If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
+
+train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
 ``` {.python}
 # DEFAULT
 trainer = Trainer(train_percent_check=1.0)
@@ -5,8 +5,6 @@ Below are all the things lightning automates for you in the validation loop.
 Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don't have to wait until a full epoch to catch possible validation issues.


-
-
 ---
 #### Check validation every n epochs
 If you have a small dataset you might want to check validation every n epochs
@@ -18,6 +16,9 @@ trainer = Trainer(check_val_every_n_epoch=1)
 ---
 #### Set how much of the validation set to check 
 If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
+
+val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_percent_check=1.0)
@@ -29,6 +30,9 @@ trainer = Trainer(val_percent_check=0.1)
 ---
 #### Set how much of the test set to check 
 If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
+
+test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
+
 ``` {.python}
 # DEFAULT
 trainer = Trainer(test_percent_check=1.0)
@@ -54,4 +58,6 @@ Lightning runs a few steps of validation in the beginning of training. This avoi
 ``` {.python}
 # DEFAULT
 trainer = Trainer(nb_sanity_val_steps=5)
-```
+```
+
+You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.
@@ -23,6 +23,9 @@ trainer = Trainer(track_grad_norm=2)
 ---
 #### Make model overfit on subset of data
 A useful debugging trick is to make your model overfit a tiny fraction of the data.
+
+setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
+
 ``` {.python}
 # DEFAULT don't overfit (ie: normal training)
 trainer = Trainer(overfit_pct=0.0)
@@ -97,6 +97,23 @@ def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
    # add as many optimizers as you want 
 ```

+This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:   
+
+```python
+# learning rate warm-up
+def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
+    # warm up lr
+    if self.trainer.global_step < 500:
+        lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
+        for pg in optimizer.param_groups:
+            pg['lr'] = lr_scale * self.hparams.learning_rate
+    
+    # update params
+    optimizer.step()
+    optimizer.zero_grad() 
+```
+
+
 ---
 #### on_before_zero_grad
 Called in the training loop after taking an optimizer step and before zeroing grads.
@@ -40,7 +40,7 @@ The main function should have 3 arguments:
 - slurm_manager: Slurm cluster manager object (can be None)
 - dict: for you to return any values you want (useful in meta-learning, otherwise set to _)    

-```{}
+```python
 def main(hparams, cluster, results_dict):
    """
    Main training routine specific for this project
@@ -1,12 +1,63 @@
 ###### New project Quick Start    
-To start a new project you define two files, a LightningModule and a Trainer file.    
+To start a new project define two files, a LightningModule and a Trainer file.    
+To illustrate Lightning power and simplicity, here's an example of a typical research flow.    

-A separate trainer file allows to run many LightningModules. Each LightningModule has the core
-logic to a particular research project.    
+###### Case 1: BERT    
+Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
+You would define a single LightningModule and use flags to switch between your different ideas.   
+```python
+class BERT(pl.LightningModule):
+    def __init__(self, model_name, task):
+        self.task = task
+    
+        if model_name == 'transformer':
+            self.net = Transformer()
+        elif model_name == 'my_cool_version':
+            self.net = MyCoolVersion()
+            
+    def training_step(self, batch, batch_nb):
+        if self.task == 'standard_bert':
+            # do standard bert training with self.net...
+            # return loss
+            
+        if self.task == 'my_cool_task':
+            # do my own version with self.net
+            # return loss
+```   

-For example, one lightningModule could be an image classifier, the other
-one could be a seq-2-seq model, both (optionally) ran by the same trainer file.
+###### Case 2: COOLER NOT BERT    
+But if you wanted to try something **completely** different, you'd define a new module for that.    
+```python

+class CoolerNotBERT(pl.LightningModule):
+    def __init__(self):
+        self.net = ...
+        
+    def training_step(self, batch, batch_nb):
+        # do some other cool task
+        # return loss   
+```   
+
+###### Rapid research flow    
+Then you could do rapid research by switching between these two and using the same trainer.   
+```python
+
+if use_bert:
+    model = BERT()
+else:
+    model = CoolerNotBERT()
+    
+trainer = Trainer(gpus=[0, 1, 2, 3], use_amp=True)
+trainer.fit(model)
+```
+
+Notice a few things about this flow:   
+1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
+2. You get free GPU and 16-bit support without writing any of that code in your model.   
+3. You also get all of the capabilities below (without coding or testing yourself).     
+
+---    
+###### Templates 
 1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
 2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_cpu_template.py) 
@@ -161,9 +161,9 @@ class LightningTemplateModel(LightningModule):
            # reduce manually when using dp
            val_acc = output['val_acc']
            if self.trainer.use_dp:
-                val_acc_mean = torch.mean(val_acc)
+                val_acc = torch.mean(val_acc)

-            val_acc_mean += val_acc_mean
+            val_acc_mean += val_acc

        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
@@ -189,16 +189,13 @@ class LightningTemplateModel(LightningModule):
        dataset = MNIST(root=self.hparams.data_root, train=train,
                        transform=transform, download=True)

-        # when using multi-node we need to add the datasampler
+        # when using multi-node (ddp) we need to add the datasampler
        train_sampler = None
        batch_size = self.hparams.batch_size

-        try:
-            if self.on_gpu:
-                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
-                batch_size = batch_size // self.trainer.world_size  # scale batch size
-        except Exception:
-            pass
+        if self.use_ddp:
+            train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
+            batch_size = batch_size // self.trainer.world_size  # scale batch size

        should_shuffle = train_sampler is None
        loader = DataLoader(
@@ -110,5 +110,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -110,5 +110,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -109,5 +109,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -2,6 +2,7 @@ site_name: PyTorch lightning Documentation
 theme:
  name: 'material'
 docs_dir: docs
+repo_name: 'williamFalcon/pytorch-lightning'
 repo_url: https://github.com/williamFalcon/pytorch-lightning
 site_dir: 'site'
 site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
@@ -1,6 +1,7 @@
-from .pt_callbacks import EarlyStopping, ModelCheckpoint
+from .pt_callbacks import EarlyStopping, ModelCheckpoint, GradientAccumulationScheduler

 __all__ = [
    'EarlyStopping',
    'ModelCheckpoint',
+    'GradientAccumulationScheduler',
 ]
@@ -1,5 +1,6 @@
 import os
 import shutil
+import warnings

 import numpy as np

@@ -254,6 +255,37 @@ class ModelCheckpoint(Callback):
                self.save_model(filepath, overwrite=False)


+class GradientAccumulationScheduler(Callback):
+    """Change gradient accumulation factor according to scheduling.
+    # Arguments
+        scheduling: dict, scheduling in format {epoch: accumulation_factor}
+    """
+    def __init__(self, scheduling: dict):
+        if scheduling == {}:  # empty dict error
+            raise TypeError("Empty dict cannot be interpreted correct")
+
+        for key in scheduling.keys():
+            if not isinstance(key, int) or not isinstance(scheduling[key], int):
+                raise TypeError("All epoches and accumulation factor must be integers")
+
+        minimal_epoch = min(scheduling.keys())
+        if minimal_epoch < 1:
+            msg = f"Epochs indexing from 1, epoch {minimal_epoch} cannot be interpreted correct"
+            raise IndexError(msg)
+        elif minimal_epoch != 1:  # if user didnt define first epoch accumulation factor
+            scheduling.update({1: 1})
+
+        self.scheduling = scheduling
+        self.epochs = sorted(scheduling.keys())
+
+    def on_epoch_begin(self, epoch, trainer):
+        epoch += 1  # indexing epochs from 1
+        for i in reversed(range(len(self.epochs))):
+            if epoch >= self.epochs[i]:
+                trainer.accumulate_grad_batches = self.scheduling.get(self.epochs[i])
+                break
+
+
 if __name__ == '__main__':
    c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
    losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
@@ -14,12 +14,13 @@ import torch.multiprocessing as mp
 import torch.distributed as dist
 from torch.optim.optimizer import Optimizer

-from pytorch_lightning.root_module.root_module import LightningModule
 from pytorch_lightning.root_module.memory import get_gpu_memory_map
 from pytorch_lightning.root_module.model_saving import TrainerIO
 from pytorch_lightning.pt_overrides.override_data_parallel import (
    LightningDistributedDataParallel, LightningDataParallel)
+from pytorch_lightning.callbacks import GradientAccumulationScheduler
 from pytorch_lightning.utilities.debugging import MisconfigurationException
+import pdb

 try:
    from apex import amp
@@ -61,7 +62,7 @@ class Trainer(TrainerIO):
                 current_gpu_name=0,
                 nb_gpu_nodes=1,
                 gpus=None,
-                 progress_bar=True,
+                 show_progress_bar=True,
                 overfit_pct=0.0,
                 track_grad_norm=-1,
                 check_val_every_n_epoch=1,
@@ -72,7 +73,7 @@ class Trainer(TrainerIO):
                 train_percent_check=1.0,
                 val_percent_check=1.0,
                 test_percent_check=1.0,
-                 val_check_interval=0.95,
+                 val_check_interval=1.0,
                 log_save_interval=100,
                 add_log_row_interval=10,
                 distributed_backend='dp',
@@ -92,7 +93,7 @@ class Trainer(TrainerIO):
        :param current_gpu_name:
        :param nb_gpu_nodes:
        :param gpus:
-        :param progress_bar:
+        :param show_progress_bar:
        :param overfit_pct:
        :param track_grad_norm:
        :param check_val_every_n_epoch:
@@ -122,7 +123,6 @@ class Trainer(TrainerIO):
        self.track_grad_norm = track_grad_norm
        self.fast_dev_run = fast_dev_run
        self.on_gpu = gpus is not None and torch.cuda.is_available()
-        self.progress_bar = progress_bar
        self.experiment = experiment
        self.exp_save_path = None
        if self.experiment is not None:
@@ -139,7 +139,13 @@ class Trainer(TrainerIO):
        self.early_stop = early_stop_callback
        self.model = None
        self.max_nb_epochs = max_nb_epochs
-        self.accumulate_grad_batches = accumulate_grad_batches
+        if isinstance(accumulate_grad_batches, dict):
+            self.accumulation_scheduler = GradientAccumulationScheduler(accumulate_grad_batches)
+        elif isinstance(accumulate_grad_batches, int):
+            schedule = {1: accumulate_grad_batches}
+            self.accumulation_scheduler = GradientAccumulationScheduler(schedule)
+        else:
+            raise TypeError("Gradient accumulation supports only int and dict types")
        self.early_stop_callback = early_stop_callback
        self.min_nb_epochs = min_nb_epochs
        self.nb_sanity_val_steps = nb_sanity_val_steps
@@ -152,6 +158,7 @@ class Trainer(TrainerIO):
        self.use_ddp = False
        self.use_dp = False
        self.single_gpu = False
+        self.testing = False

        # training bookeeping
        self.total_batch_nb = 0
@@ -159,9 +166,9 @@ class Trainer(TrainerIO):
        self.avg_loss = 0
        self.batch_nb = 0
        self.tqdm_metrics = {}
-        self.nb_val_batches = None
-        self.nb_tng_batches = None
-        self.nb_test_batches = None
+        self.nb_val_batches = 0
+        self.nb_tng_batches = 0
+        self.nb_test_batches = 0

        # gpus come in as a string.
        # if gpus = -1 then use all available devices
@@ -223,11 +230,14 @@ class Trainer(TrainerIO):

        # training state
        self.optimizers = None
-        self.prog_bar = None
        self.global_step = 0
        self.current_epoch = 0
        self.total_batches = 0

+        # can't init progress bar here because starting a new process
+        # means the prog_bar won't survive pickling
+        self.show_progress_bar = show_progress_bar
+
        # logging
        self.log_save_interval = log_save_interval
        self.val_check_interval = val_check_interval
@@ -325,7 +335,7 @@ class Trainer(TrainerIO):
    @property
    def __tng_tqdm_dic(self):
        tqdm_dic = {
-            'tng_loss': '{0:.3f}'.format(self.avg_loss),
+            'loss': '{0:.3f}'.format(self.avg_loss),
            'epoch': '{}'.format(self.current_epoch),
            'batch_nb': '{}'.format(self.batch_nb),
        }
@@ -356,20 +366,20 @@ class Trainer(TrainerIO):

        # determine number of validation batches
        # val datasets could be none, 1 or 2+
-        self.nb_val_batches = 0
        if self.val_dataloader is not None:
            self.nb_val_batches = sum(len(dataloader) for dataloader in self.val_dataloader)
-
-        self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
-        self.nb_val_batches = max(1, self.nb_val_batches)
-        self.nb_val_batches = self.nb_val_batches
+            self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
+            self.nb_val_batches = max(1, self.nb_val_batches)

        # determine number of test batches
-        self.nb_test_batches = len(self.test_dataloader) if self.test_dataloader is not None else 0
-        self.nb_test_batches = int(self.nb_test_batches * self.test_percent_check)
+        if self.test_dataloader is not None:
+            self.nb_test_batches = sum(len(dataloader) for dataloader in self.test_dataloader)
+            self.nb_test_batches = int(self.nb_test_batches * self.test_percent_check)
+            self.nb_test_batches = max(1, self.nb_test_batches)

        # determine when to check validation
        self.val_check_batch = int(self.nb_tng_batches * self.val_check_interval)
+        self.val_check_batch = max(1, self.val_check_batch)

    def __add_tqdm_metrics(self, metrics):
        for k, v in metrics.items():
@@ -378,40 +388,45 @@ class Trainer(TrainerIO):

            self.tqdm_metrics[k] = v

-    def __validation_forward(self, model, data_batch, batch_i, dataloader_i):
+    def __evaluation_forward(self, model, data_batch, batch_i, dataloader_i, test=False):
        # make dataloader_i arg in validation_step optional
        args = [data_batch, batch_i]
-        if len(self.val_dataloader) > 1:
+
+        if test and len(self.test_dataloader) > 1:
            args.append(dataloader_i)

-        if self.use_ddp:
+        elif len(self.val_dataloader) > 1:
+            args.append(dataloader_i)
+
+        # handle DP, DDP forward
+        if self.use_ddp or self.use_dp:
            output = model(*args)
-        elif self.use_dp:
-            output = model(*args)
-        elif self.single_gpu:
-            # put inputs on gpu manually
+            return output
+
+        # CPU, single GPU
+        if self.single_gpu:
+            # for single GPU put inputs on gpu manually
            gpu_id = self.data_parallel_device_ids[0]
            data_batch = self.transfer_batch_to_gpu(data_batch, gpu_id)
            args[0] = data_batch

-            # do non dp, ddp step
-            output = model.validation_step(*args)
-
+        if test:
+            output = model.test_step(*args)
        else:
-            # CPU
            output = model.validation_step(*args)

        return output

-    def validate(self, model, dataloader, max_batches, dataloader_i):
+    def evaluate(self, model, dataloader, max_batches, dataloader_i, test=False):
        """
-        Run validation code
+        Run evaluation code
        :param model: PT model
        :param dataloader: PT dataloader
        :param max_batches: Scalar
+        :param dataloader_i:
+        :param test: boolean
        :return:
        """
-
        # enable eval mode
        model.zero_grad()
        model.eval()
@@ -428,29 +443,31 @@ class Trainer(TrainerIO):
            if data_batch is None:  # pragma: no cover
                continue

-            # stop short when on fast dev run
-            if max_batches is not None and batch_i >= max_batches:
+            # stop short when on fast_dev_run (sets max_batch=1)
+            if batch_i >= max_batches:
                break

            # -----------------
-            # RUN VALIDATION STEP
+            # RUN EVALUATION STEP
            # -----------------
-            output = self.__validation_forward(model, data_batch, batch_i, dataloader_i)
+            output = self.__evaluation_forward(model, data_batch, batch_i, dataloader_i,
+                                               test)

            # track outputs for collation
            outputs.append(output)

            # batch done
-            if self.progress_bar and self.prog_bar is not None:
-                self.prog_bar.update(1)
+            if self.show_progress_bar:
+                self.progress_bar.update(1)
+
+        eval_results = {}

        # give model a chance to do something with the outputs (and method defined)
-        val_results = {}
-        if self.__is_overriden('validation_end'):
-            if self.data_parallel:
-                val_results = model.module.validation_end(outputs)
-            else:
-                val_results = model.validation_end(outputs)
+        model = self.__get_model()
+        if test and self.__is_overriden('test_end'):
+            eval_results = model.test_end(outputs)
+        elif self.__is_overriden('validation_end'):
+            eval_results = model.validation_end(outputs)

        # enable train mode again
        model.train()
@@ -458,7 +475,7 @@ class Trainer(TrainerIO):
        # enable gradients to save memory
        torch.set_grad_enabled(True)

-        return val_results
+        return eval_results

    def get_dataloaders(self, model):
        """
@@ -466,62 +483,89 @@ class Trainer(TrainerIO):
        :param model:
        :return:
        """
-        self.tng_dataloader = model.tng_dataloader

+        self.tng_dataloader = model.tng_dataloader
        self.test_dataloader = model.test_dataloader
        self.val_dataloader = model.val_dataloader

        # handle returning an actual dataloader instead of a list of loaders
+        have_test_loaders = self.test_dataloader is not None
+        if have_test_loaders and not isinstance(self.test_dataloader, list):
+            self.test_dataloader = [self.test_dataloader]
+
        have_val_loaders = self.val_dataloader is not None
        if have_val_loaders and not isinstance(self.val_dataloader, list):
            self.val_dataloader = [self.val_dataloader]

        if self.use_ddp and not isinstance(self.tng_dataloader.sampler, DistributedSampler):
            msg = """
-You're using multiple gpus and multiple nodes without using a DistributedSampler
-to assign a subset of your data to each process. To silence this warning, pass a
-DistributedSampler to your DataLoader.
+            You're using multiple gpus and multiple nodes without using a DistributedSampler
+            to assign a subset of your data to each process. To silence this warning, pass a
+            DistributedSampler to your DataLoader.

-ie: this:
-dataset = myDataset()
-dataloader = Dataloader(dataset)
+            ie: this:
+            dataset = myDataset()
+            dataloader = Dataloader(dataset)

-becomes:
-dataset = myDataset()
-dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-dataloader = Dataloader(dataset, sampler=dist_sampler)
+            becomes:
+            dataset = myDataset()
+            dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+            dataloader = Dataloader(dataset, sampler=dist_sampler)

-If you want each process to load the full dataset, ignore this warning.
-"""
+            If you want each process to load the full dataset, ignore this warning.
+            """
            warnings.warn(msg)

-        if self.use_ddp and\
-                not all(isinstance(dataloader, DistributedSampler)
-                        for dataloader in self.val_dataloader):
-            msg = """
-You're val_dataloader(s) are not all DistributedSamplers.
-You're using multiple gpus and multiple nodes without using a DistributedSampler
-to assign a subset of your data to each process. To silence this warning, pass a
-DistributedSampler to your DataLoader.
+        if self.use_ddp and self.val_dataloader is not None:
+            for dataloader in self.val_dataloader:
+                if not isinstance(dataloader, DistributedSampler):
+                    msg = """
+                    Your val_dataloader(s) are not all DistributedSamplers.
+                    You're using multiple gpus and multiple nodes without using a DistributedSampler
+                    to assign a subset of your data to each process. To silence this warning, pass a
+                    DistributedSampler to your DataLoader.

-ie: this:
-dataset = myDataset()
-dataloader = Dataloader(dataset)
+                    ie: this:
+                    dataset = myDataset()
+                    dataloader = Dataloader(dataset)

-becomes:
-dataset = myDataset()
-dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-dataloader = Dataloader(dataset, sampler=dist_sampler)
+                    becomes:
+                    dataset = myDataset()
+                    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+                    dataloader = Dataloader(dataset, sampler=dist_sampler)

-If you want each process to load the full dataset, ignore this warning.
-"""
-            warnings.warn(msg)
+                    If you want each process to load the full dataset, ignore this warning.
+                    """
+                    warnings.warn(msg)
+                    break
+
+        if self.use_ddp and self.test_dataloader is not None:
+            for dataloader in self.test_dataloader:
+                if not isinstance(dataloader, DistributedSampler):
+                    msg = """
+                    Your test_dataloader(s) are not all DistributedSamplers.
+                    You're using multiple gpus and multiple nodes without using a DistributedSampler
+                    to assign a subset of your data to each process. To silence this warning, pass a
+                    DistributedSampler to your DataLoader.
+
+                    ie: this:
+                    dataset = myDataset()
+                    dataloader = Dataloader(dataset)
+
+                    becomes:
+                    dataset = myDataset()
+                    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+                    dataloader = Dataloader(dataset, sampler=dist_sampler)
+
+                    If you want each process to load the full dataset, ignore this warning.
+                    """
+                    warnings.warn(msg)
+                    break

    # -----------------------------
    # MODEL TRAINING
    # -----------------------------
    def fit(self, model):
-
        # when using multi-node or DDP within a node start each module in a separate process
        if self.use_ddp:
            # must copy only the meta of the exp so it survives pickle/unpickle
@@ -646,7 +690,7 @@ If you want each process to load the full dataset, ignore this warning.
            self.experiment = self.experiment.get_non_ddp_exp()

        # show progbar only on prog_rank 0
-        self.prog_bar = self.prog_bar and self.node_rank == 0 and gpu_nb == 0
+        self.show_progress_bar = self.show_progress_bar and self.node_rank == 0 and gpu_nb == 0

        # determine which process we are and world size
        self.proc_rank = self.node_rank * len(self.data_parallel_device_ids) + gpu_nb
@@ -733,10 +777,15 @@ If you want each process to load the full dataset, ignore this warning.
        if self.data_parallel:
            ref_model = model.module

+        # give model convenience properties
        ref_model.trainer = self

        # set local properties on the model
        ref_model.on_gpu = self.on_gpu
+        ref_model.use_dp = self.use_dp
+        ref_model.use_ddp = self.use_ddp
+        ref_model.use_amp = self.use_amp
+        ref_model.testing = self.testing

        # transfer data loaders from model
        self.get_dataloaders(ref_model)
@@ -748,15 +797,13 @@ If you want each process to load the full dataset, ignore this warning.
        if self.proc_rank == 0 and self.print_weights_summary:
            ref_model.summarize()

-        # give model convenience properties
-        ref_model.trainer = self
-
+        # link up experiment object
        if self.experiment is not None:
            ref_model.experiment = self.experiment

-        # save exp to get started
-        if self.proc_rank == 0 and self.experiment is not None:
-            self.experiment.save()
+            # save exp to get started
+            if self.proc_rank == 0:
+                self.experiment.save()

        # track model now.
        # if cluster resets state, the model will update with the saved weights
@@ -771,11 +818,26 @@ If you want each process to load the full dataset, ignore this warning.
        if self.cluster is not None:  # pragma: no cover
            self.enable_auto_hpc_walltime_manager()

-        # run tiny validation (if validation defined) to make sure program won't crash during val
+        # progress bar init
+        if self.show_progress_bar:
+            self.progress_bar = tqdm.tqdm(0, position=self.process_position)
+
+        # when testing requested only run test and return
+        if self.testing:
+            self.__run_evaluation(test=True)
+            return
+
+        # run tiny validation (if validation defined)
+        # to make sure program won't crash during val
        ref_model.on_sanity_check_start()
-        if self.val_dataloader is not None:
+        if self.val_dataloader is not None and self.nb_sanity_val_steps > 0:
            for ds_i, dataloader in enumerate(self.val_dataloader):
-                self.validate(model, dataloader, self.nb_sanity_val_steps, ds_i)
+
+                # reset progress_bar limit for sanity check
+                if self.show_progress_bar:
+                    self.progress_bar.reset(self.nb_sanity_val_steps)
+
+                self.evaluate(model, dataloader, self.nb_sanity_val_steps, ds_i, self.testing)

        # ---------------------------
        # CORE TRAINING LOOP
@@ -794,10 +856,12 @@ If you want each process to load the full dataset, ignore this warning.
            self.total_batches = self.nb_tng_batches + self.nb_val_batches
            self.batch_loss_value = 0  # accumulated grads

-            # init progbar when requested
-            if self.progress_bar:
-                self.prog_bar = tqdm.tqdm(range(self.total_batches),
-                                          position=self.process_position)
+            # init progress_bar when requested
+            if self.show_progress_bar:
+                self.progress_bar.reset(self.total_batches)
+
+            # changing gradient according accumulation_scheduler
+            self.accumulation_scheduler.on_epoch_begin(epoch_nb, self)

            # -----------------
            # RUN TNG EPOCH
@@ -850,8 +914,10 @@ If you want each process to load the full dataset, ignore this warning.
            # RUN VAL STEP
            # ---------------
            is_val_check_batch = (batch_nb + 1) % self.val_check_batch == 0
+            can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
            if self.fast_dev_run or is_val_check_batch or early_stop_epoch:
-                self.__run_validation()
+                if can_check_epoch:
+                    self.__run_evaluation(test=self.testing)

            # when batch should be saved
            if (batch_nb + 1) % self.log_save_interval == 0 or early_stop_epoch:
@@ -896,6 +962,13 @@ If you want each process to load the full dataset, ignore this warning.
            model = self.__get_model()
            model.on_epoch_end()

+    def test(self, model=None):
+        if model is not None:
+            self.testing = True
+            self.fit(model)
+        else:
+            self.__run_evaluation(test=True)
+
    def __metrics_to_scalars(self, metrics, blacklist=set()):
        new_metrics = {}
        for k, v in metrics.items():
@@ -926,6 +999,13 @@ If you want each process to load the full dataset, ignore this warning.
                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
            return batch

+        # when tuple
+        elif isinstance(batch, tuple):
+            batch = list(batch)
+            for i, x in enumerate(batch):
+                batch[i] = self.transfer_batch_to_gpu(x, gpu_id)
+            return tuple(batch)
+
        # when dict
        elif isinstance(batch, dict):
            for k, v in batch.items():
@@ -933,6 +1013,9 @@ If you want each process to load the full dataset, ignore this warning.

            return batch

+        # nothing matches, return the value as is without transform
+        return batch
+
    def __tng_forward(self, data_batch, batch_nb, opt_idx):
        """
        Handle forward for each training case (distributed, single gpu, etc...)
@@ -1016,8 +1099,8 @@ If you want each process to load the full dataset, ignore this warning.
            if response == -1:
                return -1

-        if self.progress_bar:
-            self.prog_bar.update(1)
+        if self.show_progress_bar:
+            self.progress_bar.update(1)

        # call training_step once per optimizer
        for opt_idx, optimizer in enumerate(self.optimizers):
@@ -1066,10 +1149,10 @@ If you want each process to load the full dataset, ignore this warning.
                self.avg_loss = np.mean(self.running_loss[-100:])

                # update progbar
-                if self.progress_bar:
+                if self.show_progress_bar:
                    # add model specific metrics
                    tqdm_metrics = self.__tng_tqdm_dic
-                    self.prog_bar.set_postfix(**tqdm_metrics)
+                    self.progress_bar.set_postfix(**tqdm_metrics)

        # activate batch end hook
        if self.__is_function_implemented('on_batch_end'):
@@ -1078,41 +1161,58 @@ If you want each process to load the full dataset, ignore this warning.

        return 0

-    def __run_validation(self):
-        # decide if can check epochs
-        can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
-        if self.fast_dev_run:
-            print('skipping to check performance bc of --fast_dev_run')
-        elif not can_check_epoch:
-            return
+    def __run_evaluation(self, test=False):
+        # when testing make sure user defined a test step
+        can_run_test_step = False
+        if test:
+            can_run_test_step = self.__is_overriden('test_step') and self.__is_overriden('test_end')
+            if not can_run_test_step:
+                m = '''You called .test() without defining a test step or test_end.
+                Please define and try again'''
+                raise MisconfigurationException(m)

        # validate only if model has validation_step defined
-        if self.__is_overriden('validation_step'):
+        # test only if test_step or validation_step are defined
+        run_val_step = self.__is_overriden('validation_step')
+
+        if run_val_step or can_run_test_step:

            # hook
-            if self.__is_function_implemented('on_pre_performance_check'):
-                model = self.__get_model()
-                model.on_pre_performance_check()
+            model = self.__get_model()
+            model.on_pre_performance_check()

-            # use full val set on end of epoch
-            # use a small portion otherwise
-            max_batches = None if not self.fast_dev_run else 1
-            for ds_i, dataloader in enumerate(self.val_dataloader):
-                val_out_metrics = self.validate(self.model, dataloader, max_batches, ds_i)
-                self.__add_tqdm_metrics(val_out_metrics)
+            # select dataloaders
+            dataloaders = self.val_dataloader
+            max_batches = self.nb_val_batches

-            # hook
-            if self.__is_function_implemented('on_post_performance_check'):
-                model = self.__get_model()
+            # calculate max batches to use
+            if test:
+                dataloaders = self.test_dataloader
+                max_batches = self.nb_test_batches
+
+            # cap max batches to 1 when using fast_dev_run
+            if self.fast_dev_run:
+                max_batches = 1
+
+            for ds_i, dataloader in enumerate(dataloaders):
+                eval_out_metrics = self.evaluate(self.model,
+                                                 dataloader,
+                                                 max_batches,
+                                                 ds_i,
+                                                 test)
+
+                self.__add_tqdm_metrics(eval_out_metrics)
+
+                # hook
                model.on_post_performance_check()

-            if self.progress_bar:
+            if self.show_progress_bar:
                # add model specific metrics
                tqdm_metrics = self.__tng_tqdm_dic
-                self.prog_bar.set_postfix(**tqdm_metrics)
+                self.progress_bar.set_postfix(**tqdm_metrics)

        # model checkpointing
-        if self.proc_rank == 0 and self.checkpoint_callback is not None:
+        if self.proc_rank == 0 and self.checkpoint_callback is not None and not test:
            print('save callback...')
            self.checkpoint_callback.on_epoch_end(epoch=self.current_epoch,
                                                  logs=self.__tng_tqdm_dic)
@@ -56,6 +56,8 @@ class LightningDataParallel(DataParallel):
            # lightning
            if self.module.training:
                return self.module.training_step(*inputs[0], **kwargs[0])
+            elif self.module.testing:
+                return self.module.test_step(*inputs[0], **kwargs[0])
            else:
                return self.module.validation_step(*inputs[0], **kwargs[0])

@@ -89,6 +91,8 @@ class LightningDistributedDataParallel(DistributedDataParallel):
                # lightning
                if self.module.training:
                    output = self.module.training_step(*inputs[0], **kwargs[0])
+                elif self.module.testing:
+                    output = self.module.test_step(*inputs[0], **kwargs[0])
                else:
                    output = self.module.validation_step(*inputs[0], **kwargs[0])
            else:
@@ -153,6 +157,10 @@ def parallel_apply(modules, inputs, kwargs_tup=None, devices=None):  # pragma: n
                # CHANGE
                if module.training:
                    output = module.training_step(*input, **kwargs)
+
+                elif module.testing:
+                    output = module.test_step(*input, **kwargs)
+
                else:
                    output = module.validation_step(*input, **kwargs)
                # ---------------
@@ -1,3 +1,5 @@
+import traceback
+

 def data_loader(fn):
    """
@@ -10,8 +12,17 @@ def data_loader(fn):

    @property
    def _data_loader(self):
-        if not hasattr(self, attr_name):
-            setattr(self, attr_name, fn(self))
-        return getattr(self, attr_name)
+        try:
+            value = getattr(self, attr_name)
+        except AttributeError:
+            try:
+                value = fn(self)  # Lazy evaluation, done only once.
+            except AttributeError as e:
+                # Guard against AttributeError suppression. (Issue #142)
+                traceback.print_exc()
+                error = f'{fn.__name__}: An AttributeError was encountered: ' + str(e)
+                raise RuntimeError(error) from e
+            setattr(self, attr_name, value)  # Memoize evaluation.
+        return value

    return _data_loader
@@ -5,7 +5,7 @@ class ModelHooks(torch.nn.Module):

    def on_sanity_check_start(self):
        """
-        Called before starting validate
+        Called before starting evaluate
        :return:
        """
        pass
@@ -23,6 +23,9 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):

        # track if gpu was requested for checkpointing
        self.on_gpu = False
+        self.use_dp = False
+        self.use_ddp = False
+        self.use_amp = False

    def forward(self, *args, **kwargs):
        """
@@ -52,6 +55,16 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        pass

+    def test_step(self, *args, **kwargs):
+        """
+        return whatever outputs will need to be aggregated in test_end
+        OPTIONAL
+        :param called with batch, batch_nb
+        additional: dataset_i if multiple val datasets used
+        :return:
+        """
+        pass
+
    def validation_end(self, outputs):
        """
        Outputs has the appended output after each validation step
@@ -61,6 +74,15 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        pass

+    def test_end(self, outputs):
+        """
+        Outputs has the appended output after each test step
+        OPTIONAL
+        :param outputs:
+        :return: dic_with_metrics for tqdm
+        """
+        pass
+
    def configure_optimizers(self):
        """
        Return a list of optimizers and a list of schedulers (could be empty)
@@ -20,7 +20,7 @@ class LightningTestModel(LightningModule):
    Sample model to show how to define a template
    """

-    def __init__(self, hparams, force_remove_distributed_sampler=False):
+    def __init__(self, hparams, force_remove_distributed_sampler=False, use_two_test_sets=False):
        """
        Pass in parsed HyperOptArgumentParser to the model
        :param hparams:
@@ -28,6 +28,7 @@ class LightningTestModel(LightningModule):
        # init superclass
        super(LightningTestModel, self).__init__()
        self.hparams = hparams
+        self.use_two_test_sets = use_two_test_sets  # for some tests regarding testing

        self.batch_size = hparams.batch_size

@@ -167,12 +168,22 @@ class LightningTestModel(LightningModule):
        # if returned a scalar from validation_step, outputs is a list of tensor scalars
        # we return just the average in this case (if we want)
        # return torch.stack(outputs).mean()
-
        val_loss_mean = 0
        val_acc_mean = 0
        for output in outputs:
-            val_loss_mean += output['val_loss']
-            val_acc_mean += output['val_acc']
+            val_loss = output['val_loss']
+
+            # reduce manually when using dp
+            if self.trainer.use_dp:
+                val_loss = torch.mean(val_loss)
+            val_loss_mean += val_loss
+
+            # reduce manually when using dp
+            val_acc = output['val_acc']
+            if self.trainer.use_dp:
+                val_acc = torch.mean(val_acc)
+
+            val_acc_mean += val_acc

        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
@@ -180,6 +191,87 @@ class LightningTestModel(LightningModule):
        tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
        return tqdm_dic

+    def test_step(self, data_batch, batch_i, dataloader_i):
+        """
+        Lightning calls this inside the validation loop
+        :param data_batch:
+        :return:
+        """
+        x, y = data_batch
+        x = x.view(x.size(0), -1)
+        y_hat = self.forward(x)
+
+        loss_test = self.loss(y, y_hat)
+
+        # acc
+        labels_hat = torch.argmax(y_hat, dim=1)
+        test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
+        test_acc = torch.tensor(test_acc)
+
+        if self.on_gpu:
+            test_acc = test_acc.cuda(loss_test.device.index)
+
+        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
+        if self.trainer.use_dp:
+            loss_test = loss_test.unsqueeze(0)
+            test_acc = test_acc.unsqueeze(0)
+
+        # alternate possible outputs to test
+        if batch_i % 1 == 0:
+            output = OrderedDict({
+                'test_loss': loss_test,
+                'test_acc': test_acc,
+            })
+            return output
+        if batch_i % 2 == 0:
+            return test_acc
+
+        if batch_i % 3 == 0:
+            output = OrderedDict({
+                'test_loss': loss_test,
+                'test_acc': test_acc,
+                'test_dic': {'test_loss_a': loss_test}
+            })
+            return output
+        if batch_i % 5 == 0:
+            output = OrderedDict({
+                f'test_loss_{dataloader_i}': loss_test,
+                f'test_acc_{dataloader_i}': test_acc,
+            })
+            return output
+
+    def test_end(self, outputs):
+        """
+        Called at the end of validation to aggregate outputs
+        :param outputs: list of individual outputs of each validation step
+        :return:
+        """
+        # if returned a scalar from test_step, outputs is a list of tensor scalars
+        # we return just the average in this case (if we want)
+        # return torch.stack(outputs).mean()
+        test_loss_mean = 0
+        test_acc_mean = 0
+        for output in outputs:
+            test_loss = output['test_loss']
+
+            # reduce manually when using dp
+            if self.trainer.use_dp:
+                test_loss = torch.mean(test_loss)
+            test_loss_mean += test_loss
+
+            # reduce manually when using dp
+            test_acc = output['test_acc']
+            if self.trainer.use_dp:
+                test_acc = torch.mean(test_acc)
+
+            test_acc_mean += test_acc
+
+        test_loss_mean /= len(outputs)
+        test_acc_mean /= len(outputs)
+
+        tqdm_dic = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
+        return tqdm_dic
+
    def on_tng_metrics(self, logs):
        logs['some_tensor_to_test'] = torch.rand(1)

@@ -209,7 +301,7 @@ class LightningTestModel(LightningModule):
        batch_size = self.hparams.batch_size

        try:
-            if self.on_gpu and not self.force_remove_distributed_sampler:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
@@ -235,6 +327,8 @@ class LightningTestModel(LightningModule):

    @data_loader
    def test_dataloader(self):
+        if self.use_two_test_sets:
+            return [self.__dataloader(train=False), self.__dataloader(train=False)]
        return self.__dataloader(train=False)

    @staticmethod
@@ -181,7 +181,7 @@ class NoValEndTestModel(LightningModule):
        batch_size = self.hparams.batch_size

        try:
-            if self.on_gpu and not self.force_remove_distributed_sampler:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
@@ -138,7 +138,7 @@ class NoValModel(LightningModule):
        batch_size = self.hparams.batch_size

        try:
-            if self.on_gpu and not self.force_remove_distributed_sampler:
+            if self.use_ddp and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
        except Exception:
@@ -14,7 +14,7 @@ from setuptools import setup, find_packages
 # engineer specific practices
 setup(
    name='pytorch-lightning',
-    version='0.4.6',
+    version='0.4.8',
    description='The Keras for ML researchers using PyTorch',
    author='William Falcon',
    author_email='waf2107@columbia.edu',
@@ -30,7 +30,7 @@ setup(
    python_requires='>=3.6',
    install_requires=[
        'torch==1.2.0',
-        'tqdm',
+        'tqdm>=4.35.0',
        'test-tube>=0.6.9',
        'pandas>=0.20.3',
    ],
@@ -1,5 +1,6 @@
 from pytorch_lightning import Trainer
 from examples import LightningTemplateModel
+from pytorch_lightning.testing import LightningTestModel, NoValEndTestModel, NoValModel
 from argparse import Namespace
 from test_tube import Experiment
 from pytorch_lightning.callbacks import ModelCheckpoint
@@ -11,6 +12,8 @@ import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
+import numpy as np
+import pdb


 class CoolModel(pl.LightningModule):
@@ -72,10 +75,11 @@ def get_model():
    return model, hparams


-def get_exp(debug=True):
+def get_exp(debug=True, version=None):
    # set up exp object without actually saving logs
    root_dir = os.path.dirname(os.path.realpath(__file__))
-    exp = Experiment(debug=debug, save_dir=root_dir, name='tests_tt_dir')
+    save_dir = os.path.join(root_dir, 'save_dir')
+    exp = Experiment(debug=debug, save_dir=save_dir, name='tests_tt_dir', version=version)
    return exp


@@ -98,7 +102,7 @@ def clear_save_dir():
        shutil.rmtree(save_dir)


-def load_model(exp, save_dir):
+def load_model(exp, save_dir, on_gpu, map_location=None, module_class=LightningTemplateModel):

    # load trained model
    tags_path = exp.get_data_path(exp.name, exp.version)
@@ -107,8 +111,10 @@ def load_model(exp, save_dir):
    checkpoints = [x for x in os.listdir(save_dir) if '.ckpt' in x]
    weights_dir = os.path.join(save_dir, checkpoints[0])

-    trained_model = LightningTemplateModel.load_from_metrics(weights_path=weights_dir,
-                                                             tags_csv=tags_path, on_gpu=True)
+    trained_model = module_class.load_from_metrics(weights_path=weights_dir,
+                                                   tags_csv=tags_path,
+                                                   on_gpu=on_gpu,
+                                                   map_location=map_location)

    assert trained_model is not None, 'loading model failed'

@@ -130,47 +136,124 @@ def run_prediction(dataloader, trained_model):
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    val_acc = torch.tensor(val_acc)
    val_acc = val_acc.item()
-
-    print(val_acc)
-
    assert val_acc > 0.70, 'this model is expected to get > 0.7 in test set (it got %f)' % val_acc


-def main():
-
+# ------------------------------------------------------------------------
+def run_gpu_model_test(trainer_options, model, hparams, on_gpu=True):
    save_dir = init_save_dir()

    # exp file to get meta
    exp = get_exp(False)
+    exp.argparse(hparams)
    exp.save()

    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)

-    trainer = Trainer(
-        experiment=exp,
-        checkpoint_callback=checkpoint,
-        progress_bar=True,
-        max_nb_epochs=1,
-        gpus=[0, 1],
-        distributed_backend='dp',
-    )
-
-    model = CoolModel()
+    # add these to the trainer options
+    trainer_options['checkpoint_callback'] = checkpoint
+    trainer_options['experiment'] = exp

+    # fit model
+    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)

    # correct result and ok accuracy
    assert result == 1, 'amp + ddp model failed to complete'

    # test model loading
-    pretrained_model = load_model(exp, save_dir)
+    pretrained_model = load_model(exp, save_dir, on_gpu)

-    # test model preds
+    # test new model accuracy
    run_prediction(model.test_dataloader, pretrained_model)

+    if trainer.use_ddp:
+        # on hpc this would work fine... but need to hack it for the purpose of the test
+        trainer.model = pretrained_model
+        trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers()
+
+    # test HPC loading / saving
+    trainer.hpc_save(save_dir, exp)
+    trainer.hpc_load(save_dir, on_gpu=on_gpu)
+
    clear_save_dir()


+def assert_ok_val_acc(trainer):
+    # this model should get 0.80+ acc
+    acc = trainer.tng_tqdm_dic['val_acc']
+    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
+
+
+def assert_ok_test_acc(trainer):
+    # this model should get 0.80+ acc
+    acc = trainer.tng_tqdm_dic['test_acc']
+    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
+
+
+def get_hparams(continue_training=False, hpc_exp_number=0):
+    root_dir = os.path.dirname(os.path.realpath(__file__))
+
+    args = {
+        'drop_prob': 0.2,
+        'batch_size': 32,
+        'in_features': 28 * 28,
+        'learning_rate': 0.001 * 8,
+        'optimizer_name': 'adam',
+        'data_root': os.path.join(root_dir, 'mnist'),
+        'out_features': 10,
+        'hidden_dim': 1000}
+
+    if continue_training:
+        args['test_tube_do_checkpoint_load'] = True
+        args['hpc_exp_number'] = hpc_exp_number
+
+    hparams = Namespace(**args)
+    return hparams
+
+
+def main():
+    """Verify test() on fitted model"""
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    exp = get_exp(False)
+    exp.argparse(hparams)
+    exp.save()
+
+    # exp file to get weights
+    checkpoint = ModelCheckpoint(save_dir)
+
+    trainer_options = dict(
+        show_progress_bar=False,
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        checkpoint_callback=checkpoint,
+        experiment=exp,
+        gpus=[0, 1],
+        distributed_backend='ddp'
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # correct result and ok accuracy
+    assert result == 1, 'training failed to complete'
+    pretrained_model = load_model(exp, save_dir, on_gpu=True, module_class=LightningTestModel)
+
+    new_trainer = Trainer(**trainer_options)
+    new_trainer.test(pretrained_model)
+
+    # test we have good test accuracy
+    assert_ok_test_acc(new_trainer)
+    # clear_save_dir()
+
+
 if __name__ == '__main__':
    main()
@@ -11,7 +11,11 @@ from test_tube import Experiment, SlurmCluster
 # sys.path += [os.path.abspath('..'), os.path.abspath('../..')]
 from pytorch_lightning import Trainer
 from pytorch_lightning.testing import LightningTestModel, NoValEndTestModel, NoValModel
-from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
+from pytorch_lightning.callbacks import (
+    ModelCheckpoint,
+    EarlyStopping,
+    GradientAccumulationScheduler,
+)
 from pytorch_lightning.utilities.debugging import MisconfigurationException
 from pytorch_lightning.root_module import memory
 from pytorch_lightning.models.trainer import reduce_distributed_output
@@ -26,6 +30,270 @@ np.random.seed(SEED)
 # ------------------------------------------------------------------------
 # TESTS
 # ------------------------------------------------------------------------
+def test_running_test_pretrained_model_ddp():
+    """Verify test() on pretrained model"""
+    if not can_run_gpu_test():
+        return
+
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    exp = get_exp(False)
+    exp.argparse(hparams)
+    exp.save()
+
+    # exp file to get weights
+    checkpoint = ModelCheckpoint(save_dir)
+
+    trainer_options = dict(
+        show_progress_bar=False,
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        checkpoint_callback=checkpoint,
+        experiment=exp,
+        gpus=[0, 1],
+        distributed_backend='ddp'
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # correct result and ok accuracy
+    assert result == 1, 'training failed to complete'
+    pretrained_model = load_model(exp, save_dir, on_gpu=True, module_class=LightningTestModel)
+
+    # run test set
+    new_trainer = Trainer(**trainer_options)
+    new_trainer.test(pretrained_model)
+
+    run_prediction(model.test_dataloader, pretrained_model)
+
+    # test we have good test accuracy
+    clear_save_dir()
+
+
+def test_running_test_after_fitting():
+    """Verify test() on fitted model"""
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    exp = get_exp(False)
+    exp.argparse(hparams)
+    exp.save()
+
+    # exp file to get weights
+    checkpoint = ModelCheckpoint(save_dir)
+
+    trainer_options = dict(
+        show_progress_bar=False,
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        test_percent_check=0.2,
+        checkpoint_callback=checkpoint,
+        experiment=exp
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    assert result == 1, 'training failed to complete'
+
+    trainer.test()
+
+    # test we have good test accuracy
+    assert_ok_test_acc(trainer)
+
+    clear_save_dir()
+
+
+def test_running_test_pretrained_model():
+    """Verify test() on pretrained model"""
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    exp = get_exp(False)
+    exp.argparse(hparams)
+    exp.save()
+
+    # exp file to get weights
+    checkpoint = ModelCheckpoint(save_dir)
+
+    trainer_options = dict(
+        show_progress_bar=False,
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        checkpoint_callback=checkpoint,
+        experiment=exp
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # correct result and ok accuracy
+    assert result == 1, 'training failed to complete'
+    pretrained_model = load_model(exp, save_dir, on_gpu=False, module_class=LightningTestModel)
+
+    new_trainer = Trainer(**trainer_options)
+    new_trainer.test(pretrained_model)
+
+    # test we have good test accuracy
+    assert_ok_test_acc(new_trainer)
+    clear_save_dir()
+
+
+def test_running_test_pretrained_model_dp():
+    """Verify test() on pretrained model"""
+    if not can_run_gpu_test():
+        return
+
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    exp = get_exp(False)
+    exp.argparse(hparams)
+    exp.save()
+
+    # exp file to get weights
+    checkpoint = ModelCheckpoint(save_dir)
+
+    trainer_options = dict(
+        show_progress_bar=False,
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        checkpoint_callback=checkpoint,
+        experiment=exp,
+        gpus=[0, 1],
+        distributed_backend='dp'
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # correct result and ok accuracy
+    assert result == 1, 'training failed to complete'
+    pretrained_model = load_model(exp, save_dir, on_gpu=True, module_class=LightningTestModel)
+
+    new_trainer = Trainer(**trainer_options)
+    new_trainer.test(pretrained_model)
+
+    # test we have good test accuracy
+    assert_ok_test_acc(new_trainer)
+    clear_save_dir()
+
+
+def test_gradient_accumulation_scheduling():
+    """
+    Test grad accumulation by the freq of optimizer updates
+    """
+    # test incorrect configs
+    with pytest.raises(IndexError):
+        assert Trainer(accumulate_grad_batches={0: 3, 1: 4, 4: 6})
+        assert Trainer(accumulate_grad_batches={-2: 3})
+
+    with pytest.raises(TypeError):
+        assert Trainer(accumulate_grad_batches={})
+        assert Trainer(accumulate_grad_batches=[[2, 3], [4, 6]])
+        assert Trainer(accumulate_grad_batches={1: 2, 3.: 4})
+        assert Trainer(accumulate_grad_batches={1: 2.5, 3: 5})
+
+    # test optimizer call freq matches scheduler
+    def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i):
+        # only test the first 12 batches in epoch
+        if batch_nb < 12:
+            if epoch_nb == 0:
+                # reset counter when starting epoch
+                if batch_nb == 0:
+                    self.prev_called_batch_nb = 0
+
+                    # use this opportunity to test once
+                    assert self.trainer.accumulate_grad_batches == 1
+
+                assert batch_nb == self.prev_called_batch_nb
+                self.prev_called_batch_nb += 1
+
+            elif 1 <= epoch_nb <= 2:
+                # reset counter when starting epoch
+                if batch_nb == 1:
+                    self.prev_called_batch_nb = 1
+
+                    # use this opportunity to test once
+                    assert self.trainer.accumulate_grad_batches == 2
+
+                assert batch_nb == self.prev_called_batch_nb
+                self.prev_called_batch_nb += 2
+
+            else:
+                if batch_nb == 3:
+                    self.prev_called_batch_nb = 3
+
+                    # use this opportunity to test once
+                    assert self.trainer.accumulate_grad_batches == 4
+
+                assert batch_nb == self.prev_called_batch_nb
+                self.prev_called_batch_nb += 3
+
+        optimizer.step()
+
+        # clear gradients
+        optimizer.zero_grad()
+
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+    schedule = {1: 2, 3: 4}
+
+    trainer = Trainer(accumulate_grad_batches=schedule,
+                      train_percent_check=0.1,
+                      val_percent_check=0.1,
+                      max_nb_epochs=4)
+
+    # for the test
+    trainer.optimizer_step = optimizer_step
+    model.prev_called_batch_nb = 0
+
+    trainer.fit(model)
+
+
+def test_multi_gpu_model_ddp():
+    """
+    Make sure DDP works
+    :return:
+    """
+    if not can_run_gpu_test():
+        return
+
+    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
+    model, hparams = get_model()
+    trainer_options = dict(
+        show_progress_bar=False,
+        max_nb_epochs=1,
+        train_percent_check=0.4,
+        val_percent_check=0.2,
+        gpus=[0, 1],
+        distributed_backend='ddp'
+    )
+
+    run_gpu_model_test(trainer_options, model, hparams)
+

 def test_optimizer_return_options():

@@ -58,13 +326,7 @@ def test_optimizer_return_options():


 def test_single_gpu_batch_parse():
-    if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_ddp cannot run.'
-                      'Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_ddp cannot run.'
-                      'Rerun on a node with 2+ GPUs to run this test')
+    if not can_run_gpu_test():
        return

    trainer = Trainer()
@@ -92,6 +354,18 @@ def test_single_gpu_batch_parse():
    assert batch[0]['a'].device.index == 0 and batch[0]['a'].type() == 'torch.cuda.FloatTensor'
    assert batch[0]['b'].device.index == 0 and batch[0]['b'].type() == 'torch.cuda.FloatTensor'

+    # tuple of tensor list and list of tensor dict
+    batch = ([torch.rand(2, 3) for _ in range(2)],
+             [{'a': torch.rand(2, 3), 'b': torch.rand(2, 3)} for _ in range(2)])
+    batch = trainer.transfer_batch_to_gpu(batch, 0)
+    assert batch[0][0].device.index == 0 and batch[0][0].type() == 'torch.cuda.FloatTensor'
+
+    assert batch[1][0]['a'].device.index == 0
+    assert batch[1][0]['a'].type() == 'torch.cuda.FloatTensor'
+
+    assert batch[1][0]['b'].device.index == 0
+    assert batch[1][0]['b'].type() == 'torch.cuda.FloatTensor'
+

 def test_early_stopping_cpu_model():
    """
@@ -106,7 +380,7 @@ def test_early_stopping_cpu_model():
        overfit_pct=0.20,
        track_grad_norm=2,
        print_nan_grads=True,
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        train_percent_check=0.1,
        val_percent_check=0.1
@@ -146,7 +420,7 @@ def test_no_val_module():
    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)

-    # traning complete
+    # training complete
    assert result == 1, 'amp + ddp model failed to complete'

    # save model
@@ -253,7 +527,7 @@ def test_amp_single_gpu():
    model = LightningTestModel(hparams)

    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0],
        distributed_backend='dp',
@@ -334,13 +608,7 @@ def test_amp_gpu_ddp():
    Make sure DDP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_ddp cannot run.'
-                      'Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_ddp cannot run.'
-                      'Rerun on a node with 2+ GPUs to run this test')
+    if not can_run_gpu_test():
        return

    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
@@ -349,7 +617,7 @@ def test_amp_gpu_ddp():
    model = LightningTestModel(hparams)

    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0, 1],
        distributed_backend='ddp',
@@ -552,13 +820,7 @@ def test_amp_gpu_ddp_slurm_managed():
    Make sure DDP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_ddp cannot run.'
-                      ' Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_ddp cannot run.'
-                      ' Rerun on a node with 2+ GPUs to run this test')
+    if not can_run_gpu_test():
        return

    # simulate setting slurm flags
@@ -569,7 +831,7 @@ def test_amp_gpu_ddp_slurm_managed():
    model = LightningTestModel(hparams)

    trainer_options = dict(
-        progress_bar=True,
+        show_progress_bar=True,
        max_nb_epochs=1,
        gpus=[0],
        distributed_backend='ddp',
@@ -634,7 +896,7 @@ def test_cpu_model_with_amp():
    """

    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        max_nb_epochs=1,
        train_percent_check=0.4,
@@ -655,7 +917,7 @@ def test_cpu_model():
    """

    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        max_nb_epochs=1,
        train_percent_check=0.4,
@@ -678,7 +940,7 @@ def test_all_features_cpu_model():
        overfit_pct=0.20,
        track_grad_norm=2,
        print_nan_grads=True,
-        progress_bar=False,
+        show_progress_bar=False,
        experiment=get_exp(),
        accumulate_grad_batches=2,
        max_nb_epochs=1,
@@ -702,7 +964,7 @@ def test_single_gpu_model():
    model, hparams = get_model()

    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.1,
        val_percent_check=0.1,
@@ -717,17 +979,12 @@ def test_multi_gpu_model_dp():
    Make sure DP works
    :return:
    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_multi_gpu_model_dp cannot run.'
-                      ' Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_multi_gpu_model_dp cannot run.'
-                      ' Rerun on a node with 2+ GPUs to run this test')
+    if not can_run_gpu_test():
        return
+
    model, hparams = get_model()
    trainer_options = dict(
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        train_percent_check=0.1,
        val_percent_check=0.1,
@@ -745,14 +1002,9 @@ def test_amp_gpu_dp():
    Make sure DP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_dp cannot run.'
-                      ' Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_dp cannot run.'
-                      ' Rerun on a node with 2+ GPUs to run this test')
+    if not can_run_gpu_test():
        return
+
    model, hparams = get_model()
    trainer_options = dict(
        max_nb_epochs=1,
@@ -764,44 +1016,12 @@ def test_amp_gpu_dp():
        run_gpu_model_test(trainer_options, model, hparams)


-def test_multi_gpu_model_ddp():
-    """
-    Make sure DDP works
-    :return:
-    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_multi_gpu_model_ddp cannot run.'
-                      ' Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_multi_gpu_model_ddp cannot run.'
-                      ' Rerun on a node with 2+ GPUs to run this test')
-        return
-
-    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
-    model, hparams = get_model()
-    trainer_options = dict(
-        progress_bar=False,
-        max_nb_epochs=1,
-        train_percent_check=0.4,
-        val_percent_check=0.2,
-        gpus=[0, 1],
-        distributed_backend='ddp'
-    )
-
-    run_gpu_model_test(trainer_options, model, hparams)
-
-
 def test_ddp_sampler_error():
    """
    Make sure DDP + AMP work
    :return:
    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a node with 2+ GPUs to run this test')
+    if not can_run_gpu_test():
        return

    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
@@ -814,7 +1034,7 @@ def test_ddp_sampler_error():

    trainer = Trainer(
        experiment=exp,
-        progress_bar=False,
+        show_progress_bar=False,
        max_nb_epochs=1,
        gpus=[0, 1],
        distributed_backend='ddp',
@@ -835,7 +1055,34 @@ def test_multiple_val_dataloader():
    hparams = get_hparams()
    model = LightningTestModel(hparams)

-    save_dir = init_save_dir()
+    # exp file to get meta
+    trainer_options = dict(
+        max_nb_epochs=1,
+        val_percent_check=0.1,
+        train_percent_check=1.0,
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # verify tng completed
+    assert result == 1
+
+    # verify there are 2 val loaders
+    assert len(trainer.val_dataloader) == 2, 'Multiple val_dataloaders not initiated properly'
+
+    # make sure predictions are good for each val set
+    [run_prediction(dataloader, trainer.model) for dataloader in trainer.val_dataloader]
+
+
+def test_multiple_test_dataloader():
+    """
+    Verify multiple test_dataloader
+    :return:
+    """
+    hparams = get_hparams()
+    model = LightningTestModel(hparams, use_two_test_sets=True)

    # exp file to get meta
    trainer_options = dict(
@@ -852,10 +1099,10 @@ def test_multiple_val_dataloader():
    assert result == 1

    # verify there are 2 val loaders
-    assert len(trainer.val_dataloader) == 2, 'Multiple val_dataloaders not initiated properly'
+    assert len(trainer.test_dataloader) == 2, 'Multiple test_dataloaders not initiated properly'

-    # make sure predictions are good for each val set
-    [run_prediction(dataloader, trainer.model) for dataloader in trainer.val_dataloader]
+    # make sure predictions are good for each test set
+    [run_prediction(dataloader, trainer.model) for dataloader in trainer.test_dataloader]


 # ------------------------------------------------------------------------
@@ -886,7 +1133,7 @@ def run_gpu_model_test(trainer_options, model, hparams, on_gpu=True):
    # test model loading
    pretrained_model = load_model(exp, save_dir, on_gpu)

-    # test model preds
+    # test new model accuracy
    run_prediction(model.test_dataloader, pretrained_model)

    if trainer.use_ddp:
@@ -937,7 +1184,8 @@ def get_model(use_test_model=False):
 def get_exp(debug=True, version=None):
    # set up exp object without actually saving logs
    root_dir = os.path.dirname(os.path.realpath(__file__))
-    exp = Experiment(debug=debug, save_dir=root_dir, name='tests_tt_dir', version=version)
+    save_dir = os.path.join(root_dir, 'save_dir')
+    exp = Experiment(debug=debug, save_dir=save_dir, name='tests_tt_dir', version=version)
    return exp


@@ -946,7 +1194,8 @@ def init_save_dir():
    save_dir = os.path.join(root_dir, 'save_dir')

    if os.path.exists(save_dir):
-        shutil.rmtree(save_dir)
+        n = np.random.randint(0, 10000000, 1)[0]
+        shutil.move(save_dir, save_dir + f'_{n}')

    os.makedirs(save_dir, exist_ok=True)

@@ -957,10 +1206,11 @@ def clear_save_dir():
    root_dir = os.path.dirname(os.path.realpath(__file__))
    save_dir = os.path.join(root_dir, 'save_dir')
    if os.path.exists(save_dir):
-        shutil.rmtree(save_dir)
+        n = np.random.randint(0, 10000000, 1)[0]
+        shutil.move(save_dir, save_dir + f'_{n}')


-def load_model(exp, save_dir, on_gpu, map_location=None):
+def load_model(exp, save_dir, on_gpu, map_location=None, module_class=LightningTemplateModel):

    # load trained model
    tags_path = exp.get_data_path(exp.name, exp.version)
@@ -969,10 +1219,10 @@ def load_model(exp, save_dir, on_gpu, map_location=None):
    checkpoints = [x for x in os.listdir(save_dir) if '.ckpt' in x]
    weights_dir = os.path.join(save_dir, checkpoints[0])

-    trained_model = LightningTemplateModel.load_from_metrics(weights_path=weights_dir,
-                                                             tags_csv=tags_path,
-                                                             on_gpu=on_gpu,
-                                                             map_location=map_location)
+    trained_model = module_class.load_from_metrics(weights_path=weights_dir,
+                                                   tags_csv=tags_path,
+                                                   on_gpu=on_gpu,
+                                                   map_location=map_location)

    assert trained_model is not None, 'loading model failed'

@@ -991,19 +1241,35 @@ def run_prediction(dataloader, trained_model):

    # acc
    labels_hat = torch.argmax(y_hat, dim=1)
-    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-    val_acc = torch.tensor(val_acc)
-    val_acc = val_acc.item()
+    acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
+    acc = torch.tensor(acc)
+    acc = acc.item()

-    print(val_acc)
-
-    assert val_acc > 0.50, 'this model is expected to get > 0.50 in test set (it got %f)' % val_acc
+    assert acc > 0.50, f'this model is expected to get > 0.50 in test set (it got {acc})'


-def assert_ok_acc(trainer):
+def assert_ok_val_acc(trainer):
    # this model should get 0.80+ acc
    acc = trainer.tng_tqdm_dic['val_acc']
-    assert acc > 0.50, 'model failed to get expected 0.50 validation accuracy. Got: %f' % acc
+    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
+
+
+def assert_ok_test_acc(trainer):
+    # this model should get 0.80+ acc
+    acc = trainer.tng_tqdm_dic['test_acc']
+    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
+
+
+def can_run_gpu_test():
+    if not torch.cuda.is_available():
+        warnings.warn('test_multi_gpu_model_ddp cannot run.'
+                      ' Rerun on a GPU node to run this test')
+        return False
+    if not torch.cuda.device_count() > 1:
+        warnings.warn('test_multi_gpu_model_ddp cannot run.'
+                      ' Rerun on a node with 2+ GPUs to run this test')
+        return False
+    return True


 if __name__ == '__main__':
Author	SHA1	Message	Date
William Falcon	60384eb61e	release v0.4.8	2019-08-31 03:05:57 -04:00
William Falcon	f51b45933b	Expectopatronum implement #89 (#182 ) * rename validate -> evaluate; implement test logic; allow multiple test_loaders * add test_step and test_end to LightningModule * add in_test_mode to pretraining to implement case 2 (test pretrained model) * fix code style issues * LightningTestModel: add optional second test set, implement test_step and test_end * implemented test for multiple test_dataloaders; fixed typo * add two test cases for #89 * add documentation for test_step, test_end; fix computation of loss in validation_step example * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Added proper dp ddp routing calls for test mode * Update trainer.py * Update test_models.py * Update trainer.py * Update trainer.py * Update override_data_parallel.py * Update test_models.py * Update test_models.py * Update trainer.py * Update trainer.py * Update trainer.py * Update test_models.py * Update test_models.py * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * Update trainer.py * Update override_data_parallel.py * Update debug.py * Update lm_test_module.py * Update test_models.py	2019-08-30 18:56:09 -04:00
Stanislav	73cf47112e	Gradient accumulation callback (#150 ) * Gradient accumulation callback * little test case * typo * import fix * method name fix * fix epochs indexing from 1 * better code style * code style fix v2 :/ * change interface * fix Trainre new api in tests * trainer api bug fix * new raising error, new update method * extentions tests * a little better tests * typo fix * flack8 better * using scheduler for int and dict * typo * firs epoch bug fix * test update * empty dict exception * floats check * codestyle fix * grad counting test * someday, i will install normal linter * add more checks * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py * Update test_models.py	2019-08-30 10:56:14 -04:00
Ir1dXD	c2247350bb	feat(val_sanity): enable skipping validation sanity (#176 ) * feat(val_sanity): enable skipping validation sanity when self.nb_sanity_val_steps is 0 * docs: elaborate on skipping	2019-08-28 06:41:31 -04:00
William Falcon	67c314272b	Update setup.py (#174 )	2019-08-27 18:07:33 -04:00
Ir1dXD	da4c1e3409	docs: add repo_name in the upright corner (#171 )	2019-08-27 16:46:18 -04:00
Jirka Borovec	cd89b4ef43	move GH docs (#168 )	2019-08-27 07:10:26 -04:00
Ir1dXD	6eb6daa278	enable highlight (#170 )	2019-08-27 07:09:46 -04:00
William Falcon	c24599f5e5	release v	2019-08-24 08:13:54 -04:00
Ryan McCormick	b22e5918a9	fix python syntax in code blocks to be consistent (#166 ) A couple code blocks used "{.python}" instead of just "python" for the syntax highlighting, which doesn't render properly in GitHub markdown.	2019-08-23 21:24:18 -04:00
William Falcon	4104a0fc47	cleaned up progbar (#165 ) * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * cleaned up progbar * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * updated base files * flake 8	2019-08-23 21:23:27 -04:00
William Falcon	2ad9a9708b	Update README.md	2019-08-23 16:10:45 -04:00
William Falcon	ecce22f4de	Update README.md	2019-08-23 16:10:24 -04:00
Sebastian Præsius	b31539f62e	Guard against AttributeError in dataloaders. (#161 ) A solution for https://github.com/williamFalcon/pytorch-lightning/issues/142. Since hasattr "calls getattr(object, name) and to see whether it raises an AttributeError or not", I replaced it with a single call to getattr. See also https://stackoverflow.com/questions/24971061/python-hasattr-vs-getattr	2019-08-23 08:21:39 -04:00
William Falcon	cbb9821d9b	Cleaned up val/tng/test nb batches (#163 ) Set all to be 0 instead of None. Cleaned up val batch	2019-08-23 07:42:17 -04:00
William Falcon	a1490e993a	Update README.md	2019-08-23 03:46:27 -04:00
William Falcon	77d085134b	Update README.md	2019-08-23 03:45:14 -04:00
William Falcon	c30f69f60d	Update lightning_module_template.py	2019-08-23 02:42:40 -04:00
William Falcon	d5d47eab0d	Update lightning_module_template.py	2019-08-23 02:39:05 -04:00
Sebastian Præsius	9fc66026f1	train = False in test_dataloader (#162 ) A small change to the CoolModel example. Now test_dataloader returns the MNIST test dataset.	2019-08-22 17:44:06 -04:00
eqs	4a0b56755c	bug fix for #157 (#158 ) * Separate condition list/tuple case into separated cases * Add test for tuple of tensor list and list of tensor dict * Update test_models.py	2019-08-21 10:22:51 -04:00
William Falcon	55a804b7cf	fixes #154 (#155 ) * fixes #154 * Update trainer.py * Update trainer.py	2019-08-20 16:59:26 -04:00
William Falcon	7119ec1693	Update CONTRIBUTING.md	2019-08-20 09:51:11 -04:00
Ananya Harsh Jha	5b694c7e0e	bug fix for #138 (#143 ) * bug fix for #138 * split if for readability	2019-08-19 15:03:04 -04:00
sebftw	4bdb976284	Set val_check_interval default to 1.0. (#145 ) See discussion in https://github.com/williamFalcon/pytorch-lightning/issues/139.	2019-08-19 10:42:08 -04:00
William Falcon	4ad4588122	Update README.md	2019-08-19 07:22:02 -04:00
William Falcon	ac8186cb3c	Update README.md	2019-08-19 07:20:10 -04:00
Jirka Borovec	dbbbba35c9	add Codecov info (#144 )	2019-08-19 06:35:09 -04:00
William Falcon	f2a02881e3	Update README.md	2019-08-18 19:17:25 -04:00
William Falcon	e8c423a3b0	Update README.md	2019-08-18 19:16:57 -04:00
William Falcon	73b70584e7	Update README.md	2019-08-18 19:16:25 -04:00
William Falcon	736cf9b162	Update README.md	2019-08-18 19:16:09 -04:00
William Falcon	5771583c9d	Update README.md	2019-08-18 19:15:41 -04:00
William Falcon	64503f0d5e	Update README.md	2019-08-18 19:15:09 -04:00
William Falcon	504418d157	Update README.md	2019-08-18 19:05:13 -04:00
William Falcon	ad61b03fe9	Update README.md	2019-08-18 18:51:47 -04:00
sebftw	a7a14dadb6	F.cross_entropy(y_hat, y)(y_hat, y) typo. (#137 ) This seems to be a typo. Throws TypeError: 'Tensor' object is not callable.	2019-08-18 18:17:43 -04:00
sebftw	b2a49197e4	tensorboarX to tensorboardX (#136 ) * tensorboarX to tensorboardX * Update properties.md	2019-08-18 18:17:05 -04:00
sebftw	23a4421595	Removed redundant line. (#140 )	2019-08-18 18:16:30 -04:00
sebftw	26d3f0dbea	Error if dataset size = 1 batch. (#141 ) Fix for the bug mentioned in https://github.com/williamFalcon/pytorch-lightning/issues/139	2019-08-18 18:15:58 -04:00
Maxim Andreev	e646d745da	use val_percent_check in validation step (#135 )	2019-08-18 11:02:28 -04:00
William Falcon	9aa9a1a796	Update lightning_module_template.py	2019-08-17 11:11:07 -04:00
Ir1dXD	48de39ed50	elaborate on the correlation between overfit_pct and xxx_percent_check (#132 ) * Update Training Loop.md * update docs and elaborate on the correlation	2019-08-17 10:23:25 -04:00
William Falcon	1a31782272	fixed str crash err	2019-08-17 10:20:58 -04:00
Ir1dXD	24a97956e4	fix typo in docs (#129 ) * fix typo * fix typo * fix typo * fix list	2019-08-17 07:48:33 -04:00
William Falcon	1b7d66d089	Merge branch 'master' of https://github.com/williamFalcon/pytorch-lightning	2019-08-16 17:14:40 -04:00
William Falcon	e60e002f17	updated docs	2019-08-16 17:14:31 -04:00
William Falcon	308239cef0	allow loss to be used for early stopping (#127 )	2019-08-16 11:58:44 -04:00
William Falcon	bdd86087e6	updated docs	2019-08-16 10:07:56 -04:00
William Falcon	50f0de094f	updated docs	2019-08-16 10:07:44 -04:00
William Falcon	bc401d0f59	updated docs	2019-08-16 10:02:28 -04:00
William Falcon	4b97319c2e	updated docs	2019-08-15 21:29:25 -04:00
William Falcon	90f01c05bc	updated docs	2019-08-15 21:21:26 -04:00
William Falcon	0e92a9d7af	updated docs	2019-08-15 21:19:29 -04:00
William Falcon	81837221a4	updated docs	2019-08-15 13:59:54 -04:00
William Falcon	44da88fd15	updated docs	2019-08-15 13:59:27 -04:00