release v0.3.6.9

erge branch 'master' of https://github.com/williamFalcon/pytorch-lightning

README.md

@@ -4,10 +4,10 @@
   </a>
 </p>
 <h3 align="center">
-  Pytorch Lightning
+  PyTorch Lightning
 </h3>
 <p align="center">
-  The Keras for ML researchers using PyTorch. More control. Less boilerplate.    
+  The PyTorch Keras for ML researchers. More control. Less boilerplate.    
 </p>
 
 <p align="center">
@@ -31,9 +31,11 @@ Lightning defers training and validation loop logic to you. It guarantees correc
 
 
 ## Why do I want to use lightning?
-When starting a new project the last thing you want to do is recode a training loop, model loading/saving, distributed training, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.
+When starting a new project the last thing you want to do is recode a training loop, multi-cluster training, 16-bit precision, early-stopping, model loading/saving, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.
 
-With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: Data and training, validation loop logic. Don't worry about multiple gpus or speeding up your code, lightning will do that for you!
+With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: The data and the training/validation loop logic. 
+
+Don't worry about training on multiple gpus or speeding up your code, lightning will do that for you!
 
 ## How do I do use it?   
 
@@ -280,8 +282,8 @@ tensorboard --logdir /some/path
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
-- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)    
-- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
+- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)    
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 
 ###### Validation loop    
@@ -316,6 +318,19 @@ python single_gpu_node_template.py --gpus "0,1"
 python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'
 ```
 
+## Contributing    
+Welcome to the PTL community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!  
+
+#### Bug fixes:  
+1. Submit a github issue.   
+2. Fix it.  
+3. Submit a PR! 
+
+#### New Features:  
+1. Submit a github issue.   
+2. We'll agree on the feature scope.     
+3. Submit a PR! (with updated docs and tests 🙃).    
+
 ## Bleeding edge
 If you can't wait for the next release, install the most up to date code with:  
 ```bash

docs/LightningModule/RequiredTrainerInterface.md

@@ -300,7 +300,7 @@ def tng_dataloader(self)
 Called by lightning during training loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.
 
 ##### Return
-Pytorch DataLoader
+PyTorch DataLoader
 
 **Example**
 
@@ -327,7 +327,7 @@ def tng_dataloader(self)
 Called by lightning during validation loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.
 
 ##### Return
-Pytorch DataLoader
+PyTorch DataLoader
 
 **Example**
 
@@ -355,7 +355,7 @@ def test_dataloader(self)
 Called by lightning during test loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.
 
 ##### Return
-Pytorch DataLoader
+PyTorch DataLoader
 
 **Example**
 

docs/LightningModule/methods.md

@@ -31,7 +31,7 @@ y_hat = pretrained_model(x)
 
 | Param  | description  |
 |---|---|
-|  weights_path | Path to a pytorch checkpoint  |
+|  weights_path | Path to a PyTorch checkpoint  |
 |  tags_csv | Path to meta_tags.csv file generated by the test-tube Experiment  |
 |  on_gpu | if True, puts model on GPU. Make sure to use transforms option if model devices have changed  |
 |  map_location | A dictionary mapping saved weight GPU devices to new GPU devices |

docs/Trainer/Logging.md

@@ -52,7 +52,7 @@ Trainer(experiment=exp)
 
 ---
 ### Tensorboard support   
-The experiment object is a strict subclass of Pytorch SummaryWriter. However, this class
+The experiment object is a strict subclass of PyTorch SummaryWriter. However, this class
 also snapshots every detail about the experiment (data folder paths, code, hyperparams),
 and allows you to visualize it using tensorboard.
 ``` {.python}

docs/Trainer/hooks.md

@@ -5,7 +5,7 @@ There are cases when you might want to do something different at different parts
 To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
 
 **Contributing** If there's a hook you'd like to add, simply:    
-1. Fork PytorchLightning.    
+1. Fork PyTorchLightning.    
 2. Add the hook [here](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py).       
 3. Add the correct place in the [Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py) where it should be called.    
 

docs/Trainer/index.md

@@ -63,8 +63,8 @@ But of course the fun is in all the advanced things it can do:
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](hooks)
-- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
-- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
+- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 
 **Validation loop**    

docs/index.md

@@ -70,8 +70,8 @@ one could be a seq-2-seq model, both (optionally) ran by the same trainer file.
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
-- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
-- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
+- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 
 ###### Validation loop    

mkdocs.yml

@@ -1,10 +1,10 @@
-site_name: Pytorch lightning Documentation
+site_name: PyTorch lightning Documentation
 theme:
   name: 'material'
 docs_dir: docs
 repo_url: https://github.com/williamFalcon/pytorch-lightning
 site_dir: 'site'
-site_description: 'Documentation for Pytorch LightningModule, the researcher version of keras.'
+site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
 
 dev_addr: '0.0.0.0:8000'
 #google_analytics: ['UA-aasd', 'sitename']

pytorch_lightning/__init__.py

@@ -1,3 +1,3 @@
-from .models import Trainer
+from .models.trainer import Trainer
 from .root_module.root_module import LightningModule
 from .root_module.decorators import data_loader

pytorch_lightning/models/__init__.py

@@ -1 +0,0 @@
-from .trainer import Trainer

pytorch_lightning/models/trainer.py

@@ -772,7 +772,7 @@ class Trainer(TrainerIO):
             output = self.model.training_step(data_batch, batch_nb)
 
         try:
-            model_specific_tqdm_metrics_dic = output['tqdm_metrics']
+            model_specific_tqdm_metrics_dic = output['prog']
         except Exception as e:
             model_specific_tqdm_metrics_dic = {}
 
@@ -885,4 +885,4 @@ class Trainer(TrainerIO):
         # model checkpointing
         if self.proc_rank == 0 and self.checkpoint_callback is not None:
             print('save callback...')
-            self.checkpoint_callback.on_epoch_end(epoch=self.current_epoch, logs=self.__tng_tqdm_dic)
+            self.checkpoint_callback.on_epoch_end(epoch=self.current_epoch, logs=self.__tng_tqdm_dic)

pytorch_lightning/root_module/model_saving.py

@@ -102,13 +102,16 @@ class TrainerIO(object):
             return
 
         # allow test tube to handle model check pointing automatically
-        self.cluster.set_checkpoint_save_function(
-            self.hpc_save,
-            kwargs={
-                'folderpath': self.checkpoint_callback.filepath,
-                'experiment': self.experiment
-            }
-        )
+        # only if proc 0 so we don't trigger world_size resubmits
+        if self.proc_rank == 0:
+            self.cluster.set_checkpoint_save_function(
+                self.hpc_save,
+                kwargs={
+                    'folderpath': self.checkpoint_callback.filepath,
+                    'experiment': self.experiment
+                }
+            )
+
         self.cluster.set_checkpoint_load_function(
             self.hpc_load,
             kwargs={

pytorch_lightning/testing_models/lm_test_module.py

@@ -96,12 +96,15 @@ class LightningTestModel(LightningModule):
         if self.trainer.use_dp:
             loss_val = loss_val.unsqueeze(0)
 
-        output = OrderedDict({
-            'loss': loss_val
-        })
-
-        # can also return just a scalar instead of a dict (return loss_val)
-        return output
+        # alternate possible outputs to test
+        if self.trainer.batch_nb % 1 == 0:
+            output = OrderedDict({
+                'loss': loss_val,
+                'prog': {'some_val': loss_val * loss_val}
+            })
+            return output
+        if self.trainer.batch_nb % 2 == 0:
+            return loss_val
 
     def validation_step(self, data_batch, batch_i):
         """

setup.py

@@ -7,7 +7,7 @@ from setuptools import setup, find_packages
 # http://blog.ionelmc.ro/2014/05/25/python-packaging/
 setup(
     name="pytorch-lightning",
-    version='0.3.6.5',
+    version='0.3.6.9',
     description="The Keras for ML researchers using PyTorch",
     author="William Falcon",
     author_email="waf2107@columbia.edu",
@@ -19,7 +19,7 @@ setup(
     install_requires=[
         "torch>=1.1.0",
         "tqdm",
-        "test-tube>=0.6.7.4",
+        "test-tube>=0.6.7.6",
     ],
     packages=find_packages(),
     long_description=open("README.md", encoding="utf-8").read(),

tests/README.md

@@ -1,4 +1,4 @@
-# Pytorch-Lightning Tests
+# PyTorch-Lightning Tests
 
 ## Running tests
 The automatic travis tests ONLY run CPU-based tests. Although these cover most of the use cases,