release v0.4.1

* removed reduce on non-loss outputs from dp

* fixed val reduce

* fixed val reduce

* fixed val reduce

* fixed val reduce

.codecov.yml

@@ -0,0 +1,44 @@
+#see https://github.com/codecov/support/wiki/Codecov-Yaml
+codecov:
+  notify:
+    require_ci_to_pass: yes
+
+coverage:
+  precision: 0  # 2 = xx.xx%, 0 = xx%
+  round: nearest # how coverage is rounded: down/up/nearest
+  range: 40...100 # custom range of coverage colors from red -> yellow -> green
+  status:
+    # https://codecov.readme.io/v1.0/docs/commit-status
+    project:
+      default:
+        against: auto
+        target: 99% # specify the target coverage for each commit status
+        threshold: 20% # allow this little decrease on project
+        # https://github.com/codecov/support/wiki/Filtering-Branches
+        # branches: master
+        if_ci_failed: error
+    # https://github.com/codecov/support/wiki/Patch-Status
+    patch:
+      default:
+        against: auto
+        target: 40% # specify the target "X%" coverage to hit
+        # threshold: 50% # allow this much decrease on patch
+    changes: false
+
+parsers:
+  gcov:
+    branch_detection:
+      conditional: true
+      loop: true
+      macro: false
+      method: false
+  javascript:
+    enable_partials: false
+
+comment:
+  layout: header, diff
+  require_changes: false
+  behavior: default  # update if exists else create new
+  # branches: *
+
+

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement, help wanted
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/typos-and-doc-fixes.md

@@ -0,0 +1,17 @@
+---
+name: Typos and doc fixes
+about: Typos and doc fixes
+title: ''
+labels: typo
+assignees: ''
+
+---
+
+For typos and doc fixes, please go ahead and:
+
+1. Create an issue.
+2. Fix the typo.   
+3. Submit a PR.
+
+
+Thanks!

.gitignore

@@ -10,6 +10,7 @@ app/models/
 pip-wheel-metadata/
 test_tube_exp/
 tests/tests_tt_dir/
+tests/save_dir
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

.readthedocs.yml

@@ -16,4 +16,4 @@ formats: all
 python:
   version: 3.7
   install:
-    - requirements: docs/doc_requirements.txt
+    - requirements: docs/requirements.txt

.travis.yml

@@ -1,16 +1,46 @@
+# vim ft=yaml
+
+# After changing this file, check it on:
+#   http://yaml-online-parser.appspot.com/
+
+# See doc/travis_notes.txt for some guidelines
+
+# this file is *not* meant to cover or endorse the use of travis, but rather to
+# help confirm pull requests to this project.
+
+dist: xenial  # Ubuntu 16.04
+
+env:
+  global:
+    - DISPLAY=""
+
 language: python
-python:
-  - "3.7"
-# command to install dependencies
+
+matrix:
+  include:
+    - python: 3.6
+      env: TOXENV=py36
+    - python: 3.7
+      env: TOXENV=py37
+
+# See http://docs.travis-ci.com/user/caching/#pip-cache
 cache: pip
+
 install:
-  - pip install -e .
   - pip install -r requirements.txt
-  - pip install -U numpy
+  - pip install -r ./tests/requirements.txt
+  - pip --version ; pip list
 
-# keep build from timing out
-dist: xenial
-
-# command to run tests
 script:
-  - py.test # or py.test for Python versions 3.5 and below
+  # integration
+  - tox --sitepackages
+  - python setup.py install --dry-run
+
+after_success:
+  - coverage report
+  # disable auto coverage bc it isn't accurate since it misses gpu code.
+  # to get coverage, run local and push results
+  # - codecov
+
+notifications:
+  email: false

CONTRIBUTING.md

@@ -0,0 +1,37 @@
+# Contributing    
+Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!  
+
+## Lightning Design Principles   
+We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
+
+#### No PyTorch interference   
+We don't want to add any abstractions on top of pure PyTorch. This gives researchers all the control they need without having to learn yet another framework.    
+
+#### Simple Internal Code    
+It's useful for users to look at the code and understand very quickly what's happening. Many users won't be engineers. Thus we need to value clear, simple code over condensed ninja moves. While that's super cool, this isn't the project for that :)      
+
+#### Force User Decisions To Best Practices    
+There are 1,000 ways to do something. However, something eventually becomes standard practice that everyone does. Thus we pick one way of doing it and force everyone to do it this way. A good example is accumulated gradients. There are many ways to implement, we just pick one and force users to use that one. A bad forced decision would be to make users use a specific library to do something.    
+
+When something becomes a best practice, we add it to the framework. This likely looks like code in utils or in the model file that everyone keeps adding over and over again across projects. When this happens, bring that code inside the trainer and add a flag for it.
+
+#### Simple External API    
+What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others. Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people.    
+
+#### Gain User Trust    
+As a researcher you can't have any part of your code going wrong. So, make thorough tests that ensure an implementation of a new trick or subbtle change is correct.    
+
+## Contribution types    
+Currently looking for help implementing new features or adding bug fixes. 
+
+A lot of good work has already been done in project mechanics (requirements.txt, setup.py, pep8, badges, ci, etc...) we're in a good state there thanks to all the early contributors (even pre-beta release)!   
+
+## Bug fixes:  
+1. Submit a github issue.   
+2. Fix it.  
+3. Submit a PR! 
+
+## New Features:  
+1. Submit a github issue.   
+2. We'll agree on the feature scope.     
+3. Submit a PR! (with updated docs and tests 🙃).   

COPYING

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2019 William Falcon
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

MANIFEST.in

@@ -1,9 +1,37 @@
-graft docs
+# Manifest syntax https://docs.python.org/2/distutils/sourcedist.html
+graft wheelhouse
 
-include COPYING
-include AUTHORS
+recursive-include birl *.py
+recursive-exclude __pycache__  *.py[cod] *.orig
 
-recursive-include src/einsteinpy/tests *.py *.html
+# Include the README
+include *.md
 
-prune docs/source/examples/.ipynb_checkpoints
-global-exclude *.py[cod] __pycache__ *.so *.dylib
+# Include the license file
+include LICENSE
+
+exclude *.sh
+exclude *.toml
+exclude *.svg    
+recursive-include examples *.py
+recursive-include pytorch_lightning *.py
+
+# exclude tests from package
+recursive-exclude tests *
+exclude tests
+
+# Exclude the documentation files
+recursive-exclude docs *
+exclude docs
+
+# Include the Requirements
+include requirements.txt
+
+# Exclude build configs
+exclude *.yml
+
+prune .git
+prune .github
+prune notebook*
+prune temp*
+prune test*

README.md

@@ -1,24 +1,30 @@
-<p align="center">
-  <a href="https://williamfalcon.github.io/pytorch-lightning/">
-    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/lightning_logo.png" width="50">
-  </a>
-</p>
-<h3 align="center">
-  Pytorch Lightning
-</h3>
-<p align="center">
-  The Keras for ML researchers using PyTorch. More control. Less boilerplate.    
-</p>
+<div align="center">
 
-<p align="center">
-  <a href="https://badge.fury.io/py/pytorch-lightning"><img src="https://badge.fury.io/py/pytorch-lightning.svg" alt="PyPI version" height="18"></a>
-  <a href="https://pepy.tech/project/pytorch-lightning"><img src="https://pepy.tech/badge/pytorch-lightning" alt="PyPI version" height="18"></a>
-  <a href="https://github.com/williamFalcon/pytorch-lightning/tree/master/tests"><img src="https://github.com/williamFalcon/pytorch-lightning/blob/master/coverage.svg"></a>
-  <a href="https://travis-ci.org/williamFalcon/pytorch-lightning"><img src="https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master"></a>
-  <a href="https://williamfalcon.github.io/pytorch-lightning/"><img src="https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest"></a>
-  <a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/COPYING"><img src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
-</p>   
+![Logo](./docs/source/_static/lightning_logo_small.png)
 
+# PyTorch Lightning
+
+**The PyTorch Keras for ML researchers. More control. Less boilerplate.**
+
+
+[![PyPI Status](https://badge.fury.io/py/pytorch-lightning.svg)](https://badge.fury.io/py/pytorch-lightning)
+[![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
+[![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
+<!-- 
+removed until windows install issues resolved.
+[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning) -->
+<!-- 
+removed until codecov badge isn't empy. likely a config error showing nothing on master.
+[![codecov](https://codecov.io/gh/Borda/pytorch-lightning/branch/master/graph/badge.svg)](https://codecov.io/gh/Borda/pytorch-lightning)
+-->
+[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
+[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)
+[![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
+[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
+
+</div>
+
+Simple installation from PyPI
 ```bash
 pip install pytorch-lightning  
 ```
@@ -27,34 +33,53 @@ pip install pytorch-lightning
 **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**
 
 ## What is it?  
-Lightning defers training and validation loop logic to you. It guarantees correct, modern best practices for the core training logic.
+Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. It defers core training and validation logic to you and automates the rest. It guarantees tested, correct, modern best practices for the automated parts.
 
 
 ## Why do I want to use lightning?
-When starting a new project the last thing you want to do is recode a training loop, model loading/saving, distributed training, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.
+When starting a new project the last thing you want to do is recode a training loop, multi-cluster training, 16-bit precision, early-stopping, model loading/saving, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.
 
-With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: Data and training, validation loop logic. Don't worry about multiple gpus or speeding up your code, lightning will do that for you!
+With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: The data and the training/validation loop logic. 
 
+Don't worry about training on multiple gpus or speeding up your code, lightning will do that for you!
+
+---   
+## README Table of Contents        
+- [How do I use it](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it)     
+- [What lightning automates](https://github.com/williamFalcon/pytorch-lightning#what-does-lightning-control-for-me)    
+- [Tensorboard integration](https://github.com/williamFalcon/pytorch-lightning#tensorboard)    
+- [Lightning features](https://github.com/williamFalcon/pytorch-lightning#lightning-automates-all-of-the-following-each-is-also-configurable)    
+- [Demos](https://github.com/williamFalcon/pytorch-lightning#demo)    
+- [Tutorials](https://github.com/williamFalcon/pytorch-lightning#tutorials)
+- [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md)    
+- [Bleeding edge install](https://github.com/williamFalcon/pytorch-lightning#bleeding-edge)   
+- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)
+- [FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
+
+---   
 ## How do I do use it?   
 
 To use lightning do 2 things:  
 1. [Define a LightningModel](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
 ```python
-import pytorch_lightning as ptl
+import os
 import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
+import torchvision.transforms as transforms
 
-class CoolModel(ptl.LightningModule):
+import pytorch_lightning as pl
 
-    def __init(self):
+class CoolModel(pl.LightningModule):
+
+    def __init__(self):
         super(CoolModel, self).__init__()
         # not the best model...
         self.l1 = torch.nn.Linear(28 * 28, 10)
 
     def forward(self, x):
-        return torch.relu(self.l1(x))
+        return torch.relu(self.l1(x.view(x.size(0), -1)))
 
     def my_loss(self, y_hat, y):
         return F.cross_entropy(y_hat, y)
@@ -62,7 +87,7 @@ class CoolModel(ptl.LightningModule):
     def training_step(self, batch, batch_nb):
         x, y = batch
         y_hat = self.forward(x)
-        return {'tng_loss': self.my_loss(y_hat, y)}
+        return {'loss': self.my_loss(y_hat, y)}
 
     def validation_step(self, batch, batch_nb):
         x, y = batch
@@ -70,46 +95,67 @@ class CoolModel(ptl.LightningModule):
         return {'val_loss': self.my_loss(y_hat, y)}
 
     def validation_end(self, outputs):
-        avg_loss = torch.stack([x for x in outputs['val_loss']]).mean()
-        return avg_loss
+        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
+        return {'avg_val_loss': avg_loss}
 
     def configure_optimizers(self):
         return [torch.optim.Adam(self.parameters(), lr=0.02)]
 
-    @ptl.data_loader
+    @pl.data_loader
     def tng_dataloader(self):
-        return DataLoader(MNIST('path/to/save', train=True), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 
-    @ptl.data_loader
+    @pl.data_loader
     def val_dataloader(self):
-        return DataLoader(MNIST('path/to/save', train=False), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 
-    @ptl.data_loader
+    @pl.data_loader
     def test_dataloader(self):
-        return DataLoader(MNIST('path/to/save', train=False), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
 
 2. Fit with a [trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)    
 ```python
 from pytorch_lightning import Trainer
-from test_tube import Experiment
 
 model = CoolModel()
 
-# fit on 32 gpus across 4 nodes
-exp = Experiment(save_dir='some/dir')
-trainer = Trainer(experiment=exp, nb_gpu_nodes=4, gpus=[0,1,2,3,4,5,6,7])
+# most basic trainer, uses good defaults
+trainer = Trainer()    
+trainer.fit(model)   
+```     
 
+Or with tensorboard logger and some options turned on such as multi-gpu, etc...    
+```python   
+from test_tube import Experiment    
+
+# PyTorch summarywriter with a few bells and whistles    
+exp = Experiment(save_dir=os.getcwd())
+
+# train on cpu using only 10% of the data (for demo purposes)
+# pass in experi
+trainer = Trainer(experiment=exp, max_nb_epochs=1, train_percent_check=0.1)
+
+# train on 4 gpus
+# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3])
+
+# train on 32 gpus across 4 nodes (make sure to submit appropriate SLURM job)
+# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3, 4, 5, 6, 7], nb_gpu_nodes=4)
+
+# train (1 epoch only here for demo)
 trainer.fit(model)
 
-# see all experiment metrics here
-# tensorboard --log_dir some/dir   
-```
+# view tensorflow logs 
+print('View tensorboard logs by running\ntensorboard --logdir %s' % os.getcwd())
+print('and going to http://localhost:6006 on your browser')
+```    
 
+## What does lightning control for me?   
 
-## What does lightning control for me?
-Everything!    
-Except for these 6 core functions which you define:    
+Everything in gray!    
+You define the blue parts using the LightningModule interface:  
+
+![Ouverview](./docs/source/_static/overview_flat.jpg)
 
 ```{.python}
 # what to do in the training loop
@@ -137,7 +183,7 @@ def training_step(self, data_batch, batch_nb):
     # define your own forward and loss calculation
     hidden_states = self.encoder(x)
      
-    # even as complex as a seq-2seq + attn model
+    # even as complex as a seq-2-seq + attn model
     # (this is just a toy, non-working example to illustrate)
     start_token = '<SOS>'
     last_hidden = torch.zeros(...)
@@ -191,19 +237,11 @@ def validation_end(self, outputs):
 ## Tensorboard    
 Lightning is fully integrated with tensorboard.   
 
-<p align="center">
-  <a href="https://williamfalcon.github.io/pytorch-lightning/">
-    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/tf_loss.png" width="900px">
-  </a>
-</p>
+![tensorboard-support](./docs/source/_static/tf_loss.png)
 
 Lightning also adds a text column with all the hyperparameters for this experiment.      
 
-<p align="center">
-  <a href="https://williamfalcon.github.io/pytorch-lightning/">
-        <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/tf_tags.png" width="900px">
-  </a>
-</p>
+![tensorboard-support](./docs/source/_static/tf_tags.png)
 
 Simply note the path you set for the Experiment    
 ``` {.python}   
@@ -222,10 +260,12 @@ tensorboard --logdir /some/path
 
 ## Lightning automates all of the following ([each is also configurable](https://williamfalcon.github.io/pytorch-lightning/Trainer/)):
 
+
 ###### Checkpointing    
 
 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
+- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
 
 ###### Computing cluster (SLURM)    
 
@@ -239,7 +279,8 @@ tensorboard --logdir /some/path
 - [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
-- [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
+- [Print which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
+- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 
 
 ###### Distributed training    
@@ -254,9 +295,9 @@ tensorboard --logdir /some/path
 ###### Experiment Logging   
 
 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
-- Log arbitrary metrics
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
 - [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
+- [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
 - [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
@@ -264,16 +305,18 @@ tensorboard --logdir /some/path
 ###### Training loop    
 
 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
-- [Anneal Learning rate](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#anneal-learning-rate)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
-- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
+- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
+- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)    
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 
 ###### Validation loop    
 
 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
+- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
 - [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
@@ -287,7 +330,8 @@ pip install pytorch-lightning
 
 # clone lightning for the demo
 git clone https://github.com/williamFalcon/pytorch-lightning.git
-cd pytorch_lightning/examples/new_project_templates/
+cd pytorch-lightning
+cd examples/new_project_templates/
 
 # all of the following demos use the SAME model to show no modification needs to be made to your code
 
@@ -301,8 +345,33 @@ python single_gpu_node_template.py --gpus "0,1"
 python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'
 ```
 
+## Tutorials   
+- [Basic Lightning use](https://towardsdatascience.com/supercharge-your-ai-research-with-pytorch-lightning-337948a99eec)    
+- [9 key speed features in Pytorch-Lightning](https://towardsdatascience.com/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565)    
+- [SLURM, multi-node training with Lightning](https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd)     
+
+---   
+## FAQ    
+**Why was Lightning created?**     
+Lightning has 3 goals in mind:
+1. Maximal flexibility while abstracting out the common boilerplate across research projects.   
+2. Reproducibility. If all projects use the LightningModule template, it will be much much easier to understand what's going on and where to look! It will also mean every implementation follows a standard format.   
+3. Democratizing PyTorch power user features. Distributed training? 16-bit? know you need them but don't want to take the time to implement? All good... these come built into Lightning.    
+
+**How does Lightning compare with Ignite and fast.ai?**     
+[Here's a thorough comparison](https://medium.com/@_willfalcon/pytorch-lightning-vs-pytorch-ignite-vs-fast-ai-61dc7480ad8a).    
+
+**Is this another library I have to learn?**    
+Nope! We use pure Pytorch everywhere and don't add unecessary abstractions!   
+
+**Are there plans to support Python 2?**          
+Nope.     
+
+**Are there plans to support virtualenv?**    
+Nope. Please use anaconda or miniconda.    
+
 ## Bleeding edge
 If you can't wait for the next release, install the most up to date code with:  
 ```bash
 pip install git+https://github.com/williamFalcon/pytorch-lightning.git@master --upgrade
-```
+```

appveyor.yml

@@ -0,0 +1,66 @@
+# https://www.appveyor.com/docs/appveyor-yml/
+environment:
+
+  # SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
+  # /E:ON and /V:ON options are not enabled in the batch script interpreter
+  # See: http://stackoverflow.com/a/13751649/163740
+  CMD_IN_ENV: "cmd /E:ON /V:ON /C obvci_appveyor_python_build_env.cmd"
+
+  matrix:
+    # Pre-installed Python versions, which Appveyor may upgrade to
+    # a later point release.
+    # See: http://www.appveyor.com/docs/installed-software#python
+
+
+    # - PYTHON: "C:\\Python35-x64"
+    #   PYTHON_VERSION: "3.5.x"
+    #   PYTHON_ARCH: "64"
+    #  TOXENV: "py35"
+
+    - PYTHON: "C:\\Python36-x64"
+      PYTHON_VERSION: "3.6.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py36"
+      PIP_PYVER: "36"
+
+    - PYTHON: "C:\\Python37-x64"
+      PYTHON_VERSION: "3.7.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py37"
+      PIP_PYVER: "37"
+
+build: off
+
+# https://www.appveyor.com/docs/build-cache/
+cache:
+  - C:\ProgramData\chocolatey\bin -> appveyor.yml
+  - C:\ProgramData\chocolatey\lib -> appveyor.yml
+  - '%LOCALAPPDATA%\pip\Cache -> appveyor.yml'
+
+# scripts that run after cloning repository
+install:
+  # If there is a newer build queued for the same PR, cancel this one.
+  # The AppVeyor 'rollout builds' option is supposed to serve the same
+  # purpose but it is problematic because it tends to cancel builds pushed
+  # directly to master instead of just PR builds (or the converse).
+  - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%path%
+  - pip install -U --user pip
+  - pip install "https://download.pytorch.org/whl/cu90/torch-1.1.0-cp%PIP_PYVER%-cp%PIP_PYVER%m-win_amd%PYTHON_ARCH%.whl"
+    pip install "https://download.pytorch.org/whl/cu90/torchvision-0.3.0-cp%PIP_PYVER%-cp%PIP_PYVER%m-win_amd%PYTHON_ARCH%.whl"
+  - pip install -r requirements.txt
+  - pip install -r ./tests/requirements.txt
+
+# scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
+before_test:
+  - python --version
+  - pip --version
+  - pip list
+  - dir
+
+# to run your custom scripts instead of automatic tests
+test_script:
+  - tox --sitepackages --parallel auto
+
+on_success:
+  - coverage report
+  # - codecov

docs/LightningModule/RequiredTrainerInterface.md

@@ -3,7 +3,7 @@
 
 A lightning module is a strict superclass of nn.Module, it provides a standard interface for the trainer to interact with the model.
 
-The easiest thing to do is copy [this template](../../pytorch_lightning/examples/new_project_templates/lightning_module_template.py) and modify accordingly. 
+The easiest thing to do is copy the [minimal example](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) below and modify accordingly. 
 
 Otherwise, to Define a Lightning Module, implement the following methods:
 
@@ -27,23 +27,26 @@ Otherwise, to Define a Lightning Module, implement the following methods:
 - [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)
 
 ---
-**Minimal example**
+### Minimal example   
 ```python
-import pytorch_lightning as ptl
+import os
 import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
+import torchvision.transforms as transforms
 
-class CoolModel(ptl.LightningModule):
+import pytorch_lightning as pl
 
-    def __init(self):
+class CoolModel(pl.LightningModule):
+
+    def __init__(self):
         super(CoolModel, self).__init__()
         # not the best model...
         self.l1 = torch.nn.Linear(28 * 28, 10)
 
     def forward(self, x):
-        return torch.relu(self.l1(x))
+        return torch.relu(self.l1(x.view(x.size(0), -1)))
 
     def my_loss(self, y_hat, y):
         return F.cross_entropy(y_hat, y)
@@ -51,7 +54,7 @@ class CoolModel(ptl.LightningModule):
     def training_step(self, batch, batch_nb):
         x, y = batch
         y_hat = self.forward(x)
-        return {'tng_loss': self.my_loss(y_hat, y)}
+        return {'loss': self.my_loss(y_hat, y)}
 
     def validation_step(self, batch, batch_nb):
         x, y = batch
@@ -59,24 +62,33 @@ class CoolModel(ptl.LightningModule):
         return {'val_loss': self.my_loss(y_hat, y)}
 
     def validation_end(self, outputs):
-        avg_loss = torch.stack([x for x in outputs['val_loss']]).mean()
-        return avg_loss
+        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
+        return {'avg_val_loss': avg_loss}
 
     def configure_optimizers(self):
         return [torch.optim.Adam(self.parameters(), lr=0.02)]
 
-    @ptl.data_loader
+    @pl.data_loader
     def tng_dataloader(self):
-        return DataLoader(MNIST('path/to/save', train=True), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 
-    @ptl.data_loader
+    @pl.data_loader
     def val_dataloader(self):
-        return DataLoader(MNIST('path/to/save', train=False), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 
-    @ptl.data_loader
+    @pl.data_loader
     def test_dataloader(self):
-        return DataLoader(MNIST('path/to/save', train=False), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
+---   
+### How do these methods fit into the broader training?     
+The LightningModule interface is on the right. Each method corresponds to a part of a research project. Lightning automates everything not in blue.    
+
+<p align="center">
+  <a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg">
+    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg" height="900px">
+  </a>
+</p>   
 
 ---
 
@@ -222,26 +234,27 @@ def validation_end(self, outputs):
 def configure_optimizers(self)
 ```
 
-Set up as many optimizers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
-Lightning will call .backward() and .step() on each one.  If you use 16 bit precision it will also handle that.
+Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
+Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
 
 
 ##### Return
-List - List of optimizers
+List or Tuple - List of optimizers with an optional second list of learning-rate schedulers
 
 **Example**
 
 ``` {.python}
 # most cases
 def configure_optimizers(self):
-    opt = Adam(lr=0.01)
+    opt = Adam(self.parameters(), lr=0.01)
     return [opt]
     
-# gan example
+# gan example, with scheduler for discriminator
 def configure_optimizers(self):
-    generator_opt = Adam(lr=0.01)
-    disriminator_opt = Adam(lr=0.02)
-    return [generator_opt, disriminator_opt] 
+    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
+    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
+    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
+    return [generator_opt, disriminator_opt], [discriminator_sched]
 ```
 
 --- 
@@ -290,18 +303,18 @@ def on_load_checkpoint(self, checkpoint):
 ### tng_dataloader 
 
 ``` {.python}
-@ptl.data_loader
+@pl.data_loader
 def tng_dataloader(self)
 ```
-Called by lightning during training loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.
+Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
 
 ##### Return
-Pytorch DataLoader
+PyTorch DataLoader
 
 **Example**
 
 ``` {.python}
-@ptl.data_loader
+@pl.data_loader
 def tng_dataloader(self):
     transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
     dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
@@ -317,18 +330,18 @@ def tng_dataloader(self):
 ### val_dataloader 
 
 ``` {.python}
-@ptl.data_loader
+@pl.data_loader
 def tng_dataloader(self)
 ```
-Called by lightning during validation loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.
+Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
 
 ##### Return
-Pytorch DataLoader
+PyTorch DataLoader
 
 **Example**
 
 ``` {.python}
-@ptl.data_loader
+@pl.data_loader
 def val_dataloader(self):
     transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
     dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
@@ -345,18 +358,18 @@ def val_dataloader(self):
 ### test_dataloader 
 
 ``` {.python}
-@ptl.data_loader
+@pl.data_loader
 def test_dataloader(self)
 ```
-Called by lightning during test loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.
+Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
 
 ##### Return
-Pytorch DataLoader
+PyTorch DataLoader
 
 **Example**
 
 ``` {.python}
-@ptl.data_loader
+@pl.data_loader
 def test_dataloader(self):
     transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
     dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
@@ -428,4 +441,4 @@ def add_model_specific_args(parent_parser, root_dir):
     parser.opt_list('--batch_size', default=256, type=int, options=[32, 64, 128, 256], tunable=False)
     parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)
     return parser
-```
+```

docs/LightningModule/methods.md

@@ -31,7 +31,7 @@ y_hat = pretrained_model(x)
 
 | Param  | description  |
 |---|---|
-|  weights_path | Path to a pytorch checkpoint  |
+|  weights_path | Path to a PyTorch checkpoint  |
 |  tags_csv | Path to meta_tags.csv file generated by the test-tube Experiment  |
 |  on_gpu | if True, puts model on GPU. Make sure to use transforms option if model devices have changed  |
 |  map_location | A dictionary mapping saved weight GPU devices to new GPU devices |

docs/LightningModule/properties.md

@@ -38,3 +38,17 @@ self.trainer.current_epoch
 ...   
 ```   
 
+## Debugging   
+The LightningModule also offers these tricks to help debug.   
+
+---   
+#### example_input_array    
+In the LightningModule init, you can set a dummy tensor for this property
+to get a print out of sizes coming into and out of every layer.   
+```python
+def __init__(self):
+    # put the dimensions of the first input to your system
+    self.example_input_array = torch.rand(5, 28 * 28)
+```    
+
+

docs/Trainer/Checkpointing.md

@@ -18,5 +18,51 @@ checkpoint_callback = ModelCheckpoint(
 trainer = Trainer(checkpoint_callback=checkpoint_callback)
 ```
 
+---
+### Restoring training session 
+You might want to not only load a model but also continue training it. Use this method to
+restore the trainer state as well. This will continue from the epoch and global step you last left off.  
+However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).   
 
+Lightning will restore the session if you pass an experiment with the same version and there's a saved checkpoint.   
+``` {.python}
+from test_tube import Experiment
 
+exp = Experiment(version=a_previous_version_with_a_saved_checkpoint)
+trainer = Trainer(experiment=exp)
+
+# this fit call loads model weights and trainer state
+# the trainer continues seamlessly from where you left off
+# without having to do anything else.
+trainer.fit(model)
+```
+
+The trainer restores:  
+- global_step    
+- current_epoch    
+- All optimizers    
+- All lr_schedulers    
+- Model weights
+
+You can even change the logic of your model as long as the weights and "architecture" of 
+the system isn't different. If you add a layer, for instance, it might not work.   
+
+At a rough level, here's [what happens inside Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/model_saving.py#L63):   
+```python
+
+self.global_step = checkpoint['global_step']
+self.current_epoch = checkpoint['epoch']
+
+# restore the optimizers
+optimizer_states = checkpoint['optimizer_states']
+for optimizer, opt_state in zip(self.optimizers, optimizer_states):
+    optimizer.load_state_dict(opt_state)
+
+# restore the lr schedulers
+lr_schedulers = checkpoint['lr_schedulers']
+for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
+    scheduler.load_state_dict(lrs_state)
+
+# uses the model you passed into trainer        
+model.load_state_dict(checkpoint['state_dict'])
+```    

docs/Trainer/Distributed training.md

@@ -23,6 +23,32 @@ have configuration issues depending on your cluster.
 
 For a deeper understanding of what lightning is doing, feel free to read [this guide](https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565).   
 
+---
+#### Distributed and 16-bit precision.    
+Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
+not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.   
+
+Below are the possible configurations we support.    
+
+| 1 GPU  | 1+ GPUs  | DP  | DDP  | 16-bit  | command |
+|---|---|---|---|---|---|
+| Y  |   |   |   |  | ```Trainer(gpus=[0])``` |
+| Y  |   |   |   | Y | ```Trainer(gpus=[0], use_amp=True)``` |
+|   | Y | Y |   |   | ```Trainer(gpus=[0, ...])``` |
+|   | Y |  | Y  |  | ```Trainer(gpus=[0, ...], distributed_backend='ddp')``` |
+|   | Y |  | Y  | Y | ```Trainer(gpus=[0, ...], distributed_backend='ddp', use_amp=True)``` |
+
+
+---
+#### CUDA flags   
+CUDA flags make certain GPUs visible to your script. 
+Lightning sets these for you automatically, there's NO NEED to do this yourself.
+```python
+# lightning will set according to what you give the trainer
+# os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+# os.environ["CUDA_VISIBLE_DEVICES"] = "0"
+```
+
 ---
 #### 16-bit mixed precision
 16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.    
@@ -43,10 +69,6 @@ trainer = Trainer(amp_level='O2', use_amp=False)
 #### Single-gpu
 Make sure you're on a GPU machine. 
 ```python
-# set these flags
-os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-os.environ["CUDA_VISIBLE_DEVICES"] = "0"
-
 # DEFAULT
 trainer = Trainer(gpus=[0])
 ```
@@ -56,13 +78,6 @@ trainer = Trainer(gpus=[0])
 Make sure you're on a GPU machine. You can set as many GPUs as you want.
 In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
 ```python
-# set these flags
-# lightning sets these flags for you automatically
-# no need to set yourself
-# os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-# os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7"
-
-
 # to use DataParallel (default)
 trainer = Trainer(gpus=[0,1,2,3,4,5,6,7], distributed_backend='dp')
 
@@ -95,7 +110,7 @@ cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
 cluster.add_command('export NCCL_DEBUG=INFO')
 
 # setting a master port here is a good idea.
-cluster.add_command(f'export MASTER_PORT={PORT}')
+cluster.add_command('export MASTER_PORT=%r' % PORT)
 
 # good to load the latest NCCL version
 cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])

docs/Trainer/Logging.md

@@ -33,7 +33,7 @@ trainer = Trainer(process_position=1)
 Whenever you call .save() on the test-tube experiment it logs all the hyperparameters in current use.
 Give lightning a test-tube Experiment object to automate this for you.
 ``` {.python}
-from test-tube import Experiment
+from test_tube import Experiment
 
 exp = Experiment(...)
 Trainer(experiment=exp)
@@ -44,12 +44,48 @@ Trainer(experiment=exp)
 Whenever you call .save() on the test-tube experiment it snapshows all code and pushes to a git tag.
 Give lightning a test-tube Experiment object to automate this for you.
 ``` {.python}
-from test-tube import Experiment
+from test_tube import Experiment
 
 exp = Experiment(create_git_tag=True)
 Trainer(experiment=exp)
 ```
 
+---
+### Tensorboard support   
+In the LightningModule you can access the experiment logger by doing:
+```python
+self.experiment
+
+# add image
+# Look at PyTorch SummaryWriter docs for what you can do.   
+self.experiment.add_image(...)
+```
+
+The experiment object is a strict subclass of PyTorch SummaryWriter. However, this class
+also snapshots every detail about the experiment (data folder paths, code, hyperparams),
+and allows you to visualize it using tensorboard.
+``` {.python}
+from test_tube import Experiment, HyperOptArgumentParser
+
+# exp hyperparams
+args = HyperOptArgumentParser()
+hparams = args.parse_args()
+
+# this is a summaryWriter with nicer logging structure
+exp = Experiment(save_dir='/some/path', create_git_tag=True)
+
+# track experiment details (must be ArgumentParser or HyperOptArgumentParser).
+# each option in the parser is tracked
+exp.argparse(hparams)
+exp.tag({'description': 'running demo'})
+
+# trainer uses the exp object to log exp data
+trainer = Trainer(experiment=exp)
+trainer.fit(model)
+
+# view logs at:
+# tensorboard --logdir /some/path   
+```
 
 ---
 #### Write logs file to csv every k batches 

docs/Trainer/Training Loop.md

@@ -1,4 +1,4 @@
-The lightning training loop handles everything except the actual computations of your model. To decide what will happen in your training loop, define the [training_step function](../../Pytorch-lightning/LightningModule/#training_step).
+The lightning training loop handles everything except the actual computations of your model. To decide what will happen in your training loop, define the [training_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#training_step).
 
 Below are all the things lightning automates for you in the training loop.
 
@@ -11,17 +11,6 @@ Accumulated gradients runs K small batches of size N before doing a backwards pa
 trainer = Trainer(accumulate_grad_batches=1)
 ```
 
----
-#### Anneal Learning rate
-Cut the learning rate by 10 at every epoch listed in this list.
-``` {.python}
-# DEFAULT (don't anneal)
-trainer = Trainer(lr_scheduler_milestones=None)
-
-# cut LR by 10 at 100, 200, and 300 epochs 
-trainer = Trainer(lr_scheduler_milestones='100, 200, 300')
-```
-
 ---
 #### Force training for min or max epochs
 It can be useful to force training for a minimum number of epochs or limit to a max number

docs/Trainer/Validation loop.md

@@ -1,4 +1,4 @@
-The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the [validation_step function](../../Pytorch-lightning/LightningModule/#validation_step).
+The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the [validation_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#validation_step).
 Below are all the things lightning automates for you in the validation loop.
 
 **Note**   

docs/Trainer/hooks.md

@@ -0,0 +1,94 @@
+# Hooks
+[[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py)]   
+
+There are cases when you might want to do something different at different parts of the training/validation loop.
+To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
+
+**Contributing** If there's a hook you'd like to add, simply:    
+1. Fork PyTorchLightning.    
+2. Add the hook [here](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py).       
+3. Add the correct place in the [Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py) where it should be called.    
+
+---
+#### on_epoch_start
+Called in the training loop at the very beginning of the epoch.   
+```python
+def on_epoch_start(self):
+    # do something when the epoch starts
+```
+
+---
+#### on_epoch_end
+Called in the training loop at the very end of the epoch.   
+```python
+def on_epoch_end(self):
+    # do something when the epoch ends 
+```
+
+---
+#### on_batch_start
+Called in the training loop before anything happens for that batch.   
+```python
+def on_batch_start(self):
+    # do something when the batch starts
+```
+
+---
+#### on_batch_end
+Called in the training loop after the batch.   
+```python
+def on_batch_end(self):
+    # do something when the batch ends 
+```
+
+---
+#### on_pre_performance_check
+Called at the very beginning of the validation loop.   
+```python
+def on_pre_performance_check(self):
+    # do something before validation starts 
+```
+
+---
+#### on_post_performance_check
+Called at the very end of the validation loop.   
+```python
+def on_post_performance_check(self):
+    # do something before validation end
+```
+
+---
+#### on_tng_metrics
+Called in the training loop, right before metrics are logged.
+Although you can log at any time by using self.experiment, you can use
+this callback to modify what will be logged.
+```python
+def on_tng_metrics(self, metrics):
+    # do something before validation end
+```
+
+---
+#### on_before_zero_grad
+Called in the training loop after taking an optimizer step and before zeroing grads.
+Good place to inspect weight information with weights updated.
+
+Called once per optimizer
+```python
+def on_before_zero_grad(self, optimizer):
+    # do something with the optimizer or inspect it. 
+```
+
+---
+#### on_after_backward
+Called in the training loop after model.backward()
+This is the ideal place to inspect or log gradient information 
+```python
+def on_after_backward(self):
+    # example to inspect gradient information in tensorboard
+    if self.trainer.global_step % 25 == 0:  # don't make the tf file huge
+        params = self.state_dict()
+        for k, v in params.items():
+            grads = v
+            name = k
+            self.experiment.add_histogram(tag=name, values=grads, global_step=self.trainer.global_step)
+```

docs/Trainer/index.md

@@ -19,56 +19,61 @@ But of course the fun is in all the advanced things it can do:
 
 **Checkpointing**    
 
-- Model saving
-- Model loading 
+- [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
+- [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
+- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
 
 **Computing cluster (SLURM)**    
 
-- [Running grid search on a cluster](SLURM%20Managed%20Cluster/#running-grid-search-on-a-cluster)
-- [Walltime auto-resubmit](SLURM%20Managed%20Cluster/#walltime-auto-resubmit)   
+- [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
+- [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
 
 **Debugging**  
 
-- [Fast dev run](Debugging/#fast-dev-run)
-- [Inspect gradient norms](Debugging/#inspect-gradient-norms)
-- [Log GPU usage](Debugging/#Log-gpu-usage)
-- [Make model overfit on subset of data](Debugging/#make-model-overfit-on-subset-of-data)
-- [Print the parameter count by layer](Debugging/#print-the-parameter-count-by-layer)
-- [Pring which gradients are nan](Debugging/#print-which-gradients-are-nan)
+- [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
+- [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
+- [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
+- [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
+- [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
+- [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
+- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 
 
 **Distributed training**    
 
-- [16-bit mixed precision](Distributed%20training/#16-bit-mixed-precision)
-- [Multi-GPU](Distributed%20training/#Multi-GPU)
-- [Multi-node](Distributed%20training/#Multi-node)
-- [Single GPU](Distributed%20training/#single-gpu)
-- [Self-balancing architecture](Distributed%20training/#self-balancing-architecture)
+- [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
+- [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
+- [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
+- [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
+- [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
 
 
 **Experiment Logging**   
 
-- [Display metrics in progress bar](Logging/#display-metrics-in-progress-bar)
-- Log arbitrary metrics
-- [Log metric row every k batches](Logging/#log-metric-row-every-k-batches)
-- [Process position](Logging/#process-position)
-- [Save a snapshot of all hyperparameters](Logging/#save-a-snapshot-of-all-hyperparameters) 
-- [Snapshot code for a training run](Logging/#snapshot-code-for-a-training-run) 
-- [Write logs file to csv every k batches](Logging/#write-logs-file-to-csv-every-k-batches)
+- [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
+- [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
+- [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
+- [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
+- [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
+- [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
+- [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
 
 **Training loop**    
 
-- [Accumulate gradients](Training%20Loop/#accumulated-gradients)
-- [Anneal Learning rate](Training%20Loop/#anneal-learning-rate)
-- [Force training for min or max epochs](Training%20Loop/#force-training-for-min-or-max-epochs)
-- [Force disable early stop](Training%20Loop/#force-disable-early-stop)
-- [Use multiple optimizers (like GANs)](../Pytorch-lightning/LightningModule/#configure_optimizers)
-- [Set how much of the training set to check (1-100%)](Training%20Loop/#set-how-much-of-the-training-set-to-check)
+- [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
+- [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
+- [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
+- [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
+- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
+- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
+- [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 
 **Validation loop**    
 
-- [Check validation every n epochs](Validation%20Loop/#check-validation-every-n-epochs)
-- [Set how much of the validation set to check](Validation%20Loop/#set-how-much-of-the-validation-set-to-check)
-- [Set how much of the test set to check](Validation%20Loop/#set-how-much-of-the-test-set-to-check)
-- [Set validation check frequency within 1 training epoch](Validation%20Loop/#set-validation-check-frequency-within-1-training-epoch)
-- [Set the number of validation sanity steps](Validation%20Loop/#set-the-number-of-validation-sanity-steps)
+- [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
+- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
+- [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
+- [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
+- [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
+- [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)

docs/index.md

@@ -1,11 +1,17 @@
 ###### New project Quick Start    
-To start a new project define these two files.   
+To start a new project you define two files, a LightningModule and a Trainer file.    
 
-1. [Define a LightningModule](/pytorch-lightning/LightningModule/RequiredTrainerInterface/) 
-2. [Define a trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
-    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/examples/new_project_templates/single_cpu_template.py) 
-    - [Multi-GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/examples/new_project_templates/single_gpu_node_template.py)
-    - [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/examples/new_project_templates/multi_node_cluster_template.py)
+A separate trainer file allows to run many LightningModules. Each LightningModule has the core
+logic to a particular research project.    
+
+For example, one lightningModule could be an image classifier, the other
+one could be a seq-2-seq model, both (optionally) ran by the same trainer file.
+
+1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
+2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
+    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_cpu_template.py) 
+    - [Multi-GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_gpu_node_template.py)
+    - [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/multi_node_cluster_template.py)
 
 ###### Docs shortcuts
 - [LightningModule](LightningModule/RequiredTrainerInterface/)  
@@ -22,6 +28,7 @@ To start a new project define these two files.
 
 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
+- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
 
 ###### Computing cluster (SLURM)    
 
@@ -36,6 +43,7 @@ To start a new project define these two files.
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
 - [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
+- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 
 
 ###### Distributed training    
@@ -50,9 +58,9 @@ To start a new project define these two files.
 ###### Experiment Logging   
 
 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
-- Log arbitrary metrics
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
 - [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
+- [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
 - [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
@@ -60,16 +68,18 @@ To start a new project define these two files.
 ###### Training loop    
 
 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
-- [Anneal Learning rate](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#anneal-learning-rate)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
-- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
+- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
+- [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 
-######Validation loop    
+###### Validation loop    
 
 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
+- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
 - [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)

docs/requirements.txt

@@ -1 +1,2 @@
 mkdocs-material==4.4.0
+mkdocs==1.0.4

examples/__init__.py

@@ -1 +1,5 @@
-from .new_project_templates.lightning_module_template import LightningTemplateModel
+from .new_project_templates.lightning_module_template import LightningTemplateModel
+
+__all__ = [
+    'LightningTemplateModel'
+]

examples/new_project_templates/lightning_module_template.py

@@ -1,3 +1,6 @@
+"""
+Example template for defining a system
+"""
 import os
 from collections import OrderedDict
 import torch.nn as nn
@@ -10,7 +13,7 @@ from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
 
-import pytorch_lightning as ptl
+import pytorch_lightning as pl
 from pytorch_lightning.root_module.root_module import LightningModule
 
 
@@ -44,11 +47,13 @@ class LightningTemplateModel(LightningModule):
         Layout model
         :return:
         """
-        self.c_d1 = nn.Linear(in_features=self.hparams.in_features, out_features=self.hparams.hidden_dim)
+        self.c_d1 = nn.Linear(in_features=self.hparams.in_features,
+                              out_features=self.hparams.hidden_dim)
         self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
         self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)
 
-        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim, out_features=self.hparams.out_features)
+        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim,
+                              out_features=self.hparams.out_features)
 
     # ---------------------
     # TRAINING
@@ -146,12 +151,23 @@ class LightningTemplateModel(LightningModule):
         val_loss_mean = 0
         val_acc_mean = 0
         for output in outputs:
-            val_loss_mean += output['val_loss']
-            val_acc_mean += output['val_acc']
+            val_loss = output['val_loss']
+
+            # reduce manually when using dp
+            if self.trainer.use_dp:
+                val_loss = torch.mean(val_loss)
+            val_loss_mean += val_loss
+
+            # reduce manually when using dp
+            val_acc = output['val_acc']
+            if self.trainer.use_dp:
+                val_acc_mean = torch.mean(val_acc)
+
+            val_acc_mean += val_acc_mean
 
         val_loss_mean /= len(outputs)
         val_acc_mean /= len(outputs)
-        tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
+        tqdm_dic = {'val_loss': val_loss_mean, 'val_acc': val_acc_mean}
         return tqdm_dic
 
     # ---------------------
@@ -163,12 +179,15 @@ class LightningTemplateModel(LightningModule):
         :return: list of optimizers
         """
         optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
-        return [optimizer]
+        scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)
+        return [optimizer], [scheduler]
 
     def __dataloader(self, train):
         # init data generators
-        transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
-        dataset = MNIST(root=self.hparams.data_root, train=train, transform=transform, download=True)
+        transform = transforms.Compose([transforms.ToTensor(),
+                                        transforms.Normalize((0.5,), (1.0,))])
+        dataset = MNIST(root=self.hparams.data_root, train=train,
+                        transform=transform, download=True)
 
         # when using multi-node we need to add the datasampler
         train_sampler = None
@@ -178,7 +197,7 @@ class LightningTemplateModel(LightningModule):
             if self.on_gpu:
                 train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                 batch_size = batch_size // self.trainer.world_size  # scale batch size
-        except Exception as e:
+        except Exception:
             pass
 
         should_shuffle = train_sampler is None
@@ -191,23 +210,23 @@ class LightningTemplateModel(LightningModule):
 
         return loader
 
-    @ptl.data_loader
+    @pl.data_loader
     def tng_dataloader(self):
         print('tng data loader called')
         return self.__dataloader(train=True)
 
-    @ptl.data_loader
+    @pl.data_loader
     def val_dataloader(self):
         print('val data loader called')
         return self.__dataloader(train=False)
 
-    @ptl.data_loader
+    @pl.data_loader
     def test_dataloader(self):
         print('test data loader called')
         return self.__dataloader(train=False)
 
     @staticmethod
-    def add_model_specific_args(parent_parser, root_dir): # pragma: no cover
+    def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
         """
         Parameters you define here will be available to your model through self.hparams
         :param parent_parser:
@@ -220,20 +239,25 @@ class LightningTemplateModel(LightningModule):
         # parser.set_defaults(gradient_clip=5.0)
 
         # network params
-        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
-        parser.add_argument('--in_features', default=28*28, type=int)
+        parser.add_argument('--in_features', default=28 * 28, type=int)
         parser.add_argument('--out_features', default=10, type=int)
-        parser.add_argument('--hidden_dim', default=50000, type=int) # use 500 for CPU, 50000 for GPU to see speed difference
+        # use 500 for CPU, 50000 for GPU to see speed difference
+        parser.add_argument('--hidden_dim', default=50000, type=int)
+        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
 
         # data
         parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
 
         # training params (opt)
-        parser.opt_list('--learning_rate', default=0.001*8, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
+        parser.opt_list('--learning_rate', default=0.001 * 8, type=float,
+                        options=[0.0001, 0.0005, 0.001, 0.005],
                         tunable=False)
-        parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)
+        parser.opt_list('--optimizer_name', default='adam', type=str,
+                        options=['adam'], tunable=False)
 
-        # if using 2 nodes with 4 gpus each the batch size here (256) will be 256 / (2*8) = 16 per gpu
-        parser.opt_list('--batch_size', default=256*8, type=int, options=[32, 64, 128, 256], tunable=False,
-                        help='batch size will be divided over all the gpus being used across all nodes')
+        # if using 2 nodes with 4 gpus each the batch size here
+        #  (256) will be 256 / (2*8) = 16 per gpu
+        parser.opt_list('--batch_size', default=256 * 8, type=int,
+                        options=[32, 64, 128, 256], tunable=False,
+                        help='batch size will be divided over all gpus being used across all nodes')
         return parser

examples/new_project_templates/multi_node_cluster_template.py

@@ -1,31 +1,21 @@
+"""
+Multi-node example (GPU)
+"""
 import os
-import sys
 import numpy as np
 from time import sleep
 import torch
 
 from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
 from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
-
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 
+from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 
-# ---------------------
-# DEFINE MODEL HERE
-# ---------------------
-from lightning_module_template import LightningTemplateModel
-# ---------------------
-
-"""
-Allows training by using command line arguments
-Run by:
-# TYPE YOUR RUN COMMAND HERE
-"""
-
 
 def main_local(hparams):
     main(hparams, None, None)
@@ -122,8 +112,10 @@ def optimize_on_cluster(hyperparams):
     cluster.add_command('source activate lightning')
 
     # run only on 32GB voltas
-    cluster.add_slurm_cmd(cmd='constraint', value='volta32gb', comment='use 32gb gpus')
-    cluster.add_slurm_cmd(cmd='partition', value=hyperparams.gpu_partition, comment='use 32gb gpus')
+    cluster.add_slurm_cmd(cmd='constraint', value='volta32gb',
+                          comment='use 32gb gpus')
+    cluster.add_slurm_cmd(cmd='partition', value=hyperparams.gpu_partition,
+                          comment='use 32gb gpus')
 
     # run hopt
     # creates and submits jobs to slurm
@@ -150,15 +142,23 @@ if __name__ == '__main__':
     parent_parser.add_argument('--gpu_partition', type=str, help='consult your cluster manual')
 
     # TODO: make 1 param
-    parent_parser.add_argument('--per_experiment_nb_gpus', type=int, help='how many gpus to use in a node')
-    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node')
+    parent_parser.add_argument('--per_experiment_nb_gpus', type=int,
+                               help='how many gpus to use in a node')
+    parent_parser.add_argument('--gpus', type=str, default='-1',
+                               help='how many gpus to use in the node')
 
-    parent_parser.add_argument('--nb_gpu_nodes', type=int, default=1, help='how many nodes to use in a cluster')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
-    parent_parser.add_argument('--slurm_log_path', type=str, default=slurm_out_dir, help='where to save slurm meta')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
-    parent_parser.add_argument('--nb_hopt_trials', type=int, default=1, help='how many grid search trials to run')
+    parent_parser.add_argument('--nb_gpu_nodes', type=int, default=1,
+                               help='how many nodes to use in a cluster')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
+                               help='where to save logs')
+    parent_parser.add_argument('--slurm_log_path', type=str, default=slurm_out_dir,
+                               help='where to save slurm meta')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
+                               help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
+                               help='test tube exp name')
+    parent_parser.add_argument('--nb_hopt_trials', type=int, default=1,
+                               help='how many grid search trials to run')
 
     # allow model to overwrite or extend args
     parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)

examples/new_project_templates/single_cpu_template.py

@@ -1,24 +1,20 @@
 """
-Runs a model on a single node across N-gpus.
+Runs a model on a single node on CPU only..
 """
 import os
-import sys
 import numpy as np
-from time import sleep
 import torch
 
-from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
-
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 
+from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 
-from lightning_module_template import LightningTemplateModel
-
 
 def main(hparams):
     """
@@ -94,9 +90,12 @@ if __name__ == '__main__':
     parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
 
     # gpu args
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
+    parent_parser.add_argument('--test_tube_save_path', type=str,
+                               default=test_tube_dir, help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str,
+                               default=checkpoint_dir, help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str,
+                               default='pt_lightning_exp_a', help='test tube exp name')
 
     # allow model to overwrite or extend args
     parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -106,5 +105,5 @@ if __name__ == '__main__':
     # RUN TRAINING
     # ---------------------
     # run on HPC cluster
-    print(f'RUNNING ON CPU')
+    print('RUNNING ON CPU')
     main(hyperparams)

examples/new_project_templates/single_gpu_node_16bit_template.py

@@ -1,24 +1,20 @@
 """
-Runs a model on a single node across N-gpus.
+16-bit single node, CPU example
 """
 import os
-import sys
 import numpy as np
-from time import sleep
 import torch
 
-from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
-
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 
+from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 
-from lightning_module_template import LightningTemplateModel
-
 
 def main(hparams):
     """
@@ -96,10 +92,15 @@ if __name__ == '__main__':
     parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
 
     # gpu args
-    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node. -1 uses all the gpus on the node')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
+    parent_parser.add_argument('--gpus', type=str, default='-1',
+                               help='how many gpus to use in the node.'
+                                    'value -1 uses all the gpus on the node')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
+                               help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
+                               help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
+                               help='test tube exp name')
 
     # allow model to overwrite or extend args
     parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -109,5 +110,5 @@ if __name__ == '__main__':
     # RUN TRAINING
     # ---------------------
     # run on HPC cluster
-    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
+    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
     main(hyperparams)

examples/new_project_templates/single_gpu_node_ddp_template.py

@@ -2,23 +2,19 @@
 Runs a model on a single node across N-gpus.
 """
 import os
-import sys
 import numpy as np
-from time import sleep
 import torch
 
-from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
-
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 
+from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 
-from lightning_module_template import LightningTemplateModel
-
 
 def main(hparams):
     """
@@ -75,6 +71,7 @@ def main(hparams):
         checkpoint_callback=checkpoint,
         early_stop_callback=early_stop,
         gpus=hparams.gpus,
+        distributed_backend='ddp'
     )
 
     # ------------------------
@@ -95,10 +92,15 @@ if __name__ == '__main__':
     parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
 
     # gpu args
-    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node. -1 uses all the gpus on the node')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
+    parent_parser.add_argument('--gpus', type=str, default='-1',
+                               help='how many gpus to use in the node.'
+                                    ' value -1 uses all the gpus on the node')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
+                               help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
+                               help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
+                               help='test tube exp name')
 
     # allow model to overwrite or extend args
     parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -108,5 +110,5 @@ if __name__ == '__main__':
     # RUN TRAINING
     # ---------------------
     # run on HPC cluster
-    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
+    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
     main(hyperparams)

examples/new_project_templates/single_gpu_node_dp_template.py

@@ -1,24 +1,20 @@
 """
-Runs a model on a single node across N-gpus.
+Runs a model on a single node across N-gpus using dataParallel
 """
 import os
-import sys
 import numpy as np
-from time import sleep
 import torch
 
-from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
-
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 
+from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 
-from lightning_module_template import LightningTemplateModel
-
 
 def main(hparams):
     """
@@ -95,10 +91,15 @@ if __name__ == '__main__':
     parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
 
     # gpu args
-    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node. -1 uses all the gpus on the node')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
+    parent_parser.add_argument('--gpus', type=str, default='-1',
+                               help='how many gpus to use in the node.'
+                                    ' value -1 uses all the gpus on the node')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
+                               help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
+                               help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
+                               help='test tube exp name')
 
     # allow model to overwrite or extend args
     parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -108,5 +109,5 @@ if __name__ == '__main__':
     # RUN TRAINING
     # ---------------------
     # run on HPC cluster
-    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
+    print('RUNNING INTERACTIVE MODE ON GPUS. gpu ids: %i' % hyperparams.gpus)
     main(hyperparams)

examples/new_project_templates/trainer_cpu_template.py

@@ -3,9 +3,10 @@ import sys
 
 from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
+from pytorch_lightning.utilities.arg_parse import add_default_args
 from pytorch_lightning.callbacks.pt_callbacks import EarlyStopping, ModelCheckpoint
-from docs.source.examples.example_model import ExampleModel
+
+from examples.new_project_templates.lightning_module_template import LightningTemplateModel
 
 
 def main(hparams):
@@ -28,7 +29,7 @@ def main(hparams):
     exp.save()
 
     # build model
-    model = ExampleModel(hparams)
+    model = LightningTemplateModel(hparams)
 
     # callbacks
     early_stop = EarlyStopping(
@@ -66,7 +67,7 @@ if __name__ == '__main__':
     add_default_args(parent_parser, root_dir)
 
     # allow model to overwrite or extend args
-    parser = ExampleModel.add_model_specific_args(parent_parser)
+    parser = LightningTemplateModel.add_model_specific_args(parent_parser)
     hyperparams = parser.parse_args()
 
     # train model

mkdocs.yml

@@ -1,10 +1,10 @@
-site_name: Pytorch lightning Documentation
+site_name: PyTorch lightning Documentation
 theme:
   name: 'material'
 docs_dir: docs
 repo_url: https://github.com/williamFalcon/pytorch-lightning
 site_dir: 'site'
-site_description: 'Documentation for Pytorch LightningModule, the researcher version of keras.'
+site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
 
 dev_addr: '0.0.0.0:8000'
 #google_analytics: ['UA-aasd', 'sitename']

pytorch_lightning/__init__.py

@@ -1,3 +1,9 @@
-from .models import Trainer
+from .models.trainer import Trainer
 from .root_module.root_module import LightningModule
-from .root_module.decorators import data_loader
+from .root_module.decorators import data_loader
+
+__all__ = [
+    'Trainer',
+    'LightningModule',
+    'data_loader',
+]

pytorch_lightning/callbacks/__init__.py

@@ -1 +1,6 @@
-from .pt_callbacks import EarlyStopping, ModelCheckpoint
+from .pt_callbacks import EarlyStopping, ModelCheckpoint
+
+__all__ = [
+    'EarlyStopping',
+    'ModelCheckpoint',
+]

pytorch_lightning/callbacks/pt_callbacks.py

@@ -1,5 +1,8 @@
+import os
+import shutil
+
 import numpy as np
-import os, shutil
+
 from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel
 
 
@@ -119,9 +122,9 @@ class EarlyStopping(Callback):
         current = logs.get(self.monitor)
         stop_training = False
         if current is None:
-            print('Early stopping conditioned on metric `%s` ''which is not available. Available metrics are: %s' %
-                (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
-            )
+            print('Early stopping conditioned on metric `%s` '
+                  'which is not available. Available metrics are: %s' %
+                  (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning)
             exit(-1)
 
         if self.monitor_op(current - self.min_delta, self.best):
@@ -185,8 +188,7 @@ class ModelCheckpoint(Callback):
 
         if mode not in ['auto', 'min', 'max']:
             print('ModelCheckpoint mode %s is unknown, '
-                          'fallback to auto mode.' % (mode),
-                          RuntimeWarning)
+                  'fallback to auto mode.' % (mode), RuntimeWarning)
             mode = 'auto'
 
         if mode == 'min':
@@ -230,8 +232,8 @@ class ModelCheckpoint(Callback):
             if self.save_best_only:
                 current = logs.get(self.monitor)
                 if current is None:
-                    print('Can save best model only with %s available, '
-                                  'skipping.' % (self.monitor), RuntimeWarning)
+                    print('Can save best model only with %s available,'
+                          ' skipping.' % (self.monitor), RuntimeWarning)
                 else:
                     if self.monitor_op(current, self.best):
                         if self.verbose > 0:
@@ -260,4 +262,3 @@ if __name__ == '__main__':
         print(loss)
         if should_stop:
             break
-

pytorch_lightning/examples/new_project_templates/single_gpu_template.py

@@ -1,112 +0,0 @@
-"""
-Runs a model on a single node across N-gpus.
-"""
-import os
-import sys
-import numpy as np
-from time import sleep
-import torch
-
-from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
-from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
-
-from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
-
-SEED = 2334
-torch.manual_seed(SEED)
-np.random.seed(SEED)
-
-from lightning_module_template import LightningTemplateModel
-
-
-def main(hparams):
-    """
-    Main training routine specific for this project
-    :param hparams:
-    :return:
-    """
-    # ------------------------
-    # 1 INIT LIGHTNING MODEL
-    # ------------------------
-    print('loading model...')
-    model = LightningTemplateModel(hparams)
-    print('model built')
-
-    # ------------------------
-    # 2 INIT TEST TUBE EXP
-    # ------------------------
-
-    # init experiment
-    exp = Experiment(
-        name=hyperparams.experiment_name,
-        save_dir=hyperparams.test_tube_save_path,
-        autosave=False,
-        description='test demo'
-    )
-
-    exp.argparse(hparams)
-    exp.save()
-
-    # ------------------------
-    # 3 DEFINE CALLBACKS
-    # ------------------------
-    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
-    early_stop = EarlyStopping(
-        monitor='val_acc',
-        patience=3,
-        verbose=True,
-        mode='max'
-    )
-
-    checkpoint = ModelCheckpoint(
-        filepath=model_save_path,
-        save_best_only=True,
-        verbose=True,
-        monitor='val_loss',
-        mode='min'
-    )
-
-    # ------------------------
-    # 4 INIT TRAINER
-    # ------------------------
-    trainer = Trainer(
-        experiment=exp,
-        checkpoint_callback=checkpoint,
-        early_stop_callback=early_stop,
-        gpus=hparams.gpus,
-    )
-
-    # ------------------------
-    # 5 START TRAINING
-    # ------------------------
-    trainer.fit(model)
-
-
-if __name__ == '__main__':
-
-    # dirs
-    root_dir = os.path.dirname(os.path.realpath(__file__))
-    demo_log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
-    checkpoint_dir = os.path.join(demo_log_dir, 'model_weights')
-    test_tube_dir = os.path.join(demo_log_dir, 'test_tube_data')
-
-    # although we user hyperOptParser, we are using it only as argparse right now
-    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
-
-    # gpu args
-    parent_parser.add_argument('--gpus', type=str, default='0', help='how many gpus to use in the node. -1 uses all the gpus on the node')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
-
-    # allow model to overwrite or extend args
-    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
-    hyperparams = parser.parse_args()
-
-    # ---------------------
-    # RUN TRAINING
-    # ---------------------
-    # run on HPC cluster
-    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
-    main(hyperparams)

pytorch_lightning/models/__init__.py

@@ -1 +0,0 @@
-from .trainer import Trainer

pytorch_lightning/models/trainer.py

@@ -1,30 +1,28 @@
 """
-The trainer handles all the logic for running a val loop, training loop, distributing, etc...
+The trainer handles all the logic for running a val loop, training loop, distributing, etc.. .
 """
-import subprocess
-import traceback
-import warnings
-import os
-import pdb
-import re
 
-import torch
-from torch.utils.data.distributed import DistributedSampler
-from torch.optim.lr_scheduler import MultiStepLR
-import torch.multiprocessing as mp
-import torch.distributed as dist
+import os
+import re
+import warnings
+
 import numpy as np
 import tqdm
+import torch
+from torch.utils.data.distributed import DistributedSampler
+import torch.multiprocessing as mp
+import torch.distributed as dist
 
 from pytorch_lightning.root_module.memory import get_gpu_memory_map
 from pytorch_lightning.root_module.model_saving import TrainerIO
-from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel, LightningDataParallel
-from pytorch_lightning.utils.debugging import MisconfigurationException
+from pytorch_lightning.pt_overrides.override_data_parallel import (
+    LightningDistributedDataParallel, LightningDataParallel)
+from pytorch_lightning.utilities.debugging import MisconfigurationException
 
 try:
     from apex import amp
     APEX_AVAILABLE = True
-except Exception:
+except ImportError:
     APEX_AVAILABLE = False
 
 
@@ -52,7 +50,7 @@ def reduce_distributed_output(output, nb_gpus):
 class Trainer(TrainerIO):
 
     def __init__(self,
-                 experiment,
+                 experiment=None,
                  early_stop_callback=None,
                  checkpoint_callback=None,
                  gradient_clip=0,
@@ -67,18 +65,20 @@ class Trainer(TrainerIO):
                  check_val_every_n_epoch=1,
                  fast_dev_run=False,
                  accumulate_grad_batches=1,
-                 max_nb_epochs=1000, min_nb_epochs=1,
-                 train_percent_check=1.0, val_percent_check=1.0, test_percent_check=1.0,
+                 max_nb_epochs=1000,
+                 min_nb_epochs=1,
+                 train_percent_check=1.0,
+                 val_percent_check=1.0,
+                 test_percent_check=1.0,
                  val_check_interval=0.95,
-                 log_save_interval=100, add_log_row_interval=10,
-                 lr_scheduler_milestones=None,
+                 log_save_interval=100,
+                 add_log_row_interval=10,
                  distributed_backend='dp',
                  use_amp=False,
                  print_nan_grads=False,
                  print_weights_summary=True,
                  amp_level='O2',
                  nb_sanity_val_steps=5):
-
         """
 
         :param experiment: Test-tube experiment
@@ -104,17 +104,15 @@ class Trainer(TrainerIO):
         :param val_check_interval:
         :param log_save_interval:
         :param add_log_row_interval:
-        :param lr_scheduler_milestones:
-        :param distributed_backend: 'np' to use DistributedParallel, 'ddp' to use DistributedDataParallel
+        :param distributed_backend:
+            'do' to use DistributedParallel, 'dp' to use DistributedDataParallel, 'n' to use none
         :param use_amp:
         :param print_nan_grads:
         :param print_weights_summary:
         :param amp_level:
         :param nb_sanity_val_steps:
         """
-
         # Transfer params
-
         self.nb_gpu_nodes = nb_gpu_nodes
         self.gradient_clip = gradient_clip
         self.check_val_every_n_epoch = check_val_every_n_epoch
@@ -124,7 +122,9 @@ class Trainer(TrainerIO):
         self.on_gpu = gpus is not None and torch.cuda.is_available()
         self.progress_bar = progress_bar
         self.experiment = experiment
-        self.exp_save_path = experiment.get_data_path(experiment.name, experiment.version)
+        self.exp_save_path = None
+        if self.experiment is not None:
+            self.exp_save_path = experiment.get_data_path(experiment.name, experiment.version)
         self.cluster = cluster
         self.process_position = process_position
         self.current_gpu_name = current_gpu_name
@@ -141,7 +141,6 @@ class Trainer(TrainerIO):
         self.early_stop_callback = early_stop_callback
         self.min_nb_epochs = min_nb_epochs
         self.nb_sanity_val_steps = nb_sanity_val_steps
-        self.lr_scheduler_milestones = [] if lr_scheduler_milestones is None else [int(x.strip()) for x in lr_scheduler_milestones.split(',')]
         self.lr_schedulers = []
         self.amp_level = amp_level
         self.print_nan_grads = print_nan_grads
@@ -150,6 +149,7 @@ class Trainer(TrainerIO):
         self.node_rank = 0
         self.use_ddp = False
         self.use_dp = False
+        self.single_gpu = False
 
         # training bookeeping
         self.total_batch_nb = 0
@@ -177,13 +177,14 @@ class Trainer(TrainerIO):
 
             # set the correct cuda visible devices (using pci order)
             os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-            os.environ["CUDA_VISIBLE_DEVICES"] = ','.join([str(x) for x in self.data_parallel_device_ids])
-            print(f'VISIBLE GPUS: {os.environ["CUDA_VISIBLE_DEVICES"]}')
+            os.environ["CUDA_VISIBLE_DEVICES"] = ','.join([str(x) for x in
+                                                           self.data_parallel_device_ids])
+            print('VISIBLE GPUS: %r' % os.environ["CUDA_VISIBLE_DEVICES"])
 
         # make DP and DDP mutually exclusive
         # single GPU will also use DP with devices=[0]
-        have_gpus = self.data_parallel_device_ids is not None and len(self.data_parallel_device_ids) > 0
-        if have_gpus:
+        requested_gpus = self.data_parallel_device_ids is not None
+        if requested_gpus and len(self.data_parallel_device_ids) > 0:
             self.use_dp = distributed_backend == 'dp'
             self.use_ddp = distributed_backend == 'ddp'
 
@@ -196,6 +197,12 @@ class Trainer(TrainerIO):
                     'To silence this warning set distributed_backend=ddp'
                 warnings.warn(w)
 
+        # remove dp and ddp when requesting single gpu
+        if self.data_parallel_device_ids is not None and len(self.data_parallel_device_ids) == 1:
+            self.use_ddp = False
+            self.use_dp = False
+            self.single_gpu = True
+
         # extract SLURM flag vars
         # whenever we have the correct number of tasks, we let slurm manage processes
         # otherwise we launch the required number of processes
@@ -205,7 +212,7 @@ class Trainer(TrainerIO):
             try:
                 self.nb_slurm_tasks = int(os.environ['SLURM_NTASKS'])
                 self.is_slurm_managing_tasks = self.nb_slurm_tasks == self.nb_requested_gpus
-            except Exception as e:
+            except Exception:
                 # likely not on slurm, so set the slurm managed flag to false
                 self.is_slurm_managing_tasks = False
 
@@ -230,7 +237,8 @@ class Trainer(TrainerIO):
         self.val_dataloader = None
 
         # how much of the data to use
-        self.__determine_data_use_amount(train_percent_check, val_percent_check, test_percent_check, overfit_pct)
+        self.__determine_data_use_amount(train_percent_check, val_percent_check,
+                                         test_percent_check, overfit_pct)
         print('gpu available: {}, used: {}'.format(torch.cuda.is_available(), self.on_gpu))
 
         # 16 bit mixed precision training using apex
@@ -239,20 +247,52 @@ class Trainer(TrainerIO):
             print('using 16bit precision')
 
         if use_amp and not APEX_AVAILABLE:  # pragma: no cover
-            msg = '''
+            msg = """
             You set use_amp=True but do not have apex installed.
-            Install apex first using this guide and rerun with use_amp=True: 
+            Install apex first using this guide and rerun with use_amp=True:
             https://github.com/NVIDIA/apex#linux
-            
+
             this run will NOT use 16 bit precision
-            '''
+            """
             raise ModuleNotFoundError(msg)
 
+    def restore_state_if_existing_checkpoint(self):
+        # restore trainer state and model if there is a weight for this experiment
+        last_epoch = -1
+        last_ckpt_name = None
+
+        # do nothing if there's not dir or callback
+        no_ckpt_callback = self.checkpoint_callback is None
+        if no_ckpt_callback or not os.path.exists(self.checkpoint_callback.filepath):
+            return
+
+        # find last epoch
+        checkpoints = os.listdir(self.checkpoint_callback.filepath)
+        for name in checkpoints:
+            # ignore hpc ckpts
+            if 'hpc_' in name:
+                continue
+
+            if '.ckpt' in name:
+                epoch = name.split('epoch_')[1]
+                epoch = int(re.sub('[^0-9]', '', epoch))
+
+                if epoch > last_epoch:
+                    last_epoch = epoch
+                    last_ckpt_name = name
+
+        # restore last checkpoint
+        if last_ckpt_name is not None:
+            last_ckpt_path = os.path.join(self.checkpoint_callback.filepath, last_ckpt_name)
+            self.restore(last_ckpt_path, self.on_gpu)
+            print(f'model and trainer restored from checkpoint: {last_ckpt_path}')
+
     @property
     def data_parallel(self):
         return self.use_dp or self.use_ddp
 
-    def __determine_data_use_amount(self, train_percent_check, val_percent_check, test_percent_check, overfit_pct):
+    def __determine_data_use_amount(self, train_percent_check, val_percent_check,
+                                    test_percent_check, overfit_pct):
         """
         Use less data for debugging purposes
         """
@@ -274,13 +314,15 @@ class Trainer(TrainerIO):
 
     @property
     def __tng_tqdm_dic(self):
-        # ForkedPdb().set_trace()
         tqdm_dic = {
             'tng_loss': '{0:.3f}'.format(self.avg_loss),
-            'v_nb': '{}'.format(self.experiment.version),
             'epoch': '{}'.format(self.current_epoch),
-            'batch_nb':'{}'.format(self.batch_nb),
+            'batch_nb': '{}'.format(self.batch_nb),
         }
+
+        if self.experiment is not None:
+            tqdm_dic['v_nb'] = self.experiment.version
+
         tqdm_dic.update(self.tqdm_metrics)
 
         if self.on_gpu:
@@ -357,7 +399,15 @@ class Trainer(TrainerIO):
                 output = model(data_batch, batch_i)
             elif self.use_dp:
                 output = model(data_batch, batch_i)
-                output = reduce_distributed_output(output, len(self.data_parallel_device_ids))
+            elif self.single_gpu:
+                # put inputs on gpu manually
+                gpu_id = self.data_parallel_device_ids[0]
+                for i, x in enumerate(data_batch):
+                    if isinstance(x, torch.Tensor):
+                        data_batch[i] = x.cuda(gpu_id)
+
+                # do non dp, ddp step
+                output = model.validation_step(data_batch, batch_i)
 
             else:
                 output = model.validation_step(data_batch, batch_i)
@@ -393,18 +443,19 @@ class Trainer(TrainerIO):
         self.val_dataloader = model.val_dataloader
 
         if self.use_ddp and not isinstance(self.tng_dataloader.sampler, DistributedSampler):
-            msg = '''
-            when using multiple gpus and multiple nodes you must pass a DistributedSampler to DataLoader(sampler).
-            
-            ie: this:
-            dataset = myDataset()
-            dataloader = Dataloader(dataset)
-            
-            becomes:
-            dataset = myDataset()
-            dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-            dataloader = Dataloader(dataset, sampler=dist_sampler)
-            '''
+            msg = """
+when using multiple gpus and multiple nodes you must pass
+ a DistributedSampler to DataLoader(sampler).
+
+ie: this:
+dataset = myDataset()
+dataloader = Dataloader(dataset)
+
+becomes:
+dataset = myDataset()
+dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+dataloader = Dataloader(dataset, sampler=dist_sampler)
+"""
             raise MisconfigurationException(msg)
 
     # -----------------------------
@@ -414,19 +465,21 @@ class Trainer(TrainerIO):
 
         # when using multi-node or DDP within a node start each module in a separate process
         if self.use_ddp:
-            # must copy only the meta of the exp so it survives pickle/unpickle when going to new process
-            self.experiment = self.experiment.get_meta_copy()
+            # must copy only the meta of the exp so it survives pickle/unpickle
+            #  when going to new process
+            if self.experiment is not None:
+                self.experiment = self.experiment.get_meta_copy()
 
             if self.is_slurm_managing_tasks:
                 task = int(os.environ['SLURM_LOCALID'])
                 self.ddp_train(task, model)
             else:
-                msg = f"""
-                You requested {self.nb_requested_gpus} GPUs but launched {self.nb_slurm_tasks} slurm tasks. 
-                We will launch {self.nb_requested_gpus} processes for you. 
-                We recommend you let slurm manage the processes by setting: --ntasks-per-node={self.nb_requested_gpus}
-                If you're not using SLURM, ignore this message!
-                """
+                msg = """
+You requested %(nb_gpus)s GPUs but launched %(nb_tasks)s slurm tasks.
+We will launch %(nb_gpus)s processes for you.
+We recommend you let slurm manage the processes by setting: --ntasks-per-node=%(nb_gpus)s
+If you're not using SLURM, ignore this message!
+""" % {'nb_gpus': self.nb_requested_gpus, 'nb_tasks': self.nb_slurm_tasks}
                 warnings.warn(msg)
                 mp.spawn(self.ddp_train, nprocs=len(self.data_parallel_device_ids), args=(model, ))
 
@@ -435,15 +488,21 @@ class Trainer(TrainerIO):
         elif self.use_dp:
             self.__dp_train(model)
 
+        elif self.single_gpu:
+            self.__single_gpu_train(model)
+
         # ON CPU
         else:
             # run through amp wrapper
             if self.use_amp:
-                raise MisconfigurationException('amp + cpu is not supported. Please use a GPU option')
+                raise MisconfigurationException('amp + cpu is not supported.'
+                                                ' Please use a GPU option')
 
             # CHOOSE OPTIMIZER
-            # filter out the weights that were done on gpu so we can load on good old cpus
+            # allow for lr schedulers as well
             self.optimizers = model.configure_optimizers()
+            if len(self.optimizers) == 2:
+                self.optimizers, self.lr_schedulers = self.optimizers
 
             self.__run_pretrain_routine(model)
 
@@ -451,20 +510,42 @@ class Trainer(TrainerIO):
         # used for testing or when we need to know that training succeeded
         return 1
 
+    def __single_gpu_train(self, model):
+        # CHOOSE OPTIMIZER
+        # allow for lr schedulers as well
+        self.optimizers = model.configure_optimizers()
+        if len(self.optimizers) == 2:
+            self.optimizers, self.lr_schedulers = self.optimizers
+
+        model.cuda(self.data_parallel_device_ids[0])
+
+        if self.use_amp:
+            # An example
+            model, optimizers = amp.initialize(
+                model, self.optimizers, opt_level=self.amp_level,
+            )
+            self.optimizers = optimizers
+
+        self.__run_pretrain_routine(model)
+
     def __dp_train(self, model):
 
         # CHOOSE OPTIMIZER
-        # filter out the weights that were done on gpu so we can load on good old cpus
+        # allow for lr schedulers as well
         self.optimizers = model.configure_optimizers()
+        if len(self.optimizers) == 2:
+            self.optimizers, self.lr_schedulers = self.optimizers
 
         model.cuda(self.data_parallel_device_ids[0])
 
         # check for this bug (amp + dp + !01 doesn't work)
         # https://github.com/NVIDIA/apex/issues/227
         if self.use_dp and self.use_amp:
-            m = f'amp level {self.amp_level} with DataParallel is not supported. ' \
-                f'See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227. ' \
-                f'We recommend you switch to ddp if you want to use amp'
+            m = """
+Amp level %r with DataParallel is not supported.
+See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227.
+We recommend you switch to ddp if you want to use amp
+""" % self.amp_level
             raise MisconfigurationException(m)
 
         model = LightningDataParallel(model, device_ids=self.data_parallel_device_ids)
@@ -484,13 +565,14 @@ class Trainer(TrainerIO):
         try:
             node_id = os.environ['SLURM_NODEID']
             self.node_rank = int(node_id)
-        except Exception as e:
+        except Exception:
             self.node_rank = 0
 
         # recover original exp before went into process
         # init in write mode only on proc 0
-        self.experiment.debug = self.proc_rank > 0
-        self.experiment = self.experiment.get_non_ddp_exp()
+        if self.experiment is not None:
+            self.experiment.debug = self.proc_rank > 0
+            self.experiment = self.experiment.get_non_ddp_exp()
 
         # show progbar only on prog_rank 0
         self.prog_bar = self.prog_bar and self.node_rank == 0 and gpu_nb == 0
@@ -500,7 +582,8 @@ class Trainer(TrainerIO):
         self.world_size = self.nb_gpu_nodes * len(self.data_parallel_device_ids)
 
         # let the exp know the rank to avoid overwriting logs
-        self.experiment.rank = self.proc_rank
+        if self.experiment is not None:
+            self.experiment.rank = self.proc_rank
 
         # set up server using proc 0's ip address
         # try to init for 20 times at max in case ports are taken
@@ -508,8 +591,10 @@ class Trainer(TrainerIO):
         self.__init_tcp_connection()
 
         # CHOOSE OPTIMIZER
-        # filter out the weights that were done on gpu so we can load on good old cpus
+        # allow for lr schedulers as well
         self.optimizers = model.configure_optimizers()
+        if len(self.optimizers) == 2:
+            self.optimizers, self.lr_schedulers = self.optimizers
 
         # MODEL
         # copy model to each gpu
@@ -525,7 +610,8 @@ class Trainer(TrainerIO):
             )
             self.optimizers = optimizers
 
-        model = LightningDistributedDataParallel(model, device_ids=[gpu_nb], find_unused_parameters=True)
+        model = LightningDistributedDataParallel(model, device_ids=[gpu_nb],
+                                                 find_unused_parameters=True)
 
         # continue training routine
         self.__run_pretrain_routine(model)
@@ -541,14 +627,14 @@ class Trainer(TrainerIO):
         # sets the appropriate port
         try:
             port = os.environ['MASTER_PORT']
-        except Exception as e:
+        except Exception:
             port = 12910
-            os.environ['MASTER_PORT'] = f'{port}'
+            os.environ['MASTER_PORT'] = str(port)
 
         # figure out the root node addr
         try:
             root_node = os.environ['SLURM_NODELIST'].split(' ')[0]
-        except Exception as e:
+        except Exception:
             root_node = '127.0.0.2'
 
         root_node = self.resolve_root_node_address(root_node)
@@ -589,47 +675,50 @@ class Trainer(TrainerIO):
         # init training constants
         self.__layout_bookeeping()
 
-        # add lr schedulers
-        if self.lr_scheduler_milestones is not None:
-            for optimizer in self.optimizers:
-                scheduler = MultiStepLR(optimizer, self.lr_scheduler_milestones)
-                self.lr_schedulers.append(scheduler)
-
         # print model summary
         if self.proc_rank == 0 and self.print_weights_summary:
             ref_model.summarize()
 
         # give model convenience properties
         ref_model.trainer = self
-        ref_model.experiment = self.experiment
 
-        # run tiny validation to make sure program won't crash during val
-        _ = self.validate(model, self.val_dataloader, max_batches=self.nb_sanity_val_steps)
+        if self.experiment is not None:
+            ref_model.experiment = self.experiment
 
         # save exp to get started
-        if self.proc_rank == 0:
+        if self.proc_rank == 0 and self.experiment is not None:
             self.experiment.save()
 
         # track model now.
         # if cluster resets state, the model will update with the saved weights
         self.model = model
 
+        # restore training and model before hpc call
+        self.restore_state_if_existing_checkpoint()
+
         # enable cluster checkpointing
         # also restores training state
+        # hpc checkpoint overrides any other checkpoints loaded before
         if self.cluster is not None:  # pragma: no cover
             self.enable_auto_hpc_walltime_manager()
 
+        # run tiny validation to make sure program won't crash during val
+        ref_model.on_sanity_check_start()
+        _ = self.validate(model, self.val_dataloader, max_batches=self.nb_sanity_val_steps)
+
         # ---------------------------
         # CORE TRAINING LOOP
         # ---------------------------
+
         self.__train()
 
     def __train(self):
         # run all epochs
         for epoch_nb in range(self.current_epoch, self.max_nb_epochs):
             # update the lr scheduler
-            for lr_scheduler in self.lr_schedulers:
-                lr_scheduler.step()
+            if self.lr_schedulers is not None:
+                for lr_scheduler in self.lr_schedulers:
+                    lr_scheduler.step()
 
             model = self.__get_model()
             model.current_epoch = epoch_nb
@@ -645,7 +734,8 @@ class Trainer(TrainerIO):
 
             # init progbar when requested
             if self.progress_bar:
-                self.prog_bar = tqdm.tqdm(range(self.total_batches), position=self.process_position)
+                self.prog_bar = tqdm.tqdm(range(self.total_batches),
+                                          position=self.process_position)
 
             for batch_nb, data_batch in enumerate(self.tng_dataloader):
                 self.batch_nb = batch_nb
@@ -654,7 +744,8 @@ class Trainer(TrainerIO):
                 model = self.__get_model()
                 model.global_step = self.global_step
 
-                # stop when the flag is changed or we've gone past the amount requested in the batches
+                # stop when the flag is changed or we've gone past the amount
+                #  requested in the batches
                 self.total_batch_nb += 1
                 met_batch_limit = batch_nb > self.nb_tng_batches
                 if met_batch_limit:
@@ -675,7 +766,7 @@ class Trainer(TrainerIO):
 
                 # when batch should be saved
                 if (batch_nb + 1) % self.log_save_interval == 0 or early_stop_epoch:
-                    if self.proc_rank == 0:
+                    if self.proc_rank == 0 and self.experiment is not None:
                         self.experiment.save()
 
                 # when metrics should be logged
@@ -701,8 +792,9 @@ class Trainer(TrainerIO):
                         model.on_tng_metrics(metrics)
 
                     # log metrics
-                    scalar_metrics = self.__metrics_to_scalars(metrics, blacklist=self.__log_vals_blacklist())
-                    if self.proc_rank == 0:
+                    scalar_metrics = self.__metrics_to_scalars(
+                        metrics, blacklist=self.__log_vals_blacklist())
+                    if self.proc_rank == 0 and self.experiment is not None:
                         self.experiment.log(scalar_metrics, global_step=self.global_step)
                         self.experiment.save()
 
@@ -723,14 +815,15 @@ class Trainer(TrainerIO):
             # early stopping
             met_min_epochs = epoch_nb > self.min_nb_epochs
             if self.enable_early_stop and met_min_epochs:
-                should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch_nb, logs=self.__tng_tqdm_dic)
+                should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch_nb,
+                                                                    logs=self.__tng_tqdm_dic)
 
                 # stop training
                 stop = should_stop and met_min_epochs
                 if stop:
                     return
 
-    def __metrics_to_scalars(self, metrics, blacklist=[]):
+    def __metrics_to_scalars(self, metrics, blacklist=set()):
         new_metrics = {}
         for k, v in metrics.items():
             if type(v) is torch.Tensor:
@@ -770,23 +863,40 @@ class Trainer(TrainerIO):
             output = self.model(data_batch, batch_nb)
         elif self.use_dp:
             output = self.model(data_batch, batch_nb)
-            output = reduce_distributed_output(output, len(self.data_parallel_device_ids))
+        elif self.single_gpu:
+            gpu_id = self.data_parallel_device_ids[0]
+            for i, x in enumerate(data_batch):
+                if isinstance(x, torch.Tensor):
+                    data_batch[i] = x.cuda(gpu_id)
+            output = self.model.training_step(data_batch, batch_nb)
+
         else:
             output = self.model.training_step(data_batch, batch_nb)
 
         try:
-            model_specific_tqdm_metrics_dic = output['tqdm_metrics']
-        except Exception as e:
+            prog_output = output['prog']
+
+            # reduce prog metrics for tqdm when using dp
+            if self.use_dp:
+                nb_gpus = len(self.data_parallel_device_ids)
+                prog_output = reduce_distributed_output(prog_output, nb_gpus)
+
+            model_specific_tqdm_metrics_dic = prog_output
+        except Exception:
             model_specific_tqdm_metrics_dic = {}
 
         # if output dict doesn't have the keyword loss
         # then assume the output=loss if scalar
         try:
             loss = output['loss']
-        except Exception as e:
+        except Exception:
             if type(output) is torch.Tensor:
                 loss = output
 
+        # when using dp need to reduce the loss
+        if self.use_dp:
+            loss = reduce_distributed_output(loss, len(self.data_parallel_device_ids))
+
         self.__add_tqdm_metrics(model_specific_tqdm_metrics_dic)
 
         # backward pass
@@ -817,7 +927,7 @@ class Trainer(TrainerIO):
             # clip gradients
             if self.gradient_clip > 0:
                 model = self.__get_model()
-                torch.nn.utils.clip_grad_norm(model.parameters(), self.gradient_clip)
+                torch.nn.utils.clip_grad_norm_(model.parameters(), self.gradient_clip)
 
             # update gradients across all optimizers
             for optimizer in self.optimizers:
@@ -831,7 +941,8 @@ class Trainer(TrainerIO):
                 # clear gradients
                 optimizer.zero_grad()
 
-            # queuing loss across batches blows it up proportionally... divide out the number accumulated
+            # queuing loss across batches blows it up proportionally...
+            #  divide out the number accumulated
             self.batch_loss_value = self.batch_loss_value / self.accumulate_grad_batches
 
             # track loss
@@ -868,12 +979,12 @@ class Trainer(TrainerIO):
         # use full val set on end of epoch
         # use a small portion otherwise
         max_batches = None if not self.fast_dev_run else 1
-        model_specific_tqdm_metrics_dic = self.validate(
+        validation_results = self.validate(
             self.model,
             self.val_dataloader,
             max_batches
         )
-        self.__add_tqdm_metrics(model_specific_tqdm_metrics_dic)
+        self.__add_tqdm_metrics(validation_results)
 
         # hook
         if self.__is_function_implemented('on_post_performance_check'):
@@ -888,4 +999,5 @@ class Trainer(TrainerIO):
         # model checkpointing
         if self.proc_rank == 0 and self.checkpoint_callback is not None:
             print('save callback...')
-            self.checkpoint_callback.on_epoch_end(epoch=self.current_epoch, logs=self.__tng_tqdm_dic)
+            self.checkpoint_callback.on_epoch_end(epoch=self.current_epoch,
+                                                  logs=self.__tng_tqdm_dic)

pytorch_lightning/pt_overrides/override_data_parallel.py

@@ -6,7 +6,6 @@ from itertools import chain
 import threading
 import torch
 from torch.cuda._utils import _get_device_index
-import pdb
 
 
 def _find_tensors(obj):  # pragma: no cover
@@ -64,7 +63,6 @@ class LightningDataParallel(DataParallel):
         outputs = self.parallel_apply(replicas, inputs, kwargs)
         return self.gather(outputs, self.output_device)
 
-
     def parallel_apply(self, replicas, inputs, kwargs):
         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
 

pytorch_lightning/root_module/grads.py

@@ -1,10 +1,9 @@
-import numpy as np
-from torch import nn
-
 """
 Module to describe gradients
 """
 
+from torch import nn
+
 
 class GradInformation(nn.Module):
 
@@ -18,12 +17,13 @@ class GradInformation(nn.Module):
                     total_norm += param_norm ** norm_type
                     norm = param_norm ** (1 / norm_type)
 
-                    results['grad_{}_norm_{}'.format(norm_type, i)] = round(norm.data.cpu().numpy().flatten()[0], 3)
-                except Exception as e:
+                    grad = round(norm.data.cpu().numpy().flatten()[0], 3)
+                    results['grad_{}_norm_{}'.format(norm_type, i)] = grad
+                except Exception:
                     # this param had no grad
                     pass
 
         total_norm = total_norm ** (1. / norm_type)
-        results['grad_{}_norm_total'.format(norm_type)] = round(total_norm.data.cpu().numpy().flatten()[0], 3)
+        grad = round(total_norm.data.cpu().numpy().flatten()[0], 3)
+        results['grad_{}_norm_total'.format(norm_type)] = grad
         return results
-

pytorch_lightning/root_module/hooks.py

@@ -1,6 +1,15 @@
 import torch
 
+
 class ModelHooks(torch.nn.Module):
+
+    def on_sanity_check_start(self):
+        """
+        Called before starting validate
+        :return:
+        """
+        pass
+
     def on_batch_start(self, data_batch):
         pass
 
@@ -42,4 +51,3 @@ class ModelHooks(torch.nn.Module):
         :return:
         """
         pass
-

pytorch_lightning/root_module/memory.py

@@ -1,14 +1,14 @@
-import torch
-import gc
-import subprocess
-import numpy as np
-import pandas as pd
-
-
 '''
 Generates a summary of a model's layers and dimensionality
 '''
 
+import gc
+
+import torch
+import subprocess
+import numpy as np
+import pandas as pd
+
 
 class ModelSummary(object):
 
@@ -94,7 +94,7 @@ class ModelSummary(object):
         mods = list(self.model.modules())
         sizes = []
 
-        for i in range(1,len(mods)):
+        for i in range(1, len(mods)):
             m = mods[i]
             p = list(m.parameters())
             modsz = []
@@ -127,7 +127,7 @@ class ModelSummary(object):
         if self.model.example_input_array is not None:
             cols.extend(['In_sizes', 'Out_sizes'])
 
-        df = pd.DataFrame(np.zeros( (len(self.layer_names), len(cols))))
+        df = pd.DataFrame(np.zeros((len(self.layer_names), len(cols))))
         df.columns = cols
 
         df['Name'] = self.layer_names
@@ -152,16 +152,16 @@ class ModelSummary(object):
         self.make_summary()
 
 
-def print_mem_stack(): # pragma: no cover
+def print_mem_stack():  # pragma: no cover
     for obj in gc.get_objects():
         try:
             if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                 print(type(obj), obj.size())
-        except Exception as e:
+        except Exception:
             pass
 
 
-def count_mem_items(): # pragma: no cover
+def count_mem_items():  # pragma: no cover
     nb_params = 0
     nb_tensors = 0
     for obj in gc.get_objects():
@@ -172,7 +172,7 @@ def count_mem_items(): # pragma: no cover
                     nb_params += 1
                 else:
                     nb_tensors += 1
-        except Exception as e:
+        except Exception:
             pass
 
     return nb_params, nb_tensors
@@ -196,6 +196,6 @@ def get_gpu_memory_map():
     gpu_memory = [int(x) for x in result.strip().split('\n')]
     gpu_memory_map = {}
     for k, v in zip(range(len(gpu_memory)), gpu_memory):
-        k = f'gpu_{k}'
+        k = 'gpu_%i' % k
         gpu_memory_map[k] = v
     return gpu_memory_map

pytorch_lightning/root_module/model_saving.py

@@ -1,8 +1,10 @@
-import torch
 import os
 import re
-import pdb
-from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel, LightningDataParallel
+
+import torch
+
+from pytorch_lightning.pt_overrides.override_data_parallel import (
+    LightningDistributedDataParallel, LightningDataParallel)
 
 
 class ModelIO(object):
@@ -44,7 +46,8 @@ class ModelIO(object):
 class TrainerIO(object):
 
     def __get_model(self):
-        is_dp_module = type(self.model) is LightningDistributedDataParallel or type(self.model) is LightningDataParallel
+        is_dp_module = isinstance(self.model, (LightningDistributedDataParallel,
+                                               LightningDataParallel))
         model = self.model.module if is_dp_module else self.model
         return model
 
@@ -57,6 +60,22 @@ class TrainerIO(object):
         # do the actual save
         torch.save(checkpoint, filepath)
 
+    def restore(self, checkpoint_path, on_gpu):
+
+        if on_gpu:
+            checkpoint = torch.load(checkpoint_path)
+        else:
+            checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
+
+        # load training state (affects trainer only)
+        self.restore_training_state(checkpoint)
+
+        # load model state
+        model = self.__get_model()
+
+        # load the state_dict on the model automatically
+        model.load_state_dict(checkpoint['state_dict'])
+
     def dump_checkpoint(self):
 
         checkpoint = {
@@ -71,12 +90,20 @@ class TrainerIO(object):
             checkpoint['early_stop_callback_wait'] = self.early_stop_callback.wait
             checkpoint['early_stop_callback_patience'] = self.early_stop_callback.patience
 
+        # save optimizers
         optimizer_states = []
         for i, optimizer in enumerate(self.optimizers):
             optimizer_states.append(optimizer.state_dict())
 
         checkpoint['optimizer_states'] = optimizer_states
 
+        # save lr schedulers
+        lr_schedulers = []
+        for i, scheduler in enumerate(self.lr_schedulers):
+            lr_schedulers.append(scheduler.state_dict())
+
+        checkpoint['lr_schedulers'] = lr_schedulers
+
         # add the state_dict from the model
         model = self.__get_model()
         checkpoint['state_dict'] = model.state_dict()
@@ -94,13 +121,16 @@ class TrainerIO(object):
             return
 
         # allow test tube to handle model check pointing automatically
-        self.cluster.set_checkpoint_save_function(
-            self.hpc_save,
-            kwargs={
-                'folderpath': self.checkpoint_callback.filepath,
-                'experiment': self.experiment
-            }
-        )
+        # only if proc 0 so we don't trigger world_size resubmits
+        if self.proc_rank == 0:
+            self.cluster.set_checkpoint_save_function(
+                self.hpc_save,
+                kwargs={
+                    'folderpath': self.checkpoint_callback.filepath,
+                    'experiment': self.experiment
+                }
+            )
+
         self.cluster.set_checkpoint_load_function(
             self.hpc_load,
             kwargs={
@@ -131,6 +161,11 @@ class TrainerIO(object):
         for optimizer, opt_state in zip(self.optimizers, optimizer_states):
             optimizer.load_state_dict(opt_state)
 
+        # restore the lr schedulers
+        lr_schedulers = checkpoint['lr_schedulers']
+        for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
+            scheduler.load_state_dict(lrs_state)
+
     # ----------------------------------
     # PRIVATE OPS
     # ----------------------------------
@@ -181,15 +216,15 @@ class TrainerIO(object):
         # call model hook
         model.on_hpc_load(checkpoint)
 
-    def max_ckpt_in_folder(self, path):
+    def max_ckpt_in_folder(self, path, name_key='ckpt_'):
         files = os.listdir(path)
-        files = [x for x in files if 'ckpt_' in x]
+        files = [x for x in files if name_key in x]
         if len(files) == 0:
             return 0
 
         ckpt_vs = []
         for name in files:
-            name = name.split('ckpt_')[-1]
+            name = name.split(name_key)[-1]
             name = re.sub('[^0-9]', '', name)
             ckpt_vs.append(int(name))
 

pytorch_lightning/root_module/root_module.py

@@ -1,4 +1,5 @@
 import torch
+
 from pytorch_lightning.root_module.memory import ModelSummary
 from pytorch_lightning.root_module.grads import GradInformation
 from pytorch_lightning.root_module.model_saving import ModelIO, load_hparams_from_tags_csv
@@ -58,7 +59,7 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
 
     def configure_optimizers(self):
         """
-        Return array of optimizers
+        Return a list of optimizers and a list of schedulers (could be empty)
         :return:
         """
         raise NotImplementedError
@@ -128,6 +129,3 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
     def unfreeze(self):
         for param in self.parameters():
             param.requires_grad = True
-
-
-

pytorch_lightning/testing/lm_test_module.py

@@ -1,17 +1,18 @@
 import os
 from collections import OrderedDict
-import torch.nn as nn
-from torchvision.datasets import MNIST
-import torchvision.transforms as transforms
+
 import torch
+import torch.nn as nn
 import torch.nn.functional as F
-from test_tube import HyperOptArgumentParser
 from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
+from torchvision.datasets import MNIST
+from torchvision import transforms
+from test_tube import HyperOptArgumentParser
 
 from pytorch_lightning.root_module.root_module import LightningModule
-import pytorch_lightning as ptl
+from pytorch_lightning import data_loader
 
 
 class LightningTestModel(LightningModule):
@@ -47,11 +48,13 @@ class LightningTestModel(LightningModule):
         Layout model
         :return:
         """
-        self.c_d1 = nn.Linear(in_features=self.hparams.in_features, out_features=self.hparams.hidden_dim)
+        self.c_d1 = nn.Linear(in_features=self.hparams.in_features,
+                              out_features=self.hparams.hidden_dim)
         self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
         self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)
 
-        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim, out_features=self.hparams.out_features)
+        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim,
+                              out_features=self.hparams.out_features)
 
     # ---------------------
     # TRAINING
@@ -96,12 +99,15 @@ class LightningTestModel(LightningModule):
         if self.trainer.use_dp:
             loss_val = loss_val.unsqueeze(0)
 
-        output = OrderedDict({
-            'loss': loss_val
-        })
-
-        # can also return just a scalar instead of a dict (return loss_val)
-        return output
+        # alternate possible outputs to test
+        if self.trainer.batch_nb % 1 == 0:
+            output = OrderedDict({
+                'loss': loss_val,
+                'prog': {'some_val': loss_val * loss_val}
+            })
+            return output
+        if self.trainer.batch_nb % 2 == 0:
+            return loss_val
 
     def validation_step(self, data_batch, batch_i):
         """
@@ -129,16 +135,16 @@ class LightningTestModel(LightningModule):
             val_acc = val_acc.unsqueeze(0)
 
         # alternate possible outputs to test
-        if self.trainer.batch_nb % 1 == 0:
+        if batch_i % 1 == 0:
             output = OrderedDict({
                 'val_loss': loss_val,
                 'val_acc': val_acc,
             })
             return output
-        if self.trainer.batch_nb % 2 == 0:
+        if batch_i % 2 == 0:
             return val_acc
 
-        if self.trainer.batch_nb % 3 == 0:
+        if batch_i % 3 == 0:
             output = OrderedDict({
                 'val_loss': loss_val,
                 'val_acc': val_acc,
@@ -179,13 +185,18 @@ class LightningTestModel(LightningModule):
         return whatever optimizers we want here
         :return: list of optimizers
         """
+        # try no scheduler for this model (testing purposes)
         optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+
+        # test returning only 1 list instead of 2
         return [optimizer]
 
     def __dataloader(self, train):
         # init data generators
-        transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
-        dataset = MNIST(root=self.hparams.data_root, train=train, transform=transform, download=True)
+        transform = transforms.Compose([transforms.ToTensor(),
+                                        transforms.Normalize((0.5,), (1.0,))])
+        dataset = MNIST(root=self.hparams.data_root, train=train,
+                        transform=transform, download=True)
 
         # when using multi-node we need to add the datasampler
         train_sampler = None
@@ -195,7 +206,7 @@ class LightningTestModel(LightningModule):
             if self.on_gpu and not self.force_remove_distributed_sampler:
                 train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                 batch_size = batch_size // self.trainer.world_size  # scale batch size
-        except Exception as e:
+        except Exception:
             pass
 
         should_shuffle = train_sampler is None
@@ -208,20 +219,20 @@ class LightningTestModel(LightningModule):
 
         return loader
 
-    @ptl.data_loader
+    @data_loader
     def tng_dataloader(self):
         return self.__dataloader(train=True)
 
-    @ptl.data_loader
+    @data_loader
     def val_dataloader(self):
         return self.__dataloader(train=False)
 
-    @ptl.data_loader
+    @data_loader
     def test_dataloader(self):
         return self.__dataloader(train=False)
 
     @staticmethod
-    def add_model_specific_args(parent_parser, root_dir):
+    def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
         """
         Parameters you define here will be available to your model through self.hparams
         :param parent_parser:
@@ -235,19 +246,24 @@ class LightningTestModel(LightningModule):
 
         # network params
         parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
-        parser.add_argument('--in_features', default=28*28, type=int)
+        parser.add_argument('--in_features', default=28 * 28, type=int)
         parser.add_argument('--out_features', default=10, type=int)
-        parser.add_argument('--hidden_dim', default=50000, type=int) # use 500 for CPU, 50000 for GPU to see speed difference
+        # use 500 for CPU, 50000 for GPU to see speed difference
+        parser.add_argument('--hidden_dim', default=50000, type=int)
 
         # data
         parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
 
         # training params (opt)
-        parser.opt_list('--learning_rate', default=0.001*8, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
+        parser.opt_list('--learning_rate', default=0.001 * 8, type=float,
+                        options=[0.0001, 0.0005, 0.001, 0.005],
                         tunable=False)
-        parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)
+        parser.opt_list('--optimizer_name', default='adam', type=str,
+                        options=['adam'], tunable=False)
 
-        # if using 2 nodes with 4 gpus each the batch size here (256) will be 256 / (2*8) = 16 per gpu
-        parser.opt_list('--batch_size', default=256*8, type=int, options=[32, 64, 128, 256], tunable=False,
-                        help='batch size will be divided over all the gpus being used across all nodes')
+        # if using 2 nodes with 4 gpus each the batch size here
+        #  (256) will be 256 / (2*8) = 16 per gpu
+        parser.opt_list('--batch_size', default=256 * 8, type=int,
+                        options=[32, 64, 128, 256], tunable=False,
+                        help='batch size will be divided over all gpus being used across all nodes')
         return parser

pytorch_lightning/trainer_main.py

@@ -1,210 +0,0 @@
-import os
-import sys
-
-import torch
-import numpy as np
-from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
-from pytorch_lightning.models.trainer import Trainer
-from pytorch_lightning.utils.arg_parse import add_default_args
-from time import sleep
-
-from pytorch_lightning.callbacks.pt_callbacks import EarlyStopping, ModelCheckpoint
-SEED = 2334
-torch.manual_seed(SEED)
-np.random.seed(SEED)
-
-# ---------------------
-# DEFINE MODEL HERE
-# ---------------------
-from pytorch_lightning.models.sample_model_template.model_template import ExampleModel1
-# ---------------------
-
-AVAILABLE_MODELS = {
-    'model_1': ExampleModel1
-}
-
-
-"""
-Allows training by using command line arguments
-
-Run by: 
-# TYPE YOUR RUN COMMAND HERE
-"""
-
-
-def main_local(hparams):
-    main(hparams, None, None)
-
-
-def main(hparams, cluster, results_dict):
-    """
-    Main training routine specific for this project
-    :param hparams:
-    :return:
-    """
-    on_gpu = torch.cuda.is_available()
-    if hparams.disable_cuda:
-        on_gpu = False
-
-    device = 'cuda' if on_gpu else 'cpu'
-    hparams.__setattr__('device', device)
-    hparams.__setattr__('on_gpu', on_gpu)
-    hparams.__setattr__('nb_gpus', torch.cuda.device_count())
-    hparams.__setattr__('inference_mode', hparams.model_load_weights_path is not None)
-
-    # init experiment
-    exp = Experiment(
-        name=hparams.tt_name,
-        debug=hparams.debug,
-        save_dir=hparams.tt_save_path,
-        version=hparams.hpc_exp_number,
-        autosave=False,
-        description=hparams.tt_description
-    )
-
-    exp.argparse(hparams)
-    exp.save()
-
-    # build model
-    print('loading model...')
-    model = TRAINING_MODEL(hparams)
-    print('model built')
-
-    # callbacks
-    early_stop = EarlyStopping(
-        monitor=hparams.early_stop_metric,
-        patience=hparams.early_stop_patience,
-        verbose=True,
-        mode=hparams.early_stop_mode
-    )
-
-    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
-    checkpoint = ModelCheckpoint(
-        filepath=model_save_path,
-        save_function=None,
-        save_best_only=True,
-        verbose=True,
-        monitor=hparams.model_save_monitor_value,
-        mode=hparams.model_save_monitor_mode
-    )
-
-    # configure trainer
-    trainer = Trainer(
-        experiment=exp,
-        on_gpu=on_gpu,
-        cluster=cluster,
-        progress_bar=hparams.enable_tqdm,
-        overfit_pct=hparams.overfit,
-        track_grad_norm=hparams.track_grad_norm,
-        fast_dev_run=hparams.fast_dev_run,
-        check_val_every_n_epoch=hparams.check_val_every_n_epoch,
-        accumulate_grad_batches=hparams.accumulate_grad_batches,
-        process_position=process_position,
-        current_gpu_name=current_gpu,
-        checkpoint_callback=checkpoint,
-        early_stop_callback=early_stop,
-        enable_early_stop=hparams.enable_early_stop,
-        max_nb_epochs=hparams.max_nb_epochs,
-        min_nb_epochs=hparams.min_nb_epochs,
-        train_percent_check=hparams.train_percent_check,
-        val_percent_check=hparams.val_percent_check,
-        test_percent_check=hparams.test_percent_check,
-        val_check_interval=hparams.val_check_interval,
-        log_save_interval=hparams.log_save_interval,
-        add_log_row_interval=hparams.add_log_row_interval,
-        lr_scheduler_milestones=hparams.lr_scheduler_milestones
-    )
-
-    # train model
-    trainer.fit(model)
-
-
-def get_default_parser(strategy, root_dir):
-
-    possible_model_names = list(AVAILABLE_MODELS.keys())
-    parser = HyperOptArgumentParser(strategy=strategy, add_help=False)
-    add_default_args(parser, root_dir, possible_model_names, SEED)
-    return parser
-
-
-def get_model_name(args):
-    for i, arg in enumerate(args):
-        if 'model_name' in arg:
-            return args[i+1]
-
-
-def optimize_on_cluster(hyperparams):
-    # enable cluster training
-    cluster = SlurmCluster(
-        hyperparam_optimizer=hyperparams,
-        log_path=hyperparams.tt_save_path,
-        test_tube_exp_name=hyperparams.tt_name
-    )
-
-    # email for cluster coms
-    cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
-
-    # configure cluster
-    cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
-    cluster.job_time = '48:00:00'
-    cluster.gpu_type = '1080ti'
-    cluster.memory_mb_per_node = 48000
-
-    # any modules for code to run in env
-    cluster.add_command('source activate pytorch_lightning')
-
-    # name of exp
-    job_display_name = hyperparams.tt_name.split('_')[0]
-    job_display_name = job_display_name[0:3]
-
-    # run hopt
-    print('submitting jobs...')
-    cluster.optimize_parallel_cluster_gpu(
-        main,
-        nb_trials=hyperparams.nb_hopt_trials,
-        job_name=job_display_name
-    )
-
-
-if __name__ == '__main__':
-
-    model_name = get_model_name(sys.argv)
-
-    # use default args
-    root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
-    parent_parser = get_default_parser(strategy='random_search', root_dir=root_dir)
-
-    # allow model to overwrite or extend args
-    TRAINING_MODEL = AVAILABLE_MODELS[model_name]
-    parser = TRAINING_MODEL.add_model_specific_args(parent_parser)
-    parser.json_config('-c', '--config', default=root_dir + '/run_configs/local.json')
-    hyperparams = parser.parse_args()
-
-    # format GPU layout
-    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-    gpu_ids = hyperparams.gpus.split(';')
-
-    # RUN TRAINING
-    if hyperparams.on_cluster:
-        print('RUNNING ON SLURM CLUSTER')
-        os.environ["CUDA_VISIBLE_DEVICES"] = ','.join(gpu_ids)
-        optimize_on_cluster(hyperparams)
-
-    elif hyperparams.single_run_gpu:
-        print(f'RUNNING 1 TRIAL ON GPU. gpu: {gpu_ids[0]}')
-        os.environ["CUDA_VISIBLE_DEVICES"] = gpu_ids[0]
-        main(hyperparams, None, None)
-
-    elif hyperparams.local or hyperparams.single_run:
-        os.environ["CUDA_VISIBLE_DEVICES"] = '0'
-        print('RUNNING LOCALLY')
-        main(hyperparams, None, None)
-
-    else:
-        print(f'RUNNING MULTI GPU. GPU ids: {gpu_ids}')
-        hyperparams.optimize_parallel_gpu(
-            main_local,
-            gpu_ids=gpu_ids,
-            nb_trials=hyperparams.nb_hopt_trials,
-            nb_workers=len(gpu_ids)
-        )

pytorch_lightning/utilities/arg_parse.py

@@ -1,31 +1,48 @@
-import pdb
+"""
+List of default args which mught be useful for all the available flags
+Might need to update with the new flags
+"""
+
+import os
+
 
 def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None):
 
     # tng, test, val check intervals
-    parser.add_argument('--eval_test_set', dest='eval_test_set', action='store_true', help='true = run test set also')
-    parser.add_argument('--check_val_every_n_epoch', default=1, type=int, help='check val every n epochs')
+    parser.add_argument('--eval_test_set', dest='eval_test_set', action='store_true',
+                        help='true = run test set also')
+    parser.add_argument('--check_val_every_n_epoch', default=1, type=int,
+                        help='check val every n epochs')
     parser.opt_list('--accumulate_grad_batches', default=1, type=int, tunable=False,
-                    help='accumulates gradients k times before applying update. Simulates huge batch size')
+                    help='accumulates gradients k times before applying update.'
+                         ' Simulates huge batch size')
     parser.add_argument('--max_nb_epochs', default=200, type=int, help='cap epochs')
     parser.add_argument('--min_nb_epochs', default=2, type=int, help='min epochs')
-    parser.add_argument('--train_percent_check', default=1.0, type=float, help='how much of tng set to check')
-    parser.add_argument('--val_percent_check', default=1.0, type=float, help='how much of val set to check')
-    parser.add_argument('--test_percent_check', default=1.0, type=float, help='how much of test set to check')
+    parser.add_argument('--train_percent_check', default=1.0, type=float,
+                        help='how much of tng set to check')
+    parser.add_argument('--val_percent_check', default=1.0, type=float,
+                        help='how much of val set to check')
+    parser.add_argument('--test_percent_check', default=1.0, type=float,
+                        help='how much of test set to check')
 
-    parser.add_argument('--val_check_interval', default=0.95, type=float, help='how much within 1 epoch to check val')
-    parser.add_argument('--log_save_interval', default=100, type=int, help='how many batches between log saves')
-    parser.add_argument('--add_log_row_interval', default=100, type=int, help='add log every k batches')
+    parser.add_argument('--val_check_interval', default=0.95, type=float,
+                        help='how much within 1 epoch to check val')
+    parser.add_argument('--log_save_interval', default=100, type=int,
+                        help='how many batches between log saves')
+    parser.add_argument('--add_log_row_interval', default=100, type=int,
+                        help='add log every k batches')
 
     # early stopping
     parser.add_argument('--disable_early_stop', dest='enable_early_stop', action='store_false')
     parser.add_argument('--early_stop_metric', default='val_acc', type=str)
     parser.add_argument('--early_stop_mode', default='min', type=str)
-    parser.add_argument('--early_stop_patience', default=3, type=int, help='number of epochs until stop')
+    parser.add_argument('--early_stop_patience', default=3, type=int,
+                        help='number of epochs until stop')
 
     # gradient handling
     parser.add_argument('--gradient_clip', default=-1, type=int)
-    parser.add_argument('--track_grad_norm', default=-1, type=int, help='if > 0, will track this grad norm')
+    parser.add_argument('--track_grad_norm', default=-1, type=int,
+                        help='if > 0, will track this grad norm')
 
     # model saving
     parser.add_argument('--model_save_path', default=root_dir + '/model_weights')
@@ -41,7 +58,8 @@ def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None
     # test_tube settings
     parser.add_argument('-en', '--tt_name', default='pt_test')
     parser.add_argument('-td', '--tt_description', default='pytorch lightning test')
-    parser.add_argument('--tt_save_path', default=root_dir + '/test_tube_logs', help='logging dir')
+    parser.add_argument('--tt_save_path', default=os.path.join(root_dir, 'test_tube_logs'),
+                        help='logging dir')
     parser.add_argument('--enable_single_run', dest='single_run', action='store_true')
     parser.add_argument('--nb_hopt_trials', default=1, type=int)
     parser.add_argument('--log_stdout', dest='log_stdout', action='store_true')
@@ -52,25 +70,30 @@ def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None
     parser.add_argument('--default_tensor_type', default='torch.cuda.FloatTensor', type=str)
     parser.add_argument('--use_amp', dest='use_amp', action='store_true')
     parser.add_argument('--check_grad_nans', dest='check_grad_nans', action='store_true')
-    parser.add_argument('--amp_level', default='O2',type=str)
-
+    parser.add_argument('--amp_level', default='O2', type=str)
 
     # run on hpc
     parser.add_argument('--on_cluster', dest='on_cluster', action='store_true')
 
     # FAST training
     # use these settings to make sure network has no bugs without running a full dataset
-    parser.add_argument('--fast_dev_run', dest='fast_dev_run', default=False, action='store_true', help='runs validation after 1 tng step')
-    parser.add_argument('--enable_tqdm', dest='enable_tqdm', default=False, action='store_true', help='false removes the prog bar')
-    parser.add_argument('--overfit', default=-1, type=float, help='% of dataset to use with this option. float, or -1 for none')
+    parser.add_argument('--fast_dev_run', dest='fast_dev_run', default=False, action='store_true',
+                        help='runs validation after 1 tng step')
+    parser.add_argument('--enable_tqdm', dest='enable_tqdm', default=False, action='store_true',
+                        help='false removes the prog bar')
+    parser.add_argument('--overfit', default=-1, type=float,
+                        help='% of dataset to use with this option. float, or -1 for none')
 
     # debug args
     if rand_seed is not None:
         parser.add_argument('--random_seed', default=rand_seed, type=int)
 
-    parser.add_argument('--interactive', dest='interactive', action='store_true', help='runs on gpu without cluster')
-    parser.add_argument('--debug', dest='debug', action='store_true', help='enables/disables test tube')
-    parser.add_argument('--local', dest='local', action='store_true', help='enables local tng')
+    parser.add_argument('--interactive', dest='interactive', action='store_true',
+                        help='runs on gpu without cluster')
+    parser.add_argument('--debug', dest='debug', action='store_true',
+                        help='enables/disables test tube')
+    parser.add_argument('--local', dest='local', action='store_true',
+                        help='enables local tng')
 
     # optimizer
-    parser.add_argument('--lr_scheduler_milestones', default=None, type=str)
+    parser.add_argument('--lr_scheduler_milestones', default=None, type=str)

pytorch_lightning/utilities/debugging.py

@@ -1,5 +1,2 @@
-import pdb
-import sys
-
 class MisconfigurationException(Exception):
-    pass
+    pass

requirements.txt

@@ -1,9 +1,7 @@
-coverage==4.5.3
-mkdocs==1.0.4
-pytest==5.0.1
 scikit-learn==0.20.2
 tqdm==4.32.1
 twine==1.13.0
 numpy==1.16.4
 torch>=1.1.0
-torchvision==0.3.0
+torchvision>=0.3.0
+pandas

setup.cfg

@@ -31,6 +31,8 @@ exclude_lines =
     print(traceback.print_exc())
     return *
     raise Exception
+    raise *
+    except *
     warnings
     print
     raise RuntimeError
@@ -42,6 +44,7 @@ omit =
     pytorch_lightning/callbacks/pt_callbacks.py
     tests/test_models.py
     pytorch_lightning/testing_models/lm_test_module.py
+    pytorch_lightning/utilities/arg_parse.py
 
 [flake8]
 ignore = E731,W504,F401,F841

setup.py

@@ -1,29 +1,58 @@
 #!/usr/bin/env python
 
+# Always prefer setuptools over distutils
 from setuptools import setup, find_packages
 
 # https://packaging.python.org/guides/single-sourcing-package-version/
 
 # http://blog.ionelmc.ro/2014/05/25/python-packaging/
+
+# https://packaging.python.org/discussions/install-requires-vs-requirements /
+# keep the meta-data here for simplicity in reading this file... it's not obvious
+# what happens and to non-engineers they won't know to look in init...
+# the goal of the project is simplicity for researchers, don't want to add too much
+# engineer specific practices
 setup(
-    name="pytorch-lightning",
-    version='0.3.6.4',
-    description="The Keras for ML researchers using PyTorch",
-    author="William Falcon",
-    author_email="waf2107@columbia.edu",
-    url="https://github.com/williamFalcon/pytorch-lightning",
-    download_url="https://github.com/williamFalcon/pytorch-lightning",
-    license="MIT",
-    keywords=["deep learning", "pytorch", "AI"],
-    python_requires=">=3.5",
-    install_requires=[
-        "torch>=1.1.0",
-        "tqdm",
-        "test-tube>=0.6.7.4",
-    ],
+    name='pytorch-lightning',
+    version='0.4.0',
+    description='The Keras for ML researchers using PyTorch',
+    author='William Falcon',
+    author_email='waf2107@columbia.edu',
+    url='https://github.com/williamFalcon/pytorch-lightning',
+    download_url='https://github.com/williamFalcon/pytorch-lightning',
+    license='Apache-2',
     packages=find_packages(),
-    long_description=open("README.md", encoding="utf-8").read(),
+    long_description=open('README.md', encoding='utf-8').read(),
     long_description_content_type='text/markdown',
     include_package_data=True,
     zip_safe=False,
+    keywords=['deep learning', 'pytorch', 'AI'],
+    python_requires='>=3.6',
+    install_requires=[
+        'torch==1.1.0',
+        'tqdm',
+        'test-tube>=0.6.7.6',
+        'pandas>=0.20.3',
+    ],
+    classifiers=[
+        'Environment :: Console',
+        'Natural Language :: English',
+        # How mature is this project? Common values are
+        #   3 - Alpha, 4 - Beta, 5 - Production/Stable
+        'Development Status :: 4 - Beta',
+        # Indicate who your project is intended for
+        'Intended Audience :: Developers',
+        'Topic :: Scientific/Engineering :: Artificial Intelligence',
+        'Topic :: Scientific/Engineering :: Image Recognition',
+        'Topic :: Scientific/Engineering :: Information Analysis',
+        # Pick your license as you wish
+        'License :: OSI Approved :: BSD License',
+        'Operating System :: OS Independent',
+        # Specify the Python versions you support here. In particular, ensure
+        # that you indicate whether you support Python 2, Python 3 or both.
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7',
+    ],
 )

tests/README.md

@@ -1,4 +1,4 @@
-# Pytorch-Lightning Tests
+# PyTorch-Lightning Tests
 
 ## Running tests
 The automatic travis tests ONLY run CPU-based tests. Although these cover most of the use cases,
@@ -17,7 +17,7 @@ pip install -e .
 pip install -r requirements.txt
 
 # run tests
-py.test
+py.test -v
 ```
 
 To test models that require GPU make sure to run the above command on a GPU machine.
@@ -43,16 +43,21 @@ For each set up it also tests:
 5. simulated load from HPC signal.
 
 ## Running Coverage   
+Make sure to run coverage on a GPU machine with at least 2 GPUs and NVIDIA apex installed. 
 
 ```bash
 cd pytorch-lightning
 
 # generate coverage 
 pip install coverage
-coverage run tests/test_models.py   
+coverage run --source pytorch_lightning -m py.test pytorch_lightning tests examples -v --doctest-modules
 
 # print coverage stats
-coverage report -m   
+coverage report -m
+
+# exporting resulys
+coverage xml
+codecov -t 17327163-8cca-4a5d-86c8-ca5f2ef700bc  -v
 ```
 
 

tests/debug.py

@@ -1,24 +1,19 @@
-import pytest
 from pytorch_lightning import Trainer
-from pytorch_lightning.examples.new_project_templates.lightning_module_template import LightningTemplateModel
+from examples import LightningTemplateModel
 from argparse import Namespace
 from test_tube import Experiment
 from pytorch_lightning.callbacks import ModelCheckpoint
-import numpy as np
-import warnings
-import torch
 import os
 import shutil
-import pdb
 
-import pytorch_lightning as ptl
+import pytorch_lightning as pl
 import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 
 
-class CoolModel(ptl.LightningModule):
+class CoolModel(pl.LightningModule):
 
     def __init(self):
         super(CoolModel, self).__init__()
@@ -48,15 +43,15 @@ class CoolModel(ptl.LightningModule):
     def configure_optimizers(self):
         return [torch.optim.Adam(self.parameters(), lr=0.02)]
 
-    @ptl.data_loader
+    @pl.data_loader
     def tng_dataloader(self):
         return DataLoader(MNIST('path/to/save', train=True), batch_size=32)
 
-    @ptl.data_loader
+    @pl.data_loader
     def val_dataloader(self):
         return DataLoader(MNIST('path/to/save', train=False), batch_size=32)
 
-    @ptl.data_loader
+    @pl.data_loader
     def test_dataloader(self):
         return DataLoader(MNIST('path/to/save', train=False), batch_size=32)
 
@@ -66,8 +61,8 @@ def get_model():
     root_dir = os.path.dirname(os.path.realpath(__file__))
     hparams = Namespace(**{'drop_prob': 0.2,
                            'batch_size': 32,
-                           'in_features': 28*28,
-                           'learning_rate': 0.001*8,
+                           'in_features': 28 * 28,
+                           'learning_rate': 0.001 * 8,
                            'optimizer_name': 'adam',
                            'data_root': os.path.join(root_dir, 'mnist'),
                            'out_features': 10,
@@ -112,7 +107,8 @@ def load_model(exp, save_dir):
     checkpoints = [x for x in os.listdir(save_dir) if '.ckpt' in x]
     weights_dir = os.path.join(save_dir, checkpoints[0])
 
-    trained_model = LightningTemplateModel.load_from_metrics(weights_path=weights_dir, tags_csv=tags_path, on_gpu=True)
+    trained_model = LightningTemplateModel.load_from_metrics(weights_path=weights_dir,
+                                                             tags_csv=tags_path, on_gpu=True)
 
     assert trained_model is not None, 'loading model failed'
 
@@ -137,7 +133,7 @@ def run_prediction(dataloader, trained_model):
 
     print(val_acc)
 
-    assert val_acc > 0.70, f'this model is expected to get > 0.7 in test set (it got {val_acc})'
+    assert val_acc > 0.70, 'this model is expected to get > 0.7 in test set (it got %f)' % val_acc
 
 
 def main():

tests/requirements.txt

@@ -0,0 +1,8 @@
+tox
+coverage
+codecov
+pytest>=3.0.5
+pytest-cov
+flake8
+check-manifest
+test_tube

tests/test_models.py

@@ -1,20 +1,22 @@
-import pytest
-from pytorch_lightning import Trainer
-from pytorch_lightning.examples.new_project_templates.lightning_module_template import LightningTemplateModel
-from pytorch_lightning.testing_models.lm_test_module import LightningTestModel
+import os
+import shutil
+import warnings
 from argparse import Namespace
+
+import pytest
+import numpy as np
+import torch
 from test_tube import Experiment, SlurmCluster
+
+# sys.path += [os.path.abspath('..'), os.path.abspath('../..')]
+from pytorch_lightning import Trainer
+from pytorch_lightning.testing.lm_test_module import LightningTestModel
 from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
-from pytorch_lightning.utils.debugging import MisconfigurationException
+from pytorch_lightning.utilities.debugging import MisconfigurationException
 from pytorch_lightning.root_module import memory
 from pytorch_lightning.models.trainer import reduce_distributed_output
 from pytorch_lightning.root_module import model_saving
-import numpy as np
-import warnings
-import torch
-import os
-import shutil
-import pdb
+from examples import LightningTemplateModel
 
 SEED = 2334
 torch.manual_seed(SEED)
@@ -24,6 +26,156 @@ np.random.seed(SEED)
 # ------------------------------------------------------------------------
 # TESTS
 # ------------------------------------------------------------------------
+def test_simple_cpu():
+    """
+    Verify continue training session on CPU
+    :return:
+    """
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    trainer_options = dict(
+        max_nb_epochs=1,
+        val_percent_check=0.1,
+        train_percent_check=0.1,
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # traning complete
+    assert result == 1, 'amp + ddp model failed to complete'
+
+    clear_save_dir()
+
+
+def test_amp_single_gpu():
+    """
+    Make sure DDP + AMP work
+    :return:
+    """
+    if not torch.cuda.is_available():
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      'Rerun on a GPU node to run this test')
+        return
+    if not torch.cuda.device_count() > 1:
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      'Rerun on a node with 2+ GPUs to run this test')
+        return
+
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    trainer_options = dict(
+        progress_bar=True,
+        max_nb_epochs=1,
+        gpus=[0],
+        distributed_backend='dp',
+        use_amp=True
+    )
+
+    run_gpu_model_test(trainer_options, model, hparams)
+
+
+def test_cpu_restore_training():
+    """
+    Verify continue training session on CPU
+    :return:
+    """
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    save_dir = init_save_dir()
+
+    # exp file to get meta
+    test_exp_version = 10
+    exp = get_exp(False, version=test_exp_version)
+    exp.argparse(hparams)
+    exp.save()
+
+    trainer_options = dict(
+        max_nb_epochs=2,
+        val_check_interval=0.50,
+        val_percent_check=0.2,
+        train_percent_check=0.2,
+        experiment=exp,
+        checkpoint_callback=ModelCheckpoint(save_dir)
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+    real_global_epoch = trainer.current_epoch
+
+    # traning complete
+    assert result == 1, 'amp + ddp model failed to complete'
+
+    # wipe-out trainer and model
+    # retrain with not much data... this simulates picking training back up after slurm
+    # we want to see if the weights come back correctly
+    new_exp = get_exp(False, version=test_exp_version)
+    trainer_options = dict(
+        max_nb_epochs=2,
+        val_check_interval=0.50,
+        val_percent_check=0.2,
+        train_percent_check=0.2,
+        experiment=new_exp,
+        checkpoint_callback=ModelCheckpoint(save_dir),
+    )
+    trainer = Trainer(**trainer_options)
+    model = LightningTestModel(hparams)
+
+    # set the epoch start hook so we can predict before the model does the full training
+    def assert_good_acc():
+        assert trainer.current_epoch == real_global_epoch and trainer.current_epoch > 0
+
+        # if model and state loaded correctly, predictions will be good even though we
+        # haven't trained with the new loaded model
+        trainer.model.eval()
+        run_prediction(trainer.val_dataloader, trainer.model)
+
+    model.on_sanity_check_start = assert_good_acc
+
+    # by calling fit again, we trigger training, loading weights from the cluster
+    # and our hook to predict using current model before any more weight updates
+    trainer.fit(model)
+
+    clear_save_dir()
+
+
+def test_amp_gpu_ddp():
+    """
+    Make sure DDP + AMP work
+    :return:
+    """
+    if not torch.cuda.is_available():
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      'Rerun on a GPU node to run this test')
+        return
+    if not torch.cuda.device_count() > 1:
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      'Rerun on a node with 2+ GPUs to run this test')
+        return
+
+    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
+
+    hparams = get_hparams()
+    model = LightningTestModel(hparams)
+
+    trainer_options = dict(
+        progress_bar=True,
+        max_nb_epochs=1,
+        gpus=[0, 1],
+        distributed_backend='ddp',
+        use_amp=True
+    )
+
+    run_gpu_model_test(trainer_options, model, hparams)
+
 
 def test_cpu_slurm_save_load():
     """
@@ -78,7 +230,8 @@ def test_cpu_slurm_save_load():
     # wipe-out trainer and model
     # retrain with not much data... this simulates picking training back up after slurm
     # we want to see if the weights come back correctly
-    continue_tng_hparams = get_hparams(continue_training=True, hpc_exp_number=cluster_a.hpc_exp_number)
+    continue_tng_hparams = get_hparams(continue_training=True,
+                                       hpc_exp_number=cluster_a.hpc_exp_number)
     trainer_options = dict(
         max_nb_epochs=1,
         cluster=SlurmCluster(continue_tng_hparams),
@@ -109,11 +262,9 @@ def test_cpu_slurm_save_load():
 def test_loading_meta_tags():
     hparams = get_hparams()
 
-    save_dir = init_save_dir()
-
     # save tags
     exp = get_exp(False)
-    exp.tag({'some_str':'a_str', 'an_int': 1, 'a_float': 2.0})
+    exp.tag({'some_str': 'a_str', 'an_int': 1, 'a_float': 2.0})
     exp.argparse(hparams)
     exp.save()
 
@@ -194,7 +345,8 @@ def test_model_saving_loading():
     # load new model
     tags_path = exp.get_data_path(exp.name, exp.version)
     tags_path = os.path.join(tags_path, 'meta_tags.csv')
-    model_2 = LightningTestModel.load_from_metrics(weights_path=new_weights_path, tags_csv=tags_path, on_gpu=False)
+    model_2 = LightningTestModel.load_from_metrics(weights_path=new_weights_path,
+                                                   tags_csv=tags_path, on_gpu=False)
     model_2.eval()
 
     # make prediction
@@ -205,8 +357,6 @@ def test_model_saving_loading():
     clear_save_dir()
 
 
-
-
 def test_model_freeze_unfreeze():
     hparams = get_hparams()
     model = LightningTestModel(hparams)
@@ -221,10 +371,12 @@ def test_amp_gpu_ddp_slurm_managed():
     :return:
     """
     if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a GPU node to run this test')
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      ' Rerun on a GPU node to run this test')
         return
     if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a node with 2+ GPUs to run this test')
+        warnings.warn('test_amp_gpu_ddp cannot run.'
+                      ' Rerun on a node with 2+ GPUs to run this test')
         return
 
     # simulate setting slurm flags
@@ -280,7 +432,7 @@ def test_amp_gpu_ddp_slurm_managed():
     if trainer.use_ddp:
         # on hpc this would work fine... but need to hack it for the purpose of the test
         trainer.model = pretrained_model
-        trainer.optimizers = pretrained_model.configure_optimizers()
+        trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers()
 
     # test HPC loading / saving
     trainer.hpc_save(save_dir, exp)
@@ -388,7 +540,8 @@ def test_single_gpu_model():
     :return:
     """
     if not torch.cuda.is_available():
-        warnings.warn('test_single_gpu_model cannot run. Rerun on a GPU node to run this test')
+        warnings.warn('test_single_gpu_model cannot run.'
+                      ' Rerun on a GPU node to run this test')
         return
     model, hparams = get_model()
 
@@ -409,10 +562,12 @@ def test_multi_gpu_model_dp():
     :return:
     """
     if not torch.cuda.is_available():
-        warnings.warn('test_multi_gpu_model_dp cannot run. Rerun on a GPU node to run this test')
+        warnings.warn('test_multi_gpu_model_dp cannot run.'
+                      ' Rerun on a GPU node to run this test')
         return
     if not torch.cuda.device_count() > 1:
-        warnings.warn('test_multi_gpu_model_dp cannot run. Rerun on a node with 2+ GPUs to run this test')
+        warnings.warn('test_multi_gpu_model_dp cannot run.'
+                      ' Rerun on a node with 2+ GPUs to run this test')
         return
     model, hparams = get_model()
     trainer_options = dict(
@@ -435,10 +590,12 @@ def test_amp_gpu_dp():
     :return:
     """
     if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_dp cannot run. Rerun on a GPU node to run this test')
+        warnings.warn('test_amp_gpu_dp cannot run.'
+                      ' Rerun on a GPU node to run this test')
         return
     if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_dp cannot run. Rerun on a node with 2+ GPUs to run this test')
+        warnings.warn('test_amp_gpu_dp cannot run.'
+                      ' Rerun on a node with 2+ GPUs to run this test')
         return
     model, hparams = get_model()
     trainer_options = dict(
@@ -457,10 +614,12 @@ def test_multi_gpu_model_ddp():
     :return:
     """
     if not torch.cuda.is_available():
-        warnings.warn('test_multi_gpu_model_ddp cannot run. Rerun on a GPU node to run this test')
+        warnings.warn('test_multi_gpu_model_ddp cannot run.'
+                      ' Rerun on a GPU node to run this test')
         return
     if not torch.cuda.device_count() > 1:
-        warnings.warn('test_multi_gpu_model_ddp cannot run. Rerun on a node with 2+ GPUs to run this test')
+        warnings.warn('test_multi_gpu_model_ddp cannot run.'
+                      ' Rerun on a node with 2+ GPUs to run this test')
         return
 
     os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
@@ -477,34 +636,6 @@ def test_multi_gpu_model_ddp():
     run_gpu_model_test(trainer_options, model, hparams)
 
 
-def test_amp_gpu_ddp():
-    """
-    Make sure DDP + AMP work
-    :return:
-    """
-    if not torch.cuda.is_available():
-        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a GPU node to run this test')
-        return
-    if not torch.cuda.device_count() > 1:
-        warnings.warn('test_amp_gpu_ddp cannot run. Rerun on a node with 2+ GPUs to run this test')
-        return
-
-    os.environ['MASTER_PORT'] = str(np.random.randint(12000, 19000, 1)[0])
-
-    hparams = get_hparams()
-    model = LightningTestModel(hparams)
-
-    trainer_options = dict(
-        progress_bar=True,
-        max_nb_epochs=1,
-        gpus=[0, 1],
-        distributed_backend='ddp',
-        use_amp=True
-    )
-
-    run_gpu_model_test(trainer_options, model, hparams)
-
-
 def test_ddp_sampler_error():
     """
     Make sure DDP + AMP work
@@ -574,7 +705,7 @@ def run_gpu_model_test(trainer_options, model, hparams, on_gpu=True):
     if trainer.use_ddp:
         # on hpc this would work fine... but need to hack it for the purpose of the test
         trainer.model = pretrained_model
-        trainer.optimizers = pretrained_model.configure_optimizers()
+        trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers()
 
     # test HPC loading / saving
     trainer.hpc_save(save_dir, exp)
@@ -589,8 +720,8 @@ def get_hparams(continue_training=False, hpc_exp_number=0):
     args = {
         'drop_prob': 0.2,
         'batch_size': 32,
-        'in_features': 28*28,
-        'learning_rate': 0.001*8,
+        'in_features': 28 * 28,
+        'learning_rate': 0.001 * 8,
         'optimizer_name': 'adam',
         'data_root': os.path.join(root_dir, 'mnist'),
         'out_features': 10,
@@ -604,18 +735,22 @@ def get_hparams(continue_training=False, hpc_exp_number=0):
     return hparams
 
 
-def get_model():
+def get_model(use_test_model=False):
     # set up model with these hyperparams
     hparams = get_hparams()
-    model = LightningTemplateModel(hparams)
+
+    if use_test_model:
+        model = LightningTestModel(hparams)
+    else:
+        model = LightningTemplateModel(hparams)
 
     return model, hparams
 
 
-def get_exp(debug=True):
+def get_exp(debug=True, version=None):
     # set up exp object without actually saving logs
     root_dir = os.path.dirname(os.path.realpath(__file__))
-    exp = Experiment(debug=debug, save_dir=root_dir, name='tests_tt_dir')
+    exp = Experiment(debug=debug, save_dir=root_dir, name='tests_tt_dir', version=version)
     return exp
 
 
@@ -675,13 +810,13 @@ def run_prediction(dataloader, trained_model):
 
     print(val_acc)
 
-    assert val_acc > 0.50, f'this model is expected to get > 0.50 in test set (it got {val_acc})'
+    assert val_acc > 0.50, 'this model is expected to get > 0.50 in test set (it got %f)' % val_acc
 
 
 def assert_ok_acc(trainer):
     # this model should get 0.80+ acc
     acc = trainer.tng_tqdm_dic['val_acc']
-    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
+    assert acc > 0.50, 'model failed to get expected 0.50 validation accuracy. Got: %f' % acc
 
 
 if __name__ == '__main__':

tox.ini

@@ -0,0 +1,47 @@
+# this file is *not* meant to cover or endorse the use of tox or pytest or testing in general,
+#
+#  It's meant to show the use of:
+#
+#  - check-manifest
+#     confirm items checked into vcs are in your segdist
+#  - python setup.py check 
+#     confirm required package meta-data in setup.py
+#  - readme_renderer (when using a ReStructuredText README)
+#     confirms your long_description will render correctly on PyPI.
+#
+#  and also to help confirm pull requests to this project.
+
+[tox]
+envlist = py{35,36,37}
+
+[pytest]
+log_cli = 0
+log_cli_level = CRITICAL
+log_cli_format = %(message)s
+log_file = pytest.log
+log_file_level = DEBUG
+log_file_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)
+log_file_date_format=%Y-%m-%d %H:%M:%S
+
+[testenv]
+basepython =
+    py35: python3.5
+    py36: python3.6
+    py37: python3.7
+deps =
+    -r requirements.txt
+    -r ./tests/requirements.txt
+commands =
+    check-manifest --ignore tox.ini
+    python setup.py check -m -s
+    coverage run --source pytorch_lightning -m py.test pytorch_lightning tests examples -v --doctest-modules
+    flake8 .
+
+[flake8]
+exclude = .tox,*.egg,build,temp
+select = E,W,F
+doctests = True
+verbose = 2
+# https://pep8.readthedocs.io/en/latest/intro.html#error-codes
+format = pylint
+max-line-length = 100

update.sh

@@ -11,10 +11,7 @@ rm -rf ./dist/*
 python3 setup.py sdist
 twine upload dist/*
 
-
-
 # to update docs
 # cd to root dir
 # mkdocs gh-deploy
 
-