release v0.3.6.6

2026-09-12 12:40:20 +08:00 · 2019-07-28 12:12:14 -04:00
86 changed files with 2082 additions and 5817 deletions
@@ -1,46 +0,0 @@
-# see https://docs.codecov.io/docs/codecov-yaml
-# Validation check:
-# $ curl --data-binary @.codecov.yml https://codecov.io/validate
-
-codecov:
-  notify:
-    require_ci_to_pass: yes
-
-coverage:
-  precision: 0  # 2 = xx.xx%, 0 = xx%
-  round: nearest # how coverage is rounded: down/up/nearest
-  range: 40...100 # custom range of coverage colors from red -> yellow -> green
-  status:
-    # https://codecov.readme.io/v1.0/docs/commit-status
-    project:
-      default:
-        against: auto
-        target: 99% # specify the target coverage for each commit status
-        threshold: 20% # allow this little decrease on project
-        # https://github.com/codecov/support/wiki/Filtering-Branches
-        # branches: master
-        if_ci_failed: error
-    # https://github.com/codecov/support/wiki/Patch-Status
-    patch:
-      default:
-        against: auto
-        target: 40% # specify the target "X%" coverage to hit
-        # threshold: 50% # allow this much decrease on patch
-    changes: false
-
-parsers:
-  gcov:
-    branch_detection:
-      conditional: true
-      loop: true
-      macro: false
-      method: false
-  javascript:
-    enable_partials: false
-
-comment:
-  layout: header, diff
-  require_changes: false
-  behavior: default  # update if exists else create new
-  # branches: *
-
@@ -1,76 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, sex characteristics, gender identity and expression,
-level of experience, education, socio-economic status, nationality, personal
-appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or
- advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
- address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
- professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at waf2107@columbia.edu. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
-
-[homepage]: https://www.contributor-covenant.org
-
-For answers to common questions about this code of conduct, see
-https://www.contributor-covenant.org/faq
@@ -1,42 +0,0 @@
-# Contributing    
-Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!   
-
-## One less thing to remember    
-Simplify the API as much as possible from the user perspective. Any additions or improvements should minimize things the user needs to remember.   
-
-For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.   
-
-## Lightning Design Principles   
-We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
-
-#### No PyTorch interference   
-We don't want to add any abstractions on top of pure PyTorch. This gives researchers all the control they need without having to learn yet another framework.    
-
-#### Simple Internal Code    
-It's useful for users to look at the code and understand very quickly what's happening. Many users won't be engineers. Thus we need to value clear, simple code over condensed ninja moves. While that's super cool, this isn't the project for that :)      
-
-#### Force User Decisions To Best Practices    
-There are 1,000 ways to do something. However, something eventually becomes standard practice that everyone does. Thus we pick one way of doing it and force everyone to do it this way. A good example is accumulated gradients. There are many ways to implement, we just pick one and force users to use that one. A bad forced decision would be to make users use a specific library to do something.    
-
-When something becomes a best practice, we add it to the framework. This likely looks like code in utils or in the model file that everyone keeps adding over and over again across projects. When this happens, bring that code inside the trainer and add a flag for it.
-
-#### Simple External API    
-What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others. Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people.    
-
-#### Gain User Trust    
-As a researcher you can't have any part of your code going wrong. So, make thorough tests that ensure an implementation of a new trick or subbtle change is correct.    
-
-## Contribution types    
-Currently looking for help implementing new features or adding bug fixes. 
-
-A lot of good work has already been done in project mechanics (requirements.txt, setup.py, pep8, badges, ci, etc...) we're in a good state there thanks to all the early contributors (even pre-beta release)!   
-
-## Bug fixes:  
-1. Submit a github issue.   
-2. Fix it.  
-3. Submit a PR! 
-
-## New Features:  
-1. Submit a github issue.   
-2. We'll agree on the feature scope.     
-3. Submit a PR! (with updated docs and tests 🙃).   
@@ -1,36 +0,0 @@
---
-name: Bug report
-about: Create a report to help us improve
-title: ''
-labels: bug
-assignees: ''
-
---
-
-### Common bugs:
-1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/williamFalcon/pytorch-lightning/issues/79).    
-2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
-
-**Describe the bug**
-A clear and concise description of what the bug is.
-
-**To Reproduce**
-Steps to reproduce the behavior:
-1. Go to '...'
-2. Click on '....'
-3. Scroll down to '....'
-4. See error
-
-**Expected behavior**
-A clear and concise description of what you expected to happen.
-
-**Screenshots**
-If applicable, add screenshots to help explain your problem.
-
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
-
-**Additional context**
-Add any other context about the problem here.
@@ -1,20 +0,0 @@
---
-name: Feature request
-about: Suggest an idea for this project
-title: ''
-labels: enhancement, help wanted
-assignees: ''
-
---
-
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
-
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
-
-**Additional context**
-Add any other context or screenshots about the feature request here.
@@ -1,26 +0,0 @@
---
-name: How to question
-about: Asking how-to questions
-title: ''
-labels: question
-assignees: ''
-
---
-
-### Before asking:   
-1. search the issues.   
-2. search the docs.    
-
-If you still can't find what you need:     
-#### What is your question?    
-
-#### Code    
-Please paste a code snippet if your question requires it!   
-
-#### What have you tried?    
-
-#### What's your environment?   
- conda version (no venv)   
- PyTorch version   
- Lightning version   
- Test-tube version
@@ -1,17 +0,0 @@
---
-name: Typos and doc fixes
-about: Typos and doc fixes
-title: ''
-labels: typo
-assignees: ''
-
---
-
-For typos and doc fixes, please go ahead and:
-
-1. Create an issue.
-2. Fix the typo.   
-3. Submit a PR.
-
-
-Thanks!
@@ -1,16 +0,0 @@
-# Before submitting
-
- Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- Did you read the [contributor guideline](https://github.com/williamFalcon/pytorch-lightning/blob/master/.github/CONTRIBUTING.md)?
- Did you make sure to update the docs?   
- Did you write any new necessary tests?  
-
-## What does this PR do?
-Fixes # (issue).
-
-## PR review    
-Anyone in the community is free to review the PR once the tests have passed.     
-If we didn't discuss your PR in Github issues there's a high chance it will not be merged.   
-
-## Did you have fun?
-Make sure you had fun coding 🙃
@@ -10,8 +10,6 @@ app/models/
 pip-wheel-metadata/
 test_tube_exp/
 tests/tests_tt_dir/
-tests/save_dir
-default/

 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -16,4 +16,4 @@ formats: all
 python:
  version: 3.7
  install:
-    - requirements: docs/requirements.txt
+    - requirements: docs/doc_requirements.txt
@@ -1,70 +1,16 @@
-# vim ft=yaml
-
-# After changing this file, check it on:
-#   http://yaml-online-parser.appspot.com/
-
-# See doc/travis_notes.txt for some guidelines
-
-# this file is *not* meant to cover or endorse the use of travis, but rather to
-# help confirm pull requests to this project.
-
-env:
-  global:
-    - DISPLAY=""
-
 language: python
-
-matrix:
-  include:
-    - os: linux
-      dist: xenial  # Ubuntu 16.04
-      python: 3.6
-      env: TOXENV=py36
-    - os: linux
-      dist: bionic  # Ubuntu 18.04
-      python: 3.6
-      env: TOXENV=py36
-    - os: linux
-      dist: bionic  # Ubuntu 18.04
-      python: 3.7
-      env: TOXENV=py37
-    - os: osx
-      osx_image: xcode9.4
-      language: generic
-      env: TOXENV=py36
-      addons:
-        homebrew:
-          # update: true
-          packages: python3
-      before_install:
-        - pip3 install virtualenv
-        - virtualenv -p python3 ~/venv
-        - source ~/venv/bin/activate
-#    - os: windows
-#      language: minimal
-#      before_install:
-#        - choco install python3
-#        - export PATH="/c/Python37:/c/Python37/Scripts:$PATH"
-#      env: TOXENV=py37
-
-# See http://docs.travis-ci.com/user/caching/#pip-cache
+python:
+  - "3.7"
+# command to install dependencies
 cache: pip
-
 install:
+  - pip install -e .
  - pip install -r requirements.txt
-  - pip install -r ./tests/requirements.txt
-  - pip --version ; pip list
+  - pip install -U numpy

+# keep build from timing out
+dist: xenial
+
+# command to run tests
 script:
-  # integration
-  - tox --sitepackages
-  - python setup.py install --dry-run
-
-after_success:
-  - coverage report
-  # disable auto coverage bc it isn't accurate since it misses gpu code.
-  # to get coverage, run local and push results
-  # - codecov
-
-notifications:
-  email: false
+  - py.test # or py.test for Python versions 3.5 and below
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 William Falcon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -1,201 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
@@ -1,42 +1,9 @@
-# Manifest syntax https://docs.python.org/2/distutils/sourcedist.html
-graft wheelhouse
+graft docs

-recursive-include birl *.py
-recursive-exclude __pycache__  *.py[cod] *.orig
+include COPYING
+include AUTHORS

-# Include the README
-include *.md
+recursive-include src/einsteinpy/tests *.py *.html

-# Include the license file
-include LICENSE
-
-exclude *.sh
-exclude *.toml
-exclude *.svg    
-recursive-include pytorch_lightning *.py
-
-# include examples
-recursive-include examples *.py
-recursive-include examples *.md
-recursive-include examples *.sh
-
-# exclude tests from package
-recursive-exclude tests *
-recursive-exclude site *
-exclude tests
-
-# Exclude the documentation files
-recursive-exclude docs *
-exclude docs
-
-# Include the Requirements
-include requirements.txt
-
-# Exclude build configs
-exclude *.yml
-
-prune .git
-prune .github
-prune notebook*
-prune temp*
-prune test*
+prune docs/source/examples/.ipynb_checkpoints
+global-exclude *.py[cod] __pycache__ *.so *.dylib
@@ -1,30 +1,24 @@
-<div align="center">
+<p align="center">
+  <a href="https://williamfalcon.github.io/pytorch-lightning/">
+    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/lightning_logo.png" width="50">
+  </a>
+</p>
+<h3 align="center">
+  Pytorch Lightning
+</h3>
+<p align="center">
+  The Keras for ML researchers using PyTorch. More control. Less boilerplate.    
+</p>

-![Logo](./docs/source/_static/lightning_logo_small.png)
+<p align="center">
+  <a href="https://badge.fury.io/py/pytorch-lightning"><img src="https://badge.fury.io/py/pytorch-lightning.svg" alt="PyPI version" height="18"></a>
+  <a href="https://pepy.tech/project/pytorch-lightning"><img src="https://pepy.tech/badge/pytorch-lightning" alt="PyPI version" height="18"></a>
+  <a href="https://github.com/williamFalcon/pytorch-lightning/tree/master/tests"><img src="https://github.com/williamFalcon/pytorch-lightning/blob/master/coverage.svg"></a>
+  <a href="https://travis-ci.org/williamFalcon/pytorch-lightning"><img src="https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master"></a>
+  <a href="https://williamfalcon.github.io/pytorch-lightning/"><img src="https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest"></a>
+  <a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/COPYING"><img src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
+</p>   

-# PyTorch Lightning
-
-**The lightweight PyTorch wrapper for ML researchers. Scale your models. Write less boilerplate.**
-
-
-[![PyPI Status](https://badge.fury.io/py/pytorch-lightning.svg)](https://badge.fury.io/py/pytorch-lightning)
-[![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
-[![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
-[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning)
-[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
-[![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
-
-[![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
-[![Gitter](https://badges.gitter.im/PyTorch-Lightning/community.svg)](https://gitter.im/PyTorch-Lightning/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
-<!-- 
-removed until codecov badge isn't empy. likely a config error showing nothing on master.
-[![codecov](https://codecov.io/gh/Borda/pytorch-lightning/branch/master/graph/badge.svg)](https://codecov.io/gh/Borda/pytorch-lightning)
-->
-
-</div>
-
-Simple installation from PyPI
 ```bash
 pip install pytorch-lightning  
 ```
@@ -33,40 +27,18 @@ pip install pytorch-lightning
 **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**

 ## What is it?  
-Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.
+Lightning defers training and validation loop logic to you. It guarantees correct, modern best practices for the core training logic.

-## Starting a new project?   
-[Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     

 ## Why do I want to use lightning?
-Every research project starts the same, a model, a training loop, validation loop, etc. As your research advances, you're likely to need distributed training, 16-bit precision, checkpointing, gradient accumulation, etc.   
+When starting a new project the last thing you want to do is recode a training loop, model loading/saving, distributed training, when to validate, etc... You're likely to spend a long time ironing out all the bugs without even getting to the core of your research.

-Lightning sets up all the boilerplate state-of-the-art training for you so you can focus on the research.   
-
---
- 
-## README Table of Contents        
- [How do I use it](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it)     
- [What lightning automates](https://github.com/williamFalcon/pytorch-lightning#what-does-lightning-control-for-me)    
- [Tensorboard integration](https://github.com/williamFalcon/pytorch-lightning#tensorboard)    
- [Lightning features](https://github.com/williamFalcon/pytorch-lightning#lightning-automates-all-of-the-following-each-is-also-configurable)    
- [Demos](https://github.com/williamFalcon/pytorch-lightning#demo)    
- [Tutorials](https://github.com/williamFalcon/pytorch-lightning#tutorials)
- [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md)    
- [Bleeding edge install](https://github.com/williamFalcon/pytorch-lightning#bleeding-edge)   
- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)   
- [Asking for help](https://github.com/williamFalcon/pytorch-lightning#asking-for-help)
- [FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
-
---
+With lightning, you guarantee those parts of your code work so you can focus on what the meat of the research: Data and training, validation loop logic. Don't worry about multiple gpus or speeding up your code, lightning will do that for you!

 ## How do I do use it?   
-Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule]((https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)) which you fit using a Trainer.    
-
-The LightningModule defines a *system* such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier such as the example below.     

 To use lightning do 2 things:  
-1. [Define a LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
+1. [Define a LightningModel](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
 ```python
 import os
 import torch
@@ -75,118 +47,93 @@ from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 import torchvision.transforms as transforms

-import pytorch_lightning as pl
+import pytorch_lightning as ptl

-class CoolSystem(pl.LightningModule):
+class CoolModel(ptl.LightningModule):

    def __init__(self):
-        super(CoolSystem, self).__init__()
+        super(CoolModel, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

+    def my_loss(self, y_hat, y):
+        return F.cross_entropy(y_hat, y)
+
    def training_step(self, batch, batch_nb):
-        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
-        return {'loss': F.cross_entropy(y_hat, y)}
+        return {'loss': self.my_loss(y_hat, y)}

    def validation_step(self, batch, batch_nb):
-        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
-        return {'val_loss': F.cross_entropy(y_hat, y)}
+        return {'val_loss': self.my_loss(y_hat, y)}

    def validation_end(self, outputs):
-        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}

    def configure_optimizers(self):
-        # REQUIRED
-        # can return multiple optimizers and learning_rate schedulers
-        return torch.optim.Adam(self.parameters(), lr=0.02)
+        return [torch.optim.Adam(self.parameters(), lr=0.02)]

-    @pl.data_loader
-    def train_dataloader(self):
-        # REQUIRED
+    @ptl.data_loader
+    def tng_dataloader(self):
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

-    @pl.data_loader
+    @ptl.data_loader
    def val_dataloader(self):
-        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

-    @pl.data_loader
+    @ptl.data_loader
    def test_dataloader(self):
-        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
+
 2. Fit with a [trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)    
 ```python
 from pytorch_lightning import Trainer
+from test_tube import Experiment

-model = CoolSystem()
-
-# most basic trainer, uses good defaults
-trainer = Trainer()    
-trainer.fit(model)   
-```     
-
-Or with tensorboard logger and some options turned on such as multi-gpu, etc...
-```python   
-from test_tube import Experiment    
-
-# PyTorch summarywriter with a few bells and whistles    
+model = CoolModel()
 exp = Experiment(save_dir=os.getcwd())

 # train on cpu using only 10% of the data (for demo purposes)
-# pass in experiment for automatic tensorboard logging.    
 trainer = Trainer(experiment=exp, max_nb_epochs=1, train_percent_check=0.1)

-# train on 4 gpus (lightning chooses GPUs for you)
-# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=4)  
-
-# train on 4 gpus (you choose GPUs)
-# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 3, 7])   
+# train on 4 gpus
+# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3])

 # train on 32 gpus across 4 nodes (make sure to submit appropriate SLURM job)
-# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=8, nb_gpu_nodes=4)
+# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3, 4, 5, 6, 7], nb_gpu_nodes=4)

 # train (1 epoch only here for demo)
 trainer.fit(model)

 # view tensorflow logs 
-print('View tensorboard logs by running\ntensorboard --logdir %s' % os.getcwd())
+print(f'View tensorboard logs by running\ntensorboard --logdir {os.getcwd()}')
 print('and going to http://localhost:6006 on your browser')
-```    
-
-When you're all done you can even run the test set separately.   
-```python
-trainer.test()
 ```

-## What does lightning control for me?   

-Everything in gray!    
-You define the blue parts using the LightningModule interface:  
+## What does lightning control for me?
+Everything!    
+Except for these 6 core functions which you define:    

-![Ouverview](./docs/source/_static/overview_flat.jpg)
-
-```python
+```{.python}
 # what to do in the training loop
-def training_step(self, batch, batch_nb):
+def training_step(self, data_batch, batch_nb):

 # what to do in the validation loop
-def validation_step(self, batch, batch_nb):
+def validation_step(self, data_batch, batch_nb):

 # how to aggregate validation_step outputs
 def validation_end(self, outputs):

 # and your dataloaders
-def train_dataloader():
+def tng_dataloader():
 def val_dataloader():
 def test_dataloader():
 ```
@@ -195,13 +142,13 @@ def test_dataloader():

 ```python
 # define what happens for training here
-def training_step(self, batch, batch_nb):
-    x, y = batch
+def training_step(self, data_batch, batch_nb):
+    x, y = data_batch
    
    # define your own forward and loss calculation
    hidden_states = self.encoder(x)
     
-    # even as complex as a seq-2-seq + attn model
+    # even as complex as a seq-2seq + attn model
    # (this is just a toy, non-working example to illustrate)
    start_token = '<SOS>'
    last_hidden = torch.zeros(...)
@@ -222,8 +169,8 @@ def training_step(self, batch, batch_nb):

 ```python
 # define what happens for validation here
-def validation_step(self, batch, batch_nb):    
-    x, y = batch
+def validation_step(self, data_batch, batch_nb):    
+    x, y = data_batch
    
    # or as basic as a CNN classification
    out = self.forward(x)
@@ -248,23 +195,31 @@ def validation_end(self, outputs):

    val_loss_mean /= len(outputs)
    val_acc_mean /= len(outputs)
-    tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-    return tqdm_dict
+    tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
+    return tqdm_dic
 ```
   
 ## Tensorboard    
 Lightning is fully integrated with tensorboard.   

-![tensorboard-support](./docs/source/_static/tf_loss.png)
+<p align="center">
+  <a href="https://williamfalcon.github.io/pytorch-lightning/">
+    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/tf_loss.png" width="900px">
+  </a>
+</p>

 Lightning also adds a text column with all the hyperparameters for this experiment.      

-![tensorboard-support](./docs/source/_static/tf_tags.png)
+<p align="center">
+  <a href="https://williamfalcon.github.io/pytorch-lightning/">
+        <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/tf_tags.png" width="900px">
+  </a>
+</p>

-Simply note the path you set for the [Experiment](https://williamfalcon.github.io/test-tube/experiment_tracking/experiment/) from [test_tube](https://github.com/williamFalcon/test-tube)
-```python
+Simply note the path you set for the Experiment    
+``` {.python}   
 from test_tube import Experiment
-from pytorch_lightning import Trainer
+from pytorch-lightning import  Trainer

 exp = Experiment(save_dir='/some/path')
 trainer = Trainer(experiment=exp)
@@ -279,29 +234,27 @@ tensorboard --logdir /some/path
 ## Lightning automates all of the following ([each is also configurable](https://williamfalcon.github.io/pytorch-lightning/Trainer/)):


-#### Checkpointing    
+###### Checkpointing    

 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)

-#### Computing cluster (SLURM)    
+###### Computing cluster (SLURM)    

 - [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
 - [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   

-#### Debugging  
+###### Debugging  

 - [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
 - [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
 - [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
- [Print which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
+- [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)


-#### Distributed training    
+###### Distributed training    

 - [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
 - [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
@@ -310,7 +263,7 @@ tensorboard --logdir /some/path
 - [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)


-#### Experiment Logging   
+###### Experiment Logging   

 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
@@ -320,20 +273,18 @@ tensorboard --logdir /some/path
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)

-#### Training loop    
+###### Training loop    

 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
- [Early stopping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)    
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)

-#### Validation loop    
+###### Validation loop    

 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
@@ -342,18 +293,16 @@ tensorboard --logdir /some/path
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
 - [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)

-#### Testing loop  
- [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  
+

 ## Demo
 ```bash
 # install lightning
-pip install pytorch_lightning
+pip install pytorch-lightning

 # clone lightning for the demo
 git clone https://github.com/williamFalcon/pytorch-lightning.git
-cd pytorch-lightning
-cd examples/new_project_templates/
+cd pytorch_lightning/examples/new_project_templates/

 # all of the following demos use the SAME model to show no modification needs to be made to your code

@@ -367,78 +316,8 @@ python single_gpu_node_template.py --gpus "0,1"
 python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'
 ```

-## Tutorials   
- [Basic Lightning use](https://towardsdatascience.com/supercharge-your-ai-research-with-pytorch-lightning-337948a99eec)    
- [9 key speed features in Pytorch-Lightning](https://towardsdatascience.com/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565)    
- [SLURM, multi-node training with Lightning](https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd)     
-
---    
-## Asking for help    
-Welcome to the Lightning community!   
-
-If you have any questions, feel free to:   
-1. [read the docs](https://williamfalcon.github.io/pytorch-lightning/).     
-2. [Search through the issues](https://github.com/williamFalcon/pytorch-lightning/issues?utf8=%E2%9C%93&q=my++question).      
-3. [Ask on stackoverflow](https://stackoverflow.com/questions/ask?guided=false) with the tag pytorch-lightning.   
-
-If no one replies to you quickly enough, feel free to post the stackoverflow link to our Gitter chat!   
-
-To chat with the rest of us visit our [gitter channel](https://gitter.im/PyTorch-Lightning/community)!     
-
---   
-## FAQ    
-**How do I use Lightning for rapid research?**   
-[Here's a walk-through](https://williamfalcon.github.io/pytorch-lightning/)  
-
-**Why was Lightning created?**     
-Lightning has 3 goals in mind:
-1. Maximal flexibility while abstracting out the common boilerplate across research projects.   
-2. Reproducibility. If all projects use the LightningModule template, it will be much much easier to understand what's going on and where to look! It will also mean every implementation follows a standard format.   
-3. Democratizing PyTorch power user features. Distributed training? 16-bit? know you need them but don't want to take the time to implement? All good... these come built into Lightning.    
-
-**How does Lightning compare with Ignite and fast.ai?**     
-[Here's a thorough comparison](https://medium.com/@_willfalcon/pytorch-lightning-vs-pytorch-ignite-vs-fast-ai-61dc7480ad8a).    
-
-**Is this another library I have to learn?**    
-Nope! We use pure Pytorch everywhere and don't add unecessary abstractions!   
-
-**Are there plans to support Python 2?**          
-Nope.     
-
-**Are there plans to support virtualenv?**    
-Nope. Please use anaconda or miniconda.    
-
-**Which PyTorch versions do you support?**    
- **PyTorch 1.1.0**       
-    ```bash    
-    # install pytorch 1.1.0 using the official instructions   
-    
-    # install test-tube 0.6.7.6 which supports 1.1.0   
-    pip install test-tube==0.6.7.6   
-    
-    # install latest Lightning version without upgrading deps    
-    pip install -U --no-deps pytorch-lightning
-    ```     
- **PyTorch 1.2.0**
-    Install via pip as normal   
-
-## Custom installation
-
-### Bleeding edge
-
-If you can't wait for the next release, install the most up to date code with:
-* using GIT (locally clone whole repo with full history)
-    ```bash
-    pip install git+https://github.com/williamFalcon/pytorch-lightning.git@master --upgrade
-    ```
-* using instant zip (last state of the repo without git history)
-    ```bash
-    pip install https://github.com/williamFalcon/pytorch-lightning/archive/master.zip --upgrade
-    ```
-
-### Any release installation
-
-You can also install any past release from this repository:
+## Bleeding edge
+If you can't wait for the next release, install the most up to date code with:  
 ```bash
-pip install https://github.com/williamFalcon/pytorch-lightning/archive/0.4.4.zip --upgrade
+pip install git+https://github.com/williamFalcon/pytorch-lightning.git@master --upgrade
 ```
@@ -1,64 +0,0 @@
-# https://www.appveyor.com/docs/appveyor-yml/
-environment:
-
-  # SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
-  # /E:ON and /V:ON options are not enabled in the batch script interpreter
-  # See: http://stackoverflow.com/a/13751649/163740
-  CMD_IN_ENV: "cmd /E:ON /V:ON /C obvci_appveyor_python_build_env.cmd"
-
-  matrix:
-    # Pre-installed Python versions, which Appveyor may upgrade to
-    # a later point release.
-    # See: http://www.appveyor.com/docs/installed-software#python
-
-
-    # - PYTHON: "C:\\Python35-x64"
-    #   PYTHON_VERSION: "3.5.x"
-    #   PYTHON_ARCH: "64"
-    #  TOXENV: "py35"
-
-    - PYTHON: "C:\\Python36-x64"
-      PYTHON_VERSION: "3.6.x"
-      PYTHON_ARCH: "64"
-      TOXENV: "py36"
-      PIP_PYVER: "36"
-
-    - PYTHON: "C:\\Python37-x64"
-      PYTHON_VERSION: "3.7.x"
-      PYTHON_ARCH: "64"
-      TOXENV: "py37"
-      PIP_PYVER: "37"
-
-build: off
-
-# https://www.appveyor.com/docs/build-cache/
-cache:
-  - C:\ProgramData\chocolatey\bin -> appveyor.yml
-  - C:\ProgramData\chocolatey\lib -> appveyor.yml
-  - '%LOCALAPPDATA%\pip\Cache -> appveyor.yml'
-
-# scripts that run after cloning repository
-install:
-  # If there is a newer build queued for the same PR, cancel this one.
-  # The AppVeyor 'rollout builds' option is supposed to serve the same
-  # purpose but it is problematic because it tends to cancel builds pushed
-  # directly to master instead of just PR builds (or the converse).
-  - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%path%
-  - pip install -U --user pip
-  - pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
-  - pip install -r ./tests/requirements.txt
-
-# scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
-before_test:
-  - python --version
-  - pip --version
-  - pip list
-  - dir
-
-# to run your custom scripts instead of automatic tests
-test_script:
-  - tox --sitepackages --parallel auto
-
-on_success:
-  - coverage report
-  # - codecov
@@ -9,22 +9,22 @@ Otherwise, to Define a Lightning Module, implement the following methods:

 **Required**:  

- [training_step](RequiredTrainerInterface.md#training_step)      
- [train_dataloader](RequiredTrainerInterface.md#train_dataloader)    
- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)    
+- [training_step](RequiredTrainerInterface.md#training_step)   
+- [validation_step](RequiredTrainerInterface.md#validation_step)
+- [validation_end](RequiredTrainerInterface.md#validation_end)
+    
+- [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)
+
+- [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)
+- [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)
+- [test_dataloader](RequiredTrainerInterface.md#test_dataloader)

 **Optional**:   

- [validation_step](RequiredTrainerInterface.md#validation_step)    
- [validation_end](RequiredTrainerInterface.md#validation_end) 
- [test_step](RequiredTrainerInterface.md#test_step)    
- [test_end](RequiredTrainerInterface.md#test_end) 
- [val_dataloader](RequiredTrainerInterface.md#val_dataloader)    
- [test_dataloader](RequiredTrainerInterface.md#test_dataloader)    
- [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)    
- [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)    
- [update_training_log_metrics](RequiredTrainerInterface.md#update_training_log_metrics)    
- [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)    
+- [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)
+- [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)
+- [update_tng_log_metrics](RequiredTrainerInterface.md#update_tng_log_metrics)
+- [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)

 ---
 ### Minimal example   
@@ -36,9 +36,9 @@ from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 import torchvision.transforms as transforms

-import pytorch_lightning as pl
+import pytorch_lightning as ptl

-class CoolModel(pl.LightningModule):
+class CoolModel(ptl.LightningModule):

    def __init__(self):
        super(CoolModel, self).__init__()
@@ -48,70 +48,45 @@ class CoolModel(pl.LightningModule):
    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

+    def my_loss(self, y_hat, y):
+        return F.cross_entropy(y_hat, y)
+
    def training_step(self, batch, batch_nb):
-        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
-        return {'loss': F.cross_entropy(y_hat, y)}
+        return {'loss': self.my_loss(y_hat, y)}

    def validation_step(self, batch, batch_nb):
-        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
-        return {'val_loss': F.cross_entropy(y_hat, y)}
+        return {'val_loss': self.my_loss(y_hat, y)}

    def validation_end(self, outputs):
-        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}

-    def test_step(self, batch, batch_nb):
-        # OPTIONAL
-        x, y = batch
-        y_hat = self.forward(x)
-        return {'test_loss': F.cross_entropy(y_hat, y)}
-
-    def test_end(self, outputs):
-        # OPTIONAL
-        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
-        return {'avg_test_loss': avg_loss}
-
    def configure_optimizers(self):
-        # REQUIRED
        return [torch.optim.Adam(self.parameters(), lr=0.02)]

-    @pl.data_loader
-    def train_dataloader(self):
+    @ptl.data_loader
+    def tng_dataloader(self):
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

-    @pl.data_loader
+    @ptl.data_loader
    def val_dataloader(self):
-        # OPTIONAL
-        # can also return a list of val dataloaders
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

-    @pl.data_loader
+    @ptl.data_loader
    def test_dataloader(self):
-        # OPTIONAL
-        # can also return a list of test dataloaders
-        return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
---   
-### How do these methods fit into the broader training?     
-The LightningModule interface is on the right. Each method corresponds to a part of a research project. Lightning automates everything not in blue.    

-<p align="center">
-  <a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg">
-    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg" height="900px">
-  </a>
-</p>   
-
-## Required Methods    
+---

 ### training_step

 ``` {.python}
-def training_step(self, batch, batch_nb)
+def training_step(self, data_batch, batch_nb)
 ```

 In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
@@ -120,7 +95,7 @@ In this step you'd normally do the forward pass and calculate the loss for a bat

 | Param  | description  |
 |---|---|
-|  batch | The output of your dataloader. A tensor, tuple or list  |
+|  data_batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |

 **Return**   
@@ -130,14 +105,14 @@ Dictionary or OrderedDict
 | key  | value  | is required |
 |---|---|---|
 |  loss | tensor scalar  | Y |
-|  progress | Dict for progress bar display. Must have only tensors  | N |
+|  prog | Dict for progress bar display. Must have only tensors  | N |


 **Example**

 ``` {.python}
-def training_step(self, batch, batch_nb):
-    x, y, z = batch
+def training_step(self, data_batch, batch_nb):
+    x, y, z = data_batch
    
    # implement your own
    out = self.forward(x)
@@ -145,143 +120,46 @@ def training_step(self, batch, batch_nb):
    
    output = {
        'loss': loss, # required
-        'progress': {'training_loss': loss, 'batch_nb': batch_nb} # optional
+        'prog': {'tng_loss': loss, 'batch_nb': batch_nb} # optional
    }
    
    # return a dict
    return output
-```    
-
-If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.    
-``` {.python}
-# Multiple optimizers (ie: GANs)     
-def training_step(self, batch, batch_nb, optimizer_idx):
-    if optimizer_idx == 0:
-        # do training_step with encoder
-    if optimizer_idx == 1:
-        # do training_step with decoder    
-```    
-
--- 
-### train_dataloader 
-
-``` {.python}
-@pl.data_loader
-def train_dataloader(self)
-```
-Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
-
-##### Return
-PyTorch DataLoader
-
-**Example**
-
-``` {.python}
-@pl.data_loader
-def train_dataloader(self):
-    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
-    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
-    loader = torch.utils.data.DataLoader(
-        dataset=dataset,
-        batch_size=self.hparams.batch_size,
-        shuffle=True
-    )
-    return loader
 ```

--- 
-### configure_optimizers 
-
-``` {.python}
-def configure_optimizers(self)
-```
-
-Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
-Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
-
-**Note:** If you use multiple optimizers, training_step will have an additional ```optimizer_idx``` parameter.    
-
-
-
-##### Return    
-Return any of these 3 options:   
-Single optimizer   
-List or Tuple - List of optimizers    
-Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers
-
-**Example**
-
-``` {.python}
-# most cases
-def configure_optimizers(self):
-    opt = Adam(self.parameters(), lr=0.01)
-    return opt
-
-# multiple optimizer case (eg: GAN)
-def configure_optimizers(self):
-    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
-    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
-    return generator_opt, disriminator_opt
-    
-# example with learning_rate schedulers  
-def configure_optimizers(self):
-    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
-    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
-    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
-    return [generator_opt, disriminator_opt], [discriminator_sched]
-```  
-
-If you need to control how often those optimizers step or override the default .step() schedule, override 
-the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step) hook.   
-
-## Optional Methods    
+---

 ### validation_step

 ``` {.python}
-# if you have one val dataloader:
-def validation_step(self, batch, batch_nb)   
-
-# if you have multiple val dataloaders:  
-def validation_step(self, batch, batch_nb, dataloader_idxdx)
+def validation_step(self, data_batch, batch_nb)
 ```
-**OPTIONAL**    
-If you don't need to validate you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy. 

-When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
-
-The dict you return here will be available in the `validation_end` method. 
+In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
+This is most likely the same as your training_step. But unlike training step, the outputs from here will go to validation_end for collation.

 **Params**   

 | Param  | description  |
 |---|---|
-|  batch | The output of your dataloader. A tensor, tuple or list  |
+|  data_batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
-|  dataloader_idx | Integer displaying which dataloader this is (only if multiple val datasets used)  |

 **Return**   

 | Return  | description  | optional |
 |---|---|---|   
-|  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
+|  dict | Dict of OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |

 **Example**

 ``` {.python}
-# CASE 1: A single validation dataset
-def validation_step(self, batch, batch_nb):
-    x, y = batch
+def validation_step(self, data_batch, batch_nb):
+    x, y, z = data_batch
    
    # implement your own
    out = self.forward(x)
-    loss = self.loss(out, y)
-    
-    # log 6 example images
-    # or generated text... or whatever
-    sample_imgs = x[:6]
-    grid = torchvision.utils.make_grid(sample_imgs)
-    self.experiment.add_image('example_images', grid, 0) 
+    loss = self.loss(out, x)
    
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
@@ -296,35 +174,22 @@ def validation_step(self, batch, batch_nb):
    
    # return an optional dict
    return output
-```   
-
-If you pass in multiple validation datasets, validation_step will have an additional argument.
-
-```python
-# CASE 2: multiple validation datasets
-def validation_step(self, batch, batch_nb, dataset_idx):
-    # dataset_idx tells you which dataset this is.   
-```   
-
-The ```dataset_idx``` corresponds to the order of datasets returned in ```val_dataloader```.    
+```

 --- 
 ### validation_end

 ``` {.python}
 def validation_end(self, outputs)
-```   
-If you didn't define a validation_step, this won't be called.       
+```

-Called at the end of the validation loop with the outputs of validation_step.
-
-The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
+Called at the end of the validation loop with the output of each validation_step.

 **Params**    

 | Param  | description  |
 |---|---|
-|  outputs | List of outputs you defined in validation_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
+|  outputs | List of outputs you defined in validation_step |

 **Return**   

@@ -334,8 +199,6 @@ The outputs here are strictly for the progress bar. If you don't need to display

 **Example**

-With a single dataloader
-
 ``` {.python}
 def validation_end(self, outputs):
    """
@@ -351,173 +214,38 @@ def validation_end(self, outputs):

    val_loss_mean /= len(outputs)
    val_acc_mean /= len(outputs)
-    tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-    return tqdm_dict
+    tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
+    return tqdm_dic
 ```

-With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
-one entry per dataloader, while the inner list contains the individual outputs of 
-each validation step for that dataloader.
-
-``` {.python}
-def validation_end(self, outputs):
-    """
-    Called at the end of validation to aggregate outputs
-    :param outputs: list of list of individual outputs of each validation step
-    :return:
-    """
-    val_loss_mean = 0
-    val_acc_mean = 0
-    i = 0
-    for dataloader_outputs in outputs:
-        for output in dataloader_outputs:
-            val_loss_mean += output['val_loss']
-            val_acc_mean += output['val_acc']
-            i += 1
-
-    val_loss_mean /= i
-    val_acc_mean /= i
-    tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-    return tqdm_dict
-```
-
-### test_step
-
-``` {.python}
-# if you have one test dataloader:
-def test_step(self, batch, batch_nb)   
-
-# if you have multiple test dataloaders:  
-def test_step(self, batch, batch_nb, dataloader_idxdx)
-```
-**OPTIONAL**    
-If you don't need to test you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy.   
-
-When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
-
-The dict you return here will be available in the `test_end` method. 
-
-This function is used when you execute `trainer.test()`.
-
-**Params**   
-
-| Param  | description  |
-|---|---|
-|  batch | The output of your dataloader. A tensor, tuple or list  |
-|  batch_nb | Integer displaying which batch this is  |
-|  dataloader_idx | Integer displaying which dataloader this is (only if multiple test datasets used)  |
-
-**Return**   
-
-| Return  | description  | optional |
-|---|---|---|   
-|  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
-
-**Example**
-
-``` {.python}
-# CASE 1: A single test dataset
-def test_step(self, batch, batch_nb):
-    x, y = batch
-    
-    # implement your own
-    out = self.forward(x)
-    loss = self.loss(out, y)
-    
-    # calculate acc
-    labels_hat = torch.argmax(out, dim=1)
-    test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-    
-    # all optional...
-    # return whatever you need for the collation function test_end
-    output = OrderedDict({
-        'test_loss': loss_test,
-        'test_acc': torch.tensor(test_acc), # everything must be a tensor
-    })
-    
-    # return an optional dict
-    return output
-```   
-
-If you pass in multiple test datasets, test_step will have an additional argument.
-
-```python
-# CASE 2: multiple test datasets
-def test_step(self, batch, batch_nb, dataset_idx):
-    # dataset_idx tells you which dataset this is.   
-```   
-
-The ```dataset_idx``` corresponds to the order of datasets returned in ```test_dataloader```.    
-
 --- 
-### test_end
+### configure_optimizers 

 ``` {.python}
-def test_end(self, outputs)
-```   
-If you didn't define a test_step, this won't be called.       
+def configure_optimizers(self)
+```

-Called at the end of the test step with the output of each test_step.
+Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
+Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.

-The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    

-**Params**    
-
-| Param  | description  |
-|---|---|
-|  outputs | List of outputs you defined in test_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
-
-**Return**   
-
-| Return  | description  | optional |
-|---|---|---|   
-|  dict | Dict of OrderedDict with metrics to display in progress bar | Y |
+##### Return
+List or Tuple - List of optimizers with an optional second list of learning-rate schedulers

 **Example**

 ``` {.python}
-def test_end(self, outputs):
-    """
-    Called at the end of test to aggregate outputs
-    :param outputs: list of individual outputs of each test step
-    :return:
-    """
-    test_loss_mean = 0
-    test_acc_mean = 0
-    for output in outputs:
-        test_loss_mean += output['test_loss']
-        test_acc_mean += output['test_acc']
-
-    test_loss_mean /= len(outputs)
-    test_acc_mean /= len(outputs)
-    tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
-    return tqdm_dict
-```
-
-With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
-one entry per dataloader, while the inner list contains the individual outputs of 
-each validation step for that dataloader.
-
-``` {.python}
-def test_end(self, outputs):
-    """
-    Called at the end of test to aggregate outputs
-    :param outputs: list of individual outputs of each test step
-    :return:
-    """
-    test_loss_mean = 0
-    test_acc_mean = 0
-    i = 0
-    for dataloader_outputs in outputs:
-        for output in dataloader_outputs:
-            test_loss_mean += output['test_loss']
-            test_acc_mean += output['test_acc']
-            i += 1
-
-    test_loss_mean /= i 
-    test_acc_mean /= i
-    tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
-    return tqdm_dict
+# most cases
+def configure_optimizers(self):
+    opt = Adam(self.parameters(), lr=0.01)
+    return [opt]
+    
+# gan example, with scheduler for discriminator
+def configure_optimizers(self):
+    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
+    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
+    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
+    return [generator_opt, disriminator_opt], [discriminator_sched]
 ```

 --- 
@@ -563,24 +291,48 @@ def on_load_checkpoint(self, checkpoint):
 ```

 --- 
-### val_dataloader 
+### tng_dataloader 

 ``` {.python}
-@pl.data_loader
-def val_dataloader(self)
+@ptl.data_loader
+def tng_dataloader(self)
 ```
-**OPTIONAL**    
-If you don't need a validation dataset and a validation_step, you don't need to implement this method.    
-
-Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.   
+Called by lightning during training loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.

 ##### Return
-PyTorch DataLoader or list of PyTorch Dataloaders.    
+Pytorch DataLoader

 **Example**

 ``` {.python}
-@pl.data_loader
+@ptl.data_loader
+def tng_dataloader(self):
+    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
+    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
+    loader = torch.utils.data.DataLoader(
+        dataset=dataset,
+        batch_size=self.hparams.batch_size,
+        shuffle=True
+    )
+    return loader
+```
+
+--- 
+### val_dataloader 
+
+``` {.python}
+@ptl.data_loader
+def tng_dataloader(self)
+```
+Called by lightning during validation loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.
+
+##### Return
+Pytorch DataLoader
+
+**Example**
+
+``` {.python}
+@ptl.data_loader
 def val_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
@@ -591,35 +343,24 @@ def val_dataloader(self):
    )
    
    return loader
-
-# can also return multiple dataloaders   
-@pl.data_loader
-def val_dataloader(self):
-    return [loader_a, loader_b, ..., loader_n]   
 ```

-In the case where you return multiple val_dataloaders, the validation_step will have an arguement ```dataset_idx```
-which matches the order here.    
-
 --- 
 ### test_dataloader 

 ``` {.python}
-@pl.data_loader
+@ptl.data_loader
 def test_dataloader(self)
 ```
-**OPTIONAL**    
-If you don't need a test dataset and a test_step, you don't need to implement this method.    
-
-Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
+Called by lightning during test loop. Make sure to use the @ptl.data_loader decorator, this ensures not calling this function until the data are needed.

 ##### Return
-PyTorch DataLoader
+Pytorch DataLoader

 **Example**

 ``` {.python}
-@pl.data_loader
+@ptl.data_loader
 def test_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
@@ -633,13 +374,13 @@ def test_dataloader(self):
 ```

 --- 
-### update_training_log_metrics 
+### update_tng_log_metrics 

 ``` {.python}
-def update_training_log_metrics(self, logs)
+def update_tng_log_metrics(self, logs)
 ```
 Called by lightning right before it logs metrics for this batch.
-This is a chance to amend or add to the metrics about to be logged.
+This is a chance to ammend or add to the metrics about to be logged.

 ##### Return
 Dict 
@@ -647,7 +388,7 @@ Dict
 **Example**

 ``` {.python}
-def update_training_log_metrics(self, logs):
+def update_tng_log_metrics(self, logs):
    # modify or add to logs
    return logs
 ```
@@ -674,7 +415,7 @@ def add_model_specific_args(parent_parser, root_dir):
    parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])

    # param overwrites
-    # parser.set_defaults(gradient_clip_val=5.0)
+    # parser.set_defaults(gradient_clip=5.0)

    # network params
    parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
@@ -31,7 +31,7 @@ y_hat = pretrained_model(x)

 | Param  | description  |
 |---|---|
-|  weights_path | Path to a PyTorch checkpoint  |
+|  weights_path | Path to a pytorch checkpoint  |
 |  tags_csv | Path to meta_tags.csv file generated by the test-tube Experiment  |
 |  on_gpu | if True, puts model on GPU. Make sure to use transforms option if model devices have changed  |
 |  map_location | A dictionary mapping saved weight GPU devices to new GPU devices |
@@ -10,7 +10,7 @@ Current dtype

 ---
 #### experiment    
-An instance of test-tube Experiment which you can use to log anything for tensorboard (subclass of [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html)).   
+An instance of test-tube Experiment which you can use to log anything for tensorboarX.   
 ```{.python}
 self.experiment.add_embedding(...)   
 self.experiment.log({'val_loss': 0.9})   
@@ -22,7 +22,7 @@ self.experiment.add_scalars(...)
 Total training batches seen across all epochs   

 --- 
-#### gradient_clip_val   
+#### gradient_clip   
 The current gradient clip value    

 ---
@@ -38,17 +38,3 @@ self.trainer.current_epoch
 ...   
 ```   

-## Debugging   
-The LightningModule also offers these tricks to help debug.   
-
---   
-#### example_input_array    
-In the LightningModule init, you can set a dummy tensor for this property
-to get a print out of sizes coming into and out of every layer.   
-```python
-def __init__(self):
-    # put the dimensions of the first input to your system
-    self.example_input_array = torch.rand(5, 28 * 28)
-```    
-
-
@@ -5,66 +5,18 @@ Lightning can automate saving and loading checkpoints.
 To enable checkpointing, define the checkpoint callback and give it to the trainer.

 ``` {.python}
-from pytorch_lightning.callbacks import ModelCheckpoint
+from pytorch_lightning.utils.pt_callbacks import ModelCheckpoint

 checkpoint_callback = ModelCheckpoint(
-    filepath='/path/to/store/weights/',
+    filepath='/path/to/store/weights.ckpt',
    save_best_only=True,
    verbose=True,
    monitor='val_loss',
-    mode='min',
-    prefix=''
+    mode='min'
 )

 trainer = Trainer(checkpoint_callback=checkpoint_callback)
 ```

---
-### Restoring training session 
-You might want to not only load a model but also continue training it. Use this method to
-restore the trainer state as well. This will continue from the epoch and global step you last left off.  
-However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).   

-Lightning will restore the session if you pass an experiment with the same version and there's a saved checkpoint.   
-``` {.python}
-from test_tube import Experiment

-exp = Experiment(version=a_previous_version_with_a_saved_checkpoint)
-trainer = Trainer(experiment=exp)
-
-# this fit call loads model weights and trainer state
-# the trainer continues seamlessly from where you left off
-# without having to do anything else.
-trainer.fit(model)
-```
-
-The trainer restores:  
-
- global_step    
- current_epoch    
- All optimizers    
- All lr_schedulers    
- Model weights
-
-You can even change the logic of your model as long as the weights and "architecture" of 
-the system isn't different. If you add a layer, for instance, it might not work.   
-
-At a rough level, here's [what happens inside Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/model_saving.py#L63):   
-```python
-
-self.global_step = checkpoint['global_step']
-self.current_epoch = checkpoint['epoch']
-
-# restore the optimizers
-optimizer_states = checkpoint['optimizer_states']
-for optimizer, opt_state in zip(self.optimizers, optimizer_states):
-    optimizer.load_state_dict(opt_state)
-
-# restore the lr schedulers
-lr_schedulers = checkpoint['lr_schedulers']
-for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
-    scheduler.load_state_dict(lrs_state)
-
-# uses the model you passed into trainer        
-model.load_state_dict(checkpoint['state_dict'])
-```    
@@ -10,13 +10,10 @@ For multi-node training you must use DistributedDataParallel.

 You can toggle between each mode by setting this flag.
 ``` {.python}
-# DEFAULT (when using single GPU or no GPUs)
-trainer = Trainer(distributed_backend=None)
-
-# Change to DataParallel (gpus > 1)
+# DEFAULT uses DataParallel
 trainer = Trainer(distributed_backend='dp')

-# change to distributed data parallel (gpus > 1)
+# change to distributed data parallel
 trainer = Trainer(distributed_backend='ddp')
 ```

@@ -26,34 +23,6 @@ have configuration issues depending on your cluster.

 For a deeper understanding of what lightning is doing, feel free to read [this guide](https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565).   

---
-#### Distributed and 16-bit precision.    
-Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
-not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.   
-
-Below are the possible configurations we support.    
-
-| 1 GPU  | 1+ GPUs  | DP  | DDP  | 16-bit  | command |
-|---|---|---|---|---|---|
-| Y  |   |   |   |  | ```Trainer(gpus=1)``` |
-| Y  |   |   |   | Y | ```Trainer(gpus=1, use_amp=True)``` |
-|   | Y | Y |   |   | ```Trainer(gpus=k, distributed_backend='dp')``` |
-|   | Y |  | Y  |  | ```Trainer(gpus=k, distributed_backend='ddp')``` |
-|   | Y |  | Y  | Y | ```Trainer(gpus=k, distributed_backend='ddp', use_amp=True)``` |
-
-You also have the option of specifying which GPUs to use by passing a list:   
-
-```python
-# DEFAULT (int)
-Trainer(gpus=k)  
-
-# You specify which GPUs (don't use if running on cluster)  
-Trainer(gpus=[0, 1])  
-
-# can also be a string
-Trainer(gpus='0, 1')
-```
-
 ---
 #### CUDA flags   
 CUDA flags make certain GPUs visible to your script. 
@@ -64,9 +33,6 @@ Lightning sets these for you automatically, there's NO NEED to do this yourself.
 # os.environ["CUDA_VISIBLE_DEVICES"] = "0"
 ```

-However, when using a cluster, Lightning will NOT set these flags (and you should not either). 
-SLURM will set these for you.   
-
 ---
 #### 16-bit mixed precision
 16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.    
@@ -88,7 +54,7 @@ trainer = Trainer(amp_level='O2', use_amp=False)
 Make sure you're on a GPU machine. 
 ```python
 # DEFAULT
-trainer = Trainer(gpus=1)
+trainer = Trainer(gpus=[0])
 ```

 ---
@@ -96,11 +62,11 @@ trainer = Trainer(gpus=1)
 Make sure you're on a GPU machine. You can set as many GPUs as you want.
 In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
 ```python
-# to use DataParallel
-trainer = Trainer(gpus=8, distributed_backend='dp')
+# to use DataParallel (default)
+trainer = Trainer(gpus=[0,1,2,3,4,5,6,7], distributed_backend='dp')

 # RECOMMENDED use DistributedDataParallel
-trainer = Trainer(gpus=8, distributed_backend='ddp')
+trainer = Trainer(gpus=[0,1,2,3,4,5,6,7], distributed_backend='ddp')
 ```

 ---
@@ -108,7 +74,7 @@ trainer = Trainer(gpus=8, distributed_backend='ddp')
 Multi-node training is easily done by specifying these flags.
 ```python
 # train on 12*8 GPUs
-trainer = Trainer(gpus=8, nb_gpu_nodes=12, distributed_backend='ddp')
+trainer = Trainer(gpus=[0,1,2,3,4,5,6,7], nb_gpu_nodes=12)
 ```

 In addition, make sure to set up your SLURM job correctly via the [SlurmClusterObject](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/). In particular, specify the number of tasks per node correctly.
@@ -128,7 +94,7 @@ cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
 cluster.add_command('export NCCL_DEBUG=INFO')

 # setting a master port here is a good idea.
-cluster.add_command('export MASTER_PORT=%r' % PORT)
+cluster.add_command(f'export MASTER_PORT={PORT}')

 # good to load the latest NCCL version
 cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
@@ -140,9 +106,6 @@ cluster.per_experiment_nb_gpus = 8
 cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
 ```

-**NOTE:** When running in DDP mode, any errors in your code will show up as an NCCL issue.
-Set the ```NCCL_DEBUG=INFO``` flag to see the ACTUAL error.
-
 Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a 
 portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).   

@@ -5,7 +5,7 @@ Lighting offers a few options for logging information about model, gpu usage, et
 #### Display metrics in progress bar 
 ``` {.python}
 # DEFAULT
-trainer = Trainer(show_progress_bar=True)
+trainer = Trainer(progress_bar=True)
 ```

 ---
@@ -13,15 +13,7 @@ trainer = Trainer(show_progress_bar=True)
 Every k batches lightning will make an entry in the metrics log
 ``` {.python}
 # DEFAULT (ie: save a .csv log file every 10 batches)
-trainer = Trainer(row_log_interval=10)
-```   
-
---
-#### Log metric row every k batches 
-Logs GPU memory when metrics are logged.   
-``` {.python}
-# DEFAULT
-trainer = Trainer(log_gpu_memory=False)
+trainer = Trainer(add_log_row_interval=10)
 ```

 ---
@@ -41,7 +33,7 @@ trainer = Trainer(process_position=1)
 Whenever you call .save() on the test-tube experiment it logs all the hyperparameters in current use.
 Give lightning a test-tube Experiment object to automate this for you.
 ``` {.python}
-from test_tube import Experiment
+from test-tube import Experiment

 exp = Experiment(...)
 Trainer(experiment=exp)
@@ -52,7 +44,7 @@ Trainer(experiment=exp)
 Whenever you call .save() on the test-tube experiment it snapshows all code and pushes to a git tag.
 Give lightning a test-tube Experiment object to automate this for you.
 ``` {.python}
-from test_tube import Experiment
+from test-tube import Experiment

 exp = Experiment(create_git_tag=True)
 Trainer(experiment=exp)
@@ -60,16 +52,7 @@ Trainer(experiment=exp)

 ---
 ### Tensorboard support   
-In the LightningModule you can access the experiment logger by doing:
-```python
-self.experiment
-
-# add image
-# Look at PyTorch SummaryWriter docs for what you can do.   
-self.experiment.add_image(...)
-```
-
-The experiment object is a strict subclass of PyTorch SummaryWriter. However, this class
+The experiment object is a strict subclass of Pytorch SummaryWriter. However, this class
 also snapshots every detail about the experiment (data folder paths, code, hyperparams),
 and allows you to visualize it using tensorboard.
 ``` {.python}
@@ -1,10 +1,8 @@
 Lightning supports model training on a cluster managed by SLURM in the following cases:    

-1. Training on a single cpu or single GPU.
-2. Train on multiple GPUs on the same node using DataParallel or DistributedDataParallel
-3. Training across multiple GPUs on multiple different nodes via DistributedDataParallel.
-
-**Note: A node means a machine with multiple GPUs**
+1. Training on single or multi-cpus only.
+2. Training on single or multi-gpus on the same node.
+3. Coming SOON: Training across multiple nodes.

 ---
 #### Running grid search on a cluster
@@ -25,9 +23,6 @@ parser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4,
 hparams = parser.parse_args()    
 ```    
    
-**NOTE** You must set ```Tunable=True``` for that argument to be considered in the permutation set. Otherwise
-test-tube will use the default value. This flag is useful when you don't want to search over an argument and
-want to use the default instead.   
     
 (2). Define the cluster options in the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) (over 5 nodes and 8 gpus)    

@@ -60,8 +55,8 @@ cluster.memory_mb_per_node = 10000
 cluster.job_time = '10:00'
 ```

-(3). Make a main function with your model and trainer. Each job will call this function with a particular
-hparams configuration.    
+(3). Give trainer the cluster_manager in your main function:    
+
 ```{.python}
 from pytorch_lightning import Trainer

@@ -71,12 +66,12 @@ def train_fx(trial_hparams, cluster_manager, _):
    my_model = MyLightningModel()
    
    # give the trainer the cluster object
-    trainer = Trainer()
+    trainer = Trainer(cluster=cluster_manager)
    trainer.fit(my_model)

 ```

-(3). Start the grid/random search     
+(4). Start the grid search     
 ```{.python}
 # run the models on the cluster
 cluster.optimize_parallel_cluster_gpu(
@@ -86,27 +81,24 @@ cluster.optimize_parallel_cluster_gpu(
    job_display_name='my_exp')
 ```

-**NOTE** nb_trials specifies how many of the possible permutations to use. If using ```grid_search``` it will use
-the depth first ordering. If using ```random_search``` it will use the first k shuffled options. FYI, random search
-has been shown to be just as good as any Bayesian optimization method when using a reasonable number of samples (60),
-[see this paper for more information](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf).
+That's it! The SlurmCluster object will automatically checkpoint the lightning model and resubmit if it runs into the walltime!
+

 ---
 #### Walltime auto-resubmit
-Lightning automatically resubmits jobs when they reach the walltime. Make sure to set the SIGUSR1 signal in 
-your SLURM script.   
+Lightning automatically resubmits jobs when they reach the walltime. You get this behavior for free if you give lightning
+a slurm cluster object.

-```bash
-# 90 seconds before training ends
-#SBATCH --signal=SIGUSR1@90
+```{.python}
+def my_main_fx(hparams, slurm_manager, _):
+    trainer = Trainer(cluster=slurm_manager)
 ``` 

-When lightning receives the SIGUSR1 signal it will:
-1. save a checkpoint with 'hpc_ckpt' in the name.
-2. resubmit the job using the SLURM_JOB_ID  
-
-When the script starts again, Lightning will:
-1. search for a 'hpc_ckpt' checkpoint. 
-2. restore the model, optimizers, schedulers, epoch, etc...   
+(See the grid search example above for cluster configuration).
+With this feature lightning will:    

+1. automatically checkpoint the model
+2. checkpoint the trainer session
+3. resubmit a continuation job.
+4. load the checkpoint and trainer session in the new model

@@ -1,31 +0,0 @@
-To ensure you don't accidentally use test data to guide training decisions Lightning makes running the test set deliberate.   
-
---
-#### test
-You have two options to run the test set. 
-First case is where you test right after a full training routine.
-``` {.python}
-# run full training
-trainer.fit(model)
-
-# run test set
-trainer.test()
-```   
-
-Second case is where you load a model and run the test set   
-```{.python}
-model = MyLightningModule.load_from_metrics(
-    weights_path='/path/to/pytorch_checkpoint.ckpt',
-    tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
-    on_gpu=True,
-    map_location=None
-)
-
-# init trainer with whatever options
-trainer = Trainer(...)
-    
-# test (pass in the model)
-trainer.test(model)
-```
-In this second case, the options you pass to trainer will be used when running the test set (ie: 16-bit, dp, ddp, etc...)  
- 
@@ -19,24 +19,6 @@ It can be useful to force training for a minimum number of epochs or limit to a
 trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000)
 ```

---
-#### Early stopping
-To enable ealry-stopping, define the callback and give it to the trainer.
-``` {.python}
-from pytorch_lightning.callbacks import EarlyStopping
-
-# DEFAULTS
-early_stop_callback = EarlyStopping(
-    monitor='val_loss',
-    min_delta=0.00,
-    patience=0,
-    verbose=False,
-    mode='auto'
-)
-
-trainer = Trainer(early_stop_callback=early_stop_callback)
-```
-
 ---
 #### Force disable early stop 
 Use this to turn off early stopping and run training to the [max_epoch](#force-training-for-min-or-max-epochs)
@@ -46,18 +28,15 @@ trainer = Trainer(enable_early_stop=True)
 ```

 ---
-#### Gradient Clipping
-Gradient clipping may be enabled to avoid exploding gradients.
-Specifically, this will [clip the gradient norm computed over all model parameters *together*](https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_).
-
+#### Gradient Clipping 
+Use this to turn off early stopping and run training to the [max_epoch](#force-training-for-min-or-max-epochs)
 ``` {.python}
 # DEFAULT (ie: don't clip)
-trainer = Trainer(gradient_clip_val=0)
-
-# clip gradients with norm above 0.5
-trainer = Trainer(gradient_clip_val=0.5)
+trainer = Trainer(gradient_clip=0)
 ```

+
+
 ---
 #### Inspect gradient norms
 Looking at grad norms can help you figure out where training might be going wrong.
@@ -72,10 +51,7 @@ trainer = Trainer(track_grad_norm=2)

 ---
 #### Set how much of the training set to check
-If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
-
-train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
-
+If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag
 ``` {.python}
 # DEFAULT
 trainer = Trainer(train_percent_check=1.0)
@@ -5,6 +5,8 @@ Below are all the things lightning automates for you in the validation loop.
 Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don't have to wait until a full epoch to catch possible validation issues.


+
+
 ---
 #### Check validation every n epochs
 If you have a small dataset you might want to check validation every n epochs
@@ -16,9 +18,6 @@ trainer = Trainer(check_val_every_n_epoch=1)
 ---
 #### Set how much of the validation set to check 
 If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
-
-val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
-
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_percent_check=1.0)
@@ -30,9 +29,6 @@ trainer = Trainer(val_percent_check=0.1)
 ---
 #### Set how much of the test set to check 
 If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
-
-test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
-
 ``` {.python}
 # DEFAULT
 trainer = Trainer(test_percent_check=1.0)
@@ -58,6 +54,4 @@ Lightning runs a few steps of validation in the beginning of training. This avoi
 ``` {.python}
 # DEFAULT
 trainer = Trainer(nb_sanity_val_steps=5)
-```
-
-You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.
+```
@@ -23,9 +23,6 @@ trainer = Trainer(track_grad_norm=2)
 ---
 #### Make model overfit on subset of data
 A useful debugging trick is to make your model overfit a tiny fraction of the data.
-
-setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
-
 ``` {.python}
 # DEFAULT don't overfit (ie: normal training)
 trainer = Trainer(overfit_pct=0.0)
@@ -5,7 +5,7 @@ There are cases when you might want to do something different at different parts
 To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.

 **Contributing** If there's a hook you'd like to add, simply:    
-1. Fork PyTorchLightning.    
+1. Fork PytorchLightning.    
 2. Add the hook [here](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py).       
 3. Add the correct place in the [Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py) where it should be called.    

@@ -18,7 +18,7 @@ def on_epoch_start(self):
 ```

 ---
-#### on_epoch_end
+#### on_batch_end
 Called in the training loop at the very end of the epoch.   
 ```python
 def on_epoch_end(self):
@@ -33,14 +33,6 @@ def on_batch_start(self):
    # do something when the batch starts
 ```

---
-#### on_batch_end
-Called in the training loop after the batch.   
-```python
-def on_batch_end(self):
-    # do something when the batch ends 
-```
-
 ---
 #### on_pre_performance_check
 Called at the very beginning of the validation loop.   
@@ -58,62 +50,15 @@ def on_post_performance_check(self):
 ```

 ---
-#### on_training_metrics
+#### on_tng_metrics
 Called in the training loop, right before metrics are logged.
 Although you can log at any time by using self.experiment, you can use
 this callback to modify what will be logged.
 ```python
-def on_training_metrics(self, metrics):
+def on_tng_metrics(self, metrics):
    # do something before validation end
 ```

---   
-#### optimizer_step 
-Calls .step() and .zero_grad for each optimizer.  
-You can override this method to adjust how you do the optimizer step for each optimizer
-
-Called once per optimizer
-```python
-# DEFAULT
-def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
-    optimizer.step()   
-    optimizer.zero_grad()   
-    
-# Alternating schedule for optimizer steps (ie: GANs)    
-def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
-    # update generator opt every 2 steps
-    if optimizer_i == 0:
-        if batch_nb % 2 == 0 :
-            optimizer.step()
-            optimizer.zero_grad()
-   
-    # update discriminator opt every 4 steps
-    if optimizer_i == 1:
-        if batch_nb % 4 == 0 :
-            optimizer.step()
-            optimizer.zero_grad()    
-    
-    # ...
-    # add as many optimizers as you want 
-```
-
-This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:   
-
-```python
-# learning rate warm-up
-def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
-    # warm up lr
-    if self.trainer.global_step < 500:
-        lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
-        for pg in optimizer.param_groups:
-            pg['lr'] = lr_scale * self.hparams.learning_rate
-    
-    # update params
-    optimizer.step()
-    optimizer.zero_grad() 
-```
-
-
 ---
 #### on_before_zero_grad
 Called in the training loop after taking an optimizer step and before zeroing grads.
@@ -19,44 +19,42 @@ But of course the fun is in all the advanced things it can do:

 **Checkpointing**    

- [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
- [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
+- Model saving
+- Model loading 

 **Computing cluster (SLURM)**    

- [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
- [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
+- [Running grid search on a cluster](SLURM%20Managed%20Cluster/#running-grid-search-on-a-cluster)
+- [Walltime auto-resubmit](SLURM%20Managed%20Cluster/#walltime-auto-resubmit)   

 **Debugging**  

- [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
- [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
- [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
- [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
- [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
- [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
+- [Fast dev run](Debugging/#fast-dev-run)
+- [Inspect gradient norms](Debugging/#inspect-gradient-norms)
+- [Log GPU usage](Debugging/#Log-gpu-usage)
+- [Make model overfit on subset of data](Debugging/#make-model-overfit-on-subset-of-data)
+- [Print the parameter count by layer](Debugging/#print-the-parameter-count-by-layer)
+- [Pring which gradients are nan](Debugging/#print-which-gradients-are-nan)


 **Distributed training**    

- [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
- [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
- [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
- [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
- [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
+- [16-bit mixed precision](Distributed%20training/#16-bit-mixed-precision)
+- [Multi-GPU](Distributed%20training/#Multi-GPU)
+- [Multi-node](Distributed%20training/#Multi-node)
+- [Single GPU](Distributed%20training/#single-gpu)
+- [Self-balancing architecture](Distributed%20training/#self-balancing-architecture)


 **Experiment Logging**   

- [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
- [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
- [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
+- [Display metrics in progress bar](Logging/#display-metrics-in-progress-bar)
+- [Log metric row every k batches](Logging/#log-metric-row-every-k-batches)
+- [Process position](Logging/#process-position)
 - [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
- [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
- [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
- [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
+- [Save a snapshot of all hyperparameters](Logging/#save-a-snapshot-of-all-hyperparameters) 
+- [Snapshot code for a training run](Logging/#snapshot-code-for-a-training-run) 
+- [Write logs file to csv every k batches](Logging/#write-logs-file-to-csv-every-k-batches)

 **Training loop**    

@@ -64,22 +62,16 @@ But of course the fun is in all the advanced things it can do:
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
+- [Hooks](hooks)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)

 **Validation loop**    

- [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
- [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
- [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
- [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
- [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
- [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
-
-
-**Testing loop**    
-
- [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  
+- [Check validation every n epochs](Validation%20Loop/#check-validation-every-n-epochs)
+- [Hooks](hooks)
+- [Set how much of the validation set to check](Validation%20Loop/#set-how-much-of-the-validation-set-to-check)
+- [Set how much of the test set to check](Validation%20Loop/#set-how-much-of-the-test-set-to-check)
+- [Set validation check frequency within 1 training epoch](Validation%20Loop/#set-validation-check-frequency-within-1-training-epoch)
+- [Set the number of validation sanity steps](Validation%20Loop/#set-the-number-of-validation-sanity-steps)
@@ -1,2 +1 @@
 mkdocs-material==4.4.0
-mkdocs==1.0.4
@@ -3,7 +3,7 @@ In 99% of cases you want to just copy [this template](https://github.com/william

 ```bash
 # get a copy of the module template
-wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/examples/new_project_templates/lightning_module_template.py
+wget https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/lightning_module_template.py
 ```

 ---    
@@ -40,7 +40,7 @@ The main function should have 3 arguments:
 - slurm_manager: Slurm cluster manager object (can be None)
 - dict: for you to return any values you want (useful in meta-learning, otherwise set to _)    

-```python
+```{}
 def main(hparams, cluster, results_dict):
    """
    Main training routine specific for this project
@@ -1,68 +1,17 @@
 ###### New project Quick Start    
-To start a new project define two files, a LightningModule and a Trainer file.    
-To illustrate Lightning power and simplicity, here's an example of a typical research flow.    
+To start a new project you define two files, a LightningModule and a Trainer file.    

-###### Case 1: BERT    
-Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
-You would define a single LightningModule and use flags to switch between your different ideas.   
-```python
-class BERT(pl.LightningModule):
-    def __init__(self, model_name, task):
-        self.task = task
-    
-        if model_name == 'transformer':
-            self.net = Transformer()
-        elif model_name == 'my_cool_version':
-            self.net = MyCoolVersion()
-            
-    def training_step(self, batch, batch_nb):
-        if self.task == 'standard_bert':
-            # do standard bert training with self.net...
-            # return loss
-            
-        if self.task == 'my_cool_task':
-            # do my own version with self.net
-            # return loss
-```   
+A separate trainer file allows to run many LightningModules. Each LightningModule has the core
+logic to a particular research project.    

-###### Case 2: COOLER NOT BERT    
-But if you wanted to try something **completely** different, you'd define a new module for that.    
-```python
+For example, one lightningModule could be an image classifier, the other
+one could be a seq-2-seq model, both (optionally) ran by the same trainer file.

-class CoolerNotBERT(pl.LightningModule):
-    def __init__(self):
-        self.net = ...
-        
-    def training_step(self, batch, batch_nb):
-        # do some other cool task
-        # return loss   
-```   
-
-###### Rapid research flow    
-Then you could do rapid research by switching between these two and using the same trainer.   
-```python
-
-if use_bert:
-    model = BERT()
-else:
-    model = CoolerNotBERT()
-    
-trainer = Trainer(gpus=[0, 1, 2, 3], use_amp=True)
-trainer.fit(model)
-```
-
-Notice a few things about this flow:   
-1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
-2. You get free GPU and 16-bit support without writing any of that code in your model.   
-3. You also get all of the capabilities below (without coding or testing yourself).     
-
---    
-###### Templates 
 1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
 2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
-    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_cpu_template.py) 
-    - [Multi-GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_gpu_node_template.py)
-    - [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/multi_node_cluster_template.py)
+    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/examples/new_project_templates/single_cpu_template.py) 
+    - [Multi-GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/examples/new_project_templates/single_gpu_node_template.py)
+    - [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/examples/new_project_templates/multi_node_cluster_template.py)

 ###### Docs shortcuts
 - [LightningModule](LightningModule/RequiredTrainerInterface/)  
@@ -79,7 +28,6 @@ Notice a few things about this flow:

 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)

 ###### Computing cluster (SLURM)    

@@ -94,7 +42,6 @@ Notice a few things about this flow:
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
 - [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
- [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)


 ###### Distributed training    
@@ -120,14 +67,12 @@ Notice a few things about this flow:

 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
- [Early stopping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
+- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/Pytorch-Lightning/LightningModule/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)

 ###### Validation loop    

@@ -138,5 +83,3 @@ Notice a few things about this flow:
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
 - [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)

-###### Testing loop  
- [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  
@@ -1,107 +0,0 @@
-# Multi-node examples
-Use these templates for multi-node training. 
-The main complexity around cluster training is how you submit the SLURM jobs.  
-
-## Test-tube   
-Lightning uses test-tube to submit SLURM jobs and to run hyperparameter searches on a cluster.  
-
-To run a hyperparameter search, we normally add the values to search to the Hyperparameter optimizer 
-```python
-from test_tube import HyperOptArgumentParser
-
-parser = HyperOptArgumentParser(strategy='grid_search')
-parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=True)
-parser.opt_list('--learning_rate', default=0.001, type=float,
-                        options=[0.0001, 0.0005, 0.001],
-                        tunable=True)
-                        
-# give your model a chance to add its own parameters
-parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
-
-# parse args
-hyperparams = parser.parse_args()
-```
-
-The above sets up a grid search on learning rate and drop probability. You can now add this object to the 
-cluster object to perform the grid search:   
-```python
-cluster = SlurmCluster(
-    hyperparam_optimizer=hyperparams,
-    log_path='/path/to/log/slurm/files',
-)
-
-# ... configure cluster options
-
-# run grid search on cluster
-nb_trials = 6   # (2 drop probs * 3 lrs)
-cluster.optimize_parallel_cluster_gpu(
-    YourMainFunction,
-    nb_trials=nb_trials,
-    job_name=hyperparams.experiment_name
-)
-```
-
-Running the above will launch 6 jobs, each with a different drop prob and learning rate combination.   
-The ```tunable``` parameter must be set to True to add that argument to the space of options, otherwise
-Test-Tube will use the ```default=value```.    
-
-
-## SLURM Flags   
-However you decide to submit your jobs, debugging requires a few flags. Without these flags, you'll
-see a nccl error instead of the actual error which caused the bug.   
-
-```sh
-export NCCL_DEBUG=INFO
-export PYTHONFAULTHANDLER=1
-```
-
-On some clusters you might need to set the network interface with this flag.   
-```sh
-export NCCL_SOCKET_IFNAME=^docker0,lo
-```   
-
-You might also need to load the latest version of NCCL  
-```sh
-module load NCCL/2.4.7-1-cuda.10.0
-```
-
-Finally, you must set the master port (usually a random number between 12k and 20k).   
-```sh
-# random port between 12k and 20k
-export MASTER_PORT=$((12000 + RANDOM % 20000))$   
-```
-
-## Simplest example.   
-1. Modify this script with your CoolModel file.   
-2. Update and submit [this bash script](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/multi_node_examples/minimal_multi_node_demo_script.sh)   
-```bash
-squeue minimal_multi_node_demo_script.sh
-```
-
-## Grid search on a cluster   
-
-#### Option 1: Run on cluster using your own SLURM script    
-The trainer and model will work on a cluster if you configure your SLURM script correctly.   
-
-1. Update [this demo slurm script](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/multi_node_examples/demo_script.sh).  
-2. Submit the script   
-```bash
-$ squeue demo_script.sh
-```
-
-Most people have some way they automatically generate their own scripts.  
-To run a grid search this way, you'd need a way to automatically generate scripts using all the combinations of 
-hyperparameters to search over.   
-
-#### Option 2: Use test-tube for SLURM script
-With test tube we can automatically generate slurm scripts for different hyperparameter options.   
-
-To run this demo:    
-```bash
-source activate YourCondaEnv
-
-python multi_node_cluster_auto_slurm.py --email your@email.com --gpu_partition your_partition --conda_env YourCondaEnv
-```
-
-That will submit 6 jobs. Each job will have a specific combination of hyperparams. Each job will also run on 2 nodes
-where each node has 8 gpus.   
@@ -1,66 +0,0 @@
-#!/bin/bash
-#
-# Auto-generated by test-tube (https://github.com/williamFalcon/test-tube)
-#################
-
-# set a job name
-#SBATCH --job-name=lightning_test
-#################
-
-# a file for job output, you can check job progress
-#SBATCH --output=/slurm_output_%j.out
-#################
-
-# a file for errors
-#SBATCH --error=/slurm_output_%j.err
-#################
-
-# time needed for job
-#SBATCH --time=01:00:00
-#################
-
-# gpus per node
-#SBATCH --gres=gpu:8
-#################
-
-# cpus per job
-#SBATCH --cpus-per-task=10
-#################
-
-# number of requested nodes
-#SBATCH --nodes=2
-#################
-
-# memory per node (0 means all)
-#SBATCH --mem=0
-#################
-
-# slurm will send a signal this far out before it kills the job
-#SBATCH --signal=USR1@300
-#################
-
-# comment
-#SBATCH --comment=lightning_demo
-#################
-
-# 1 task per gpu
-#SBATCH --ntasks-per-node=8
-#################
-
-source activate YourEnv
-
-# debugging flags (optional)
-export NCCL_DEBUG=INFO
-export PYTHONFAULTHANDLER=1
-
-# on your cluster you might need these:
-# set the network interface
-export NCCL_SOCKET_IFNAME=^docker0,lo
-
-# might need the latest cuda
-module load NCCL/2.4.7-1-cuda.10.0
-
-# random port between 12k and 20k
-export MASTER_PORT=$((12000 + RANDOM % 20000))$
-
-srun python multi_node_own_slurm_script.py
@@ -1,24 +0,0 @@
-from pytorch_lightning import Trainer
-from test_tube import Experiment
-import os
-
-
-def main():
-    # use the cool model from the main README.md
-    model = CoolModel()  # noqa: F821
-    exp = Experiment(save_dir=os.getcwd())
-
-    # train on 4 GPUs across 4 nodes
-    trainer = Trainer(
-        experiment=exp,
-        distributed_backend='ddp',
-        max_nb_epochs=10,
-        gpus=4,
-        nb_gpu_nodes=4
-    )
-
-    trainer.fit(model)
-
-
-if __name__ == '__main__':
-    main()
@@ -1,30 +0,0 @@
-#!/bin/bash -l
-
-# SLURM SUBMIT SCRIPT
-#SBATCH --nodes=4
-#SBATCH --gres=gpu:4
-#SBATCH --ntasks-per-node=4
-#SBATCH --mem=0
-#SBATCH --time=0-02:00:00
-
-# activate conda env
-conda activate my_env
-
-# -------------------------
-# debugging flags (optional)
-# export NCCL_DEBUG=INFO
-# export PYTHONFAULTHANDLER=1
-
-# on your cluster you might need these:
-# set the network interface
-# export NCCL_SOCKET_IFNAME=^docker0,lo
-
-# might need the latest cuda
-# module load NCCL/2.4.7-1-cuda.10.0
-# -------------------------
-
-# random port between 12k and 20k
-export MASTER_PORT=$((12000 + RANDOM % 20000))
-
-# run script from above
-python minimal_multi_node_demo.py
@@ -1,70 +0,0 @@
-"""
-Multi-node example (GPU)
-"""
-import os
-import numpy as np
-import torch
-
-from test_tube import HyperOptArgumentParser, Experiment
-from pytorch_lightning import Trainer
-from examples.new_project_templates.lightning_module_template import LightningTemplateModel
-
-SEED = 2334
-torch.manual_seed(SEED)
-np.random.seed(SEED)
-
-
-def main(hparams):
-    """
-    Main training routine specific for this project
-    :param hparams:
-    :return:
-    """
-    # ------------------------
-    # 1 INIT LIGHTNING MODEL
-    # ------------------------
-    model = LightningTemplateModel(hparams)
-
-    # ------------------------
-    # 2 INIT TEST TUBE EXP
-    # ------------------------
-    # init experiment
-    exp = Experiment(
-        name='test_exp',
-        save_dir=hyperparams.log_dir,
-        autosave=False,
-        description='test demo'
-    )
-
-    # ------------------------
-    # 2 INIT TRAINER
-    # ------------------------
-    trainer = Trainer(
-        experiment=exp,
-        gpus=8,
-        nb_gpu_nodes=2
-    )
-
-    # ------------------------
-    # 5 START TRAINING
-    # ------------------------
-    trainer.fit(model)
-
-
-if __name__ == '__main__':
-    # use current dir for logging
-    root_dir = os.path.dirname(os.path.realpath(__file__))
-    log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
-
-    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
-    parent_parser.add_argument('--log_dir', type=str, default=log_dir,
-                               help='where to save logs')
-
-    # allow model to overwrite or extend args
-    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
-    hyperparams = parser.parse_args()
-
-    # ---------------------
-    # RUN TRAINING
-    # ---------------------
-    main(hyperparams)
@@ -1,178 +0,0 @@
-"""
-To run this template just do:  
-python gan.py   
-
-After a few epochs, launch tensorboard to see the images being generated at every batch.   
-
-tensorboard --logdir default
-"""
-from argparse import ArgumentParser
-import os
-import numpy as np
-
-import torchvision
-import torchvision.transforms as transforms
-from torchvision.datasets import MNIST
-
-from torch.utils.data import DataLoader
-
-import torch.nn as nn
-import torch.nn.functional as F
-import torch
-
-import pytorch_lightning as pl
-from test_tube import Experiment
-
-
-class Generator(nn.Module):
-    def __init__(self, latent_dim, img_shape):
-        super(Generator, self).__init__()
-        self.img_shape = img_shape
-
-        def block(in_feat, out_feat, normalize=True):
-            layers = [nn.Linear(in_feat, out_feat)]
-            if normalize:
-                layers.append(nn.BatchNorm1d(out_feat, 0.8))
-            layers.append(nn.LeakyReLU(0.2, inplace=True))
-            return layers
-
-        self.model = nn.Sequential(
-            *block(latent_dim, 128, normalize=False),
-            *block(128, 256),
-            *block(256, 512),
-            *block(512, 1024),
-            nn.Linear(1024, int(np.prod(img_shape))),
-            nn.Tanh()
-        )
-
-    def forward(self, z):
-        img = self.model(z)
-        img = img.view(img.size(0), *self.img_shape)
-        return img
-
-
-class Discriminator(nn.Module):
-    def __init__(self, img_shape):
-        super(Discriminator, self).__init__()
-
-        self.model = nn.Sequential(
-            nn.Linear(int(np.prod(img_shape)), 512),
-            nn.LeakyReLU(0.2, inplace=True),
-            nn.Linear(512, 256),
-            nn.LeakyReLU(0.2, inplace=True),
-            nn.Linear(256, 1),
-            nn.Sigmoid(),
-        )
-
-    def forward(self, img):
-        img_flat = img.view(img.size(0), -1)
-        validity = self.model(img_flat)
-
-        return validity
-
-
-class GAN(pl.LightningModule):
-
-    def __init__(self, hparams):
-        super(GAN, self).__init__()
-        self.hparams = hparams
-
-        # networks
-        mnist_shape = (1, 28, 28)
-        self.generator = Generator(latent_dim=hparams.latent_dim, img_shape=mnist_shape)
-        self.discriminator = Discriminator(img_shape=mnist_shape)
-
-        # cache for generated images
-        self.generated_imgs = None
-
-    def forward(self, z):
-        return self.generator(z)
-
-    def adversarial_loss(self, y_hat, y):
-        return F.binary_cross_entropy(y_hat, y)
-
-    def training_step(self, batch, batch_nb, optimizer_i):
-        imgs, _ = batch
-
-        # train generator
-        if optimizer_i == 0:
-            # sample noise
-            z = torch.randn(imgs.shape[0], self.hparams.latent_dim)
-
-            # match gpu device (or keep as cpu)
-            if self.on_gpu:
-                z = z.cuda(imgs.device.index)
-
-            # generate images
-            self.generated_imgs = self.forward(z)
-
-            # log sampled images
-            sample_imgs = self.generated_imgs[:6]
-            grid = torchvision.utils.make_grid(sample_imgs)
-            self.experiment.add_image('generated_images', grid, 0)
-
-            # ground truth result (ie: all fake)
-            valid = torch.ones(imgs.size(0), 1)
-
-            # adversarial loss is binary cross-entropy
-            g_loss = self.adversarial_loss(self.discriminator(self.generated_imgs), valid)
-
-            return g_loss
-
-        # train discriminator
-        if optimizer_i == 1:
-            # Measure discriminator's ability to classify real from generated samples
-
-            # how well can it label as real?
-            valid = torch.ones(imgs.size(0), 1)
-            real_loss = self.adversarial_loss(self.discriminator(imgs), valid)
-
-            # how well can it label as fake?
-            fake = torch.zeros(imgs.size(0), 1)
-            fake_loss = self.adversarial_loss(self.discriminator(self.generated_imgs.detach()), fake)
-
-            # discriminator loss is the average of these
-            d_loss = (real_loss + fake_loss) / 2
-
-            return d_loss
-
-    def configure_optimizers(self):
-        lr = self.hparams.lr
-        b1 = self.hparams.b1
-        b2 = self.hparams.b2
-
-        opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
-        opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))
-        return [opt_g, opt_d], []
-
-    @pl.data_loader
-    def train_dataloader(self):
-        transform = transforms.Compose([transforms.ToTensor(),
-                                        transforms.Normalize([0.5], [0.5])])
-        dataset = MNIST(os.getcwd(), train=True, download=True, transform=transform)
-        return DataLoader(dataset, batch_size=self.hparams.batch_size)
-
-
-def main(hparams):
-    # save tensorboard logs
-    exp = Experiment(save_dir=os.getcwd())
-
-    # init model
-    model = GAN(hparams)
-
-    # fit trainer on CPU
-    trainer = pl.Trainer(experiment=exp, max_nb_epochs=200)
-    trainer.fit(model)
-
-
-if __name__ == '__main__':
-    parser = ArgumentParser()
-    parser.add_argument("--batch_size", type=int, default=64, help="size of the batches")
-    parser.add_argument("--lr", type=float, default=0.0002, help="adam: learning rate")
-    parser.add_argument("--b1", type=float, default=0.5, help="adam: decay of first order momentum of gradient")
-    parser.add_argument("--b2", type=float, default=0.999, help="adam: decay of first order momentum of gradient")
-    parser.add_argument("--latent_dim", type=int, default=100, help="dimensionality of the latent space")
-
-    hparams = parser.parse_args()
-
-    main(hparams)
@@ -1,16 +1,10 @@
-site_name: PyTorch lightning Documentation
+site_name: Pytorch lightning Documentation
 theme:
  name: 'material'
 docs_dir: docs
-repo_name: 'williamFalcon/pytorch-lightning'
 repo_url: https://github.com/williamFalcon/pytorch-lightning
 site_dir: 'site'
-site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
+site_description: 'Documentation for Pytorch LightningModule, the researcher version of keras.'

 dev_addr: '0.0.0.0:8000'
 #google_analytics: ['UA-aasd', 'sitename']
-
-markdown_extensions:
-  - codehilite:
-      guess_lang: false
-      linenums: true
@@ -1,9 +1,3 @@
-from .trainer.trainer import Trainer
+from .models import Trainer
 from .root_module.root_module import LightningModule
-from .root_module.decorators import data_loader
-
-__all__ = [
-    'Trainer',
-    'LightningModule',
-    'data_loader',
-]
+from .root_module.decorators import data_loader
@@ -1,7 +1 @@
-from .pt_callbacks import EarlyStopping, ModelCheckpoint, GradientAccumulationScheduler
-
-__all__ = [
-    'EarlyStopping',
-    'ModelCheckpoint',
-    'GradientAccumulationScheduler',
-]
+from .pt_callbacks import EarlyStopping, ModelCheckpoint
@@ -1,9 +1,5 @@
-import os
-import shutil
-import warnings
-
 import numpy as np
-
+import os, shutil
 from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel


@@ -12,6 +8,7 @@ class Callback(object):
    # Properties
        params: dict. Training parameters
            (eg. verbosity, batch size, number of epochs...).
+        model: instance of `keras.models.Model`.
            Reference of the model being trained.
    The `logs` dictionary that callback methods
    take as argument will contain keys for quantities relevant to
@@ -122,9 +119,9 @@ class EarlyStopping(Callback):
        current = logs.get(self.monitor)
        stop_training = False
        if current is None:
-            print('Early stopping conditioned on metric `%s` '
-                  'which is not available. Available metrics are: %s' %
-                  (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning)
+            print('Early stopping conditioned on metric `%s` ''which is not available. Available metrics are: %s' %
+                (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
+            )
            exit(-1)

        if self.monitor_op(current - self.min_delta, self.best):
@@ -188,7 +185,8 @@ class ModelCheckpoint(Callback):

        if mode not in ['auto', 'min', 'max']:
            print('ModelCheckpoint mode %s is unknown, '
-                  'fallback to auto mode.' % (mode), RuntimeWarning)
+                          'fallback to auto mode.' % (mode),
+                          RuntimeWarning)
            mode = 'auto'

        if mode == 'min':
@@ -232,8 +230,8 @@ class ModelCheckpoint(Callback):
            if self.save_best_only:
                current = logs.get(self.monitor)
                if current is None:
-                    print('Can save best model only with %s available,'
-                          ' skipping.' % (self.monitor), RuntimeWarning)
+                    print('Can save best model only with %s available, '
+                                  'skipping.' % (self.monitor), RuntimeWarning)
                else:
                    if self.monitor_op(current, self.best):
                        if self.verbose > 0:
@@ -254,37 +252,6 @@ class ModelCheckpoint(Callback):
                self.save_model(filepath, overwrite=False)


-class GradientAccumulationScheduler(Callback):
-    """Change gradient accumulation factor according to scheduling.
-    # Arguments
-        scheduling: dict, scheduling in format {epoch: accumulation_factor}
-    """
-    def __init__(self, scheduling: dict):
-        if scheduling == {}:  # empty dict error
-            raise TypeError("Empty dict cannot be interpreted correct")
-
-        for key in scheduling.keys():
-            if not isinstance(key, int) or not isinstance(scheduling[key], int):
-                raise TypeError("All epoches and accumulation factor must be integers")
-
-        minimal_epoch = min(scheduling.keys())
-        if minimal_epoch < 1:
-            msg = f"Epochs indexing from 1, epoch {minimal_epoch} cannot be interpreted correct"
-            raise IndexError(msg)
-        elif minimal_epoch != 1:  # if user didnt define first epoch accumulation factor
-            scheduling.update({1: 1})
-
-        self.scheduling = scheduling
-        self.epochs = sorted(scheduling.keys())
-
-    def on_epoch_begin(self, epoch, trainer):
-        epoch += 1  # indexing epochs from 1
-        for i in reversed(range(len(self.epochs))):
-            if epoch >= self.epochs[i]:
-                trainer.accumulate_grad_batches = self.scheduling.get(self.epochs[i])
-                break
-
-
 if __name__ == '__main__':
    c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
    losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
@@ -293,3 +260,4 @@ if __name__ == '__main__':
        print(loss)
        if should_stop:
            break
+
@@ -1,5 +1 @@
-from .new_project_templates.lightning_module_template import LightningTemplateModel
-
-__all__ = [
-    'LightningTemplateModel'
-]
+from .new_project_templates.lightning_module_template import LightningTemplateModel
@@ -1,6 +1,3 @@
-"""
-Example template for defining a system
-"""
 import os
 from collections import OrderedDict
 import torch.nn as nn
@@ -13,7 +10,7 @@ from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler

-import pytorch_lightning as pl
+import pytorch_lightning as ptl
 from pytorch_lightning.root_module.root_module import LightningModule


@@ -47,13 +44,11 @@ class LightningTemplateModel(LightningModule):
        Layout model
        :return:
        """
-        self.c_d1 = nn.Linear(in_features=self.hparams.in_features,
-                              out_features=self.hparams.hidden_dim)
+        self.c_d1 = nn.Linear(in_features=self.hparams.in_features, out_features=self.hparams.hidden_dim)
        self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
        self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)

-        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim,
-                              out_features=self.hparams.out_features)
+        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim, out_features=self.hparams.out_features)

    # ---------------------
    # TRAINING
@@ -79,14 +74,14 @@ class LightningTemplateModel(LightningModule):
        nll = F.nll_loss(logits, labels)
        return nll

-    def training_step(self, batch, batch_idx):
+    def training_step(self, data_batch, batch_i):
        """
        Lightning calls this inside the training loop
-        :param batch:
+        :param data_batch:
        :return:
        """
        # forward pass
-        x, y = batch
+        x, y = data_batch
        x = x.view(x.size(0), -1)

        y_hat = self.forward(x)
@@ -105,13 +100,13 @@ class LightningTemplateModel(LightningModule):
        # can also return just a scalar instead of a dict (return loss_val)
        return output

-    def validation_step(self, batch, batch_idx):
+    def validation_step(self, data_batch, batch_i):
        """
        Lightning calls this inside the validation loop
-        :param batch:
+        :param data_batch:
        :return:
        """
-        x, y = batch
+        x, y = data_batch
        x = x.view(x.size(0), -1)
        y_hat = self.forward(x)

@@ -151,24 +146,13 @@ class LightningTemplateModel(LightningModule):
        val_loss_mean = 0
        val_acc_mean = 0
        for output in outputs:
-            val_loss = output['val_loss']
-
-            # reduce manually when using dp
-            if self.trainer.use_dp:
-                val_loss = torch.mean(val_loss)
-            val_loss_mean += val_loss
-
-            # reduce manually when using dp
-            val_acc = output['val_acc']
-            if self.trainer.use_dp:
-                val_acc = torch.mean(val_acc)
-
-            val_acc_mean += val_acc
+            val_loss_mean += output['val_loss']
+            val_acc_mean += output['val_acc']

        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
-        tqdm_dict = {'val_loss': val_loss_mean, 'val_acc': val_acc_mean}
-        return tqdm_dict
+        tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
+        return tqdm_dic

    # ---------------------
    # TRAINING SETUP
@@ -184,18 +168,19 @@ class LightningTemplateModel(LightningModule):

    def __dataloader(self, train):
        # init data generators
-        transform = transforms.Compose([transforms.ToTensor(),
-                                        transforms.Normalize((0.5,), (1.0,))])
-        dataset = MNIST(root=self.hparams.data_root, train=train,
-                        transform=transform, download=True)
+        transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
+        dataset = MNIST(root=self.hparams.data_root, train=train, transform=transform, download=True)

-        # when using multi-node (ddp) we need to add the datasampler
+        # when using multi-node we need to add the datasampler
        train_sampler = None
        batch_size = self.hparams.batch_size

-        if self.use_ddp:
-            train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
-            batch_size = batch_size // self.trainer.world_size  # scale batch size
+        try:
+            if self.on_gpu:
+                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
+                batch_size = batch_size // self.trainer.world_size  # scale batch size
+        except Exception as e:
+            pass

        should_shuffle = train_sampler is None
        loader = DataLoader(
@@ -207,23 +192,23 @@ class LightningTemplateModel(LightningModule):

        return loader

-    @pl.data_loader
-    def train_dataloader(self):
-        print('training data loader called')
+    @ptl.data_loader
+    def tng_dataloader(self):
+        print('tng data loader called')
        return self.__dataloader(train=True)

-    @pl.data_loader
+    @ptl.data_loader
    def val_dataloader(self):
        print('val data loader called')
        return self.__dataloader(train=False)

-    @pl.data_loader
+    @ptl.data_loader
    def test_dataloader(self):
        print('test data loader called')
        return self.__dataloader(train=False)

    @staticmethod
-    def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
+    def add_model_specific_args(parent_parser, root_dir): # pragma: no cover
        """
        Parameters you define here will be available to your model through self.hparams
        :param parent_parser:
@@ -233,28 +218,22 @@ class LightningTemplateModel(LightningModule):
        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])

        # param overwrites
-        # parser.set_defaults(gradient_clip_val=5.0)
+        # parser.set_defaults(gradient_clip=5.0)

        # network params
-        parser.add_argument('--in_features', default=28 * 28, type=int)
+        parser.add_argument('--in_features', default=28*28, type=int)
        parser.add_argument('--out_features', default=10, type=int)
-        # use 500 for CPU, 50000 for GPU to see speed difference
-        parser.add_argument('--hidden_dim', default=50000, type=int)
-        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=True)
-        parser.opt_list('--learning_rate', default=0.001 * 8, type=float,
-                        options=[0.0001, 0.0005, 0.001],
-                        tunable=True)
+        parser.add_argument('--hidden_dim', default=50000, type=int) # use 500 for CPU, 50000 for GPU to see speed difference

        # data
        parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)

        # training params (opt)
-        parser.opt_list('--optimizer_name', default='adam', type=str,
-                        options=['adam'], tunable=False)
+        parser.opt_list('--learning_rate', default=0.001*8, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
+                        tunable=False)
+        parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)

-        # if using 2 nodes with 4 gpus each the batch size here
-        #  (256) will be 256 / (2*8) = 16 per gpu
-        parser.opt_list('--batch_size', default=256 * 8, type=int,
-                        options=[32, 64, 128, 256], tunable=False,
-                        help='batch size will be divided over all gpus being used across all nodes')
+        # if using 2 nodes with 4 gpus each the batch size here (256) will be 256 / (2*8) = 16 per gpu
+        parser.opt_list('--batch_size', default=256*8, type=int, options=[32, 64, 128, 256], tunable=False,
+                        help='batch size will be divided over all the gpus being used across all nodes')
        return parser
@@ -1,28 +1,37 @@
-"""
-Multi-node example (GPU)
-"""
 import os
+import sys
 import numpy as np
 from time import sleep
 import torch

 from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
-from pytorch_lightning import Trainer
+from pytorch_lightning.models.trainer import Trainer
+from pytorch_lightning.utils.arg_parse import add_default_args
+
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint

-from examples.new_project_templates.lightning_module_template import LightningTemplateModel
-
-PORT = np.random.randint(12000, 20000, 1)[0]
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)

+# ---------------------
+# DEFINE MODEL HERE
+# ---------------------
+from lightning_module_template import LightningTemplateModel
+# ---------------------
+
+"""
+Allows training by using command line arguments
+Run by:
+# TYPE YOUR RUN COMMAND HERE
+"""
+

 def main_local(hparams):
    main(hparams, None, None)


-def main(hparams, cluster):
+def main(hparams, cluster, results_dict):
    """
    Main training routine specific for this project
    :param hparams:
@@ -48,7 +57,6 @@ def main(hparams, cluster):
        name=hyperparams.experiment_name,
        save_dir=hyperparams.test_tube_save_path,
        autosave=False,
-        version=hparams.hpc_exp_number,  # match the slurm job version number
        description='test demo'
    )

@@ -79,9 +87,10 @@ def main(hparams, cluster):
    # ------------------------
    trainer = Trainer(
        experiment=exp,
+        cluster=cluster,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
-        gpus=hparams.per_experiment_nb_gpus,
+        gpus=hparams.gpus,
        nb_gpu_nodes=hyperparams.nb_gpu_nodes
    )

@@ -100,7 +109,7 @@ def optimize_on_cluster(hyperparams):
    )

    # email for cluster coms
-    cluster.notify_job_status(email=hyperparams.email, on_done=True, on_fail=True)
+    cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)

    # configure cluster
    cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
@@ -110,36 +119,17 @@ def optimize_on_cluster(hyperparams):
    cluster.memory_mb_per_node = 0

    # any modules for code to run in env
-    cluster.add_command(f'source activate {hyperparams.conda_env}')
-
-    # set DDP master port
-    cluster.add_command(f'export MASTER_PORT={PORT}')
-
-    # OPTIONAL for debugging
-    # without these flags errors in your code will 
-    # appear to be nccl errors
-    cluster.add_command('export NCCL_DEBUG=INFO')
-    cluster.add_command('export PYTHONFAULTHANDLER=1')
-
-    # depending on your cluster config, you probably want
-    # to limit the wired connection device
-    # cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
-
-    # depending on your cluster, you might need to load
-    # the latest NCCL version
-    # cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
+    cluster.add_command('source activate lightning')

    # run only on 32GB voltas
-    cluster.add_slurm_cmd(cmd='constraint', value='volta32gb',
-                          comment='use 32gb gpus')
-    cluster.add_slurm_cmd(cmd='partition', value=hyperparams.gpu_partition,
-                          comment='use 32gb gpus')
+    cluster.add_slurm_cmd(cmd='constraint', value='volta32gb', comment='use 32gb gpus')
+    cluster.add_slurm_cmd(cmd='partition', value=hyperparams.gpu_partition, comment='use 32gb gpus')

    # run hopt
    # creates and submits jobs to slurm
    cluster.optimize_parallel_cluster_gpu(
        main,
-        nb_trials=hyperparams.num_hyperparam_trials,
+        nb_trials=hyperparams.nb_hopt_trials,
        job_name=hyperparams.experiment_name
    )

@@ -157,28 +147,19 @@ if __name__ == '__main__':
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)

    # cluster args not defined inside the model
-
-    parent_parser.add_argument('--per_experiment_nb_gpus', type=int,
-                               default=8, help='how many gpus to use in a node')
-    parent_parser.add_argument('--nb_gpu_nodes', type=int, default=2,
-                               help='how many nodes to use in a cluster')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
-                               help='where to save logs')
-    parent_parser.add_argument('--slurm_log_path', type=str, default=slurm_out_dir,
-                               help='where to save slurm meta')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
-                               help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
-                               help='test tube exp name')
-    parent_parser.add_argument('--num_hyperparam_trials', type=int, default=6,
-                               help='how many grid search trials to run')
-
-    parent_parser.add_argument('--email', type=str, default='add@email.com',
-                               help='email for jobs')
-    parent_parser.add_argument('--conda_env', type=str, default='base',
-                               help='email for jobs')
    parent_parser.add_argument('--gpu_partition', type=str, help='consult your cluster manual')

+    # TODO: make 1 param
+    parent_parser.add_argument('--per_experiment_nb_gpus', type=int, help='how many gpus to use in a node')
+    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node')
+
+    parent_parser.add_argument('--nb_gpu_nodes', type=int, default=1, help='how many nodes to use in a cluster')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
+    parent_parser.add_argument('--slurm_log_path', type=str, default=slurm_out_dir, help='where to save slurm meta')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
+    parent_parser.add_argument('--nb_hopt_trials', type=int, default=1, help='how many grid search trials to run')
+
    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
@@ -1,20 +1,24 @@
 """
-Runs a model on a single node on CPU only..
+Runs a model on a single node across N-gpus.
 """
 import os
+import sys
 import numpy as np
+from time import sleep
 import torch

-from test_tube import HyperOptArgumentParser, Experiment
-from pytorch_lightning import Trainer
-from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
+from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from pytorch_lightning.models.trainer import Trainer
+from pytorch_lightning.utils.arg_parse import add_default_args

-from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint

 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)

+from lightning_module_template import LightningTemplateModel
+

 def main(hparams):
    """
@@ -25,11 +29,14 @@ def main(hparams):
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
+    print('loading model...')
    model = LightningTemplateModel(hparams)
+    print('model built')

    # ------------------------
-    # 2 INIT EXP
+    # 2 INIT TEST TUBE EXP
    # ------------------------
+
    # init experiment
    exp = Experiment(
        name=hyperparams.experiment_name,
@@ -87,12 +94,9 @@ if __name__ == '__main__':
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)

    # gpu args
-    parent_parser.add_argument('--test_tube_save_path', type=str,
-                               default=test_tube_dir, help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str,
-                               default=checkpoint_dir, help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str,
-                               default='pt_lightning_exp_a', help='test tube exp name')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')

    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -102,5 +106,5 @@ if __name__ == '__main__':
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
-    print('RUNNING ON CPU')
+    print(f'RUNNING ON CPU')
    main(hyperparams)
@@ -1,20 +1,24 @@
 """
-16-bit single node, CPU example
+Runs a model on a single node across N-gpus.
 """
 import os
+import sys
 import numpy as np
+from time import sleep
 import torch

-from test_tube import HyperOptArgumentParser, Experiment
-from pytorch_lightning import Trainer
-from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
+from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from pytorch_lightning.models.trainer import Trainer
+from pytorch_lightning.utils.arg_parse import add_default_args

-from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint

 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)

+from lightning_module_template import LightningTemplateModel
+

 def main(hparams):
    """
@@ -92,15 +96,10 @@ if __name__ == '__main__':
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)

    # gpu args
-    parent_parser.add_argument('--gpus', type=str, default='-1',
-                               help='how many gpus to use in the node.'
-                                    'value -1 uses all the gpus on the node')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
-                               help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
-                               help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
-                               help='test tube exp name')
+    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node. -1 uses all the gpus on the node')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')

    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -1,20 +1,24 @@
 """
-Runs a model on a single node across N-gpus using dataParallel
+Runs a model on a single node across N-gpus.
 """
 import os
+import sys
 import numpy as np
+from time import sleep
 import torch

-from test_tube import HyperOptArgumentParser, Experiment
-from pytorch_lightning import Trainer
-from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
+from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from pytorch_lightning.models.trainer import Trainer
+from pytorch_lightning.utils.arg_parse import add_default_args

-from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint

 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)

+from lightning_module_template import LightningTemplateModel
+

 def main(hparams):
    """
@@ -71,7 +75,6 @@ def main(hparams):
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
        gpus=hparams.gpus,
-        distributed_backend=hparams.dist_backend,
    )

    # ------------------------
@@ -92,17 +95,10 @@ if __name__ == '__main__':
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)

    # gpu args
-    parent_parser.add_argument('--gpus', type=str, default='-1',
-                               help='how many gpus to use in the node.'
-                                    ' value -1 uses all the gpus on the node')
-    parent_parser.add_argument('--dist_backend', type=str, default='dp',
-                               help='When using multiple GPUs set Trainer(distributed_backend=dp) (or ddp)')
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
-                               help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
-                               help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
-                               help='test tube exp name')
+    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node. -1 uses all the gpus on the node')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')

    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -2,19 +2,23 @@
 Runs a model on a single node across N-gpus.
 """
 import os
+import sys
 import numpy as np
+from time import sleep
 import torch

-from test_tube import HyperOptArgumentParser, Experiment
-from pytorch_lightning import Trainer
-from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
+from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from pytorch_lightning.models.trainer import Trainer
+from pytorch_lightning.utils.arg_parse import add_default_args

-from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint

 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)

+from lightning_module_template import LightningTemplateModel
+

 def main(hparams):
    """
@@ -71,7 +75,6 @@ def main(hparams):
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
        gpus=hparams.gpus,
-        distributed_backend=hparams.dist_backend
    )

    # ------------------------
@@ -92,17 +95,10 @@ if __name__ == '__main__':
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)

    # gpu args
-    parent_parser.add_argument('--gpus', type=str, default='-1',
-                               help='how many gpus to use in the node.'
-                                    ' value -1 uses all the gpus on the node')
-    parent_parser.add_argument('--dist_backend', type=str, default='ddp',
-                                help='When using multiple GPUs set Trainer(distributed_backend=dp) (or ddp)')     
-    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
-                               help='where to save logs')
-    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
-                               help='where to save model')
-    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
-                               help='test tube exp name')
+    parent_parser.add_argument('--gpus', type=str, default='-1', help='how many gpus to use in the node. -1 uses all the gpus on the node')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')

    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
@@ -0,0 +1,112 @@
+"""
+Runs a model on a single node across N-gpus.
+"""
+import os
+import sys
+import numpy as np
+from time import sleep
+import torch
+
+from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
+from pytorch_lightning.models.trainer import Trainer
+from pytorch_lightning.utils.arg_parse import add_default_args
+
+from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
+
+SEED = 2334
+torch.manual_seed(SEED)
+np.random.seed(SEED)
+
+from lightning_module_template import LightningTemplateModel
+
+
+def main(hparams):
+    """
+    Main training routine specific for this project
+    :param hparams:
+    :return:
+    """
+    # ------------------------
+    # 1 INIT LIGHTNING MODEL
+    # ------------------------
+    print('loading model...')
+    model = LightningTemplateModel(hparams)
+    print('model built')
+
+    # ------------------------
+    # 2 INIT TEST TUBE EXP
+    # ------------------------
+
+    # init experiment
+    exp = Experiment(
+        name=hyperparams.experiment_name,
+        save_dir=hyperparams.test_tube_save_path,
+        autosave=False,
+        description='test demo'
+    )
+
+    exp.argparse(hparams)
+    exp.save()
+
+    # ------------------------
+    # 3 DEFINE CALLBACKS
+    # ------------------------
+    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
+    early_stop = EarlyStopping(
+        monitor='val_acc',
+        patience=3,
+        verbose=True,
+        mode='max'
+    )
+
+    checkpoint = ModelCheckpoint(
+        filepath=model_save_path,
+        save_best_only=True,
+        verbose=True,
+        monitor='val_loss',
+        mode='min'
+    )
+
+    # ------------------------
+    # 4 INIT TRAINER
+    # ------------------------
+    trainer = Trainer(
+        experiment=exp,
+        checkpoint_callback=checkpoint,
+        early_stop_callback=early_stop,
+        gpus=hparams.gpus,
+    )
+
+    # ------------------------
+    # 5 START TRAINING
+    # ------------------------
+    trainer.fit(model)
+
+
+if __name__ == '__main__':
+
+    # dirs
+    root_dir = os.path.dirname(os.path.realpath(__file__))
+    demo_log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
+    checkpoint_dir = os.path.join(demo_log_dir, 'model_weights')
+    test_tube_dir = os.path.join(demo_log_dir, 'test_tube_data')
+
+    # although we user hyperOptParser, we are using it only as argparse right now
+    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
+
+    # gpu args
+    parent_parser.add_argument('--gpus', type=str, default='0', help='how many gpus to use in the node. -1 uses all the gpus on the node')
+    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir, help='where to save logs')
+    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir, help='where to save model')
+    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a', help='test tube exp name')
+
+    # allow model to overwrite or extend args
+    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
+    hyperparams = parser.parse_args()
+
+    # ---------------------
+    # RUN TRAINING
+    # ---------------------
+    # run on HPC cluster
+    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
+    main(hyperparams)
@@ -2,11 +2,10 @@ import os
 import sys

 from test_tube import HyperOptArgumentParser, Experiment
-from pytorch_lightning import Trainer
-from pytorch_lightning.utilities.arg_parse import add_default_args
+from pytorch_lightning.models.trainer import Trainer
+from pytorch_lightning.utils.arg_parse import add_default_args
 from pytorch_lightning.callbacks.pt_callbacks import EarlyStopping, ModelCheckpoint
-
-from examples.new_project_templates.lightning_module_template import LightningTemplateModel
+from docs.source.examples.example_model import ExampleModel


 def main(hparams):
@@ -29,7 +28,7 @@ def main(hparams):
    exp.save()

    # build model
-    model = LightningTemplateModel(hparams)
+    model = ExampleModel(hparams)

    # callbacks
    early_stop = EarlyStopping(
@@ -67,7 +66,7 @@ if __name__ == '__main__':
    add_default_args(parent_parser, root_dir)

    # allow model to overwrite or extend args
-    parser = LightningTemplateModel.add_model_specific_args(parent_parser)
+    parser = ExampleModel.add_model_specific_args(parent_parser)
    hyperparams = parser.parse_args()

    # train model
@@ -0,0 +1 @@
+from .trainer import Trainer
@@ -0,0 +1,888 @@
+"""
+The trainer handles all the logic for running a val loop, training loop, distributing, etc...
+"""
+import subprocess
+import traceback
+import warnings
+import os
+import pdb
+import re
+
+import torch
+from torch.utils.data.distributed import DistributedSampler
+import torch.multiprocessing as mp
+import torch.distributed as dist
+import numpy as np
+import tqdm
+
+from pytorch_lightning.root_module.memory import get_gpu_memory_map
+from pytorch_lightning.root_module.model_saving import TrainerIO
+from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel, LightningDataParallel
+from pytorch_lightning.utils.debugging import MisconfigurationException
+
+try:
+    from apex import amp
+    APEX_AVAILABLE = True
+except Exception:
+    APEX_AVAILABLE = False
+
+
+def reduce_distributed_output(output, nb_gpus):
+    if nb_gpus <= 1:
+        return output
+
+    # when using DP, we get one output per gpu
+    # average outputs and return
+    if type(output) is torch.Tensor:
+        return output.mean()
+
+    for k, v in output.items():
+        # recurse on nested dics
+        if isinstance(output[k], dict):
+            output[k] = reduce_distributed_output(output[k], nb_gpus)
+
+        # reduce only metrics that have the same nb of gpus
+        elif output[k].size(0) == nb_gpus:
+            reduced = torch.mean(output[k])
+            output[k] = reduced
+    return output
+
+
+class Trainer(TrainerIO):
+
+    def __init__(self,
+                 experiment,
+                 early_stop_callback=None,
+                 checkpoint_callback=None,
+                 gradient_clip=0,
+                 cluster=None,
+                 process_position=0,
+                 current_gpu_name=0,
+                 nb_gpu_nodes=1,
+                 gpus=None,
+                 progress_bar=True,
+                 overfit_pct=0.0,
+                 track_grad_norm=-1,
+                 check_val_every_n_epoch=1,
+                 fast_dev_run=False,
+                 accumulate_grad_batches=1,
+                 max_nb_epochs=1000, min_nb_epochs=1,
+                 train_percent_check=1.0, val_percent_check=1.0, test_percent_check=1.0,
+                 val_check_interval=0.95,
+                 log_save_interval=100, add_log_row_interval=10,
+                 distributed_backend='dp',
+                 use_amp=False,
+                 print_nan_grads=False,
+                 print_weights_summary=True,
+                 amp_level='O2',
+                 nb_sanity_val_steps=5):
+
+        """
+
+        :param experiment: Test-tube experiment
+        :param early_stop_callback: from pytorch_lightning import EarlyStopping
+        :param checkpoint_callback: from pytorch_lightning import Checkpoint
+        :param gradient_clip:
+        :param cluster:
+        :param process_position:
+        :param current_gpu_name:
+        :param nb_gpu_nodes:
+        :param gpus:
+        :param progress_bar:
+        :param overfit_pct:
+        :param track_grad_norm:
+        :param check_val_every_n_epoch:
+        :param fast_dev_run:
+        :param accumulate_grad_batches:
+        :param max_nb_epochs:
+        :param min_nb_epochs:
+        :param train_percent_check:
+        :param val_percent_check:
+        :param test_percent_check:
+        :param val_check_interval:
+        :param log_save_interval:
+        :param add_log_row_interval:
+        :param distributed_backend: 'np' to use DistributedParallel, 'ddp' to use DistributedDataParallel
+        :param use_amp:
+        :param print_nan_grads:
+        :param print_weights_summary:
+        :param amp_level:
+        :param nb_sanity_val_steps:
+        """
+
+        # Transfer params
+
+        self.nb_gpu_nodes = nb_gpu_nodes
+        self.gradient_clip = gradient_clip
+        self.check_val_every_n_epoch = check_val_every_n_epoch
+        self.enable_early_stop = early_stop_callback is not None
+        self.track_grad_norm = track_grad_norm
+        self.fast_dev_run = fast_dev_run
+        self.on_gpu = gpus is not None and torch.cuda.is_available()
+        self.progress_bar = progress_bar
+        self.experiment = experiment
+        self.exp_save_path = experiment.get_data_path(experiment.name, experiment.version)
+        self.cluster = cluster
+        self.process_position = process_position
+        self.current_gpu_name = current_gpu_name
+        self.print_weights_summary = print_weights_summary
+        self.checkpoint_callback = checkpoint_callback
+
+        if self.checkpoint_callback is not None:
+            self.checkpoint_callback.save_function = self.save_checkpoint
+
+        self.early_stop = early_stop_callback
+        self.model = None
+        self.max_nb_epochs = max_nb_epochs
+        self.accumulate_grad_batches = accumulate_grad_batches
+        self.early_stop_callback = early_stop_callback
+        self.min_nb_epochs = min_nb_epochs
+        self.nb_sanity_val_steps = nb_sanity_val_steps
+        self.lr_schedulers = []
+        self.amp_level = amp_level
+        self.print_nan_grads = print_nan_grads
+        self.data_parallel_device_ids = None
+        self.world_size = 1
+        self.node_rank = 0
+        self.use_ddp = False
+        self.use_dp = False
+
+        # training bookeeping
+        self.total_batch_nb = 0
+        self.running_loss = []
+        self.avg_loss = 0
+        self.batch_nb = 0
+        self.tqdm_metrics = {}
+        self.nb_val_batches = None
+        self.nb_tng_batches = None
+        self.nb_test_batches = None
+
+        # gpus come in as a string.
+        # if gpus = -1 then use all available devices
+        # otherwise, split the string using commas
+        if gpus is not None:
+            if type(gpus) is list:
+                self.data_parallel_device_ids = gpus
+            elif type(gpus) is str:
+                if gpus == '-1':
+                    self.data_parallel_device_ids = list(range(0, torch.cuda.device_count()))
+                else:
+                    self.data_parallel_device_ids = [int(x.strip()) for x in gpus.split(',')]
+            else:
+                raise Exception('gpus has to be a string or list of ids')
+
+            # set the correct cuda visible devices (using pci order)
+            os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+            os.environ["CUDA_VISIBLE_DEVICES"] = ','.join([str(x) for x in self.data_parallel_device_ids])
+            print(f'VISIBLE GPUS: {os.environ["CUDA_VISIBLE_DEVICES"]}')
+
+        # make DP and DDP mutually exclusive
+        # single GPU will also use DP with devices=[0]
+        have_gpus = self.data_parallel_device_ids is not None and len(self.data_parallel_device_ids) > 0
+        if have_gpus:
+            self.use_dp = distributed_backend == 'dp'
+            self.use_ddp = distributed_backend == 'ddp'
+
+            # use ddp automatically if nb_gpu_nodes > 1
+            if nb_gpu_nodes > 1 and self.use_dp:  # pragma: no cover
+                self.use_ddp = True
+                self.use_dp = False
+                w = 'DataParallel does not support nb_gpu_nodes > 1. ' \
+                    'Switching to DistributedDataParallel for you. ' \
+                    'To silence this warning set distributed_backend=ddp'
+                warnings.warn(w)
+
+        # extract SLURM flag vars
+        # whenever we have the correct number of tasks, we let slurm manage processes
+        # otherwise we launch the required number of processes
+        if self.use_ddp:
+            self.nb_requested_gpus = len(self.data_parallel_device_ids) * self.nb_gpu_nodes
+            self.nb_slurm_tasks = 0
+            try:
+                self.nb_slurm_tasks = int(os.environ['SLURM_NTASKS'])
+                self.is_slurm_managing_tasks = self.nb_slurm_tasks == self.nb_requested_gpus
+            except Exception as e:
+                # likely not on slurm, so set the slurm managed flag to false
+                self.is_slurm_managing_tasks = False
+
+        # process info
+        self.proc_rank = 0
+
+        # training state
+        self.optimizers = None
+        self.prog_bar = None
+        self.global_step = 0
+        self.current_epoch = 0
+        self.total_batches = 0
+
+        # logging
+        self.log_save_interval = log_save_interval
+        self.val_check_interval = val_check_interval
+        self.add_log_row_interval = add_log_row_interval
+
+        # dataloaders
+        self.tng_dataloader = None
+        self.test_dataloader = None
+        self.val_dataloader = None
+
+        # how much of the data to use
+        self.__determine_data_use_amount(train_percent_check, val_percent_check, test_percent_check, overfit_pct)
+        print('gpu available: {}, used: {}'.format(torch.cuda.is_available(), self.on_gpu))
+
+        # 16 bit mixed precision training using apex
+        self.use_amp = use_amp and APEX_AVAILABLE
+        if self.use_amp:
+            print('using 16bit precision')
+
+        if use_amp and not APEX_AVAILABLE:  # pragma: no cover
+            msg = '''
+            You set use_amp=True but do not have apex installed.
+            Install apex first using this guide and rerun with use_amp=True: 
+            https://github.com/NVIDIA/apex#linux
+            
+            this run will NOT use 16 bit precision
+            '''
+            raise ModuleNotFoundError(msg)
+
+    @property
+    def data_parallel(self):
+        return self.use_dp or self.use_ddp
+
+    def __determine_data_use_amount(self, train_percent_check, val_percent_check, test_percent_check, overfit_pct):
+        """
+        Use less data for debugging purposes
+        """
+        self.train_percent_check = train_percent_check
+        self.val_percent_check = val_percent_check
+        self.test_percent_check = test_percent_check
+        if overfit_pct > 0:
+            self.train_percent_check = overfit_pct
+            self.val_percent_check = overfit_pct
+            self.test_percent_check = overfit_pct
+
+    def __get_model(self):
+        return self.model.module if self.data_parallel else self.model
+
+    def __is_function_implemented(self, f_name):
+        model = self.__get_model()
+        f_op = getattr(model, f_name, None)
+        return callable(f_op)
+
+    @property
+    def __tng_tqdm_dic(self):
+        # ForkedPdb().set_trace()
+        tqdm_dic = {
+            'tng_loss': '{0:.3f}'.format(self.avg_loss),
+            'v_nb': '{}'.format(self.experiment.version),
+            'epoch': '{}'.format(self.current_epoch),
+            'batch_nb':'{}'.format(self.batch_nb),
+        }
+        tqdm_dic.update(self.tqdm_metrics)
+
+        if self.on_gpu:
+            tqdm_dic['gpu'] = '{}'.format(self.current_gpu_name)
+
+        return tqdm_dic
+
+    @property
+    def tng_tqdm_dic(self):
+        """
+        Read-only for tqdm metrics
+        :return:
+        """
+        return self.__tng_tqdm_dic
+
+    def __layout_bookeeping(self):
+
+        # determine number of training batches
+        self.nb_tng_batches = len(self.tng_dataloader)
+        self.nb_tng_batches = int(self.nb_tng_batches * self.train_percent_check)
+
+        # determine number of validation batches
+        self.nb_val_batches = len(self.val_dataloader)
+        self.nb_val_batches = int(self.nb_val_batches * self.val_percent_check)
+        self.nb_val_batches = max(1, self.nb_val_batches)
+        self.nb_val_batches = self.nb_val_batches
+
+        # determine number of test batches
+        self.nb_test_batches = len(self.test_dataloader)
+        self.nb_test_batches = int(self.nb_test_batches * self.test_percent_check)
+
+        # determine when to check validation
+        self.val_check_batch = int(self.nb_tng_batches * self.val_check_interval)
+
+    def __add_tqdm_metrics(self, metrics):
+        for k, v in metrics.items():
+            if type(v) is torch.Tensor:
+                v = v.item()
+
+            self.tqdm_metrics[k] = v
+
+    def validate(self, model, dataloader, max_batches):
+        """
+        Run validation code
+        :param model: PT model
+        :param dataloader: PT dataloader
+        :param max_batches: Scalar
+        :return:
+        """
+        # enable eval mode
+        model.zero_grad()
+        model.eval()
+
+        # disable gradients to save memory
+        torch.set_grad_enabled(False)
+
+        # bookkeeping
+        outputs = []
+
+        # run training
+        for batch_i, data_batch in enumerate(dataloader):
+
+            if data_batch is None:  # pragma: no cover
+                continue
+
+            # stop short when on fast dev run
+            if max_batches is not None and batch_i >= max_batches:
+                break
+
+            # -----------------
+            # RUN VALIDATION STEP
+            # -----------------
+            if self.use_ddp:
+                output = model(data_batch, batch_i)
+            elif self.use_dp:
+                output = model(data_batch, batch_i)
+                output = reduce_distributed_output(output, len(self.data_parallel_device_ids))
+
+            else:
+                output = model.validation_step(data_batch, batch_i)
+
+            outputs.append(output)
+
+            # batch done
+            if self.progress_bar and self.prog_bar is not None:
+                self.prog_bar.update(1)
+
+        # give model a chance to do something with the outputs
+        if self.data_parallel:
+            val_results = model.module.validation_end(outputs)
+        else:
+            val_results = model.validation_end(outputs)
+
+        # enable train mode again
+        model.train()
+
+        # enable gradients to save memory
+        torch.set_grad_enabled(True)
+
+        return val_results
+
+    def get_dataloaders(self, model):
+        """
+        Dataloaders are provided by the model
+        :param model:
+        :return:
+        """
+        self.tng_dataloader = model.tng_dataloader
+        self.test_dataloader = model.test_dataloader
+        self.val_dataloader = model.val_dataloader
+
+        if self.use_ddp and not isinstance(self.tng_dataloader.sampler, DistributedSampler):
+            msg = '''
+            when using multiple gpus and multiple nodes you must pass a DistributedSampler to DataLoader(sampler).
+            
+            ie: this:
+            dataset = myDataset()
+            dataloader = Dataloader(dataset)
+            
+            becomes:
+            dataset = myDataset()
+            dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+            dataloader = Dataloader(dataset, sampler=dist_sampler)
+            '''
+            raise MisconfigurationException(msg)
+
+    # -----------------------------
+    # MODEL TRAINING
+    # -----------------------------
+    def fit(self, model):
+
+        # when using multi-node or DDP within a node start each module in a separate process
+        if self.use_ddp:
+            # must copy only the meta of the exp so it survives pickle/unpickle when going to new process
+            self.experiment = self.experiment.get_meta_copy()
+
+            if self.is_slurm_managing_tasks:
+                task = int(os.environ['SLURM_LOCALID'])
+                self.ddp_train(task, model)
+            else:
+                msg = f"""
+                You requested {self.nb_requested_gpus} GPUs but launched {self.nb_slurm_tasks} slurm tasks. 
+                We will launch {self.nb_requested_gpus} processes for you. 
+                We recommend you let slurm manage the processes by setting: --ntasks-per-node={self.nb_requested_gpus}
+                If you're not using SLURM, ignore this message!
+                """
+                warnings.warn(msg)
+                mp.spawn(self.ddp_train, nprocs=len(self.data_parallel_device_ids), args=(model, ))
+
+        # 1 gpu or dp option triggers training using DP module
+        # easier to avoid NCCL issues
+        elif self.use_dp:
+            self.__dp_train(model)
+
+        # ON CPU
+        else:
+            # run through amp wrapper
+            if self.use_amp:
+                raise MisconfigurationException('amp + cpu is not supported. Please use a GPU option')
+
+            # CHOOSE OPTIMIZER
+            # allow for lr schedulers as well
+            self.optimizers = model.configure_optimizers()
+            if len(self.optimizers) == 2:
+                self.optimizers, self.lr_schedulers = self.optimizers
+
+            self.__run_pretrain_routine(model)
+
+        # return 1 when finished
+        # used for testing or when we need to know that training succeeded
+        return 1
+
+    def __dp_train(self, model):
+
+        # CHOOSE OPTIMIZER
+        # allow for lr schedulers as well
+        self.optimizers = model.configure_optimizers()
+        if len(self.optimizers) == 2:
+            self.optimizers, self.lr_schedulers = self.optimizers
+
+        model.cuda(self.data_parallel_device_ids[0])
+
+        # check for this bug (amp + dp + !01 doesn't work)
+        # https://github.com/NVIDIA/apex/issues/227
+        if self.use_dp and self.use_amp:
+            m = f'amp level {self.amp_level} with DataParallel is not supported. ' \
+                f'See this note from NVIDIA for more info: https://github.com/NVIDIA/apex/issues/227. ' \
+                f'We recommend you switch to ddp if you want to use amp'
+            raise MisconfigurationException(m)
+
+        model = LightningDataParallel(model, device_ids=self.data_parallel_device_ids)
+
+        self.__run_pretrain_routine(model)
+
+    def ddp_train(self, gpu_nb, model):
+        """
+        Entry point into a DP thread
+        :param gpu_nb:
+        :param model:
+        :param cluster_obj:
+        :return:
+        """
+        # node rank using relative slurm id
+        # otherwise default to node rank 0
+        try:
+            node_id = os.environ['SLURM_NODEID']
+            self.node_rank = int(node_id)
+        except Exception as e:
+            self.node_rank = 0
+
+        # recover original exp before went into process
+        # init in write mode only on proc 0
+        self.experiment.debug = self.proc_rank > 0
+        self.experiment = self.experiment.get_non_ddp_exp()
+
+        # show progbar only on prog_rank 0
+        self.prog_bar = self.prog_bar and self.node_rank == 0 and gpu_nb == 0
+
+        # determine which process we are and world size
+        self.proc_rank = self.node_rank * len(self.data_parallel_device_ids) + gpu_nb
+        self.world_size = self.nb_gpu_nodes * len(self.data_parallel_device_ids)
+
+        # let the exp know the rank to avoid overwriting logs
+        self.experiment.rank = self.proc_rank
+
+        # set up server using proc 0's ip address
+        # try to init for 20 times at max in case ports are taken
+        # where to store ip_table
+        self.__init_tcp_connection()
+
+        # CHOOSE OPTIMIZER
+        # allow for lr schedulers as well
+        self.optimizers = model.configure_optimizers()
+        if len(self.optimizers) == 2:
+            self.optimizers, self.lr_schedulers = self.optimizers
+
+        # MODEL
+        # copy model to each gpu
+        torch.cuda.set_device(gpu_nb)
+        model.cuda(gpu_nb)
+
+        # AMP
+        # run through amp wrapper before going to distributed DP
+        if self.use_amp:
+            # An example
+            model, optimizers = amp.initialize(
+                model, self.optimizers, opt_level=self.amp_level,
+            )
+            self.optimizers = optimizers
+
+        model = LightningDistributedDataParallel(model, device_ids=[gpu_nb], find_unused_parameters=True)
+
+        # continue training routine
+        self.__run_pretrain_routine(model)
+
+    def __init_tcp_connection(self):
+        """
+        Connect all procs in the world using the env:// init
+        Use the first node as the root address
+        :param port:
+        :param tries:
+        :return:
+        """
+        # sets the appropriate port
+        try:
+            port = os.environ['MASTER_PORT']
+        except Exception as e:
+            port = 12910
+            os.environ['MASTER_PORT'] = f'{port}'
+
+        # figure out the root node addr
+        try:
+            root_node = os.environ['SLURM_NODELIST'].split(' ')[0]
+        except Exception as e:
+            root_node = '127.0.0.2'
+
+        root_node = self.resolve_root_node_address(root_node)
+        os.environ['MASTER_ADDR'] = root_node
+
+        dist.init_process_group("nccl", rank=self.proc_rank, world_size=self.world_size)
+
+    def resolve_root_node_address(self, root_node):
+        if '[' in root_node:
+            name = root_node.split('[')[0]
+            number = root_node.split(',')[0]
+            if '-' in number:
+                number = number.split('-')[0]
+
+            number = re.sub('[^0-9]', '', number)
+            root_node = name + number
+
+        return root_node
+
+    def __run_pretrain_routine(self, model):
+        """
+        Sanity check a few things before starting actual training
+        :param model:
+        :return:
+        """
+        ref_model = model
+        if self.data_parallel:
+            ref_model = model.module
+
+        ref_model.trainer = self
+
+        # set local properties on the model
+        ref_model.on_gpu = self.on_gpu
+
+        # transfer data loaders from model
+        self.get_dataloaders(ref_model)
+
+        # init training constants
+        self.__layout_bookeeping()
+
+        # print model summary
+        if self.proc_rank == 0 and self.print_weights_summary:
+            ref_model.summarize()
+
+        # give model convenience properties
+        ref_model.trainer = self
+        ref_model.experiment = self.experiment
+
+        # run tiny validation to make sure program won't crash during val
+        _ = self.validate(model, self.val_dataloader, max_batches=self.nb_sanity_val_steps)
+
+        # save exp to get started
+        if self.proc_rank == 0:
+            self.experiment.save()
+
+        # track model now.
+        # if cluster resets state, the model will update with the saved weights
+        self.model = model
+
+        # enable cluster checkpointing
+        # also restores training state
+        if self.cluster is not None:  # pragma: no cover
+            self.enable_auto_hpc_walltime_manager()
+
+        # ---------------------------
+        # CORE TRAINING LOOP
+        # ---------------------------
+        self.__train()
+
+    def __train(self):
+        # run all epochs
+        for epoch_nb in range(self.current_epoch, self.max_nb_epochs):
+            # update the lr scheduler
+            if self.lr_schedulers is not None:
+                for lr_scheduler in self.lr_schedulers:
+                    lr_scheduler.step()
+
+            model = self.__get_model()
+            model.current_epoch = epoch_nb
+
+            # hook
+            if self.__is_function_implemented('on_epoch_start'):
+                model = self.__get_model()
+                model.on_epoch_start()
+
+            self.current_epoch = epoch_nb
+            self.total_batches = self.nb_tng_batches + self.nb_val_batches
+            self.batch_loss_value = 0  # accumulated grads
+
+            # init progbar when requested
+            if self.progress_bar:
+                self.prog_bar = tqdm.tqdm(range(self.total_batches), position=self.process_position)
+
+            for batch_nb, data_batch in enumerate(self.tng_dataloader):
+                self.batch_nb = batch_nb
+                self.global_step += 1
+
+                model = self.__get_model()
+                model.global_step = self.global_step
+
+                # stop when the flag is changed or we've gone past the amount requested in the batches
+                self.total_batch_nb += 1
+                met_batch_limit = batch_nb > self.nb_tng_batches
+                if met_batch_limit:
+                    break
+
+                # ---------------
+                # RUN TRAIN STEP
+                # ---------------
+                batch_result = self.__run_tng_batch(data_batch, batch_nb)
+                early_stop_epoch = batch_result == -1
+
+                # ---------------
+                # RUN VAL STEP
+                # ---------------
+                is_val_check_batch = (batch_nb + 1) % self.val_check_batch == 0
+                if self.fast_dev_run or is_val_check_batch or early_stop_epoch:
+                    self.__run_validation()
+
+                # when batch should be saved
+                if (batch_nb + 1) % self.log_save_interval == 0 or early_stop_epoch:
+                    if self.proc_rank == 0:
+                        self.experiment.save()
+
+                # when metrics should be logged
+                if batch_nb % self.add_log_row_interval == 0 or early_stop_epoch:
+                    # count items in memory
+                    # nb_params, nb_tensors = count_mem_items()
+
+                    model = self.__get_model()
+                    metrics = self.__tng_tqdm_dic
+
+                    # add gpu memory
+                    if self.on_gpu:
+                        mem_map = get_gpu_memory_map()
+                        metrics.update(mem_map)
+
+                    # add norms
+                    if self.track_grad_norm > 0:
+                        model = self.__get_model()
+                        grad_norm_dic = model.grad_norm(self.track_grad_norm)
+                        metrics.update(grad_norm_dic)
+
+                    if self.__is_function_implemented('on_tng_metrics'):
+                        model.on_tng_metrics(metrics)
+
+                    # log metrics
+                    scalar_metrics = self.__metrics_to_scalars(metrics, blacklist=self.__log_vals_blacklist())
+                    if self.proc_rank == 0:
+                        self.experiment.log(scalar_metrics, global_step=self.global_step)
+                        self.experiment.save()
+
+                # hook
+                if self.__is_function_implemented('on_batch_end'):
+                    model = self.__get_model()
+                    model.on_batch_end()
+
+                # end epoch early
+                if early_stop_epoch:
+                    break
+
+            # hook
+            if self.__is_function_implemented('on_epoch_end'):
+                model = self.__get_model()
+                model.on_epoch_end()
+
+            # early stopping
+            met_min_epochs = epoch_nb > self.min_nb_epochs
+            if self.enable_early_stop and met_min_epochs:
+                should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch_nb, logs=self.__tng_tqdm_dic)
+
+                # stop training
+                stop = should_stop and met_min_epochs
+                if stop:
+                    return
+
+    def __metrics_to_scalars(self, metrics, blacklist=[]):
+        new_metrics = {}
+        for k, v in metrics.items():
+            if type(v) is torch.Tensor:
+                v = v.item()
+
+            if type(v) is dict:
+                v = self.__metrics_to_scalars(v)
+
+            if k not in blacklist:
+                new_metrics[k] = float(v)
+
+        return new_metrics
+
+    def __log_vals_blacklist(self):
+        """avoid logging some vals lightning uses to maintain state"""
+        blacklist = {'batch_nb', 'v_nb', 'gpu'}
+        return blacklist
+
+    def __run_tng_batch(self, data_batch, batch_nb):
+        if data_batch is None:
+            return 0
+
+        # hook
+        if self.__is_function_implemented('on_batch_start'):
+            model_ref = self.__get_model()
+            response = model_ref.on_batch_start(data_batch)
+
+            if response == -1:
+                return -1
+
+        if self.progress_bar:
+            self.prog_bar.update(1)
+
+        # forward pass
+        # return a scalar value and a dic with tqdm metrics
+        if self.use_ddp:
+            output = self.model(data_batch, batch_nb)
+        elif self.use_dp:
+            output = self.model(data_batch, batch_nb)
+            output = reduce_distributed_output(output, len(self.data_parallel_device_ids))
+        else:
+            output = self.model.training_step(data_batch, batch_nb)
+
+        try:
+            model_specific_tqdm_metrics_dic = output['prog']
+        except Exception as e:
+            model_specific_tqdm_metrics_dic = {}
+
+        # if output dict doesn't have the keyword loss
+        # then assume the output=loss if scalar
+        try:
+            loss = output['loss']
+        except Exception as e:
+            if type(output) is torch.Tensor:
+                loss = output
+
+        self.__add_tqdm_metrics(model_specific_tqdm_metrics_dic)
+
+        # backward pass
+        if self.use_amp:
+            # scale loss when using amp
+            for optimizer in self.optimizers:
+                with amp.scale_loss(loss, optimizer) as scaled_loss:
+                    scaled_loss.backward()
+        else:
+            loss.backward()
+
+        # insert after step hook
+        if self.__is_function_implemented('on_after_backward'):
+            model_ref = self.__get_model()
+            response = model_ref.on_after_backward()
+
+        if self.print_nan_grads:
+            model = self.__get_model()
+            for param in model.parameters():
+                print(param.grad.float().sum())
+
+        # avoid memory leaks
+        self.batch_loss_value += loss.item()
+
+        # gradient update with accumulated gradients
+        if (self.batch_nb + 1) % self.accumulate_grad_batches == 0:
+
+            # clip gradients
+            if self.gradient_clip > 0:
+                model = self.__get_model()
+                torch.nn.utils.clip_grad_norm(model.parameters(), self.gradient_clip)
+
+            # update gradients across all optimizers
+            for optimizer in self.optimizers:
+                optimizer.step()
+
+                # insert after step hook
+                if self.__is_function_implemented('on_before_zero_grad'):
+                    model_ref = self.__get_model()
+                    response = model_ref.on_before_zero_grad(optimizer)
+
+                # clear gradients
+                optimizer.zero_grad()
+
+            # queuing loss across batches blows it up proportionally... divide out the number accumulated
+            self.batch_loss_value = self.batch_loss_value / self.accumulate_grad_batches
+
+            # track loss
+            self.running_loss.append(self.batch_loss_value)
+            self.batch_loss_value = 0
+            self.avg_loss = np.mean(self.running_loss[-100:])
+
+            # update progbar
+            if self.progress_bar:
+                # add model specific metrics
+                tqdm_metrics = self.__tng_tqdm_dic
+                self.prog_bar.set_postfix(**tqdm_metrics)
+
+        # activate batch end hook
+        if self.__is_function_implemented('on_batch_end'):
+            model = self.__get_model()
+            model.on_batch_end()
+
+        return 0
+
+    def __run_validation(self):
+        # decide if can check epochs
+        can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
+        if self.fast_dev_run:
+            print('skipping to check performance bc of --fast_dev_run')
+        elif not can_check_epoch:
+            return
+
+        # hook
+        if self.__is_function_implemented('on_pre_performance_check'):
+            model = self.__get_model()
+            model.on_pre_performance_check()
+
+        # use full val set on end of epoch
+        # use a small portion otherwise
+        max_batches = None if not self.fast_dev_run else 1
+        model_specific_tqdm_metrics_dic = self.validate(
+            self.model,
+            self.val_dataloader,
+            max_batches
+        )
+        self.__add_tqdm_metrics(model_specific_tqdm_metrics_dic)
+
+        # hook
+        if self.__is_function_implemented('on_post_performance_check'):
+            model = self.__get_model()
+            model.on_post_performance_check()
+
+        if self.progress_bar:
+            # add model specific metrics
+            tqdm_metrics = self.__tng_tqdm_dic
+            self.prog_bar.set_postfix(**tqdm_metrics)
+
+        # model checkpointing
+        if self.proc_rank == 0 and self.checkpoint_callback is not None:
+            print('save callback...')
+            self.checkpoint_callback.on_epoch_end(epoch=self.current_epoch, logs=self.__tng_tqdm_dic)
@@ -6,6 +6,7 @@ from itertools import chain
 import threading
 import torch
 from torch.cuda._utils import _get_device_index
+import pdb


 def _find_tensors(obj):  # pragma: no cover
@@ -56,8 +57,6 @@ class LightningDataParallel(DataParallel):
            # lightning
            if self.module.training:
                return self.module.training_step(*inputs[0], **kwargs[0])
-            elif self.module.testing:
-                return self.module.test_step(*inputs[0], **kwargs[0])
            else:
                return self.module.validation_step(*inputs[0], **kwargs[0])

@@ -65,6 +64,7 @@ class LightningDataParallel(DataParallel):
        outputs = self.parallel_apply(replicas, inputs, kwargs)
        return self.gather(outputs, self.output_device)

+
    def parallel_apply(self, replicas, inputs, kwargs):
        return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])

@@ -91,8 +91,6 @@ class LightningDistributedDataParallel(DistributedDataParallel):
                # lightning
                if self.module.training:
                    output = self.module.training_step(*inputs[0], **kwargs[0])
-                elif self.module.testing:
-                    output = self.module.test_step(*inputs[0], **kwargs[0])
                else:
                    output = self.module.validation_step(*inputs[0], **kwargs[0])
            else:
@@ -157,10 +155,6 @@ def parallel_apply(modules, inputs, kwargs_tup=None, devices=None):  # pragma: n
                # CHANGE
                if module.training:
                    output = module.training_step(*input, **kwargs)
-
-                elif module.testing:
-                    output = module.test_step(*input, **kwargs)
-
                else:
                    output = module.validation_step(*input, **kwargs)
                # ---------------
@@ -1,5 +1,3 @@
-import traceback
-

 def data_loader(fn):
    """
@@ -12,17 +10,8 @@ def data_loader(fn):

    @property
    def _data_loader(self):
-        try:
-            value = getattr(self, attr_name)
-        except AttributeError:
-            try:
-                value = fn(self)  # Lazy evaluation, done only once.
-            except AttributeError as e:
-                # Guard against AttributeError suppression. (Issue #142)
-                traceback.print_exc()
-                error = f'{fn.__name__}: An AttributeError was encountered: ' + str(e)
-                raise RuntimeError(error) from e
-            setattr(self, attr_name, value)  # Memoize evaluation.
-        return value
+        if not hasattr(self, attr_name):
+            setattr(self, attr_name, fn(self))
+        return getattr(self, attr_name)

    return _data_loader
@@ -1,9 +1,10 @@
+import numpy as np
+from torch import nn
+
 """
 Module to describe gradients
 """

-from torch import nn
-

 class GradInformation(nn.Module):

@@ -17,13 +18,12 @@ class GradInformation(nn.Module):
                    total_norm += param_norm ** norm_type
                    norm = param_norm ** (1 / norm_type)

-                    grad = round(norm.data.cpu().numpy().flatten()[0], 3)
-                    results['grad_{}_norm_{}'.format(norm_type, i)] = grad
-                except Exception:
+                    results['grad_{}_norm_{}'.format(norm_type, i)] = round(norm.data.cpu().numpy().flatten()[0], 3)
+                except Exception as e:
                    # this param had no grad
                    pass

        total_norm = total_norm ** (1. / norm_type)
-        grad = round(total_norm.data.cpu().numpy().flatten()[0], 3)
-        results['grad_{}_norm_total'.format(norm_type)] = grad
+        results['grad_{}_norm_total'.format(norm_type)] = round(total_norm.data.cpu().numpy().flatten()[0], 3)
        return results
+
@@ -1,16 +1,7 @@
 import torch

-
 class ModelHooks(torch.nn.Module):
-
-    def on_sanity_check_start(self):
-        """
-        Called before starting evaluate
-        :return:
-        """
-        pass
-
-    def on_batch_start(self, batch):
+    def on_batch_start(self, data_batch):
        pass

    def on_batch_end(self):
@@ -28,7 +19,7 @@ class ModelHooks(torch.nn.Module):
    def on_post_performance_check(self):
        pass

-    def on_training_metrics(self, metrics):
+    def on_tng_metrics(self, metrics):
        pass

    def on_before_zero_grad(self, optimizer):
@@ -51,3 +42,4 @@ class ModelHooks(torch.nn.Module):
        :return:
        """
        pass
+
@@ -1,15 +1,15 @@
-'''
-Generates a summary of a model's layers and dimensionality
-'''
-
-import gc
-
 import torch
+import gc
 import subprocess
 import numpy as np
 import pandas as pd


+'''
+Generates a summary of a model's layers and dimensionality
+'''
+
+
 class ModelSummary(object):

    def __init__(self, model):
@@ -94,7 +94,7 @@ class ModelSummary(object):
        mods = list(self.model.modules())
        sizes = []

-        for i in range(1, len(mods)):
+        for i in range(1,len(mods)):
            m = mods[i]
            p = list(m.parameters())
            modsz = []
@@ -127,7 +127,7 @@ class ModelSummary(object):
        if self.model.example_input_array is not None:
            cols.extend(['In_sizes', 'Out_sizes'])

-        df = pd.DataFrame(np.zeros((len(self.layer_names), len(cols))))
+        df = pd.DataFrame(np.zeros( (len(self.layer_names), len(cols))))
        df.columns = cols

        df['Name'] = self.layer_names
@@ -152,16 +152,16 @@ class ModelSummary(object):
        self.make_summary()


-def print_mem_stack():  # pragma: no cover
+def print_mem_stack(): # pragma: no cover
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                print(type(obj), obj.size())
-        except Exception:
+        except Exception as e:
            pass


-def count_mem_items():  # pragma: no cover
+def count_mem_items(): # pragma: no cover
    nb_params = 0
    nb_tensors = 0
    for obj in gc.get_objects():
@@ -172,7 +172,7 @@ def count_mem_items():  # pragma: no cover
                    nb_params += 1
                else:
                    nb_tensors += 1
-        except Exception:
+        except Exception as e:
            pass

    return nb_params, nb_tensors
@@ -196,6 +196,6 @@ def get_gpu_memory_map():
    gpu_memory = [int(x) for x in result.strip().split('\n')]
    gpu_memory_map = {}
    for k, v in zip(range(len(gpu_memory)), gpu_memory):
-        k = 'gpu_%i' % k
+        k = f'gpu_{k}'
        gpu_memory_map[k] = v
    return gpu_memory_map
@@ -1,3 +1,10 @@
+import torch
+import os
+import re
+import pdb
+from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel, LightningDataParallel
+
+
 class ModelIO(object):

    def on_load_checkpoint(self, checkpoint):
@@ -32,3 +39,201 @@ class ModelIO(object):
        :return:
        """
        pass
+
+
+class TrainerIO(object):
+
+    def __get_model(self):
+        is_dp_module = type(self.model) is LightningDistributedDataParallel or type(self.model) is LightningDataParallel
+        model = self.model.module if is_dp_module else self.model
+        return model
+
+    # --------------------
+    # MODEL SAVE CHECKPOINT
+    # --------------------
+    def save_checkpoint(self, filepath):
+        checkpoint = self.dump_checkpoint()
+
+        # do the actual save
+        torch.save(checkpoint, filepath)
+
+    def dump_checkpoint(self):
+
+        checkpoint = {
+            'epoch': self.current_epoch,
+            'global_step': self.global_step
+        }
+
+        if self.checkpoint_callback is not None:
+            checkpoint['checkpoint_callback_best'] = self.checkpoint_callback.best
+
+        if self.early_stop_callback is not None:
+            checkpoint['early_stop_callback_wait'] = self.early_stop_callback.wait
+            checkpoint['early_stop_callback_patience'] = self.early_stop_callback.patience
+
+        # save optimizers
+        optimizer_states = []
+        for i, optimizer in enumerate(self.optimizers):
+            optimizer_states.append(optimizer.state_dict())
+
+        checkpoint['optimizer_states'] = optimizer_states
+        
+        # save lr schedulers
+        lr_schedulers = []
+        for i, scheduler in enumerate(self.lr_schedulers):
+            lr_schedulers.append(scheduler.state_dict())
+
+        checkpoint['lr_schedulers'] = lr_schedulers
+
+        # add the state_dict from the model
+        model = self.__get_model()
+        checkpoint['state_dict'] = model.state_dict()
+
+        # give the model a chance to add a few things
+        model.on_save_checkpoint(checkpoint)
+
+        return checkpoint
+
+    # --------------------
+    # HPC IO
+    # --------------------
+    def enable_auto_hpc_walltime_manager(self):
+        if self.cluster is None:
+            return
+
+        # allow test tube to handle model check pointing automatically
+        self.cluster.set_checkpoint_save_function(
+            self.hpc_save,
+            kwargs={
+                'folderpath': self.checkpoint_callback.filepath,
+                'experiment': self.experiment
+            }
+        )
+        self.cluster.set_checkpoint_load_function(
+            self.hpc_load,
+            kwargs={
+                'folderpath': self.checkpoint_callback.filepath,
+                'on_gpu': self.on_gpu
+            }
+        )
+
+    def restore_training_state(self, checkpoint):
+        """
+        Restore trainer state.
+        Model will get its change to update
+        :param checkpoint:
+        :return:
+        """
+        if self.checkpoint_callback is not None:
+            self.checkpoint_callback.best = checkpoint['checkpoint_callback_best']
+
+        if self.early_stop_callback is not None:
+            self.early_stop_callback.wait = checkpoint['early_stop_callback_wait']
+            self.early_stop_callback.patience = checkpoint['early_stop_callback_patience']
+
+        self.global_step = checkpoint['global_step']
+        self.current_epoch = checkpoint['epoch']
+
+        # restore the optimizers
+        optimizer_states = checkpoint['optimizer_states']
+        for optimizer, opt_state in zip(self.optimizers, optimizer_states):
+            optimizer.load_state_dict(opt_state)
+        
+        # restore the lr schedulers
+        lr_schedulers = checkpoint['lr_schedulers']
+        for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
+            scheduler.load_state_dict(lrs_state)
+
+    # ----------------------------------
+    # PRIVATE OPS
+    # ----------------------------------
+    def hpc_save(self, folderpath, experiment):
+        # make sure the checkpoint folder exists
+        os.makedirs(folderpath, exist_ok=True)
+
+        # save exp to make sure we get all the metrics
+        experiment.save()
+
+        # close experiment to avoid issues
+        experiment.close()
+
+        ckpt_number = self.max_ckpt_in_folder(folderpath) + 1
+
+        if not os.path.exists(folderpath):
+            os.makedirs(folderpath, exist_ok=True)
+        filepath = '{}/hpc_ckpt_{}.ckpt'.format(folderpath, ckpt_number)
+
+        # give model a chance to do something on hpc_save
+        model = self.__get_model()
+        checkpoint = self.dump_checkpoint()
+
+        model.on_hpc_save(checkpoint)
+
+        # do the actual save
+        torch.save(checkpoint, filepath)
+
+        return filepath
+
+    def hpc_load(self, folderpath, on_gpu):
+        filepath = '{}/hpc_ckpt_{}.ckpt'.format(folderpath, self.max_ckpt_in_folder(folderpath))
+
+        if on_gpu:
+            checkpoint = torch.load(filepath)
+        else:
+            checkpoint = torch.load(filepath, map_location=lambda storage, loc: storage)
+
+        # load training state (affects trainer only)
+        self.restore_training_state(checkpoint)
+
+        # load model state
+        model = self.__get_model()
+
+        # load the state_dict on the model automatically
+        model.load_state_dict(checkpoint['state_dict'])
+
+        # call model hook
+        model.on_hpc_load(checkpoint)
+
+    def max_ckpt_in_folder(self, path):
+        files = os.listdir(path)
+        files = [x for x in files if 'ckpt_' in x]
+        if len(files) == 0:
+            return 0
+
+        ckpt_vs = []
+        for name in files:
+            name = name.split('ckpt_')[-1]
+            name = re.sub('[^0-9]', '', name)
+            ckpt_vs.append(int(name))
+
+        return max(ckpt_vs)
+
+
+def load_hparams_from_tags_csv(tags_csv):
+    from argparse import Namespace
+    import pandas as pd
+
+    tags_df = pd.read_csv(tags_csv)
+    dic = tags_df.to_dict(orient='records')
+
+    ns_dict = {row['key']: convert(row['value']) for row in dic}
+
+    ns = Namespace(**ns_dict)
+    return ns
+
+
+def convert(val):
+    constructors = [int, float, str]
+
+    if type(val) is str:
+        if val.lower() == 'true':
+            return True
+        if val.lower() == 'false':
+            return False
+
+    for c in constructors:
+        try:
+            return c(val)
+        except ValueError:
+            pass
+    return val
@@ -1,9 +1,7 @@
 import torch
-
 from pytorch_lightning.root_module.memory import ModelSummary
 from pytorch_lightning.root_module.grads import GradInformation
-from pytorch_lightning.trainer.trainer_io import load_hparams_from_tags_csv
-from pytorch_lightning.root_module.model_saving import ModelIO
+from pytorch_lightning.root_module.model_saving import ModelIO, load_hparams_from_tags_csv
 from pytorch_lightning.root_module.hooks import ModelHooks
 from pytorch_lightning.root_module.decorators import data_loader

@@ -24,9 +22,6 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):

        # track if gpu was requested for checkpointing
        self.on_gpu = False
-        self.use_dp = False
-        self.use_ddp = False
-        self.use_amp = False

    def forward(self, *args, **kwargs):
        """
@@ -37,52 +32,29 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        raise NotImplementedError

-    def training_step(self, *args, **kwargs):
+    def validation_step(self, data_batch, batch_nb):
        """
-        return loss, dict with metrics for tqdm
-        :param called with batch, batch_nb
-        additional: optimizer_i if multiple optimizers used
+        return whatever outputs will need to be aggregated in validation_end
+        :param data_batch:
        :return:
        """
        raise NotImplementedError

-    def validation_step(self, *args, **kwargs):
-        """
-        return whatever outputs will need to be aggregated in validation_end
-        OPTIONAL
-        :param called with batch, batch_nb
-        additional: dataset_i if multiple val datasets used
-        :return:
-        """
-        pass
-
-    def test_step(self, *args, **kwargs):
-        """
-        return whatever outputs will need to be aggregated in test_end
-        OPTIONAL
-        :param called with batch, batch_nb
-        additional: dataset_i if multiple val datasets used
-        :return:
-        """
-        pass
-
    def validation_end(self, outputs):
        """
        Outputs has the appended output after each validation step
-        OPTIONAL
        :param outputs:
        :return: dic_with_metrics for tqdm
        """
-        pass
+        raise NotImplementedError

-    def test_end(self, outputs):
+    def training_step(self, data_batch, batch_nb):
        """
-        Outputs has the appended output after each test step
-        OPTIONAL
-        :param outputs:
-        :return: dic_with_metrics for tqdm
+        return loss, dict with metrics for tqdm
+        :param data_batch:
+        :return:
        """
-        pass
+        raise NotImplementedError

    def configure_optimizers(self):
        """
@@ -91,24 +63,10 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        """
        raise NotImplementedError

-    def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i):
-        """
-        Do something instead of the standard optimizer behavior
-        :param epoch_nb:
-        :param batch_nb:
-        :param optimizer:
-        :param optimizer_i:
-        :return:
-        """
-        optimizer.step()
-
-        # clear gradients
-        optimizer.zero_grad()
-
    @data_loader
-    def train_dataloader(self):
+    def tng_dataloader(self):
        """
-        Implement a PyTorch DataLoader
+        Implement a function to load an h5py of this data
        :return:
        """
        raise NotImplementedError
@@ -116,21 +74,21 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
    @data_loader
    def test_dataloader(self):
        """
-        Implement a PyTorch DataLoader
+        Implement a function to load an h5py of this data
        :return:
        """
-        return None
+        raise NotImplementedError

    @data_loader
    def val_dataloader(self):
        """
-        Implement a PyTorch DataLoader
+        Implement a function to load an h5py of this data
        :return:
        """
-        return None
+        raise NotImplementedError

    @classmethod
-    def load_from_metrics(cls, weights_path, tags_csv, on_gpu):
+    def load_from_metrics(cls, weights_path, tags_csv, on_gpu, map_location=None):
        """
        Primary way of loading model from csv weights path
        :param weights_path:
@@ -142,9 +100,13 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
        hparams = load_hparams_from_tags_csv(tags_csv)
        hparams.__setattr__('on_gpu', on_gpu)

-        # load on CPU only to avoid OOM issues
-        # then its up to user to put back on GPUs
-        checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)
+        if on_gpu:
+            if map_location is not None:
+                checkpoint = torch.load(weights_path, map_location=map_location)
+            else:
+                checkpoint = torch.load(weights_path)
+        else:
+            checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)

        # load the state_dict on the model automatically
        model = cls(hparams)
@@ -166,3 +128,6 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
    def unfreeze(self):
        for param in self.parameters():
            param.requires_grad = True
+
+
+
@@ -1,12 +0,0 @@
-from .lm_test_module import LightningTestModel
-from .lm_test_module_base import LightningTestModelBase
-from .lm_test_module_mixins import (
-    LightningValidationStepMixin,
-    LightningValidationMixin,
-    LightningValidationStepMultipleDataloadersMixin,
-    LightningValidationMultipleDataloadersMixin,
-    LightningTestStepMixin,
-    LightningTestMixin,
-    LightningTestStepMultipleDataloadersMixin,
-    LightningTestMultipleDataloadersMixin,
-)
@@ -1,27 +0,0 @@
-import os
-from collections import OrderedDict
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-from torch import optim
-from torch.utils.data import DataLoader
-from torch.utils.data.distributed import DistributedSampler
-from torchvision.datasets import MNIST
-from torchvision import transforms
-from test_tube import HyperOptArgumentParser
-
-from pytorch_lightning.root_module.root_module import LightningModule
-from pytorch_lightning import data_loader
-
-from .lm_test_module_base import LightningTestModelBase
-from .lm_test_module_mixins import LightningValidationMixin, LightningTestMixin
-
-
-class LightningTestModel(LightningValidationMixin, LightningTestMixin, LightningTestModelBase):
-    """
-    Most common test case. Validation and test dataloaders
-    """
-
-    def on_training_metrics(self, logs):
-        logs['some_tensor_to_test'] = torch.rand(1)
@@ -1,387 +0,0 @@
-import os
-from collections import OrderedDict
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-from torch import optim
-from torch.utils.data import DataLoader
-from torch.utils.data.distributed import DistributedSampler
-from torchvision.datasets import MNIST
-from torchvision import transforms
-from test_tube import HyperOptArgumentParser
-
-from pytorch_lightning.root_module.root_module import LightningModule
-from pytorch_lightning import data_loader
-
-
-class LightningValidationStepMixin:
-    """
-    Add val_dataloader and validation_step methods for the case
-    when val_dataloader returns a single dataloader
-    """
-
-    @data_loader
-    def val_dataloader(self):
-        return self._dataloader(train=False)
-
-    def validation_step(self, batch, batch_idx):
-        """
-        Lightning calls this inside the validation loop
-        :param batch:
-        :return:
-        """
-        x, y = batch
-        x = x.view(x.size(0), -1)
-        y_hat = self.forward(x)
-
-        loss_val = self.loss(y, y_hat)
-
-        # acc
-        labels_hat = torch.argmax(y_hat, dim=1)
-        val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-        val_acc = torch.tensor(val_acc)
-
-        if self.on_gpu:
-            val_acc = val_acc.cuda(loss_val.device.index)
-
-        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp:
-            loss_val = loss_val.unsqueeze(0)
-            val_acc = val_acc.unsqueeze(0)
-
-        # alternate possible outputs to test
-        if batch_idx % 1 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-            })
-            return output
-        if batch_idx % 2 == 0:
-            return val_acc
-
-        if batch_idx % 3 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-                'test_dic': {'val_loss_a': loss_val}
-            })
-            return output
-
-
-class LightningValidationMixin(LightningValidationStepMixin):
-    """
-    Add val_dataloader, validation_step, and validation_end methods for the case
-    when val_dataloader returns a single dataloader
-    """
-
-    def validation_end(self, outputs):
-        """
-        Called at the end of validation to aggregate outputs
-        :param outputs: list of individual outputs of each validation step
-        :return:
-        """
-        # if returned a scalar from validation_step, outputs is a list of tensor scalars
-        # we return just the average in this case (if we want)
-        # return torch.stack(outputs).mean()
-        val_loss_mean = 0
-        val_acc_mean = 0
-        for output in outputs:
-            val_loss = output['val_loss']
-
-            # reduce manually when using dp
-            if self.trainer.use_dp:
-                val_loss = torch.mean(val_loss)
-            val_loss_mean += val_loss
-
-            # reduce manually when using dp
-            val_acc = output['val_acc']
-            if self.trainer.use_dp:
-                val_acc = torch.mean(val_acc)
-
-            val_acc_mean += val_acc
-
-        val_loss_mean /= len(outputs)
-        val_acc_mean /= len(outputs)
-
-        tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-        return tqdm_dict
-
-
-class LightningValidationStepMultipleDataloadersMixin:
-    """
-    Add val_dataloader and validation_step methods for the case
-    when val_dataloader returns multiple dataloaders
-    """
-
-    @data_loader
-    def val_dataloader(self):
-        return [self._dataloader(train=False), self._dataloader(train=False)]
-
-    def validation_step(self, batch, batch_idx, dataloader_idx):
-        """
-        Lightning calls this inside the validation loop
-        :param batch:
-        :return:
-        """
-        x, y = batch
-        x = x.view(x.size(0), -1)
-        y_hat = self.forward(x)
-
-        loss_val = self.loss(y, y_hat)
-
-        # acc
-        labels_hat = torch.argmax(y_hat, dim=1)
-        val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-        val_acc = torch.tensor(val_acc)
-
-        if self.on_gpu:
-            val_acc = val_acc.cuda(loss_val.device.index)
-
-        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp:
-            loss_val = loss_val.unsqueeze(0)
-            val_acc = val_acc.unsqueeze(0)
-
-        # alternate possible outputs to test
-        if batch_idx % 1 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-            })
-            return output
-        if batch_idx % 2 == 0:
-            return val_acc
-
-        if batch_idx % 3 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-                'test_dic': {'val_loss_a': loss_val}
-            })
-            return output
-        if batch_idx % 5 == 0:
-            output = OrderedDict({
-                f'val_loss_{dataloader_idx}': loss_val,
-                f'val_acc_{dataloader_idx}': val_acc,
-            })
-            return output
-
-
-class LightningValidationMultipleDataloadersMixin(LightningValidationStepMultipleDataloadersMixin):
-    """
-    Add val_dataloader, validation_step, and validation_end methods for the case
-    when val_dataloader returns multiple dataloaders
-    """
-
-    def validation_end(self, outputs):
-        """
-        Called at the end of validation to aggregate outputs
-        :param outputs: list of individual outputs of each validation step
-        :return:
-        """
-        # if returned a scalar from validation_step, outputs is a list of tensor scalars
-        # we return just the average in this case (if we want)
-        # return torch.stack(outputs).mean()
-        val_loss_mean = 0
-        val_acc_mean = 0
-        i = 0
-        for dl_output in outputs:
-            for output in dl_output:
-                val_loss = output['val_loss']
-
-                # reduce manually when using dp
-                if self.trainer.use_dp:
-                    val_loss = torch.mean(val_loss)
-                val_loss_mean += val_loss
-
-                # reduce manually when using dp
-                val_acc = output['val_acc']
-                if self.trainer.use_dp:
-                    val_acc = torch.mean(val_acc)
-
-                val_acc_mean += val_acc
-                i += 1
-
-        val_loss_mean /= i
-        val_acc_mean /= i
-
-        tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-        return tqdm_dict
-
-
-class LightningTestStepMixin:
-
-    @data_loader
-    def test_dataloader(self):
-        return self._dataloader(train=False)
-
-    def test_step(self, batch, batch_idx):
-        """
-        Lightning calls this inside the validation loop
-        :param batch:
-        :return:
-        """
-        x, y = batch
-        x = x.view(x.size(0), -1)
-        y_hat = self.forward(x)
-
-        loss_test = self.loss(y, y_hat)
-
-        # acc
-        labels_hat = torch.argmax(y_hat, dim=1)
-        test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-        test_acc = torch.tensor(test_acc)
-
-        if self.on_gpu:
-            test_acc = test_acc.cuda(loss_test.device.index)
-
-        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp:
-            loss_test = loss_test.unsqueeze(0)
-            test_acc = test_acc.unsqueeze(0)
-
-        # alternate possible outputs to test
-        if batch_idx % 1 == 0:
-            output = OrderedDict({
-                'test_loss': loss_test,
-                'test_acc': test_acc,
-            })
-            return output
-        if batch_idx % 2 == 0:
-            return test_acc
-
-        if batch_idx % 3 == 0:
-            output = OrderedDict({
-                'test_loss': loss_test,
-                'test_acc': test_acc,
-                'test_dic': {'test_loss_a': loss_test}
-            })
-            return output
-
-
-class LightningTestMixin(LightningTestStepMixin):
-    def test_end(self, outputs):
-        """
-        Called at the end of validation to aggregate outputs
-        :param outputs: list of individual outputs of each validation step
-        :return:
-        """
-        # if returned a scalar from test_step, outputs is a list of tensor scalars
-        # we return just the average in this case (if we want)
-        # return torch.stack(outputs).mean()
-        test_loss_mean = 0
-        test_acc_mean = 0
-        for output in outputs:
-            test_loss = output['test_loss']
-
-            # reduce manually when using dp
-            if self.trainer.use_dp:
-                test_loss = torch.mean(test_loss)
-            test_loss_mean += test_loss
-
-            # reduce manually when using dp
-            test_acc = output['test_acc']
-            if self.trainer.use_dp:
-                test_acc = torch.mean(test_acc)
-
-            test_acc_mean += test_acc
-
-        test_loss_mean /= len(outputs)
-        test_acc_mean /= len(outputs)
-
-        tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
-        return tqdm_dict
-
-
-class LightningTestStepMultipleDataloadersMixin:
-
-    @data_loader
-    def test_dataloader(self):
-        return [self._dataloader(train=False), self._dataloader(train=False)]
-
-    def test_step(self, batch, batch_idx, dataloader_idx):
-        """
-        Lightning calls this inside the validation loop
-        :param batch:
-        :return:
-        """
-        x, y = batch
-        x = x.view(x.size(0), -1)
-        y_hat = self.forward(x)
-
-        loss_test = self.loss(y, y_hat)
-
-        # acc
-        labels_hat = torch.argmax(y_hat, dim=1)
-        test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-        test_acc = torch.tensor(test_acc)
-
-        if self.on_gpu:
-            test_acc = test_acc.cuda(loss_test.device.index)
-
-        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp:
-            loss_test = loss_test.unsqueeze(0)
-            test_acc = test_acc.unsqueeze(0)
-
-        # alternate possible outputs to test
-        if batch_idx % 1 == 0:
-            output = OrderedDict({
-                'test_loss': loss_test,
-                'test_acc': test_acc,
-            })
-            return output
-        if batch_idx % 2 == 0:
-            return test_acc
-
-        if batch_idx % 3 == 0:
-            output = OrderedDict({
-                'test_loss': loss_test,
-                'test_acc': test_acc,
-                'test_dic': {'test_loss_a': loss_test}
-            })
-            return output
-        if batch_idx % 5 == 0:
-            output = OrderedDict({
-                f'test_loss_{dataloader_idx}': loss_test,
-                f'test_acc_{dataloader_idx}': test_acc,
-            })
-            return output
-
-
-class LightningTestMultipleDataloadersMixin(LightningTestStepMultipleDataloadersMixin):
-    def test_end(self, outputs):
-        """
-        Called at the end of validation to aggregate outputs
-        :param outputs: list of individual outputs of each validation step
-        :return:
-        """
-        # if returned a scalar from test_step, outputs is a list of tensor scalars
-        # we return just the average in this case (if we want)
-        # return torch.stack(outputs).mean()
-        test_loss_mean = 0
-        test_acc_mean = 0
-        i = 0
-        for dl_output in outputs:
-            for output in dl_output:
-                test_loss = output['test_loss']
-
-                # reduce manually when using dp
-                if self.trainer.use_dp:
-                    test_loss = torch.mean(test_loss)
-                test_loss_mean += test_loss
-
-                # reduce manually when using dp
-                test_acc = output['test_acc']
-                if self.trainer.use_dp:
-                    test_acc = torch.mean(test_acc)
-
-                test_acc_mean += test_acc
-                i += 1
-
-        test_loss_mean /= i
-        test_acc_mean /= i
-
-        tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
-        return tqdm_dict
@@ -1,24 +1,22 @@
 import os
 from collections import OrderedDict
-
-import torch
 import torch.nn as nn
+from torchvision.datasets import MNIST
+import torchvision.transforms as transforms
+import torch
 import torch.nn.functional as F
+from test_tube import HyperOptArgumentParser
 from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
-from torchvision.datasets import MNIST
-from torchvision import transforms
-from test_tube import HyperOptArgumentParser

 from pytorch_lightning.root_module.root_module import LightningModule
-from pytorch_lightning import data_loader
+import pytorch_lightning as ptl


-class LightningTestModelBase(LightningModule):
+class LightningTestModel(LightningModule):
    """
-    Base LightningModule for testing. Implements only the required
-    interface
+    Sample model to show how to define a template
    """

    def __init__(self, hparams, force_remove_distributed_sampler=False):
@@ -27,7 +25,7 @@ class LightningTestModelBase(LightningModule):
        :param hparams:
        """
        # init superclass
-        super(LightningTestModelBase, self).__init__()
+        super(LightningTestModel, self).__init__()
        self.hparams = hparams

        self.batch_size = hparams.batch_size
@@ -49,13 +47,11 @@ class LightningTestModelBase(LightningModule):
        Layout model
        :return:
        """
-        self.c_d1 = nn.Linear(in_features=self.hparams.in_features,
-                              out_features=self.hparams.hidden_dim)
+        self.c_d1 = nn.Linear(in_features=self.hparams.in_features, out_features=self.hparams.hidden_dim)
        self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
        self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)

-        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim,
-                              out_features=self.hparams.out_features)
+        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim, out_features=self.hparams.out_features)

    # ---------------------
    # TRAINING
@@ -81,14 +77,14 @@ class LightningTestModelBase(LightningModule):
        nll = F.nll_loss(logits, labels)
        return nll

-    def training_step(self, batch, batch_idx):
+    def training_step(self, data_batch, batch_i):
        """
        Lightning calls this inside the training loop
-        :param batch:
+        :param data_batch:
        :return:
        """
        # forward pass
-        x, y = batch
+        x, y = data_batch
        x = x.view(x.size(0), -1)

        y_hat = self.forward(x)
@@ -104,12 +100,80 @@ class LightningTestModelBase(LightningModule):
        if self.trainer.batch_nb % 1 == 0:
            output = OrderedDict({
                'loss': loss_val,
-                'progress': {'some_val': loss_val * loss_val}
+                'prog': {'some_val': loss_val * loss_val}
            })
            return output
        if self.trainer.batch_nb % 2 == 0:
            return loss_val

+    def validation_step(self, data_batch, batch_i):
+        """
+        Lightning calls this inside the validation loop
+        :param data_batch:
+        :return:
+        """
+        x, y = data_batch
+        x = x.view(x.size(0), -1)
+        y_hat = self.forward(x)
+
+        loss_val = self.loss(y, y_hat)
+
+        # acc
+        labels_hat = torch.argmax(y_hat, dim=1)
+        val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
+        val_acc = torch.tensor(val_acc)
+
+        if self.on_gpu:
+            val_acc = val_acc.cuda(loss_val.device.index)
+
+        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
+        if self.trainer.use_dp:
+            loss_val = loss_val.unsqueeze(0)
+            val_acc = val_acc.unsqueeze(0)
+
+        # alternate possible outputs to test
+        if self.trainer.batch_nb % 1 == 0:
+            output = OrderedDict({
+                'val_loss': loss_val,
+                'val_acc': val_acc,
+            })
+            return output
+        if self.trainer.batch_nb % 2 == 0:
+            return val_acc
+
+        if self.trainer.batch_nb % 3 == 0:
+            output = OrderedDict({
+                'val_loss': loss_val,
+                'val_acc': val_acc,
+                'test_dic': {'val_loss_a': loss_val}
+            })
+            return output
+
+    def validation_end(self, outputs):
+        """
+        Called at the end of validation to aggregate outputs
+        :param outputs: list of individual outputs of each validation step
+        :return:
+        """
+        # if returned a scalar from validation_step, outputs is a list of tensor scalars
+        # we return just the average in this case (if we want)
+        # return torch.stack(outputs).mean()
+
+        val_loss_mean = 0
+        val_acc_mean = 0
+        for output in outputs:
+            val_loss_mean += output['val_loss']
+            val_acc_mean += output['val_acc']
+
+        val_loss_mean /= len(outputs)
+        val_acc_mean /= len(outputs)
+
+        tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
+        return tqdm_dic
+
+    def on_tng_metrics(self, logs):
+        logs['some_tensor_to_test'] = torch.rand(1)
+
    # ---------------------
    # TRAINING SETUP
    # ---------------------
@@ -122,24 +186,22 @@ class LightningTestModelBase(LightningModule):
        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)

        # test returning only 1 list instead of 2
-        return optimizer
+        return [optimizer]

-    def _dataloader(self, train):
+    def __dataloader(self, train):
        # init data generators
-        transform = transforms.Compose([transforms.ToTensor(),
-                                        transforms.Normalize((0.5,), (1.0,))])
-        dataset = MNIST(root=self.hparams.data_root, train=train,
-                        transform=transform, download=True)
+        transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
+        dataset = MNIST(root=self.hparams.data_root, train=train, transform=transform, download=True)

        # when using multi-node we need to add the datasampler
        train_sampler = None
        batch_size = self.hparams.batch_size

        try:
-            if self.use_ddp and not self.force_remove_distributed_sampler:
+            if self.on_gpu and not self.force_remove_distributed_sampler:
                train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
                batch_size = batch_size // self.trainer.world_size  # scale batch size
-        except Exception:
+        except Exception as e:
            pass

        should_shuffle = train_sampler is None
@@ -152,12 +214,20 @@ class LightningTestModelBase(LightningModule):

        return loader

-    @data_loader
-    def train_dataloader(self):
-        return self._dataloader(train=True)
+    @ptl.data_loader
+    def tng_dataloader(self):
+        return self.__dataloader(train=True)
+
+    @ptl.data_loader
+    def val_dataloader(self):
+        return self.__dataloader(train=False)
+
+    @ptl.data_loader
+    def test_dataloader(self):
+        return self.__dataloader(train=False)

    @staticmethod
-    def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
+    def add_model_specific_args(parent_parser, root_dir):
        """
        Parameters you define here will be available to your model through self.hparams
        :param parent_parser:
@@ -167,28 +237,23 @@ class LightningTestModelBase(LightningModule):
        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])

        # param overwrites
-        # parser.set_defaults(gradient_clip_val=5.0)
+        # parser.set_defaults(gradient_clip=5.0)

        # network params
        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
-        parser.add_argument('--in_features', default=28 * 28, type=int)
+        parser.add_argument('--in_features', default=28*28, type=int)
        parser.add_argument('--out_features', default=10, type=int)
-        # use 500 for CPU, 50000 for GPU to see speed difference
-        parser.add_argument('--hidden_dim', default=50000, type=int)
+        parser.add_argument('--hidden_dim', default=50000, type=int) # use 500 for CPU, 50000 for GPU to see speed difference

        # data
        parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)

        # training params (opt)
-        parser.opt_list('--learning_rate', default=0.001 * 8, type=float,
-                        options=[0.0001, 0.0005, 0.001, 0.005],
+        parser.opt_list('--learning_rate', default=0.001*8, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
                        tunable=False)
-        parser.opt_list('--optimizer_name', default='adam', type=str,
-                        options=['adam'], tunable=False)
+        parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)

-        # if using 2 nodes with 4 gpus each the batch size here
-        #  (256) will be 256 / (2*8) = 16 per gpu
-        parser.opt_list('--batch_size', default=256 * 8, type=int,
-                        options=[32, 64, 128, 256], tunable=False,
-                        help='batch size will be divided over all gpus being used across all nodes')
+        # if using 2 nodes with 4 gpus each the batch size here (256) will be 256 / (2*8) = 16 per gpu
+        parser.opt_list('--batch_size', default=256*8, type=int, options=[32, 64, 128, 256], tunable=False,
+                        help='batch size will be divided over all the gpus being used across all nodes')
        return parser
@@ -1,325 +0,0 @@
-import os
-import re
-import signal
-import pdb
-from subprocess import call
-
-import torch
-
-from pytorch_lightning.pt_overrides.override_data_parallel import (
-    LightningDistributedDataParallel, LightningDataParallel)
-
-
-class TrainerIO(object):
-
-    def __get_model(self):
-        is_dp_module = isinstance(self.model, (LightningDistributedDataParallel,
-                                               LightningDataParallel))
-        model = self.model.module if is_dp_module else self.model
-        return model
-
-    # --------------------
-    # CHECK-POINTING
-    # --------------------
-    def restore_weights(self, model):
-        """
-        To restore weights we have two cases.
-        First, if we use the same experiment version, then restore the latest ckpt.
-        AFTER that, if we find weights from hpc checkpoint, then restore that.
-        :param model:
-        :return:
-        """
-        # restore weights if same exp version
-        self.restore_state_if_checkpoint_exists(model)
-
-        # if script called from hpc resubmit, load weights
-        self.restore_hpc_weights_if_needed(model)
-
-    def restore_state_if_checkpoint_exists(self, model):
-        # do nothing if there's not dir or callback
-        no_ckpt_callback = self.checkpoint_callback is None
-        if no_ckpt_callback or not os.path.exists(self.checkpoint_callback.filepath):
-            return
-
-        # restore trainer state and model if there is a weight for this experiment
-        last_epoch = -1
-        last_ckpt_name = None
-
-        # find last epoch
-        checkpoints = os.listdir(self.checkpoint_callback.filepath)
-        for name in checkpoints:
-            # ignore hpc ckpts
-            if 'hpc_' in name:
-                continue
-
-            if '.ckpt' in name:
-                epoch = name.split('epoch_')[1]
-                epoch = int(re.sub('[^0-9]', '', epoch))
-
-                if epoch > last_epoch:
-                    last_epoch = epoch
-                    last_ckpt_name = name
-
-        # restore last checkpoint
-        if last_ckpt_name is not None:
-            last_ckpt_path = os.path.join(self.checkpoint_callback.filepath, last_ckpt_name)
-            self.restore(last_ckpt_path, self.on_gpu)
-            print(f'model and trainer restored from checkpoint: {last_ckpt_path}')
-
-    # --------------------
-    # HPC SIGNAL HANDLING
-    # --------------------
-    def register_slurm_signal_handlers(self):
-        # see if we're using slurm (not interactive)
-        on_slurm = False
-        try:
-            job_name = os.environ['SLURM_JOB_NAME']
-            if job_name != 'bash':
-                on_slurm = True
-        except Exception as e:
-            pass
-
-        if on_slurm:
-            print('set slurm handle signals')
-            signal.signal(signal.SIGUSR1, self.sig_handler)
-            signal.signal(signal.SIGTERM, self.term_handler)
-
-    def sig_handler(self, signum, frame):
-        if self.proc_rank == 0:
-            # save weights
-            print('handling SIGUSR1')
-            self.hpc_save(self.weights_save_path, self.experiment)
-
-            # find job id
-            job_id = os.environ['SLURM_JOB_ID']
-            cmd = 'scontrol requeue {}'.format(job_id)
-
-            # requeue job
-            print('\nrequeing job {}...'.format(job_id))
-            result = call(cmd, shell=True)
-
-            # print result text
-            if result == 0:
-                print('requeued exp ', job_id)
-            else:
-                print('requeue failed...')
-
-            # close experiment to avoid issues
-            self.experiment.close()
-
-    def term_handler(self, signum, frame):
-        # save
-        print("bypassing sigterm")
-
-    # --------------------
-    # MODEL SAVE CHECKPOINT
-    # --------------------
-    def save_checkpoint(self, filepath):
-        checkpoint = self.dump_checkpoint()
-
-        # do the actual save
-        torch.save(checkpoint, filepath)
-
-    def restore(self, checkpoint_path, on_gpu):
-
-        # if on_gpu:
-        #     checkpoint = torch.load(checkpoint_path)
-        # else:
-        # load on CPU first
-        checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
-
-        # load model state
-        model = self.__get_model()
-
-        # load the state_dict on the model automatically
-        model.load_state_dict(checkpoint['state_dict'])
-        if on_gpu:
-            model.cuda(self.root_gpu)
-
-        # load training state (affects trainer only)
-        self.restore_training_state(checkpoint)
-
-    def dump_checkpoint(self):
-
-        checkpoint = {
-            'epoch': self.current_epoch,
-            'global_step': self.global_step
-        }
-
-        if self.checkpoint_callback is not None:
-            checkpoint['checkpoint_callback_best'] = self.checkpoint_callback.best
-
-        if self.early_stop_callback is not None:
-            checkpoint['early_stop_callback_wait'] = self.early_stop_callback.wait
-            checkpoint['early_stop_callback_patience'] = self.early_stop_callback.patience
-
-        # save optimizers
-        optimizer_states = []
-        for i, optimizer in enumerate(self.optimizers):
-            optimizer_states.append(optimizer.state_dict())
-
-        checkpoint['optimizer_states'] = optimizer_states
-
-        # save lr schedulers
-        lr_schedulers = []
-        for i, scheduler in enumerate(self.lr_schedulers):
-            lr_schedulers.append(scheduler.state_dict())
-
-        checkpoint['lr_schedulers'] = lr_schedulers
-
-        # add the state_dict from the model
-        model = self.__get_model()
-        checkpoint['state_dict'] = model.state_dict()
-
-        # give the model a chance to add a few things
-        model.on_save_checkpoint(checkpoint)
-
-        return checkpoint
-
-    # --------------------
-    # HPC IO
-    # --------------------
-    def restore_hpc_weights_if_needed(self, model):
-        """
-        If there is a set of hpc weights, use as signal to restore model
-        :param model:
-        :return:
-        """
-        # look for hpc weights
-        folderpath = self.weights_save_path
-        if os.path.exists(folderpath):
-            files = os.listdir(folderpath)
-            hpc_weight_paths = [x for x in files if 'hpc_ckpt' in x]
-
-            # if hpc weights exist restore model
-            if len(hpc_weight_paths) > 0:
-                self.hpc_load(folderpath, self.on_gpu)
-
-    def restore_training_state(self, checkpoint):
-        """
-        Restore trainer state.
-        Model will get its change to update
-        :param checkpoint:
-        :return:
-        """
-        if self.checkpoint_callback is not None:
-            self.checkpoint_callback.best = checkpoint['checkpoint_callback_best']
-
-        if self.early_stop_callback is not None:
-            self.early_stop_callback.wait = checkpoint['early_stop_callback_wait']
-            self.early_stop_callback.patience = checkpoint['early_stop_callback_patience']
-
-        self.global_step = checkpoint['global_step']
-        self.current_epoch = checkpoint['epoch']
-
-        # restore the optimizers
-        optimizer_states = checkpoint['optimizer_states']
-        for optimizer, opt_state in zip(self.optimizers, optimizer_states):
-            optimizer.load_state_dict(opt_state)
-
-            # move optimizer to GPU 1 weight at a time
-            # avoids OOM
-            if self.root_gpu is not None:
-                for state in optimizer.state.values():
-                    for k, v in state.items():
-                        if isinstance(v, torch.Tensor):
-                            state[k] = v.cuda(self.root_gpu)
-
-        # restore the lr schedulers
-        lr_schedulers = checkpoint['lr_schedulers']
-        for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
-            scheduler.load_state_dict(lrs_state)
-
-    # ----------------------------------
-    # PRIVATE OPS
-    # ----------------------------------
-    def hpc_save(self, folderpath, experiment):
-        # make sure the checkpoint folder exists
-        os.makedirs(folderpath, exist_ok=True)
-
-        # save exp to make sure we get all the metrics
-        experiment.save()
-
-        ckpt_number = self.max_ckpt_in_folder(folderpath) + 1
-
-        if not os.path.exists(folderpath):
-            os.makedirs(folderpath, exist_ok=True)
-        filepath = '{}/hpc_ckpt_{}.ckpt'.format(folderpath, ckpt_number)
-
-        # give model a chance to do something on hpc_save
-        model = self.__get_model()
-        checkpoint = self.dump_checkpoint()
-
-        model.on_hpc_save(checkpoint)
-
-        # do the actual save
-        torch.save(checkpoint, filepath)
-
-        return filepath
-
-    def hpc_load(self, folderpath, on_gpu):
-        filepath = '{}/hpc_ckpt_{}.ckpt'.format(folderpath, self.max_ckpt_in_folder(folderpath))
-
-        # load on CPU first
-        checkpoint = torch.load(filepath, map_location=lambda storage, loc: storage)
-
-        # load model state
-        model = self.__get_model()
-
-        # load the state_dict on the model automatically
-        model.load_state_dict(checkpoint['state_dict'])
-
-        if self.root_gpu is not None:
-            model.cuda(self.root_gpu)
-
-        # load training state (affects trainer only)
-        self.restore_training_state(checkpoint)
-
-        # call model hook
-        model.on_hpc_load(checkpoint)
-
-        print(f'restored hpc model from: {filepath}')
-
-    def max_ckpt_in_folder(self, path, name_key='ckpt_'):
-        files = os.listdir(path)
-        files = [x for x in files if name_key in x]
-        if len(files) == 0:
-            return 0
-
-        ckpt_vs = []
-        for name in files:
-            name = name.split(name_key)[-1]
-            name = re.sub('[^0-9]', '', name)
-            ckpt_vs.append(int(name))
-
-        return max(ckpt_vs)
-
-
-def load_hparams_from_tags_csv(tags_csv):
-    from argparse import Namespace
-    import pandas as pd
-
-    tags_df = pd.read_csv(tags_csv)
-    dic = tags_df.to_dict(orient='records')
-
-    ns_dict = {row['key']: convert(row['value']) for row in dic}
-
-    ns = Namespace(**ns_dict)
-    return ns
-
-
-def convert(val):
-    constructors = [int, float, str]
-
-    if type(val) is str:
-        if val.lower() == 'true':
-            return True
-        if val.lower() == 'false':
-            return False
-
-    for c in constructors:
-        try:
-            return c(val)
-        except ValueError:
-            pass
-    return val
@@ -1,48 +1,31 @@
-"""
-List of default args which mught be useful for all the available flags
-Might need to update with the new flags
-"""
-
-import os
-
+import pdb

 def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None):

-    # training, test, val check intervals
-    parser.add_argument('--eval_test_set', dest='eval_test_set', action='store_true',
-                        help='true = run test set also')
-    parser.add_argument('--check_val_every_n_epoch', default=1, type=int,
-                        help='check val every n epochs')
+    # tng, test, val check intervals
+    parser.add_argument('--eval_test_set', dest='eval_test_set', action='store_true', help='true = run test set also')
+    parser.add_argument('--check_val_every_n_epoch', default=1, type=int, help='check val every n epochs')
    parser.opt_list('--accumulate_grad_batches', default=1, type=int, tunable=False,
-                    help='accumulates gradients k times before applying update.'
-                         ' Simulates huge batch size')
+                    help='accumulates gradients k times before applying update. Simulates huge batch size')
    parser.add_argument('--max_nb_epochs', default=200, type=int, help='cap epochs')
    parser.add_argument('--min_nb_epochs', default=2, type=int, help='min epochs')
-    parser.add_argument('--train_percent_check', default=1.0, type=float,
-                        help='how much of training set to check')
-    parser.add_argument('--val_percent_check', default=1.0, type=float,
-                        help='how much of val set to check')
-    parser.add_argument('--test_percent_check', default=1.0, type=float,
-                        help='how much of test set to check')
+    parser.add_argument('--train_percent_check', default=1.0, type=float, help='how much of tng set to check')
+    parser.add_argument('--val_percent_check', default=1.0, type=float, help='how much of val set to check')
+    parser.add_argument('--test_percent_check', default=1.0, type=float, help='how much of test set to check')

-    parser.add_argument('--val_check_interval', default=0.95, type=float,
-                        help='how much within 1 epoch to check val')
-    parser.add_argument('--log_save_interval', default=100, type=int,
-                        help='how many batches between log saves')
-    parser.add_argument('--row_log_interval', default=100, type=int,
-                        help='add log every k batches')
+    parser.add_argument('--val_check_interval', default=0.95, type=float, help='how much within 1 epoch to check val')
+    parser.add_argument('--log_save_interval', default=100, type=int, help='how many batches between log saves')
+    parser.add_argument('--add_log_row_interval', default=100, type=int, help='add log every k batches')

    # early stopping
    parser.add_argument('--disable_early_stop', dest='enable_early_stop', action='store_false')
    parser.add_argument('--early_stop_metric', default='val_acc', type=str)
    parser.add_argument('--early_stop_mode', default='min', type=str)
-    parser.add_argument('--early_stop_patience', default=3, type=int,
-                        help='number of epochs until stop')
+    parser.add_argument('--early_stop_patience', default=3, type=int, help='number of epochs until stop')

    # gradient handling
-    parser.add_argument('--gradient_clip_val', default=-1, type=int)
-    parser.add_argument('--track_grad_norm', default=-1, type=int,
-                        help='if > 0, will track this grad norm')
+    parser.add_argument('--gradient_clip', default=-1, type=int)
+    parser.add_argument('--track_grad_norm', default=-1, type=int, help='if > 0, will track this grad norm')

    # model saving
    parser.add_argument('--model_save_path', default=root_dir + '/model_weights')
@@ -58,8 +41,7 @@ def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None
    # test_tube settings
    parser.add_argument('-en', '--tt_name', default='pt_test')
    parser.add_argument('-td', '--tt_description', default='pytorch lightning test')
-    parser.add_argument('--tt_save_path', default=os.path.join(root_dir, 'test_tube_logs'),
-                        help='logging dir')
+    parser.add_argument('--tt_save_path', default=root_dir + '/test_tube_logs', help='logging dir')
    parser.add_argument('--enable_single_run', dest='single_run', action='store_true')
    parser.add_argument('--nb_hopt_trials', default=1, type=int)
    parser.add_argument('--log_stdout', dest='log_stdout', action='store_true')
@@ -70,30 +52,25 @@ def add_default_args(parser, root_dir, rand_seed=None, possible_model_names=None
    parser.add_argument('--default_tensor_type', default='torch.cuda.FloatTensor', type=str)
    parser.add_argument('--use_amp', dest='use_amp', action='store_true')
    parser.add_argument('--check_grad_nans', dest='check_grad_nans', action='store_true')
-    parser.add_argument('--amp_level', default='O2', type=str)
+    parser.add_argument('--amp_level', default='O2',type=str)
+

    # run on hpc
    parser.add_argument('--on_cluster', dest='on_cluster', action='store_true')

    # FAST training
    # use these settings to make sure network has no bugs without running a full dataset
-    parser.add_argument('--fast_dev_run', dest='fast_dev_run', default=False, action='store_true',
-                        help='runs validation after 1 training step')
-    parser.add_argument('--enable_tqdm', dest='enable_tqdm', default=False, action='store_true',
-                        help='false removes the progress bar')
-    parser.add_argument('--overfit', default=-1, type=float,
-                        help='% of dataset to use with this option. float, or -1 for none')
+    parser.add_argument('--fast_dev_run', dest='fast_dev_run', default=False, action='store_true', help='runs validation after 1 tng step')
+    parser.add_argument('--enable_tqdm', dest='enable_tqdm', default=False, action='store_true', help='false removes the prog bar')
+    parser.add_argument('--overfit', default=-1, type=float, help='% of dataset to use with this option. float, or -1 for none')

    # debug args
    if rand_seed is not None:
        parser.add_argument('--random_seed', default=rand_seed, type=int)

-    parser.add_argument('--interactive', dest='interactive', action='store_true',
-                        help='runs on gpu without cluster')
-    parser.add_argument('--debug', dest='debug', action='store_true',
-                        help='enables/disables test tube')
-    parser.add_argument('--local', dest='local', action='store_true',
-                        help='enables local training')
+    parser.add_argument('--interactive', dest='interactive', action='store_true', help='runs on gpu without cluster')
+    parser.add_argument('--debug', dest='debug', action='store_true', help='enables/disables test tube')
+    parser.add_argument('--local', dest='local', action='store_true', help='enables local tng')

    # optimizer
-    parser.add_argument('--lr_scheduler_milestones', default=None, type=str)
+    parser.add_argument('--lr_scheduler_milestones', default=None, type=str)
@@ -1,2 +1,5 @@
+import pdb
+import sys
+
 class MisconfigurationException(Exception):
-    pass
+    pass
@@ -1,7 +1,9 @@
+coverage==4.5.3
+mkdocs==1.0.4
+pytest==5.0.1
 scikit-learn==0.20.2
 tqdm==4.32.1
 twine==1.13.0
 numpy==1.16.4
 torch>=1.1.0
-torchvision>=0.3.0
-pandas
+torchvision==0.3.0
@@ -31,8 +31,6 @@ exclude_lines =
    print(traceback.print_exc())
    return *
    raise Exception
-    raise *
-    except *
    warnings
    print
    raise RuntimeError
@@ -44,8 +42,6 @@ omit =
    pytorch_lightning/callbacks/pt_callbacks.py
    tests/test_models.py
    pytorch_lightning/testing_models/lm_test_module.py
-    pytorch_lightning/utilities/arg_parse.py
-    examples/templates

 [flake8]
 ignore = E731,W504,F401,F841
@@ -1,58 +1,29 @@
 #!/usr/bin/env python

-# Always prefer setuptools over distutils
 from setuptools import setup, find_packages

 # https://packaging.python.org/guides/single-sourcing-package-version/

 # http://blog.ionelmc.ro/2014/05/25/python-packaging/
-
-# https://packaging.python.org/discussions/install-requires-vs-requirements /
-# keep the meta-data here for simplicity in reading this file... it's not obvious
-# what happens and to non-engineers they won't know to look in init ...
-# the goal of the project is simplicity for researchers, don't want to add too much
-# engineer specific practices
 setup(
-    name='pytorch-lightning',
-    version='0.5.0',
-    description='The Keras for ML researchers using PyTorch',
-    author='William Falcon',
-    author_email='waf2107@columbia.edu',
-    url='https://github.com/williamFalcon/pytorch-lightning',
-    download_url='https://github.com/williamFalcon/pytorch-lightning',
-    license='Apache-2',
+    name="pytorch-lightning",
+    version='0.3.6.6',
+    description="The Keras for ML researchers using PyTorch",
+    author="William Falcon",
+    author_email="waf2107@columbia.edu",
+    url="https://github.com/williamFalcon/pytorch-lightning",
+    download_url="https://github.com/williamFalcon/pytorch-lightning",
+    license="MIT",
+    keywords=["deep learning", "pytorch", "AI"],
+    python_requires=">=3.5",
+    install_requires=[
+        "torch>=1.1.0",
+        "tqdm",
+        "test-tube>=0.6.7.4",
+    ],
    packages=find_packages(),
-    long_description=open('README.md', encoding='utf-8').read(),
+    long_description=open("README.md", encoding="utf-8").read(),
    long_description_content_type='text/markdown',
    include_package_data=True,
    zip_safe=False,
-    keywords=['deep learning', 'pytorch', 'AI'],
-    python_requires='>=3.6',
-    install_requires=[
-        'torch==1.2.0',
-        'tqdm>=4.35.0',
-        'test-tube>=0.6.9',
-        'pandas>=0.20.3',
-    ],
-    classifiers=[
-        'Environment :: Console',
-        'Natural Language :: English',
-        # How mature is this project? Common values are
-        #   3 - Alpha, 4 - Beta, 5 - Production/Stable
-        'Development Status :: 4 - Beta',
-        # Indicate who your project is intended for
-        'Intended Audience :: Developers',
-        'Topic :: Scientific/Engineering :: Artificial Intelligence',
-        'Topic :: Scientific/Engineering :: Image Recognition',
-        'Topic :: Scientific/Engineering :: Information Analysis',
-        # Pick your license as you wish
-        'License :: OSI Approved :: BSD License',
-        'Operating System :: OS Independent',
-        # Specify the Python versions you support here. In particular, ensure
-        # that you indicate whether you support Python 2, Python 3 or both.
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
-    ],
 )
@@ -1,4 +1,4 @@
-# PyTorch-Lightning Tests
+# Pytorch-Lightning Tests

 ## Running tests
 The automatic travis tests ONLY run CPU-based tests. Although these cover most of the use cases,
@@ -17,7 +17,7 @@ pip install -e .
 pip install -r requirements.txt

 # run tests
-py.test -v
+py.test
 ```

 To test models that require GPU make sure to run the above command on a GPU machine.
@@ -43,21 +43,16 @@ For each set up it also tests:
 5. simulated load from HPC signal.

 ## Running Coverage   
-Make sure to run coverage on a GPU machine with at least 2 GPUs and NVIDIA apex installed. 

 ```bash
 cd pytorch-lightning

 # generate coverage 
 pip install coverage
-coverage run --source pytorch_lightning -m py.test pytorch_lightning tests examples -v --doctest-modules
+coverage run tests/test_models.py   

 # print coverage stats
-coverage report -m
-
-# exporting resulys
-coverage xml
-codecov -t 17327163-8cca-4a5d-86c8-ca5f2ef700bc  -v
+coverage report -m   
 ```


@@ -1,22 +1,24 @@
+import pytest
 from pytorch_lightning import Trainer
-from examples import LightningTemplateModel
-from pytorch_lightning.testing import LightningTestModel
+from pytorch_lightning.examples.new_project_templates.lightning_module_template import LightningTemplateModel
 from argparse import Namespace
 from test_tube import Experiment
 from pytorch_lightning.callbacks import ModelCheckpoint
+import numpy as np
+import warnings
+import torch
 import os
 import shutil
+import pdb

-import pytorch_lightning as pl
+import pytorch_lightning as ptl
 import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
-import numpy as np
-import pdb


-class CoolModel(pl.LightningModule):
+class CoolModel(ptl.LightningModule):

    def __init(self):
        super(CoolModel, self).__init__()
@@ -32,7 +34,7 @@ class CoolModel(pl.LightningModule):
    def training_step(self, batch, batch_nb):
        x, y = batch
        y_hat = self.forward(x)
-        return {'training_loss': self.my_loss(y_hat, y)}
+        return {'tng_loss': self.my_loss(y_hat, y)}

    def validation_step(self, batch, batch_nb):
        x, y = batch
@@ -46,15 +48,15 @@ class CoolModel(pl.LightningModule):
    def configure_optimizers(self):
        return [torch.optim.Adam(self.parameters(), lr=0.02)]

-    @pl.data_loader
-    def train_dataloader(self):
+    @ptl.data_loader
+    def tng_dataloader(self):
        return DataLoader(MNIST('path/to/save', train=True), batch_size=32)

-    @pl.data_loader
+    @ptl.data_loader
    def val_dataloader(self):
        return DataLoader(MNIST('path/to/save', train=False), batch_size=32)

-    @pl.data_loader
+    @ptl.data_loader
    def test_dataloader(self):
        return DataLoader(MNIST('path/to/save', train=False), batch_size=32)

@@ -64,8 +66,8 @@ def get_model():
    root_dir = os.path.dirname(os.path.realpath(__file__))
    hparams = Namespace(**{'drop_prob': 0.2,
                           'batch_size': 32,
-                           'in_features': 28 * 28,
-                           'learning_rate': 0.001 * 8,
+                           'in_features': 28*28,
+                           'learning_rate': 0.001*8,
                           'optimizer_name': 'adam',
                           'data_root': os.path.join(root_dir, 'mnist'),
                           'out_features': 10,
@@ -75,11 +77,10 @@ def get_model():
    return model, hparams


-def get_exp(debug=True, version=None):
+def get_exp(debug=True):
    # set up exp object without actually saving logs
    root_dir = os.path.dirname(os.path.realpath(__file__))
-    save_dir = os.path.join(root_dir, 'save_dir')
-    exp = Experiment(debug=debug, save_dir=save_dir, name='tests_tt_dir', version=version)
+    exp = Experiment(debug=debug, save_dir=root_dir, name='tests_tt_dir')
    return exp


@@ -102,7 +103,7 @@ def clear_save_dir():
        shutil.rmtree(save_dir)


-def load_model(exp, save_dir, on_gpu, map_location=None, module_class=LightningTemplateModel):
+def load_model(exp, save_dir):

    # load trained model
    tags_path = exp.get_data_path(exp.name, exp.version)
@@ -111,10 +112,7 @@ def load_model(exp, save_dir, on_gpu, map_location=None, module_class=LightningT
    checkpoints = [x for x in os.listdir(save_dir) if '.ckpt' in x]
    weights_dir = os.path.join(save_dir, checkpoints[0])

-    trained_model = module_class.load_from_metrics(weights_path=weights_dir,
-                                                   tags_csv=tags_path,
-                                                   on_gpu=on_gpu,
-                                                   map_location=map_location)
+    trained_model = LightningTemplateModel.load_from_metrics(weights_path=weights_dir, tags_csv=tags_path, on_gpu=True)

    assert trained_model is not None, 'loading model failed'

@@ -136,160 +134,45 @@ def run_prediction(dataloader, trained_model):
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    val_acc = torch.tensor(val_acc)
    val_acc = val_acc.item()
-    assert val_acc > 0.70, 'this model is expected to get > 0.7 in test set (it got %f)' % val_acc
+
+    print(val_acc)
+
+    assert val_acc > 0.70, f'this model is expected to get > 0.7 in test set (it got {val_acc})'


-# ------------------------------------------------------------------------
-def run_gpu_model_test(trainer_options, model, hparams, on_gpu=True):
+def main():
+
    save_dir = init_save_dir()

    # exp file to get meta
    exp = get_exp(False)
-    exp.argparse(hparams)
    exp.save()

    # exp file to get weights
    checkpoint = ModelCheckpoint(save_dir)

-    # add these to the trainer options
-    trainer_options['checkpoint_callback'] = checkpoint
-    trainer_options['experiment'] = exp
+    trainer = Trainer(
+        experiment=exp,
+        checkpoint_callback=checkpoint,
+        progress_bar=True,
+        max_nb_epochs=1,
+        gpus=[0, 1],
+        distributed_backend='dp',
+    )
+
+    model = CoolModel()

-    # fit model
-    trainer = Trainer(**trainer_options)
    result = trainer.fit(model)

    # correct result and ok accuracy
    assert result == 1, 'amp + ddp model failed to complete'

    # test model loading
-    pretrained_model = load_model(exp, save_dir, on_gpu)
+    pretrained_model = load_model(exp, save_dir)

-    # test new model accuracy
+    # test model preds
    run_prediction(model.test_dataloader, pretrained_model)

-    if trainer.use_ddp:
-        # on hpc this would work fine... but need to hack it for the purpose of the test
-        trainer.model = pretrained_model
-        trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers()
-
-    # test HPC loading / saving
-    trainer.hpc_save(save_dir, exp)
-    trainer.hpc_load(save_dir, on_gpu=on_gpu)
-
-    clear_save_dir()
-
-
-def assert_ok_val_acc(trainer):
-    # this model should get 0.80+ acc
-    acc = trainer.training_tqdm_dict['val_acc']
-    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
-
-
-def assert_ok_test_acc(trainer):
-    # this model should get 0.80+ acc
-    acc = trainer.training_tqdm_dict['test_acc']
-    assert acc > 0.50, f'model failed to get expected 0.50 validation accuracy. Got: {acc}'
-
-
-def get_hparams(continue_training=False, hpc_exp_number=0):
-    root_dir = os.path.dirname(os.path.realpath(__file__))
-
-    args = {
-        'drop_prob': 0.2,
-        'batch_size': 32,
-        'in_features': 28 * 28,
-        'learning_rate': 0.001 * 8,
-        'optimizer_name': 'adam',
-        'data_root': os.path.join(root_dir, 'mnist'),
-        'out_features': 10,
-        'hidden_dim': 1000}
-
-    if continue_training:
-        args['test_tube_do_checkpoint_load'] = True
-        args['hpc_exp_number'] = hpc_exp_number
-
-    hparams = Namespace(**args)
-    return hparams
-
-
-def main():
-    """
-    Make sure DDP + AMP continue training correctly
-    :return:
-    """
-    hparams = get_hparams()
-    model = LightningTestModel(hparams)
-
-    trainer_options = dict(
-        show_progress_bar=True,
-        max_nb_epochs=4,
-        gpus=2,
-        distributed_backend='dp',
-    )
-
-    save_dir = init_save_dir()
-
-    # exp file to get meta
-    exp = get_exp(False)
-    exp.argparse(hparams)
-    exp.save()
-
-    # exp file to get weights
-    checkpoint = ModelCheckpoint(save_dir)
-
-    # add these to the trainer options
-    trainer_options['experiment'] = exp
-    trainer_options['checkpoint_callback'] = checkpoint
-
-    # fit model
-    trainer = Trainer(**trainer_options)
-    trainer.is_slurm_managing_tasks = True
-    result = trainer.fit(model)
-
-    # track epoch before saving
-    real_global_epoch = trainer.current_epoch
-
-    # correct result and ok accuracy
-    assert result == 1, 'amp + dp model failed to complete'
-
-    # ---------------------------
-    # HPC LOAD/SAVE
-    # ---------------------------
-    # save
-    trainer.hpc_save(save_dir, exp)
-
-    # init new trainer
-    new_exp = get_exp(False, version=exp.version)
-    trainer_options['experiment'] = new_exp
-    trainer_options['checkpoint_callback'] = ModelCheckpoint(save_dir)
-    trainer_options['train_percent_check'] = 0.2
-    trainer_options['val_percent_check'] = 0.2
-    trainer_options['max_nb_epochs'] = 1
-    new_trainer = Trainer(**trainer_options)
-
-    # set the epoch start hook so we can predict before the model does the full training
-    def assert_good_acc():
-        assert trainer.current_epoch == real_global_epoch and trainer.current_epoch > 0
-
-        # if model and state loaded correctly, predictions will be good even though we
-        # haven't trained with the new loaded model
-        dp_model = new_trainer.model
-        dp_model.eval()
-
-        _ = [run_prediction(dataloader, dp_model, dp=True) for dataloader in trainer.val_dataloader]
-
-    # new model
-    model = LightningTestModel(hparams)
-    model.on_sanity_check_start = assert_good_acc
-
-    # fit new model which should load hpc weights
-    new_trainer.fit(model)
-
-    # test freeze on gpu
-    model.freeze()
-    model.unfreeze()
-
    clear_save_dir()


@@ -1,8 +0,0 @@
-tox
-coverage
-codecov
-pytest>=3.0.5
-pytest-cov
-flake8
-check-manifest
-test_tube
@@ -1,47 +0,0 @@
-# this file is *not* meant to cover or endorse the use of tox or pytest or testing in general,
-#
-#  It's meant to show the use of:
-#
-#  - check-manifest
-#     confirm items checked into vcs are in your segdist
-#  - python setup.py check 
-#     confirm required package meta-data in setup.py
-#  - readme_renderer (when using a ReStructuredText README)
-#     confirms your long_description will render correctly on PyPI.
-#
-#  and also to help confirm pull requests to this project.
-
-[tox]
-envlist = py{35,36,37}
-
-[pytest]
-log_cli = 0
-log_cli_level = CRITICAL
-log_cli_format = %(message)s
-log_file = pytest.log
-log_file_level = DEBUG
-log_file_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)
-log_file_date_format=%Y-%m-%d %H:%M:%S
-
-[testenv]
-basepython =
-    py35: python3.5
-    py36: python3.6
-    py37: python3.7
-deps =
-    -r requirements.txt
-    -r ./tests/requirements.txt
-commands =
-    check-manifest --ignore tox.ini
-    python setup.py check -m -s
-    flake8 .
-    coverage run --source pytorch_lightning -m py.test pytorch_lightning tests examples -v --doctest-modules
-
-[flake8]
-exclude = .tox,*.egg,build,temp,examples/*
-select = E,W,F
-doctests = True
-verbose = 2
-# https://pep8.readthedocs.io/en/latest/intro.html#error-codes
-format = pylint
-max-line-length = 100
@@ -11,7 +11,10 @@ rm -rf ./dist/*
 python3 setup.py sdist
 twine upload dist/*

+
+
 # to update docs
 # cd to root dir
 # mkdocs gh-deploy

+