release v0.5.1.2

Merge branch 'master' into docs
cleaned up test_tube logger
2026-09-12 12:40:20 +08:00 · 2019-10-06 00:08:17 -04:00 · 2019-10-05 23:51:36 -04:00 · 2019-10-05 23:46:09 -04:00 · 2019-10-05 23:35:10 -04:00 · 2019-10-05 23:23:38 -04:00
127 changed files with 6105 additions and 9415 deletions
@@ -1,90 +0,0 @@
 # Python CircleCI 2.0 configuration file
 #
 # Check https://circleci.com/docs/2.0/language-python/ for more details
 #
 version: 2.0
 references:
  install_deps: &install_deps
    run:
      name: Install Dependences
      command: |
        pip install "$TORCH_VERSION" --user
        # this is temporal fix til test-tube is not merged and released
        pip install -r requirements.txt --user
        sudo pip install pytest pytest-cov pytest-flake8
        pip install -r ./tests/requirements.txt --user
  tests_format: &tests_format
   run:
     name: Tests and formating
     command: |
       python --version ; pip --version ; pip list
       py.test pytorch_lightning tests pl_examples -v --doctest-modules --junitxml=test-reports/pytest_junit.xml --flake8
     no_output_timeout: 15m
  make_docs: &make_docs
   run:
     name: Make Documentation
     command: |
       # sudo apt-get install pandoc
       pip install -r requirements.txt --user
       sudo pip install -r docs/requirements.txt
       # sphinx-apidoc -o ./docs/source ./pytorch_lightning **/test_* --force --follow-links
       cd docs; make clean ; make html
 jobs:
  Build-Docs:
    docker:
      - image: circleci/python:3.7
    steps:
      - checkout
      - *make_docs
  PyTorch:
    docker:
      - image: circleci/python:3.7
    environment:
      - TORCH_VERSION: "torch"
    steps: &steps
      - checkout
      - *install_deps
      - *tests_format
      - store_test_results:
          path: test-reports
      - store_artifacts:
          path: test-reports
  PyTorch-v1.1:
    docker:
      - image: circleci/python:3.6
    environment:
      - TORCH_VERSION: "torch>=1.1, <1.2"
    steps: *steps
  PyTorch-v1.2:
    docker:
      - image: circleci/python:3.6
    environment:
      - TORCH_VERSION: "torch>=1.2, <1.3"
    steps: *steps
  PyTorch-v1.3:
    docker:
      - image: circleci/python:3.6
    environment:
      - TORCH_VERSION: "torch>=1.3, <1.4"
    steps: *steps
 workflows:
  version: 2
  build:
    jobs:
      - Build-Docs
      - PyTorch-v1.1
      - PyTorch-v1.2
      - PyTorch-v1.3
@@ -1,59 +0,0 @@
 # How to become a core contributor
 Thanks for your interest in joining the Lightning team! We’re a rapidly growing project which is poised to become the go-to framework for DL researchers! 
 We're currently recruiting for a team of 5 core maintainers. 
 As a core maintainer you will have a strong say in the direction of the project. Big changes will require a majority of maintainers to agree.
 ### Code of conduct  
 First and foremost, you'll be evaluated against [these core values](https://github.com/williamFalcon/pytorch-lightning/blob/master/.github/CONTRIBUTING.md). Any code we commit or feature we add needs to align with those core values.
 ### The bar for joining the team   
 Lightning is being used to solve really hard problems at the top AI labs in the world. As such, the bar for adding team members is extremely high. Candidates must have solid engineering skills, have a good eye for user experience, and must be a power user of Lightning and PyTorch.
 With that said, the Lightning team will be diverse and a reflection of an inclusive AI community. You don't have to be an engineer to conntribute! Scientists with great usability intuition and PyTorch ninja skills are welcomed!
 ### Responsibilities:
 The responsibilities mainly revolve around 3 things.
 #### Github issues
 - Here we want to help users have an amazing experience. These range from questions from new people getting into DL to questions from researchers about doing something esoteric with Lightning
 Often, these issues require some sort of bug fix, document clarification or new functionality to be scoped out.
 - To become a core member you must resolve at least 10 Github issues which align with the API design goals for Lightning. By the end of these 10 issues I should feel comfortable in the way you answer user questions
 Pleasant/helpful tone.
 - Can abstract from that issue or bug into functionality that might solve other related issues or makes the platform more flexible.
 - Don’t make users feel like they don’t know what they’re doing. We’re here to help and to make everyone’s experience delightful.
 #### Pull requests
 - Here we need to ensure the code that enters Lightning is high quality. For each PR we need to:
 - Make sure code coverage does not decrease
 - Documents are updated
 - Code is elegant and simple
 - Code is NOT overly engineered or hard to read
 - Ask yourself, could a non-engineer understand what’s happening here?
 - Make sure new tests are written
 - Is this NECESSARY for Lightning? There are some PRs which are just purely about adding engineering complexity which have no place in Lightning. 
 Guidance
 - Some other PRs are for people who are wanting to get involved and add something unnecessary. We do want their help though! So don’t approve the PR, but direct them to a Github issue that they might be interested in helping with instead!
 - To be considered for core contributor, please review 10 PRs and help the authors land it on master. Once you've finished the review, ping me 
 for a sanity check. At the end of 10 PRs if your PR reviews are inline with expectations described above, then you can merge PRs on your own going forward,
 otherwise we'll do a few more until we're both comfortable :)
 #### Project directions
 There are some big decisions which the project must make. For these I expect core contributors to have something meaningful to add if it’s their area of expertise.
 #### Diversity
 Lightning should reflect the broader community it serves. As such we should have scientists/researchers from 
 different fields contributing!   
 The first 5 core contributors will fit this profile. Thus if you overlap strongly with experiences and expertise as someone else on the team, you might have to wait until the next set of contributors are added.
 #### Summary: Requirements to apply
 - Solve 10 Github issues. The goal is to be inline with expectations for solving issues by the last one so you can do them on your own. If not, I might ask you to solve a few more specific ones.
 - Do 10 PR reviews. The goal is to be inline with expectations for solving issues by the last one so you can do them on your own. If not, I might ask you to solve a few more specific ones.
 If you want to be considered, ping me on gitter and start [tracking your progress here](https://docs.google.com/spreadsheets/d/15D58gp8DvI0Z6qbbYVRuaWioiwzafcP58-UlbuO_CMU/edit?usp=sharing).
@@ -1,14 +1,15 @@
 # Contributing    
 Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!   
-## Main Core Value: One less thing to remember    
+## One less thing to remember    
 Simplify the API as much as possible from the user perspective. Any additions or improvements should minimize things the user needs to remember.   
-For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.  
+For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.   
 ## Lightning Design Principles   
 We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
-#### No PyTorch Interference   
+
 #### No PyTorch interference   
 We don't want to add any abstractions on top of pure PyTorch. This gives researchers all the control they need without having to learn yet another framework.    
 #### Simple Internal Code    
@@ -20,25 +21,17 @@ There are 1,000 ways to do something. However, something eventually becomes stan
 When something becomes a best practice, we add it to the framework. This likely looks like code in utils or in the model file that everyone keeps adding over and over again across projects. When this happens, bring that code inside the trainer and add a flag for it.
 #### Simple External API    
-What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others. Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people. 
+What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others. Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people.    
 #### Backward-compatible API   
 We all hate updating our deep learning packages because we don't want to refactor a bunch of stuff. In Lightning, we make sure every change we make which could break an API is backwards compatible with good deprecation warnings.
 You shouldn't be afraid to upgrade Lightning :)
 #### Gain User Trust    
 As a researcher you can't have any part of your code going wrong. So, make thorough tests that ensure an implementation of a new trick or subbtle change is correct.    
-#### Interoperability  
+## Contribution types    
 Have a favorite feature from other libraries like fast.ai or transformers? Those should just work with lightning as well. Grab your favorite model or learning rate scheduler from your favorite library and run it in Lightning.
 ## Contribution Types    
 Currently looking for help implementing new features or adding bug fixes. 
 A lot of good work has already been done in project mechanics (requirements.txt, setup.py, pep8, badges, ci, etc...) we're in a good state there thanks to all the early contributors (even pre-beta release)!   
-## Bug Fixes:  
+## Bug fixes:  
 1. Submit a github issue.   
 2. Fix it.  
 3. Submit a PR! 
@@ -47,7 +40,3 @@ A lot of good work has already been done in project mechanics (requirements.txt,
 1. Submit a github issue.   
 2. We'll agree on the feature scope.     
 3. Submit a PR! (with updated docs and tests 🙃).   
 ## Coding Styleguide
 1. Test the code with flake8.
 2. Use f-strings.
@@ -11,52 +11,26 @@ assignees: ''
 1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/williamFalcon/pytorch-lightning/issues/79).    
 2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
-## 🐛 Bug
+**Describe the bug**
-
+A clear and concise description of what the bug is.
 <!-- A clear and concise description of what the bug is. -->
 ### To Reproduce
 **To Reproduce**
 Steps to reproduce the behavior:
 1. Go to '...'
-2. Run '....'
+2. Click on '....'
 3. Scroll down to '....'
 4. See error
-<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
+**Expected behavior**
 A clear and concise description of what you expected to happen.
 **Screenshots**
 If applicable, add screenshots to help explain your problem.
-#### Code sample
+**Desktop (please complete the following information):**
-<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
+ - OS: [e.g. iOS]
-Minimal means having the shortest code but still preserving the bug. -->
+ - Browser [e.g. chrome, safari]
 - Version [e.g. 22]
-### Expected behavior
+**Additional context**
-
+Add any other context about the problem here.
 <!-- A clear and concise description of what you expected to happen. -->
 ### Environment
 Please copy and paste the output from our
 [environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
 (or fill out the checklist below manually).
 You can get the script and run it with:
 ```
 wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
 # For security purposes, please check the contents of collect_env.py before running it.
 python collect_env.py
 ```
 - PyTorch Version (e.g., 1.0):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:
 ### Additional context
 <!-- Add any other context about the problem here. -->
@@ -7,21 +7,14 @@ assignees: ''
 ---
-## 🚀 Feature
+**Is your feature request related to a problem? Please describe.**
-<!-- A clear and concise description of the feature proposal -->
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-### Motivation
+**Describe the solution you'd like**
 A clear and concise description of what you want to happen.
-<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
+**Describe alternatives you've considered**
 A clear and concise description of any alternative solutions or features you've considered.
-### Pitch
+**Additional context**
-
+Add any other context or screenshots about the feature request here.
 <!-- A clear and concise description of what you want to happen. -->
 ### Alternatives
 <!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
 ### Additional context
 <!-- Add any other context or screenshots about the feature request here. -->
@@ -7,24 +7,20 @@ assignees: ''
 ---
 ## ❓ Questions and Help
 ### Before asking:   
 1. search the issues.   
 2. search the docs.    
-<!-- If you still can't find what you need: -->
+If you still can't find what you need:     
 #### What is your question?    
-#### What is your question?
+#### Code    
 Please paste a code snippet if your question requires it!   
-#### Code
+#### What have you tried?    
-<!-- Please paste a code snippet if your question requires it! -->   
+#### What's your environment?   
-
+- conda version (no venv)   
-#### What have you tried?
+- PyTorch version   
-
+- Lightning version   
-#### What's your environment?
+- Test-tube version
 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]
@@ -7,12 +7,11 @@ assignees: ''
 ---
 ## 📚 Documentation
 For typos and doc fixes, please go ahead and:
 1. Create an issue.
 2. Fix the typo.   
 3. Submit a PR.
 Thanks!
@@ -1,16 +1,16 @@
 # Before submitting
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
+- Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [ ] Did you read the [contributor guideline](https://github.com/williamFalcon/pytorch-lightning/blob/master/.github/CONTRIBUTING.md)?
+- Did you read the [contributor guideline](https://github.com/williamFalcon/pytorch-lightning/blob/master/.github/CONTRIBUTING.md)?
- [ ] Did you make sure to update the docs?   
+- Did you make sure to update the docs?   
- [ ] Did you write any new necessary tests?  
+- Did you write any new necessary tests?  
 ## What does this PR do?
 Fixes # (issue).
 ## PR review    
 Anyone in the community is free to review the PR once the tests have passed.     
-If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
+If we didn't discuss your PR in Github issues there's a high chance it will not be merged.   
 ## Did you have fun?
 Make sure you had fun coding 🙃
@@ -12,8 +12,6 @@ test_tube_exp/
 tests/tests_tt_dir/
 tests/save_dir
 default/
 lightning_logs/
 tests/tests/
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -5,13 +5,9 @@
 # Required
 version: 2
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
  configuration: docs/source/conf.py
 # Build documentation with MkDocs
-#mkdocs:
+mkdocs:
-#  configuration: mkdocs.yml
+  configuration: mkdocs.yml
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: all
@@ -20,5 +16,4 @@ formats: all
 python:
  version: 3.7
  install:
-    #- requirements: requirements.txt
+    - requirements: docs/requirements.txt
    - requirements: docs/requirements.txt
@@ -1,9 +1,5 @@
 # use this to run tests
 rm -rf _ckpt_*
 rm -rf tests/save_dir*
 rm -rf tests/mlruns_*
-rm -rf tests/cometruns*
+coverage run --source pytorch_lightning -m py.test pytorch_lightning tests examples -v --doctest-modules   
 rm -rf tests/tests/*
 rm -rf lightning_logs
 coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules
 coverage report -m
@@ -16,20 +16,16 @@ language: python
 matrix:
  include:
-    - dist: xenial  # Ubuntu 16.04
+    - os: linux
-      python: 3.6
+      dist: xenial  # Ubuntu 16.04
      env:
        - TOXENV=py36
        - MIN_REQUIREMENTS=1
    - dist: xenial  # Ubuntu 16.04
      python: 3.7
      env:
        - TOXENV=py37
        - MIN_REQUIREMENTS=1
    - dist: bionic  # Ubuntu 18.04
      python: 3.6
      env: TOXENV=py36
-    - dist: bionic  # Ubuntu 18.04
+    - os: linux
      dist: bionic  # Ubuntu 18.04
      python: 3.6
      env: TOXENV=py36
    - os: linux
      dist: bionic  # Ubuntu 18.04
      python: 3.7
      env: TOXENV=py37
    - os: osx
@@ -39,7 +35,7 @@ matrix:
      addons:
        homebrew:
          # update: true
-          packages: python3.6
+          packages: python3
      before_install:
        - pip3 install virtualenv
        - virtualenv -p python3 ~/venv
@@ -55,29 +51,14 @@ matrix:
 cache: pip
 install:
-  - pip install future  # needed for `builtins`
+  - pip install -r requirements.txt
-  - sudo pip install tox
+  - pip install -r ./tests/requirements.txt
-
+  - pip --version ; pip list
 before_script:
  # rewrite all minimal requirements as strict
  - if [[ "${MIN_REQUIREMENTS}" == "1" ]]; then
      python -c "req = open('requirements.txt').read().replace('>', '=') ; open('requirements-ci.txt', 'w').write(req)" ;
    else
      cp requirements.txt requirements-ci.txt ;
    fi
  - pip install -r requirements-ci.txt -U
 script:
  # integration
  - tox --sitepackages
-
+  - python setup.py install --dry-run
  #- python setup.py install --dry-run --user
  - virtualenv vEnv ;
    source vEnv/bin/activate
  - pip install --editable . ;
    cd .. & python -c "import pytorch_lightning ; print(pytorch_lightning.__version__)"
  - deactivate ;
    rm -rf vEnv
 after_success:
  - coverage report
@@ -16,9 +16,9 @@ exclude *.svg
 recursive-include pytorch_lightning *.py
 # include examples
-recursive-include pl_examples *.py
+recursive-include examples *.py
-recursive-include pl_examples *.md
+recursive-include examples *.md
-recursive-include pl_examples *.sh
+recursive-include examples *.sh
 # exclude tests from package
 recursive-exclude tests *
@@ -37,7 +37,6 @@ exclude *.yml
 prune .git
 prune .github
 prune .circleci
 prune notebook*
 prune temp*
 prune test*
@@ -1,6 +1,6 @@
 <div align="center">
-![Logo](docs/source/_static/images/lightning_logo_small.png)
+![Logo](./docs/source/_static/lightning_logo_small.png)
 # PyTorch Lightning
@@ -10,15 +10,13 @@
 [![PyPI Status](https://badge.fury.io/py/pytorch-lightning.svg)](https://badge.fury.io/py/pytorch-lightning)
 [![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
 [![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
-[![Build status](https://ci.appveyor.com/api/projects/status/NEW-PROJECT-ID?svg=true)](https://ci.appveyor.com/project/williamFalcon/pytorch-lightning)
+[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning)
-[![Coverage](docs/source/_static/images/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
+[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
 [![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
 [![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
-[![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/pytorch-lightning/shared_invite/enQtODU5ODIyNTUzODQwLTFkMDg5Mzc1MDBmNjEzMDgxOTVmYTdhYjA1MDdmODUyOTg2OGQ1ZWZkYTQzODhhNzdhZDA3YmNhMDhlMDY4YzQ)
+[![Gitter](https://badges.gitter.im/PyTorch-Lightning/community.svg)](https://gitter.im/PyTorch-Lightning/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
 [![Next Release](https://img.shields.io/badge/Next%20Release-Dec%206-<COLOR>.svg)](https://shields.io/)
 <!-- 
 removed until codecov badge isn't empy. likely a config error showing nothing on master.
 [![codecov](https://codecov.io/gh/Borda/pytorch-lightning/branch/master/graph/badge.svg)](https://codecov.io/gh/Borda/pytorch-lightning)
@@ -34,15 +32,9 @@ pip install pytorch-lightning
 ## Docs   
 **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**
 ## Demo  
 [Copy and run this COLAB!](https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=HOk9c4_35FKg)
 ## What is it?  
 Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.
 ## How much effort is it to convert?
 You're probably tired of switching frameworks at this point. But it is a very quick process to refactor into the Lightning format. [Check out this tutorial](https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538)
 ## Starting a new project?   
 [Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     
@@ -60,7 +52,7 @@ Lightning sets up all the boilerplate state-of-the-art training for you so you c
 - [Lightning features](https://github.com/williamFalcon/pytorch-lightning#lightning-automates-all-of-the-following-each-is-also-configurable)    
 - [Examples](https://github.com/williamFalcon/pytorch-lightning#examples)    
 - [Tutorials](https://github.com/williamFalcon/pytorch-lightning#tutorials)
- [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/.github/CONTRIBUTING.md)
+- [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md)    
 - [Bleeding edge install](https://github.com/williamFalcon/pytorch-lightning#bleeding-edge)   
 - [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)   
 - [Asking for help](https://github.com/williamFalcon/pytorch-lightning#asking-for-help)
@@ -69,85 +61,85 @@ Lightning sets up all the boilerplate state-of-the-art training for you so you c
 ---
 ## How do I do use it?   
-Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/) which you fit using a Trainer.
+Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule]((https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)) which you fit using a Trainer.    
 The LightningModule defines a *system* such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier such as the example below.     
 To use lightning do 2 things:  
-1. [Define a LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)
+1. [Define a LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)     
-**WARNING:** This syntax is for version 0.5.0+ where abbreviations were removed.
+
-    ```python
+**WARNING:** This syntax is for version 0.5.0+ where abbreviations were removed.    
-    import os
+```python
-    
+import os
-    import torch
+import torch
-    from torch.nn import functional as F
+from torch.nn import functional as F
-    from torch.utils.data import DataLoader
+from torch.utils.data import DataLoader
-    from torchvision.datasets import MNIST
+from torchvision.datasets import MNIST
-    from torchvision import transforms
+import torchvision.transforms as transforms
-    
+
-    import pytorch_lightning as pl
+import pytorch_lightning as pl
-    
+
-    class CoolSystem(pl.LightningModule):
+class CoolSystem(pl.LightningModule):
-    
+
-        def __init__(self):
+    def __init__(self):
-            super(CoolSystem, self).__init__()
+        super(CoolSystem, self).__init__()
-            # not the best model...
+        # not the best model...
-            self.l1 = torch.nn.Linear(28 * 28, 10)
+        self.l1 = torch.nn.Linear(28 * 28, 10)
-    
+
-        def forward(self, x):
+    def forward(self, x):
-            return torch.relu(self.l1(x.view(x.size(0), -1)))
+        return torch.relu(self.l1(x.view(x.size(0), -1)))
-    
+
-        def training_step(self, batch, batch_idx):
+    def training_step(self, batch, batch_nb):
-            # REQUIRED
+        # REQUIRED
-            x, y = batch
+        x, y = batch
-            y_hat = self.forward(x)
+        y_hat = self.forward(x)
-            loss = F.cross_entropy(y_hat, y)
+        loss = F.cross_entropy(y_hat, y)
-            tensorboard_logs = {'train_loss': loss}
+        tensorboard_logs = {'train_loss': loss}
-            return {'loss': loss, 'log': tensorboard_logs}
+        return {'loss': loss, 'log': tensorboard_logs}
-    
+
-        def validation_step(self, batch, batch_idx):
+    def validation_step(self, batch, batch_nb):
-            # OPTIONAL
+        # OPTIONAL
-            x, y = batch
+        x, y = batch
-            y_hat = self.forward(x)
+        y_hat = self.forward(x)
-            return {'val_loss': F.cross_entropy(y_hat, y)}
+        return {'val_loss': F.cross_entropy(y_hat, y)}
-    
+
-        def validation_end(self, outputs):
+    def validation_end(self, outputs):
-            # OPTIONAL
+        # OPTIONAL
-            avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
+        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
-            tensorboard_logs = {'val_loss': avg_loss}
+        tensorboard_logs = {'val_loss': avg_loss}
-            return {'avg_val_loss': avg_loss, 'log': tensorboard_logs}
+        return {'avg_val_loss': avg_loss, 'log': tensorboard_logs}
-    
+
-        def configure_optimizers(self):
+    def configure_optimizers(self):
-            # REQUIRED
+        # REQUIRED
-            # can return multiple optimizers and learning_rate schedulers
+        # can return multiple optimizers and learning_rate schedulers
-            # (LBFGS it is automatically supported, no need for closure function)
+        # (LBFGS it is automatically supported, no need for closure function)
-            return torch.optim.Adam(self.parameters(), lr=0.02)
+        return torch.optim.Adam(self.parameters(), lr=0.02)
-    
+
-        @pl.data_loader
+    @pl.data_loader
-        def train_dataloader(self):
+    def train_dataloader(self):
-            # REQUIRED
+        # REQUIRED
-            return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
-    
+
-        @pl.data_loader
+    @pl.data_loader
-        def val_dataloader(self):
+    def val_dataloader(self):
-            # OPTIONAL
+        # OPTIONAL
-            return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
-    
+
-        @pl.data_loader
+    @pl.data_loader
-        def test_dataloader(self):
+    def test_dataloader(self):
-            # OPTIONAL
+        # OPTIONAL
-            return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
+        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
-    ```
+```
 2. Fit with a [trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)    
-    ```python
+```python
-    from pytorch_lightning import Trainer
+from pytorch_lightning import Trainer
-    
+
-    model = CoolSystem()
+model = CoolSystem()
-    
+
-    # most basic trainer, uses good defaults
+# most basic trainer, uses good defaults
-    trainer = Trainer()    
+trainer = Trainer()    
-    trainer.fit(model)   
+trainer.fit(model)   
-    ```
+```     
 Trainer sets up a tensorboard logger, early stopping and checkpointing by default (you can modify all of them or
 use something other than tensorboard).   
@@ -155,24 +147,24 @@ use something other than tensorboard).
 Here are more advanced examples
 ```python   
 # train on cpu using only 10% of the data (for demo purposes)
-trainer = Trainer(max_epochs=1, train_percent_check=0.1)
+trainer = Trainer(max_nb_epochs=1, train_percent_check=0.1)
 # train on 4 gpus (lightning chooses GPUs for you)
-# trainer = Trainer(max_epochs=1, gpus=4, distributed_backend='ddp')  
+# trainer = Trainer(max_nb_epochs=1, gpus=4)  
 # train on 4 gpus (you choose GPUs)
-# trainer = Trainer(max_epochs=1, gpus=[0, 1, 3, 7], distributed_backend='ddp')   
+# trainer = Trainer(max_nb_epochs=1, gpus=[0, 1, 3, 7])   
 # train on 32 gpus across 4 nodes (make sure to submit appropriate SLURM job)
-# trainer = Trainer(max_epochs=1, gpus=8, num_gpu_nodes=4, distributed_backend='ddp')
+# trainer = Trainer(max_nb_epochs=1, gpus=8, nb_gpu_nodes=4)
 # train (1 epoch only here for demo)
 trainer.fit(model)
 # view tensorboard logs 
-logging.info(f'View tensorboard logs by running\ntensorboard --logdir {os.getcwd()}')
+print('View tensorboard logs by running\ntensorboard --logdir %s' % os.getcwd())
-logging.info('and going to http://localhost:6006 on your browser')
+print('and going to http://localhost:6006 on your browser')
-```
+```    
 When you're all done you can even run the test set separately.   
 ```python
@@ -184,14 +176,14 @@ trainer.test()
 Everything in gray!    
 You define the blue parts using the LightningModule interface:  
-![Overview](docs/source/_static/images/overview_flat.jpg)
+![Overview](./docs/source/_static/overview_flat.jpg)
 ```python
 # what to do in the training loop
-def training_step(self, batch, batch_idx):
+def training_step(self, batch, batch_nb):
 # what to do in the validation loop
-def validation_step(self, batch, batch_idx):
+def validation_step(self, batch, batch_nb):
 # how to aggregate validation_step outputs
 def validation_end(self, outputs):
@@ -206,7 +198,7 @@ def test_dataloader():
 ```python
 # define what happens for training here
-def training_step(self, batch, batch_idx):
+def training_step(self, batch, batch_nb):
    x, y = batch
    # define your own forward and loss calculation
@@ -233,7 +225,7 @@ def training_step(self, batch, batch_idx):
 ```python
 # define what happens for validation here
-def validation_step(self, batch, batch_idx):    
+def validation_step(self, batch, batch_nb):    
    x, y = batch
    # or as basic as a CNN classification
@@ -267,11 +259,11 @@ def validation_end(self, outputs):
 ## Tensorboard    
 Lightning is fully integrated with tensorboard, MLFlow and supports any logging module.   
-![tensorboard-support](docs/source/_static/images/tf_loss.png)
+![tensorboard-support](./docs/source/_static/tf_loss.png)
 Lightning also adds a text column with all the hyperparameters for this experiment.      
-![tensorboard-support](docs/source/_static/images/tf_tags.png)
+![tensorboard-support](./docs/source/_static/tf_tags.png)
 ## Lightning automates all of the following ([each is also configurable](https://williamfalcon.github.io/pytorch-lightning/Trainer/)):
@@ -300,7 +292,6 @@ Lightning also adds a text column with all the hyperparameters for this experime
 #### Distributed training    
 - [Implement Your Own Distributed (DDP) training](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#init_ddp_connection)
 - [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
 - [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
 - [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
@@ -344,18 +335,17 @@ Lightning also adds a text column with all the hyperparameters for this experime
 - [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  
 ## Examples   
- [GAN](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/domain_templates/gan.py)    
+- [GAN](https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/domain_templates/gan.py)    
- [MNIST](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/basic_examples)      
+- [MNIST](https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/basic_examples)      
 - [Other projects using Lightning](https://github.com/williamFalcon/pytorch-lightning/network/dependents?package_id=UGFja2FnZS0zNzE3NDU4OTM%3D)    
- [Multi-node](https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples/multi_node_examples)   
+- [Multi-node](https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/multi_node_examples)   
 ## Tutorials   
 - [Basic Lightning use](https://towardsdatascience.com/supercharge-your-ai-research-with-pytorch-lightning-337948a99eec)    
 - [9 key speed features in Pytorch-Lightning](https://towardsdatascience.com/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565)    
 - [SLURM, multi-node training with Lightning](https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd)     
---
+---    
 ## Asking for help    
 Welcome to the Lightning community!   
@@ -402,7 +392,7 @@ Nope. Please use anaconda or miniconda.
    # install latest Lightning version without upgrading deps    
    pip install -U --no-deps pytorch-lightning
    ```     
- **PyTorch 1.2.0, 1.3.0,**
+- **PyTorch 1.2.0**
    Install via pip as normal   
 ## Custom installation
@@ -425,16 +415,3 @@ You can also install any past release from this repository:
 ```bash
 pip install https://github.com/williamFalcon/pytorch-lightning/archive/0.4.4.zip --upgrade
 ```
 ## Bibtex
 If you want to cite the framework feel free to use this (but only if you loved it 😊):
 ```
@misc{Falcon2019,
  author = {Falcon, W.A.},
  title = {PyTorch Lightning},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/williamFalcon/pytorch-lightning}}
 }
 ```
@@ -44,13 +44,11 @@ install:
  # purpose but it is problematic because it tends to cancel builds pushed
  # directly to master instead of just PR builds (or the converse).
  - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%path%
-  #- pip install -U --user "pip<19.3"
+  - pip install -U --user pip
-  - python -m pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
+  - pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
-  - python -m pip install -r ./tests/requirements.txt
+  - pip install -r ./tests/requirements.txt
  - python -m pip install pytest-flake8
-# scripts to run before tests (working directory and environment changes
+# scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
 # are persisted from the previous steps such as "before_build")
 before_test:
  - python --version
  - pip --version
@@ -59,9 +57,7 @@ before_test:
 # to run your custom scripts instead of automatic tests
 test_script:
-  - coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules --flake8
+  - tox --sitepackages --parallel auto
  #- python setup.py sdist
  #- twine check dist/*
 on_success:
  - coverage report
@@ -0,0 +1,707 @@
 # Lightning Module interface
 [[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/root_module.py)]
 A lightning module is a strict superclass of nn.Module, it provides a standard interface for the trainer to interact with the model.
 The easiest thing to do is copy the [minimal example](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) below and modify accordingly. 
 Otherwise, to Define a Lightning Module, implement the following methods:
 **Required**:  
 - [training_step](RequiredTrainerInterface.md#training_step)      
 - [train_dataloader](RequiredTrainerInterface.md#train_dataloader)    
 - [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)    
 **Optional**:   
 - [validation_step](RequiredTrainerInterface.md#validation_step)    
 - [validation_end](RequiredTrainerInterface.md#validation_end) 
 - [test_step](RequiredTrainerInterface.md#test_step)    
 - [test_end](RequiredTrainerInterface.md#test_end) 
 - [val_dataloader](RequiredTrainerInterface.md#val_dataloader)    
 - [test_dataloader](RequiredTrainerInterface.md#test_dataloader)    
 - [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)    
 - [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)    
 - [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)    
 ---
 ### Minimal example   
 ```python
 import os
 import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 import torchvision.transforms as transforms
 import pytorch_lightning as pl
 class CoolModel(pl.LightningModule):
    def __init__(self):
        super(CoolModel, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)
    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))
    def training_step(self, batch, batch_nb):
        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
        return {'loss': F.cross_entropy(y_hat, y)}
    def validation_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
        return {'val_loss': F.cross_entropy(y_hat, y)}
    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}
    def test_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
        return {'test_loss': F.cross_entropy(y_hat, y)}
    def test_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
        return {'avg_test_loss': avg_loss}
    def configure_optimizers(self):
        # REQUIRED
        return torch.optim.Adam(self.parameters(), lr=0.02)
    @pl.data_loader
    def train_dataloader(self):
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
    @pl.data_loader
    def val_dataloader(self):
        # OPTIONAL
        # can also return a list of val dataloaders
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
    @pl.data_loader
    def test_dataloader(self):
        # OPTIONAL
        # can also return a list of test dataloaders
        return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
 ---   
 ### How do these methods fit into the broader training?     
 The LightningModule interface is on the right. Each method corresponds to a part of a research project. Lightning automates everything not in blue.    
 <p align="center">
  <a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg">
    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg" height="900px">
  </a>
 </p>   
 ## Required Methods    
 ### training_step
 ``` {.python}
 def training_step(self, batch, batch_nb)
 ```
 In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
 **Params**    
 | Param  | description  |
 |---|---|
 |  batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
 **Return**   
 Dictionary or OrderedDict   
 | key  | value  | is required |
 |---|---|---|
 |  loss | tensor scalar  | Y |
 |  progress_bar | Dict for progress bar display. Must have only tensors  | N |
 |  log | Dict of metrics to add to logger. Must have only tensors (no images, etc)  | N |
 **Example**
 ``` {.python}
 def training_step(self, batch, batch_nb):
    x, y, z = batch
    # implement your own
    out = self.forward(x)
    loss = self.loss(out, x)
    output = {
        'loss': loss, # required
        'progress_bar': {'training_loss': loss}, # optional (MUST ALL BE TENSORS)
        'log': {'training_loss': loss} # optional (MUST ALL BE TENSORS)
    }
    # return a dict
    return output
 ```    
 If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.    
 ``` {.python}
 # Multiple optimizers (ie: GANs)     
 def training_step(self, batch, batch_nb, optimizer_idx):
    if optimizer_idx == 0:
        # do training_step with encoder
    if optimizer_idx == 1:
        # do training_step with decoder    
 ```    
 You can also return a -1 instead of a dict to stop the current loop. This is useful if you want to
 break out of the current training epoch early.
 --- 
 ### train_dataloader 
 ``` {.python}
@pl.data_loader
 def train_dataloader(self)
 ```
 Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.   
 If you want to change the data during every epoch DON'T use the data_loader decorator.   
 ##### Return
 PyTorch DataLoader
 **Example**
 ``` {.python}
@pl.data_loader
 def train_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.hparams.batch_size,
        shuffle=True
    )
    return loader
 ```
 --- 
 ### configure_optimizers 
 ``` {.python}
 def configure_optimizers(self)
 ```
 Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
 Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
 **Note:** If you use multiple optimizers, training_step will have an additional ```optimizer_idx``` parameter.    
 **Note 2:** If you use LBFGS lightning handles the closure function automatically for you.
 ##### Return    
 Return any of these 3 options:   
 Single optimizer   
 List or Tuple - List of optimizers    
 Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers
 **Example**
 ``` {.python}
 # most cases
 def configure_optimizers(self):
    opt = Adam(self.parameters(), lr=0.01)
    return opt
 # multiple optimizer case (eg: GAN)
 def configure_optimizers(self):
    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
    return generator_opt, disriminator_opt
 # example with learning_rate schedulers  
 def configure_optimizers(self):
    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
    return [generator_opt, disriminator_opt], [discriminator_sched]
 ```  
 If you need to control how often those optimizers step or override the default .step() schedule, override 
 the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step) hook.   
 ## Optional Methods    
 ### validation_step
 ``` {.python}
 # if you have one val dataloader:
 def validation_step(self, batch, batch_nb)   
 # if you have multiple val dataloaders:  
 def validation_step(self, batch, batch_nb, dataloader_idxdx)
 ```
 **OPTIONAL**    
 If you don't need to validate you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
 When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
 The dict you return here will be available in the `validation_end` method. 
 **Params**   
 | Param  | description  |
 |---|---|
 |  batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
 |  dataloader_idx | Integer displaying which dataloader this is (only if multiple val datasets used)  |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict or OrderedDict - passed to the validation_end step | N |
 **Example**
 ``` {.python}
 # CASE 1: A single validation dataset
 def validation_step(self, batch, batch_nb):
    x, y = batch
    # implement your own
    out = self.forward(x)
    loss = self.loss(out, y)
    # log 6 example images
    # or generated text... or whatever
    sample_imgs = x[:6]
    grid = torchvision.utils.make_grid(sample_imgs)
    self.logger.experiment.add_image('example_images', grid, 0) 
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    # all optional...
    # return whatever you need for the collation function validation_end
    output = OrderedDict({
        'val_loss': loss_val,
        'val_acc': torch.tensor(val_acc), # everything must be a tensor
    })
    # return an optional dict
    return output
 ```   
 If you pass in multiple validation datasets, validation_step will have an additional argument.
 ```python
 # CASE 2: multiple validation datasets
 def validation_step(self, batch, batch_nb, dataset_idx):
    # dataset_idx tells you which dataset this is.   
 ```   
 The ```dataset_idx``` corresponds to the order of datasets returned in ```val_dataloader```.    
 --- 
 ### validation_end
 ``` {.python}
 def validation_end(self, outputs)
 ```   
 If you didn't define a validation_step, this won't be called.       
 Called at the end of the validation loop with the outputs of validation_step.
 The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
 **Params**    
 | Param  | description  |
 |---|---|
 |  outputs | List of outputs you defined in validation_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
 **Return**   
 Dictionary or OrderedDict   
 | key  | value  | is required |
 |---|---|---|
 |  progress_bar | Dict for progress bar display. Must have only tensors  | N |
 |  log | Dict of metrics to add to logger. Must have only tensors (no images, etc)  | N |
 **Example**
 With a single dataloader
 ``` {.python}
 def validation_end(self, outputs):
    """
    Called at the end of validation to aggregate outputs
    :param outputs: list of individual outputs of each validation step
    :return:
    """
    val_loss_mean = 0
    val_acc_mean = 0
    for output in outputs:
        val_loss_mean += output['val_loss']
        val_acc_mean += output['val_acc']
    val_loss_mean /= len(outputs)
    val_acc_mean /= len(outputs)
    tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
    # show val_loss and val_acc in progress bar but only log val_loss
    results = {
        'progress_bar': tqdm_dict,
        'log': {'val_loss': val_loss_mean.item()}
    }
    return results
 ```
 With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
 one entry per dataloader, while the inner list contains the individual outputs of 
 each validation step for that dataloader.
 ``` {.python}
 def validation_end(self, outputs):
    """
    Called at the end of validation to aggregate outputs
    :param outputs: list of list of individual outputs of each validation step
    :return:
    """
    val_loss_mean = 0
    val_acc_mean = 0
    i = 0
    for dataloader_outputs in outputs:
        for output in dataloader_outputs:
            val_loss_mean += output['val_loss']
            val_acc_mean += output['val_acc']
            i += 1
    val_loss_mean /= i
    val_acc_mean /= i
    tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
    # show val_loss and val_acc in progress bar but only log val_loss
    results = {
        'progress_bar': tqdm_dict,
        'log': {'val_loss': val_loss_mean.item()}
    }
    return results
 ```
 ### test_step
 ``` {.python}
 # if you have one test dataloader:
 def test_step(self, batch, batch_nb)   
 # if you have multiple test dataloaders:  
 def test_step(self, batch, batch_nb, dataloader_idxdx)
 ```
 **OPTIONAL**    
 If you don't need to test you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy.   
 When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.
 The dict you return here will be available in the `test_end` method. 
 This function is used when you execute `trainer.test()`.
 **Params**   
 | Param  | description  |
 |---|---|
 |  batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
 |  dataloader_idx | Integer displaying which dataloader this is (only if multiple test datasets used)  |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
 **Example**
 ``` {.python}
 # CASE 1: A single test dataset
 def test_step(self, batch, batch_nb):
    x, y = batch
    # implement your own
    out = self.forward(x)
    loss = self.loss(out, y)
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    # all optional...
    # return whatever you need for the collation function test_end
    output = OrderedDict({
        'test_loss': loss_test,
        'test_acc': torch.tensor(test_acc), # everything must be a tensor
    })
    # return an optional dict
    return output
 ```   
 If you pass in multiple test datasets, test_step will have an additional argument.
 ```python
 # CASE 2: multiple test datasets
 def test_step(self, batch, batch_nb, dataset_idx):
    # dataset_idx tells you which dataset this is.   
 ```   
 The ```dataset_idx``` corresponds to the order of datasets returned in ```test_dataloader```.    
 --- 
 ### test_end
 ``` {.python}
 def test_end(self, outputs)
 ```   
 If you didn't define a test_step, this won't be called.       
 Called at the end of the test step with the output of each test_step.
 The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
 **Params**    
 | Param  | description  |
 |---|---|
 |  outputs | List of outputs you defined in test_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict of OrderedDict with metrics to display in progress bar | Y |
 **Example**
 ``` {.python}
 def test_end(self, outputs):
    """
    Called at the end of test to aggregate outputs
    :param outputs: list of individual outputs of each test step
    :return:
    """
    test_loss_mean = 0
    test_acc_mean = 0
    for output in outputs:
        test_loss_mean += output['test_loss']
        test_acc_mean += output['test_acc']
    test_loss_mean /= len(outputs)
    test_acc_mean /= len(outputs)
    tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
    # show test_loss and test_acc in progress bar but only log test_loss
    results = {
        'progress_bar': tqdm_dict,
        'log': {'test_loss': val_loss_mean.item()}
    }
    return results
 ```
 With multiple dataloaders, `outputs` will be a list of lists. The outer list contains
 one entry per dataloader, while the inner list contains the individual outputs of 
 each validation step for that dataloader.
 ``` {.python}
 def test_end(self, outputs):
    """
    Called at the end of test to aggregate outputs
    :param outputs: list of individual outputs of each test step
    :return:
    """
    test_loss_mean = 0
    test_acc_mean = 0
    i = 0
    for dataloader_outputs in outputs:
        for output in dataloader_outputs:
            test_loss_mean += output['test_loss']
            test_acc_mean += output['test_acc']
            i += 1
    test_loss_mean /= i 
    test_acc_mean /= i
    tqdm_dict = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
    # show test_loss and test_acc in progress bar but only log test_loss
    results = {
        'progress_bar': tqdm_dict,
        'log': {'test_loss': val_loss_mean.item()}
    }
    return results
 ```
 --- 
 ### on_save_checkpoint 
 ``` {.python}
 def on_save_checkpoint(self, checkpoint)
 ```
 Called by lightning to checkpoint your model. Lightning saves the training state (current epoch, global_step, etc)
 and also saves the model state_dict. If you want to save anything else, use this method to add your own
 key-value pair.
 ##### Return
 Nothing
 **Example**
 ``` {.python}
 def on_save_checkpoint(self, checkpoint):
    # 99% of use cases you don't need to implement this method 
    checkpoint['something_cool_i_want_to_save'] = my_cool_pickable_object
 ```
 --- 
 ### on_load_checkpoint 
 ``` {.python}
 def on_load_checkpoint(self, checkpoint)
 ```
 Called by lightning to restore your model. Lighting auto-restores global step, epoch, etc...
 It also restores the model state_dict.
 If you saved something with **on_save_checkpoint** this is your chance to restore this.
 ##### Return
 Nothing  
 **Example**
 ``` {.python}
 def on_load_checkpoint(self, checkpoint):
    # 99% of the time you don't need to implement this method
    self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']
 ```
 --- 
 ### val_dataloader 
 ``` {.python}
@pl.data_loader
 def val_dataloader(self)
 ```
 **OPTIONAL**    
 If you don't need a validation dataset and a validation_step, you don't need to implement this method.    
 Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.   
 If you want to change the data during every epoch DON'T use the data_loader decorator.   
 ##### Return
 PyTorch DataLoader or list of PyTorch Dataloaders.    
 **Example**
 ``` {.python}
@pl.data_loader
 def val_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.hparams.batch_size,
        shuffle=True
    )
    return loader
 # can also return multiple dataloaders   
@pl.data_loader
 def val_dataloader(self):
    return [loader_a, loader_b, ..., loader_n]   
 ```
 In the case where you return multiple val_dataloaders, the validation_step will have an arguement ```dataset_idx```
 which matches the order here.    
 --- 
 ### test_dataloader 
 ``` {.python}
@pl.data_loader
 def test_dataloader(self)
 ```
 **OPTIONAL**    
 If you don't need a test dataset and a test_step, you don't need to implement this method.    
 Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
 If you want to change the data during every epoch DON'T use the data_loader decorator.   
 ##### Return
 PyTorch DataLoader
 **Example**
 ``` {.python}
@pl.data_loader
 def test_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.hparams.batch_size,
        shuffle=True
    )
    return loader
 ```
 --- 
 ### add_model_specific_args 
 ``` {.python}
@staticmethod
 def add_model_specific_args(parent_parser, root_dir)
 ```
 Lightning has a list of default argparse commands.
 This method is your chance to add or modify commands specific to your model.
 The [hyperparameter argument parser](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/) is available anywhere in your model by calling self.hparams.
 ##### Return
 An argument parser
 **Example**
 ``` {.python}
@staticmethod
 def add_model_specific_args(parent_parser, root_dir):
    parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
    # param overwrites
    # parser.set_defaults(gradient_clip_val=5.0)
    # network params
    parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
    parser.add_argument('--in_features', default=28*28)
    parser.add_argument('--out_features', default=10)
    parser.add_argument('--hidden_dim', default=50000) # use 500 for CPU, 50000 for GPU to see speed difference
    # data
    parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
    # training params (opt)
    parser.opt_list('--learning_rate', default=0.001, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
                    tunable=False)
    parser.opt_list('--batch_size', default=256, type=int, options=[32, 64, 128, 256], tunable=False)
    parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)
    return parser
 ```
@@ -0,0 +1,50 @@
 Lightning modules are strict superclasses of torch.nn.Module. A LightningModule offers the following in addition to that API.
 ---    
 ### freeze 
 Freeze all params for inference
 ```{.python}
 model = MyLightningModule(...)
 model.freeze()
 ```
 ---    
 ### load_from_metrics
 This is the easiest/fastest way which uses the meta_tags.csv file from test-tube to rebuild the model.
 The meta_tags.csv file can be found in the test-tube experiment save_dir.       
 ```{.python}
 pretrained_model = MyLightningModule.load_from_metrics(
    weights_path='/path/to/pytorch_checkpoint.ckpt',
    tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
    on_gpu=True,
    map_location=None
 )
 # predict
 pretrained_model.eval()
 pretrained_model.freeze()
 y_hat = pretrained_model(x)
 ```
 **Params**    
 | Param  | description  |
 |---|---|
 |  weights_path | Path to a PyTorch checkpoint  |
 |  tags_csv | Path to meta_tags.csv file generated by the test-tube Experiment  |
 |  on_gpu | if True, puts model on GPU. Make sure to use transforms option if model devices have changed  |
 |  map_location | A dictionary mapping saved weight GPU devices to new GPU devices |
 **Returns**    
 LightningModule - The pretrained LightningModule
 ---    
 ### unfreeze 
 Unfreeze all params for inference
 ```{.python}
 model = MyLightningModule(...)
 model.unfreeze()
 ```
@@ -0,0 +1,64 @@
 A LightningModule has the following properties which you can access at any time
 --- 
 #### current_epoch
 The current epoch   
 ---
 #### dtype    
 Current dtype    
 ---
 #### logger    
 A reference to the logger you passed into trainer. 
 Passing a logger is optional. If you don't pass one in, Lightning will create one for you automatically.
 This logger saves logs to '''/os.getcwd()/lightning_logs'''
 ```python
 Trainer(logger=your_logger)
 ```    
 Call it from anywhere in your LightningModule to add metrics, images, etc... whatever your logger supports.  
 Here is an example using the TestTubeLogger (which is a wrapper on [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html) with versioned folder structure).     
 ```{.python}
 # if logger is a tensorboard logger or TestTubeLogger
 self.logger.experiment.add_embedding(...)   
 self.logger.experiment.log({'val_loss': 0.9})   
 self.logger.experiment.add_scalars(...)   
 ```
 --- 
 #### global_step   
 Total training batches seen across all epochs   
 --- 
 #### gradient_clip_val   
 The current gradient clip value    
 ---
 #### on_gpu    
 True if your model is currently running on GPUs. Useful to set flags around the LightningModule for different CPU vs GPU behavior.    
 ---
 #### trainer    
 Last resort access to any state the trainer has. Changing certain properties here could affect your training run.
 ```{.python}   
 self.trainer.optimizers
 self.trainer.current_epoch
 ...   
 ```   
 ## Debugging   
 The LightningModule also offers these tricks to help debug.   
 ---   
 #### example_input_array    
 In the LightningModule init, you can set a dummy tensor for this property
 to get a print out of sizes coming into and out of every layer.   
 ```python
 def __init__(self):
    # put the dimensions of the first input to your system
    self.example_input_array = torch.rand(5, 28 * 28)
 ```    
@@ -1,19 +0,0 @@
 # Minimal makefile for Sphinx documentation
 #
 # You can set these variables from the command line.
 SPHINXOPTS    =
 SPHINXBUILD   = sphinx-build
 SOURCEDIR     = source
 BUILDDIR      = build
 # Put it first so that "make" without argument is like "make help".
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 .PHONY: help Makefile
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
@@ -0,0 +1,77 @@
 Lightning can automate saving and loading checkpoints.
 ---
 ### Model saving
 Checkpointing is enabled by default to the current working directory.
 To change the checkpoint path pass in :
 ```python
 Trainer(default_save_path='/your/path/to/save/checkpoints')
 ```
 To modify the behavior of checkpointing pass in your own callback.
 ``` {.python}
 from pytorch_lightning.callbacks import ModelCheckpoint
 # DEFAULTS used by the Trainer
 checkpoint_callback = ModelCheckpoint(
    filepath=os.getcwd(),
    save_best_only=True,
    verbose=True,
    monitor='val_loss',
    mode='min',
    prefix=''
 )
 trainer = Trainer(checkpoint_callback=checkpoint_callback)
 ```
 ---
 ### Restoring training session 
 You might want to not only load a model but also continue training it. Use this method to
 restore the trainer state as well. This will continue from the epoch and global step you last left off.  
 However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).   
 Lightning will restore the session if you pass an experiment with the same version and there's a saved checkpoint.   
 ``` {.python}
 from test_tube import Experiment
 exp = Experiment(version=a_previous_version_with_a_saved_checkpoint)
 trainer = Trainer(experiment=exp)
 # this fit call loads model weights and trainer state
 # the trainer continues seamlessly from where you left off
 # without having to do anything else.
 trainer.fit(model)
 ```
 The trainer restores:  
 - global_step    
 - current_epoch    
 - All optimizers    
 - All lr_schedulers    
 - Model weights
 You can even change the logic of your model as long as the weights and "architecture" of 
 the system isn't different. If you add a layer, for instance, it might not work.   
 At a rough level, here's [what happens inside Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/model_saving.py#L63):   
 ```python
 self.global_step = checkpoint['global_step']
 self.current_epoch = checkpoint['epoch']
 # restore the optimizers
 optimizer_states = checkpoint['optimizer_states']
 for optimizer, opt_state in zip(self.optimizers, optimizer_states):
    optimizer.load_state_dict(opt_state)
 # restore the lr schedulers
 lr_schedulers = checkpoint['lr_schedulers']
 for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
    scheduler.load_state_dict(lrs_state)
 # uses the model you passed into trainer        
 model.load_state_dict(checkpoint['state_dict'])
 ```    
@@ -0,0 +1,255 @@
 Lightning makes multi-gpu training and 16 bit training trivial.
 *Note:*   
 None of the flags below require changing anything about your lightningModel definition. 
 ---
 #### Choosing a backend  
 Lightning supports two backends. DataParallel and DistributedDataParallel. Both can be used for single-node multi-GPU training.
 For multi-node training you must use DistributedDataParallel.   
 ##### DataParallel (dp)   
 Splits a batch across multiple GPUs on the same node. Cannot be used for multi-node training.   
 ##### DistributedDataParallel (ddp)   
 Trains a copy of the model on each GPU and only syncs gradients. If used with DistributedSampler, each GPU trains
 on a subset of the full dataset.  
 ##### DistributedDataParallel-2 (ddp2)   
 Works like DDP, except each node trains a single copy of the model using ALL GPUs on that node. 
 Very useful when dealing with negative samples, etc...
 You can toggle between each mode by setting this flag.
 ``` {.python}
 # DEFAULT (when using single GPU or no GPUs)
 trainer = Trainer(distributed_backend=None)
 # Change to DataParallel (gpus > 1)
 trainer = Trainer(distributed_backend='dp')
 # change to distributed data parallel (gpus > 1)
 trainer = Trainer(distributed_backend='ddp')
 # change to distributed data parallel (gpus > 1)
 trainer = Trainer(distributed_backend='ddp2')
 ```
 If you request multiple nodes, the back-end will auto-switch to ddp.
 We recommend you use DistributedDataparallel even for single-node multi-GPU training. It is MUCH faster than DP but *may*
 have configuration issues depending on your cluster.
 For a deeper understanding of what lightning is doing, feel free to read [this guide](https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565).   
 ---
 #### Distributed and 16-bit precision.    
 Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
 not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.   
 Below are the possible configurations we support.    
 | 1 GPU  | 1+ GPUs  | DP  | DDP  | 16-bit  | command |
 |---|---|---|---|---|---|
 | Y  |   |   |   |  | ```Trainer(gpus=1)``` |
 | Y  |   |   |   | Y | ```Trainer(gpus=1, use_amp=True)``` |
 |   | Y | Y |   |   | ```Trainer(gpus=k, distributed_backend='dp')``` |
 |   | Y |  | Y  |  | ```Trainer(gpus=k, distributed_backend='ddp')``` |
 |   | Y |  | Y  | Y | ```Trainer(gpus=k, distributed_backend='ddp', use_amp=True)``` |
 You also have the option of specifying which GPUs to use by passing a list:   
 ```python
 # DEFAULT (int)
 Trainer(gpus=k)  
 # You specify which GPUs (don't use if running on cluster)  
 Trainer(gpus=[0, 1])  
 # can also be a string
 Trainer(gpus='0, 1')
 ```
 ---
 #### CUDA flags   
 CUDA flags make certain GPUs visible to your script. 
 Lightning sets these for you automatically, there's NO NEED to do this yourself.
 ```python
 # lightning will set according to what you give the trainer
 # os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
 # os.environ["CUDA_VISIBLE_DEVICES"] = "0"
 ```
 However, when using a cluster, Lightning will NOT set these flags (and you should not either). 
 SLURM will set these for you.   
 ---
 #### 16-bit mixed precision
 16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.    
 First, install apex (if install fails, look [here](https://github.com/NVIDIA/apex)):
 ```bash
 $ git clone https://github.com/NVIDIA/apex
 $ cd apex
 # ------------------------
 # OPTIONAL: on your cluster you might need to load cuda 10 or 9
 # depending on how you installed PyTorch
 # see available modules
 module avail
 # load correct cuda before install
 module load cuda-10.0
 # ------------------------
 # make sure you've loaded a cuda version > 4.0 and < 7.0
 module load gcc-6.1.0
 $ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
 ```
 then set this use_amp to True.
 ``` {.python}
 # DEFAULT
 trainer = Trainer(amp_level='O2', use_amp=False)
 ```
 ---
 #### Single-gpu
 Make sure you're on a GPU machine. 
 ```python
 # DEFAULT
 trainer = Trainer(gpus=1)
 ```
 ---
 #### multi-gpu 
 Make sure you're on a GPU machine. You can set as many GPUs as you want.
 In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
 ```python
 # to use DataParallel
 trainer = Trainer(gpus=8, distributed_backend='dp')
 # RECOMMENDED use DistributedDataParallel
 trainer = Trainer(gpus=8, distributed_backend='ddp')
 ```
 ---
 #### Multi-node
 Multi-node training is easily done by specifying these flags. 
 ```python
 # train on 12*8 GPUs
 trainer = Trainer(gpus=8, nb_gpu_nodes=12, distributed_backend='ddp')
 ```
 You must configure your job submission script correctly for the trainer to work. Here is an example
 script for the above trainer configuration.   
 ```sh
 #!/bin/bash -l
 # SLURM SUBMIT SCRIPT
 #SBATCH --nodes=12
 #SBATCH --gres=gpu:8
 #SBATCH --ntasks-per-node=8
 #SBATCH --mem=0
 #SBATCH --time=0-02:00:00
 # activate conda env
 conda activate my_env
 # -------------------------
 # OPTIONAL
 # -------------------------
 # debugging flags (optional)
 # export NCCL_DEBUG=INFO
 # export PYTHONFAULTHANDLER=1
 # PyTorch comes with prebuilt NCCL support... but if you have issues with it
 # you might need to load the latest version from your  modules
 # module load NCCL/2.4.7-1-cuda.10.0
 # on your cluster you might need these:
 # set the network interface
 # export NCCL_SOCKET_IFNAME=^docker0,lo
 # -------------------------
 # random port between 12k and 20k
 export MASTER_PORT=$((12000 + RANDOM % 20000))
 # run script from above
 python my_main_file.py
 ```
 **NOTE:** When running in DDP mode, any errors in your code will show up as an NCCL issue.
 Set the ```NCCL_DEBUG=INFO``` flag to see the ACTUAL error.
 Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a 
 portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).   
 ```python
 # ie: this:
 dataset = myDataset()
 dataloader = Dataloader(dataset)
 # becomes:
 dataset = myDataset()
 dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
 dataloader = Dataloader(dataset, sampler=dist_sampler)
 ```
 #### Auto-slurm-job-submission
 Instead of manually building SLURM scripts, you can use the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) to
 do this for you. The SlurmCluster can also run a grid search if you pass in a [HyperOptArgumentParser](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/).
 Here is an example where you run a grid search of 9 combinations of hyperparams.
 [The full examples are here](https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/new_project_templates/multi_node_examples).
 ```python
 # grid search 3 values of learning rate and 3 values of number of layers for your net
 # this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
 parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
 parser.opt_list('--learning_rate', default=0.001, type=float, options=[1e-3, 1e-2, 1e-1], tunable=True)
 parser.opt_list('--layers', default=1, type=float, options=[16, 32, 64], tunable=True)
 hyperparams = parser.parse_args()
 # Slurm cluster submits 9 jobs, each with a set of hyperparams
 cluster = SlurmCluster(
    hyperparam_optimizer=hyperparams,
    log_path='/some/path/to/save',
 )
 # OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
 # which interface your nodes use for communication
 cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
 # see output of the NCCL connection process
 # NCCL is how the nodes talk to each other
 cluster.add_command('export NCCL_DEBUG=INFO')
 # setting a master port here is a good idea.
 cluster.add_command('export MASTER_PORT=%r' % PORT)
 # ************** DON'T FORGET THIS ***************
 # MUST load the latest NCCL version
 cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
 # configure cluster
 cluster.per_experiment_nb_nodes = 12 
 cluster.per_experiment_nb_gpus = 8
 cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')  
 # submit a script with 9 combinations of hyper params
 # (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
 cluster.optimize_parallel_cluster_gpu(
    main,
    nb_trials=9, # how many permutations of the grid search to run
    job_name='name_for_squeue'
 )
 ```
 The other option is that you generate scripts on your own via a bash command or use another library...
 ---
 #### Self-balancing architecture
 Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.   
 COMING SOON.
@@ -0,0 +1,177 @@
 Lighting offers options for logging information about model, gpu usage, etc, via several different logging frameworks. It also offers printing options for training monitoring.
 ---   
 ### default_save_path   
 Lightning sets a default TestTubeLogger and CheckpointCallback for you which log to
 ```os.getcwd()``` by default. To modify the logging path you can set:
 ```python
 Trainer(default_save_path='/your/path/to/save/checkpoints')
 ```
 If you need more custom behavior (different paths for both, different metrics, etc...)
 from the logger and the checkpointCallback, pass in your own instances as explained below.
 ---
 ### Setting up logging
 The trainer inits a default logger for you (TestTubeLogger). All logs will
 go to the current working directory under a folder named ```os.getcwd()/lightning_logs``. 
 If you want to modify the default logging behavior even more, pass in a logger
 (which should inherit from `LightningBaseLogger`).   
 ```{.python}
 my_logger = MyLightningLogger(...)
 trainer = Trainer(logger=my_logger)
 ```
 The path in this logger will overwrite default_save_path.
 Lightning supports several common experiment tracking frameworks out of the box
 ---
 #### Test tube
 Log using [test tube](https://williamfalcon.github.io/test-tube/).
 ```{.python}
 from pytorch_lightning.logging import TestTubeLogger
 tt_logger = TestTubeLogger(
    save_dir=".",
    name="default",
    debug=False,
    create_git_tag=False
 )
 trainer = Trainer(logger=tt_logger)
 ```
 ---
 #### MLFlow
 Log using [mlflow](https://mlflow.org)
 ```{.python}
 from pytorch_lightning.logging import MLFlowLogger
 mlf_logger = MLFlowLogger(
    experiment_name="default",
    tracking_uri="file:/."
 )
 trainer = Trainer(logger=mlf_logger)
 ```
 ---
 #### Custom logger
 You can implement your own logger by writing a class that inherits from
 `LightningLoggerBase`. Use the `rank_zero_only` decorator to make sure that
 only the first process in DDP training logs data.
 ```{.python}
 from pytorch_lightning.logging import LightningLoggerBase, rank_zero_only
 class MyLogger(LightningLoggerBase):
    @rank_zero_only
    def log_hyperparams(self, params):
        # params is an argparse.Namespace
        # your code to record hyperparameters goes here
        pass
    @rank_zero_only
    def log_metrics(self, metrics, step_num):
        # metrics is a dictionary of metric names and values
        # your code to record metrics goes here
        pass
    def save(self):
        # Optional. Any code necessary to save logger data goes here
        pass
    @rank_zero_only
    def finalize(self, status):
        # Optional. Any code that needs to be run after training
        # finishes goes here
 ```
 If you write a logger than may be useful to others, please send
 a pull request to add it to Lighting!
 ---
 #### Using loggers
 You can call the logger anywhere from your LightningModule by doing:
 ```python
 self.logger
 # add an image if using TestTubeLogger
 self.logger.experiment.add_image(...)
 ```
 #### Display metrics in progress bar 
 ``` {.python}
 # DEFAULT
 trainer = Trainer(show_progress_bar=True)
 ```
 ---
 #### Log metric row every k batches 
 Every k batches lightning will make an entry in the metrics log
 ``` {.python}
 # DEFAULT (ie: save a .csv log file every 10 batches)
 trainer = Trainer(row_log_interval=10)
 ```   
 ---
 #### Log GPU memory
 Logs GPU memory when metrics are logged.   
 ``` {.python}
 # DEFAULT
 trainer = Trainer(log_gpu_memory=None)
 # log only the min/max utilization
 trainer = Trainer(log_gpu_memory='min_max')
 # log all the GPU memory (if on DDP, logs only that node)
 trainer = Trainer(log_gpu_memory='all')
 ```
 ---
 #### Process position
 When running multiple models on the same machine we want to decide which progress bar to use.
 Lightning will stack progress bars according to this value. 
 ``` {.python}
 # DEFAULT
 trainer = Trainer(process_position=0)
 # if this is the second model on the node, show the second progress bar below
 trainer = Trainer(process_position=1)
 ```
 ---
 #### Save a snapshot of all hyperparameters 
 Automatically log hyperparameters stored in the `hparams` attribute as an `argparse.Namespace` 
 ``` {.python}
 class MyModel(pl.Lightning):
    def __init__(self, hparams):
        self.hparams = hparams
    ...
 args = parser.parse_args()
 model = MyModel(args)
 logger = TestTubeLogger(...)
 t = Trainer(logger=logger)
 trainer.fit(model)
 ```
 ---
 #### Write logs file to csv every k batches 
 Every k batches, lightning will write the new logs to disk
 ``` {.python}
 # DEFAULT (ie: save a .csv log file every 100 batches)
 trainer = Trainer(log_save_interval=100)
 ```
@@ -0,0 +1,112 @@
 Lightning supports model training on a cluster managed by SLURM in the following cases:    
 1. Training on a single cpu or single GPU.
 2. Train on multiple GPUs on the same node using DataParallel or DistributedDataParallel
 3. Training across multiple GPUs on multiple different nodes via DistributedDataParallel.
 **Note: A node means a machine with multiple GPUs**
 ---
 #### Running grid search on a cluster
 To use lightning to run a hyperparameter search (grid-search or random-search) on a cluster do 4 things:   
 (1). Define the parameters for the grid search    
 ```{.python}
 from test_tube import HyperOptArgumentParser
 # subclass of argparse
 parser = HyperOptArgumentParser(strategy='random_search')
 parser.add_argument('--learning_rate', default=0.002, type=float, help='the learning rate')
 # let's enable optimizing over the number of layers in the network
 parser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4, 8])
 hparams = parser.parse_args()    
 ```    
 **NOTE** You must set ```Tunable=True``` for that argument to be considered in the permutation set. Otherwise
 test-tube will use the default value. This flag is useful when you don't want to search over an argument and
 want to use the default instead.   
 (2). Define the cluster options in the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) (over 5 nodes and 8 gpus)    
 ```{.python}
 from test_tube.hpc import SlurmCluster
 # hyperparameters is a test-tube hyper params object
 # see https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/
 hyperparams = args.parse()
 # init cluster
 cluster = SlurmCluster(
    hyperparam_optimizer=hyperparams,
    log_path='/path/to/log/results/to',
    python_cmd='python3'
 )
 # let the cluster know where to email for a change in job status (ie: complete, fail, etc...)
 cluster.notify_job_status(email='some@email.com', on_done=True, on_fail=True)
 # set the job options. In this instance, we'll run 20 different models
 # each with its own set of hyperparameters giving each one 1 GPU (ie: taking up 20 GPUs)
 cluster.per_experiment_nb_gpus = 8
 cluster.per_experiment_nb_nodes = 5
 # we'll request 10GB of memory per node
 cluster.memory_mb_per_node = 10000
 # set a walltime of 10 minues
 cluster.job_time = '10:00'
 ```
 (3). Make a main function with your model and trainer. Each job will call this function with a particular
 hparams configuration.    
 ```{.python}
 from pytorch_lightning import Trainer
 def train_fx(trial_hparams, cluster_manager, _):
    # hparams has a specific set of hyperparams
    my_model = MyLightningModel()
    # give the trainer the cluster object
    trainer = Trainer()
    trainer.fit(my_model)
 ```
 (3). Start the grid/random search     
 ```{.python}
 # run the models on the cluster
 cluster.optimize_parallel_cluster_gpu(
    train_fx, 
    nb_trials=20, 
    job_name='my_grid_search_exp_name', 
    job_display_name='my_exp')
 ```
 **NOTE** nb_trials specifies how many of the possible permutations to use. If using ```grid_search``` it will use
 the depth first ordering. If using ```random_search``` it will use the first k shuffled options. FYI, random search
 has been shown to be just as good as any Bayesian optimization method when using a reasonable number of samples (60),
 [see this paper for more information](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf).
 ---
 #### Walltime auto-resubmit
 Lightning automatically resubmits jobs when they reach the walltime. Make sure to set the SIGUSR1 signal in 
 your SLURM script.   
 ```bash
 # 90 seconds before training ends
 #SBATCH --signal=SIGUSR1@90
 ``` 
 When lightning receives the SIGUSR1 signal it will:
 1. save a checkpoint with 'hpc_ckpt' in the name.
 2. resubmit the job using the SLURM_JOB_ID  
 When the script starts again, Lightning will:
 1. search for a 'hpc_ckpt' checkpoint. 
 2. restore the model, optimizers, schedulers, epoch, etc...   
@@ -0,0 +1,31 @@
 To ensure you don't accidentally use test data to guide training decisions Lightning makes running the test set deliberate.   
 ---
 #### test
 You have two options to run the test set. 
 First case is where you test right after a full training routine.
 ``` {.python}
 # run full training
 trainer.fit(model)
 # run test set
 trainer.test()
 ```   
 Second case is where you load a model and run the test set   
 ```{.python}
 model = MyLightningModule.load_from_metrics(
    weights_path='/path/to/pytorch_checkpoint.ckpt',
    tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
    on_gpu=True,
    map_location=None
 )
 # init trainer with whatever options
 trainer = Trainer(...)
 # test (pass in the model)
 trainer.test(model)
 ```
 In this second case, the options you pass to trainer will be used when running the test set (ie: 16-bit, dp, ddp, etc...)  
@@ -0,0 +1,86 @@
 The lightning training loop handles everything except the actual computations of your model. To decide what will happen in your training loop, define the [training_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#training_step).
 Below are all the things lightning automates for you in the training loop.
 ---
 #### Accumulated gradients  
 Accumulated gradients runs K small batches of size N before doing a backwards pass. The effect is a large effective batch size of size KxN. 
 ``` {.python}
 # DEFAULT (ie: no accumulated grads)
 trainer = Trainer(accumulate_grad_batches=1)
 ```
 ---
 #### Force training for min or max epochs
 It can be useful to force training for a minimum number of epochs or limit to a max number
 ``` {.python}
 # DEFAULT
 trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000)
 ```
 ---
 #### Early stopping
 The trainer already sets up default early stopping for you. 
 To modify this behavior, pass in your own EarlyStopping callback.
 ``` {.python}
 from pytorch_lightning.callbacks import EarlyStopping
 # DEFAULTS used by Trainer
 early_stop_callback = EarlyStopping(
    monitor='val_loss',
    min_delta=0.00,
    patience=3,
    verbose=False,
    mode='min'
 )
 trainer = Trainer(early_stop_callback=early_stop_callback)
 ```
 ---
 #### Force disable early stop 
 Use this to turn off early stopping and run training to the [max_epoch](#force-training-for-min-or-max-epochs)
 ``` {.python}
 # DEFAULT
 trainer = Trainer(enable_early_stop=True)
 ```
 ---
 #### Gradient Clipping
 Gradient clipping may be enabled to avoid exploding gradients.
 Specifically, this will [clip the gradient norm computed over all model parameters *together*](https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_).
 ``` {.python}
 # DEFAULT (ie: don't clip)
 trainer = Trainer(gradient_clip_val=0)
 # clip gradients with norm above 0.5
 trainer = Trainer(gradient_clip_val=0.5)
 ```
 ---
 #### Inspect gradient norms
 Looking at grad norms can help you figure out where training might be going wrong.
 ``` {.python}
 # DEFAULT (-1 doesn't track norms)
 trainer = Trainer(track_grad_norm=-1)
 # track the LP norm (P=2 here)
 trainer = Trainer(track_grad_norm=2)
 ```
 ---
 #### Set how much of the training set to check
 If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
 train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(train_percent_check=1.0)
 # check 10% only
 trainer = Trainer(train_percent_check=0.1)
 ```
@@ -0,0 +1,63 @@
 The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the [validation_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#validation_step).
 Below are all the things lightning automates for you in the validation loop.
 **Note**   
 Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don't have to wait until a full epoch to catch possible validation issues.
 ---
 #### Check validation every n epochs
 If you have a small dataset you might want to check validation every n epochs
 ``` {.python}
 # DEFAULT
 trainer = Trainer(check_val_every_n_epoch=1)
 ```
 ---
 #### Set how much of the validation set to check 
 If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
 val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_percent_check=1.0)
 # check 10% only
 trainer = Trainer(val_percent_check=0.1)
 ```
 ---
 #### Set how much of the test set to check 
 If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
 test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(test_percent_check=1.0)
 # check 10% only
 trainer = Trainer(test_percent_check=0.1)
 ```
 ---
 ####  Set validation check frequency within 1 training epoch
 For large datasets it's often desirable to check validation multiple times within a training loop
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_check_interval=0.95)
 # check every .25 of an epoch 
 trainer = Trainer(val_check_interval=0.25)
 ```
 ---
 ####  Set the number of validation sanity steps
 Lightning runs a few steps of validation in the beginning of training. This avoids crashing in the validation loop sometime deep into a lengthy training loop.
 ``` {.python}
 # DEFAULT
 trainer = Trainer(nb_sanity_val_steps=5)
 ```
 You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.
@@ -0,0 +1,51 @@
 These flags are useful to help debug a model.
 ---
 #### Fast dev run 
 This flag is meant for debugging a full train/val/test loop. It'll activate callbacks, everything but only with 1 training and 1 validation batch.
 Use this to debug a full run of your program quickly
 ``` {.python}
 # DEFAULT
 trainer = Trainer(fast_dev_run=False)
 ```
 ---
 #### Inspect gradient norms
 Looking at grad norms can help you figure out where training might be going wrong.
 ``` {.python}
 # DEFAULT (-1 doesn't track norms)
 trainer = Trainer(track_grad_norm=-1)
 # track the LP norm (P=2 here)
 trainer = Trainer(track_grad_norm=2)
 ```
 ---
 #### Make model overfit on subset of data
 A useful debugging trick is to make your model overfit a tiny fraction of the data.
 setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
 ``` {.python}
 # DEFAULT don't overfit (ie: normal training)
 trainer = Trainer(overfit_pct=0.0)
 # overfit on 1% of data 
 trainer = Trainer(overfit_pct=0.01)
 ```
 ---
 #### Print the parameter count by layer
 By default lightning prints a list of parameters *and submodules* when it starts training.
 ---
 #### Print which gradients are nan 
 This option prints a list of tensors with nan gradients.
 ``` {.python}
 # DEFAULT
 trainer = Trainer(print_nan_grads=False)
 ```
 ---
 #### Log GPU usage
 Lightning automatically logs gpu usage to the test tube logs. It'll only do it at the metric logging interval, so it doesn't slow down training.
@@ -0,0 +1,131 @@
 # Hooks
 [[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py)]   
 There are cases when you might want to do something different at different parts of the training/validation loop.
 To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
 **Contributing** If there's a hook you'd like to add, simply:    
 1. Fork PyTorchLightning.    
 2. Add the hook [here](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py).       
 3. Add the correct place in the [Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py) where it should be called.    
 ---
 #### on_epoch_start
 Called in the training loop at the very beginning of the epoch.   
 ```python
 def on_epoch_start(self):
    # do something when the epoch starts
 ```
 ---
 #### on_epoch_end
 Called in the training loop at the very end of the epoch.   
 ```python
 def on_epoch_end(self):
    # do something when the epoch ends 
 ```
 ---
 #### on_batch_start
 Called in the training loop before anything happens for that batch.   
 ```python
 def on_batch_start(self):
    # do something when the batch starts
 ```
 ---
 #### on_batch_end
 Called in the training loop after the batch.   
 ```python
 def on_batch_end(self):
    # do something when the batch ends 
 ```
 ---
 #### on_pre_performance_check
 Called at the very beginning of the validation loop.   
 ```python
 def on_pre_performance_check(self):
    # do something before validation starts 
 ```
 ---
 #### on_post_performance_check
 Called at the very end of the validation loop.   
 ```python
 def on_post_performance_check(self):
    # do something before validation end
 ```
 ---
 #### optimizer_step 
 Calls .step() and .zero_grad for each optimizer.  
 You can override this method to adjust how you do the optimizer step for each optimizer
 Called once per optimizer
 ```python
 # DEFAULT
 def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
    optimizer.step()   
    optimizer.zero_grad()   
 # Alternating schedule for optimizer steps (ie: GANs)    
 def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
    # update generator opt every 2 steps
    if optimizer_i == 0:
        if batch_nb % 2 == 0 :
            optimizer.step()
            optimizer.zero_grad()
    # update discriminator opt every 4 steps
    if optimizer_i == 1:
        if batch_nb % 4 == 0 :
            optimizer.step()
            optimizer.zero_grad()    
    # ...
    # add as many optimizers as you want 
 ```
 This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:   
 ```python
 # learning rate warm-up
 def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
    # warm up lr
    if self.trainer.global_step < 500:
        lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
        for pg in optimizer.param_groups:
            pg['lr'] = lr_scale * self.hparams.learning_rate
    # update params
    optimizer.step()
    optimizer.zero_grad() 
 ```
 ---
 #### on_before_zero_grad
 Called in the training loop after taking an optimizer step and before zeroing grads.
 Good place to inspect weight information with weights updated.
 Called once per optimizer
 ```python
 def on_before_zero_grad(self, optimizer):
    # do something with the optimizer or inspect it. 
 ```
 ---
 #### on_after_backward
 Called in the training loop after model.backward()
 This is the ideal place to inspect or log gradient information 
 ```python
 def on_after_backward(self):
    # example to inspect gradient information in tensorboard
    if self.trainer.global_step % 25 == 0:  # don't make the tf file huge
        params = self.state_dict()
        for k, v in params.items():
            grads = v
            name = k
            self.logger.experiment.add_histogram(tag=name, values=grads, global_step=self.trainer.global_step)
 ```
@@ -0,0 +1,87 @@
 # Trainer
 [[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py)]
 The lightning trainer abstracts best practices for running a training, val, test routine. It calls parts of your model when it wants to hand over full control and otherwise makes training assumptions which are now standard practice in AI research.
 This is the basic use of the trainer:
 ``` {.python}
 from pytorch_lightning import Trainer
 model = LightningTemplate()
 trainer = Trainer()
 trainer.fit(model)
 ```
 But of course the fun is in all the advanced things it can do:
 **Checkpointing**    
 - [Checkpoint callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)    
 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
 - [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
 **Computing cluster (SLURM)**    
 - [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
 - [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
 **Debugging**  
 - [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
 - [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
 - [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
 - [Print which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
 - [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 **Distributed training**    
 - [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
 - [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
 - [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
 - [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
 - [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
 **Experiment Logging**   
 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
 - [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
 - [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
 - [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
 **Training loop**    
 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)    
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Early stopping callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)    
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 **Validation loop**    
 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
 - [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
 - [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
 **Testing loop**    
 - [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  
@@ -0,0 +1,131 @@
 ### Template model definition
 In 99% of cases you want to just copy [this template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/lightning_module_template.py) to start a new lightningModule and change the core of what your model is actually trying to do.
 ```bash
 # get a copy of the module template
 wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/examples/new_project_templates/lightning_module_template.py
 ```
 ---    
 ### Trainer Example 
 ** \_\_main__ function**    
 Normally, we want to let the \_\_main__ function start the training.
 Inside the main we parse training arguments with whatever hyperparameters we want. Your LightningModule will have a 
 chance to add hyperparameters.   
 ```{.python}
 from test_tube import HyperOptArgumentParser
 if __name__ == '__main__':
    # use default args given by lightning
    root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
    parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
    add_default_args(parent_parser, root_dir)
    # allow model to overwrite or extend args
    parser = ExampleModel.add_model_specific_args(parent_parser)
    hyperparams = parser.parse_args()
    # train model
    main(hyperparams)
 ```
 **Main Function**      
 The main function is your entry into the program. This is where you init your model, checkpoint directory, and launch the training.
 The main function should have 3 arguments:   
 - hparams: a configuration of hyperparameters.    
 - slurm_manager: Slurm cluster manager object (can be None)
 - dict: for you to return any values you want (useful in meta-learning, otherwise set to _)    
 ```python
 def main(hparams, cluster, results_dict):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # build model
    model = MyLightningModule(hparams)
    # configure trainer
    trainer = Trainer()
    # train model
    trainer.fit(model)
 ```
 The __main__ function will start training on your **main** function. If you use the HyperParameterOptimizer
 in hyper parameter optimization mode, this main function will get one set of hyperparameters. If you use it as a simple
 argument parser you get the default arguments in the argument parser.
 So, calling main(hyperparams) runs the model with the default argparse arguments.       
 ```{.python}
 main(hyperparams)
 ```
 ---
 #### CPU hyperparameter search      
 ```{.python}
 # run a grid search over 20 hyperparameter combinations.
 hyperparams.optimize_parallel_cpu(
    main_local,
    nb_trials=20,
    nb_workers=1
 )
 ```
 ---
 #### Hyperparameter search on a single or multiple GPUs       
 ```{.python}
 # run a grid search over 20 hyperparameter combinations.
 hyperparams.optimize_parallel_gpu(
    main_local,
    nb_trials=20,
    nb_workers=1,
    gpus=[0,1,2,3]
 )
 ```
 ---
 #### Hyperparameter search on a SLURM HPC cluster   
 ```{.python}    
 def optimize_on_cluster(hyperparams):
    # enable cluster training
    cluster = SlurmCluster(
        hyperparam_optimizer=hyperparams,
        log_path=hyperparams.tt_save_path,
        test_tube_exp_name=hyperparams.tt_name
    )
    # email for cluster coms
    cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
    # configure cluster
    cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
    cluster.job_time = '48:00:00'
    cluster.gpu_type = '1080ti'
    cluster.memory_mb_per_node = 48000
    # any modules for code to run in env
    cluster.add_command('source activate pytorch_lightning')
    # name of exp
    job_display_name = hyperparams.tt_name.split('_')[0]
    job_display_name = job_display_name[0:3]
    # run hopt
    print('submitting jobs...')
    cluster.optimize_parallel_cluster_gpu(
        main,
        nb_trials=hyperparams.nb_hopt_trials,
        job_name=job_display_name
    )
 # run cluster hyperparameter search    
 optimize_on_cluster(hyperparams)
 ```
@@ -0,0 +1,142 @@
 ###### New project Quick Start    
 To start a new project define two files, a LightningModule and a Trainer file.    
 To illustrate Lightning power and simplicity, here's an example of a typical research flow.    
 ###### Case 1: BERT    
 Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
 You would define a single LightningModule and use flags to switch between your different ideas.   
 ```python
 class BERT(pl.LightningModule):
    def __init__(self, model_name, task):
        self.task = task
        if model_name == 'transformer':
            self.net = Transformer()
        elif model_name == 'my_cool_version':
            self.net = MyCoolVersion()
    def training_step(self, batch, batch_nb):
        if self.task == 'standard_bert':
            # do standard bert training with self.net...
            # return loss
        if self.task == 'my_cool_task':
            # do my own version with self.net
            # return loss
 ```   
 ###### Case 2: COOLER NOT BERT    
 But if you wanted to try something **completely** different, you'd define a new module for that.    
 ```python
 class CoolerNotBERT(pl.LightningModule):
    def __init__(self):
        self.net = ...
    def training_step(self, batch, batch_nb):
        # do some other cool task
        # return loss   
 ```   
 ###### Rapid research flow    
 Then you could do rapid research by switching between these two and using the same trainer.   
 ```python
 if use_bert:
    model = BERT()
 else:
    model = CoolerNotBERT()
 trainer = Trainer(gpus=4, use_amp=True)
 trainer.fit(model)
 ```
 Notice a few things about this flow:   
 1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
 2. You get free GPU and 16-bit support without writing any of that code in your model.   
 3. You also get all of the capabilities below (without coding or testing yourself).     
 ---    
 ###### Templates 
 1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
 2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
    - [Basic CPU, GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/basic_examples) 
    - [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/tree/master/examples/multi_node_examples)
 ###### Docs shortcuts
 - [LightningModule](LightningModule/RequiredTrainerInterface/)  
 - [Trainer](Trainer/)  
 ###### Quick start examples 
 - [CPU example](examples/Examples/#cpu-hyperparameter-search)   
 - [Hyperparameter search on single GPU](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)    
 - [Hyperparameter search on multiple GPUs on same node](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)  
 - [Hyperparameter search on a SLURM HPC cluster](examples/Examples/#Hyperparameter search on a SLURM HPC cluster)      
 ###### Checkpointing    
 - [Checkpoint callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)    
 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
 - [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
 ###### Computing cluster (SLURM)    
 - [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
 - [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
 ###### Debugging  
 - [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
 - [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
 - [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
 - [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
 - [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 ###### Distributed training    
 - [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
 - [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
 - [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
 - [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
 - [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
 ###### Experiment Logging   
 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
 - [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
 - [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
 - [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
 ###### Training loop    
 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Early stopping callback](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#early-stopping)    
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 ###### Validation loop    
 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
 - [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
 - [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
 ###### Testing loop  
 - [Run test set](https://williamfalcon.github.io/pytorch-lightning/Trainer/Testing%20loop/)  
@@ -1,35 +0,0 @@
@ECHO OFF
 pushd %~dp0
 REM Command file for Sphinx documentation
 if "%SPHINXBUILD%" == "" (
 	set SPHINXBUILD=sphinx-build
 )
 set SOURCEDIR=source
 set BUILDDIR=build
 if "%1" == "" goto help
 %SPHINXBUILD% >NUL 2>NUL
 if errorlevel 9009 (
 	echo.
 	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
 	echo.installed, then set the SPHINXBUILD environment variable to point
 	echo.to the full path of the 'sphinx-build' executable. Alternatively you
 	echo.may add the Sphinx directory to PATH.
 	echo.
 	echo.If you don't have Sphinx installed, grab it from
 	echo.http://sphinx-doc.org/
 	exit /b 1
 )
 %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
 goto end
 :help
 %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
 :end
 popd
@@ -1,9 +1,2 @@
-sphinx>=1.8.3
+mkdocs-material==4.4.0
-recommonmark  # fails with badges
+mkdocs==1.0.4
 m2r  # fails with multi-line text
 nbsphinx
 pandoc
 docutils
 git+https://github.com/Borda/lightning_sphinx_theme.git
 sphinxcontrib-fulltoc
 sphinxcontrib-mockautodoc
@@ -1,17 +0,0 @@
 {%- set external_urls = {
  'github': 'https://github.com/williamFalcon/pytorch-lightning',
  'github_issues': 'https://github.com/williamFalcon/pytorch-lightning/issues',
  'contributing': 'https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md',
  'docs': 'https://williamfalcon.github.io/pytorch-lightning',
  'twitter': 'https://twitter.com/PyTorchLightnin',
  'discuss': 'https://discuss.pytorch.org',
  'tutorials': 'https://williamfalcon.github.io/pytorch-lightning/',
  'previous_pytorch_versions': 'https://williamfalcon.github.io/pytorch-lightning/',
  'home': 'https://williamfalcon.github.io/pytorch-lightning/',
  'get_started': 'https://williamfalcon.github.io/pytorch-lightning/',
  'features': 'https://williamfalcon.github.io/pytorch-lightning/',
  'blog': 'https://williamfalcon.github.io/pytorch-lightning/',
  'resources': 'https://williamfalcon.github.io/pytorch-lightning/',
  'support': 'https://williamfalcon.github.io/pytorch-lightning/',
 }
 -%}
@@ -1,357 +0,0 @@
 # -*- coding: utf-8 -*-
 #
 # Configuration file for the Sphinx documentation builder.
 #
 # This file does only contain a selection of the most common options. For a
 # full list see the documentation:
 # http://www.sphinx-doc.org/en/master/config
 # -- Path setup --------------------------------------------------------------
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 import os
 import sys
 import glob
 import shutil
 import inspect
 # import m2r
 import builtins
 import pt_lightning_sphinx_theme
 PATH_HERE = os.path.abspath(os.path.dirname(__file__))
 PATH_ROOT = os.path.join(PATH_HERE, '..', '..')
 sys.path.insert(0, os.path.abspath(PATH_ROOT))
 builtins.__LIGHTNING_SETUP__ = True
 import pytorch_lightning  # noqa: E402
 # -- Project documents -------------------------------------------------------
 # # export the documentation
 # with open('intro.rst', 'w') as fp:
 #     intro = pytorch_lightning.__doc__.replace(os.linesep + ' ', '')
 #     fp.write(m2r.convert(intro))
 #     # fp.write(pytorch_lightning.__doc__)
 # # export the READme
 # with open(os.path.join(PATH_ROOT, 'README.md'), 'r') as fp:
 #     readme = fp.read()
 # # replace all paths to relative
 # for ndir in (os.path.basename(p) for p in glob.glob(os.path.join(PATH_ROOT, '*'))
 #              if os.path.isdir(p)):
 #     readme = readme.replace('](%s/' % ndir, '](%s/%s/' % (PATH_ROOT, ndir))
 # with open('readme.md', 'w') as fp:
 #     fp.write(readme)
 for md in glob.glob(os.path.join(PATH_ROOT, '.github', '*.md')):
    shutil.copy(md, os.path.join(PATH_HERE, os.path.basename(md)))
 # -- Project information -----------------------------------------------------
 project = 'PyTorch-Lightning'
 copyright = pytorch_lightning.__copyright__
 author = pytorch_lightning.__author__
 # The short X.Y version
 version = pytorch_lightning.__version__
 # The full version, including alpha/beta/rc tags
 release = pytorch_lightning.__version__
 # -- General configuration ---------------------------------------------------
 # If your documentation needs a minimal Sphinx version, state it here.
 needs_sphinx = '1.4'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
    'sphinx.ext.autodoc',
    'sphinxcontrib.mockautodoc',
    # 'sphinxcontrib.fulltoc',  # breaks pytorch-theme with unexpected kw argument 'titles_only'
    'sphinx.ext.doctest',
    'sphinx.ext.intersphinx',
    'sphinx.ext.todo',
    'sphinx.ext.coverage',
    'sphinx.ext.linkcode',
    'sphinx.ext.autosummary',
    'sphinx.ext.napoleon',
    'recommonmark',
    # 'm2r',
    'nbsphinx',
 ]
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 # https://berkeley-stat159-f17.github.io/stat159-f17/lectures/14-sphinx..html#conf.py-(cont.)
 # https://stackoverflow.com/questions/38526888/embed-ipython-notebook-in-sphinx-document
 # I execute the notebooks manually in advance. If notebooks test the code,
 # they should be run at build time.
 nbsphinx_execute = 'never'
 nbsphinx_allow_errors = True
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
 # source_suffix = ['.rst', '.md']
 # source_suffix = ['.rst', '.md', '.ipynb']
 source_suffix = {
    '.rst': 'restructuredtext',
    '.txt': 'markdown',
    '.md': 'markdown',
    '.ipynb': 'nbsphinx',
 }
 # The master toctree document.
 master_doc = 'index'
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
 language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = ['*.test_*']
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = None
 # -- Options for HTML output -------------------------------------------------
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 # http://www.sphinx-doc.org/en/master/usage/theming.html#builtin-themes
 # html_theme = 'bizstyle'
 # https://sphinx-themes.org
 html_theme = 'pt_lightning_sphinx_theme'
 html_theme_path = [pt_lightning_sphinx_theme.get_html_theme_path()]
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 html_theme_options = {
    'pytorch_project': pytorch_lightning.__homepage__,
    'canonical_url': pytorch_lightning.__homepage__,
    'collapse_navigation': False,
    'display_version': True,
    'logo_only': False,
 }
 html_logo = '_static/images/lightning_logo_small.png'
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.
 #
 # The default sidebars (for documents that don't match any pattern) are
 # defined by theme itself.  Builtin themes are using these templates by
 # default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
 # 'searchbox.html']``.
 #
 # html_sidebars = {}
 # -- Options for HTMLHelp output ---------------------------------------------
 # Output file base name for HTML help builder.
 htmlhelp_basename = project + '-doc'
 # -- Options for LaTeX output ------------------------------------------------
 latex_elements = {
    # The paper size ('letterpaper' or 'a4paper').
    # 'papersize': 'letterpaper',
    # The font size ('10pt', '11pt' or '12pt').
    # 'pointsize': '10pt',
    # Additional stuff for the LaTeX preamble.
    # 'preamble': '',
    # Latex figure (float) alignment
    'figure_align': 'htbp',
 }
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
    (master_doc, project + '.tex', project + ' Documentation', author, 'manual'),
 ]
 # -- Options for manual page output ------------------------------------------
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
    (master_doc, project, project + ' Documentation', [author], 1)
 ]
 # -- Options for Texinfo output ----------------------------------------------
 # Grouping the document tree into Texinfo files. List of tuples
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
    (master_doc, project, project + ' Documentation', author, project,
     'One line description of project.', 'Miscellaneous'),
 ]
 # -- Options for Epub output -------------------------------------------------
 # Bibliographic Dublin Core info.
 epub_title = project
 # The unique identifier of the text. This can be a ISBN number
 # or the project homepage.
 #
 # epub_identifier = ''
 # A unique identification for the text.
 #
 # epub_uid = ''
 # A list of files that should not be packed into the epub file.
 epub_exclude_files = ['search.html']
 # -- Extension configuration -------------------------------------------------
 # -- Options for intersphinx extension ---------------------------------------
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {'https://docs.python.org/': None}
 # -- Options for todo extension ----------------------------------------------
 # If true, `todo` and `todoList` produce output, else they produce nothing.
 todo_include_todos = True
 # https://github.com/rtfd/readthedocs.org/issues/1139
 # I use sphinx-apidoc to auto-generate API documentation for my project.
 # Right now I have to commit these auto-generated files to my repository
 # so that RTD can build them into HTML docs. It'd be cool if RTD could run
 # sphinx-apidoc for me, since it's easy to forget to regen API docs
 # and commit them to my repo after making changes to my code.
 PACKAGES = [
    pytorch_lightning.__name__,
    'pl_examples',
 ]
 def run_apidoc(_):
    for pkg in PACKAGES:
        argv = ['-e', '-o', PATH_HERE, os.path.join(PATH_HERE, PATH_ROOT, pkg),
                '**/test_*', '--force', '--private', '--module-first']
        try:
            # Sphinx 1.7+
            from sphinx.ext import apidoc
            apidoc.main(argv)
        except ImportError:
            # Sphinx 1.6 (and earlier)
            from sphinx import apidoc
            argv.insert(0, apidoc.__file__)
            apidoc.main(argv)
 def setup(app):
    app.connect('builder-inited', run_apidoc)
 # copy all notebooks to local folder
 path_nbs = os.path.join(PATH_HERE, 'notebooks')
 if not os.path.isdir(path_nbs):
    os.mkdir(path_nbs)
 for path_ipynb in glob.glob(os.path.join(PATH_ROOT, 'notebooks', '*.ipynb')):
    path_ipynb2 = os.path.join(path_nbs, os.path.basename(path_ipynb))
    shutil.copy(path_ipynb, path_ipynb2)
 # Ignoring Third-party packages
 # https://stackoverflow.com/questions/15889621/sphinx-how-to-exclude-imports-in-automodule
 MOCK_REQUIRE_PACKAGES = []
 with open(os.path.join(PATH_ROOT, 'requirements.txt'), 'r') as fp:
    for ln in fp.readlines():
        found = [ln.index(ch) for ch in list(',=<>#') if ch in ln]
        pkg = ln[:min(found)] if found else ln
        if pkg.rstrip():
            MOCK_REQUIRE_PACKAGES.append(pkg.rstrip())
 # TODO: better parse from package since the import name and package name may differ
 MOCK_MANUAL_PACKAGES = ['torch', 'torchvision', 'sklearn', 'test_tube', 'mlflow', 'comet_ml']
 autodoc_mock_imports = MOCK_REQUIRE_PACKAGES + MOCK_MANUAL_PACKAGES
 # for mod_name in MOCK_REQUIRE_PACKAGES:
 #     sys.modules[mod_name] = mock.Mock()
 # Options for the linkcode extension
 # ----------------------------------
 github_user = 'williamFalcon'
 github_repo = project
 # Resolve function
 # This function is used to populate the (source) links in the API
 def linkcode_resolve(domain, info):
    def find_source():
        # try to find the file and line number, based on code from numpy:
        # https://github.com/numpy/numpy/blob/master/doc/source/conf.py#L286
        obj = sys.modules[info['module']]
        for part in info['fullname'].split('.'):
            obj = getattr(obj, part)
        fname = inspect.getsourcefile(obj)
        # https://github.com/rtfd/readthedocs.org/issues/5735
        if any([s in fname for s in ('readthedocs', 'checkouts')]):
            # /home/docs/checkouts/readthedocs.org/user_builds/pytorch_lightning/checkouts/
            #  devel/pytorch_lightning/utilities/cls_experiment.py#L26-L176
            path_top = os.path.abspath(os.path.join('..', '..', '..'))
            fname = os.path.relpath(fname, start=path_top)
        else:
            # Local build, imitate master
            fname = 'master/' + os.path.relpath(fname, start=os.path.abspath('..'))
        source, lineno = inspect.getsourcelines(obj)
        return fname, lineno, lineno + len(source) - 1
    if domain != 'py' or not info['module']:
        return None
    try:
        filename = '%s#L%d-L%d' % find_source()
    except Exception:
        filename = info['module'].replace('.', '/') + '.py'
    # import subprocess
    # tag = subprocess.Popen(['git', 'rev-parse', 'HEAD'], stdout=subprocess.PIPE,
    #                        universal_newlines=True).communicate()[0][:-1]
    return "https://github.com/%s/%s/blob/%s" \
           % (github_user, github_repo, filename)
 autodoc_member_order = 'groupwise'
 autoclass_content = 'both'
 autodoc_default_flags = [
    'members', 'undoc-members', 'show-inheritance', 'private-members',
    # 'special-members', 'inherited-members'
 ]
@@ -1,8 +0,0 @@
 Documentation
 =============
 .. toctree::
   :maxdepth: 4
   pytorch_lightning
@@ -1,8 +0,0 @@
 Examples & Tutorials
 ====================
 .. toctree::
   :maxdepth: 3
   pl_examples
@@ -1,40 +0,0 @@
 .. PyTorch-Lightning documentation master file, created by
   sphinx-quickstart on Fri Nov 15 07:48:22 2019.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.
 Welcome to PyTorch-Lightning!
 =============================
 .. toctree::
   :maxdepth: 4
   :name: start
   :caption: Quick Start
   new-project
   examples
 .. toctree::
   :maxdepth: 4
   :name: docs
   :caption: Docs
   documentation
 .. toctree::
   :maxdepth: 1
   :name: community
   :caption: Community
   CODE_OF_CONDUCT.md
   CONTRIBUTING.md
   BECOMING_A_CORE_CONTRIBUTOR.md
 Indices and tables
 ------------------
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
@@ -1,71 +0,0 @@
 Quick Start
 ===========
 To start a new project define two files, a LightningModule and a Trainer file.    
 To illustrate Lightning power and simplicity, here's an example of a typical research flow.    
 Case 1: BERT
 ------------
 Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
 You would define a single LightningModule and use flags to switch between your different ideas.
 .. code-block:: python
    class BERT(pl.LightningModule):
        def __init__(self, model_name, task):
            self.task = task
            if model_name == 'transformer':
                self.net = Transformer()
            elif model_name == 'my_cool_version':
                self.net = MyCoolVersion()
        def training_step(self, batch, batch_idx):
            if self.task == 'standard_bert':
                # do standard bert training with self.net...
                # return loss
            if self.task == 'my_cool_task':
                # do my own version with self.net
                # return loss
 Case 2: COOLER NOT BERT
 -----------------------
 But if you wanted to try something **completely** different, you'd define a new module for that.
 .. code-block:: python
    class CoolerNotBERT(pl.LightningModule):
        def __init__(self):
            self.net = ...
        def training_step(self, batch, batch_idx):
            # do some other cool task
            # return loss
 Rapid research flow
 -------------------
 Then you could do rapid research by switching between these two and using the same trainer.
 .. code-block:: python
    if use_bert:
        model = BERT()
    else:
        model = CoolerNotBERT()
    trainer = Trainer(gpus=4, use_amp=True)
    trainer.fit(model)
 **Notice a few things about this flow:**
 1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
 2. You get free GPU and 16-bit support without writing any of that code in your model.   
 3. You also get all of the capabilities below (without coding or testing yourself).     
@@ -0,0 +1,5 @@
 from .basic_examples.lightning_module_template import LightningTemplateModel
 __all__ = [
    'LightningTemplateModel'
 ]
@@ -2,13 +2,12 @@
 Runs a model on a single node across N-gpus.
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
-from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
+from argparse import ArgumentParser
 from pytorch_lightning import Trainer
 from examples.basic_examples.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
@@ -2,13 +2,12 @@
 Runs a model on a single node across N-gpus.
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
-from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
+from argparse import ArgumentParser
 from pytorch_lightning import Trainer
 from examples.basic_examples.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
@@ -2,21 +2,19 @@
 Example template for defining a system
 """
 import os
 import logging
 from argparse import ArgumentParser
 from collections import OrderedDict
 import torch
 import torch.nn as nn
-import torch.nn.functional as F
+from torchvision.datasets import MNIST
 import torchvision.transforms as transforms
 import torch
 import torch.nn.functional as F
 from argparse import ArgumentParser
 from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
 from torchvision.datasets import MNIST
 import pytorch_lightning as pl
-from pytorch_lightning.core.lightning import LightningModule
+from pytorch_lightning.root_module.root_module import LightningModule
 class LightningTemplateModel(LightningModule):
@@ -100,11 +98,8 @@ class LightningTemplateModel(LightningModule):
        if self.trainer.use_dp or self.trainer.use_ddp2:
            loss_val = loss_val.unsqueeze(0)
        tqdm_dict = {'train_loss': loss_val}
        output = OrderedDict({
-            'loss': loss_val,
+            'loss': loss_val
            'progress_bar': tqdm_dict,
            'log': tqdm_dict
        })
        # can also return just a scalar instead of a dict (return loss_val)
@@ -159,7 +154,7 @@ class LightningTemplateModel(LightningModule):
            val_loss = output['val_loss']
            # reduce manually when using dp
-            if self.trainer.use_dp or self.trainer.use_ddp2:
+            if self.trainer.use_dp:
                val_loss = torch.mean(val_loss)
            val_loss_mean += val_loss
@@ -173,7 +168,7 @@ class LightningTemplateModel(LightningModule):
        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
        tqdm_dict = {'val_loss': val_loss_mean, 'val_acc': val_acc_mean}
-        result = {'progress_bar': tqdm_dict, 'log': tqdm_dict, 'val_loss': val_loss_mean}
+        result = {'progress_bar': tqdm_dict, 'logs': tqdm_dict}
        return result
    # ---------------------
@@ -215,17 +210,17 @@ class LightningTemplateModel(LightningModule):
    @pl.data_loader
    def train_dataloader(self):
-        logging.info('training data loader called')
+        print('training data loader called')
        return self.__dataloader(train=True)
    @pl.data_loader
    def val_dataloader(self):
-        logging.info('val data loader called')
+        print('val data loader called')
        return self.__dataloader(train=False)
    @pl.data_loader
    def test_dataloader(self):
-        logging.info('test data loader called')
+        print('test data loader called')
        return self.__dataloader(train=False)
    @staticmethod
@@ -1,25 +1,27 @@
 """
-To run this template just do:
+To run this template just do:  
-python gan.py
+python gan.py   
-After a few epochs, launch tensorboard to see the images being generated at every batch.
+After a few epochs, launch tensorboard to see the images being generated at every batch.   
 tensorboard --logdir default
 """
 import os
 from argparse import ArgumentParser
-from collections import OrderedDict
+import os
 import numpy as np
-import torch
+
 import torch.nn as nn
 import torch.nn.functional as F
 import torchvision
 import torchvision.transforms as transforms
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 from torch.utils.data import DataLoader
 import torch.nn as nn
 import torch.nn.functional as F
 import torch
 import pytorch_lightning as pl
 from test_tube import Experiment
 class Generator(nn.Module):
@@ -82,7 +84,6 @@ class GAN(pl.LightningModule):
        # cache for generated images
        self.generated_imgs = None
        self.last_imgs = None
    def forward(self, z):
        return self.generator(z)
@@ -90,12 +91,11 @@ class GAN(pl.LightningModule):
    def adversarial_loss(self, y_hat, y):
        return F.binary_cross_entropy(y_hat, y)
-    def training_step(self, batch, batch_idx, optimizer_idx):
+    def training_step(self, batch, batch_nb, optimizer_i):
        imgs, _ = batch
        self.last_imgs = imgs
        # train generator
-        if optimizer_idx == 0:
+        if optimizer_i == 0:
            # sample noise
            z = torch.randn(imgs.shape[0], self.hparams.latent_dim)
@@ -107,54 +107,34 @@ class GAN(pl.LightningModule):
            self.generated_imgs = self.forward(z)
            # log sampled images
-            # sample_imgs = self.generated_imgs[:6]
+            sample_imgs = self.generated_imgs[:6]
-            # grid = torchvision.utils.make_grid(sample_imgs)
+            grid = torchvision.utils.make_grid(sample_imgs)
-            # self.logger.experiment.add_image('generated_images', grid, 0)
+            self.logger.experiment.add_image('generated_images', grid, 0)
            # ground truth result (ie: all fake)
            # put on GPU because we created this tensor inside training_loop
            valid = torch.ones(imgs.size(0), 1)
            if self.on_gpu:
                valid = valid.cuda(imgs.device.index)
            # adversarial loss is binary cross-entropy
            g_loss = self.adversarial_loss(self.discriminator(self.generated_imgs), valid)
-            tqdm_dict = {'g_loss': g_loss}
+
-            output = OrderedDict({
+            return g_loss
                'loss': g_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            })
            return output
        # train discriminator
-        if optimizer_idx == 1:
+        if optimizer_i == 1:
            # Measure discriminator's ability to classify real from generated samples
            # how well can it label as real?
            valid = torch.ones(imgs.size(0), 1)
            if self.on_gpu:
                valid = valid.cuda(imgs.device.index)
            real_loss = self.adversarial_loss(self.discriminator(imgs), valid)
            # how well can it label as fake?
            fake = torch.zeros(imgs.size(0), 1)
-            if self.on_gpu:
+            fake_loss = self.adversarial_loss(self.discriminator(self.generated_imgs.detach()), fake)
                fake = fake.cuda(imgs.device.index)
            fake_loss = self.adversarial_loss(
                self.discriminator(self.generated_imgs.detach()), fake)
            # discriminator loss is the average of these
            d_loss = (real_loss + fake_loss) / 2
-            tqdm_dict = {'d_loss': d_loss}
+
-            output = OrderedDict({
+            return d_loss
                'loss': d_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            })
            return output
    def configure_optimizers(self):
        lr = self.hparams.lr
@@ -172,32 +152,16 @@ class GAN(pl.LightningModule):
        dataset = MNIST(os.getcwd(), train=True, download=True, transform=transform)
        return DataLoader(dataset, batch_size=self.hparams.batch_size)
    def on_epoch_end(self):
        z = torch.randn(8, self.hparams.latent_dim)
        # match gpu device (or keep as cpu)
        if self.on_gpu:
            z = z.cuda(self.last_imgs.device.index)
        # log sampled images
        sample_imgs = self.forward(z)
        grid = torchvision.utils.make_grid(sample_imgs)
        self.logger.experiment.add_image(f'generated_images', grid, self.current_epoch)
 def main(hparams):
-    # ------------------------
+    # save tensorboard logs
-    # 1 INIT LIGHTNING MODEL
+    exp = Experiment(save_dir=os.getcwd())
-    # ------------------------
+
    # init model
    model = GAN(hparams)
-    # ------------------------
+    # fit trainer on CPU
-    # 2 INIT TRAINER
+    trainer = pl.Trainer(experiment=exp, max_nb_epochs=200)
    # ------------------------
    trainer = pl.Trainer()
    # ------------------------
    # 3 START TRAINING
    # ------------------------
    trainer.fit(model)
@@ -205,12 +169,9 @@ if __name__ == '__main__':
    parser = ArgumentParser()
    parser.add_argument("--batch_size", type=int, default=64, help="size of the batches")
    parser.add_argument("--lr", type=float, default=0.0002, help="adam: learning rate")
-    parser.add_argument("--b1", type=float, default=0.5,
+    parser.add_argument("--b1", type=float, default=0.5, help="adam: decay of first order momentum of gradient")
-                        help="adam: decay of first order momentum of gradient")
+    parser.add_argument("--b2", type=float, default=0.999, help="adam: decay of first order momentum of gradient")
-    parser.add_argument("--b2", type=float, default=0.999,
+    parser.add_argument("--latent_dim", type=int, default=100, help="dimensionality of the latent space")
                        help="adam: decay of first order momentum of gradient")
    parser.add_argument("--latent_dim", type=int, default=100,
                        help="dimensionality of the latent space")
    hparams = parser.parse_args()
@@ -2,13 +2,12 @@
 Multi-node example (GPU)
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
-from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
+from argparse import ArgumentParser
 from pytorch_lightning import Trainer
 from examples.basic_examples.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
@@ -31,7 +30,7 @@ def main(hparams):
    # ------------------------
    trainer = Trainer(
        gpus=2,
-        num_nodes=2,
+        nb_gpu_nodes=2,
        distributed_backend='ddp2'
    )
@@ -42,6 +41,7 @@ def main(hparams):
 if __name__ == '__main__':
    root_dir = os.path.dirname(os.path.realpath(__file__))
    parent_parser = ArgumentParser(add_help=False)
@@ -2,13 +2,12 @@
 Multi-node example (GPU)
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
-from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
+from argparse import ArgumentParser
 from pytorch_lightning import Trainer
 from examples.basic_examples.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
@@ -31,7 +30,7 @@ def main(hparams):
    # ------------------------
    trainer = Trainer(
        gpus=2,
-        num_nodes=2,
+        nb_gpu_nodes=2,
        distributed_backend='ddp'
    )
@@ -42,6 +41,7 @@ def main(hparams):
 if __name__ == '__main__':
    root_dir = os.path.dirname(os.path.realpath(__file__))
    parent_parser = ArgumentParser(add_help=False)
@@ -0,0 +1,16 @@
 site_name: PyTorch lightning Documentation
 theme:
  name: 'material'
 docs_dir: docs
 repo_name: 'williamFalcon/pytorch-lightning'
 repo_url: https://github.com/williamFalcon/pytorch-lightning
 site_dir: 'site'
 site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
 dev_addr: '0.0.0.0:8000'
 #google_analytics: ['UA-aasd', 'sitename']
 markdown_extensions:
  - codehilite:
      guess_lang: false
      linenums: true
@@ -1,146 +0,0 @@
 """
 Template model definition
 -------------------------
 In 99% of cases you want to just copy `one of the examples
 <https://github.com/williamFalcon/pytorch-lightning/tree/master/pl_examples>`_
 to start a new lightningModule and change the core of what your model is actually trying to do.
 .. code-block:: bash
    # get a copy of the module template
    wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/pl_examples/new_project_templates/lightning_module_template.py  # noqa: E501
 Trainer Example
 ---------------
 **`__main__` function**
 Normally, we want to let the `__main__` function start the training.
 Inside the main we parse training arguments with whatever hyperparameters we want.
 Your LightningModule will have a chance to add hyperparameters.
 .. code-block:: python
    from test_tube import HyperOptArgumentParser
    if __name__ == '__main__':
        # use default args given by lightning
        root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
        parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
        add_default_args(parent_parser, root_dir)
        # allow model to overwrite or extend args
        parser = ExampleModel.add_model_specific_args(parent_parser)
        hyperparams = parser.parse_args()
        # train model
        main(hyperparams)
 **Main Function**
 The main function is your entry into the program. This is where you init your model, checkpoint directory,
 and launch the training. The main function should have 3 arguments:
 - hparams: a configuration of hyperparameters.
 - slurm_manager: Slurm cluster manager object (can be None)
 - dict: for you to return any values you want (useful in meta-learning, otherwise set to)
 .. code-block:: python
    def main(hparams, cluster, results_dict):
        # build model
        model = MyLightningModule(hparams)
        # configure trainer
        trainer = Trainer()
        # train model
        trainer.fit(model)
 The `__main__` function will start training on your **main** function.
 If you use the HyperParameterOptimizer in hyper parameter optimization mode,
 this main function will get one set of hyperparameters. If you use it as a simple
 argument parser you get the default arguments in the argument parser.
 So, calling main(hyperparams) runs the model with the default argparse arguments.::
    main(hyperparams)
 CPU hyperparameter search
 -------------------------
 .. code-block:: python
    # run a grid search over 20 hyperparameter combinations.
    hyperparams.optimize_parallel_cpu(
        main_local,
        nb_trials=20,
        nb_workers=1
    )
 Hyperparameter search on a single or multiple GPUs
 --------------------------------------------------
 .. code-block:: python
    # run a grid search over 20 hyperparameter combinations.
    hyperparams.optimize_parallel_gpu(
        main_local,
        nb_trials=20,
        nb_workers=1,
        gpus=[0,1,2,3]
    )
 Hyperparameter search on a SLURM HPC cluster
 --------------------------------------------
 .. code-block:: python
    def optimize_on_cluster(hyperparams):
        # enable cluster training
        cluster = SlurmCluster(
            hyperparam_optimizer=hyperparams,
            log_path=hyperparams.tt_save_path,
            test_tube_exp_name=hyperparams.tt_name
        )
        # email for cluster coms
        cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
        # configure cluster
        cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
        cluster.job_time = '48:00:00'
        cluster.gpu_type = '1080ti'
        cluster.memory_mb_per_node = 48000
        # any modules for code to run in env
        cluster.add_command('source activate pytorch_lightning')
        # name of exp
        job_display_name = hyperparams.tt_name.split('_')[0]
        job_display_name = job_display_name[0:3]
        # run hopt
        logging.info('submitting jobs...')
        cluster.optimize_parallel_cluster_gpu(
            main,
            nb_trials=hyperparams.nb_hopt_trials,
            job_name=job_display_name
        )
    # run cluster hyperparameter search
    optimize_on_cluster(hyperparams)
 """
 from .basic_examples.lightning_module_template import LightningTemplateModel
 __all__ = [
    'LightningTemplateModel'
 ]
@@ -1,248 +0,0 @@
 """
 This example is largely adapted from https://github.com/pytorch/examples/blob/master/imagenet/main.py
 """
 import argparse
 import os
 import random
 from collections import OrderedDict
 import torch
 import torch.backends.cudnn as cudnn
 import torch.nn.parallel
 import torch.nn.functional as F
 import torch.optim as optim
 import torch.optim.lr_scheduler as lr_scheduler
 import torch.utils.data
 import torch.utils.data.distributed
 import torchvision.transforms as transforms
 import torchvision.models as models
 import torchvision.datasets as datasets
 import pytorch_lightning as pl
 # pull out resnet names from torchvision models
 MODEL_NAMES = sorted(
    name for name in models.__dict__
    if name.islower() and not name.startswith("__") and callable(models.__dict__[name])
 )
 class ImageNetLightningModel(pl.LightningModule):
    def __init__(self, hparams):
        super(ImageNetLightningModel, self).__init__()
        self.hparams = hparams
        self.model = models.__dict__[self.hparams.arch](pretrained=self.hparams.pretrained)
    def training_step(self, batch, batch_idx):
        images, target = batch
        output = self.model(images)
        loss_val = F.cross_entropy(output, target)
        acc1, acc5 = self.__accuracy(output, target, topk=(1, 5))
        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
        if self.trainer.use_dp or self.trainer.use_ddp2:
            loss_val = loss_val.unsqueeze(0)
            acc1 = acc1.unsqueeze(0)
            acc5 = acc5.unsqueeze(0)
        tqdm_dict = {'train_loss': loss_val}
        output = OrderedDict({
            'loss': loss_val,
            'acc1': acc1,
            'acc5': acc5,
            'progress_bar': tqdm_dict,
            'log': tqdm_dict
        })
        return output
    def validation_step(self, batch, batch_idx):
        images, target = batch
        output = self.model(images)
        loss_val = F.cross_entropy(output, target)
        acc1, acc5 = self.__accuracy(output, target, topk=(1, 5))
        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
        if self.trainer.use_dp or self.trainer.use_ddp2:
            loss_val = loss_val.unsqueeze(0)
            acc1 = acc1.unsqueeze(0)
            acc5 = acc5.unsqueeze(0)
        output = OrderedDict({
            'val_loss': loss_val,
            'val_acc1': acc1,
            'val_acc5': acc5,
        })
        return output
    def validation_end(self, outputs):
        tqdm_dict = {}
        for metric_name in ["val_loss", "val_acc1", "val_acc5"]:
            metric_total = 0
            for output in outputs:
                metric_value = output[metric_name]
                # reduce manually when using dp
                if self.trainer.use_dp or self.trainer.use_ddp2:
                    metric_value = torch.mean(metric_value)
                metric_total += metric_value
            tqdm_dict[metric_name] = metric_total / len(outputs)
        result = {'progress_bar': tqdm_dict, 'log': tqdm_dict, 'val_loss': tqdm_dict["val_loss"]}
        return result
    @classmethod
    def __accuracy(cls, output, target, topk=(1,)):
        """Computes the accuracy over the k top predictions for the specified values of k"""
        with torch.no_grad():
            maxk = max(topk)
            batch_size = target.size(0)
            _, pred = output.topk(maxk, 1, True, True)
            pred = pred.t()
            correct = pred.eq(target.view(1, -1).expand_as(pred))
            res = []
            for k in topk:
                correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
                res.append(correct_k.mul_(100.0 / batch_size))
            return res
    def configure_optimizers(self):
        optimizer = optim.SGD(
            self.parameters(),
            lr=self.hparams.lr,
            momentum=self.hparams.momentum,
            weight_decay=self.hparams.weight_decay
        )
        scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.1)
        return [optimizer], [scheduler]
    @pl.data_loader
    def train_dataloader(self):
        normalize = transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225],
        )
        train_dir = os.path.join(self.hparams.data, 'train')
        train_dataset = datasets.ImageFolder(
            train_dir,
            transforms.Compose([
                transforms.RandomResizedCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize,
            ]))
        if self.use_ddp:
            train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
        else:
            train_sampler = None
        train_loader = torch.utils.data.DataLoader(
            dataset=train_dataset,
            batch_size=self.hparams.batch_size,
            shuffle=(train_sampler is None),
            num_workers=0,
            sampler=train_sampler
        )
        return train_loader
    @pl.data_loader
    def val_dataloader(self):
        normalize = transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225],
        )
        val_dir = os.path.join(self.hparams.data, 'val')
        val_loader = torch.utils.data.DataLoader(
            datasets.ImageFolder(val_dir, transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize,
            ])),
            batch_size=self.hparams.batch_size,
            shuffle=False,
            num_workers=0,
        )
        return val_loader
    @staticmethod
    def add_model_specific_args(parent_parser):  # pragma: no cover
        parser = argparse.ArgumentParser(parents=[parent_parser])
        parser.add_argument('-a', '--arch', metavar='ARCH', default='resnet18', choices=MODEL_NAMES,
                            help='model architecture: ' +
                                 ' | '.join(MODEL_NAMES) +
                                 ' (default: resnet18)')
        parser.add_argument('--epochs', default=90, type=int, metavar='N',
                            help='number of total epochs to run')
        parser.add_argument('--seed', type=int, default=None,
                            help='seed for initializing training. ')
        parser.add_argument('-b', '--batch-size', default=256, type=int,
                            metavar='N',
                            help='mini-batch size (default: 256), this is the total '
                                 'batch size of all GPUs on the current node when '
                                 'using Data Parallel or Distributed Data Parallel')
        parser.add_argument('--lr', '--learning-rate', default=0.1, type=float,
                            metavar='LR', help='initial learning rate', dest='lr')
        parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
                            help='momentum')
        parser.add_argument('--wd', '--weight-decay', default=1e-4, type=float,
                            metavar='W', help='weight decay (default: 1e-4)',
                            dest='weight_decay')
        parser.add_argument('--pretrained', dest='pretrained', action='store_true',
                            help='use pre-trained model')
        return parser
 def get_args():
    parent_parser = argparse.ArgumentParser(add_help=False)
    parent_parser.add_argument('--data-path', metavar='DIR', type=str,
                               help='path to dataset')
    parent_parser.add_argument('--save-path', metavar='DIR', default=".", type=str,
                               help='path to save output')
    parent_parser.add_argument('--gpus', type=int, default=1,
                               help='how many gpus')
    parent_parser.add_argument('--distributed-backend', type=str, default='dp', choices=('dp', 'ddp', 'ddp2'),
                               help='supports three options dp, ddp, ddp2')
    parent_parser.add_argument('--use-16bit', dest='use-16bit', action='store_true',
                               help='if true uses 16 bit precision')
    parent_parser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true',
                               help='evaluate model on validation set')
    parser = ImageNetLightningModel.add_model_specific_args(parent_parser)
    return parser.parse_args()
 def main(hparams):
    model = ImageNetLightningModel(hparams)
    if hparams.seed is not None:
        random.seed(hparams.seed)
        torch.manual_seed(hparams.seed)
        cudnn.deterministic = True
    trainer = pl.Trainer(
        default_save_path=hparams.save_path,
        gpus=hparams.gpus,
        max_epochs=hparams.epochs,
        distributed_backend=hparams.distributed_backend,
        use_amp=hparams.use_16bit
    )
    if hparams.evaluate:
        trainer.run_evaluation()
    else:
        trainer.fit(model)
 if __name__ == '__main__':
    main(get_args())
@@ -1,38 +1,9 @@
-"""Package info"""
+from .trainer.trainer import Trainer
 from .root_module.root_module import LightningModule
 from .root_module.decorators import data_loader
-__version__ = '0.5.3.2'
+__all__ = [
-__author__ = 'William Falcon et al.'
+    'Trainer',
-__author_email__ = 'waf2107@columbia.edu'
+    'LightningModule',
-__license__ = 'Apache-2.0'
+    'data_loader',
-__copyright__ = 'Copyright (c) 2018-2019, %s.' % __author__
+]
 __homepage__ = 'https://github.com/williamFalcon/pytorch-lightning'
 # this has to be simple string, see: https://github.com/pypa/twine/issues/522
 __docs__ = "PyTorch Lightning is the lightweight PyTorch wrapper for ML researchers." \
           " Scale your models. Write less boilerplate."
 try:
    # This variable is injected in the __builtins__ by the build
    # process. It used to enable importing subpackages of skimage when
    # the binaries are not built
    __LIGHTNING_SETUP__
 except NameError:
    __LIGHTNING_SETUP__ = False
 if __LIGHTNING_SETUP__:
    import sys
    sys.stderr.write('Partial import of skimage during the build process.\n')
    # We are not importing the rest of the scikit during the build
    # process, as it may not be compiled yet
 else:
    from .trainer.trainer import Trainer
    from .core.lightning import LightningModule
    from .core.decorators import data_loader
    import logging
    __all__ = [
        'Trainer',
        'LightningModule',
        'data_loader',
    ]
    logging.basicConfig(level=logging.INFO)
@@ -1,33 +1,32 @@
 import os
 import shutil
 import logging
 import warnings
 import numpy as np
-from pytorch_lightning.overrides.data_parallel import LightningDistributedDataParallel
+from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel
 class Callback(object):
    """Abstract base class used to build new callbacks.
    # Properties
-        * params: dict. Training parameters
+        params: dict. Training parameters
            (eg. verbosity, batch size, number of epochs...).
            Reference of the model being trained.
-
+    The `logs` dictionary that callback methods
-    The `logs` dictionary that callback methods take as argument will contain keys
+    take as argument will contain keys for quantities relevant to
-     for quantities relevant to the current batch or epoch.
+    the current batch or epoch.
-    Currently, the `.fit()` method of the `Sequential` model class will include the following
+    Currently, the `.fit()` method of the `Sequential` model class
-     quantities in the `logs` that it passes to its callbacks:
+    will include the following quantities in the `logs` that
-    * on_epoch_end: logs include `acc` and `loss`, and
+    it passes to its callbacks:
        on_epoch_end: logs include `acc` and `loss`, and
            optionally include `val_loss`
            (if validation is enabled in `fit`), and `val_acc`
            (if validation and accuracy monitoring are enabled).
-    * on_batch_begin: logs include `size`,
+        on_batch_begin: logs include `size`,
            the number of samples in the current batch.
-    * on_batch_end: logs include `loss`, and optionally `acc`
+        on_batch_end: logs include `loss`, and optionally `acc`
            (if accuracy monitoring is enabled).
    """
    def __init__(self):
@@ -63,7 +62,6 @@ class Callback(object):
 class EarlyStopping(Callback):
    """Stop training when a monitored quantity has stopped improving.
    # Arguments
        monitor: quantity to be monitored.
        min_delta: minimum change in the monitored quantity
@@ -80,7 +78,6 @@ class EarlyStopping(Callback):
            monitored has stopped increasing; in `auto`
            mode, the direction is automatically inferred
            from the name of the monitored quantity.
    """
    def __init__(self, monitor='val_loss',
@@ -95,7 +92,7 @@ class EarlyStopping(Callback):
        self.stopped_epoch = 0
        if mode not in ['auto', 'min', 'max']:
-            logging.info(f'EarlyStopping mode {mode} is unknown, fallback to auto mode.')
+            print('EarlyStopping mode %s is unknown, fallback to auto mode.' % mode)
            mode = 'auto'
        if mode == 'min':
@@ -125,12 +122,10 @@ class EarlyStopping(Callback):
        current = logs.get(self.monitor)
        stop_training = False
        if current is None:
-            warnings.warn(
+            print('Early stopping conditioned on metric `%s` '
-                f'Early stopping conditioned on metric `{self.monitor}`'
+                  'which is not available. Available metrics are: %s' %
-                f' which is not available. Available metrics are: {",".join(list(logs.keys()))}',
+                  (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning)
-                RuntimeWarning)
+            exit(-1)
            stop_training = True
            return stop_training
        if self.monitor_op(current - self.min_delta, self.best):
            self.best = current
@@ -146,34 +141,26 @@ class EarlyStopping(Callback):
    def on_train_end(self, logs=None):
        if self.stopped_epoch > 0 and self.verbose > 0:
-            logging.info(f'Epoch {self.stopped_epoch + 1:05d}: early stopping')
+            print('Epoch %05d: early stopping' % (self.stopped_epoch + 1))
 class ModelCheckpoint(Callback):
    """Save the model after every epoch.
-
+    `filepath` can contain named formatting options,
    The `filepath` can contain named formatting options,
    which will be filled the value of `epoch` and
    keys in `logs` (passed in `on_epoch_end`).
    For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`,
    then the model checkpoints will be saved with the epoch number and
    the validation loss in the filename.
    # Arguments
        filepath: string, path to save the model file.
        monitor: quantity to monitor.
        verbose: verbosity mode, 0 or 1.
-        save_top_k: if `save_top_k == k`,
+        save_best_only: if `save_best_only=True`,
-            the best k models according to
+            the latest best model according to
-            the quantity monitored will be saved.
+            the quantity monitored will not be overwritten.
            if `save_top_k == 0`, no models are saved.
            if `save_top_k == -1`, all models are saved.
            Please note that the monitors are checked every `period` epochs.
            if `save_top_k >= 2` and the callback is called multiple
            times inside an epoch, the name of the saved file will be
            appended with a version count starting with `v0`.
        mode: one of {auto, min, max}.
-            If `save_top_k != 0`, the decision
+            If `save_best_only=True`, the decision
            to overwrite the current save file is made
            based on either the maximization or the
            minimization of the monitored quantity. For `val_acc`,
@@ -184,159 +171,94 @@ class ModelCheckpoint(Callback):
            saved (`model.save_weights(filepath)`), else the full model
            is saved (`model.save(filepath)`).
        period: Interval (number of epochs) between checkpoints.
    """
    def __init__(self, filepath, monitor='val_loss', verbose=0,
-                 save_top_k=1, save_weights_only=False,
+                 save_best_only=False, save_weights_only=False,
                 mode='auto', period=1, prefix=''):
        super(ModelCheckpoint, self).__init__()
        if (
            save_top_k and
            os.path.isdir(filepath) and
            len(os.listdir(filepath)) > 0
        ):
            warnings.warn(
                f"Checkpoint directory {filepath} exists and is not empty with save_top_k != 0."
                "All files in this directory will be deleted when a checkpoint is saved!"
            )
        self.monitor = monitor
        self.verbose = verbose
        self.filepath = filepath
-        os.makedirs(filepath, exist_ok=True)
+        self.save_best_only = save_best_only
        self.save_top_k = save_top_k
        self.save_weights_only = save_weights_only
        self.period = period
-        self.epochs_since_last_check = 0
+        self.epochs_since_last_save = 0
        self.prefix = prefix
        self.best_k_models = {}
        # {filename: monitor}
        self.kth_best_model = ''
        self.best = 0
        if mode not in ['auto', 'min', 'max']:
-            warnings.warn(
+            print('ModelCheckpoint mode %s is unknown, '
-                f'ModelCheckpoint mode {mode} is unknown, '
+                  'fallback to auto mode.' % (mode), RuntimeWarning)
                'fallback to auto mode.', RuntimeWarning)
            mode = 'auto'
        if mode == 'min':
            self.monitor_op = np.less
-            self.kth_value = np.Inf
+            self.best = np.Inf
            self.mode = 'min'
        elif mode == 'max':
            self.monitor_op = np.greater
-            self.kth_value = -np.Inf
+            self.best = -np.Inf
            self.mode = 'max'
        else:
            if 'acc' in self.monitor or self.monitor.startswith('fmeasure'):
                self.monitor_op = np.greater
-                self.kth_value = -np.Inf
+                self.best = -np.Inf
                self.mode = 'max'
            else:
                self.monitor_op = np.less
-                self.kth_value = np.Inf
+                self.best = np.Inf
                self.mode = 'min'
-    def _del_model(self, filepath):
+    def save_model(self, filepath, overwrite):
-        dirpath = os.path.dirname(filepath)
+        dirpath = '/'.join(filepath.split('/')[:-1])
        # make paths
-        os.makedirs(dirpath, exist_ok=True)
+        os.makedirs(os.path.dirname(filepath), exist_ok=True)
-        try:
+        if overwrite:
-            shutil.rmtree(filepath)
+            for filename in os.listdir(dirpath):
-        except OSError:
+                if self.prefix in filename:
-            os.remove(filepath)
+                    path_to_delete = os.path.join(dirpath, filename)
-
+                    try:
-    def _save_model(self, filepath):
+                        shutil.rmtree(path_to_delete)
-        dirpath = os.path.dirname(filepath)
+                    except OSError:
-
+                        os.remove(path_to_delete)
        # make paths
        os.makedirs(dirpath, exist_ok=True)
        # delegate the saving to the model
        self.save_function(filepath)
    def check_monitor_top_k(self, current):
        less_than_k_models = len(self.best_k_models.keys()) < self.save_top_k
        if less_than_k_models:
            return True
        return self.monitor_op(current, self.best_k_models[self.kth_best_model])
    def on_epoch_end(self, epoch, logs=None):
        logs = logs or {}
-        self.epochs_since_last_check += 1
+        self.epochs_since_last_save += 1
-
+        if self.epochs_since_last_save >= self.period:
-        if self.save_top_k == 0:
+            self.epochs_since_last_save = 0
-            # no models are saved
+            filepath = '{}/{}_ckpt_epoch_{}.ckpt'.format(self.filepath, self.prefix, epoch + 1)
-            return
+            if self.save_best_only:
        if self.epochs_since_last_check >= self.period:
            self.epochs_since_last_check = 0
            filepath = f'{self.filepath}/{self.prefix}_ckpt_epoch_{epoch}.ckpt'
            version_cnt = 0
            while os.path.isfile(filepath):
                # this epoch called before
                filepath = f'{self.filepath}/{self.prefix}_ckpt_epoch_{epoch}_v{version_cnt}.ckpt'
                version_cnt += 1
            if self.save_top_k != -1:
                current = logs.get(self.monitor)
                if current is None:
-                    warnings.warn(
+                    print('Can save best model only with %s available,'
-                        f'Can save best model only with {self.monitor} available,'
+                          ' skipping.' % (self.monitor), RuntimeWarning)
                        ' skipping.', RuntimeWarning)
                else:
-                    if self.check_monitor_top_k(current):
+                    if self.monitor_op(current, self.best):
                        # remove kth
                        if len(self.best_k_models.keys()) == self.save_top_k:
                            delpath = self.kth_best_model
                            self.best_k_models.pop(self.kth_best_model)
                            self._del_model(delpath)
                        self.best_k_models[filepath] = current
                        if len(self.best_k_models.keys()) == self.save_top_k:
                            # monitor dict has reached k elements
                            if self.mode == 'min':
                                self.kth_best_model = max(self.best_k_models, key=self.best_k_models.get)
                            else:
                                self.kth_best_model = min(self.best_k_models, key=self.best_k_models.get)
                            self.kth_value = self.best_k_models[self.kth_best_model]
                        if self.mode == 'min':
                            self.best = min(self.best_k_models.values())
                        else:
                            self.best = max(self.best_k_models.values())
                        if self.verbose > 0:
-                            logging.info(
+                            print('\nEpoch %05d: %s improved from %0.5f to %0.5f,'
-                                f'\nEpoch {epoch:05d}: {self.monitor} reached'
+                                  ' saving model to %s'
-                                f' {current:0.5f} (best {self.best:0.5f}), saving model to'
+                                  % (epoch + 1, self.monitor, self.best,
-                                f' {filepath} as top {self.save_top_k}')
+                                     current, filepath))
-                        self._save_model(filepath)
+                        self.best = current
                        self.save_model(filepath, overwrite=True)
                    else:
                        if self.verbose > 0:
-                            logging.info(
+                            print('\nEpoch %05d: %s did not improve' %
-                                f'\nEpoch {epoch:05d}: {self.monitor}'
+                                  (epoch + 1, self.monitor))
                                f' was not in top {self.save_top_k}')
            else:
                if self.verbose > 0:
-                    logging.info(f'\nEpoch {epoch:05d}: saving model to {filepath}')
+                    print('\nEpoch %05d: saving model to %s' % (epoch + 1, filepath))
-                self._save_model(filepath)
+                self.save_model(filepath, overwrite=False)
 class GradientAccumulationScheduler(Callback):
    """Change gradient accumulation factor according to scheduling.
    # Arguments
        scheduling: dict, scheduling in format {epoch: accumulation_factor}
    """
    def __init__(self, scheduling: dict):
        if scheduling == {}:  # empty dict error
            raise TypeError("Empty dict cannot be interpreted correct")
@@ -363,11 +285,11 @@ class GradientAccumulationScheduler(Callback):
                break
-# if __name__ == '__main__':
+if __name__ == '__main__':
-#     c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
+    c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
-#     losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
+    losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
-#     for i, loss in enumerate(losses):
+    for i, loss in enumerate(losses):
-#         should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
+        should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
-#         logging.info(loss)
+        print(loss)
-#         if should_stop:
+        if should_stop:
-#             break
+            break
@@ -1,150 +0,0 @@
 """
 Lightning Module interface
 ==========================
 A lightning module is a strict superclass of nn.Module, it provides a standard interface
 for the trainer to interact with the model.
 The easiest thing to do is copy the minimal example below and modify accordingly.
 Otherwise, to Define a Lightning Module, implement the following methods:
 Minimal example
 ---------------
 .. code-block:: python
    import os
    import torch
    from torch.nn import functional as F
    from torch.utils.data import DataLoader
    from torchvision.datasets import MNIST
    import torchvision.transforms as transforms
    import pytorch_lightning as pl
    class CoolModel(pl.LightningModule):
        def __init__(self):
            super(CoolModel, self).__init__()
            # not the best model...
            self.l1 = torch.nn.Linear(28 * 28, 10)
        def forward(self, x):
            return torch.relu(self.l1(x.view(x.size(0), -1)))
        def training_step(self, batch, batch_idx):
            # REQUIRED
            x, y = batch
            y_hat = self.forward(x)
            return {'loss': F.cross_entropy(y_hat, y)}
        def validation_step(self, batch, batch_idx):
            # OPTIONAL
            x, y = batch
            y_hat = self.forward(x)
            return {'val_loss': F.cross_entropy(y_hat, y)}
        def validation_end(self, outputs):
            # OPTIONAL
            avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
            return {'avg_val_loss': avg_loss}
        def test_step(self, batch, batch_idx):
            # OPTIONAL
            x, y = batch
            y_hat = self.forward(x)
            return {'test_loss': F.cross_entropy(y_hat, y)}
        def test_end(self, outputs):
            # OPTIONAL
            avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
            return {'avg_test_loss': avg_loss}
        def configure_optimizers(self):
            # REQUIRED
            return torch.optim.Adam(self.parameters(), lr=0.02)
        @pl.data_loader
        def train_dataloader(self):
            return DataLoader(MNIST(os.getcwd(), train=True, download=True,
                              transform=transforms.ToTensor()), batch_size=32)
        @pl.data_loader
        def val_dataloader(self):
            # OPTIONAL
            # can also return a list of val dataloaders
            return DataLoader(MNIST(os.getcwd(), train=True, download=True,
                              transform=transforms.ToTensor()), batch_size=32)
        @pl.data_loader
        def test_dataloader(self):
            # OPTIONAL
            # can also return a list of test dataloaders
            return DataLoader(MNIST(os.getcwd(), train=False, download=True,
                              transform=transforms.ToTensor()), batch_size=32)
 How do these methods fit into the broader training?
 ---------------------------------------------------
 The LightningModule interface is on the right. Each method corresponds
 to a part of a research project. Lightning automates everything not in blue.
 .. figure::  docs/source/_static/images/overview_flat.jpg
   :align:   center
   Overview.
 Optional Methods
 ----------------
 **add_model_specific_args**
 .. code-block:: python
    @staticmethod
    def add_model_specific_args(parent_parser, root_dir)
 Lightning has a list of default argparse commands.
 This method is your chance to add or modify commands specific to your model.
 The `hyperparameter argument parser
  <https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser>`_
 is available anywhere in your model by calling self.hparams.
 **Return**
 An argument parser
 **Example**
 .. code-block:: python
    @staticmethod
    def add_model_specific_args(parent_parser, root_dir):
        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
        # param overwrites
        # parser.set_defaults(gradient_clip_val=5.0)
        # network params
        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
        parser.add_argument('--in_features', default=28*28)
        parser.add_argument('--out_features', default=10)
        # use 500 for CPU, 50000 for GPU to see speed difference
        parser.add_argument('--hidden_dim', default=50000)
        # data
        parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
        # training params (opt)
        parser.opt_list('--learning_rate', default=0.001, type=float,
                        options=[0.0001, 0.0005, 0.001, 0.005], tunable=False)
        parser.opt_list('--batch_size', default=256, type=int,
                        options=[32, 64, 128, 256], tunable=False)
        parser.opt_list('--optimizer_name', default='adam', type=str,
                        options=['adam'], tunable=False)
        return parser
 """
@@ -1,153 +0,0 @@
 """
 # Hooks
 There are cases when you might want to do something different at different parts of the training/validation loop.
 To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
 **Contributing** If there's a hook you'd like to add, simply:
 1. Fork PyTorchLightning.
 2. Add the hook :py:mod:`pytorch_lightning.base_module.hooks.py`.
 3. Add the correct place in the :py:mod:`pytorch_lightning.models.trainer` where it should be called.
 """
 import torch
 try:
    from apex import amp
    APEX_AVAILABLE = True
 except ImportError:
    APEX_AVAILABLE = False
 class ModelHooks(torch.nn.Module):
    def on_sanity_check_start(self):
        """
        Called before starting evaluate
        .. warning:: will be deprecated.
        :return:
        """
        pass
    def on_train_start(self):
        """Called at the beginning of training before sanity check
        :return:
        """
        # do something at the start of training
        pass
    def on_train_end(self):
        """
        Called at the end of training before logger experiment is closed
        :return:
        """
        # do something at the end of training
        pass
    def on_batch_start(self, batch):
        """Called in the training loop before anything happens for that batch.
        :param batch:
        :return:
        """
        # do something when the batch starts
        pass
    def on_batch_end(self):
        """Called in the training loop after the batch."""
        # do something when the batch ends
        pass
    def on_epoch_start(self):
        """Called in the training loop at the very beginning of the epoch."""
        # do something when the epoch starts
        pass
    def on_epoch_end(self):
        """Called in the training loop at the very end of the epoch."""
        # do something when the epoch ends
        pass
    def on_pre_performance_check(self):
        """Called at the very beginning of the validation loop."""
        # do something before validation starts
        pass
    def on_post_performance_check(self):
        """Called at the very end of the validation loop."""
        # do something before validation end
        pass
    def on_before_zero_grad(self, optimizer):
        """Called after optimizer.step() and before optimizer.zero_grad()
        Called in the training loop after taking an optimizer step and before zeroing grads.
        Good place to inspect weight information with weights updated.
        for optimizer in optimizers::
            optimizer.step()
            model.on_before_zero_grad(optimizer) # < ---- called here
            optimizer.zero_grad
        :param optimizer:
        :return:
        """
        # do something with the optimizer or inspect it.
        pass
    def on_after_backward(self):
        """Called after loss.backward() and before optimizers do anything.
        :return:
        Called in the training loop after model.backward()
        This is the ideal place to inspect or log gradient information
        .. code-block:: python
            def on_after_backward(self):
                # example to inspect gradient information in tensorboard
                if self.trainer.global_step % 25 == 0:  # don't make the tf file huge
                    params = self.state_dict()
                    for k, v in params.items():
                        grads = v
                        name = k
                        self.logger.experiment.add_histogram(tag=name, values=grads,
                                                             global_step=self.trainer.global_step)
        """
        pass
    def backward(self, use_amp, loss, optimizer):
        """Override backward with your own implementation if you need to
        :param use_amp: Whether amp was requested or not
        :param loss: Loss is already scaled by accumulated grads
        :param optimizer: Current optimizer being used
        :return:
        Called to perform backward step.
        Feel free to override as needed.
        The loss passed in has already been scaled for accumulated gradients if requested.
        .. code-block:: python
            def backward(self, use_amp, loss, optimizer):
                if use_amp:
                    with amp.scale_loss(loss, optimizer) as scaled_loss:
                        scaled_loss.backward()
                else:
                    loss.backward()
        """
        if use_amp:
            with amp.scale_loss(loss, optimizer) as scaled_loss:
                scaled_loss.backward()
        else:
            loss.backward()
@@ -1,10 +0,0 @@
 """
 .. warning:: `model_saving` module has been renamed to `saving` since v0.5.3 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`model_saving` module has been renamed to `saving` since v0.5.3"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.core.saving import ModelIO  # noqa: E402
@@ -1,10 +0,0 @@
 """
 .. warning:: `root_module` module has been renamed to `lightning` since v0.5.3 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`root_module` module has been renamed to `lightning` since v0.5.3"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.core.lightning import LightningModule  # noqa: E402
@@ -1,189 +1,10 @@
 """
 Lighting offers options for logging information about model, gpu usage, etc,
 via several different logging frameworks. It also offers printing options for training monitoring.
 **default_save_path**
 Lightning sets a default TestTubeLogger and CheckpointCallback for you which log to
 `os.getcwd()` by default. To modify the logging path you can set::
    Trainer(default_save_path='/your/path/to/save/checkpoints')
 If you need more custom behavior (different paths for both, different metrics, etc...)
 from the logger and the checkpointCallback, pass in your own instances as explained below.
 Setting up logging
 ------------------
 The trainer inits a default logger for you (TestTubeLogger). All logs will
 go to the current working directory under a folder named `os.getcwd()/lightning_logs`.
 If you want to modify the default logging behavior even more, pass in a logger
 (which should inherit from `LightningBaseLogger`).
 .. code-block:: python
    my_logger = MyLightningLogger(...)
    trainer = Trainer(logger=my_logger)
 The path in this logger will overwrite `default_save_path`.
 Lightning supports several common experiment tracking frameworks out of the box
 Custom logger
 -------------
 You can implement your own logger by writing a class that inherits from
 `LightningLoggerBase`. Use the `rank_zero_only` decorator to make sure that
 only the first process in DDP training logs data.
 .. code-block:: python
    from pytorch_lightning.logging import LightningLoggerBase, rank_zero_only
    class MyLogger(LightningLoggerBase):
        @rank_zero_only
        def log_hyperparams(self, params):
            # params is an argparse.Namespace
            # your code to record hyperparameters goes here
            pass
        @rank_zero_only
        def log_metrics(self, metrics, step):
            # metrics is a dictionary of metric names and values
            # your code to record metrics goes here
            pass
        def save(self):
            # Optional. Any code necessary to save logger data goes here
            pass
        @rank_zero_only
        def finalize(self, status):
            # Optional. Any code that needs to be run after training
            # finishes goes here
 If you write a logger than may be useful to others, please send
 a pull request to add it to Lighting!
 Using loggers
 -------------
 You can call the logger anywhere from your LightningModule by doing:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.whatever_method_summary_writer_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.add_histogram(...)
 Display metrics in progress bar
 -------------------------------
 .. code-block:: python
    # DEFAULT
    trainer = Trainer(show_progress_bar=True)
 Log metric row every k batches
 ------------------------------
 Every k batches lightning will make an entry in the metrics log
 .. code-block:: python
    # DEFAULT (ie: save a .csv log file every 10 batches)
    trainer = Trainer(row_log_interval=10)
 Log GPU memory
 --------------
 Logs GPU memory when metrics are logged.
 .. code-block:: python
    # DEFAULT
    trainer = Trainer(log_gpu_memory=None)
    # log only the min/max utilization
    trainer = Trainer(log_gpu_memory='min_max')
    # log all the GPU memory (if on DDP, logs only that node)
    trainer = Trainer(log_gpu_memory='all')
 Process position
 ----------------
 When running multiple models on the same machine we want to decide which progress bar to use.
 Lightning will stack progress bars according to this value.
 .. code-block:: python
    # DEFAULT
    trainer = Trainer(process_position=0)
    # if this is the second model on the node, show the second progress bar below
    trainer = Trainer(process_position=1)
 Save a snapshot of all hyperparameters
 --------------------------------------
 Automatically log hyperparameters stored in the `hparams` attribute as an `argparse.Namespace`
 .. code-block:: python
    class MyModel(pl.Lightning):
        def __init__(self, hparams):
            self.hparams = hparams
        ...
    args = parser.parse_args()
    model = MyModel(args)
    logger = TestTubeLogger(...)
    t = Trainer(logger=logger)
    trainer.fit(model)
 Write logs file to csv every k batches
 --------------------------------------
 Every k batches, lightning will write the new logs to disk
 .. code-block:: python
    # DEFAULT (ie: save a .csv log file every 100 batches)
    trainer = Trainer(log_save_interval=100)
 """
 from os import environ
 from .base import LightningLoggerBase, rank_zero_only
 from .tensorboard import TensorBoardLogger
 try:
-    from .test_tube import TestTubeLogger
+    from .test_tube_logger import TestTubeLogger
-except ImportError:
+except ModuleNotFoundError:
    pass
 try:
-    from .mlflow import MLFlowLogger
+    from .mlflow_logger import MLFlowLogger
-except ImportError:
+except ModuleNotFoundError:
    pass
 try:
    # needed to prevent ImportError and duplicated logs.
    environ["COMET_DISABLE_AUTO_LOGGING"] = "1"
    from .comet import CometLogger
 except ImportError:
    del environ["COMET_DISABLE_AUTO_LOGGING"]
@@ -1,9 +1,8 @@
 from abc import ABC
 from functools import wraps
 def rank_zero_only(fn):
-    """Decorate a logger method to run it only on the process with rank 0.
+    """Decorate a logger method to run it only on the process with rank 0
    :param fn: Function to decorate
    """
@@ -16,62 +15,57 @@ def rank_zero_only(fn):
    return wrapped_fn
-class LightningLoggerBase(ABC):
+class LightningLoggerBase(object):
-    """Base class for experiment loggers."""
+    """Base class for experiment loggers"""
    def __init__(self):
        self._rank = 0
-    @property
+    def log_metrics(self, metrics, step_num):
-    def experiment(self):
+        """Record metrics
        raise NotImplementedError()
-    def log_metrics(self, metrics, step):
+        :param metric: Dictionary with metric names as keys and measured
-        """Record metrics.
+        quanties as values
-
+        :param step_num: Step number at which the metrics should be recorded
        :param float metric: Dictionary with metric names as keys and measured quanties as values
        :param int|None step: Step number at which the metrics should be recorded
        """
        raise NotImplementedError()
    def log_hyperparams(self, params):
-        """Record hyperparameters.
+        """Record hyperparameters
        :param params: argparse.Namespace containing the hyperparameters
        """
        raise NotImplementedError()
    def save(self):
-        """Save log data."""
+        """Save log data"""
        pass
    def finalize(self, status):
-        """Do any processing that is necessary to finalize an experiment.
+        """Do any processing that is necessary to finalize an experiment
        :param status: Status that the experiment finished with (e.g. success, failed, aborted)
        """
        pass
    def close(self):
-        """Do any cleanup that is necessary to close an experiment."""
+        """Do any cleanup that is necessary to close an experiment"""
        pass
    @property
    def rank(self):
-        """Process rank. In general, metrics should only be logged by the process with rank 0."""
+        """
        Process rank. In general, metrics should only be logged by the process
        with rank 0
        """
        return self._rank
    @rank.setter
    def rank(self, value):
-        """Set the process rank."""
+        """Set the process rank"""
        self._rank = value
    @property
    def name(self):
        """Return the experiment name."""
        raise NotImplementedError("Sub-classes must provide a name property")
    @property
    def version(self):
-        """Return the experiment version."""
+        """Return the experiment version"""
-        raise NotImplementedError("Sub-classes must provide a version property")
+        return None
@@ -1,174 +0,0 @@
 """
 Log using `comet <https://www.comet.ml>`_
 Comet logger can be used in either online or offline mode.
 To log in online mode, CometLogger requries an API key:
 .. code-block:: python
    from pytorch_lightning.logging import CometLogger
    # arguments made to CometLogger are passed on to the comet_ml.Experiment class
    comet_logger = CometLogger(
        api_key=os.environ["COMET_KEY"],
        workspace=os.environ["COMET_WORKSPACE"], # Optional
        project_name="default_project", # Optional
        rest_api_key=os.environ["COMET_REST_KEY"], # Optional
        experiment_name="default" # Optional
    )
    trainer = Trainer(logger=comet_logger)
 To log in offline mode, CometLogger requires a path to a local directory:
 .. code-block:: python
    from pytorch_lightning.logging import CometLogger
    # arguments made to CometLogger are passed on to the comet_ml.Experiment class
    comet_logger = CometLogger(
        save_dir=".",
        workspace=os.environ["COMET_WORKSPACE"], # Optional
        project_name="default_project", # Optional
        rest_api_key=os.environ["COMET_REST_KEY"], # Optional
        experiment_name="default" # Optional
    )
    trainer = Trainer(logger=comet_logger)
 Use the logger anywhere in you LightningModule as follows:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.whatever_comet_ml_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.whatever_comet_ml_supports(...)
 """
 from logging import getLogger
 try:
    from comet_ml import Experiment as CometExperiment
    from comet_ml import OfflineExperiment as CometOfflineExperiment
    try:
        from comet_ml.api import API
    except ImportError:
        # For more information, see: https://www.comet.ml/docs/python-sdk/releases/#release-300
        from comet_ml.papi import API
 except ImportError:
    raise ImportError('Missing comet_ml package.')
 from torch import is_tensor
 from .base import LightningLoggerBase, rank_zero_only
 from ..utilities.debugging import MisconfigurationException
 logger = getLogger(__name__)
 class CometLogger(LightningLoggerBase):
    def __init__(self, api_key=None, save_dir=None, workspace=None,
                 rest_api_key=None, project_name=None, experiment_name=None, **kwargs):
        """Initialize a Comet.ml logger.
        Requires either an API Key (online mode) or a local directory path (offline mode)
        :param str api_key: Required in online mode. API key, found on Comet.ml
        :param str save_dir: Required in offline mode. The path for the directory to save local comet logs
        :param str workspace: Optional. Name of workspace for this user
        :param str project_name: Optional. Send your experiment to a specific project.
        Otherwise will be sent to Uncategorized Experiments.
        If project name does not already exists Comet.ml will create a new project.
        :param str rest_api_key: Optional. Rest API key found in Comet.ml settings.
        This is used to determine version number
        :param str experiment_name: Optional. String representing the name for this particular experiment on Comet.ml
        """
        super().__init__()
        self._experiment = None
        # Determine online or offline mode based on which arguments were passed to CometLogger
        if save_dir is not None and api_key is not None:
            # If arguments are passed for both save_dir and api_key, preference is given to online mode
            self.mode = "online"
            self.api_key = api_key
        elif api_key is not None:
            self.mode = "online"
            self.api_key = api_key
        elif save_dir is not None:
            self.mode = "offline"
            self.save_dir = save_dir
        else:
            # If neither api_key nor save_dir are passed as arguments, raise an exception
            raise MisconfigurationException("CometLogger requires either api_key or save_dir during initialization.")
        logger.info(f"CometLogger will be initialized in {self.mode} mode")
        self.workspace = workspace
        self.project_name = project_name
        self._kwargs = kwargs
        if rest_api_key is not None:
            # Comet.ml rest API, used to determine version number
            self.rest_api_key = rest_api_key
            self.comet_api = API(self.rest_api_key)
        else:
            self.rest_api_key = None
            self.comet_api = None
        if experiment_name:
            try:
                self.name = experiment_name
            except TypeError as e:
                logger.exception("Failed to set experiment name for comet.ml logger")
    @property
    def experiment(self):
        if self._experiment is not None:
            return self._experiment
        if self.mode == "online":
            self._experiment = CometExperiment(
                api_key=self.api_key,
                workspace=self.workspace,
                project_name=self.project_name,
                **self._kwargs
            )
        else:
            self._experiment = CometOfflineExperiment(
                offline_directory=self.save_dir,
                workspace=self.workspace,
                project_name=self.project_name,
                **self._kwargs
            )
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        self.experiment.log_parameters(vars(params))
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        # Comet.ml expects metrics to be a dictionary of detached tensors on CPU
        for key, val in metrics.items():
            if is_tensor(val):
                metrics[key] = val.cpu().detach()
        self.experiment.log_metrics(metrics, step=step)
    @rank_zero_only
    def finalize(self, status):
        self.experiment.end()
    @property
    def name(self):
        return self.experiment.project_name
    @name.setter
    def name(self, value):
        self.experiment.set_name(value)
    @property
    def version(self):
        return self.experiment.id
@@ -1,10 +0,0 @@
 """
 .. warning:: `comet_logger` module has been renamed to `comet` since v0.5.3 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`comet_logger` module has been renamed to `comet` since v0.5.3"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.logging.comet import CometLogger  # noqa: E402
@@ -1,100 +0,0 @@
 """
 Log using `mlflow <https://mlflow.org>'_
 .. code-block:: python
    from pytorch_lightning.logging import MLFlowLogger
    mlf_logger = MLFlowLogger(
        experiment_name="default",
        tracking_uri="file:/."
    )
    trainer = Trainer(logger=mlf_logger)
 Use the logger anywhere in you LightningModule as follows:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.whatever_ml_flow_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.whatever_ml_flow_supports(...)
 """
 from logging import getLogger
 from time import time
 try:
    import mlflow
 except ImportError:
    raise ImportError('Missing mlflow package.')
 from .base import LightningLoggerBase, rank_zero_only
 logger = getLogger(__name__)
 class MLFlowLogger(LightningLoggerBase):
    def __init__(self, experiment_name, tracking_uri=None, tags=None):
        super().__init__()
        self._mlflow_client = mlflow.tracking.MlflowClient(tracking_uri)
        self.experiment_name = experiment_name
        self._run_id = None
        self.tags = tags
    @property
    def experiment(self):
        return self._mlflow_client
    @property
    def run_id(self):
        if self._run_id is not None:
            return self._run_id
        expt = self._mlflow_client.get_experiment_by_name(self.experiment_name)
        if expt:
            self._expt_id = expt.experiment_id
        else:
            logger.warning(f"Experiment with name f{self.experiment_name} not found. Creating it.")
            self._expt_id = self._mlflow_client.create_experiment(name=self.experiment_name)
        run = self._mlflow_client.create_run(experiment_id=self._expt_id, tags=self.tags)
        self._run_id = run.info.run_id
        return self._run_id
    @rank_zero_only
    def log_hyperparams(self, params):
        for k, v in vars(params).items():
            self.experiment.log_param(self.run_id, k, v)
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        timestamp_ms = int(time() * 1000)
        for k, v in metrics.items():
            if isinstance(v, str):
                logger.warning(
                    f"Discarding metric with string value {k}={v}"
                )
                continue
            self.experiment.log_metric(self.run_id, k, v, timestamp_ms, step)
    def save(self):
        pass
    @rank_zero_only
    def finalize(self, status="FINISHED"):
        if status == 'success':
            status = 'FINISHED'
        self.experiment.set_terminated(self.run_id, status)
    @property
    def name(self):
        return self.experiment_name
    @property
    def version(self):
        return self._run_id
@@ -1,10 +1,56 @@
-"""
+from time import time
-.. warning:: `mlflow_logger` module has been renamed to `mlflow` since v0.5.3 and will be removed in v0.8.0
+from logging import getLogger
 """
-import warnings
+import mlflow
-warnings.warn("`mlflow_logger` module has been renamed to `mlflow` since v0.5.3"
+from .base import LightningLoggerBase, rank_zero_only
              " and will be removed in v0.8.0", DeprecationWarning)
-from pytorch_lightning.logging.mlflow import MLFlowLogger  # noqa: E402
+logger = getLogger(__name__)
 class MLFlowLogger(LightningLoggerBase):
    def __init__(self, experiment_name, tracking_uri=None):
        super().__init__()
        self.client = mlflow.tracking.MlflowClient(tracking_uri)
        self.experiment_name = experiment_name
        self._run_id = None
    @property
    def run_id(self):
        if self._run_id is not None:
            return self._run_id
        experiment = self.client.get_experiment_by_name(self.experiment_name)
        if experiment is None:
            logger.warning(
                f"Experiment with name f{self.experiment_name} not found. Creating it."
            )
            self.client.create_experiment(self.experiment_name)
            experiment = self.client.get_experiment_by_name(self.experiment_name)
        run = self.client.create_run(experiment.experiment_id)
        self._run_id = run.info.run_id
        return self._run_id
    @rank_zero_only
    def log_hyperparams(self, params):
        for k, v in vars(params).items():
            self.client.log_param(self.run_id, k, v)
    @rank_zero_only
    def log_metrics(self, metrics, step_num=None):
        timestamp_ms = int(time() * 1000)
        for k, v in metrics.items():
            if isinstance(v, str):
                logger.warning(
                    f"Discarding metric with string value {k}={v}"
                )
                continue
            self.client.log_metric(self.run_id, k, v, timestamp_ms, step_num)
    def save(self):
        pass
    @rank_zero_only
    def finalize(self, status="FINISHED"):
        self.client.set_terminated(self.run_id, status)
@@ -1,114 +0,0 @@
 import os
 from warnings import warn
 import torch
 from pkg_resources import parse_version
 from torch.utils.tensorboard import SummaryWriter
 from .base import LightningLoggerBase, rank_zero_only
 class TensorBoardLogger(LightningLoggerBase):
    r"""Log to local file system in TensorBoard format
    Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to
    `os.path.join(save_dir, name, version)`
    :example:
    .. code-block:: python
        logger = TensorBoardLogger("tb_logs", name="my_model")
        trainer = Trainer(logger=logger)
        trainer.train(model)
    :param str save_dir: Save directory
    :param str name: Experiment name. Defaults to "default".
    :param int version: Experiment version. If version is not specified the logger inspects the save
        directory for existing versions, then automatically assigns the next available version.
    :param \**kwargs: Other arguments are passed directly to the :class:`SummaryWriter` constructor.
    """
    def __init__(self, save_dir, name="default", version=None, **kwargs):
        super().__init__()
        self.save_dir = save_dir
        self._name = name
        self._version = version
        self._experiment = None
        self.kwargs = kwargs
    @property
    def experiment(self):
        """The underlying :class:`torch.utils.tensorboard.SummaryWriter`.
        :rtype: torch.utils.tensorboard.SummaryWriter
        """
        if self._experiment is not None:
            return self._experiment
        root_dir = os.path.join(self.save_dir, self.name)
        os.makedirs(root_dir, exist_ok=True)
        log_dir = os.path.join(root_dir, str(self.version))
        self._experiment = SummaryWriter(log_dir=log_dir, **self.kwargs)
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        if parse_version(torch.__version__) < parse_version("1.3.0"):
            warn(
                f"Hyperparameter logging is not available for Torch version {torch.__version__}."
                " Skipping log_hyperparams. Upgrade to Torch 1.3.0 or above to enable"
                " hyperparameter logging."
            )
            # TODO: some alternative should be added
            return
        try:
            # in case converting from namespace, todo: rather test if it is namespace
            params = vars(params)
        except TypeError:
            pass
        if params is not None:
            # `add_hparams` requires both - hparams and metric
            self.experiment.add_hparams(hparam_dict=dict(params), metric_dict={})
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        for k, v in metrics.items():
            if isinstance(v, torch.Tensor):
                v = v.item()
            self.experiment.add_scalar(k, v, step)
    @rank_zero_only
    def save(self):
        try:
            self.experiment.flush()
        except AttributeError:
            # you are using PT version (<v1.2) which does not have implemented flush
            self.experiment._get_file_writer().flush()
    @rank_zero_only
    def finalize(self, status):
        self.save()
    @property
    def name(self):
        return self._name
    @property
    def version(self):
        if self._version is None:
            self._version = self._get_next_version()
        return self._version
    def _get_next_version(self):
        root_dir = os.path.join(self.save_dir, self.name)
        existing_versions = [
            int(d) for d in os.listdir(root_dir) if os.path.isdir(os.path.join(root_dir, d)) and d.isdigit()
        ]
        if len(existing_versions) == 0:
            return 0
        else:
            return max(existing_versions) + 1
@@ -1,140 +0,0 @@
 """
 Log using `test tube <https://williamfalcon.github.io/test-tube>'_. Test tube logger is
 a strict subclass of `PyTorch SummaryWriter <https://pytorch.org/docs/stable/tensorboard.html>`_, refer to their
 documentation for all supported operations. The TestTubeLogger adds a nicer folder structure
 to manage experiments and snapshots all hyperparameters you pass to a LightningModule.
 .. code-block:: python
    from pytorch_lightning.logging import TestTubeLogger
    tt_logger = TestTubeLogger(
        save_dir=".",
        name="default",
        debug=False,
        create_git_tag=False
    )
    trainer = Trainer(logger=tt_logger)
 Use the logger anywhere in you LightningModule as follows:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.whatever_method_summary_writer_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.add_histogram(...)
 """
 try:
    from test_tube import Experiment
 except ImportError:
    raise ImportError('Missing test-tube package.')
 from .base import LightningLoggerBase, rank_zero_only
 class TestTubeLogger(LightningLoggerBase):
    __test__ = False
    def __init__(
            self, save_dir, name="default", description=None, debug=False,
            version=None, create_git_tag=False
    ):
        super().__init__()
        self.save_dir = save_dir
        self._name = name
        self.description = description
        self.debug = debug
        self._version = version
        self.create_git_tag = create_git_tag
        self._experiment = None
    @property
    def experiment(self):
        if self._experiment is not None:
            return self._experiment
        self._experiment = Experiment(
            save_dir=self.save_dir,
            name=self._name,
            debug=self.debug,
            version=self.version,
            description=self.description,
            create_git_tag=self.create_git_tag,
            rank=self.rank,
        )
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.experiment.argparse(params)
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.experiment.log(metrics, global_step=step)
    @rank_zero_only
    def save(self):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.experiment.save()
    @rank_zero_only
    def finalize(self, status):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.save()
        self.close()
    @rank_zero_only
    def close(self):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        exp = self.experiment
        exp.close()
    @property
    def rank(self):
        return self._rank
    @rank.setter
    def rank(self, value):
        self._rank = value
        if self._experiment is not None:
            self.experiment.rank = value
    @property
    def name(self):
        if self._experiment is None:
            return self._name
        else:
            return self.experiment.name
    @property
    def version(self):
        if self._experiment is None:
            return self._version
        else:
            return self.experiment.version
    # Test tube experiments are not pickleable, so we need to override a few
    # methods to get DDP working. See
    # https://docs.python.org/3/library/pickle.html#handling-stateful-objects
    # for more info.
    def __getstate__(self):
        state = self.__dict__.copy()
        state["_experiment"] = self.experiment.get_meta_copy()
        return state
    def __setstate__(self, state):
        self._experiment = state["_experiment"].get_non_ddp_exp()
        del state["_experiment"]
        self.__dict__.update(state)
@@ -1,10 +1,93 @@
-"""
+import os.path
-.. warning:: `test_tube_logger` module has been renamed to `test_tube` since v0.5.3 and will be removed in v0.8.0
+from copy import copy
 """
-import warnings
+from .base import LightningLoggerBase, rank_zero_only
-warnings.warn("`test_tube_logger` module has been renamed to `test_tube` since v0.5.3"
+from test_tube import Experiment
              " and will be removed in v0.8.0", DeprecationWarning)
-from pytorch_lightning.logging.test_tube import TestTubeLogger  # noqa: E402
+
 class TestTubeLogger(LightningLoggerBase):
    __test__ = False
    def __init__(
        self, save_dir, name="default", description=None, debug=False,
            version=None, create_git_tag=False
    ):
        super().__init__()
        self.save_dir = save_dir
        self.name = name
        self.description = description
        self.debug = debug
        self._version = version
        self.create_git_tag = create_git_tag
        self._experiment = None
    @property
    def experiment(self):
        if self._experiment is not None:
            return self._experiment
        self._experiment = Experiment(
            save_dir=self.save_dir,
            name=self.name,
            debug=self.debug,
            version=self.version,
            description=self.description,
            create_git_tag=self.create_git_tag,
            rank=self.rank,
        )
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        self.experiment.argparse(params)
    @rank_zero_only
    def log_metrics(self, metrics, step_num=None):
        self.experiment.log(metrics, global_step=step_num)
    @rank_zero_only
    def save(self):
        self.experiment.save()
    @rank_zero_only
    def finalize(self, status):
        self.save()
        self.close()
    def close(self):
        self.experiment.close()
    @property
    def rank(self):
        if self._experiment is None:
            return self._rank
        else:
            return self.experiment.rank
    @rank.setter
    def rank(self, value):
        if self._experiment is None:
            self._rank = value
        else:
            return self.experiment.rank
    @property
    def version(self):
        if self._experiment is None:
            return self._version
        else:
            return self.experiment.version
    # Test tube experiments are not pickleable, so we need to override a few
    # methods to get DDP working. See
    # https://docs.python.org/3/library/pickle.html#handling-stateful-objects
    # for more info.
    def __getstate__(self):
        state = self.__dict__.copy()
        state["_experiment"] = self.experiment.get_meta_copy()
        return state
    def __setstate__(self, state):
        self._experiment = state["_experiment"].get_non_ddp_exp()
        del state["_experiment"]
        self.__dict__.update(state)
@@ -1,12 +0,0 @@
 """
 .. warning:: `override_data_parallel` module has been renamed to `data_parallel` since v0.5.3
 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`override_data_parallel` module has been renamed to `data_parallel` since v0.5.3"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.overrides.data_parallel import (  # noqa: E402
    get_a_var, parallel_apply, LightningDataParallel, LightningDistributedDataParallel)
@@ -1,10 +0,0 @@
 """
 .. warning:: `pt_overrides` package has been renamed to `overrides` since v0.5.3 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`pt_overrides` package has been renamed to `overrides` since v0.5.3"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.overrides import override_data_parallel  # noqa: E402
@@ -1,11 +1,11 @@
 import itertools
 import threading
 from itertools import chain
 import torch
 from torch.cuda._utils import _get_device_index
 from torch.nn import DataParallel
 from torch.nn.parallel import DistributedDataParallel
 import itertools
 from itertools import chain
 import threading
 import torch
 from torch.cuda._utils import _get_device_index
 def _find_tensors(obj):  # pragma: no cover
@@ -1,11 +0,0 @@
 """
 .. warning:: `root_module` package has been renamed to `core` since v0.5.3 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`root_module` package has been renamed to `core` since v0.5.3"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.core import (  # noqa: E402
    decorators, grads, hooks, root_module, memory, model_saving)
@@ -17,9 +17,9 @@ def data_loader(fn):
            try:
                value = fn(self)  # Lazy evaluation, done only once.
                if (
-                        value is not None and
+                    value is not None and
-                        not isinstance(value, list) and
+                    not isinstance(value, list) and
-                        fn.__name__ in ['test_dataloader', 'val_dataloader']
+                    fn.__name__ in['test_dataloader', 'val_dataloader']
                ):
                    value = [value]
            except AttributeError as e:
@@ -0,0 +1,50 @@
 import torch
 class ModelHooks(torch.nn.Module):
    def on_sanity_check_start(self):
        """
        Called before starting evaluate
        :return:
        """
        pass
    def on_batch_start(self, batch):
        pass
    def on_batch_end(self):
        pass
    def on_epoch_start(self):
        pass
    def on_epoch_end(self):
        pass
    def on_pre_performance_check(self):
        pass
    def on_post_performance_check(self):
        pass
    def on_before_zero_grad(self, optimizer):
        """
        Called after optimizer.step() and before optimizer.zero_grad()
        for optimizer in optimizers:
            optimizer.step()
            model.on_before_zero_grad(optimizer) # < ---- called here
            optimizer.zero_grad
        :param optimizer:
        :return:
        """
        pass
    def on_after_backward(self):
        """
        Called after loss.backward() and before optimizers do anything
        :return:
        """
        pass
@@ -3,23 +3,20 @@ Generates a summary of a model's layers and dimensionality
 '''
 import gc
 import os
 import subprocess
 import torch
 import subprocess
 import numpy as np
 import pandas as pd
 import torch
 import logging
 class ModelSummary(object):
-    def __init__(self, model, mode='full'):
+    def __init__(self, model):
        '''
        Generates summaries of model layers and dimensions.
        '''
        self.model = model
        self.mode = mode
        self.in_sizes = []
        self.out_sizes = []
@@ -31,50 +28,29 @@ class ModelSummary(object):
    def __repr__(self):
        return self.summary.__str__()
    def named_modules(self):
        if self.mode == 'full':
            mods = self.model.named_modules()
            mods = list(mods)[1:]  # do not include root module (LightningModule)
        elif self.mode == 'top':
            # the children are the top-level modules
            mods = self.model.named_children()
        else:
            mods = []
        return list(mods)
    def get_variable_sizes(self):
        '''Run sample input through each layer to get output sizes'''
-        mods = self.named_modules()
+        mods = list(self.model.modules())
        in_sizes = []
        out_sizes = []
        input_ = self.model.example_input_array
        if self.model.on_gpu:
-            device = next(self.model.parameters()).get_device()
+            input_ = input_.cuda(0)
            # test if input is a list or a tuple
            if isinstance(input_, (list, tuple)):
                input_ = [input_i.cuda(device) if torch.is_tensor(input_i) else input_i
                          for input_i in input_]
            else:
                input_ = input_.cuda(device)
        if self.model.trainer.use_amp:
-            # test if it is not a list or a tuple
+            input_ = input_.half()
            if isinstance(input_, (list, tuple)):
                input_ = [input_i.half() if torch.is_tensor(input_i) else input_i
                          for input_i in input_]
            else:
                input_ = input_.half()
        with torch.no_grad():
-            for _, m in mods:
+            for i in range(1, len(mods)):
-                if isinstance(input_, (list, tuple)):  # pragma: no cover
+                m = mods[i]
                if type(input_) is list or type(input_) is tuple:  # pragma: no cover
                    out = m(*input_)
                else:
                    out = m(input_)
-                if isinstance(input_, (list, tuple)):  # pragma: no cover
+                if type(input_) is tuple or type(input_) is list:  # pragma: no cover
                    in_size = []
                    for x in input_:
                        if type(x) is list:
@@ -86,7 +62,7 @@ class ModelSummary(object):
                in_sizes.append(in_size)
-                if isinstance(out, (list, tuple)):  # pragma: no cover
+                if type(out) is tuple or type(out) is list:  # pragma: no cover
                    out_size = np.asarray([x.size() for x in out])
                else:
                    out_size = np.array(out.size())
@@ -96,17 +72,16 @@ class ModelSummary(object):
        self.in_sizes = in_sizes
        self.out_sizes = out_sizes
        assert len(in_sizes) == len(out_sizes)
        return
    def get_layer_names(self):
        '''Collect Layer Names'''
-        mods = self.named_modules()
+        mods = list(self.model.named_modules())
        names = []
        layers = []
-        for name, m in mods:
+        for m in mods[1:]:
-            names += [name]
+            names += [m[0]]
-            layers += [str(m.__class__)]
+            layers += [str(m[1].__class__)]
        layer_types = [x.split('.')[-1][:-2] for x in layers]
@@ -116,9 +91,11 @@ class ModelSummary(object):
    def get_parameter_sizes(self):
        '''Get sizes of all parameters in `model`'''
-        mods = self.named_modules()
+        mods = list(self.model.modules())
        sizes = []
-        for _, m in mods:
+
        for i in range(1, len(mods)):
            m = mods[i]
            p = list(m.parameters())
            modsz = []
            for j in range(len(p)):
@@ -156,9 +133,9 @@ class ModelSummary(object):
        df['Name'] = self.layer_names
        df['Type'] = self.layer_types
        df['Params'] = self.param_nums
        df['Params'] = df['Params'].map(get_human_readable_count)
        if self.model.example_input_array is not None:
            df['In_sizes'] = self.in_sizes
            df['Out_sizes'] = self.out_sizes
@@ -179,26 +156,26 @@ def print_mem_stack():  # pragma: no cover
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
-                logging.info(type(obj), obj.size())
+                print(type(obj), obj.size())
        except Exception:
            pass
 def count_mem_items():  # pragma: no cover
-    num_params = 0
+    nb_params = 0
-    num_tensors = 0
+    nb_tensors = 0
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                obj_type = str(type(obj))
                if 'parameter' in obj_type:
-                    num_params += 1
+                    nb_params += 1
                else:
-                    num_tensors += 1
+                    nb_tensors += 1
        except Exception:
            pass
-    return num_params, num_tensors
+    return nb_params, nb_tensors
 def get_memory_profile(mode):
@@ -211,10 +188,19 @@ def get_memory_profile(mode):
    memory_map = get_gpu_memory_map()
    if mode == 'min_max':
-        min_index, min_memory = min(memory_map.items(), key=lambda item: item[1])
+        min_mem = 1000000
-        max_index, max_memory = max(memory_map.items(), key=lambda item: item[1])
+        min_k = None
        max_mem = 0
        max_k = None
        for k, v in memory_map:
            if v > max_mem:
                max_mem = v
                max_k = k
            if v < min_mem:
                min_mem = v
                min_k = k
-        memory_map = {min_index: min_memory, max_index: max_memory}
+        memory_map = {min_k: min_mem, max_k: max_mem}
    return memory_map
@@ -228,41 +214,15 @@ def get_gpu_memory_map():
        Keys are device ids as integers.
        Values are memory usage as integers in MB.
    """
-    result = subprocess.run(
+    result = subprocess.check_output(
        [
-            'nvidia-smi',
+            'nvidia-smi', '--query-gpu=memory.used',
-            '--query-gpu=memory.used',
+            '--format=csv,nounits,noheader'
-            '--format=csv,nounits,noheader',
+        ], encoding='utf-8')
        ],
        encoding='utf-8',
        capture_output=True,
        check=True)
    # Convert lines into a dictionary
-    gpu_memory = [int(x) for x in result.stdout.strip().split(os.linesep)]
+    gpu_memory = [int(x) for x in result.strip().split('\n')]
-    gpu_memory_map = {f'gpu_{index}': memory for index, memory in enumerate(gpu_memory)}
+    gpu_memory_map = {}
    for k, v in zip(range(len(gpu_memory)), gpu_memory):
        k = f'gpu_{k}'
        gpu_memory_map[k] = v
    return gpu_memory_map
 def get_human_readable_count(number):
    """
    Abbreviates an integer number with K, M, B, T for thousands, millions,
    billions and trillions, respectively.
    Examples:
        123     -> 123
        1234    -> 1 K       (one thousand)
        2e6     -> 2 M       (two million)
        3e9     -> 3 B       (three billion)
        4e12    -> 4 T       (four trillion)
        5e15    -> 5,000 T
    :param number: a positive integer number
    :returns a string formatted according to the pattern described above.
    """
    assert number >= 0
    labels = [' ', 'K', 'M', 'B', 'T']
    num_digits = int(np.floor(np.log10(number)) + 1 if number > 0 else 1)
    num_groups = int(np.ceil(num_digits / 3))
    num_groups = min(num_groups, len(labels))  # don't abbreviate beyond trillions
    shift = -3 * (num_groups - 1)
    number = number * (10 ** shift)
    index = num_groups - 1
    return f'{int(number):,d} {labels[index]}'
@@ -0,0 +1,172 @@
 import torch
 from pytorch_lightning.root_module.memory import ModelSummary
 from pytorch_lightning.root_module.grads import GradInformation
 from pytorch_lightning.trainer.trainer_io import load_hparams_from_tags_csv
 from pytorch_lightning.root_module.model_saving import ModelIO
 from pytorch_lightning.root_module.hooks import ModelHooks
 from pytorch_lightning.root_module.decorators import data_loader
 class LightningModule(GradInformation, ModelIO, ModelHooks):
    def __init__(self, *args, **kwargs):
        super(LightningModule, self).__init__(*args, **kwargs)
        self.dtype = torch.FloatTensor
        self.exp_save_path = None
        self.current_epoch = 0
        self.global_step = 0
        self.loaded_optimizer_states_dict = {}
        self.trainer = None
        self.logger = None
        self.example_input_array = None
        # track if gpu was requested for checkpointing
        self.on_gpu = False
        self.use_dp = False
        self.use_ddp = False
        self.use_ddp2 = False
        self.use_amp = False
    def forward(self, *args, **kwargs):
        """
        Expand model in into whatever you need.
        Also need to return the target
        :param x:
        :return:
        """
        raise NotImplementedError
    def training_step(self, *args, **kwargs):
        """
        return loss, dict with metrics for tqdm
        :param called with batch, batch_nb
        additional: optimizer_i if multiple optimizers used
        :return:
        """
        raise NotImplementedError
    def validation_step(self, *args, **kwargs):
        """
        return whatever outputs will need to be aggregated in validation_end
        OPTIONAL
        :param called with batch, batch_nb
        additional: dataset_i if multiple val datasets used
        :return:
        """
        pass
    def test_step(self, *args, **kwargs):
        """
        return whatever outputs will need to be aggregated in test_end
        OPTIONAL
        :param called with batch, batch_nb
        additional: dataset_i if multiple val datasets used
        :return:
        """
        pass
    def validation_end(self, outputs):
        """
        Outputs has the appended output after each validation step
        OPTIONAL
        :param outputs:
        :return: dic_with_metrics for tqdm
        """
        pass
    def test_end(self, outputs):
        """
        Outputs has the appended output after each test step
        OPTIONAL
        :param outputs:
        :return: dic_with_metrics for tqdm
        """
        pass
    def configure_optimizers(self):
        """
        Return a list of optimizers and a list of schedulers (could be empty)
        :return:
        """
        raise NotImplementedError
    def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i, second_order_closure=None):
        """
        Do something instead of the standard optimizer behavior
        :param epoch_nb:
        :param batch_nb:
        :param optimizer:
        :param optimizer_i:
        :param second_order_closure: closure for second order methods
        :return:
        """
        if isinstance(optimizer, torch.optim.LBFGS):
            optimizer.step(second_order_closure)
        else:
            optimizer.step()
        # clear gradients
        optimizer.zero_grad()
    @data_loader
    def train_dataloader(self):
        """
        Implement a PyTorch DataLoader
        :return:
        """
        raise NotImplementedError
    @data_loader
    def test_dataloader(self):
        """
        Implement a PyTorch DataLoader
        :return:
        """
        return None
    @data_loader
    def val_dataloader(self):
        """
        Implement a PyTorch DataLoader
        :return:
        """
        return None
    @classmethod
    def load_from_metrics(cls, weights_path, tags_csv):
        """
        Primary way of loading model from csv weights path
        :param weights_path:
        :param tags_csv:
        :param map_location: dic for mapping storage {'cuda:1':'cuda:0'}
        :return:
        """
        hparams = load_hparams_from_tags_csv(tags_csv)
        hparams.__setattr__('on_gpu', False)
        # load on CPU only to avoid OOM issues
        # then its up to user to put back on GPUs
        checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)
        # load the state_dict on the model automatically
        model = cls(hparams)
        model.load_state_dict(checkpoint['state_dict'])
        # give model a chance to load something
        model.on_load_checkpoint(checkpoint)
        return model
    def summarize(self):
        model_summary = ModelSummary(self)
        print(model_summary)
    def freeze(self):
        for param in self.parameters():
            param.requires_grad = False
    def unfreeze(self):
        for param in self.parameters():
            param.requires_grad = True
@@ -1,6 +1,6 @@
-from .model import LightningTestModel
+from .lm_test_module import LightningTestModel
-from .model_base import LightningTestModelBase
+from .lm_test_module_base import LightningTestModelBase
-from .model_mixins import (
+from .lm_test_module_mixins import (
    LightningValidationStepMixin,
    LightningValidationMixin,
    LightningValidationStepMultipleDataloadersMixin,
@@ -0,0 +1,27 @@
 import os
 from collections import OrderedDict
 import torch
 import torch.nn as nn
 import torch.nn.functional as F
 from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
 from torchvision.datasets import MNIST
 from torchvision import transforms
 from test_tube import HyperOptArgumentParser
 from pytorch_lightning.root_module.root_module import LightningModule
 from pytorch_lightning import data_loader
 from .lm_test_module_base import LightningTestModelBase
 from .lm_test_module_mixins import LightningValidationMixin, LightningTestMixin
 class LightningTestModel(LightningValidationMixin, LightningTestMixin, LightningTestModelBase):
    """
    Most common test case. Validation and test dataloaders
    """
    def on_training_metrics(self, logs):
        logs['some_tensor_to_test'] = torch.rand(1)
@@ -7,32 +7,12 @@ import torch.nn.functional as F
 from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
 from torchvision import transforms
 from torchvision.datasets import MNIST
-try:
+from torchvision import transforms
-    from test_tube import HyperOptArgumentParser
+from test_tube import HyperOptArgumentParser
 except ImportError:
    # TODO: this should be discussed and moved out of this package
    raise ImportError('Missing test-tube package.')
 from pytorch_lightning.root_module.root_module import LightningModule
 from pytorch_lightning import data_loader
 from pytorch_lightning.core.lightning import LightningModule
 class TestingMNIST(MNIST):
    def __init__(self, root, train=True, transform=None, target_transform=None,
                 download=False, num_samples=8000):
        super(TestingMNIST, self).__init__(
            root,
            train=train,
            transform=transform,
            target_transform=target_transform,
            download=download
        )
        # take just a subset of MNIST dataset
        self.data = self.data[:num_samples]
        self.targets = self.targets[:num_samples]
 class LightningTestModelBase(LightningModule):
@@ -121,15 +101,14 @@ class LightningTestModelBase(LightningModule):
            loss_val = loss_val.unsqueeze(0)
        # alternate possible outputs to test
-        if self.trainer.batch_idx % 1 == 0:
+        if self.trainer.batch_nb % 1 == 0:
            output = OrderedDict({
                'loss': loss_val,
-                'progress_bar': {'some_val': loss_val * loss_val},
+                'progress_bar': {'some_val': loss_val * loss_val}
                'log': {'train_some_val': loss_val * loss_val},
            })
            return output
-        if self.trainer.batch_idx % 2 == 0:
+        if self.trainer.batch_nb % 2 == 0:
            return loss_val
    # ---------------------
@@ -153,8 +132,8 @@ class LightningTestModelBase(LightningModule):
        # init data generators
        transform = transforms.Compose([transforms.ToTensor(),
                                        transforms.Normalize((0.5,), (1.0,))])
-        dataset = TestingMNIST(root=self.hparams.data_root, train=train,
+        dataset = MNIST(root=self.hparams.data_root, train=train,
-                               transform=transform, download=True, num_samples=2000)
+                        transform=transform, download=True)
        # when using multi-node we need to add the datasampler
        train_sampler = None
@@ -1,7 +1,17 @@
 import os
 from collections import OrderedDict
 import torch
 import torch.nn as nn
 import torch.nn.functional as F
 from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
 from torchvision.datasets import MNIST
 from torchvision import transforms
 from test_tube import HyperOptArgumentParser
 from pytorch_lightning.root_module.root_module import LightningModule
 from pytorch_lightning import data_loader
@@ -80,13 +90,13 @@ class LightningValidationMixin(LightningValidationStepMixin):
            val_loss = output['val_loss']
            # reduce manually when using dp
-            if self.trainer.use_dp or self.trainer.use_ddp2:
+            if self.trainer.use_dp:
                val_loss = torch.mean(val_loss)
            val_loss_mean += val_loss
            # reduce manually when using dp
            val_acc = output['val_acc']
-            if self.trainer.use_dp or self.trainer.use_ddp2:
+            if self.trainer.use_dp:
                val_acc = torch.mean(val_acc)
            val_acc_mean += val_acc
@@ -95,7 +105,7 @@ class LightningValidationMixin(LightningValidationStepMixin):
        val_acc_mean /= len(outputs)
        tqdm_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-        results = {'progress_bar': tqdm_dict, 'log': tqdm_dict}
+        results = {'progress_bar': tqdm_dict}
        return results
@@ -1,13 +0,0 @@
 import torch
 from .model_base import LightningTestModelBase
 from .model_mixins import LightningValidationMixin, LightningTestMixin
 class LightningTestModel(LightningValidationMixin, LightningTestMixin, LightningTestModelBase):
    """
    Most common test case. Validation and test dataloaders
    """
    def on_training_metrics(self, logs):
        logs['some_tensor_to_test'] = torch.rand(1)
@@ -1,19 +0,0 @@
 """
 # Trainer
 The lightning trainer abstracts best practices for running a training, val, test routine.
 It calls parts of your model when it wants to hand over full control and otherwise makes
 training assumptions which are now standard practice in AI research.
 This is the basic use of the trainer:
 .. code-block:: python
    from pytorch_lightning import Trainer
    model = LightningTemplate()
    trainer = Trainer()
    trainer.fit(model)
 """
@@ -1,27 +0,0 @@
 from abc import ABC
 try:
    from apex import amp
    APEX_AVAILABLE = True
 except ImportError:
    APEX_AVAILABLE = False
 import logging
 class TrainerAMPMixin(ABC):
    def init_amp(self, use_amp):
        self.use_amp = use_amp and APEX_AVAILABLE
        if self.use_amp:
            logging.info('using 16bit precision')
        if use_amp and not APEX_AVAILABLE:  # pragma: no cover
            msg = """
            You set `use_amp=True` but do not have apex installed.
            Install apex first using this guide and rerun with use_amp=True:
            https://github.com/NVIDIA/apex#linux
            this run will NOT use 16 bit precision
            """
            raise ModuleNotFoundError(msg)
@@ -1,82 +0,0 @@
 import os
 from abc import ABC
 from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
 from pytorch_lightning.logging import TestTubeLogger
 class TrainerCallbackConfigMixin(ABC):
    def __init__(self):
        # this is just a summary on variables used in this abstract class,
        #  the proper values/initialisation should be done in child class
        self.default_save_path = None
        self.save_checkpoint = None
        self.slurm_job_id = None
    def configure_checkpoint_callback(self):
        """
        Weight path set in this priority:
        Checkpoint_callback's path (if passed in).
        User provided weights_saved_path
        Otherwise use os.getcwd()
        """
        if self.checkpoint_callback is True:
            # init a default one
            if self.logger is not None:
                ckpt_path = os.path.join(
                    self.default_save_path,
                    self.logger.name,
                    f'version_{self.logger.version}',
                    "checkpoints"
                )
            else:
                ckpt_path = os.path.join(self.default_save_path, "checkpoints")
            self.checkpoint_callback = ModelCheckpoint(
                filepath=ckpt_path
            )
        elif self.checkpoint_callback is False:
            self.checkpoint_callback = None
        if self.checkpoint_callback:
            # set the path for the callbacks
            self.checkpoint_callback.save_function = self.save_checkpoint
            # if checkpoint callback used, then override the weights path
            self.weights_save_path = self.checkpoint_callback.filepath
        # if weights_save_path is still none here, set to current working dir
        if self.weights_save_path is None:
            self.weights_save_path = self.default_save_path
    def configure_early_stopping(self, early_stop_callback, logger):
        if early_stop_callback is True:
            self.early_stop_callback = EarlyStopping(
                monitor='val_loss',
                patience=3,
                verbose=True,
                mode='min'
            )
            self.enable_early_stop = True
        elif not early_stop_callback:
            self.early_stop_callback = None
            self.enable_early_stop = False
        else:
            self.early_stop_callback = early_stop_callback
            self.enable_early_stop = True
        # configure logger
        if logger is True:
            # default logger
            self.logger = TestTubeLogger(
                save_dir=self.default_save_path,
                version=self.slurm_job_id,
                name='lightning_logs'
            )
            self.logger.rank = 0
        elif logger is False:
            self.logger = None
        else:
            self.logger = logger
            self.logger.rank = 0
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
William Falcon	8c00c4a3cd	release v0.5.1.2	2019-10-06 00:08:17 -04:00
William Falcon	3b9e97fb96	Merge branch 'master' into docs	2019-10-05 23:51:36 -04:00
William Falcon	438a4f74d0	cleaned up test_tube logger	2019-10-05 23:46:09 -04:00
William Falcon	d0a9c92326	cleaned up test_tube logger	2019-10-05 23:35:10 -04:00
William Falcon	d0a2438e22	cleaned up test_tube logger	2019-10-05 23:23:38 -04:00
William Falcon	4e1c90d892	cleaning up docs	2019-10-05 20:50:40 -04:00
William Falcon	dccab6ce19	cleaning up demos	2019-10-05 16:38:48 -04:00
William Falcon	602236ecf2	cleaning up demos	2019-10-05 16:35:39 -04:00
William Falcon	f1caeb49cc	cleaning up demos	2019-10-05 16:33:49 -04:00
William Falcon	1c45edd192	cleaning up demos	2019-10-05 16:30:27 -04:00
William Falcon	dd85e1c600	cleaning up demos	2019-10-05 16:26:16 -04:00
William Falcon	ae8f6866bb	cleaning up demos	2019-10-05 16:25:17 -04:00
William Falcon	35d85b2de1	cleaning up demos	2019-10-05 16:24:02 -04:00
William Falcon	d63c19539f	cleaning up demos	2019-10-05 16:21:19 -04:00
William Falcon	0d86302e05	cleaning up demos	2019-10-05 16:19:53 -04:00
William Falcon	d179ad6d1d	cleaning up demos	2019-10-05 16:16:48 -04:00
William Falcon	221c10c8c3	cleaning up demos	2019-10-05 16:14:05 -04:00
William Falcon	90faf9118c	cleaning up demos	2019-10-05 16:12:29 -04:00
William Falcon	a00e24ed80	cleaning up demos	2019-10-05 16:09:48 -04:00
William Falcon	5c99da5861	cleaning up demos	2019-10-05 16:07:00 -04:00
William Falcon	d3ca836024	cleaning up demos	2019-10-05 16:05:42 -04:00
William Falcon	d6c79644af	cleaning up demos	2019-10-05 16:01:27 -04:00
William Falcon	56179bdbbb	cleaning up demos	2019-10-05 16:00:32 -04:00
William Falcon	a3df04f049	cleaning up demos	2019-10-05 15:57:31 -04:00
William Falcon	170262ff73	cleaning up demos	2019-10-05 15:54:48 -04:00
William Falcon	18f58d4bf4	cleaning up demos	2019-10-05 15:52:52 -04:00
William Falcon	7b982f2280	cleaning up demos	2019-10-05 15:48:15 -04:00
William Falcon	7b5a40b170	cleaning up demos	2019-10-05 15:45:41 -04:00
William Falcon	e7f15e4e89	cleaning up demos	2019-10-05 15:39:37 -04:00
William Falcon	c0c1cf058f	cleaning up demos	2019-10-05 15:36:12 -04:00
William Falcon	6be5fb3cf7	cleaning up demos	2019-10-05 15:32:54 -04:00
William Falcon	45f36d3e74	cleaning up demos	2019-10-05 15:21:32 -04:00
William Falcon	0994cd9b09	cleaning up demos	2019-10-05 15:20:54 -04:00
William Falcon	f02856c0bc	cleaning up demos	2019-10-05 15:15:28 -04:00
William Falcon	67de4c241e	cleaning up demos	2019-10-05 15:14:34 -04:00
William Falcon	a7e154e643	cleaning up demos	2019-10-05 15:08:39 -04:00
William Falcon	f2fb219bfe	cleaning up demos	2019-10-05 15:05:23 -04:00
William Falcon	e2bf0cda07	cleaning up demos	2019-10-05 15:04:22 -04:00
William Falcon	149628a003	cleaning up demos	2019-10-05 15:02:46 -04:00
William Falcon	ac4216f95c	cleaning up demos	2019-10-05 15:00:58 -04:00