release v0.4.8

Expectopatronum implement #89 (#182 )
* rename validate -> evaluate; implement test logic; allow multiple test_loaders * add test_step and test_end to LightningModule * add in_test_mode to pretraining to implement case 2 (test pretrained model) * fix code style issues * LightningTestModel: add optional second test set, implement test_step and test_end * implemented test for multiple test_dataloaders; fixed typo * add two test cases for #89 * add documentation for test_step, test_end; fix computation of loss in validation_step example * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Update trainer.py * Added proper dp ddp routing calls for test mode * Update trainer.py * Update test_models.py * Update trainer.py * Update trainer.py * Update override_data_parallel.py * Update test_models.py * Update test_models.py * Update trainer.py * Update trainer.py * Update trainer.py * Update test_models.py * Update test_models.py * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * Update trainer.py * Update override_data_parallel.py * Update debug.py * Update lm_test_module.py * Update test_models.py
2026-09-12 12:40:20 +08:00 · 2019-08-31 03:05:57 -04:00 · 2019-08-30 18:56:09 -04:00
151 changed files with 7057 additions and 12498 deletions
@@ -1,116 +0,0 @@
 # Python CircleCI 2.0 configuration file
 #
 # Check https://circleci.com/docs/2.0/language-python/ for more details
 #
 version: 2.0
 references:
  install_deps: &install_deps
    run:
      name: Install Dependences
      command: |
        pip install "$TORCH_VERSION" --user
        # this is temporal fix til test-tube is not merged and released
        pip install -r requirements.txt --user
        sudo pip install pytest pytest-cov pytest-flake8
        pip install -r ./tests/requirements.txt --user
  tests: &tests
   run:
     name: Testing
     command: |
       python --version ; pip --version ; pip list
       py.test pytorch_lightning tests pl_examples -v --doctest-modules --junitxml=test-reports/pytest_junit.xml
     no_output_timeout: 15m
  format: &format
   run:
     name: Formatting
     command: |
       python --version ; pip --version ; pip list
       flake8
  make_docs: &make_docs
   run:
     name: Make Documentation
     command: |
       # sudo apt-get install pandoc
       pip install -r requirements.txt --user
       sudo pip install -r docs/requirements.txt
       # sphinx-apidoc -o ./docs/source ./pytorch_lightning **/test_* --force --follow-links
       cd docs; make clean ; make html
 jobs:
  Build-Docs:
    docker:
      - image: circleci/python:3.7
    steps:
      - checkout
      - *make_docs
  Formatting:
    docker:
      - image: circleci/python:3.7
    environment:
        - TORCH_VERSION: "torch"
    steps:
      - checkout
      - *install_deps
      - *format
  PyTorch:
    docker:
      - image: circleci/python:3.7
    environment:
      - TORCH_VERSION: "torch"
    steps: &steps
      - checkout
      - *install_deps
      - *tests
      - store_test_results:
          path: test-reports
      - store_artifacts:
          path: test-reports
  PyTorch-v1.1:
    docker:
      - image: circleci/python:3.6
    environment:
      - TORCH_VERSION: "torch>=1.1, <1.2"
    steps: *steps
  PyTorch-v1.2:
    docker:
      - image: circleci/python:3.6
    environment:
      - TORCH_VERSION: "torch>=1.2, <1.3"
    steps: *steps
  PyTorch-v1.3:
    docker:
      - image: circleci/python:3.6
    environment:
      - TORCH_VERSION: "torch>=1.3, <1.4"
    steps: *steps
  PyTorch-v1.4:
    docker:
      - image: circleci/python:3.6
    environment:
      - TORCH_VERSION: "torch>=1.4, <1.5"
    steps: *steps
 workflows:
  version: 2
  build:
    jobs:
      - Formatting
      - Build-Docs
      - PyTorch-v1.1
      - PyTorch-v1.2
      - PyTorch-v1.3
      - PyTorch-v1.4
@@ -1,59 +0,0 @@
 # How to become a core contributor
 Thanks for your interest in joining the Lightning team! We’re a rapidly growing project which is poised to become the go-to framework for DL researchers! 
 We're currently recruiting for a team of 5 core maintainers. 
 As a core maintainer you will have a strong say in the direction of the project. Big changes will require a majority of maintainers to agree.
 ### Code of conduct  
 First and foremost, you'll be evaluated against [these core values](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md). Any code we commit or feature we add needs to align with those core values.
 ### The bar for joining the team   
 Lightning is being used to solve really hard problems at the top AI labs in the world. As such, the bar for adding team members is extremely high. Candidates must have solid engineering skills, have a good eye for user experience, and must be a power user of Lightning and PyTorch.
 With that said, the Lightning team will be diverse and a reflection of an inclusive AI community. You don't have to be an engineer to conntribute! Scientists with great usability intuition and PyTorch ninja skills are welcomed!
 ### Responsibilities:
 The responsibilities mainly revolve around 3 things.
 #### Github issues
 - Here we want to help users have an amazing experience. These range from questions from new people getting into DL to questions from researchers about doing something esoteric with Lightning
 Often, these issues require some sort of bug fix, document clarification or new functionality to be scoped out.
 - To become a core member you must resolve at least 10 Github issues which align with the API design goals for Lightning. By the end of these 10 issues I should feel comfortable in the way you answer user questions
 Pleasant/helpful tone.
 - Can abstract from that issue or bug into functionality that might solve other related issues or makes the platform more flexible.
 - Don’t make users feel like they don’t know what they’re doing. We’re here to help and to make everyone’s experience delightful.
 #### Pull requests
 - Here we need to ensure the code that enters Lightning is high quality. For each PR we need to:
 - Make sure code coverage does not decrease
 - Documents are updated
 - Code is elegant and simple
 - Code is NOT overly engineered or hard to read
 - Ask yourself, could a non-engineer understand what’s happening here?
 - Make sure new tests are written
 - Is this NECESSARY for Lightning? There are some PRs which are just purely about adding engineering complexity which have no place in Lightning. 
 Guidance
 - Some other PRs are for people who are wanting to get involved and add something unnecessary. We do want their help though! So don’t approve the PR, but direct them to a Github issue that they might be interested in helping with instead!
 - To be considered for core contributor, please review 10 PRs and help the authors land it on master. Once you've finished the review, ping me 
 for a sanity check. At the end of 10 PRs if your PR reviews are inline with expectations described above, then you can merge PRs on your own going forward,
 otherwise we'll do a few more until we're both comfortable :)
 #### Project directions
 There are some big decisions which the project must make. For these I expect core contributors to have something meaningful to add if it’s their area of expertise.
 #### Diversity
 Lightning should reflect the broader community it serves. As such we should have scientists/researchers from 
 different fields contributing!   
 The first 5 core contributors will fit this profile. Thus if you overlap strongly with experiences and expertise as someone else on the team, you might have to wait until the next set of contributors are added.
 #### Summary: Requirements to apply
 - Solve 10 Github issues. The goal is to be inline with expectations for solving issues by the last one so you can do them on your own. If not, I might ask you to solve a few more specific ones.
 - Do 10 PR reviews. The goal is to be inline with expectations for solving issues by the last one so you can do them on your own. If not, I might ask you to solve a few more specific ones.
 If you want to be considered, ping me on gitter and start [tracking your progress here](https://docs.google.com/spreadsheets/d/15D58gp8DvI0Z6qbbYVRuaWioiwzafcP58-UlbuO_CMU/edit?usp=sharing).
@@ -1,14 +1,15 @@
 # Contributing    
 Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!   
-## Main Core Value: One less thing to remember    
+## One less thing to remember    
 Simplify the API as much as possible from the user perspective. Any additions or improvements should minimize things the user needs to remember.   
-For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.  
+For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.   
 ## Lightning Design Principles   
 We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
-#### No PyTorch Interference   
+
 #### No PyTorch interference   
 We don't want to add any abstractions on top of pure PyTorch. This gives researchers all the control they need without having to learn yet another framework.    
 #### Simple Internal Code    
@@ -20,25 +21,17 @@ There are 1,000 ways to do something. However, something eventually becomes stan
 When something becomes a best practice, we add it to the framework. This likely looks like code in utils or in the model file that everyone keeps adding over and over again across projects. When this happens, bring that code inside the trainer and add a flag for it.
 #### Simple External API    
-What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others. Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people. 
+What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others. Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people.    
 #### Backward-compatible API   
 We all hate updating our deep learning packages because we don't want to refactor a bunch of stuff. In Lightning, we make sure every change we make which could break an API is backwards compatible with good deprecation warnings.
 You shouldn't be afraid to upgrade Lightning :)
 #### Gain User Trust    
 As a researcher you can't have any part of your code going wrong. So, make thorough tests that ensure an implementation of a new trick or subbtle change is correct.    
-#### Interoperability  
+## Contribution types    
 Have a favorite feature from other libraries like fast.ai or transformers? Those should just work with lightning as well. Grab your favorite model or learning rate scheduler from your favorite library and run it in Lightning.
 ## Contribution Types    
 Currently looking for help implementing new features or adding bug fixes. 
 A lot of good work has already been done in project mechanics (requirements.txt, setup.py, pep8, badges, ci, etc...) we're in a good state there thanks to all the early contributors (even pre-beta release)!   
-## Bug Fixes:  
+## Bug fixes:  
 1. Submit a github issue.   
 2. Fix it.  
 3. Submit a PR! 
@@ -47,7 +40,3 @@ A lot of good work has already been done in project mechanics (requirements.txt,
 1. Submit a github issue.   
 2. We'll agree on the feature scope.     
 3. Submit a PR! (with updated docs and tests 🙃).   
 ## Coding Styleguide
 1. Test the code with flake8.
 2. Use f-strings.
@@ -8,55 +8,29 @@ assignees: ''
 ---
 ### Common bugs:
-1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
+1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/williamFalcon/pytorch-lightning/issues/79).    
-2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
+2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
-## 🐛 Bug
+**Describe the bug**
-
+A clear and concise description of what the bug is.
 <!-- A clear and concise description of what the bug is. -->
 ### To Reproduce
 **To Reproduce**
 Steps to reproduce the behavior:
 1. Go to '...'
-2. Run '....'
+2. Click on '....'
 3. Scroll down to '....'
 4. See error
-<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
+**Expected behavior**
 A clear and concise description of what you expected to happen.
 **Screenshots**
 If applicable, add screenshots to help explain your problem.
-#### Code sample
+**Desktop (please complete the following information):**
-<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
+ - OS: [e.g. iOS]
-Minimal means having the shortest code but still preserving the bug. -->
+ - Browser [e.g. chrome, safari]
 - Version [e.g. 22]
-### Expected behavior
+**Additional context**
-
+Add any other context about the problem here.
 <!-- A clear and concise description of what you expected to happen. -->
 ### Environment
 Please copy and paste the output from our
 [environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
 (or fill out the checklist below manually).
 You can get the script and run it with:
 ```
 wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
 # For security purposes, please check the contents of collect_env.py before running it.
 python collect_env.py
 ```
 - PyTorch Version (e.g., 1.0):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:
 ### Additional context
 <!-- Add any other context about the problem here. -->
@@ -7,21 +7,14 @@ assignees: ''
 ---
-## 🚀 Feature
+**Is your feature request related to a problem? Please describe.**
-<!-- A clear and concise description of the feature proposal -->
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-### Motivation
+**Describe the solution you'd like**
 A clear and concise description of what you want to happen.
-<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
+**Describe alternatives you've considered**
 A clear and concise description of any alternative solutions or features you've considered.
-### Pitch
+**Additional context**
-
+Add any other context or screenshots about the feature request here.
 <!-- A clear and concise description of what you want to happen. -->
 ### Alternatives
 <!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
 ### Additional context
 <!-- Add any other context or screenshots about the feature request here. -->
@@ -7,24 +7,20 @@ assignees: ''
 ---
 ## ❓ Questions and Help
 ### Before asking:   
 1. search the issues.   
 2. search the docs.    
-<!-- If you still can't find what you need: -->
+If you still can't find what you need:     
 #### What is your question?    
-#### What is your question?
+#### Code    
 Please paste a code snippet if your question requires it!   
-#### Code
+#### What have you tried?    
-<!-- Please paste a code snippet if your question requires it! -->   
+#### What's your environment?   
-
+- conda version (no venv)   
-#### What have you tried?
+- PyTorch version   
-
+- Lightning version   
-#### What's your environment?
+- Test-tube version
 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]
@@ -7,12 +7,11 @@ assignees: ''
 ---
 ## 📚 Documentation
 For typos and doc fixes, please go ahead and:
 1. Create an issue.
 2. Fix the typo.   
 3. Submit a PR.
 Thanks!
@@ -1,16 +0,0 @@
 # Before submitting
 - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
 - [ ] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md)?
 - [ ] Did you make sure to update the docs?   
 - [ ] Did you write any new necessary tests?  
 ## What does this PR do?
 Fixes # (issue).
 ## PR review    
 Anyone in the community is free to review the PR once the tests have passed.     
 If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
 ## Did you have fun?
 Make sure you had fun coding 🙃
@@ -1,29 +1,25 @@
 # project
 .DS_Store
 .data/
 run_configs/
 test_tube_logs/
 test_tube_data/
 datasets/
 model_weights/
 app/models/
 pip-wheel-metadata/
 lightning_logs/
 # Test-tube
 test_tube_logs/
 test_tube_data/
 test_tube_exp/
-
+tests/tests_tt_dir/
-# Documentations
+tests/save_dir
-docs/source/pl_examples*.rst
+default/
 docs/source/pytorch_lightning*.rst
 tests/tests/
 /docs/source/*.md
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
 *$py.class
 example.py
 timit_data/
-
+LJSpeech-1.1/
 # C extensions
 *.so
@@ -32,6 +28,7 @@ timit_data/
 # Distribution / packaging
 .Python
 env/
 ide_layouts/
 build/
 develop-eggs/
@@ -43,6 +40,7 @@ lib/
 lib64/
 parts/
 sdist/
 var/
 wheels/
 *.egg-info/
 .installed.cfg
@@ -68,9 +66,6 @@ nosetests.xml
 coverage.xml
 *.cover
 .hypothesis/
 tests/tests_tt_dir/
 tests/save_dir
 tests/tests/
 # Translations
 *.mo
@@ -88,7 +83,7 @@ instance/
 .scrapy
 # Sphinx documentation
-docs/build/
+docs/_build/
 # PyBuilder
 target/
@@ -110,7 +105,6 @@ celerybeat-schedule
 # virtualenv
 .venv
 env/
 venv/
 ENV/
@@ -128,6 +122,4 @@ ENV/
 .mypy_cache/
 # data
 .data/
 datasets/
 mnist/
@@ -5,13 +5,9 @@
 # Required
 version: 2
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
  configuration: docs/source/conf.py
 # Build documentation with MkDocs
-#mkdocs:
+mkdocs:
-#  configuration: mkdocs.yml
+  configuration: mkdocs.yml
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: all
@@ -20,5 +16,4 @@ formats: all
 python:
  version: 3.7
  install:
-    - requirements: docs/requirements.txt
+    - requirements: docs/requirements.txt
    #- requirements: requirements.txt
@@ -1,10 +0,0 @@
 # use this to run tests
 rm -rf _ckpt_*
 rm -rf tests/save_dir*
 rm -rf tests/mlruns_*
 rm -rf tests/cometruns*
 rm -rf tests/wandb*
 rm -rf tests/tests/*
 rm -rf lightning_logs
 coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules
 coverage report -m
@@ -8,6 +8,8 @@
 # this file is *not* meant to cover or endorse the use of travis, but rather to
 # help confirm pull requests to this project.
 dist: xenial  # Ubuntu 16.04
 env:
  global:
    - DISPLAY=""
@@ -16,69 +18,23 @@ language: python
 matrix:
  include:
-    - dist: xenial  # Ubuntu 16.04
+    - python: 3.6
      python: 3.6
      env:
        - TOXENV=py36
        - MIN_REQUIREMENTS=1
    - dist: xenial  # Ubuntu 16.04
      python: 3.7
      env:
        - TOXENV=py37
        - MIN_REQUIREMENTS=1
    - dist: bionic  # Ubuntu 18.04
      python: 3.6
      env: TOXENV=py36
-    - dist: bionic  # Ubuntu 18.04
+    - python: 3.7
      python: 3.7
      env: TOXENV=py37
    - os: osx
      # https://blog.travis-ci.com/2019-08-07-extensive-python-testing-on-travis-ci
      osx_image: xcode10.3
      language: generic
      env: TOXENV=py37
      #addons:
      #  homebrew:
      #    # update: true
      #    packages: python3.7
      before_install:
        - pip3 install virtualenv
        - virtualenv -p python3 ~/venv
        - source ~/venv/bin/activate
 #    - os: windows
 #      language: minimal
 #      before_install:
 #        - choco install python3
 #        - export PATH="/c/Python37:/c/Python37/Scripts:$PATH"
 #      env: TOXENV=py37
 # See http://docs.travis-ci.com/user/caching/#pip-cache
 cache: pip
 install:
-  - pip install future  # needed for `builtins`
+  - pip install -r requirements.txt
-  - sudo pip install tox
+  - pip install -r ./tests/requirements.txt
-
+  - pip --version ; pip list
 before_script:
  # rewrite all minimal requirements as strict
  - if [[ "${MIN_REQUIREMENTS}" == "1" ]]; then
      python -c "req = open('requirements.txt').read().replace('>', '=') ; open('requirements-ci.txt', 'w').write(req)" ;
    else
      cp requirements.txt requirements-ci.txt ;
    fi
  - pip install -r requirements-ci.txt -U
 script:
  # integration
  - tox --sitepackages
-
+  - python setup.py install --dry-run
  #- python setup.py install --dry-run --user
  - virtualenv vEnv ;
    source vEnv/bin/activate
  - pip install --editable . ;
    cd .. & python -c "import pytorch_lightning ; print(pytorch_lightning.__version__)"
  - deactivate ;
    rm -rf vEnv
 after_success:
  - coverage report
@@ -1,6 +1,7 @@
 # Manifest syntax https://docs.python.org/2/distutils/sourcedist.html
 graft wheelhouse
 recursive-include birl *.py
 recursive-exclude __pycache__  *.py[cod] *.orig
 # Include the README
@@ -12,16 +13,11 @@ include LICENSE
 exclude *.sh
 exclude *.toml
 exclude *.svg    
 recursive-include examples *.py
 recursive-include pytorch_lightning *.py
 # include examples
 recursive-include pl_examples *.py
 recursive-include pl_examples *.md
 recursive-include pl_examples *.sh
 # exclude tests from package
 recursive-exclude tests *
 recursive-exclude site *
 exclude tests
 # Exclude the documentation files
@@ -36,7 +32,6 @@ exclude *.yml
 prune .git
 prune .github
 prune .circleci
 prune notebook*
 prune temp*
 prune test*
@@ -1,24 +1,22 @@
 <div align="center">
-![Logo](docs/source/_static/images/lightning_logo_small.png)
+![Logo](./docs/source/_static/lightning_logo_small.png)
 # PyTorch Lightning
-**The lightweight PyTorch wrapper for ML researchers. Scale your models. Write less boilerplate.**
+**The PyTorch Keras for ML researchers. More control. Less boilerplate.**
 [![PyPI Status](https://badge.fury.io/py/pytorch-lightning.svg)](https://badge.fury.io/py/pytorch-lightning)
 [![PyPI Status](https://pepy.tech/badge/pytorch-lightning)](https://pepy.tech/project/pytorch-lightning)
-[![Build Status](https://travis-ci.org/PytorchLightning/pytorch-lightning.svg?branch=master)](https://travis-ci.org/PytorchLightning/pytorch-lightning)
+[![Build Status](https://travis-ci.org/williamFalcon/pytorch-lightning.svg?branch=master)](https://travis-ci.org/williamFalcon/pytorch-lightning)
-[![Build status](https://ci.appveyor.com/api/projects/status/NEW-PROJECT-ID?svg=true)](https://ci.appveyor.com/project/PytorchLightning/pytorch-lightning)
+[![Build status](https://ci.appveyor.com/api/projects/status/rum89d7hq8l1kfye?svg=true)](https://ci.appveyor.com/project/Borda/pytorch-lightning)
-[![Coverage](docs/source/_static/images/coverage.svg)](https://github.com/PytorchLightning/pytorch-lightning/tree/master/tests#running-coverage)
+[![Coverage](https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/coverage.svg)](https://github.com/williamFalcon/pytorch-lightning/tree/master/tests#running-coverage)
 [![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
 [![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
-[![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/pytorch-lightning/shared_invite/enQtODU5ODIyNTUzODQwLTFkMDg5Mzc1MDBmNjEzMDgxOTVmYTdhYjA1MDdmODUyOTg2OGQ1ZWZkYTQzODhhNzdhZDA3YmNhMDhlMDY4YzQ)
+[![Gitter](https://badges.gitter.im/PyTorch-Lightning/community.svg)](https://gitter.im/PyTorch-Lightning/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/PytorchLightning/pytorch-lightning/blob/master/LICENSE)
+[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/williamFalcon/pytorch-lightning/blob/master/LICENSE)
 [![Next Release](https://img.shields.io/badge/Next%20Release-Feb%206-<COLOR>.svg)](https://shields.io/)
 <!-- 
 removed until codecov badge isn't empy. likely a config error showing nothing on master.
 [![codecov](https://codecov.io/gh/Borda/pytorch-lightning/branch/master/graph/badge.svg)](https://codecov.io/gh/Borda/pytorch-lightning)
@@ -32,189 +30,164 @@ pip install pytorch-lightning
 ```
 ## Docs   
-[jan 20, 2020]   
+**[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)**
 **[Old docs (some links might be broken)](https://pytorch-lightning.readthedocs.io/en/stable)
 ###### As a temporary hack, when you get the 404, replace williamfalcon.github.io with pytorchlightning.github.io.   
 **[New docs, CURRENTLY DEBUGING](https://pytorch-lightning.rtfd.io/en/latest)**   
 ## Demo  
 [Copy and run this COLAB!](https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=HOk9c4_35FKg)
 ## What is it?  
-Lightning is a very lightweight wrapper on PyTorch that decouples the science code from the engineering code. It's more of a style-guide than a framework. By refactoring your code, we can automate most of the non-research code.  
+Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.
 To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/PytorchLightning/pytorch-lightning#how-do-i-do-use-it) format (the science) and Lightning will automate the rest (the engineering). Lightning guarantees tested, correct, modern best practices for the automated parts.
 - If you are a researcher, Lightning is infinitely flexible, you can modify everything down to the way .backward is called or distributed is set up. 
 - If you are a scientist or production team, lightning is very simple to use with best practice defaults.
 ## What does lightning control for me?   
 Everything in Blue!   
 This is how lightning separates the science (red) from the engineering (blue).
 ![Overview](docs/source/_static/images/pl.gif)
 ## How much effort is it to convert?
 You're probably tired of switching frameworks at this point. But it is a very quick process to refactor into the Lightning format (ie: hours). [Check out this tutorial](https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538)
 ## Starting a new project?   
-[Use our seed-project aimed at reproducibility!](https://github.com/PytorchLightning/pytorch-lightning-conference-seed)     
+[Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     
 ## Why do I want to use lightning?
 Every research project starts the same, a model, a training loop, validation loop, etc. As your research advances, you're likely to need distributed training, 16-bit precision, checkpointing, gradient accumulation, etc.   
 Lightning sets up all the boilerplate state-of-the-art training for you so you can focus on the research.   
---
+---   
 ## README Table of Contents        
- [How do I use it](https://github.com/PytorchLightning/pytorch-lightning#how-do-i-do-use-it)     
+- [How do I use it](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it)     
- [What lightning automates](https://github.com/PytorchLightning/pytorch-lightning#what-does-lightning-control-for-me)    
+- [What lightning automates](https://github.com/williamFalcon/pytorch-lightning#what-does-lightning-control-for-me)    
- [Tensorboard integration](https://github.com/PytorchLightning/pytorch-lightning#tensorboard)    
+- [Tensorboard integration](https://github.com/williamFalcon/pytorch-lightning#tensorboard)    
- [Lightning features](https://github.com/PytorchLightning/pytorch-lightning#lightning-automates-all-of-the-following-each-is-also-configurable)    
+- [Lightning features](https://github.com/williamFalcon/pytorch-lightning#lightning-automates-all-of-the-following-each-is-also-configurable)    
- [Examples](https://github.com/PytorchLightning/pytorch-lightning#examples)    
+- [Demos](https://github.com/williamFalcon/pytorch-lightning#demo)    
- [Tutorials](https://github.com/PytorchLightning/pytorch-lightning#tutorials)
+- [Tutorials](https://github.com/williamFalcon/pytorch-lightning#tutorials)
- [Contributing](https://github.com/PytorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md)
+- [Contributing](https://github.com/williamFalcon/pytorch-lightning/blob/master/CONTRIBUTING.md)    
- [Bleeding edge install](https://github.com/PytorchLightning/pytorch-lightning#bleeding-edge)   
+- [Bleeding edge install](https://github.com/williamFalcon/pytorch-lightning#bleeding-edge)   
- [Lightning Design Principles](https://github.com/PytorchLightning/pytorch-lightning#lightning-design-principles)   
+- [Lightning Design Principles](https://github.com/williamFalcon/pytorch-lightning#lightning-design-principles)   
- [Asking for help](https://github.com/PytorchLightning/pytorch-lightning#asking-for-help)
+- [Asking for help](https://github.com/williamFalcon/pytorch-lightning#asking-for-help)
- [FAQ](https://github.com/PytorchLightning/pytorch-lightning#faq)    
+- [FAQ](https://github.com/williamFalcon/pytorch-lightning#faq)    
 ---
 ---   
 ## How do I do use it?   
-Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule](https://pytorch-lightning.rtfd.io/en/latest/LightningModule/RequiredTrainerInterface/) which you fit using a Trainer.
+Think about Lightning as refactoring your research code instead of using a new framework. The research code goes into a [LightningModule]((https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)) which you fit using a Trainer.    
 The LightningModule defines a *system* such as seq-2-seq, GAN, etc... It can ALSO define a simple classifier such as the example below.     
 To use lightning do 2 things:  
-1. [Define a LightningModule](https://pytorch-lightning.rtfd.io/en/latest/LightningModule/RequiredTrainerInterface/)
+1. [Define a LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/)         
-**WARNING:** This syntax is for version 0.5.0+ where abbreviations were removed.
+```python
-    ```python
+import os
-    import os
+import torch
-    
+from torch.nn import functional as F
-    import torch
+from torch.utils.data import DataLoader
-    from torch.nn import functional as F
+from torchvision.datasets import MNIST
-    from torch.utils.data import DataLoader
+import torchvision.transforms as transforms
    from torchvision.datasets import MNIST
    from torchvision import transforms
    import pytorch_lightning as pl
    class CoolSystem(pl.LightningModule):
        def __init__(self):
            super(CoolSystem, self).__init__()
            # not the best model...
            self.l1 = torch.nn.Linear(28 * 28, 10)
        def forward(self, x):
            return torch.relu(self.l1(x.view(x.size(0), -1)))
        def training_step(self, batch, batch_idx):
            # REQUIRED
            x, y = batch
            y_hat = self.forward(x)
            loss = F.cross_entropy(y_hat, y)
            tensorboard_logs = {'train_loss': loss}
            return {'loss': loss, 'log': tensorboard_logs}
        def validation_step(self, batch, batch_idx):
            # OPTIONAL
            x, y = batch
            y_hat = self.forward(x)
            return {'val_loss': F.cross_entropy(y_hat, y)}
        def validation_end(self, outputs):
            # OPTIONAL
            avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
            tensorboard_logs = {'val_loss': avg_loss}
            return {'avg_val_loss': avg_loss, 'log': tensorboard_logs}
        def test_step(self, batch, batch_idx):
            # OPTIONAL
            x, y = batch
            y_hat = self.forward(x)
            return {'test_loss': F.cross_entropy(y_hat, y)}
        def test_end(self, outputs):
            # OPTIONAL
            avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
            tensorboard_logs = {'test_loss': avg_loss}
            return {'avg_test_loss': avg_loss, 'log': tensorboard_logs}
        def configure_optimizers(self):
            # REQUIRED
            # can return multiple optimizers and learning_rate schedulers
            # (LBFGS it is automatically supported, no need for closure function)
            return torch.optim.Adam(self.parameters(), lr=0.02)
        @pl.data_loader
        def train_dataloader(self):
            # REQUIRED
            return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
        @pl.data_loader
        def val_dataloader(self):
            # OPTIONAL
            return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
        @pl.data_loader
        def test_dataloader(self):
            # OPTIONAL
            return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
    ```
 2. Fit with a [trainer](https://pytorch-lightning.rtfd.io/en/latest/Trainer/)    
    ```python
    from pytorch_lightning import Trainer
    model = CoolSystem()
    # most basic trainer, uses good defaults
    trainer = Trainer()    
    trainer.fit(model)   
    ```
-Trainer sets up a tensorboard logger, early stopping and checkpointing by default (you can modify all of them or
+import pytorch_lightning as pl
 use something other than tensorboard).   
-Here are more advanced examples
+class CoolSystem(pl.LightningModule):
    def __init__(self):
        super(CoolSystem, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)
    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))
    def training_step(self, batch, batch_nb):
        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
        return {'loss': F.cross_entropy(y_hat, y)}
    def validation_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
        return {'val_loss': F.cross_entropy(y_hat, y)}
    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}
    def configure_optimizers(self):
        # REQUIRED
        # can return multiple optimizers and learning_rate schedulers
        return torch.optim.Adam(self.parameters(), lr=0.02)
    @pl.data_loader
    def tng_dataloader(self):
        # REQUIRED
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
    @pl.data_loader
    def val_dataloader(self):
        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
    @pl.data_loader
    def test_dataloader(self):
        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
 2. Fit with a [trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)    
 ```python
 from pytorch_lightning import Trainer
 model = CoolSystem()
 # most basic trainer, uses good defaults
 trainer = Trainer()    
 trainer.fit(model)   
 ```     
 Or with tensorboard logger and some options turned on such as multi-gpu, etc...    
 ```python   
 from test_tube import Experiment    
 # PyTorch summarywriter with a few bells and whistles    
 exp = Experiment(save_dir=os.getcwd())
 # train on cpu using only 10% of the data (for demo purposes)
-trainer = Trainer(max_epochs=1, train_percent_check=0.1)
+# pass in experiment for automatic tensorboard logging.    
 trainer = Trainer(experiment=exp, max_nb_epochs=1, train_percent_check=0.1)
-# train on 4 gpus (lightning chooses GPUs for you)
+# train on 4 gpus
-# trainer = Trainer(max_epochs=1, gpus=4, distributed_backend='ddp')  
+# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3])
 # train on 4 gpus (you choose GPUs)
 # trainer = Trainer(max_epochs=1, gpus=[0, 1, 3, 7], distributed_backend='ddp')   
 # train on 32 gpus across 4 nodes (make sure to submit appropriate SLURM job)
-# trainer = Trainer(max_epochs=1, gpus=8, num_gpu_nodes=4, distributed_backend='ddp')
+# trainer = Trainer(experiment=exp, max_nb_epochs=1, gpus=[0, 1, 2, 3, 4, 5, 6, 7], nb_gpu_nodes=4)
 # train (1 epoch only here for demo)
 trainer.fit(model)
-# view tensorboard logs 
+# view tensorflow logs 
-logging.info(f'View tensorboard logs by running\ntensorboard --logdir {os.getcwd()}')
+print('View tensorboard logs by running\ntensorboard --logdir %s' % os.getcwd())
-logging.info('and going to http://localhost:6006 on your browser')
+print('and going to http://localhost:6006 on your browser')
-```
+```    
 ## What does lightning control for me?   
 Everything in gray!    
 You define the blue parts using the LightningModule interface:  
 ![Ouverview](./docs/source/_static/overview_flat.jpg)
 When you're all done you can even run the test set separately.   
 ```python
-trainer.test()
+# what to do in the training loop
 def training_step(self, data_batch, batch_nb):
 # what to do in the validation loop
 def validation_step(self, data_batch, batch_nb):
 # how to aggregate validation_step outputs
 def validation_end(self, outputs):
 # and your dataloaders
 def tng_dataloader():
 def val_dataloader():
 def test_dataloader():
 ```
 **Could be as complex as seq-2-seq + attention**    
 ```python
 # define what happens for training here
-def training_step(self, batch, batch_idx):
+def training_step(self, data_batch, batch_nb):
-    x, y = batch
+    x, y = data_batch
    # define your own forward and loss calculation
    hidden_states = self.encoder(x)
@@ -240,8 +213,8 @@ def training_step(self, batch, batch_idx):
 ```python
 # define what happens for validation here
-def validation_step(self, batch, batch_idx):    
+def validation_step(self, data_batch, batch_nb):    
-    x, y = batch
+    x, y = data_batch
    # or as basic as a CNN classification
    out = self.forward(x)
@@ -266,64 +239,144 @@ def validation_end(self, outputs):
    val_loss_mean /= len(outputs)
    val_acc_mean /= len(outputs)
-    logs = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
+    tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
-    result = {'log': logs}
+    return tqdm_dic
    return result
 ```
 ## Tensorboard    
-Lightning is fully integrated with tensorboard, MLFlow and supports any logging module.   
+Lightning is fully integrated with tensorboard.   
-![tensorboard-support](docs/source/_static/images/tf_loss.png)
+![tensorboard-support](./docs/source/_static/tf_loss.png)
 Lightning also adds a text column with all the hyperparameters for this experiment.      
-![tensorboard-support](docs/source/_static/images/tf_tags.png)
+![tensorboard-support](./docs/source/_static/tf_tags.png)
-## Lightning automates all of the following ([each is also configurable](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.trainer.html)):
+Simply note the path you set for the Experiment    
 ```python
 from test_tube import Experiment
 from pytorch-lightning import  Trainer
 exp = Experiment(save_dir='/some/path')
 trainer = Trainer(experiment=exp)
 ...
 ```   
 And run tensorboard from that dir   
 ```bash
 tensorboard --logdir /some/path     
 ```    
 ## Lightning automates all of the following ([each is also configurable](https://williamfalcon.github.io/pytorch-lightning/Trainer/)):
- [Running grid search on a cluster](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.trainer.distrib_data_parallel.html)  
+###### Checkpointing    
 - [Fast dev run](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.utilities.debugging.html)
 - [Logging](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.logging.html)
 - [Implement Your Own Distributed (DDP) training](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.core.lightning.html#pytorch_lightning.core.lightning.LightningModule.configure_ddp)
 - [Multi-GPU & Multi-node](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.trainer.distrib_parts.html)
 - [Training loop](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.trainer.training_loop.html)
 - [Hooks](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.core.hooks.html)
 - [Configure optimizers](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.core.lightning.html#pytorch_lightning.core.lightning.LightningModule.configure_optimizers)
 - [Validations](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.trainer.evaluation_loop.html)
 - [Model saving & Restoring training session](https://pytorch-lightning.rtfd.io/en/latest/pytorch_lightning.trainer.training_io.html)
-## Examples   
+- [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
- [GAN](https://github.com/PytorchLightning/pytorch-lightning/tree/master/pl_examples/domain_templates/gan.py)    
+- [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
- [MNIST](https://github.com/PytorchLightning/pytorch-lightning/tree/master/pl_examples/basic_examples)      
+- [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
- [Other projects using Lightning](https://github.com/PytorchLightning/pytorch-lightning/network/dependents?package_id=UGFja2FnZS0zNzE3NDU4OTM%3D)    
+
- [Multi-node](https://github.com/PytorchLightning/pytorch-lightning/tree/master/pl_examples/multi_node_examples)   
+###### Computing cluster (SLURM)    
 - [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
 - [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
 ###### Debugging  
 - [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
 - [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
 - [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
 - [Print which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
 - [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 ###### Distributed training    
 - [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
 - [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
 - [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
 - [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
 - [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
 ###### Experiment Logging   
 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
 - [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
 - [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
 - [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
 ###### Training loop    
 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)    
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 ###### Validation loop    
 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
 - [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
 - [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
 ## Demo
 ```bash
 # install lightning
 pip install pytorch-lightning
 # clone lightning for the demo
 git clone https://github.com/williamFalcon/pytorch-lightning.git
 cd pytorch-lightning
 cd examples/new_project_templates/
 # all of the following demos use the SAME model to show no modification needs to be made to your code
 # train on cpu 
 python single_cpu_template.py
 # train on multiple-gpus 
 python single_gpu_node_template.py --gpus "0,1"
 # train on 32 gpus on a cluster (run on a SLURM managed cluster)
 python multi_node_cluster_template.py --nb_gpu_nodes 4 --gpus '0,1,2,3,4,5,6,7'
 ```
 ## Tutorials   
 - [Basic Lightning use](https://towardsdatascience.com/supercharge-your-ai-research-with-pytorch-lightning-337948a99eec)    
 - [9 key speed features in Pytorch-Lightning](https://towardsdatascience.com/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565)    
 - [SLURM, multi-node training with Lightning](https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd)     
---
+---    
 ## Asking for help    
 Welcome to the Lightning community!   
 If you have any questions, feel free to:   
-1. [read the docs](https://pytorch-lightning.rtfd.io/en/latest/).     
+1. [read the docs](https://williamfalcon.github.io/pytorch-lightning/).     
-2. [Search through the issues](https://github.com/PytorchLightning/pytorch-lightning/issues?utf8=%E2%9C%93&q=my++question).      
+2. [Search through the issues](https://github.com/williamFalcon/pytorch-lightning/issues?utf8=%E2%9C%93&q=my++question).      
 3. [Ask on stackoverflow](https://stackoverflow.com/questions/ask?guided=false) with the tag pytorch-lightning.   
 If no one replies to you quickly enough, feel free to post the stackoverflow link to our Gitter chat!   
-To chat with the rest of us visit our [gitter channel](https://gitter.im/PyTorch-Lightning/community)!     
+To chat with the rest of us visit our [gitter channel](https://gitter.im/PyTorch-Lightning/community?utm_source=share-link&utm_medium=link&utm_campaign=share-link)!     
 ---   
 ## FAQ    
 **How do I use Lightning for rapid research?**   
-[Here's a walk-through](https://pytorch-lightning.rtfd.io/en/latest/)  
+[Here's a walk-through](https://williamfalcon.github.io/pytorch-lightning/)  
 **Why was Lightning created?**     
 Lightning has 3 goals in mind:
@@ -344,49 +397,22 @@ Nope.
 Nope. Please use anaconda or miniconda.    
 **Which PyTorch versions do you support?**    
- **PyTorch 1.1.0**       
+##### PyTorch 1.1.0       
-    ```bash    
+```bash    
-    # install pytorch 1.1.0 using the official instructions   
+# install pytorch 1.1.0 using the official instructions   
    # install test-tube 0.6.7.6 which supports 1.1.0   
    pip install test-tube==0.6.7.6   
    # install latest Lightning version without upgrading deps    
    pip install -U --no-deps pytorch-lightning
    ```     
 - **PyTorch 1.2.0, 1.3.0,**
    Install via pip as normal   
-## Custom installation
+# install test-tube 0.6.7.6 which supports 1.1.0   
 pip install test-tube==0.6.7.6   
-### Bleeding edge
+# install latest Lightning version without upgrading deps    
 pip install -U --no-deps pytorch-lightning
 ```     
-If you can't wait for the next release, install the most up to date code with:
+##### PyTorch 1.2.0   
-* using GIT (locally clone whole repo with full history)
+Install via pip as normal   
    ```bash
    pip install git+https://github.com/PytorchLightning/pytorch-lightning.git@master --upgrade
    ```
 * using instant zip (last state of the repo without git history)
    ```bash
    pip install https://github.com/PytorchLightning/pytorch-lightning/archive/master.zip --upgrade
    ```
-### Any release installation
+## Bleeding edge
-
+If you can't wait for the next release, install the most up to date code with:  
 You can also install any past release `0.X.Y` from this repository:
 ```bash
-pip install https://github.com/PytorchLightning/pytorch-lightning/archive/0.X.Y.zip --upgrade
+pip install git+https://github.com/williamFalcon/pytorch-lightning.git@master --upgrade
 ```
 ## Bibtex
 If you want to cite the framework feel free to use this (but only if you loved it 😊):
 ```
@misc{Falcon2019,
  author = {Falcon, W.A. et al.},
  title = {PyTorch Lightning},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/PytorchLightning/pytorch-lightning}}
 }
 ```
@@ -44,13 +44,11 @@ install:
  # purpose but it is problematic because it tends to cancel builds pushed
  # directly to master instead of just PR builds (or the converse).
  - SET PATH=%PYTHON%;%PYTHON%\\Scripts;%path%
-  #- pip install -U --user "pip<19.3"
+  - pip install -U --user pip
-  - python -m pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
+  - pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
-  - python -m pip install -r ./tests/requirements.txt
+  - pip install -r ./tests/requirements.txt
  - python -m pip install pytest-flake8
-# scripts to run before tests (working directory and environment changes
+# scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
 # are persisted from the previous steps such as "before_build")
 before_test:
  - python --version
  - pip --version
@@ -59,9 +57,7 @@ before_test:
 # to run your custom scripts instead of automatic tests
 test_script:
-  - coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules --flake8
+  - tox --sitepackages --parallel auto
  #- python setup.py sdist
  #- twine check dist/*
 on_success:
  - coverage report
@@ -0,0 +1,640 @@
 # Lightning Module interface
 [[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/root_module.py)]
 A lightning module is a strict superclass of nn.Module, it provides a standard interface for the trainer to interact with the model.
 The easiest thing to do is copy the [minimal example](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) below and modify accordingly. 
 Otherwise, to Define a Lightning Module, implement the following methods:
 **Required**:  
 - [training_step](RequiredTrainerInterface.md#training_step)      
 - [tng_dataloader](RequiredTrainerInterface.md#tng_dataloader)    
 - [configure_optimizers](RequiredTrainerInterface.md#configure_optimizers)    
 **Optional**:   
 - [validation_step](RequiredTrainerInterface.md#validation_step)    
 - [validation_end](RequiredTrainerInterface.md#validation_end) 
 - [test_step](RequiredTrainerInterface.md#test_step)    
 - [test_end](RequiredTrainerInterface.md#test_end) 
 - [val_dataloader](RequiredTrainerInterface.md#val_dataloader)    
 - [test_dataloader](RequiredTrainerInterface.md#test_dataloader)    
 - [on_save_checkpoint](RequiredTrainerInterface.md#on_save_checkpoint)    
 - [on_load_checkpoint](RequiredTrainerInterface.md#on_load_checkpoint)    
 - [update_tng_log_metrics](RequiredTrainerInterface.md#update_tng_log_metrics)    
 - [add_model_specific_args](RequiredTrainerInterface.md#add_model_specific_args)    
 ---
 ### Minimal example   
 ```python
 import os
 import torch
 from torch.nn import functional as F
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 import torchvision.transforms as transforms
 import pytorch_lightning as pl
 class CoolModel(pl.LightningModule):
    def __init__(self):
        super(CoolModel, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)
    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))
    def training_step(self, batch, batch_nb):
        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
        return {'loss': F.cross_entropy(y_hat, y)}
    def validation_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
        return {'val_loss': F.cross_entropy(y_hat, y)}
    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'avg_val_loss': avg_loss}
    def test_step(self, batch, batch_nb):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
        return {'test_loss': F.cross_entropy(y_hat, y)}
    def test_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
        return {'avg_test_loss': avg_loss}
    def configure_optimizers(self):
        # REQUIRED
        return [torch.optim.Adam(self.parameters(), lr=0.02)]
    @pl.data_loader
    def tng_dataloader(self):
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
    @pl.data_loader
    def val_dataloader(self):
        # OPTIONAL
        # can also return a list of val dataloaders
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
    @pl.data_loader
    def test_dataloader(self):
        # OPTIONAL
        # can also return a list of test dataloaders
        return DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=transforms.ToTensor()), batch_size=32)
 ```
 ---   
 ### How do these methods fit into the broader training?     
 The LightningModule interface is on the right. Each method corresponds to a part of a research project. Lightning automates everything not in blue.    
 <p align="center">
  <a href="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg">
    <img alt="" src="https://github.com/williamFalcon/pytorch-lightning/blob/master/docs/source/_static/overview_flat.jpg" height="900px">
  </a>
 </p>   
 ## Required Methods    
 ### training_step
 ``` {.python}
 def training_step(self, data_batch, batch_nb)
 ```
 In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
 **Params**    
 | Param  | description  |
 |---|---|
 |  data_batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
 **Return**   
 Dictionary or OrderedDict   
 | key  | value  | is required |
 |---|---|---|
 |  loss | tensor scalar  | Y |
 |  prog | Dict for progress bar display. Must have only tensors  | N |
 **Example**
 ``` {.python}
 def training_step(self, data_batch, batch_nb):
    x, y, z = data_batch
    # implement your own
    out = self.forward(x)
    loss = self.loss(out, x)
    output = {
        'loss': loss, # required
        'prog': {'tng_loss': loss, 'batch_nb': batch_nb} # optional
    }
    # return a dict
    return output
 ```    
 If you define multiple optimizers, this step will also be called with an additional ```optimizer_idx``` param.    
 ``` {.python}
 # Multiple optimizers (ie: GANs)     
 def training_step(self, data_batch, batch_nb, optimizer_idx):
    if optimizer_idx == 0:
        # do training_step with encoder
    if optimizer_idx == 1:
        # do training_step with decoder    
 ```    
 --- 
 ### tng_dataloader 
 ``` {.python}
@pl.data_loader
 def tng_dataloader(self)
 ```
 Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
 ##### Return
 PyTorch DataLoader
 **Example**
 ``` {.python}
@pl.data_loader
 def tng_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.hparams.batch_size,
        shuffle=True
    )
    return loader
 ```
 --- 
 ### configure_optimizers 
 ``` {.python}
 def configure_optimizers(self)
 ```
 Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple. 
 Lightning will call .backward() and .step() on each one in every epoch.  If you use 16 bit precision it will also handle that.
 **Note:** If you use multiple optimizers, training_step will have an additional ```optimizer_idx``` parameter.    
 ##### Return    
 Return any of these 3 options:   
 Single optimizer   
 List or Tuple - List of optimizers    
 Two lists - The first list has multiple optimizers, the second a list of learning-rate schedulers
 **Example**
 ``` {.python}
 # most cases
 def configure_optimizers(self):
    opt = Adam(self.parameters(), lr=0.01)
    return opt
 # multiple optimizer case (eg: GAN)
 def configure_optimizers(self):
    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
    return generator_opt, disriminator_opt
 # example with learning_rate schedulers  
 def configure_optimizers(self):
    generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
    disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
    discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
    return [generator_opt, disriminator_opt], [discriminator_sched]
 ```  
 If you need to control how often those optimizers step or override the default .step() schedule, override 
 the [optimizer_step](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step) hook.   
 ## Optional Methods    
 ### validation_step
 ``` {.python}
 # if you have one val dataloader:
 def validation_step(self, data_batch, batch_nb)   
 # if you have multiple val dataloaders:  
 def validation_step(self, data_batch, batch_nb, dataloader_idx)
 ```
 **OPTIONAL**    
 If you don't need to validate you don't need to implement this method.    
 In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
 The dict you return here will be available in the `validation_end` method. 
 **Params**   
 | Param  | description  |
 |---|---|
 |  data_batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
 |  dataloader_i | Integer displaying which dataloader this is (only if multiple val datasets used)  |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
 **Example**
 ``` {.python}
 # CASE 1: A single validation dataset
 def validation_step(self, data_batch, batch_nb):
    x, y = data_batch
    # implement your own
    out = self.forward(x)
    loss = self.loss(out, y)
    # log 6 example images
    # or generated text... or whatever
    sample_imgs = x[:6]
    grid = torchvision.utils.make_grid(sample_imgs)
    self.experiment.add_image('example_images', grid, 0) 
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    # all optional...
    # return whatever you need for the collation function validation_end
    output = OrderedDict({
        'val_loss': loss_val,
        'val_acc': torch.tensor(val_acc), # everything must be a tensor
    })
    # return an optional dict
    return output
 ```   
 If you pass in multiple validation datasets, validation_step will have an additional argument.
 ```python
 # CASE 2: multiple validation datasets
 def validation_step(self, data_batch, batch_nb, dataset_idx):
    # dataset_idx tells you which dataset this is.   
 ```   
 The ```dataset_idx``` corresponds to the order of datasets returned in ```val_dataloader```.    
 --- 
 ### validation_end
 ``` {.python}
 def validation_end(self, outputs)
 ```   
 If you didn't define a validation_step, this won't be called.       
 Called at the end of the validation loop with the output of each validation_step.  Called once per validation dataset.   
 The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
 **Params**    
 | Param  | description  |
 |---|---|
 |  outputs | List of outputs you defined in validation_step |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict of OrderedDict with metrics to display in progress bar | Y |
 **Example**
 ``` {.python}
 def validation_end(self, outputs):
    """
    Called at the end of validation to aggregate outputs
    :param outputs: list of individual outputs of each validation step
    :return:
    """
    val_loss_mean = 0
    val_acc_mean = 0
    for output in outputs:
        val_loss_mean += output['val_loss']
        val_acc_mean += output['val_acc']
    val_loss_mean /= len(outputs)
    val_acc_mean /= len(outputs)
    tqdm_dic = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
    return tqdm_dic
 ```
 ### test_step
 ``` {.python}
 # if you have one test dataloader:
 def test_step(self, data_batch, batch_nb)   
 # if you have multiple test dataloaders:  
 def test_step(self, data_batch, batch_nb, dataloader_idx)
 ```
 **OPTIONAL**    
 If you don't need to test you don't need to implement this method.    
 In this step you'd normally generate examples or calculate anything of interest such as accuracy. 
 The dict you return here will be available in the `test_end` method. 
 This function is used when you execute `trainer.test()`.
 **Params**   
 | Param  | description  |
 |---|---|
 |  data_batch | The output of your dataloader. A tensor, tuple or list  |
 |  batch_nb | Integer displaying which batch this is  |
 |  dataloader_i | Integer displaying which dataloader this is (only if multiple test datasets used)  |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict or OrderedDict with metrics to display in progress bar. All keys must be tensors. | Y |
 **Example**
 ``` {.python}
 # CASE 1: A single test dataset
 def test_step(self, data_batch, batch_nb):
    x, y = data_batch
    # implement your own
    out = self.forward(x)
    loss = self.loss(out, y)
    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    test_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
    # all optional...
    # return whatever you need for the collation function test_end
    output = OrderedDict({
        'test_loss': loss_test,
        'test_acc': torch.tensor(test_acc), # everything must be a tensor
    })
    # return an optional dict
    return output
 ```   
 If you pass in multiple test datasets, test_step will have an additional argument.
 ```python
 # CASE 2: multiple test datasets
 def test_step(self, data_batch, batch_nb, dataset_idx):
    # dataset_idx tells you which dataset this is.   
 ```   
 The ```dataset_idx``` corresponds to the order of datasets returned in ```test_dataloader```.    
 --- 
 ### test_end
 ``` {.python}
 def test_end(self, outputs)
 ```   
 If you didn't define a test_step, this won't be called.       
 Called at the end of the test step with the output of each test_step.  Called once per test dataset.   
 The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    
 **Params**    
 | Param  | description  |
 |---|---|
 |  outputs | List of outputs you defined test_step |
 **Return**   
 | Return  | description  | optional |
 |---|---|---|   
 |  dict | Dict of OrderedDict with metrics to display in progress bar | Y |
 **Example**
 ``` {.python}
 def test_end(self, outputs):
    """
    Called at the end of test to aggregate outputs
    :param outputs: list of individual outputs of each test step
    :return:
    """
    test_loss_mean = 0
    test_acc_mean = 0
    for output in outputs:
        test_loss_mean += output['test_loss']
        test_acc_mean += output['test_acc']
    test_loss_mean /= len(outputs)
    test_acc_mean /= len(outputs)
    tqdm_dic = {'test_loss': test_loss_mean.item(), 'test_acc': test_acc_mean.item()}
    return tqdm_dic
 ```
 --- 
 ### on_save_checkpoint 
 ``` {.python}
 def on_save_checkpoint(self, checkpoint)
 ```
 Called by lightning to checkpoint your model. Lightning saves the training state (current epoch, global_step, etc)
 and also saves the model state_dict. If you want to save anything else, use this method to add your own
 key-value pair.
 ##### Return
 Nothing
 **Example**
 ``` {.python}
 def on_save_checkpoint(self, checkpoint):
    # 99% of use cases you don't need to implement this method 
    checkpoint['something_cool_i_want_to_save'] = my_cool_pickable_object
 ```
 --- 
 ### on_load_checkpoint 
 ``` {.python}
 def on_load_checkpoint(self, checkpoint)
 ```
 Called by lightning to restore your model. Lighting auto-restores global step, epoch, etc...
 It also restores the model state_dict.
 If you saved something with **on_save_checkpoint** this is your chance to restore this.
 ##### Return
 Nothing  
 **Example**
 ``` {.python}
 def on_load_checkpoint(self, checkpoint):
    # 99% of the time you don't need to implement this method
    self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']
 ```
 --- 
 ### val_dataloader 
 ``` {.python}
@pl.data_loader
 def val_dataloader(self)
 ```
 **OPTIONAL**    
 If you don't need a validation dataset and a validation_step, you don't need to implement this method.    
 Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.   
 ##### Return
 PyTorch DataLoader or list of PyTorch Dataloaders.    
 **Example**
 ``` {.python}
@pl.data_loader
 def val_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.hparams.batch_size,
        shuffle=True
    )
    return loader
 # can also return multiple dataloaders   
@pl.data_loader
 def val_dataloader(self):
    return [loader_a, loader_b, ..., loader_n]   
 ```
 In the case where you return multiple val_dataloaders, the validation_step will have an arguement ```dataset_idx```
 which matches the order here.    
 --- 
 ### test_dataloader 
 ``` {.python}
@pl.data_loader
 def test_dataloader(self)
 ```
 **OPTIONAL**    
 If you don't need a test dataset and a test_step, you don't need to implement this method.    
 Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
 ##### Return
 PyTorch DataLoader
 **Example**
 ``` {.python}
@pl.data_loader
 def test_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform, download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.hparams.batch_size,
        shuffle=True
    )
    return loader
 ```
 --- 
 ### update_tng_log_metrics 
 ``` {.python}
 def update_tng_log_metrics(self, logs)
 ```
 Called by lightning right before it logs metrics for this batch.
 This is a chance to amend or add to the metrics about to be logged.
 ##### Return
 Dict 
 **Example**
 ``` {.python}
 def update_tng_log_metrics(self, logs):
    # modify or add to logs
    return logs
 ```
 --- 
 ### add_model_specific_args 
 ``` {.python}
@staticmethod
 def add_model_specific_args(parent_parser, root_dir)
 ```
 Lightning has a list of default argparse commands.
 This method is your chance to add or modify commands specific to your model.
 The [hyperparameter argument parser](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/) is available anywhere in your model by calling self.hparams.
 ##### Return
 An argument parser
 **Example**
 ``` {.python}
@staticmethod
 def add_model_specific_args(parent_parser, root_dir):
    parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
    # param overwrites
    # parser.set_defaults(gradient_clip=5.0)
    # network params
    parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
    parser.add_argument('--in_features', default=28*28)
    parser.add_argument('--out_features', default=10)
    parser.add_argument('--hidden_dim', default=50000) # use 500 for CPU, 50000 for GPU to see speed difference
    # data
    parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
    # training params (opt)
    parser.opt_list('--learning_rate', default=0.001, type=float, options=[0.0001, 0.0005, 0.001, 0.005],
                    tunable=False)
    parser.opt_list('--batch_size', default=256, type=int, options=[32, 64, 128, 256], tunable=False)
    parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False)
    return parser
 ```
@@ -0,0 +1,50 @@
 Lightning modules are strict superclasses of torch.nn.Module. A LightningModule offers the following in addition to that API.
 ---    
 ### freeze 
 Freeze all params for inference
 ```{.python}
 model = MyLightningModule(...)
 model.freeze()
 ```
 ---    
 ### load_from_metrics
 This is the easiest/fastest way which uses the meta_tags.csv file from test-tube to rebuild the model.
 The meta_tags.csv file can be found in the test-tube experiment save_dir.       
 ```{.python}
 pretrained_model = MyLightningModule.load_from_metrics(
    weights_path='/path/to/pytorch_checkpoint.ckpt',
    tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv',
    on_gpu=True,
    map_location=None
 )
 # predict
 pretrained_model.eval()
 pretrained_model.freeze()
 y_hat = pretrained_model(x)
 ```
 **Params**    
 | Param  | description  |
 |---|---|
 |  weights_path | Path to a PyTorch checkpoint  |
 |  tags_csv | Path to meta_tags.csv file generated by the test-tube Experiment  |
 |  on_gpu | if True, puts model on GPU. Make sure to use transforms option if model devices have changed  |
 |  map_location | A dictionary mapping saved weight GPU devices to new GPU devices |
 **Returns**    
 LightningModule - The pretrained LightningModule
 ---    
 ### unfreeze 
 Unfreeze all params for inference
 ```{.python}
 model = MyLightningModule(...)
 model.unfreeze()
 ```
@@ -0,0 +1,54 @@
 A LightningModule has the following properties which you can access at any time
 --- 
 #### current_epoch
 The current epoch   
 ---
 #### dtype    
 Current dtype    
 ---
 #### experiment    
 An instance of test-tube Experiment which you can use to log anything for tensorboard (subclass of [PyTorch SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html)).   
 ```{.python}
 self.experiment.add_embedding(...)   
 self.experiment.log({'val_loss': 0.9})   
 self.experiment.add_scalars(...)   
 ```
 --- 
 #### global_step   
 Total training batches seen across all epochs   
 --- 
 #### gradient_clip   
 The current gradient clip value    
 ---
 #### on_gpu    
 True if your model is currently running on GPUs. Useful to set flags around the LightningModule for different CPU vs GPU behavior.    
 ---
 #### trainer    
 Last resort access to any state the trainer has. Changing certain properties here could affect your training run.
 ```{.python}   
 self.trainer.optimizers
 self.trainer.current_epoch
 ...   
 ```   
 ## Debugging   
 The LightningModule also offers these tricks to help debug.   
 ---   
 #### example_input_array    
 In the LightningModule init, you can set a dummy tensor for this property
 to get a print out of sizes coming into and out of every layer.   
 ```python
 def __init__(self):
    # put the dimensions of the first input to your system
    self.example_input_array = torch.rand(5, 28 * 28)
 ```    
@@ -1,19 +0,0 @@
 # Minimal makefile for Sphinx documentation
 #
 # You can set these variables from the command line.
 SPHINXOPTS    =
 SPHINXBUILD   = sphinx-build
 SOURCEDIR     = source
 BUILDDIR      = build
 # Put it first so that "make" without argument is like "make help".
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 .PHONY: help Makefile
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
@@ -0,0 +1,69 @@
 Lightning can automate saving and loading checkpoints.
 ---
 ### Model saving
 To enable checkpointing, define the checkpoint callback and give it to the trainer.
 ``` {.python}
 from pytorch_lightning.callbacks import ModelCheckpoint
 checkpoint_callback = ModelCheckpoint(
    filepath='/path/to/store/weights.ckpt',
    save_best_only=True,
    verbose=True,
    monitor='val_loss',
    mode='min'
 )
 trainer = Trainer(checkpoint_callback=checkpoint_callback)
 ```
 ---
 ### Restoring training session 
 You might want to not only load a model but also continue training it. Use this method to
 restore the trainer state as well. This will continue from the epoch and global step you last left off.  
 However, the dataloaders will start from the first batch again (if you shuffled it shouldn't matter).   
 Lightning will restore the session if you pass an experiment with the same version and there's a saved checkpoint.   
 ``` {.python}
 from test_tube import Experiment
 exp = Experiment(version=a_previous_version_with_a_saved_checkpoint)
 trainer = Trainer(experiment=exp)
 # this fit call loads model weights and trainer state
 # the trainer continues seamlessly from where you left off
 # without having to do anything else.
 trainer.fit(model)
 ```
 The trainer restores:  
 - global_step    
 - current_epoch    
 - All optimizers    
 - All lr_schedulers    
 - Model weights
 You can even change the logic of your model as long as the weights and "architecture" of 
 the system isn't different. If you add a layer, for instance, it might not work.   
 At a rough level, here's [what happens inside Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/model_saving.py#L63):   
 ```python
 self.global_step = checkpoint['global_step']
 self.current_epoch = checkpoint['epoch']
 # restore the optimizers
 optimizer_states = checkpoint['optimizer_states']
 for optimizer, opt_state in zip(self.optimizers, optimizer_states):
    optimizer.load_state_dict(opt_state)
 # restore the lr schedulers
 lr_schedulers = checkpoint['lr_schedulers']
 for scheduler, lrs_state in zip(self.lr_schedulers, lr_schedulers):
    scheduler.load_state_dict(lrs_state)
 # uses the model you passed into trainer        
 model.load_state_dict(checkpoint['state_dict'])
 ```    
@@ -0,0 +1,143 @@
 Lightning makes multi-gpu training and 16 bit training trivial.
 *Note:*   
 None of the flags below require changing anything about your lightningModel definition. 
 ---
 #### Choosing a backend  
 Lightning supports two backends. DataParallel and DistributedDataParallel. Both can be used for single-node multi-GPU training.
 For multi-node training you must use DistributedDataParallel.   
 You can toggle between each mode by setting this flag.
 ``` {.python}
 # DEFAULT uses DataParallel
 trainer = Trainer(distributed_backend='dp')
 # change to distributed data parallel
 trainer = Trainer(distributed_backend='ddp')
 ```
 If you request multiple nodes, the back-end will auto-switch to ddp.
 We recommend you use DistributedDataparallel even for single-node multi-GPU training. It is MUCH faster than DP but *may*
 have configuration issues depending on your cluster.
 For a deeper understanding of what lightning is doing, feel free to read [this guide](https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565).   
 ---
 #### Distributed and 16-bit precision.    
 Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
 not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.   
 Below are the possible configurations we support.    
 | 1 GPU  | 1+ GPUs  | DP  | DDP  | 16-bit  | command |
 |---|---|---|---|---|---|
 | Y  |   |   |   |  | ```Trainer(gpus=[0])``` |
 | Y  |   |   |   | Y | ```Trainer(gpus=[0], use_amp=True)``` |
 |   | Y | Y |   |   | ```Trainer(gpus=[0, ...])``` |
 |   | Y |  | Y  |  | ```Trainer(gpus=[0, ...], distributed_backend='ddp')``` |
 |   | Y |  | Y  | Y | ```Trainer(gpus=[0, ...], distributed_backend='ddp', use_amp=True)``` |
 ---
 #### CUDA flags   
 CUDA flags make certain GPUs visible to your script. 
 Lightning sets these for you automatically, there's NO NEED to do this yourself.
 ```python
 # lightning will set according to what you give the trainer
 # os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
 # os.environ["CUDA_VISIBLE_DEVICES"] = "0"
 ```
 ---
 #### 16-bit mixed precision
 16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.    
 First, install apex (if install fails, look [here](https://github.com/NVIDIA/apex)):
 ```bash
 $ git clone https://github.com/NVIDIA/apex
 $ cd apex
 $ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
 ```
 then set this use_amp to True.
 ``` {.python}
 # DEFAULT
 trainer = Trainer(amp_level='O2', use_amp=False)
 ```
 ---
 #### Single-gpu
 Make sure you're on a GPU machine. 
 ```python
 # DEFAULT
 trainer = Trainer(gpus=[0])
 ```
 ---
 #### multi-gpu 
 Make sure you're on a GPU machine. You can set as many GPUs as you want.
 In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
 ```python
 # to use DataParallel (default)
 trainer = Trainer(gpus=[0,1,2,3,4,5,6,7], distributed_backend='dp')
 # RECOMMENDED use DistributedDataParallel
 trainer = Trainer(gpus=[0,1,2,3,4,5,6,7], distributed_backend='ddp')
 ```
 ---
 #### Multi-node
 Multi-node training is easily done by specifying these flags.
 ```python
 # train on 12*8 GPUs
 trainer = Trainer(gpus=[0,1,2,3,4,5,6,7], nb_gpu_nodes=12)
 ```
 In addition, make sure to set up your SLURM job correctly via the [SlurmClusterObject](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/). In particular, specify the number of tasks per node correctly.
 ```python
 cluster = SlurmCluster(
    hyperparam_optimizer=test_tube.HyperOptArgumentParser(),
    log_path='/some/path/to/save',
 )
 # OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
 # which interface your nodes use for communication
 cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
 # see output of the NCCL connection process
 # NCCL is how the nodes talk to each other
 cluster.add_command('export NCCL_DEBUG=INFO')
 # setting a master port here is a good idea.
 cluster.add_command('export MASTER_PORT=%r' % PORT)
 # good to load the latest NCCL version
 cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
 # configure cluster
 cluster.per_experiment_nb_nodes = 12 
 cluster.per_experiment_nb_gpus = 8
 cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
 ```
 Finally, make sure to add a distributed sampler to your dataset. The distributed sampler copies a 
 portion of your dataset onto each GPU. (World_size = gpus_per_node * nb_nodes).   
 ```python
 # ie: this:
 dataset = myDataset()
 dataloader = Dataloader(dataset)
 # becomes:
 dataset = myDataset()
 dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
 dataloader = Dataloader(dataset, sampler=dist_sampler)
 ```
 ---
 #### Self-balancing architecture
 Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.   
 COMING SOON.
@@ -0,0 +1,97 @@
 Lighting offers a few options for logging information about model, gpu usage, etc (via test-tube). It also offers printing options for training monitoring.
 ---
 #### Display metrics in progress bar 
 ``` {.python}
 # DEFAULT
 trainer = Trainer(show_progress_bar=True)
 ```
 ---
 #### Log metric row every k batches 
 Every k batches lightning will make an entry in the metrics log
 ``` {.python}
 # DEFAULT (ie: save a .csv log file every 10 batches)
 trainer = Trainer(add_log_row_interval=10)
 ```
 ---
 #### Process position
 When running multiple models on the same machine we want to decide which progress bar to use.
 Lightning will stack progress bars according to this value. 
 ``` {.python}
 # DEFAULT
 trainer = Trainer(process_position=0)
 # if this is the second model on the node, show the second progress bar below
 trainer = Trainer(process_position=1)
 ```
 ---
 #### Save a snapshot of all hyperparameters 
 Whenever you call .save() on the test-tube experiment it logs all the hyperparameters in current use.
 Give lightning a test-tube Experiment object to automate this for you.
 ``` {.python}
 from test_tube import Experiment
 exp = Experiment(...)
 Trainer(experiment=exp)
 ```
 ---
 #### Snapshot code for a training run
 Whenever you call .save() on the test-tube experiment it snapshows all code and pushes to a git tag.
 Give lightning a test-tube Experiment object to automate this for you.
 ``` {.python}
 from test_tube import Experiment
 exp = Experiment(create_git_tag=True)
 Trainer(experiment=exp)
 ```
 ---
 ### Tensorboard support   
 In the LightningModule you can access the experiment logger by doing:
 ```python
 self.experiment
 # add image
 # Look at PyTorch SummaryWriter docs for what you can do.   
 self.experiment.add_image(...)
 ```
 The experiment object is a strict subclass of PyTorch SummaryWriter. However, this class
 also snapshots every detail about the experiment (data folder paths, code, hyperparams),
 and allows you to visualize it using tensorboard.
 ``` {.python}
 from test_tube import Experiment, HyperOptArgumentParser
 # exp hyperparams
 args = HyperOptArgumentParser()
 hparams = args.parse_args()
 # this is a summaryWriter with nicer logging structure
 exp = Experiment(save_dir='/some/path', create_git_tag=True)
 # track experiment details (must be ArgumentParser or HyperOptArgumentParser).
 # each option in the parser is tracked
 exp.argparse(hparams)
 exp.tag({'description': 'running demo'})
 # trainer uses the exp object to log exp data
 trainer = Trainer(experiment=exp)
 trainer.fit(model)
 # view logs at:
 # tensorboard --logdir /some/path   
 ```
 ---
 #### Write logs file to csv every k batches 
 Every k batches, lightning will write the new logs to disk
 ``` {.python}
 # DEFAULT (ie: save a .csv log file every 100 batches)
 trainer = Trainer(log_save_interval=100)
 ```
@@ -0,0 +1,104 @@
 Lightning supports model training on a cluster managed by SLURM in the following cases:    
 1. Training on single or multi-cpus only.
 2. Training on single or multi-gpus on the same node.
 3. Coming SOON: Training across multiple nodes.
 ---
 #### Running grid search on a cluster
 To use lightning to run a hyperparameter search (grid-search or random-search) on a cluster do 4 things:   
 (1). Define the parameters for the grid search    
 ```{.python}
 from test_tube import HyperOptArgumentParser
 # subclass of argparse
 parser = HyperOptArgumentParser(strategy='random_search')
 parser.add_argument('--learning_rate', default=0.002, type=float, help='the learning rate')
 # let's enable optimizing over the number of layers in the network
 parser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4, 8])
 hparams = parser.parse_args()    
 ```    
 (2). Define the cluster options in the [SlurmCluster object](https://williamfalcon.github.io/test-tube/hpc/SlurmCluster/) (over 5 nodes and 8 gpus)    
 ```{.python}
 from test_tube.hpc import SlurmCluster
 # hyperparameters is a test-tube hyper params object
 # see https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/
 hyperparams = args.parse()
 # init cluster
 cluster = SlurmCluster(
    hyperparam_optimizer=hyperparams,
    log_path='/path/to/log/results/to',
    python_cmd='python3'
 )
 # let the cluster know where to email for a change in job status (ie: complete, fail, etc...)
 cluster.notify_job_status(email='some@email.com', on_done=True, on_fail=True)
 # set the job options. In this instance, we'll run 20 different models
 # each with its own set of hyperparameters giving each one 1 GPU (ie: taking up 20 GPUs)
 cluster.per_experiment_nb_gpus = 8
 cluster.per_experiment_nb_nodes = 5
 # we'll request 10GB of memory per node
 cluster.memory_mb_per_node = 10000
 # set a walltime of 10 minues
 cluster.job_time = '10:00'
 ```
 (3). Give trainer the cluster_manager in your main function:    
 ```{.python}
 from pytorch_lightning import Trainer
 def train_fx(trial_hparams, cluster_manager, _):
    # hparams has a specific set of hyperparams
    my_model = MyLightningModel()
    # give the trainer the cluster object
    trainer = Trainer(cluster=cluster_manager)
    trainer.fit(my_model)
 ```
 (4). Start the grid search     
 ```{.python}
 # run the models on the cluster
 cluster.optimize_parallel_cluster_gpu(
    train_fx, 
    nb_trials=20, 
    job_name='my_grid_search_exp_name', 
    job_display_name='my_exp')
 ```
 That's it! The SlurmCluster object will automatically checkpoint the lightning model and resubmit if it runs into the walltime!
 ---
 #### Walltime auto-resubmit
 Lightning automatically resubmits jobs when they reach the walltime. You get this behavior for free if you give lightning
 a slurm cluster object.
 ```{.python}
 def my_main_fx(hparams, slurm_manager, _):
    trainer = Trainer(cluster=slurm_manager)
 ``` 
 (See the grid search example above for cluster configuration).
 With this feature lightning will:    
 1. automatically checkpoint the model
 2. checkpoint the trainer session
 3. resubmit a continuation job.
 4. load the checkpoint and trainer session in the new model
@@ -0,0 +1,67 @@
 The lightning training loop handles everything except the actual computations of your model. To decide what will happen in your training loop, define the [training_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#training_step).
 Below are all the things lightning automates for you in the training loop.
 ---
 #### Accumulated gradients  
 Accumulated gradients runs K small batches of size N before doing a backwards pass. The effect is a large effective batch size of size KxN. 
 ``` {.python}
 # DEFAULT (ie: no accumulated grads)
 trainer = Trainer(accumulate_grad_batches=1)
 ```
 ---
 #### Force training for min or max epochs
 It can be useful to force training for a minimum number of epochs or limit to a max number
 ``` {.python}
 # DEFAULT
 trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000)
 ```
 ---
 #### Force disable early stop 
 Use this to turn off early stopping and run training to the [max_epoch](#force-training-for-min-or-max-epochs)
 ``` {.python}
 # DEFAULT
 trainer = Trainer(enable_early_stop=True)
 ```
 ---
 #### Gradient Clipping
 Gradient clipping may be enabled to avoid exploding gradients.
 Specifically, this will [clip the gradient norm computed over all model parameters *together*](https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_).
 ``` {.python}
 # DEFAULT (ie: don't clip)
 trainer = Trainer(gradient_clip=0)
 # clip gradients with norm above 0.5
 trainer = Trainer(gradient_clip=0.5)
 ```
 ---
 #### Inspect gradient norms
 Looking at grad norms can help you figure out where training might be going wrong.
 ``` {.python}
 # DEFAULT (-1 doesn't track norms)
 trainer = Trainer(track_grad_norm=-1)
 # track the LP norm (P=2 here)
 trainer = Trainer(track_grad_norm=2)
 ```
 ---
 #### Set how much of the training set to check
 If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag.
 train_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(train_percent_check=1.0)
 # check 10% only
 trainer = Trainer(train_percent_check=0.1)
 ```
@@ -0,0 +1,63 @@
 The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the [validation_step function](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#validation_step).
 Below are all the things lightning automates for you in the validation loop.
 **Note**   
 Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don't have to wait until a full epoch to catch possible validation issues.
 ---
 #### Check validation every n epochs
 If you have a small dataset you might want to check validation every n epochs
 ``` {.python}
 # DEFAULT
 trainer = Trainer(check_val_every_n_epoch=1)
 ```
 ---
 #### Set how much of the validation set to check 
 If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag
 val_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_percent_check=1.0)
 # check 10% only
 trainer = Trainer(val_percent_check=0.1)
 ```
 ---
 #### Set how much of the test set to check 
 If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag
 test_percent_check will be overwritten by overfit_pct if `overfit_pct > 0`
 ``` {.python}
 # DEFAULT
 trainer = Trainer(test_percent_check=1.0)
 # check 10% only
 trainer = Trainer(test_percent_check=0.1)
 ```
 ---
 ####  Set validation check frequency within 1 training epoch
 For large datasets it's often desirable to check validation multiple times within a training loop
 ``` {.python}
 # DEFAULT
 trainer = Trainer(val_check_interval=0.95)
 # check every .25 of an epoch 
 trainer = Trainer(val_check_interval=0.25)
 ```
 ---
 ####  Set the number of validation sanity steps
 Lightning runs a few steps of validation in the beginning of training. This avoids crashing in the validation loop sometime deep into a lengthy training loop.
 ``` {.python}
 # DEFAULT
 trainer = Trainer(nb_sanity_val_steps=5)
 ```
 You can use `Trainer(nb_sanity_val_steps=0)` to skip the sanity check.
@@ -0,0 +1,51 @@
 These flags are useful to help debug a model.
 ---
 #### Fast dev run 
 This flag is meant for debugging a full train/val/test loop. It'll activate callbacks, everything but only with 1 training and 1 validation batch.
 Use this to debug a full run of your program quickly
 ``` {.python}
 # DEFAULT
 trainer = Trainer(fast_dev_run=False)
 ```
 ---
 #### Inspect gradient norms
 Looking at grad norms can help you figure out where training might be going wrong.
 ``` {.python}
 # DEFAULT (-1 doesn't track norms)
 trainer = Trainer(track_grad_norm=-1)
 # track the LP norm (P=2 here)
 trainer = Trainer(track_grad_norm=2)
 ```
 ---
 #### Make model overfit on subset of data
 A useful debugging trick is to make your model overfit a tiny fraction of the data.
 setting `overfit_pct > 0` will overwrite train_percent_check, val_percent_check, test_percent_check
 ``` {.python}
 # DEFAULT don't overfit (ie: normal training)
 trainer = Trainer(overfit_pct=0.0)
 # overfit on 1% of data 
 trainer = Trainer(overfit_pct=0.01)
 ```
 ---
 #### Print the parameter count by layer
 By default lightning prints a list of parameters *and submodules* when it starts training.
 ---
 #### Print which gradients are nan 
 This option prints a list of tensors with nan gradients.
 ``` {.python}
 # DEFAULT
 trainer = Trainer(print_nan_grads=False)
 ```
 ---
 #### Log GPU usage
 Lightning automatically logs gpu usage to the test tube logs. It'll only do it at the metric logging interval, so it doesn't slow down training.
@@ -0,0 +1,141 @@
 # Hooks
 [[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py)]   
 There are cases when you might want to do something different at different parts of the training/validation loop.
 To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
 **Contributing** If there's a hook you'd like to add, simply:    
 1. Fork PyTorchLightning.    
 2. Add the hook [here](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/hooks.py).       
 3. Add the correct place in the [Trainer](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py) where it should be called.    
 ---
 #### on_epoch_start
 Called in the training loop at the very beginning of the epoch.   
 ```python
 def on_epoch_start(self):
    # do something when the epoch starts
 ```
 ---
 #### on_epoch_end
 Called in the training loop at the very end of the epoch.   
 ```python
 def on_epoch_end(self):
    # do something when the epoch ends 
 ```
 ---
 #### on_batch_start
 Called in the training loop before anything happens for that batch.   
 ```python
 def on_batch_start(self):
    # do something when the batch starts
 ```
 ---
 #### on_batch_end
 Called in the training loop after the batch.   
 ```python
 def on_batch_end(self):
    # do something when the batch ends 
 ```
 ---
 #### on_pre_performance_check
 Called at the very beginning of the validation loop.   
 ```python
 def on_pre_performance_check(self):
    # do something before validation starts 
 ```
 ---
 #### on_post_performance_check
 Called at the very end of the validation loop.   
 ```python
 def on_post_performance_check(self):
    # do something before validation end
 ```
 ---
 #### on_tng_metrics
 Called in the training loop, right before metrics are logged.
 Although you can log at any time by using self.experiment, you can use
 this callback to modify what will be logged.
 ```python
 def on_tng_metrics(self, metrics):
    # do something before validation end
 ```
 ---   
 #### optimizer_step 
 Calls .step() and .zero_grad for each optimizer.  
 You can override this method to adjust how you do the optimizer step for each optimizer
 Called once per optimizer
 ```python
 # DEFAULT
 def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
    optimizer.step()   
    optimizer.zero_grad()   
 # Alternating schedule for optimizer steps (ie: GANs)    
 def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
    # update generator opt every 2 steps
    if optimizer_i == 0:
        if batch_nb % 2 == 0 :
            optimizer.step()
            optimizer.zero_grad()
    # update discriminator opt every 4 steps
    if optimizer_i == 1:
        if batch_nb % 4 == 0 :
            optimizer.step()
            optimizer.zero_grad()    
    # ...
    # add as many optimizers as you want 
 ```
 This step allows you to do a lot of non-standard training tricks such as learning-rate warm-up:   
 ```python
 # learning rate warm-up
 def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i):
    # warm up lr
    if self.trainer.global_step < 500:
        lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
        for pg in optimizer.param_groups:
            pg['lr'] = lr_scale * self.hparams.learning_rate
    # update params
    optimizer.step()
    optimizer.zero_grad() 
 ```
 ---
 #### on_before_zero_grad
 Called in the training loop after taking an optimizer step and before zeroing grads.
 Good place to inspect weight information with weights updated.
 Called once per optimizer
 ```python
 def on_before_zero_grad(self, optimizer):
    # do something with the optimizer or inspect it. 
 ```
 ---
 #### on_after_backward
 Called in the training loop after model.backward()
 This is the ideal place to inspect or log gradient information 
 ```python
 def on_after_backward(self):
    # example to inspect gradient information in tensorboard
    if self.trainer.global_step % 25 == 0:  # don't make the tf file huge
        params = self.state_dict()
        for k, v in params.items():
            grads = v
            name = k
            self.experiment.add_histogram(tag=name, values=grads, global_step=self.trainer.global_step)
 ```
@@ -0,0 +1,80 @@
 # Trainer
 [[Github Code](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/models/trainer.py)]
 The lightning trainer abstracts best practices for running a training, val, test routine. It calls parts of your model when it wants to hand over full control and otherwise makes training assumptions which are now standard practice in AI research.
 This is the basic use of the trainer:
 ``` {.python}
 from pytorch_lightning import Trainer
 model = LightningTemplate()
 trainer = Trainer()
 trainer.fit(model)
 ```
 But of course the fun is in all the advanced things it can do:
 **Checkpointing**    
 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
 - [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
 **Computing cluster (SLURM)**    
 - [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
 - [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
 **Debugging**  
 - [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
 - [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
 - [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
 - [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
 - [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 **Distributed training**    
 - [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
 - [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
 - [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
 - [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
 - [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
 **Experiment Logging**   
 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
 - [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
 - [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
 - [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
 **Training loop**    
 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 **Validation loop**    
 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
 - [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
 - [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
@@ -0,0 +1,171 @@
 ### Template model definition
 In 99% of cases you want to just copy [this template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/lightning_module_template.py) to start a new lightningModule and change the core of what your model is actually trying to do.
 ```bash
 # get a copy of the module template
 wget https://raw.githubusercontent.com/williamFalcon/pytorch-lightning/master/examples/new_project_templates/lightning_module_template.py
 ```
 ---    
 ### Trainer Example 
 ** \_\_main__ function**    
 Normally, we want to let the \_\_main__ function start the training.
 Inside the main we parse training arguments with whatever hyperparameters we want. Your LightningModule will have a 
 chance to add hyperparameters.   
 ```{.python}
 from test_tube import HyperOptArgumentParser
 if __name__ == '__main__':
    # use default args given by lightning
    root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
    parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
    add_default_args(parent_parser, root_dir)
    # allow model to overwrite or extend args
    parser = ExampleModel.add_model_specific_args(parent_parser)
    hyperparams = parser.parse_args()
    # train model
    main(hyperparams)
 ```
 **Main Function**      
 The main function is your entry into the program. This is where you init your model, checkpoint directory, and launch the training.
 The main function should have 3 arguments:   
 - hparams: a configuration of hyperparameters.    
 - slurm_manager: Slurm cluster manager object (can be None)
 - dict: for you to return any values you want (useful in meta-learning, otherwise set to _)    
 ```python
 def main(hparams, cluster, results_dict):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # init experiment
    log_dir = os.path.dirname(os.path.realpath(__file__))
    exp = Experiment(
        name='test_tube_exp',
        debug=True,
        save_dir=log_dir,
        version=0,
        autosave=False,
        description='test demo'
    )
    # set the hparams for the experiment
    exp.argparse(hparams)
    exp.save()
    # build model
    model = MyLightningModule(hparams)
    # callbacks
    early_stop = EarlyStopping(
        monitor=hparams.early_stop_metric,
        patience=hparams.early_stop_patience,
        verbose=True,
        mode=hparams.early_stop_mode
    )
    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
    checkpoint = ModelCheckpoint(
        filepath=model_save_path,
        save_function=None,
        save_best_only=True,
        verbose=True,
        monitor=hparams.model_save_monitor_value,
        mode=hparams.model_save_monitor_mode
    )
    # configure trainer
    trainer = Trainer(
        experiment=exp,
        cluster=cluster,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
    )
    # train model
    trainer.fit(model)
 ```
 The __main__ function will start training on your **main** function. If you use the HyperParameterOptimizer
 in hyper parameter optimization mode, this main function will get one set of hyperparameters. If you use it as a simple
 argument parser you get the default arguments in the argument parser.
 So, calling main(hyperparams) runs the model with the default argparse arguments.       
 ```{.python}
 main(hyperparams)
 ```
 ---
 #### CPU hyperparameter search      
 ```{.python}
 # run a grid search over 20 hyperparameter combinations.
 hyperparams.optimize_parallel_cpu(
    main_local,
    nb_trials=20,
    nb_workers=1
 )
 ```
 ---
 #### Hyperparameter search on a single or multiple GPUs       
 ```{.python}
 # run a grid search over 20 hyperparameter combinations.
 hyperparams.optimize_parallel_gpu(
    main_local,
    nb_trials=20,
    nb_workers=1,
    gpus=[0,1,2,3]
 )
 ```
 ---
 #### Hyperparameter search on a SLURM HPC cluster   
 ```{.python}    
 def optimize_on_cluster(hyperparams):
    # enable cluster training
    cluster = SlurmCluster(
        hyperparam_optimizer=hyperparams,
        log_path=hyperparams.tt_save_path,
        test_tube_exp_name=hyperparams.tt_name
    )
    # email for cluster coms
    cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
    # configure cluster
    cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
    cluster.job_time = '48:00:00'
    cluster.gpu_type = '1080ti'
    cluster.memory_mb_per_node = 48000
    # any modules for code to run in env
    cluster.add_command('source activate pytorch_lightning')
    # name of exp
    job_display_name = hyperparams.tt_name.split('_')[0]
    job_display_name = job_display_name[0:3]
    # run hopt
    print('submitting jobs...')
    cluster.optimize_parallel_cluster_gpu(
        main,
        nb_trials=hyperparams.nb_hopt_trials,
        job_name=job_display_name
    )
 # run cluster hyperparameter search    
 optimize_on_cluster(hyperparams)
 ```
@@ -0,0 +1,139 @@
 ###### New project Quick Start    
 To start a new project define two files, a LightningModule and a Trainer file.    
 To illustrate Lightning power and simplicity, here's an example of a typical research flow.    
 ###### Case 1: BERT    
 Let's say you're working on something like BERT but want to try different ways of training or even different networks.  
 You would define a single LightningModule and use flags to switch between your different ideas.   
 ```python
 class BERT(pl.LightningModule):
    def __init__(self, model_name, task):
        self.task = task
        if model_name == 'transformer':
            self.net = Transformer()
        elif model_name == 'my_cool_version':
            self.net = MyCoolVersion()
    def training_step(self, batch, batch_nb):
        if self.task == 'standard_bert':
            # do standard bert training with self.net...
            # return loss
        if self.task == 'my_cool_task':
            # do my own version with self.net
            # return loss
 ```   
 ###### Case 2: COOLER NOT BERT    
 But if you wanted to try something **completely** different, you'd define a new module for that.    
 ```python
 class CoolerNotBERT(pl.LightningModule):
    def __init__(self):
        self.net = ...
    def training_step(self, batch, batch_nb):
        # do some other cool task
        # return loss   
 ```   
 ###### Rapid research flow    
 Then you could do rapid research by switching between these two and using the same trainer.   
 ```python
 if use_bert:
    model = BERT()
 else:
    model = CoolerNotBERT()
 trainer = Trainer(gpus=[0, 1, 2, 3], use_amp=True)
 trainer.fit(model)
 ```
 Notice a few things about this flow:   
 1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.   
 2. You get free GPU and 16-bit support without writing any of that code in your model.   
 3. You also get all of the capabilities below (without coding or testing yourself).     
 ---    
 ###### Templates 
 1. [MNIST LightningModule](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#minimal-example) 
 2. [Trainer](https://williamfalcon.github.io/pytorch-lightning/Trainer/)
    - [Basic CPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_cpu_template.py) 
    - [Multi-GPU Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/single_gpu_node_template.py)
    - [GPU cluster Trainer Template](https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/multi_node_cluster_template.py)
 ###### Docs shortcuts
 - [LightningModule](LightningModule/RequiredTrainerInterface/)  
 - [Trainer](Trainer/)  
 ###### Quick start examples 
 - [CPU example](examples/Examples/#cpu-hyperparameter-search)   
 - [Hyperparameter search on single GPU](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)    
 - [Hyperparameter search on multiple GPUs on same node](examples/Examples/#hyperparameter-search-on-a-single-or-multiple-gpus)  
 - [Hyperparameter search on a SLURM HPC cluster](examples/Examples/#Hyperparameter search on a SLURM HPC cluster)      
 ###### Checkpointing    
 - [Model saving](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#model-saving)
 - [Model loading](https://williamfalcon.github.io/pytorch-lightning/LightningModule/methods/#load-from-metrics) 
 - [Restoring training session](https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/#restoring-training-session)
 ###### Computing cluster (SLURM)    
 - [Running grid search on a cluster](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#running-grid-search-on-a-cluster) 
 - [Walltime auto-resubmit](https://williamfalcon.github.io/pytorch-lightning/Trainer/SLURM%20Managed%20Cluster#walltime-auto-resubmit)   
 ###### Debugging  
 - [Fast dev run](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#fast-dev-run)
 - [Inspect gradient norms](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#inspect-gradient-norms)
 - [Log GPU usage](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#Log-gpu-usage)
 - [Make model overfit on subset of data](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#make-model-overfit-on-subset-of-data)
 - [Print the parameter count by layer](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-the-parameter-count-by-layer)
 - [Pring which gradients are nan](https://williamfalcon.github.io/pytorch-lightning/Trainer/debugging/#print-which-gradients-are-nan)
 - [Print input and output size of every module in system](https://williamfalcon.github.io/pytorch-lightning/LightningModule/properties/#example_input_array)
 ###### Distributed training    
 - [16-bit mixed precision](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#16-bit-mixed-precision)
 - [Multi-GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-GPU)
 - [Multi-node](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#Multi-node)
 - [Single GPU](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#single-gpu)
 - [Self-balancing architecture](https://williamfalcon.github.io/pytorch-lightning/Trainer/Distributed%20training/#self-balancing-architecture)
 ###### Experiment Logging   
 - [Display metrics in progress bar](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#display-metrics-in-progress-bar)
 - [Log metric row every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#log-metric-row-every-k-batches)
 - [Process position](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#process-position)
 - [Tensorboard support](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#tensorboard-support)
 - [Save a snapshot of all hyperparameters](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#save-a-snapshot-of-all-hyperparameters) 
 - [Snapshot code for a training run](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#snapshot-code-for-a-training-run) 
 - [Write logs file to csv every k batches](https://williamfalcon.github.io/pytorch-lightning/Trainer/Logging/#write-logs-file-to-csv-every-k-batches)
 ###### Training loop    
 - [Accumulate gradients](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#accumulated-gradients)
 - [Force training for min or max epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-training-for-min-or-max-epochs)
 - [Force disable early stop](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#force-disable-early-stop)
 - [Gradient Clipping](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#gradient-clipping)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Learning rate scheduling](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
 - [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
 - [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
 ###### Validation loop    
 - [Check validation every n epochs](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#check-validation-every-n-epochs)
 - [Hooks](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/)
 - [Set how much of the validation set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-validation-set-to-check)
 - [Set how much of the test set to check](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-how-much-of-the-test-set-to-check)
 - [Set validation check frequency within 1 training epoch](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-validation-check-frequency-within-1-training-epoch)
 - [Set the number of validation sanity steps](https://williamfalcon.github.io/pytorch-lightning/Trainer/Validation%20loop/#set-the-number-of-validation-sanity-steps)
@@ -1,35 +0,0 @@
@ECHO OFF
 pushd %~dp0
 REM Command file for Sphinx documentation
 if "%SPHINXBUILD%" == "" (
 	set SPHINXBUILD=sphinx-build
 )
 set SOURCEDIR=source
 set BUILDDIR=build
 if "%1" == "" goto help
 %SPHINXBUILD% >NUL 2>NUL
 if errorlevel 9009 (
 	echo.
 	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
 	echo.installed, then set the SPHINXBUILD environment variable to point
 	echo.to the full path of the 'sphinx-build' executable. Alternatively you
 	echo.may add the Sphinx directory to PATH.
 	echo.
 	echo.If you don't have Sphinx installed, grab it from
 	echo.http://sphinx-doc.org/
 	exit /b 1
 )
 %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
 goto end
 :help
 %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
 :end
 popd
@@ -1,9 +1,2 @@
-sphinx>=1.8.3
+mkdocs-material==4.4.0
-recommonmark  # fails with badges
+mkdocs==1.0.4
 m2r  # fails with multi-line text
 nbsphinx
 pandoc
 docutils
 git+https://github.com/PytorchLightning/lightning_sphinx_theme.git
 sphinxcontrib-fulltoc
 sphinxcontrib-mockautodoc
@@ -1,62 +0,0 @@
 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
 <svg
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:cc="http://creativecommons.org/ns#"
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
   id="svg"
   version="1.1"
   width="16.000004"
   height="15.999986"
   viewBox="0 0 16.000004 15.999986"
   sodipodi:docname="lightning_icon.svg"
   inkscape:version="0.92.3 (2405546, 2018-03-11)">
  <metadata
     id="metadata13">
    <rdf:RDF>
      <cc:Work
         rdf:about="">
        <dc:format>image/svg+xml</dc:format>
        <dc:type
           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
        <dc:title></dc:title>
      </cc:Work>
    </rdf:RDF>
  </metadata>
  <defs
     id="defs11" />
  <sodipodi:namedview
     pagecolor="#ffffff"
     bordercolor="#666666"
     borderopacity="1"
     objecttolerance="10"
     gridtolerance="10"
     guidetolerance="10"
     inkscape:pageopacity="0"
     inkscape:pageshadow="2"
     inkscape:window-width="1920"
     inkscape:window-height="1028"
     id="namedview9"
     showgrid="false"
     inkscape:zoom="0.59"
     inkscape:cx="-669.05062"
     inkscape:cy="373.84245"
     inkscape:window-x="0"
     inkscape:window-y="0"
     inkscape:window-maximized="1"
     inkscape:current-layer="svg" />
  <path
     style="fill:#fbfbfb;fill-rule:evenodd;stroke:none;stroke-width:0.04002798"
     inkscape:connector-curvature="0"
     d="m 8.987101,1.723485 c -0.05588,0.03422 -4.121881,4.096544 -4.184645,4.180924 -0.02317,0.0311 -0.04587,0.06016 -0.05044,0.06456 -0.0087,0.0084 -0.07477,0.145063 -0.09679,0.20014 -0.05848,0.146583 -0.05804,0.44387 0.001,0.592413 0.08426,0.21243 0.08826,0.216754 1.576864,1.706274 0.779463,0.779947 1.41719,1.426877 1.41719,1.437604 0,0.0232 -0.253177,0.79848 -0.273873,0.838707 -0.0079,0.0153 -0.01433,0.04087 -0.01433,0.05684 0,0.01597 -0.0059,0.03587 -0.01313,0.04423 -0.0072,0.0084 -0.03678,0.09086 -0.06568,0.18333 -0.02893,0.09246 -0.05904,0.180647 -0.06693,0.195937 -0.0079,0.0153 -0.01437,0.04087 -0.01437,0.05684 0,0.01597 -0.0059,0.03586 -0.01313,0.04423 -0.0072,0.0084 -0.03679,0.09086 -0.06569,0.18333 -0.02893,0.09246 -0.05904,0.180643 -0.06693,0.195937 -0.0079,0.0153 -0.01437,0.04187 -0.01437,0.05908 0,0.0172 -0.0072,0.03574 -0.016,0.04119 -0.0088,0.0054 -0.016,0.02607 -0.016,0.04579 0,0.01973 -0.006,0.04271 -0.0134,0.05108 -0.0074,0.0084 -0.04439,0.112477 -0.08222,0.23136 -0.03787,0.118884 -0.151103,0.461124 -0.251693,0.760534 -0.489984,1.45874 -0.462444,1.36155 -0.413611,1.45938 0.06917,0.138657 0.23128,0.199741 0.358251,0.134974 0.07057,-0.03602 4.143298,-4.099985 4.245368,-4.236242 0.03382,-0.04515 0.09094,-0.165796 0.109916,-0.232123 0.0088,-0.03083 0.0243,-0.08498 0.03442,-0.120363 0.03346,-0.11668 0.0068,-0.361134 -0.0566,-0.520084 C 10.880518,9.229614 10.738898,9.079187 9.372744,7.714673 8.601524,6.944416 7.970523,6.302806 7.970523,6.288916 c 0,-0.01393 0.02817,-0.107833 0.0626,-0.208663 0.03442,-0.100834 0.07881,-0.237367 0.09859,-0.303414 0.0198,-0.06605 0.04207,-0.12693 0.04947,-0.135293 0.0074,-0.0084 0.0135,-0.03133 0.0135,-0.05108 0,-0.01973 0.0072,-0.04035 0.016,-0.04579 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04804 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04804 0.0088,-0.0054 0.016,-0.02707 0.016,-0.04803 0,-0.02097 0.0072,-0.04259 0.016,-0.04803 0.0088,-0.0054 0.016,-0.02397 0.016,-0.04119 0,-0.0172 0.0065,-0.04379 0.0144,-0.05908 0.0079,-0.0153 0.119204,-0.34484 0.247334,-0.73231 C 9.064507,2.979766 9.220177,2.513319 9.28226,2.330632 9.408267,1.960092 9.41367,1.921146 9.35255,1.826839 9.27225,1.703032 9.099973,1.654399 8.986893,1.723566"
     id="path0" />
  <path
     style="fill:#540c8c;fill-rule:evenodd;stroke:none;stroke-width:0.04002798"
     inkscape:connector-curvature="0"
     d="m 0.07719102,0.01733399 c -0.02187,0.0111 -0.04875,0.03799 -0.05984,0.05984 -0.0161,0.03173 -0.01937,1.62421701 -0.01633,7.94479601 l 0.0038,7.905086 0.03647,0.03646 0.03646,0.03647 H 8.00241 15.927073 l 0.03646,-0.03647 0.03647,-0.03646 V 8.002393 0.07773399 l -0.03647,-0.03646 -0.03646,-0.03647 -7.905086,-0.0038 c -6.320579,-0.003 -7.91305298,2.4e-4 -7.94479598,0.01633 M 9.193764,1.668208 c 0.259903,0.09046 0.275193,0.212427 0.09363,0.74628 C 8.845834,3.776859 8.388843,5.102846 7.991127,6.302606 L 9.415644,7.72492 c 1.24415,1.242111 1.51682,1.523547 1.51682,1.565414 0,0.0051 0.0133,0.03987 0.02953,0.07718 0.12913,0.296607 0.0877,0.664983 -0.103314,0.91872 -0.141456,0.187933 -4.207341,4.228478 -4.273468,4.246848 -0.139417,0.03871 -0.248653,-0.006 -0.34324,-0.140417 -0.07665,-0.108996 -0.06985,-0.137256 0.287004,-1.194633 0.34663,-1.101761 0.75901,-2.243218 1.08916,-3.290661 0,-0.0078 -0.636164,-0.650377 -1.413707,-1.427921 C 4.877658,7.152643 4.728155,6.995813 4.673718,6.87361 4.661948,6.84718 4.645988,6.81305 4.638168,6.79776 4.630368,6.78246 4.624038,6.75689 4.624038,6.74092 c 0,-0.01597 -0.0076,-0.03659 -0.01687,-0.04587 -0.02253,-0.02253 -0.02253,-0.436904 0,-0.45944 0.0093,-0.0093 0.01687,-0.0327 0.01687,-0.05204 0,-0.0363 0.06917,-0.178363 0.130414,-0.267907 0.07965,-0.1164 4.221831,-4.237681 4.259458,-4.237921 0.02047,-1.2e-4 0.04803,-0.0072 0.06124,-0.01577 0.03147,-0.02033 0.04415,-0.01967 0.118603,0.0062"
     id="path1"
     sodipodi:nodetypes="ccscccccccccccscccccscccccccccsssscccc" />
 </svg>
@@ -1,61 +0,0 @@
 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
 <svg
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:cc="http://creativecommons.org/ns#"
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
   id="svg"
   version="1.1"
   width="400"
   height="400"
   viewBox="0, 0, 400,400"
   sodipodi:docname="lightning_logo.svg"
   inkscape:version="0.92.3 (2405546, 2018-03-11)">
  <metadata
     id="metadata13">
    <rdf:RDF>
      <cc:Work
         rdf:about="">
        <dc:format>image/svg+xml</dc:format>
        <dc:type
           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
      </cc:Work>
    </rdf:RDF>
  </metadata>
  <defs
     id="defs11" />
  <sodipodi:namedview
     pagecolor="#ffffff"
     bordercolor="#666666"
     borderopacity="1"
     objecttolerance="10"
     gridtolerance="10"
     guidetolerance="10"
     inkscape:pageopacity="0"
     inkscape:pageshadow="2"
     inkscape:window-width="1920"
     inkscape:window-height="1028"
     id="namedview9"
     showgrid="false"
     inkscape:zoom="9.44"
     inkscape:cx="203.07907"
     inkscape:cy="335.32491"
     inkscape:window-x="0"
     inkscape:window-y="0"
     inkscape:window-maximized="1"
     inkscape:current-layer="svg" />
  <path
     style="fill:#fbfbfb;fill-rule:evenodd;stroke:none"
     inkscape:connector-curvature="0"
     d="m 224.6,43.137 c -1.396,0.855 -102.975,102.342 -104.543,104.45 -0.579,0.777 -1.146,1.503 -1.26,1.613 -0.218,0.21 -1.868,3.624 -2.418,5 -1.461,3.662 -1.45,11.089 0.022,14.8 2.105,5.307 2.205,5.415 39.394,42.627 19.473,19.485 35.405,35.647 35.405,35.915 0,0.58 -6.325,19.948 -6.842,20.953 -0.197,0.382 -0.358,1.021 -0.358,1.42 0,0.399 -0.147,0.896 -0.328,1.105 -0.18,0.209 -0.919,2.27 -1.641,4.58 -0.723,2.31 -1.475,4.513 -1.672,4.895 -0.198,0.382 -0.359,1.021 -0.359,1.42 0,0.399 -0.147,0.896 -0.328,1.105 -0.18,0.209 -0.919,2.27 -1.641,4.58 -0.723,2.31 -1.475,4.513 -1.672,4.895 -0.198,0.382 -0.359,1.046 -0.359,1.476 0,0.43 -0.18,0.893 -0.4,1.029 -0.22,0.136 -0.4,0.651 -0.4,1.144 0,0.493 -0.151,1.067 -0.335,1.276 -0.184,0.209 -1.109,2.81 -2.054,5.78 -0.946,2.97 -3.775,11.52 -6.288,19 -12.241,36.443 -11.553,34.015 -10.333,36.459 1.728,3.464 5.778,4.99 8.95,3.372 1.763,-0.9 103.51,-102.428 106.06,-105.832 0.845,-1.128 2.272,-4.142 2.746,-5.799 0.22,-0.77 0.607,-2.123 0.86,-3.007 0.836,-2.915 0.171,-9.022 -1.414,-12.993 -1.493,-3.741 -5.031,-7.499 -39.161,-41.588 C 214.964,173.569 199.2,157.54 199.2,157.193 c 0,-0.348 0.704,-2.694 1.564,-5.213 0.86,-2.519 1.969,-5.93 2.463,-7.58 0.495,-1.65 1.051,-3.171 1.236,-3.38 0.186,-0.209 0.337,-0.783 0.337,-1.276 0,-0.493 0.18,-1.008 0.4,-1.144 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.676 0.4,-1.2 0,-0.524 0.18,-1.064 0.4,-1.2 0.22,-0.136 0.4,-0.599 0.4,-1.029 0,-0.43 0.162,-1.094 0.36,-1.476 0.197,-0.382 2.978,-8.615 6.179,-18.295 3.2,-9.68 7.089,-21.333 8.64,-25.897 3.148,-9.257 3.283,-10.23 1.756,-12.586 -2.006,-3.093 -6.31,-4.308 -9.135,-2.58"
     id="path0" />
  <path
     style="fill:#540c8c;fill-rule:evenodd;stroke:none"
     inkscape:connector-curvature="0"
     d="M 2.008,0.513 C 1.462,0.79 0.79,1.462 0.513,2.008 0.111,2.801 0.029,42.585 0.105,200.489 L 0.2,397.978 1.111,398.889 2.022,399.8 H 200 397.978 l 0.911,-0.911 0.911,-0.911 V 200 2.022 L 398.889,1.111 397.978,0.2 200.489,0.105 C 42.585,0.029 2.801,0.111 2.008,0.513 m 227.755,41.243 c 6.493,2.26 6.875,5.307 2.339,18.644 -11.0313,34.035452 -22.44803,67.16196 -32.384,97.135 l 35.588,35.533 c 31.082,31.031 37.894,38.062 37.894,39.108 0,0.128 0.332,0.996 0.738,1.928 3.226,7.41 2.191,16.613 -2.581,22.952 -3.534,4.695 -105.11,105.638 -106.762,106.097 -3.483,0.967 -6.212,-0.15 -8.575,-3.508 -1.915,-2.723 -1.745,-3.429 7.17,-29.845 8.65971,-27.52475 18.96205,-56.04122 27.21,-82.209 0,-0.195 -15.893,-16.248 -35.318,-35.673 -33.146,-33.147 -36.881,-37.065 -38.241,-40.118 -0.294,-0.66 -0.693,-1.513 -0.888,-1.895 -0.194,-0.382 -0.353,-1.021 -0.353,-1.42 0,-0.399 -0.189,-0.914 -0.421,-1.146 -0.563,-0.563 -0.563,-10.915 0,-11.478 0.232,-0.232 0.421,-0.817 0.421,-1.3 0,-0.907 1.728,-4.456 3.258,-6.693 C 120.848,144.96 224.33,42 225.27,41.994 c 0.511,-0.003 1.2,-0.181 1.53,-0.394 0.786,-0.508 1.103,-0.491 2.963,0.156"
     id="path1"
     sodipodi:nodetypes="ccscccccccccccscccccscccccccccsssscccc" />
 </svg>
@@ -1,62 +0,0 @@
 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
 <svg
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:cc="http://creativecommons.org/ns#"
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
   id="svg"
   version="1.1"
   width="47.999985"
   height="47.999943"
   viewBox="0 0 47.999985 47.999943"
   sodipodi:docname="lightning_logo.svg"
   inkscape:version="0.92.3 (2405546, 2018-03-11)">
  <metadata
     id="metadata13">
    <rdf:RDF>
      <cc:Work
         rdf:about="">
        <dc:format>image/svg+xml</dc:format>
        <dc:type
           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
        <dc:title />
      </cc:Work>
    </rdf:RDF>
  </metadata>
  <defs
     id="defs11" />
  <sodipodi:namedview
     pagecolor="#ffffff"
     bordercolor="#666666"
     borderopacity="1"
     objecttolerance="10"
     gridtolerance="10"
     guidetolerance="10"
     inkscape:pageopacity="0"
     inkscape:pageshadow="2"
     inkscape:window-width="1920"
     inkscape:window-height="1028"
     id="namedview9"
     showgrid="false"
     inkscape:zoom="0.59"
     inkscape:cx="-347.96588"
     inkscape:cy="389.84243"
     inkscape:window-x="0"
     inkscape:window-y="0"
     inkscape:window-maximized="1"
     inkscape:current-layer="svg" />
  <path
     style="fill:#fbfbfb;fill-rule:evenodd;stroke:none;stroke-width:0.12008391"
     inkscape:connector-curvature="0"
     d="m 26.961294,5.1704519 c -0.16764,0.10267 -12.36564,12.2896301 -12.55393,12.5427701 -0.0695,0.0933 -0.13762,0.18048 -0.15131,0.19369 -0.0262,0.0252 -0.22432,0.43519 -0.29036,0.60042 -0.17544,0.43975 -0.17412,1.33161 0.003,1.77724 0.25278,0.63729 0.26479,0.65026 4.73059,5.11882 2.33839,2.33984 4.25157,4.28063 4.25157,4.31281 0,0.0696 -0.75953,2.39544 -0.82162,2.51612 -0.0237,0.0459 -0.043,0.12261 -0.043,0.17052 0,0.0479 -0.0177,0.1076 -0.0394,0.13269 -0.0216,0.0251 -0.11035,0.27259 -0.19705,0.54999 -0.0868,0.27739 -0.17713,0.54194 -0.20078,0.58781 -0.0238,0.0459 -0.0431,0.1226 -0.0431,0.17052 0,0.0479 -0.0177,0.10759 -0.0394,0.13269 -0.0216,0.0251 -0.11036,0.27259 -0.19706,0.54999 -0.0868,0.27739 -0.17712,0.54193 -0.20078,0.58781 -0.0238,0.0459 -0.0431,0.1256 -0.0431,0.17724 0,0.0516 -0.0216,0.10723 -0.048,0.12357 -0.0264,0.0163 -0.048,0.0782 -0.048,0.13737 0,0.0592 -0.0181,0.12813 -0.0402,0.15323 -0.0221,0.0251 -0.13318,0.33743 -0.24666,0.69408 -0.1136,0.35665 -0.45331,1.38337 -0.75508,2.2816 -1.46995,4.37622 -1.38733,4.08465 -1.24083,4.37814 0.2075,0.41597 0.69384,0.59922 1.07475,0.40492 0.21171,-0.10807 12.42989,-12.29995 12.7361,-12.70872 0.10147,-0.13545 0.27283,-0.49739 0.32975,-0.69637 0.0264,-0.0925 0.0729,-0.25493 0.10327,-0.36109 0.10039,-0.35004 0.0205,-1.0834 -0.1698,-1.56025 -0.17928,-0.44923 -0.60414,-0.90051 -4.7026,-4.99405 -2.31366,-2.31077 -4.20666,-4.2356 -4.20666,-4.27727 0,-0.0418 0.0845,-0.3235 0.18781,-0.62599 0.10327,-0.3025 0.23644,-0.7121 0.29577,-0.91024 0.0594,-0.19814 0.1262,-0.38079 0.14842,-0.40588 0.0223,-0.0251 0.0405,-0.094 0.0405,-0.15323 0,-0.0592 0.0216,-0.12105 0.048,-0.13738 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.14411 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.14411 0.0264,-0.0163 0.048,-0.0812 0.048,-0.1441 0,-0.0629 0.0216,-0.12777 0.048,-0.1441 0.0264,-0.0163 0.048,-0.0719 0.048,-0.12356 0,-0.0516 0.0195,-0.13137 0.0432,-0.17725 0.0237,-0.0459 0.35761,-1.03452 0.742,-2.19693 0.38427,-1.1624101 0.85128,-2.5617501 1.03753,-3.1098101 0.37802,-1.11162 0.39423,-1.22846 0.21087,-1.51138 -0.24089,-0.37142 -0.75773,-0.51732 -1.09697,-0.30982"
     id="path0" />
  <path
     style="fill:#540c8c;fill-rule:evenodd;stroke:none;stroke-width:0.12008391"
     inkscape:connector-curvature="0"
     d="m 0.2315739,0.05200186 c -0.0656,0.0333 -0.14626,0.11396 -0.17952,0.17952 -0.0483,0.0952 -0.0581,4.87265004 -0.049,23.83438014 l 0.0114,23.71525 0.1094,0.10939 0.10939,0.1094 h 23.7739701 23.77398 l 0.10939,-0.1094 0.1094,-0.10939 V 24.007172 0.23320186 l -0.1094,-0.10939 -0.10939,-0.1094 -23.71525,-0.0114 c -18.9617301,-0.009 -23.7391501,7.2e-4 -23.8343801,0.049 M 27.581274,5.0046319 c 0.77971,0.27139 0.82558,0.63728 0.28088,2.23884 -1.32468,4.0871101 -2.69565,8.0650701 -3.8888,11.6643501 l 4.27355,4.26694 c 3.73245,3.72633 4.55046,4.57064 4.55046,4.69624 0,0.0154 0.0399,0.11961 0.0886,0.23153 0.38739,0.88982 0.2631,1.99495 -0.30994,2.75616 -0.42437,0.5638 -12.62202,12.68543 -12.8204,12.74054 -0.41825,0.11613 -0.74596,-0.018 -1.02972,-0.42125 -0.22996,-0.32699 -0.20954,-0.41177 0.86101,-3.5839 1.03989,-3.30528 2.27703,-6.72965 3.26748,-9.87198 0,-0.0234 -1.90849,-1.95113 -4.24112,-4.28376 -3.98031,-3.98042 -4.42882,-4.45091 -4.59213,-4.81752 -0.0353,-0.0793 -0.0832,-0.18169 -0.10664,-0.22756 -0.0233,-0.0459 -0.0424,-0.12261 -0.0424,-0.17052 0,-0.0479 -0.0227,-0.10976 -0.0506,-0.13762 -0.0676,-0.0676 -0.0676,-1.31071 0,-1.37832 0.0279,-0.0279 0.0506,-0.0981 0.0506,-0.15611 0,-0.10891 0.20751,-0.53509 0.39124,-0.80372 0.23896,-0.3492 12.66549,-12.7130401 12.77837,-12.7137601 0.0614,-3.6e-4 0.1441,-0.0217 0.18372,-0.0473 0.0944,-0.061 0.13246,-0.059 0.35581,0.0187"
     id="path1"
     sodipodi:nodetypes="ccscccccccccccscccccscccccccccsssscccc" />
 </svg>
@@ -1,17 +0,0 @@
 {%- set external_urls = {
  'github': 'https://github.com/PytorchLightning/pytorch-lightning',
  'github_issues': 'https://github.com/PytorchLightning/pytorch-lightning/issues',
  'contributing': 'https://github.com/PytorchLightning/pytorch-lightning/blob/master/CONTRIBUTING.md',
  'docs': 'https://pytorch-lightning.rtfd.io/en/latest',
  'twitter': 'https://twitter.com/PyTorchLightnin',
  'discuss': 'https://discuss.pytorch.org',
  'tutorials': 'https://pytorch-lightning.rtfd.io/en/latest/',
  'previous_pytorch_versions': 'https://pytorch-lightning.rtfd.io/en/latest/',
  'home': 'https://pytorch-lightning.rtfd.io/en/latest/',
  'get_started': 'https://pytorch-lightning.rtfd.io/en/latest/',
  'features': 'https://pytorch-lightning.rtfd.io/en/latest/',
  'blog': 'https://pytorch-lightning.rtfd.io/en/latest/',
  'resources': 'https://pytorch-lightning.rtfd.io/en/latest/',
  'support': 'https://pytorch-lightning.rtfd.io/en/latest/',
 }
 -%}
@@ -1,14 +0,0 @@
 .. role:: hidden
    :class: hidden-section
 Callbacks
 ===========
 .. automodule:: pytorch_lightning.callbacks
   :exclude-members:
        _del_model,
        _save_model,
        on_epoch_end,
        on_train_end,
        on_epoch_begin,
        check_monitor_top_k,
        on_train_begin,
@@ -1,21 +0,0 @@
 Multi-gpu (same node) training
 ==============================
 Multi-node training
 ====================
 16-bit precision
 =================
 gradient clipping
 =================
 modifying training via hooks
 =============================
 .. toctree::
   :maxdepth: 3
   pl_examples
@@ -1,350 +0,0 @@
 # -*- coding: utf-8 -*-
 #
 # Configuration file for the Sphinx documentation builder.
 #
 # This file does only contain a selection of the most common options. For a
 # full list see the documentation:
 # http://www.sphinx-doc.org/en/master/config
 # -- Path setup --------------------------------------------------------------
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 import os
 import sys
 import glob
 import shutil
 import inspect
 # import m2r
 import builtins
 import pt_lightning_sphinx_theme
 PATH_HERE = os.path.abspath(os.path.dirname(__file__))
 PATH_ROOT = os.path.join(PATH_HERE, '..', '..')
 sys.path.insert(0, os.path.abspath(PATH_ROOT))
 builtins.__LIGHTNING_SETUP__ = True
 import pytorch_lightning  # noqa: E402
 # -- Project documents -------------------------------------------------------
 # # export the documentation
 # with open('intro.rst', 'w') as fp:
 #     intro = pytorch_lightning.__doc__.replace(os.linesep + ' ', '')
 #     fp.write(m2r.convert(intro))
 #     # fp.write(pytorch_lightning.__doc__)
 # # export the READme
 # with open(os.path.join(PATH_ROOT, 'README.md'), 'r') as fp:
 #     readme = fp.read()
 # # replace all paths to relative
 # for ndir in (os.path.basename(p) for p in glob.glob(os.path.join(PATH_ROOT, '*'))
 #              if os.path.isdir(p)):
 #     readme = readme.replace('](%s/' % ndir, '](%s/%s/' % (PATH_ROOT, ndir))
 # with open('readme.md', 'w') as fp:
 #     fp.write(readme)
 for md in glob.glob(os.path.join(PATH_ROOT, '.github', '*.md')):
    shutil.copy(md, os.path.join(PATH_HERE, os.path.basename(md)))
 # -- Project information -----------------------------------------------------
 project = 'PyTorch-Lightning'
 copyright = pytorch_lightning.__copyright__
 author = pytorch_lightning.__author__
 # The short X.Y version
 version = pytorch_lightning.__version__
 # The full version, including alpha/beta/rc tags
 release = pytorch_lightning.__version__
 # -- General configuration ---------------------------------------------------
 # If your documentation needs a minimal Sphinx version, state it here.
 needs_sphinx = '1.4'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
    'sphinx.ext.autodoc',
    'sphinxcontrib.mockautodoc',
    # 'sphinxcontrib.fulltoc',  # breaks pytorch-theme with unexpected kw argument 'titles_only'
    'sphinx.ext.doctest',
    'sphinx.ext.intersphinx',
    'sphinx.ext.todo',
    'sphinx.ext.coverage',
    'sphinx.ext.linkcode',
    'sphinx.ext.autosummary',
    'sphinx.ext.napoleon',
    'recommonmark',
    'sphinx.ext.autosectionlabel',
    # 'm2r',
    'nbsphinx',
 ]
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 # https://berkeley-stat159-f17.github.io/stat159-f17/lectures/14-sphinx..html#conf.py-(cont.)
 # https://stackoverflow.com/questions/38526888/embed-ipython-notebook-in-sphinx-document
 # I execute the notebooks manually in advance. If notebooks test the code,
 # they should be run at build time.
 nbsphinx_execute = 'never'
 nbsphinx_allow_errors = True
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
 # source_suffix = ['.rst', '.md']
 # source_suffix = ['.rst', '.md', '.ipynb']
 source_suffix = {
    '.rst': 'restructuredtext',
    '.txt': 'markdown',
    '.md': 'markdown',
    '.ipynb': 'nbsphinx',
 }
 # The master toctree document.
 master_doc = 'index'
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
 language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = ['*.test_*']
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = None
 # -- Options for HTML output -------------------------------------------------
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 # http://www.sphinx-doc.org/en/master/usage/theming.html#builtin-themes
 # html_theme = 'bizstyle'
 # https://sphinx-themes.org
 html_theme = 'pt_lightning_sphinx_theme'
 html_theme_path = [pt_lightning_sphinx_theme.get_html_theme_path()]
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 html_theme_options = {
    'pytorch_project': pytorch_lightning.__homepage__,
    'canonical_url': pytorch_lightning.__homepage__,
    'collapse_navigation': False,
    'display_version': True,
    'logo_only': False,
 }
 html_logo = '_static/images/lightning_logo-name.svg'
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.
 #
 # The default sidebars (for documents that don't match any pattern) are
 # defined by theme itself.  Builtin themes are using these templates by
 # default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
 # 'searchbox.html']``.
 #
 # html_sidebars = {}
 # -- Options for HTMLHelp output ---------------------------------------------
 # Output file base name for HTML help builder.
 htmlhelp_basename = project + '-doc'
 # -- Options for LaTeX output ------------------------------------------------
 latex_elements = {
    # The paper size ('letterpaper' or 'a4paper').
    # 'papersize': 'letterpaper',
    # The font size ('10pt', '11pt' or '12pt').
    # 'pointsize': '10pt',
    # Additional stuff for the LaTeX preamble.
    # 'preamble': '',
    # Latex figure (float) alignment
    'figure_align': 'htbp',
 }
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
    (master_doc, project + '.tex', project + ' Documentation', author, 'manual'),
 ]
 # -- Options for manual page output ------------------------------------------
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
    (master_doc, project, project + ' Documentation', [author], 1)
 ]
 # -- Options for Texinfo output ----------------------------------------------
 # Grouping the document tree into Texinfo files. List of tuples
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
    (master_doc, project, project + ' Documentation', author, project,
     'One line description of project.', 'Miscellaneous'),
 ]
 # -- Options for Epub output -------------------------------------------------
 # Bibliographic Dublin Core info.
 epub_title = project
 # The unique identifier of the text. This can be a ISBN number
 # or the project homepage.
 #
 # epub_identifier = ''
 # A unique identification for the text.
 #
 # epub_uid = ''
 # A list of files that should not be packed into the epub file.
 epub_exclude_files = ['search.html']
 # -- Extension configuration -------------------------------------------------
 # -- Options for intersphinx extension ---------------------------------------
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {'https://docs.python.org/': None}
 # -- Options for todo extension ----------------------------------------------
 # If true, `todo` and `todoList` produce output, else they produce nothing.
 todo_include_todos = True
 # https://github.com/rtfd/readthedocs.org/issues/1139
 # I use sphinx-apidoc to auto-generate API documentation for my project.
 # Right now I have to commit these auto-generated files to my repository
 # so that RTD can build them into HTML docs. It'd be cool if RTD could run
 # sphinx-apidoc for me, since it's easy to forget to regen API docs
 # and commit them to my repo after making changes to my code.
 PACKAGES = [
    pytorch_lightning.__name__,
    'pl_examples',
 ]
 def run_apidoc(_):
    for pkg in PACKAGES:
        argv = ['-e', '-o', PATH_HERE, os.path.join(PATH_HERE, PATH_ROOT, pkg),
                '**/test_*', '--force', '--private', '--module-first']
        try:
            # Sphinx 1.7+
            from sphinx.ext import apidoc
            apidoc.main(argv)
        except ImportError:
            # Sphinx 1.6 (and earlier)
            from sphinx import apidoc
            argv.insert(0, apidoc.__file__)
            apidoc.main(argv)
 def setup(app):
    app.connect('builder-inited', run_apidoc)
 # copy all notebooks to local folder
 path_nbs = os.path.join(PATH_HERE, 'notebooks')
 if not os.path.isdir(path_nbs):
    os.mkdir(path_nbs)
 for path_ipynb in glob.glob(os.path.join(PATH_ROOT, 'notebooks', '*.ipynb')):
    path_ipynb2 = os.path.join(path_nbs, os.path.basename(path_ipynb))
    shutil.copy(path_ipynb, path_ipynb2)
 # Ignoring Third-party packages
 # https://stackoverflow.com/questions/15889621/sphinx-how-to-exclude-imports-in-automodule
 MOCK_REQUIRE_PACKAGES = []
 with open(os.path.join(PATH_ROOT, 'requirements.txt'), 'r') as fp:
    for ln in fp.readlines():
        found = [ln.index(ch) for ch in list(',=<>#') if ch in ln]
        pkg = ln[:min(found)] if found else ln
        if pkg.rstrip():
            MOCK_REQUIRE_PACKAGES.append(pkg.rstrip())
 # TODO: better parse from package since the import name and package name may differ
 MOCK_MANUAL_PACKAGES = ['torch', 'torchvision', 'sklearn', 'test_tube', 'mlflow', 'comet_ml', 'wandb', 'neptune']
 autodoc_mock_imports = MOCK_REQUIRE_PACKAGES + MOCK_MANUAL_PACKAGES
 # for mod_name in MOCK_REQUIRE_PACKAGES:
 #     sys.modules[mod_name] = mock.Mock()
 # Options for the linkcode extension
 # ----------------------------------
 github_user = 'PyTorchLightning'
 github_repo = project
 # Resolve function
 # This function is used to populate the (source) links in the API
 def linkcode_resolve(domain, info):
    def find_source():
        # try to find the file and line number, based on code from numpy:
        # https://github.com/numpy/numpy/blob/master/doc/source/conf.py#L286
        obj = sys.modules[info['module']]
        for part in info['fullname'].split('.'):
            obj = getattr(obj, part)
        fname = inspect.getsourcefile(obj)
        # https://github.com/rtfd/readthedocs.org/issues/5735
        if any([s in fname for s in ('readthedocs', 'checkouts')]):
            # /home/docs/checkouts/readthedocs.org/user_builds/pytorch_lightning/checkouts/
            #  devel/pytorch_lightning/utilities/cls_experiment.py#L26-L176
            path_top = os.path.abspath(os.path.join('..', '..', '..'))
            fname = os.path.relpath(fname, start=path_top)
        else:
            # Local build, imitate master
            fname = 'master/' + os.path.relpath(fname, start=os.path.abspath('..'))
        source, lineno = inspect.getsourcelines(obj)
        return fname, lineno, lineno + len(source) - 1
    if domain != 'py' or not info['module']:
        return None
    try:
        filename = '%s#L%d-L%d' % find_source()
    except Exception:
        filename = info['module'].replace('.', '/') + '.py'
    # import subprocess
    # tag = subprocess.Popen(['git', 'rev-parse', 'HEAD'], stdout=subprocess.PIPE,
    #                        universal_newlines=True).communicate()[0][:-1]
    return "https://github.com/%s/%s/blob/%s" \
           % (github_user, github_repo, filename)
 autodoc_member_order = 'groupwise'
 autoclass_content = 'both'
 autodoc_default_flags = [
    'members', 'undoc-members', 'show-inheritance', 'private-members',
    # 'special-members', 'inherited-members'
 ]
@@ -1,8 +0,0 @@
 Documentation
 =============
 .. toctree::
   :maxdepth: 4
   pytorch_lightning
@@ -1,34 +0,0 @@
 GAN
 ====
 .. toctree::
   :maxdepth: 3
   pl_examples.domain_templates.gan
 MNIST
 ====
 .. toctree::
   :maxdepth: 3
   pl_examples.basic_examples.lightning_module_template
 Multi-node (ddp) MNIST
 ====
 .. toctree::
   :maxdepth: 3
   pl_examples.multi_node_examples.multi_node_ddp_demo
 Multi-node (ddp2) MNIST
 ====
 .. toctree::
   :maxdepth: 3
   pl_examples.multi_node_examples.multi_node_ddp2_demo
 Imagenet
 ====
 .. toctree::
   :maxdepth: 3
   pl_examples.full_examples.imagenet.imagenet_example
@@ -1,63 +0,0 @@
 .. PyTorch-Lightning documentation master file, created by
   sphinx-quickstart on Fri Nov 15 07:48:22 2019.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.
 PyTorch-Lightning Documentation
 =============================
 .. toctree::
   :maxdepth: 1
   :name: start
   :caption: Start Here
   new-project
 .. toctree::
   :maxdepth: 4
   :name: docs
   :caption: Python API
   callbacks
   lightning-module
   logging
   trainer
 .. toctree::
   :maxdepth: 1
   :name: Examples
   :caption: Examples
   examples
 .. toctree::
   :maxdepth: 1
   :name: Tutorials
   :caption: Tutorials
   tutorials
 .. toctree::
   :maxdepth: 1
   :name: Common Use Cases
   :caption: Common Use Cases
   common-cases
 .. toctree::
   :maxdepth: 1
   :name: community
   :caption: Community
   CODE_OF_CONDUCT.md
   CONTRIBUTING.md
   BECOMING_A_CORE_CONTRIBUTOR.md
   governance.md
 Indices and tables
 ------------------
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
@@ -1,10 +0,0 @@
 .. role:: hidden
    :class: hidden-section
 LightningModule
 ===========
 .. automodule:: pytorch_lightning.core
   :exclude-members:
        _abc_impl,
        summarize,
@@ -1,12 +0,0 @@
 .. role:: hidden
    :class: hidden-section
 Logging
 ===========
 .. automodule:: pytorch_lightning.logging
   :exclude-members:
        _abc_impl,
        _save_model,
        on_epoch_end,
        on_train_end,
        on_epoch_begin,
@@ -1,7 +0,0 @@
 pl_examples
 ===========
 .. toctree::
   :maxdepth: 4
   pl_examples
@@ -1,72 +0,0 @@
 Quick Start
 ===========
 | To start a new project define two files, a LightningModule and a Trainer file.
 | To illustrate the power of Lightning and its simplicity, here's an example of a typical research flow.
 Case 1: BERT
 ------------
 | Let's say you're working on something like BERT but want to try different ways of training or even different networks.
 | You would define a single LightningModule and use flags to switch between your different ideas.
 .. code-block:: python
    class BERT(pl.LightningModule):
        def __init__(self, model_name, task):
            self.task = task
            if model_name == 'transformer':
                self.net = Transformer()
            elif model_name == 'my_cool_version':
                self.net = MyCoolVersion()
        def training_step(self, batch, batch_idx):
            if self.task == 'standard_bert':
                # do standard bert training with self.net...
                # return loss
            if self.task == 'my_cool_task':
                # do my own version with self.net
                # return loss
 Case 2: COOLER NOT BERT
 -----------------------
 But if you wanted to try something **completely** different, you'd define a new module for that.
 .. code-block:: python
    class CoolerNotBERT(pl.LightningModule):
        def __init__(self):
            self.net = ...
        def training_step(self, batch, batch_idx):
            # do some other cool task
            # return loss
 Rapid research flow
 -------------------
 Then you could do rapid research by switching between these two and using the same trainer.
 .. code-block:: python
    if use_bert:
        model = BERT()
    else:
        model = CoolerNotBERT()
    trainer = Trainer(gpus=4, use_amp=True)
    trainer.fit(model)
 **Notice a few things about this flow:**
 1. You're writing pure PyTorch... no unnecessary abstractions or new libraries to learn.
 2. You get free GPU and 16-bit support without writing any of that code in your model.
 3. You also get early stopping, multi-gpu training, 16-bit and MUCH more without coding anything!
@@ -1,21 +0,0 @@
 .. role:: hidden
    :class: hidden-section
 Trainer
 ===========
 .. automodule:: pytorch_lightning.trainer
   :members: fit, test
   :exclude-members: 
        run_pretrain_routine, 
        _abc_impl, 
        _Trainer__set_root_gpu, 
        _Trainer__init_optimizers,
        _Trainer__parse_gpu_ids,
        _Trainer__configure_schedulers,
        data_parallel,
        num_gpus,
        slurm_job_id,
        tng_tqdm_dic,
        training_tqdm_dict,
        init_optimizers,
        configure_schedulers
@@ -1,20 +0,0 @@
 Refactoring PyTorch into Lightning
 ==================================
 `Tutorial <https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538>`_
 Start a research project
 =========================
 `Research seed <https://github.com/PytorchLightning/pytorch-lightning-conference-seed>`_
 Basic Lightning use
 ====================
 `Tutorial <https://towardsdatascience.com/supercharge-your-ai-research-with-pytorch-lightning-337948a99eec>`_
 9 key Lightning  tricks
 ========================
 `Tutorial <9 key speed features in Pytorch-Lightning>`_
 Multi-node training on SLURM
 =============================
 `Tutorial <https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd>`_
@@ -0,0 +1,5 @@
 from .new_project_templates.lightning_module_template import LightningTemplateModel
 __all__ = [
    'LightningTemplateModel'
 ]
@@ -1,22 +1,20 @@
 """
 Example template for defining a system
 """
 import logging
 import os
 from argparse import ArgumentParser
 from collections import OrderedDict
 import torch
 import torch.nn as nn
-import torch.nn.functional as F
+from torchvision.datasets import MNIST
 import torchvision.transforms as transforms
 import torch
 import torch.nn.functional as F
 from test_tube import HyperOptArgumentParser
 from torch import optim
 from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
 from torchvision.datasets import MNIST
 import pytorch_lightning as pl
-from pytorch_lightning.core.lightning import LightningModule
+from pytorch_lightning.root_module.root_module import LightningModule
 class LightningTemplateModel(LightningModule):
@@ -81,14 +79,14 @@ class LightningTemplateModel(LightningModule):
        nll = F.nll_loss(logits, labels)
        return nll
-    def training_step(self, batch, batch_idx):
+    def training_step(self, data_batch, batch_i):
        """
        Lightning calls this inside the training loop
-        :param batch:
+        :param data_batch:
        :return:
        """
        # forward pass
-        x, y = batch
+        x, y = data_batch
        x = x.view(x.size(0), -1)
        y_hat = self.forward(x)
@@ -97,26 +95,23 @@ class LightningTemplateModel(LightningModule):
        loss_val = self.loss(y, y_hat)
        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp or self.trainer.use_ddp2:
+        if self.trainer.use_dp:
            loss_val = loss_val.unsqueeze(0)
        tqdm_dict = {'train_loss': loss_val}
        output = OrderedDict({
-            'loss': loss_val,
+            'loss': loss_val
            'progress_bar': tqdm_dict,
            'log': tqdm_dict
        })
        # can also return just a scalar instead of a dict (return loss_val)
        return output
-    def validation_step(self, batch, batch_idx):
+    def validation_step(self, data_batch, batch_i):
        """
        Lightning calls this inside the validation loop
-        :param batch:
+        :param data_batch:
        :return:
        """
-        x, y = batch
+        x, y = data_batch
        x = x.view(x.size(0), -1)
        y_hat = self.forward(x)
@@ -131,7 +126,7 @@ class LightningTemplateModel(LightningModule):
            val_acc = val_acc.cuda(loss_val.device.index)
        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp or self.trainer.use_ddp2:
+        if self.trainer.use_dp:
            loss_val = loss_val.unsqueeze(0)
            val_acc = val_acc.unsqueeze(0)
@@ -159,22 +154,21 @@ class LightningTemplateModel(LightningModule):
            val_loss = output['val_loss']
            # reduce manually when using dp
-            if self.trainer.use_dp or self.trainer.use_ddp2:
+            if self.trainer.use_dp:
                val_loss = torch.mean(val_loss)
            val_loss_mean += val_loss
            # reduce manually when using dp
            val_acc = output['val_acc']
-            if self.trainer.use_dp or self.trainer.use_ddp2:
+            if self.trainer.use_dp:
                val_acc = torch.mean(val_acc)
            val_acc_mean += val_acc
        val_loss_mean /= len(outputs)
        val_acc_mean /= len(outputs)
-        tqdm_dict = {'val_loss': val_loss_mean, 'val_acc': val_acc_mean}
+        tqdm_dic = {'val_loss': val_loss_mean, 'val_acc': val_acc_mean}
-        result = {'progress_bar': tqdm_dict, 'log': tqdm_dict, 'val_loss': val_loss_mean}
+        return tqdm_dic
        return result
    # ---------------------
    # TRAINING SETUP
@@ -195,37 +189,37 @@ class LightningTemplateModel(LightningModule):
        dataset = MNIST(root=self.hparams.data_root, train=train,
                        transform=transform, download=True)
-        # when using multi-node (ddp) we need to add the  datasampler
+        # when using multi-node (ddp) we need to add the datasampler
        train_sampler = None
        batch_size = self.hparams.batch_size
        if self.use_ddp:
-            train_sampler = DistributedSampler(dataset)
+            train_sampler = DistributedSampler(dataset, rank=self.trainer.proc_rank)
            batch_size = batch_size // self.trainer.world_size  # scale batch size
        should_shuffle = train_sampler is None
        loader = DataLoader(
            dataset=dataset,
            batch_size=batch_size,
            shuffle=should_shuffle,
-            sampler=train_sampler,
+            sampler=train_sampler
            num_workers=0
        )
        return loader
    @pl.data_loader
-    def train_dataloader(self):
+    def tng_dataloader(self):
-        logging.info('training data loader called')
+        print('tng data loader called')
        return self.__dataloader(train=True)
    @pl.data_loader
    def val_dataloader(self):
-        logging.info('val data loader called')
+        print('val data loader called')
        return self.__dataloader(train=False)
    @pl.data_loader
    def test_dataloader(self):
-        logging.info('test data loader called')
+        print('test data loader called')
        return self.__dataloader(train=False)
    @staticmethod
@@ -236,23 +230,31 @@ class LightningTemplateModel(LightningModule):
        :param root_dir:
        :return:
        """
-        parser = ArgumentParser(parents=[parent_parser])
+        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
        # param overwrites
-        # parser.set_defaults(gradient_clip_val=5.0)
+        # parser.set_defaults(gradient_clip=5.0)
        # network params
        parser.add_argument('--in_features', default=28 * 28, type=int)
        parser.add_argument('--out_features', default=10, type=int)
        # use 500 for CPU, 50000 for GPU to see speed difference
        parser.add_argument('--hidden_dim', default=50000, type=int)
-        parser.add_argument('--drop_prob', default=0.2, type=float)
+        parser.opt_list('--drop_prob', default=0.2, options=[0.2, 0.5], type=float, tunable=False)
        parser.add_argument('--learning_rate', default=0.001, type=float)
        # data
        parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
        # training params (opt)
-        parser.add_argument('--optimizer_name', default='adam', type=str)
+        parser.opt_list('--learning_rate', default=0.001 * 8, type=float,
-        parser.add_argument('--batch_size', default=64, type=int)
+                        options=[0.0001, 0.0005, 0.001, 0.005],
                        tunable=False)
        parser.opt_list('--optimizer_name', default='adam', type=str,
                        options=['adam'], tunable=False)
        # if using 2 nodes with 4 gpus each the batch size here
        #  (256) will be 256 / (2*8) = 16 per gpu
        parser.opt_list('--batch_size', default=256 * 8, type=int,
                        options=[32, 64, 128, 256], tunable=False,
                        help='batch size will be divided over all gpus being used across all nodes')
        return parser
@@ -0,0 +1,172 @@
 """
 Multi-node example (GPU)
 """
 import os
 import numpy as np
 from time import sleep
 import torch
 from test_tube import HyperOptArgumentParser, Experiment, SlurmCluster
 from pytorch_lightning.models.trainer import Trainer
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 from examples.new_project_templates.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main_local(hparams):
    main(hparams, None, None)
 def main(hparams, cluster, results_dict):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    print('loading model...')
    model = LightningTemplateModel(hparams)
    print('model built')
    # ------------------------
    # 2 INIT TEST TUBE EXP
    # ------------------------
    # when using grid search, it's possible for all models to start at once
    # and use the same test tube experiment version
    relative_node_id = int(os.environ['SLURM_NODEID'])
    sleep(relative_node_id + 1)
    # init experiment
    exp = Experiment(
        name=hyperparams.experiment_name,
        save_dir=hyperparams.test_tube_save_path,
        autosave=False,
        description='test demo'
    )
    exp.argparse(hparams)
    exp.save()
    # ------------------------
    # 3 DEFINE CALLBACKS
    # ------------------------
    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
    early_stop = EarlyStopping(
        monitor='val_acc',
        patience=3,
        verbose=True,
        mode='max'
    )
    checkpoint = ModelCheckpoint(
        filepath=model_save_path,
        save_best_only=True,
        verbose=True,
        monitor='val_loss',
        mode='min'
    )
    # ------------------------
    # 4 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        experiment=exp,
        cluster=cluster,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
        gpus=hparams.gpus,
        nb_gpu_nodes=hyperparams.nb_gpu_nodes
    )
    # ------------------------
    # 5 START TRAINING
    # ------------------------
    trainer.fit(model)
 def optimize_on_cluster(hyperparams):
    # enable cluster training
    # log all scripts to the test tube folder
    cluster = SlurmCluster(
        hyperparam_optimizer=hyperparams,
        log_path=hyperparams.slurm_log_path,
    )
    # email for cluster coms
    cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
    # configure cluster
    cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
    cluster.per_experiment_nb_nodes = hyperparams.nb_gpu_nodes
    cluster.job_time = '2:00:00'
    cluster.gpu_type = 'volta'
    cluster.memory_mb_per_node = 0
    # any modules for code to run in env
    cluster.add_command('source activate lightning')
    # run only on 32GB voltas
    cluster.add_slurm_cmd(cmd='constraint', value='volta32gb',
                          comment='use 32gb gpus')
    cluster.add_slurm_cmd(cmd='partition', value=hyperparams.gpu_partition,
                          comment='use 32gb gpus')
    # run hopt
    # creates and submits jobs to slurm
    cluster.optimize_parallel_cluster_gpu(
        main,
        nb_trials=hyperparams.nb_hopt_trials,
        job_name=hyperparams.experiment_name
    )
 if __name__ == '__main__':
    # use default args
    root_dir = os.path.dirname(os.path.realpath(__file__))
    demo_log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
    checkpoint_dir = os.path.join(demo_log_dir, 'model_weights')
    test_tube_dir = os.path.join(demo_log_dir, 'test_tube_data')
    slurm_out_dir = os.path.join(demo_log_dir, 'slurm_scripts')
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
    # cluster args not defined inside the model
    parent_parser.add_argument('--gpu_partition', type=str, help='consult your cluster manual')
    # TODO: make 1 param
    parent_parser.add_argument('--per_experiment_nb_gpus', type=int,
                               help='how many gpus to use in a node')
    parent_parser.add_argument('--gpus', type=str, default='-1',
                               help='how many gpus to use in the node')
    parent_parser.add_argument('--nb_gpu_nodes', type=int, default=1,
                               help='how many nodes to use in a cluster')
    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
                               help='where to save logs')
    parent_parser.add_argument('--slurm_log_path', type=str, default=slurm_out_dir,
                               help='where to save slurm meta')
    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
                               help='where to save model')
    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
                               help='test tube exp name')
    parent_parser.add_argument('--nb_hopt_trials', type=int, default=1,
                               help='how many grid search trials to run')
    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
    print('RUNNING ON SLURM CLUSTER')
    optimize_on_cluster(hyperparams)
@@ -0,0 +1,109 @@
 """
 Runs a model on a single node on CPU only..
 """
 import os
 import numpy as np
 import torch
 from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 from examples.new_project_templates.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    print('loading model...')
    model = LightningTemplateModel(hparams)
    print('model built')
    # ------------------------
    # 2 INIT TEST TUBE EXP
    # ------------------------
    # init experiment
    exp = Experiment(
        name=hyperparams.experiment_name,
        save_dir=hyperparams.test_tube_save_path,
        autosave=False,
        description='test demo'
    )
    exp.argparse(hparams)
    exp.save()
    # ------------------------
    # 3 DEFINE CALLBACKS
    # ------------------------
    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
    early_stop = EarlyStopping(
        monitor='val_acc',
        patience=3,
        verbose=True,
        mode='max'
    )
    checkpoint = ModelCheckpoint(
        filepath=model_save_path,
        save_best_only=True,
        verbose=True,
        monitor='val_loss',
        mode='min'
    )
    # ------------------------
    # 4 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        experiment=exp,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
    )
    # ------------------------
    # 5 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    # dirs
    root_dir = os.path.dirname(os.path.realpath(__file__))
    demo_log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
    checkpoint_dir = os.path.join(demo_log_dir, 'model_weights')
    test_tube_dir = os.path.join(demo_log_dir, 'test_tube_data')
    # although we user hyperOptParser, we are using it only as argparse right now
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
    # gpu args
    parent_parser.add_argument('--test_tube_save_path', type=str,
                               default=test_tube_dir, help='where to save logs')
    parent_parser.add_argument('--model_save_path', type=str,
                               default=checkpoint_dir, help='where to save model')
    parent_parser.add_argument('--experiment_name', type=str,
                               default='pt_lightning_exp_a', help='test tube exp name')
    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
    print('RUNNING ON CPU')
    main(hyperparams)
@@ -0,0 +1,114 @@
 """
 16-bit single node, CPU example
 """
 import os
 import numpy as np
 import torch
 from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 from examples.new_project_templates.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    print('loading model...')
    model = LightningTemplateModel(hparams)
    print('model built')
    # ------------------------
    # 2 INIT TEST TUBE EXP
    # ------------------------
    # init experiment
    exp = Experiment(
        name=hyperparams.experiment_name,
        save_dir=hyperparams.test_tube_save_path,
        autosave=False,
        description='test demo'
    )
    exp.argparse(hparams)
    exp.save()
    # ------------------------
    # 3 DEFINE CALLBACKS
    # ------------------------
    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
    early_stop = EarlyStopping(
        monitor='val_acc',
        patience=3,
        verbose=True,
        mode='max'
    )
    checkpoint = ModelCheckpoint(
        filepath=model_save_path,
        save_best_only=True,
        verbose=True,
        monitor='val_loss',
        mode='min'
    )
    # ------------------------
    # 4 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        experiment=exp,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
        gpus=hparams.gpus,
        use_amp=True
    )
    # ------------------------
    # 5 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    # dirs
    root_dir = os.path.dirname(os.path.realpath(__file__))
    demo_log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
    checkpoint_dir = os.path.join(demo_log_dir, 'model_weights')
    test_tube_dir = os.path.join(demo_log_dir, 'test_tube_data')
    # although we user hyperOptParser, we are using it only as argparse right now
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
    # gpu args
    parent_parser.add_argument('--gpus', type=str, default='-1',
                               help='how many gpus to use in the node.'
                                    'value -1 uses all the gpus on the node')
    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
                               help='where to save logs')
    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
                               help='where to save model')
    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
                               help='test tube exp name')
    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -0,0 +1,114 @@
 """
 Runs a model on a single node across N-gpus.
 """
 import os
 import numpy as np
 import torch
 from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 from examples.new_project_templates.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    print('loading model...')
    model = LightningTemplateModel(hparams)
    print('model built')
    # ------------------------
    # 2 INIT TEST TUBE EXP
    # ------------------------
    # init experiment
    exp = Experiment(
        name=hyperparams.experiment_name,
        save_dir=hyperparams.test_tube_save_path,
        autosave=False,
        description='test demo'
    )
    exp.argparse(hparams)
    exp.save()
    # ------------------------
    # 3 DEFINE CALLBACKS
    # ------------------------
    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
    early_stop = EarlyStopping(
        monitor='val_acc',
        patience=3,
        verbose=True,
        mode='max'
    )
    checkpoint = ModelCheckpoint(
        filepath=model_save_path,
        save_best_only=True,
        verbose=True,
        monitor='val_loss',
        mode='min'
    )
    # ------------------------
    # 4 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        experiment=exp,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
        gpus=hparams.gpus,
        distributed_backend='ddp'
    )
    # ------------------------
    # 5 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    # dirs
    root_dir = os.path.dirname(os.path.realpath(__file__))
    demo_log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
    checkpoint_dir = os.path.join(demo_log_dir, 'model_weights')
    test_tube_dir = os.path.join(demo_log_dir, 'test_tube_data')
    # although we user hyperOptParser, we are using it only as argparse right now
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
    # gpu args
    parent_parser.add_argument('--gpus', type=str, default='-1',
                               help='how many gpus to use in the node.'
                                    ' value -1 uses all the gpus on the node')
    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
                               help='where to save logs')
    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
                               help='where to save model')
    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
                               help='test tube exp name')
    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -0,0 +1,113 @@
 """
 Runs a model on a single node across N-gpus using dataParallel
 """
 import os
 import numpy as np
 import torch
 from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 from examples.new_project_templates.lightning_module_template import LightningTemplateModel
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    print('loading model...')
    model = LightningTemplateModel(hparams)
    print('model built')
    # ------------------------
    # 2 INIT TEST TUBE EXP
    # ------------------------
    # init experiment
    exp = Experiment(
        name=hyperparams.experiment_name,
        save_dir=hyperparams.test_tube_save_path,
        autosave=False,
        description='test demo'
    )
    exp.argparse(hparams)
    exp.save()
    # ------------------------
    # 3 DEFINE CALLBACKS
    # ------------------------
    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
    early_stop = EarlyStopping(
        monitor='val_acc',
        patience=3,
        verbose=True,
        mode='max'
    )
    checkpoint = ModelCheckpoint(
        filepath=model_save_path,
        save_best_only=True,
        verbose=True,
        monitor='val_loss',
        mode='min'
    )
    # ------------------------
    # 4 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        experiment=exp,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
        gpus=hparams.gpus,
    )
    # ------------------------
    # 5 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    # dirs
    root_dir = os.path.dirname(os.path.realpath(__file__))
    demo_log_dir = os.path.join(root_dir, 'pt_lightning_demo_logs')
    checkpoint_dir = os.path.join(demo_log_dir, 'model_weights')
    test_tube_dir = os.path.join(demo_log_dir, 'test_tube_data')
    # although we user hyperOptParser, we are using it only as argparse right now
    parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
    # gpu args
    parent_parser.add_argument('--gpus', type=str, default='-1',
                               help='how many gpus to use in the node.'
                                    ' value -1 uses all the gpus on the node')
    parent_parser.add_argument('--test_tube_save_path', type=str, default=test_tube_dir,
                               help='where to save logs')
    parent_parser.add_argument('--model_save_path', type=str, default=checkpoint_dir,
                               help='where to save model')
    parent_parser.add_argument('--experiment_name', type=str, default='pt_lightning_exp_a',
                               help='test tube exp name')
    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    # run on HPC cluster
    print(f'RUNNING INTERACTIVE MODE ON GPUS. gpu ids: {hyperparams.gpus}')
    main(hyperparams)
@@ -0,0 +1,74 @@
 import os
 import sys
 from test_tube import HyperOptArgumentParser, Experiment
 from pytorch_lightning.models.trainer import Trainer
 from pytorch_lightning.utilities.arg_parse import add_default_args
 from pytorch_lightning.callbacks.pt_callbacks import EarlyStopping, ModelCheckpoint
 from examples.new_project_templates.lightning_module_template import LightningTemplateModel
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # init experiment
    exp = Experiment(
        name=hparams.tt_name,
        debug=hparams.debug,
        save_dir=hparams.tt_save_path,
        version=hparams.hpc_exp_number,
        autosave=False,
        description=hparams.tt_description
    )
    exp.argparse(hparams)
    exp.save()
    # build model
    model = LightningTemplateModel(hparams)
    # callbacks
    early_stop = EarlyStopping(
        monitor='val_acc',
        patience=3,
        mode='min',
        verbose=True,
    )
    model_save_path = '{}/{}/{}'.format(hparams.model_save_path, exp.name, exp.version)
    checkpoint = ModelCheckpoint(
        filepath=model_save_path,
        save_best_only=True,
        verbose=True,
        monitor='val_acc',
        mode='min'
    )
    # configure trainer
    trainer = Trainer(
        experiment=exp,
        checkpoint_callback=checkpoint,
        early_stop_callback=early_stop,
    )
    # train model
    trainer.fit(model)
 if __name__ == '__main__':
    # use default args given by lightning
    root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
    parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
    add_default_args(parent_parser, root_dir)
    # allow model to overwrite or extend args
    parser = LightningTemplateModel.add_model_specific_args(parent_parser)
    hyperparams = parser.parse_args()
    # train model
    main(hyperparams)
@@ -1,25 +1,27 @@
 """
-To run this template just do:
+To run this template just do:  
-python gan.py
+python gan.py   
-After a few epochs, launch tensorboard to see the images being generated at every batch.
+After a few epochs, launch tensorboard to see the images being generated at every batch.   
 tensorboard --logdir default
 """
 import os
 from argparse import ArgumentParser
-from collections import OrderedDict
+import os
 import numpy as np
-import torch
+
 import torch.nn as nn
 import torch.nn.functional as F
 import torchvision
 import torchvision.transforms as transforms
 from torch.utils.data import DataLoader
 from torchvision.datasets import MNIST
 from torch.utils.data import DataLoader
 import torch.nn as nn
 import torch.nn.functional as F
 import torch
 import pytorch_lightning as pl
 from test_tube import Experiment
 class Generator(nn.Module):
@@ -82,7 +84,6 @@ class GAN(pl.LightningModule):
        # cache for generated images
        self.generated_imgs = None
        self.last_imgs = None
    def forward(self, z):
        return self.generator(z)
@@ -90,12 +91,11 @@ class GAN(pl.LightningModule):
    def adversarial_loss(self, y_hat, y):
        return F.binary_cross_entropy(y_hat, y)
-    def training_step(self, batch, batch_idx, optimizer_idx):
+    def training_step(self, batch, batch_nb, optimizer_i):
        imgs, _ = batch
        self.last_imgs = imgs
        # train generator
-        if optimizer_idx == 0:
+        if optimizer_i == 0:
            # sample noise
            z = torch.randn(imgs.shape[0], self.hparams.latent_dim)
@@ -107,54 +107,34 @@ class GAN(pl.LightningModule):
            self.generated_imgs = self.forward(z)
            # log sampled images
-            # sample_imgs = self.generated_imgs[:6]
+            sample_imgs = self.generated_imgs[:6]
-            # grid = torchvision.utils.make_grid(sample_imgs)
+            grid = torchvision.utils.make_grid(sample_imgs)
-            # self.logger.experiment.add_image('generated_images', grid, 0)
+            self.experiment.add_image('generated_images', grid, 0)
            # ground truth result (ie: all fake)
            # put on GPU because we created this tensor inside training_loop
            valid = torch.ones(imgs.size(0), 1)
            if self.on_gpu:
                valid = valid.cuda(imgs.device.index)
            # adversarial loss is binary cross-entropy
            g_loss = self.adversarial_loss(self.discriminator(self.generated_imgs), valid)
-            tqdm_dict = {'g_loss': g_loss}
+
-            output = OrderedDict({
+            return g_loss
                'loss': g_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            })
            return output
        # train discriminator
-        if optimizer_idx == 1:
+        if optimizer_i == 1:
            # Measure discriminator's ability to classify real from generated samples
            # how well can it label as real?
            valid = torch.ones(imgs.size(0), 1)
            if self.on_gpu:
                valid = valid.cuda(imgs.device.index)
            real_loss = self.adversarial_loss(self.discriminator(imgs), valid)
            # how well can it label as fake?
            fake = torch.zeros(imgs.size(0), 1)
-            if self.on_gpu:
+            fake_loss = self.adversarial_loss(self.discriminator(self.generated_imgs.detach()), fake)
                fake = fake.cuda(imgs.device.index)
            fake_loss = self.adversarial_loss(
                self.discriminator(self.generated_imgs.detach()), fake)
            # discriminator loss is the average of these
            d_loss = (real_loss + fake_loss) / 2
-            tqdm_dict = {'d_loss': d_loss}
+
-            output = OrderedDict({
+            return d_loss
                'loss': d_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            })
            return output
    def configure_optimizers(self):
        lr = self.hparams.lr
@@ -166,38 +146,22 @@ class GAN(pl.LightningModule):
        return [opt_g, opt_d], []
    @pl.data_loader
-    def train_dataloader(self):
+    def tng_dataloader(self):
        transform = transforms.Compose([transforms.ToTensor(),
                                        transforms.Normalize([0.5], [0.5])])
        dataset = MNIST(os.getcwd(), train=True, download=True, transform=transform)
        return DataLoader(dataset, batch_size=self.hparams.batch_size)
    def on_epoch_end(self):
        z = torch.randn(8, self.hparams.latent_dim)
        # match gpu device (or keep as cpu)
        if self.on_gpu:
            z = z.cuda(self.last_imgs.device.index)
        # log sampled images
        sample_imgs = self.forward(z)
        grid = torchvision.utils.make_grid(sample_imgs)
        self.logger.experiment.add_image(f'generated_images', grid, self.current_epoch)
 def main(hparams):
-    # ------------------------
+    # save tensorboard logs
-    # 1 INIT LIGHTNING MODEL
+    exp = Experiment(save_dir=os.getcwd())
-    # ------------------------
+
    # init model
    model = GAN(hparams)
-    # ------------------------
+    # fit trainer on CPU
-    # 2 INIT TRAINER
+    trainer = pl.Trainer(experiment=exp, max_nb_epochs=200)
    # ------------------------
    trainer = pl.Trainer()
    # ------------------------
    # 3 START TRAINING
    # ------------------------
    trainer.fit(model)
@@ -205,12 +169,9 @@ if __name__ == '__main__':
    parser = ArgumentParser()
    parser.add_argument("--batch_size", type=int, default=64, help="size of the batches")
    parser.add_argument("--lr", type=float, default=0.0002, help="adam: learning rate")
-    parser.add_argument("--b1", type=float, default=0.5,
+    parser.add_argument("--b1", type=float, default=0.5, help="adam: decay of first order momentum of gradient")
-                        help="adam: decay of first order momentum of gradient")
+    parser.add_argument("--b2", type=float, default=0.999, help="adam: decay of first order momentum of gradient")
-    parser.add_argument("--b2", type=float, default=0.999,
+    parser.add_argument("--latent_dim", type=int, default=100, help="dimensionality of the latent space")
                        help="adam: decay of first order momentum of gradient")
    parser.add_argument("--latent_dim", type=int, default=100,
                        help="dimensionality of the latent space")
    hparams = parser.parse_args()
@@ -0,0 +1,16 @@
 site_name: PyTorch lightning Documentation
 theme:
  name: 'material'
 docs_dir: docs
 repo_name: 'williamFalcon/pytorch-lightning'
 repo_url: https://github.com/williamFalcon/pytorch-lightning
 site_dir: 'site'
 site_description: 'Documentation for PyTorch LightningModule, the researcher version of keras.'
 dev_addr: '0.0.0.0:8000'
 #google_analytics: ['UA-aasd', 'sitename']
 markdown_extensions:
  - codehilite:
      guess_lang: false
      linenums: true
@@ -1,11 +0,0 @@
 # Examples   
 This folder has 3 sections:   
 ### Domain templates   
 These are templates to show common approaches such as GANs and RL.   
 ### Basic examples   
 These show the most common use of Lightning for either CPU or GPU training.   
 ### Multi-node examples   
 These show how to run jobs on a GPU cluster using lightning.
@@ -1,146 +0,0 @@
 """
 Template model definition
 -------------------------
 In 99% of cases you want to just copy `one of the examples
 <https://github.com/PyTorchLightning/pytorch-lightning/tree/master/pl_examples>`_
 to start a new lightningModule and change the core of what your model is actually trying to do.
 .. code-block:: bash
    # get a copy of the module template
    wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/pl_examples/new_project_templates/lightning_module_template.py  # noqa: E501
 Trainer Example
 ---------------
 **`__main__` function**
 Normally, we want to let the `__main__` function start the training.
 Inside the main we parse training arguments with whatever hyperparameters we want.
 Your LightningModule will have a chance to add hyperparameters.
 .. code-block:: python
    from test_tube import HyperOptArgumentParser
    if __name__ == '__main__':
        # use default args given by lightning
        root_dir = os.path.split(os.path.dirname(sys.modules['__main__'].__file__))[0]
        parent_parser = HyperOptArgumentParser(strategy='random_search', add_help=False)
        add_default_args(parent_parser, root_dir)
        # allow model to overwrite or extend args
        parser = ExampleModel.add_model_specific_args(parent_parser)
        hyperparams = parser.parse_args()
        # train model
        main(hyperparams)
 **Main Function**
 The main function is your entry into the program. This is where you init your model, checkpoint directory,
 and launch the training. The main function should have 3 arguments:
 - hparams: a configuration of hyperparameters.
 - slurm_manager: Slurm cluster manager object (can be None)
 - dict: for you to return any values you want (useful in meta-learning, otherwise set to)
 .. code-block:: python
    def main(hparams, cluster, results_dict):
        # build model
        model = MyLightningModule(hparams)
        # configure trainer
        trainer = Trainer()
        # train model
        trainer.fit(model)
 The `__main__` function will start training on your **main** function.
 If you use the HyperParameterOptimizer in hyper parameter optimization mode,
 this main function will get one set of hyperparameters. If you use it as a simple
 argument parser you get the default arguments in the argument parser.
 So, calling main(hyperparams) runs the model with the default argparse arguments.::
    main(hyperparams)
 CPU hyperparameter search
 -------------------------
 .. code-block:: python
    # run a grid search over 20 hyperparameter combinations.
    hyperparams.optimize_parallel_cpu(
        main_local,
        nb_trials=20,
        nb_workers=1
    )
 Hyperparameter search on a single or multiple GPUs
 --------------------------------------------------
 .. code-block:: python
    # run a grid search over 20 hyperparameter combinations.
    hyperparams.optimize_parallel_gpu(
        main_local,
        nb_trials=20,
        nb_workers=1,
        gpus=[0,1,2,3]
    )
 Hyperparameter search on a SLURM HPC cluster
 --------------------------------------------
 .. code-block:: python
    def optimize_on_cluster(hyperparams):
        # enable cluster training
        cluster = SlurmCluster(
            hyperparam_optimizer=hyperparams,
            log_path=hyperparams.tt_save_path,
            test_tube_exp_name=hyperparams.tt_name
        )
        # email for cluster coms
        cluster.notify_job_status(email='add_email_here', on_done=True, on_fail=True)
        # configure cluster
        cluster.per_experiment_nb_gpus = hyperparams.per_experiment_nb_gpus
        cluster.job_time = '48:00:00'
        cluster.gpu_type = '1080ti'
        cluster.memory_mb_per_node = 48000
        # any modules for code to run in env
        cluster.add_command('source activate pytorch_lightning')
        # name of exp
        job_display_name = hyperparams.tt_name.split('_')[0]
        job_display_name = job_display_name[0:3]
        # run hopt
        logging.info('submitting jobs...')
        cluster.optimize_parallel_cluster_gpu(
            main,
            nb_trials=hyperparams.nb_hopt_trials,
            job_name=job_display_name
        )
    # run cluster hyperparameter search
    optimize_on_cluster(hyperparams)
 """
 from .basic_examples.lightning_module_template import LightningTemplateModel
 __all__ = [
    'LightningTemplateModel'
 ]
@@ -1,39 +0,0 @@
 # Basic Examples   
 Use these examples to test how lightning works.   
 #### Test on CPU  
 ```bash
 python cpu_template.py
 ```
 ---   
 #### Train on a single GPU
 ```bash
 python gpu_template.py --gpus 1
 ```   
 ---    
 #### DataParallel (dp)   
 Train on multiple GPUs using DataParallel.
 ```bash
 python gpu_template.py --gpus 2 --distributed_backend dp
 ```   
 ---
 #### DistributedDataParallel (ddp)    
 Train on multiple GPUs using DistributedDataParallel   
 ```bash
 python gpu_template.py --gpus 2 --distributed_backend ddp
 ```
 ---
 #### DistributedDataParallel+DP (ddp2)    
 Train on multiple GPUs using DistributedDataParallel + dataparallel.
 On a single node, uses all GPUs for 1 model. Then shares gradient information
 across nodes.   
 ```bash
 python gpu_template.py --gpus 2 --distributed_backend ddp2
 ```
@@ -1,54 +0,0 @@
 """
 Runs a model on a single node across N-gpus.
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
 from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
 from pytorch_lightning import Trainer
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    model = LightningTemplateModel(hparams)
    # ------------------------
    # 2 INIT TRAINER
    # ------------------------
    trainer = Trainer()
    # ------------------------
    # 3 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    # ------------------------
    # TRAINING ARGUMENTS
    # ------------------------
    # these are project-wide arguments
    root_dir = os.path.dirname(os.path.realpath(__file__))
    parent_parser = ArgumentParser(add_help=False)
    # each LightningModule defines arguments relevant to it
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    main(hyperparams)
@@ -1,79 +0,0 @@
 """
 Runs a model on a single node across N-gpus.
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
 from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
 from pytorch_lightning import Trainer
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    model = LightningTemplateModel(hparams)
    # ------------------------
    # 2 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        gpus=hparams.gpus,
        distributed_backend=hparams.distributed_backend,
        use_amp=hparams.use_16bit
    )
    # ------------------------
    # 3 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    # ------------------------
    # TRAINING ARGUMENTS
    # ------------------------
    # these are project-wide arguments
    root_dir = os.path.dirname(os.path.realpath(__file__))
    parent_parser = ArgumentParser(add_help=False)
    # gpu args
    parent_parser.add_argument(
        '--gpus',
        type=int,
        default=2,
        help='how many gpus'
    )
    parent_parser.add_argument(
        '--distributed_backend',
        type=str,
        default='dp',
        help='supports three options dp, ddp, ddp2'
    )
    parent_parser.add_argument(
        '--use_16bit',
        dest='use_16bit',
        action='store_true',
        help='if true uses 16 bit precision'
    )
    # each LightningModule defines arguments relevant to it
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    main(hyperparams)
@@ -1,246 +0,0 @@
 """
 This example is largely adapted from https://github.com/pytorch/examples/blob/master/imagenet/main.py
 """
 import argparse
 import os
 import random
 from collections import OrderedDict
 import torch
 import torch.backends.cudnn as cudnn
 import torch.nn.functional as F
 import torch.nn.parallel
 import torch.optim as optim
 import torch.optim.lr_scheduler as lr_scheduler
 import torch.utils.data
 import torch.utils.data.distributed
 import torchvision.datasets as datasets
 import torchvision.models as models
 import torchvision.transforms as transforms
 import pytorch_lightning as pl
 # pull out resnet names from torchvision models
 MODEL_NAMES = sorted(
    name for name in models.__dict__
    if name.islower() and not name.startswith("__") and callable(models.__dict__[name])
 )
 class ImageNetLightningModel(pl.LightningModule):
    def __init__(self, hparams):
        super(ImageNetLightningModel, self).__init__()
        self.hparams = hparams
        self.model = models.__dict__[self.hparams.arch](pretrained=self.hparams.pretrained)
    def training_step(self, batch, batch_idx):
        images, target = batch
        output = self.model(images)
        loss_val = F.cross_entropy(output, target)
        acc1, acc5 = self.__accuracy(output, target, topk=(1, 5))
        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
        if self.trainer.use_dp or self.trainer.use_ddp2:
            loss_val = loss_val.unsqueeze(0)
            acc1 = acc1.unsqueeze(0)
            acc5 = acc5.unsqueeze(0)
        tqdm_dict = {'train_loss': loss_val}
        output = OrderedDict({
            'loss': loss_val,
            'acc1': acc1,
            'acc5': acc5,
            'progress_bar': tqdm_dict,
            'log': tqdm_dict
        })
        return output
    def validation_step(self, batch, batch_idx):
        images, target = batch
        output = self.model(images)
        loss_val = F.cross_entropy(output, target)
        acc1, acc5 = self.__accuracy(output, target, topk=(1, 5))
        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
        if self.trainer.use_dp or self.trainer.use_ddp2:
            loss_val = loss_val.unsqueeze(0)
            acc1 = acc1.unsqueeze(0)
            acc5 = acc5.unsqueeze(0)
        output = OrderedDict({
            'val_loss': loss_val,
            'val_acc1': acc1,
            'val_acc5': acc5,
        })
        return output
    def validation_end(self, outputs):
        tqdm_dict = {}
        for metric_name in ["val_loss", "val_acc1", "val_acc5"]:
            metric_total = 0
            for output in outputs:
                metric_value = output[metric_name]
                # reduce manually when using dp
                if self.trainer.use_dp or self.trainer.use_ddp2:
                    metric_value = torch.mean(metric_value)
                metric_total += metric_value
            tqdm_dict[metric_name] = metric_total / len(outputs)
        result = {'progress_bar': tqdm_dict, 'log': tqdm_dict, 'val_loss': tqdm_dict["val_loss"]}
        return result
    @classmethod
    def __accuracy(cls, output, target, topk=(1,)):
        """Computes the accuracy over the k top predictions for the specified values of k"""
        with torch.no_grad():
            maxk = max(topk)
            batch_size = target.size(0)
            _, pred = output.topk(maxk, 1, True, True)
            pred = pred.t()
            correct = pred.eq(target.view(1, -1).expand_as(pred))
            res = []
            for k in topk:
                correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
                res.append(correct_k.mul_(100.0 / batch_size))
            return res
    def configure_optimizers(self):
        optimizer = optim.SGD(
            self.parameters(),
            lr=self.hparams.lr,
            momentum=self.hparams.momentum,
            weight_decay=self.hparams.weight_decay
        )
        scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.1)
        return [optimizer], [scheduler]
    @pl.data_loader
    def train_dataloader(self):
        normalize = transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225],
        )
        train_dir = os.path.join(self.hparams.data, 'train')
        train_dataset = datasets.ImageFolder(
            train_dir,
            transforms.Compose([
                transforms.RandomResizedCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize,
            ]))
        if self.use_ddp:
            train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
        else:
            train_sampler = None
        train_loader = torch.utils.data.DataLoader(
            dataset=train_dataset,
            batch_size=self.hparams.batch_size,
            shuffle=(train_sampler is None),
            num_workers=0,
            sampler=train_sampler
        )
        return train_loader
    @pl.data_loader
    def val_dataloader(self):
        normalize = transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225],
        )
        val_dir = os.path.join(self.hparams.data, 'val')
        val_loader = torch.utils.data.DataLoader(
            datasets.ImageFolder(val_dir, transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize,
            ])),
            batch_size=self.hparams.batch_size,
            shuffle=False,
            num_workers=0,
        )
        return val_loader
    @staticmethod
    def add_model_specific_args(parent_parser):  # pragma: no cover
        parser = argparse.ArgumentParser(parents=[parent_parser])
        parser.add_argument('-a', '--arch', metavar='ARCH', default='resnet18', choices=MODEL_NAMES,
                            help='model architecture: ' +
                                 ' | '.join(MODEL_NAMES) +
                                 ' (default: resnet18)')
        parser.add_argument('--epochs', default=90, type=int, metavar='N',
                            help='number of total epochs to run')
        parser.add_argument('--seed', type=int, default=None,
                            help='seed for initializing training. ')
        parser.add_argument('-b', '--batch-size', default=256, type=int,
                            metavar='N',
                            help='mini-batch size (default: 256), this is the total '
                                 'batch size of all GPUs on the current node when '
                                 'using Data Parallel or Distributed Data Parallel')
        parser.add_argument('--lr', '--learning-rate', default=0.1, type=float,
                            metavar='LR', help='initial learning rate', dest='lr')
        parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
                            help='momentum')
        parser.add_argument('--wd', '--weight-decay', default=1e-4, type=float,
                            metavar='W', help='weight decay (default: 1e-4)',
                            dest='weight_decay')
        parser.add_argument('--pretrained', dest='pretrained', action='store_true',
                            help='use pre-trained model')
        return parser
 def get_args():
    parent_parser = argparse.ArgumentParser(add_help=False)
    parent_parser.add_argument('--data-path', metavar='DIR', type=str,
                               help='path to dataset')
    parent_parser.add_argument('--save-path', metavar='DIR', default=".", type=str,
                               help='path to save output')
    parent_parser.add_argument('--gpus', type=int, default=1,
                               help='how many gpus')
    parent_parser.add_argument('--distributed-backend', type=str, default='dp', choices=('dp', 'ddp', 'ddp2'),
                               help='supports three options dp, ddp, ddp2')
    parent_parser.add_argument('--use-16bit', dest='use-16bit', action='store_true',
                               help='if true uses 16 bit precision')
    parent_parser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true',
                               help='evaluate model on validation set')
    parser = ImageNetLightningModel.add_model_specific_args(parent_parser)
    return parser.parse_args()
 def main(hparams):
    model = ImageNetLightningModel(hparams)
    if hparams.seed is not None:
        random.seed(hparams.seed)
        torch.manual_seed(hparams.seed)
        cudnn.deterministic = True
    trainer = pl.Trainer(
        default_save_path=hparams.save_path,
        gpus=hparams.gpus,
        max_epochs=hparams.epochs,
        distributed_backend=hparams.distributed_backend,
        use_amp=hparams.use_16bit
    )
    if hparams.evaluate:
        trainer.run_evaluation()
    else:
        trainer.fit(model)
 if __name__ == '__main__':
    main(get_args())
@@ -1,21 +0,0 @@
 # Multi-node example   
 This demo launches a job using 2 GPUs on 2 different nodes (4 GPUs total).
 To run this demo do the following:
 1. Log into the jumphost node of your SLURM-managed cluster.  
 2. Create a conda environment with Lightning and a GPU PyTorch version.   
 3. Choose a script to submit    
 #### DDP  
 Submit this job to run with distributedDataParallel (2 nodes, 2 gpus each)
 ```bash
 sbatch ddp_job_submit.sh YourEnv
 ```
 #### DDP2  
 Submit this job to run with a different implementation of distributedDataParallel.
 In this version, each node acts like DataParallel but syncs across nodes like DDP.
 ```bash
 sbatch ddp2_job_submit.sh YourEnv
 ```
@@ -1,27 +0,0 @@
 #!/bin/bash -l
 # SLURM SUBMIT SCRIPT
 #SBATCH --nodes=2
 #SBATCH --gres=gpu:2
 #SBATCH --ntasks-per-node=1
 #SBATCH --mem=0
 #SBATCH --time=0-02:00:00
 # activate conda env
 source activate $1
 # -------------------------
 # debugging flags (optional)
 export NCCL_DEBUG=INFO
 export PYTHONFAULTHANDLER=1
 # on your cluster you might need these:
 # set the network interface
 # export NCCL_SOCKET_IFNAME=^docker0,lo
 # might need the latest cuda
 # module load NCCL/2.4.7-1-cuda.10.0
 # -------------------------
 # run script from above
 srun python3 multi_node_ddp2_demo.py
@@ -1,27 +0,0 @@
 #!/bin/bash -l
 # SLURM SUBMIT SCRIPT
 #SBATCH --nodes=2
 #SBATCH --gres=gpu:2
 #SBATCH --ntasks-per-node=2
 #SBATCH --mem=0
 #SBATCH --time=0-02:00:00
 # activate conda env
 source activate $1
 # -------------------------
 # debugging flags (optional)
 export NCCL_DEBUG=INFO
 export PYTHONFAULTHANDLER=1
 # on your cluster you might need these:
 # set the network interface
 # export NCCL_SOCKET_IFNAME=^docker0,lo
 # might need the latest cuda
 # module load NCCL/2.4.7-1-cuda.10.0
 # -------------------------
 # run script from above
 srun python3 multi_node_ddp_demo.py
@@ -1,55 +0,0 @@
 """
 Multi-node example (GPU)
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
 from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
 from pytorch_lightning import Trainer
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    model = LightningTemplateModel(hparams)
    # ------------------------
    # 2 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        gpus=2,
        num_nodes=2,
        distributed_backend='ddp2'
    )
    # ------------------------
    # 3 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    root_dir = os.path.dirname(os.path.realpath(__file__))
    parent_parser = ArgumentParser(add_help=False)
    # each LightningModule defines arguments relevant to it
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    main(hyperparams)
@@ -1,55 +0,0 @@
 """
 Multi-node example (GPU)
 """
 import os
 from argparse import ArgumentParser
 import numpy as np
 import torch
 from pl_examples.basic_examples.lightning_module_template import LightningTemplateModel
 from pytorch_lightning import Trainer
 SEED = 2334
 torch.manual_seed(SEED)
 np.random.seed(SEED)
 def main(hparams):
    """
    Main training routine specific for this project
    :param hparams:
    :return:
    """
    # ------------------------
    # 1 INIT LIGHTNING MODEL
    # ------------------------
    model = LightningTemplateModel(hparams)
    # ------------------------
    # 2 INIT TRAINER
    # ------------------------
    trainer = Trainer(
        gpus=2,
        num_nodes=2,
        distributed_backend='ddp'
    )
    # ------------------------
    # 3 START TRAINING
    # ------------------------
    trainer.fit(model)
 if __name__ == '__main__':
    root_dir = os.path.dirname(os.path.realpath(__file__))
    parent_parser = ArgumentParser(add_help=False)
    # each LightningModule defines arguments relevant to it
    parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
    hyperparams = parser.parse_args()
    # ---------------------
    # RUN TRAINING
    # ---------------------
    main(hyperparams)
@@ -1,38 +1,9 @@
-"""Package info"""
+from .models.trainer import Trainer
 from .root_module.root_module import LightningModule
 from .root_module.decorators import data_loader
-__version__ = '0.6.0'
+__all__ = [
-__author__ = 'William Falcon et al.'
+    'Trainer',
-__author_email__ = 'waf2107@columbia.edu'
+    'LightningModule',
-__license__ = 'Apache-2.0'
+    'data_loader',
-__copyright__ = 'Copyright (c) 2018-2019, %s.' % __author__
+]
 __homepage__ = 'https://github.com/PyTorchLightning/pytorch-lightning'
 # this has to be simple string, see: https://github.com/pypa/twine/issues/522
 __docs__ = "PyTorch Lightning is the lightweight PyTorch wrapper for ML researchers." \
           " Scale your models. Write less boilerplate."
 try:
    # This variable is injected in the __builtins__ by the build
    # process. It used to enable importing subpackages of skimage when
    # the binaries are not built
    __LIGHTNING_SETUP__
 except NameError:
    __LIGHTNING_SETUP__ = False
 if __LIGHTNING_SETUP__:
    import sys
    sys.stderr.write('Partial import of torchlightning during the build process.\n')
    # We are not importing the rest of the scikit during the build
    # process, as it may not be compiled yet
 else:
    from .trainer.trainer import Trainer
    from .core.lightning import LightningModule
    from .core.decorators import data_loader
    import logging
    __all__ = [
        'Trainer',
        'LightningModule',
        'data_loader',
    ]
    logging.basicConfig(level=logging.INFO)
@@ -1,20 +1,33 @@
 """
 Callbacks
 ====================================
 Callbacks supported by Lightning
 """
 import os
 import shutil
 import logging
 import warnings
 import numpy as np
-from pytorch_lightning.overrides.data_parallel import LightningDistributedDataParallel
+from pytorch_lightning.pt_overrides.override_data_parallel import LightningDistributedDataParallel
 class Callback(object):
-    r"""Abstract base class used to build new callbacks.
+    """Abstract base class used to build new callbacks.
    # Properties
        params: dict. Training parameters
            (eg. verbosity, batch size, number of epochs...).
        model: instance of `keras.models.Model`.
            Reference of the model being trained.
    The `logs` dictionary that callback methods
    take as argument will contain keys for quantities relevant to
    the current batch or epoch.
    Currently, the `.fit()` method of the `Sequential` model class
    will include the following quantities in the `logs` that
    it passes to its callbacks:
        on_epoch_end: logs include `acc` and `loss`, and
            optionally include `val_loss`
            (if validation is enabled in `fit`), and `val_acc`
            (if validation and accuracy monitoring are enabled).
        on_batch_begin: logs include `size`,
            the number of samples in the current batch.
        on_batch_end: logs include `loss`, and optionally `acc`
            (if accuracy monitoring is enabled).
    """
    def __init__(self):
@@ -30,30 +43,12 @@ class Callback(object):
        self.model = model
    def on_epoch_begin(self, epoch, logs=None):
        """
        called when the epoch begins
        Args:
            epoch (int): current epoch
            logs (dict): key-value pairs of quantities to monitor
            Example:
                on_epoch_begin(epoch=2, logs={'val_loss': 0.2})
        """
        pass
    def on_epoch_end(self, epoch, logs=None):
        pass
    def on_batch_begin(self, batch, logs=None):
        """
        called when the batch starts.
        Args:
            batch (Tensor): current batch tensor
            logs (dict): key-value pairs of quantities to monitor
        """
        pass
    def on_batch_end(self, batch, logs=None):
@@ -67,33 +62,23 @@ class Callback(object):
 class EarlyStopping(Callback):
-    r"""
+    """Stop training when a monitored quantity has stopped improving.
-    Stop training when a monitored quantity has stopped improving.
+    # Arguments
-
+        monitor: quantity to be monitored.
-    Args:
+        min_delta: minimum change in the monitored quantity
        monitor (str): quantity to be monitored.
        min_delta (float): minimum change in the monitored quantity
            to qualify as an improvement, i.e. an absolute
            change of less than min_delta, will count as no
            improvement.
-        patience (int): number of epochs with no improvement
+        patience: number of epochs with no improvement
            after which training will be stopped.
-        verbose (bool): verbosity mode.
+        verbose: verbosity mode.
-        mode (str): one of {auto, min, max}. In `min` mode,
+        mode: one of {auto, min, max}. In `min` mode,
            training will stop when the quantity
            monitored has stopped decreasing; in `max`
            mode it will stop when the quantity
            monitored has stopped increasing; in `auto`
            mode, the direction is automatically inferred
            from the name of the monitored quantity.
    Example::
        from pytorch_lightning import Trainer
        from pytorch_lightning.callbacks import EarlyStopping
        early_stopping = EarlyStopping('val_loss')
        Trainer(early_stop_callback=early_stopping)
    """
    def __init__(self, monitor='val_loss',
@@ -108,7 +93,7 @@ class EarlyStopping(Callback):
        self.stopped_epoch = 0
        if mode not in ['auto', 'min', 'max']:
-            logging.info(f'EarlyStopping mode {mode} is unknown, fallback to auto mode.')
+            print('EarlyStopping mode %s is unknown, fallback to auto mode.' % mode)
            mode = 'auto'
        if mode == 'min':
@@ -138,12 +123,10 @@ class EarlyStopping(Callback):
        current = logs.get(self.monitor)
        stop_training = False
        if current is None:
-            warnings.warn(
+            print('Early stopping conditioned on metric `%s` '
-                f'Early stopping conditioned on metric `{self.monitor}`'
+                  'which is not available. Available metrics are: %s' %
-                f' which is not available. Available metrics are: {",".join(list(logs.keys()))}',
+                  (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning)
-                RuntimeWarning)
+            exit(-1)
            stop_training = True
            return stop_training
        if self.monitor_op(current - self.min_delta, self.best):
            self.best = current
@@ -159,217 +142,124 @@ class EarlyStopping(Callback):
    def on_train_end(self, logs=None):
        if self.stopped_epoch > 0 and self.verbose > 0:
-            logging.info(f'Epoch {self.stopped_epoch + 1:05d}: early stopping')
+            print('Epoch %05d: early stopping' % (self.stopped_epoch + 1))
 class ModelCheckpoint(Callback):
-    r"""
+    """Save the model after every epoch.
-
+    `filepath` can contain named formatting options,
-    Save the model after every epoch.
+    which will be filled the value of `epoch` and
-
+    keys in `logs` (passed in `on_epoch_end`).
-    Args:
+    For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`,
-        filepath (str): path to save the model file.
+    then the model checkpoints will be saved with the epoch number and
-            Can contain named formatting options to be auto-filled.
+    the validation loss in the filename.
-
+    # Arguments
-            Example::
+        filepath: string, path to save the model file.
-
+        monitor: quantity to monitor.
-                # save epoch and val_loss in name
+        verbose: verbosity mode, 0 or 1.
-                ModelCheckpoint(filepath='{epoch:02d}-{val_loss:.2f}.hdf5')
+        save_best_only: if `save_best_only=True`,
-                # saves file like: /path/epoch_2-val_loss_0.2.hdf5
+            the latest best model according to
-        monitor (str): quantity to monitor.
+            the quantity monitored will not be overwritten.
-        verbose (bool): verbosity mode, 0 or 1.
+        mode: one of {auto, min, max}.
-        save_top_k (int): if `save_top_k == k`,
+            If `save_best_only=True`, the decision
            the best k models according to
            the quantity monitored will be saved.
            if `save_top_k == 0`, no models are saved.
            if `save_top_k == -1`, all models are saved.
            Please note that the monitors are checked every `period` epochs.
            if `save_top_k >= 2` and the callback is called multiple
            times inside an epoch, the name of the saved file will be
            appended with a version count starting with `v0`.
        mode (str): one of {auto, min, max}.
            If `save_top_k != 0`, the decision
            to overwrite the current save file is made
            based on either the maximization or the
            minimization of the monitored quantity. For `val_acc`,
            this should be `max`, for `val_loss` this should
            be `min`, etc. In `auto` mode, the direction is
            automatically inferred from the name of the monitored quantity.
-        save_weights_only (bool): if True, then only the model's weights will be
+        save_weights_only: if True, then only the model's weights will be
            saved (`model.save_weights(filepath)`), else the full model
            is saved (`model.save(filepath)`).
-        period (int): Interval (number of epochs) between checkpoints.
+        period: Interval (number of epochs) between checkpoints.
    Example::
        from pytorch_lightning import Trainer
        from pytorch_lightning.callbacks import ModelCheckpoint
        checkpoint_callback = ModelCheckpoint(filepath='my_path')
        Trainer(checkpoint_callback=checkpoint_callback)
        # saves checkpoints to my_path whenever 'val_loss' has a new min
    """
    def __init__(self, filepath, monitor='val_loss', verbose=0,
-                 save_top_k=1, save_weights_only=False,
+                 save_best_only=False, save_weights_only=False,
                 mode='auto', period=1, prefix=''):
        super(ModelCheckpoint, self).__init__()
        if (
            save_top_k and
            os.path.isdir(filepath) and
            len(os.listdir(filepath)) > 0
        ):
            warnings.warn(
                f"Checkpoint directory {filepath} exists and is not empty with save_top_k != 0."
                "All files in this directory will be deleted when a checkpoint is saved!"
            )
        self.monitor = monitor
        self.verbose = verbose
        self.filepath = filepath
-        os.makedirs(filepath, exist_ok=True)
+        self.save_best_only = save_best_only
        self.save_top_k = save_top_k
        self.save_weights_only = save_weights_only
        self.period = period
-        self.epochs_since_last_check = 0
+        self.epochs_since_last_save = 0
        self.prefix = prefix
        self.best_k_models = {}
        # {filename: monitor}
        self.kth_best_model = ''
        self.best = 0
        if mode not in ['auto', 'min', 'max']:
-            warnings.warn(
+            print('ModelCheckpoint mode %s is unknown, '
-                f'ModelCheckpoint mode {mode} is unknown, '
+                  'fallback to auto mode.' % (mode), RuntimeWarning)
                'fallback to auto mode.', RuntimeWarning)
            mode = 'auto'
        if mode == 'min':
            self.monitor_op = np.less
-            self.kth_value = np.Inf
+            self.best = np.Inf
            self.mode = 'min'
        elif mode == 'max':
            self.monitor_op = np.greater
-            self.kth_value = -np.Inf
+            self.best = -np.Inf
            self.mode = 'max'
        else:
            if 'acc' in self.monitor or self.monitor.startswith('fmeasure'):
                self.monitor_op = np.greater
-                self.kth_value = -np.Inf
+                self.best = -np.Inf
                self.mode = 'max'
            else:
                self.monitor_op = np.less
-                self.kth_value = np.Inf
+                self.best = np.Inf
                self.mode = 'min'
-    def _del_model(self, filepath):
+    def save_model(self, filepath, overwrite):
-        dirpath = os.path.dirname(filepath)
+        dirpath = '/'.join(filepath.split('/')[:-1])
        # make paths
-        os.makedirs(dirpath, exist_ok=True)
+        os.makedirs(os.path.dirname(filepath), exist_ok=True)
-        try:
+        if overwrite:
-            shutil.rmtree(filepath)
+            for filename in os.listdir(dirpath):
-        except OSError:
+                if self.prefix in filename:
-            os.remove(filepath)
+                    path_to_delete = os.path.join(dirpath, filename)
-
+                    try:
-    def _save_model(self, filepath):
+                        shutil.rmtree(path_to_delete)
-        dirpath = os.path.dirname(filepath)
+                    except OSError:
-
+                        os.remove(path_to_delete)
        # make paths
        os.makedirs(dirpath, exist_ok=True)
        # delegate the saving to the model
        self.save_function(filepath)
    def check_monitor_top_k(self, current):
        less_than_k_models = len(self.best_k_models.keys()) < self.save_top_k
        if less_than_k_models:
            return True
        return self.monitor_op(current, self.best_k_models[self.kth_best_model])
    def on_epoch_end(self, epoch, logs=None):
        logs = logs or {}
-        self.epochs_since_last_check += 1
+        self.epochs_since_last_save += 1
-
+        if self.epochs_since_last_save >= self.period:
-        if self.save_top_k == 0:
+            self.epochs_since_last_save = 0
-            # no models are saved
+            filepath = '{}/{}_ckpt_epoch_{}.ckpt'.format(self.filepath, self.prefix, epoch + 1)
-            return
+            if self.save_best_only:
        if self.epochs_since_last_check >= self.period:
            self.epochs_since_last_check = 0
            filepath = f'{self.filepath}/{self.prefix}_ckpt_epoch_{epoch}.ckpt'
            version_cnt = 0
            while os.path.isfile(filepath):
                # this epoch called before
                filepath = f'{self.filepath}/{self.prefix}_ckpt_epoch_{epoch}_v{version_cnt}.ckpt'
                version_cnt += 1
            if self.save_top_k != -1:
                current = logs.get(self.monitor)
                if current is None:
-                    warnings.warn(
+                    print('Can save best model only with %s available,'
-                        f'Can save best model only with {self.monitor} available,'
+                          ' skipping.' % (self.monitor), RuntimeWarning)
                        ' skipping.', RuntimeWarning)
                else:
-                    if self.check_monitor_top_k(current):
+                    if self.monitor_op(current, self.best):
                        # remove kth
                        if len(self.best_k_models.keys()) == self.save_top_k:
                            delpath = self.kth_best_model
                            self.best_k_models.pop(self.kth_best_model)
                            self._del_model(delpath)
                        self.best_k_models[filepath] = current
                        if len(self.best_k_models.keys()) == self.save_top_k:
                            # monitor dict has reached k elements
                            if self.mode == 'min':
                                self.kth_best_model = max(self.best_k_models, key=self.best_k_models.get)
                            else:
                                self.kth_best_model = min(self.best_k_models, key=self.best_k_models.get)
                            self.kth_value = self.best_k_models[self.kth_best_model]
                        if self.mode == 'min':
                            self.best = min(self.best_k_models.values())
                        else:
                            self.best = max(self.best_k_models.values())
                        if self.verbose > 0:
-                            logging.info(
+                            print('\nEpoch %05d: %s improved from %0.5f to %0.5f,'
-                                f'\nEpoch {epoch:05d}: {self.monitor} reached'
+                                  ' saving model to %s'
-                                f' {current:0.5f} (best {self.best:0.5f}), saving model to'
+                                  % (epoch + 1, self.monitor, self.best,
-                                f' {filepath} as top {self.save_top_k}')
+                                     current, filepath))
-                        self._save_model(filepath)
+                        self.best = current
                        self.save_model(filepath, overwrite=True)
                    else:
                        if self.verbose > 0:
-                            logging.info(
+                            print('\nEpoch %05d: %s did not improve' %
-                                f'\nEpoch {epoch:05d}: {self.monitor}'
+                                  (epoch + 1, self.monitor))
                                f' was not in top {self.save_top_k}')
            else:
                if self.verbose > 0:
-                    logging.info(f'\nEpoch {epoch:05d}: saving model to {filepath}')
+                    print('\nEpoch %05d: saving model to %s' % (epoch + 1, filepath))
-                self._save_model(filepath)
+                self.save_model(filepath, overwrite=False)
 class GradientAccumulationScheduler(Callback):
-    r"""
+    """Change gradient accumulation factor according to scheduling.
-    Change gradient accumulation factor according to scheduling.
+    # Arguments
-
+        scheduling: dict, scheduling in format {epoch: accumulation_factor}
    Args:
        scheduling (dict): scheduling in format {epoch: accumulation_factor}
    Example::
        from pytorch_lightning import Trainer
        from pytorch_lightning.callbacks import GradientAccumulationScheduler
        # at epoch 5 start accumulating every 2 batches
        accumulator = GradientAccumulationScheduler(scheduling: {5: 2})
        Trainer(accumulate_grad_batches=accumulator)
    """
    def __init__(self, scheduling: dict):
        if scheduling == {}:  # empty dict error
            raise TypeError("Empty dict cannot be interpreted correct")
@@ -396,11 +286,11 @@ class GradientAccumulationScheduler(Callback):
                break
-# if __name__ == '__main__':
+if __name__ == '__main__':
-#     c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
+    c = EarlyStopping(min_delta=0.9, patience=2, verbose=True)
-#     losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
+    losses = [10, 9, 8, 8, 6, 4.3, 5, 4.4, 2.8, 2.5]
-#     for i, loss in enumerate(losses):
+    for i, loss in enumerate(losses):
-#         should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
+        should_stop = c.on_epoch_end(i, logs={'val_loss': loss})
-#         logging.info(loss)
+        print(loss)
-#         if should_stop:
+        if should_stop:
-#             break
+            break
@@ -1,100 +0,0 @@
 """
 A LightningModule is a strict superclass of torch.nn.Module but provides an interface to standardize
 the "ingredients" for a research or production system.
 - The model/system definition (__init__)
 - The model/system computations (forward)
 - What happens in the training loop (training_step, training_end)
 - What happens in the validation loop (validation_step, validation_end)
 - What happens in the test loop (test_step, test_end)
 - What optimizers to use (configure_optimizers)
 - What data to use (train_dataloader, val_dataloader, test_dataloader)
 Most methods are optional. Here's a minimal example.
 .. code-block:: python
    import os
    import torch
    from torch.nn import functional as F
    from torch.utils.data import DataLoader
    from torchvision.datasets import MNIST
    import torchvision.transforms as transforms
    import pytorch_lightning as pl
    class CoolModel(pl.LightningModule):
        def __init__(self):
            super(CoolModel, self).__init__()
            self.l1 = torch.nn.Linear(28 * 28, 10)
        def forward(self, x):
            return torch.relu(self.l1(x.view(x.size(0), -1)))
        def training_step(self, batch, batch_idx):
            x, y = batch
            y_hat = self.forward(x)
            return {'loss': F.cross_entropy(y_hat, y)}
        def validation_step(self, batch, batch_idx):
            # OPTIONAL
            x, y = batch
            y_hat = self.forward(x)
            return {'val_loss': F.cross_entropy(y_hat, y)}
        def validation_end(self, outputs):
            # OPTIONAL
            val_loss_mean = torch.stack([x['val_loss'] for x in outputs]).mean()
            return {'val_loss': val_loss_mean}
        def test_step(self, batch, batch_idx):
            # OPTIONAL
            x, y = batch
            y_hat = self.forward(x)
            return {'test_loss': F.cross_entropy(y_hat, y)}
        def test_end(self, outputs):
            # OPTIONAL
            test_loss_mean = torch.stack([x['test_loss'] for x in outputs]).mean()
            return {'test_loss': test_loss_mean}
        def configure_optimizers(self):
            # REQUIRED
            return torch.optim.Adam(self.parameters(), lr=0.02)
        @pl.data_loader
        def train_dataloader(self):
            return DataLoader(MNIST(os.getcwd(), train=True, download=True,
                              transform=transforms.ToTensor()), batch_size=32)
        @pl.data_loader
        def val_dataloader(self):
            # OPTIONAL
            # can also return a list of val dataloaders
            return DataLoader(MNIST(os.getcwd(), train=True, download=True,
                              transform=transforms.ToTensor()), batch_size=32)
        @pl.data_loader
        def test_dataloader(self):
            # OPTIONAL
            # can also return a list of test dataloaders
            return DataLoader(MNIST(os.getcwd(), train=False, download=True,
                              transform=transforms.ToTensor()), batch_size=32)
 Once you've defined the LightningModule, fit  it using a trainer.
 .. code-block:: python
   trainer = pl.Trainer()
   model = CoolModel()
   trainer.fit(model)
 Check out this
 `COLAB <https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=HOk9c4_35FKg>`_
 for a live demo.
 """
 from .lightning import LightningModule
 __all__ = ['LightningModule']
@@ -1,154 +0,0 @@
 """
 Hooks
 =====
 There are cases when you might want to do something different at different parts of the training/validation loop.
 To enable a hook, simply override the method in your LightningModule and the trainer will call it at the correct time.
 **Contributing** If there's a hook you'd like to add, simply:
 1. Fork PyTorchLightning.
 2. Add the hook :py:mod:`pytorch_lightning.base_module.hooks.py`.
 3. Add the correct place in the :py:mod:`pytorch_lightning.models.trainer` where it should be called.
 """
 import torch
 try:
    from apex import amp
    APEX_AVAILABLE = True
 except ImportError:
    APEX_AVAILABLE = False
 class ModelHooks(torch.nn.Module):
    def on_sanity_check_start(self):
        """
        Called before starting evaluate
        .. warning:: will be deprecated.
        :return:
        """
        pass
    def on_train_start(self):
        """Called at the beginning of training before sanity check
        :return:
        """
        # do something at the start of training
        pass
    def on_train_end(self):
        """
        Called at the end of training before logger experiment is closed
        :return:
        """
        # do something at the end of training
        pass
    def on_batch_start(self, batch):
        """Called in the training loop before anything happens for that batch.
        :param batch:
        :return:
        """
        # do something when the batch starts
        pass
    def on_batch_end(self):
        """Called in the training loop after the batch."""
        # do something when the batch ends
        pass
    def on_epoch_start(self):
        """Called in the training loop at the very beginning of the epoch."""
        # do something when the epoch starts
        pass
    def on_epoch_end(self):
        """Called in the training loop at the very end of the epoch."""
        # do something when the epoch ends
        pass
    def on_pre_performance_check(self):
        """Called at the very beginning of the validation loop."""
        # do something before validation starts
        pass
    def on_post_performance_check(self):
        """Called at the very end of the validation loop."""
        # do something before validation end
        pass
    def on_before_zero_grad(self, optimizer):
        """Called after optimizer.step() and before optimizer.zero_grad()
        Called in the training loop after taking an optimizer step and before zeroing grads.
        Good place to inspect weight information with weights updated.
        for optimizer in optimizers::
            optimizer.step()
            model.on_before_zero_grad(optimizer) # < ---- called here
            optimizer.zero_grad
        :param optimizer:
        :return:
        """
        # do something with the optimizer or inspect it.
        pass
    def on_after_backward(self):
        """Called after loss.backward() and before optimizers do anything.
        :return:
        Called in the training loop after model.backward()
        This is the ideal place to inspect or log gradient information
        .. code-block:: python
            def on_after_backward(self):
                # example to inspect gradient information in tensorboard
                if self.trainer.global_step % 25 == 0:  # don't make the tf file huge
                    params = self.state_dict()
                    for k, v in params.items():
                        grads = v
                        name = k
                        self.logger.experiment.add_histogram(tag=name, values=grads,
                                                             global_step=self.trainer.global_step)
        """
        pass
    def backward(self, use_amp, loss, optimizer):
        """Override backward with your own implementation if you need to
        :param use_amp: Whether amp was requested or not
        :param loss: Loss is already scaled by accumulated grads
        :param optimizer: Current optimizer being used
        :return:
        Called to perform backward step.
        Feel free to override as needed.
        The loss passed in has already been scaled for accumulated gradients if requested.
        .. code-block:: python
            def backward(self, use_amp, loss, optimizer):
                if use_amp:
                    with amp.scale_loss(loss, optimizer) as scaled_loss:
                        scaled_loss.backward()
                else:
                    loss.backward()
        """
        if use_amp:
            with amp.scale_loss(loss, optimizer) as scaled_loss:
                scaled_loss.backward()
        else:
            loss.backward()
@@ -1,10 +0,0 @@
 """
 .. warning:: `model_saving` module has been renamed to `saving` since v0.6.0 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`model_saving` module has been renamed to `saving` since v0.6.0"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.core.saving import ModelIO  # noqa: E402
@@ -1,8 +0,0 @@
 """
 .. warning:: `root_module` module has been renamed to `lightning` since v0.6.0 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`root_module` module has been renamed to `lightning` since v0.6.0"
              " and will be removed in v0.8.0", DeprecationWarning)
@@ -1,34 +0,0 @@
 class ModelIO(object):
    def on_load_checkpoint(self, checkpoint):
        """
        Do something with the checkpoint
        Gives model a chance to load something before state_dict is restored
        :param checkpoint:
        :return:
        """
        pass
    def on_save_checkpoint(self, checkpoint):
        """
        Give the model a chance to add something to the checkpoint.
        state_dict is already there
        """
        pass
    # -------------------------
    # OPTIONAL HOOKS
    # -------------------------
    def on_hpc_save(self, checkpoint):
        """
        Hook to do whatever you need right before Slurm manager saves the model
        :return:
        """
        pass
    def on_hpc_load(self, checkpoint):
        """
        Hook to do whatever you need right before Slurm manager loads the model
        :return:
        """
        pass
@@ -1,116 +0,0 @@
 """
 Lightning supports most popular logging frameworks (Tensorboard, comet, weights and biases, etc...).
 To use a logger, simply pass it into the trainer.
 .. code-block:: python
    from pytorch_lightning import logging
    # lightning uses tensorboard by default
    tb_logger = logging.TensorBoardLogger()
    trainer = Trainer(logger=tb_logger)
    # or choose from any of the others such as MLFlow, Comet, Neptune, Wandb
    comet_logger = logging.CometLogger()
    trainer = Trainer(logger=comet_logger)
 .. note:: All loggers log by default to `os.getcwd()`. To change the path without creating a logger set
    Trainer(default_save_path='/your/path/to/save/checkpoints')
 Custom logger
 -------------
 You can implement your own logger by writing a class that inherits from
 `LightningLoggerBase`. Use the `rank_zero_only` decorator to make sure that
 only the first process in DDP training logs data.
 .. code-block:: python
    from pytorch_lightning.logging import LightningLoggerBase, rank_zero_only
    class MyLogger(LightningLoggerBase):
        @rank_zero_only
        def log_hyperparams(self, params):
            # params is an argparse.Namespace
            # your code to record hyperparameters goes here
            pass
        @rank_zero_only
        def log_metrics(self, metrics, step):
            # metrics is a dictionary of metric names and values
            # your code to record metrics goes here
            pass
        def save(self):
            # Optional. Any code necessary to save logger data goes here
            pass
        @rank_zero_only
        def finalize(self, status):
            # Optional. Any code that needs to be run after training
            # finishes goes here
 If you write a logger than may be useful to others, please send
 a pull request to add it to Lighting!
 Using loggers
 -------------
 Call the logger anywhere from your LightningModule by doing:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.whatever_method_summary_writer_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.add_histogram(...)
 Supported Loggers
 -----------------
 """
 from os import environ
 from .base import LightningLoggerBase, rank_zero_only
 from .tensorboard import TensorBoardLogger
 all = []
 try:
    # needed to prevent ImportError and duplicated logs.
    environ["COMET_DISABLE_AUTO_LOGGING"] = "1"
    from .comet import CometLogger
    all.append('CometLogger')
 except ImportError:
    del environ["COMET_DISABLE_AUTO_LOGGING"]
 try:
    from .mlflow import MLFlowLogger
    all.append('MLFlowLogger')
 except ImportError:
    pass
 try:
    from .neptune import NeptuneLogger
    all.append('NeptuneLogger')
 except ImportError:
    pass
 all.append('TensorBoardLogger')
 try:
    from .test_tube import TestTubeLogger
    all.append('TestTubeLogger')
 except ImportError:
    pass
 try:
    from .wandb import WandbLogger
    all.append('WandbLogger')
 except ImportError:
    pass
 __all__ = all
@@ -1,77 +0,0 @@
 from abc import ABC
 from functools import wraps
 def rank_zero_only(fn):
    """Decorate a logger method to run it only on the process with rank 0.
    :param fn: Function to decorate
    """
    @wraps(fn)
    def wrapped_fn(self, *args, **kwargs):
        if self.rank == 0:
            fn(self, *args, **kwargs)
    return wrapped_fn
 class LightningLoggerBase(ABC):
    """Base class for experiment loggers."""
    def __init__(self):
        self._rank = 0
    @property
    def experiment(self):
        raise NotImplementedError()
    def log_metrics(self, metrics, step):
        """Record metrics.
        :param float metric: Dictionary with metric names as keys and measured quanties as values
        :param int|None step: Step number at which the metrics should be recorded
        """
        raise NotImplementedError()
    def log_hyperparams(self, params):
        """Record hyperparameters.
        :param params: argparse.Namespace containing the hyperparameters
        """
        raise NotImplementedError()
    def save(self):
        """Save log data."""
        pass
    def finalize(self, status):
        """Do any processing that is necessary to finalize an experiment.
        :param status: Status that the experiment finished with (e.g. success, failed, aborted)
        """
        pass
    def close(self):
        """Do any cleanup that is necessary to close an experiment."""
        pass
    @property
    def rank(self):
        """Process rank. In general, metrics should only be logged by the process with rank 0."""
        return self._rank
    @rank.setter
    def rank(self, value):
        """Set the process rank."""
        self._rank = value
    @property
    def name(self):
        """Return the experiment name."""
        raise NotImplementedError("Sub-classes must provide a name property")
    @property
    def version(self):
        """Return the experiment version."""
        raise NotImplementedError("Sub-classes must provide a version property")
@@ -1,170 +0,0 @@
 from logging import getLogger
 try:
    from comet_ml import Experiment as CometExperiment
    from comet_ml import OfflineExperiment as CometOfflineExperiment
    try:
        from comet_ml.api import API
    except ImportError:
        # For more information, see: https://www.comet.ml/docs/python-sdk/releases/#release-300
        from comet_ml.papi import API
 except ImportError:
    raise ImportError('Missing comet_ml package.')
 from torch import is_tensor
 from .base import LightningLoggerBase, rank_zero_only
 from ..utilities.debugging import MisconfigurationException
 logger = getLogger(__name__)
 class CometLogger(LightningLoggerBase):
    def __init__(self, api_key=None, save_dir=None, workspace=None,
                 rest_api_key=None, project_name=None, experiment_name=None, **kwargs):
        r"""
        Log using `comet <https://www.comet.ml>`_.
        Requires either an API Key (online mode) or a local directory path (offline mode)
        .. code-block:: python
            # ONLINE MODE
            from pytorch_lightning.logging import CometLogger
            # arguments made to CometLogger are passed on to the comet_ml.Experiment class
            comet_logger = CometLogger(
                api_key=os.environ["COMET_KEY"],
                workspace=os.environ["COMET_WORKSPACE"], # Optional
                project_name="default_project", # Optional
                rest_api_key=os.environ["COMET_REST_KEY"], # Optional
                experiment_name="default" # Optional
            )
            trainer = Trainer(logger=comet_logger)
        .. code-block:: python
            # OFFLINE MODE
            from pytorch_lightning.logging import CometLogger
            # arguments made to CometLogger are passed on to the comet_ml.Experiment class
            comet_logger = CometLogger(
                save_dir=".",
                workspace=os.environ["COMET_WORKSPACE"], # Optional
                project_name="default_project", # Optional
                rest_api_key=os.environ["COMET_REST_KEY"], # Optional
                experiment_name="default" # Optional
            )
            trainer = Trainer(logger=comet_logger)
        Args:
            api_key (str): Required in online mode. API key, found on Comet.ml
            save_dir (str): Required in offline mode. The path for the directory to save local comet logs
            workspace (str): Optional. Name of workspace for this user
            project_name (str): Optional. Send your experiment to a specific project.
            Otherwise will be sent to Uncategorized Experiments.
            If project name does not already exists Comet.ml will create a new project.
            rest_api_key (str): Optional. Rest API key found in Comet.ml settings.
                This is used to determine version number
            experiment_name (str): Optional. String representing the name for this particular experiment on Comet.ml
        """
        super().__init__()
        self._experiment = None
        # Determine online or offline mode based on which arguments were passed to CometLogger
        if save_dir is not None and api_key is not None:
            # If arguments are passed for both save_dir and api_key, preference is given to online mode
            self.mode = "online"
            self.api_key = api_key
        elif api_key is not None:
            self.mode = "online"
            self.api_key = api_key
        elif save_dir is not None:
            self.mode = "offline"
            self.save_dir = save_dir
        else:
            # If neither api_key nor save_dir are passed as arguments, raise an exception
            raise MisconfigurationException("CometLogger requires either api_key or save_dir during initialization.")
        logger.info(f"CometLogger will be initialized in {self.mode} mode")
        self.workspace = workspace
        self.project_name = project_name
        self._kwargs = kwargs
        if rest_api_key is not None:
            # Comet.ml rest API, used to determine version number
            self.rest_api_key = rest_api_key
            self.comet_api = API(self.rest_api_key)
        else:
            self.rest_api_key = None
            self.comet_api = None
        if experiment_name:
            try:
                self.name = experiment_name
            except TypeError as e:
                logger.exception("Failed to set experiment name for comet.ml logger")
    @property
    def experiment(self):
        r"""
        Actual comet object. To use comet features do the following.
        Example::
            self.logger.experiment.some_comet_function()
        """
        if self._experiment is not None:
            return self._experiment
        if self.mode == "online":
            self._experiment = CometExperiment(
                api_key=self.api_key,
                workspace=self.workspace,
                project_name=self.project_name,
                **self._kwargs
            )
        else:
            self._experiment = CometOfflineExperiment(
                offline_directory=self.save_dir,
                workspace=self.workspace,
                project_name=self.project_name,
                **self._kwargs
            )
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        self.experiment.log_parameters(vars(params))
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        # Comet.ml expects metrics to be a dictionary of detached tensors on CPU
        for key, val in metrics.items():
            if is_tensor(val):
                metrics[key] = val.cpu().detach()
        self.experiment.log_metrics(metrics, step=step)
    @rank_zero_only
    def finalize(self, status):
        self.experiment.end()
    @property
    def name(self):
        return self.experiment.project_name
    @name.setter
    def name(self, value):
        self.experiment.set_name(value)
    @property
    def version(self):
        return self.experiment.id
@@ -1,10 +0,0 @@
 """
 .. warning:: `comet_logger` module has been renamed to `comet` since v0.6.0 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`comet_logger` module has been renamed to `comet` since v0.6.0"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.logging.comet import CometLogger  # noqa: E402
@@ -1,118 +0,0 @@
 """
 Log using `mlflow <https://mlflow.org>'_
 .. code-block:: python
    from pytorch_lightning.logging import MLFlowLogger
    mlf_logger = MLFlowLogger(
        experiment_name="default",
        tracking_uri="file:/."
    )
    trainer = Trainer(logger=mlf_logger)
 Use the logger anywhere in you LightningModule as follows:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.whatever_ml_flow_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.whatever_ml_flow_supports(...)
 """
 from logging import getLogger
 from time import time
 try:
    import mlflow
 except ImportError:
    raise ImportError('Missing mlflow package.')
 from .base import LightningLoggerBase, rank_zero_only
 logger = getLogger(__name__)
 class MLFlowLogger(LightningLoggerBase):
    def __init__(self, experiment_name, tracking_uri=None, tags=None):
        r"""
        Logs using MLFlow
        Args:
            experiment_name (str): The name of the experiment
            tracking_uri (str): where this should track
            tags (dict): todo this param
        """
        super().__init__()
        self._mlflow_client = mlflow.tracking.MlflowClient(tracking_uri)
        self.experiment_name = experiment_name
        self._run_id = None
        self.tags = tags
    @property
    def experiment(self):
        r"""
        Actual mlflow object. To use mlflow features do the following.
        Example::
            self.logger.experiment.some_mlflow_function()
        """
        return self._mlflow_client
    @property
    def run_id(self):
        if self._run_id is not None:
            return self._run_id
        expt = self._mlflow_client.get_experiment_by_name(self.experiment_name)
        if expt:
            self._expt_id = expt.experiment_id
        else:
            logger.warning(f"Experiment with name {self.experiment_name} not found. Creating it.")
            self._expt_id = self._mlflow_client.create_experiment(name=self.experiment_name)
        run = self._mlflow_client.create_run(experiment_id=self._expt_id, tags=self.tags)
        self._run_id = run.info.run_id
        return self._run_id
    @rank_zero_only
    def log_hyperparams(self, params):
        for k, v in vars(params).items():
            self.experiment.log_param(self.run_id, k, v)
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        timestamp_ms = int(time() * 1000)
        for k, v in metrics.items():
            if isinstance(v, str):
                logger.warning(
                    f"Discarding metric with string value {k}={v}"
                )
                continue
            self.experiment.log_metric(self.run_id, k, v, timestamp_ms, step)
    def save(self):
        pass
    @rank_zero_only
    def finalize(self, status="FINISHED"):
        if status == 'success':
            status = 'FINISHED'
        self.experiment.set_terminated(self.run_id, status)
    @property
    def name(self):
        return self.experiment_name
    @property
    def version(self):
        return self._run_id
@@ -1,10 +0,0 @@
 """
 .. warning:: `mlflow_logger` module has been renamed to `mlflow` since v0.6.0 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`mlflow_logger` module has been renamed to `mlflow` since v0.6.0"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.logging.mlflow import MLFlowLogger  # noqa: E402
@@ -1,286 +0,0 @@
 """
 Log using `neptune <https://www.neptune.ml>`_
 Neptune logger can be used in the online mode or offline (silent) mode.
 To log experiment data in online mode, NeptuneLogger requries an API key:
 .. code-block:: python
    from pytorch_lightning.logging import NeptuneLogger
    # arguments made to NeptuneLogger are passed on to the neptune.experiments.Experiment class
    neptune_logger = NeptuneLogger(
        api_key=os.environ["NEPTUNE_API_TOKEN"],
        project_name="USER_NAME/PROJECT_NAME",
        experiment_name="default", # Optional,
        params={"max_epochs": 10}, # Optional,
        tags=["pytorch-lightning","mlp"] # Optional,
    )
    trainer = Trainer(max_epochs=10, logger=neptune_logger)
 Use the logger anywhere in you LightningModule as follows:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.log_metric("acc_train", acc_train) # log metrics
        self.logger.experiment.log_image("worse_predictions", prediction_image) # log images
        self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint
        self.logger.experiment.whatever_neptune_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.log_metric("acc_train", acc_train) # log metrics
        self.logger.experiment.log_image("worse_predictions", prediction_image) # log images
        self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint
        self.logger.experiment.whatever_neptune_supports(...)
 """
 from logging import getLogger
 try:
    import neptune
 except ImportError:
    raise ImportError('Missing neptune package. Run `pip install neptune-client`')
 from torch import is_tensor
 # from .base import LightningLoggerBase, rank_zero_only
 from pytorch_lightning.logging.base import LightningLoggerBase, rank_zero_only
 logger = getLogger(__name__)
 class NeptuneLogger(LightningLoggerBase):
    def __init__(self, api_key=None, project_name=None, offline_mode=False,
                 experiment_name=None, upload_source_files=None,
                 params=None, properties=None, tags=None, **kwargs):
        r"""
        Initialize a neptune.ml logger.
        .. note:: Requires either an API Key (online mode) or a local directory path (offline mode)
        .. code-block:: python
            # ONLINE MODE
            from pytorch_lightning.logging import NeptuneLogger
            # arguments made to NeptuneLogger are passed on to the neptune.experiments.Experiment class
            neptune_logger = NeptuneLogger(
                api_key=os.environ["NEPTUNE_API_TOKEN"],
                project_name="USER_NAME/PROJECT_NAME",
                experiment_name="default", # Optional,
                params={"max_epochs": 10}, # Optional,
                tags=["pytorch-lightning","mlp"] # Optional,
            )
            trainer = Trainer(max_epochs=10, logger=neptune_logger)
        .. code-block:: python
            # OFFLINE MODE
            from pytorch_lightning.logging import NeptuneLogger
            # arguments made to NeptuneLogger are passed on to the neptune.experiments.Experiment class
            neptune_logger = NeptuneLogger(
                project_name="USER_NAME/PROJECT_NAME",
                experiment_name="default", # Optional,
                params={"max_epochs": 10}, # Optional,
                tags=["pytorch-lightning","mlp"] # Optional,
            )
            trainer = Trainer(max_epochs=10, logger=neptune_logger)
        Args:
            api_key (str | None): Required in online mode. Neputne API token, found on https://neptune.ml.
                Read how to get your API key
                https://docs.neptune.ml/python-api/tutorials/get-started.html#copy-api-token.
            project_name (str): Required in online mode. Qualified name of a project in a form of
               "namespace/project_name" for example "tom/minst-classification".
               If None, the value of NEPTUNE_PROJECT environment variable will be taken.
               You need to create the project in https://neptune.ml first.
            offline_mode (bool): Optional default False. If offline_mode=True no logs will be send to neptune.
               Usually used for debug purposes.
            experiment_name (str|None): Optional. Editable name of the experiment.
               Name is displayed in the experiment’s Details (Metadata section) and in experiments view as a column.
            upload_source_files (list|None): Optional. List of source files to be uploaded.
               Must be list of str or single str. Uploaded sources are displayed in the experiment’s Source code tab.
               If None is passed, Python file from which experiment was created will be uploaded.
               Pass empty list ([]) to upload no files. Unix style pathname pattern expansion is supported.
               For example, you can pass '*.py' to upload all python source files from the current directory.
               For recursion lookup use '**/*.py' (for Python 3.5 and later). For more information see glob library.
            params (dict|None): Optional. Parameters of the experiment. After experiment creation params are read-only.
               Parameters are displayed in the experiment’s Parameters section and each key-value pair can be
               viewed in experiments view as a column.
            properties (dict|None): Optional default is {}. Properties of the experiment.
               They are editable after experiment is created. Properties are displayed in the experiment’s Details and
               each key-value pair can be viewed in experiments view as a column.
            tags (list|None): Optional default []. Must be list of str. Tags of the experiment.
               They are editable after experiment is created (see: append_tag() and remove_tag()).
               Tags are displayed in the experiment’s Details and can be viewed in experiments view as a column.
        """
        super().__init__()
        self.api_key = api_key
        self.project_name = project_name
        self.offline_mode = offline_mode
        self.experiment_name = experiment_name
        self.upload_source_files = upload_source_files
        self.params = params
        self.properties = properties
        self.tags = tags
        self._experiment = None
        self._kwargs = kwargs
        if offline_mode:
            self.mode = "offline"
            neptune.init(project_qualified_name='dry-run/project',
                         backend=neptune.OfflineBackend())
        else:
            self.mode = "online"
            neptune.init(api_token=self.api_key,
                         project_qualified_name=self.project_name)
        logger.info(f"NeptuneLogger was initialized in {self.mode} mode")
    @property
    def experiment(self):
        r"""
        Actual neptune object. To use neptune features do the following.
        Example::
            self.logger.experiment.some_neptune_function()
        """
        if self._experiment is not None:
            return self._experiment
        else:
            self._experiment = neptune.create_experiment(name=self.experiment_name,
                                                         params=self.params,
                                                         properties=self.properties,
                                                         tags=self.tags,
                                                         upload_source_files=self.upload_source_files,
                                                         **self._kwargs)
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        for key, val in vars(params).items():
            self.experiment.set_property(f"param__{key}", val)
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        """Log metrics (numeric values) in Neptune experiments
        :param float metric: Dictionary with metric names as keys and measured quanties as values
        :param int|None step: Step number at which the metrics should be recorded, must be strictly increasing
        """
        for key, val in metrics.items():
            if is_tensor(val):
                val = val.cpu().detach()
            if step is None:
                self.experiment.log_metric(key, val)
            else:
                self.experiment.log_metric(key, x=step, y=val)
    @rank_zero_only
    def finalize(self, status):
        self.experiment.stop()
    @property
    def name(self):
        if self.mode == "offline":
            return "offline-name"
        else:
            return self.experiment.name
    @property
    def version(self):
        if self.mode == "offline":
            return "offline-id-1234"
        else:
            return self.experiment.id
    @rank_zero_only
    def log_metric(self, metric_name, metric_value, step=None):
        """Log metrics (numeric values) in Neptune experiments
        :param str metric_name:  The name of log, i.e. mse, loss, accuracy.
        :param str metric_value: The value of the log (data-point).
        :param int|None step: Step number at which the metrics should be recorded, must be strictly increasing
        """
        if step is None:
            self.experiment.log_metric(metric_name, metric_value)
        else:
            self.experiment.log_metric(metric_name, x=step, y=metric_value)
    @rank_zero_only
    def log_text(self, log_name, text, step=None):
        """Log text data in Neptune experiment
        :param str log_name:  The name of log, i.e. mse, my_text_data, timing_info.
        :param str text: The value of the log (data-point).
        :param int|None step: Step number at which the metrics should be recorded, must be strictly increasing
        """
        if step is None:
            self.experiment.log_metric(log_name, text)
        else:
            self.experiment.log_metric(log_name, x=step, y=text)
    @rank_zero_only
    def log_image(self, log_name, image, step=None):
        """Log image data in Neptune experiment
        :param str log_name: The name of log, i.e. bboxes, visualisations, sample_images.
        :param str|PIL.Image|matplotlib.figure.Figure image: The value of the log (data-point).
           Can be one of the following types: PIL image, matplotlib.figure.Figure, path to image file (str)
        :param int|None step: Step number at which the metrics should be recorded, must be strictly increasing
        """
        if step is None:
            self.experiment.log_image(log_name, image)
        else:
            self.experiment.log_image(log_name, x=step, y=image)
    @rank_zero_only
    def log_artifact(self, artifact, destination=None):
        """Save an artifact (file) in Neptune experiment storage.
        :param str artifact: A path to the file in local filesystem.
        :param str|None destination: Optional default None.
           A destination path. If None is passed, an artifact file name will be used.
        """
        self.experiment.log_artifact(artifact, destination)
    @rank_zero_only
    def set_property(self, key, value):
        """Set key-value pair as Neptune experiment property.
        :param str key: Property key.
        :param obj value: New value of a property.
        """
        self.experiment.set_property(key, value)
    @rank_zero_only
    def append_tags(self, tags):
        """appends tags to neptune experiment
        :param str|tuple|list(str) tags: Tags to add to the current experiment.
           If str is passed, singe tag is added.
           If multiple - comma separated - str are passed, all of them are added as tags.
           If list of str is passed, all elements of the list are added as tags.
        """
        if not isinstance(tags, (list, set, tuple)):
            tags = [tags]  # make it as an iterable is if it is not yet
        self.experiment.append_tags(*tags)
@@ -1,142 +0,0 @@
 import os
 from warnings import warn
 from argparse import Namespace
 from pkg_resources import parse_version
 import torch
 import pandas as pd
 from torch.utils.tensorboard import SummaryWriter
 from .base import LightningLoggerBase, rank_zero_only
 class TensorBoardLogger(LightningLoggerBase):
    r"""
    Log to local file system in TensorBoard format
    Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to
    `os.path.join(save_dir, name, version)`
    Example
    --------
    .. code-block:: python
        logger = TensorBoardLogger("tb_logs", name="my_model")
        trainer = Trainer(logger=logger)
        trainer.train(model)
    Args:
        save_dir (str): Save directory
        name (str): Experiment name. Defaults to "default".
        version (int): Experiment version. If version is not specified the logger inspects the save
        directory for existing versions, then automatically assigns the next available version.
        \**kwargs  (dict): Other arguments are passed directly to the :class:`SummaryWriter` constructor.
    """
    NAME_CSV_TAGS = 'meta_tags.csv'
    def __init__(self, save_dir, name="default", version=None, **kwargs):
        super().__init__()
        self.save_dir = save_dir
        self._name = name
        self._version = version
        self._experiment = None
        self.tags = {}
        self.kwargs = kwargs
    @property
    def experiment(self):
        r"""
         Actual tensorboard object. To use tensorboard features do the following.
         Example::
             self.logger.experiment.some_tensorboard_function()
         """
        if self._experiment is not None:
            return self._experiment
        root_dir = os.path.join(self.save_dir, self.name)
        os.makedirs(root_dir, exist_ok=True)
        log_dir = os.path.join(root_dir, "version_" + str(self.version))
        self._experiment = SummaryWriter(log_dir=log_dir, **self.kwargs)
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        if params is None:
            return
        # in case converting from namespace
        if isinstance(params, Namespace):
            params = vars(params)
        params = dict(params)
        if parse_version(torch.__version__) < parse_version("1.3.0"):
            warn(
                f"Hyperparameter logging is not available for Torch version {torch.__version__}."
                " Skipping log_hyperparams. Upgrade to Torch 1.3.0 or above to enable"
                " hyperparameter logging."
            )
        else:
            # `add_hparams` requires both - hparams and metric
            self.experiment.add_hparams(hparam_dict=params, metric_dict={})
        # some alternative should be added
        self.tags.update(params)
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        for k, v in metrics.items():
            if isinstance(v, torch.Tensor):
                v = v.item()
            self.experiment.add_scalar(k, v, step)
    @rank_zero_only
    def save(self):
        try:
            self.experiment.flush()
        except AttributeError:
            # you are using PT version (<v1.2) which does not have implemented flush
            self.experiment._get_file_writer().flush()
        # create a preudo standard path ala test-tube
        dir_path = os.path.join(self.save_dir, self.name, 'version_%s' % self.version)
        if not os.path.isdir(dir_path):
            dir_path = self.save_dir
        # prepare the file path
        meta_tags_path = os.path.join(dir_path, self.NAME_CSV_TAGS)
        # save the metatags file
        df = pd.DataFrame({'key': list(self.tags.keys()),
                           'value': list(self.tags.values())})
        df.to_csv(meta_tags_path, index=False)
    @rank_zero_only
    def finalize(self, status):
        self.save()
    @property
    def name(self):
        return self._name
    @property
    def version(self):
        if self._version is None:
            self._version = self._get_next_version()
        return self._version
    def _get_next_version(self):
        root_dir = os.path.join(self.save_dir, self.name)
        existing_versions = []
        for d in os.listdir(root_dir):
            if os.path.isdir(os.path.join(root_dir, d)) and d.startswith("version_"):
                existing_versions.append(int(d.split("_")[1]))
        if len(existing_versions) == 0:
            return 0
        else:
            return max(existing_versions) + 1
@@ -1,178 +0,0 @@
 """
 Log using `test tube <https://williamfalcon.github.io/test-tube>'_. Test tube logger is
 a strict subclass of `PyTorch SummaryWriter <https://pytorch.org/docs/stable/tensorboard.html>`_, refer to their
 documentation for all supported operations. The TestTubeLogger adds a nicer folder structure
 to manage experiments and snapshots all hyperparameters you pass to a LightningModule.
 .. code-block:: python
    from pytorch_lightning.logging import TestTubeLogger
    tt_logger = TestTubeLogger(
        save_dir=".",
        name="default",
        debug=False,
        create_git_tag=False
    )
    trainer = Trainer(logger=tt_logger)
 Use the logger anywhere in you LightningModule as follows:
 .. code-block:: python
    def train_step(...):
        # example
        self.logger.experiment.whatever_method_summary_writer_supports(...)
    def any_lightning_module_function_or_hook(...):
        self.logger.experiment.add_histogram(...)
 """
 try:
    from test_tube import Experiment
 except ImportError:
    raise ImportError('Missing test-tube package.')
 from .base import LightningLoggerBase, rank_zero_only
 class TestTubeLogger(LightningLoggerBase):
    r"""
    Log to local file system in TensorBoard format but using a nicer folder structure.
    Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to
    `os.path.join(save_dir, name, version)`
    Example
    --------
    .. code-block:: python
        logger = TestTubeLogger("tt_logs", name="my_exp_name")
        trainer = Trainer(logger=logger)
        trainer.train(model)
    Args:
        save_dir (str): Save directory
        name (str): Experiment name. Defaults to "default".
        description (str): A short snippet about this experiment
        debug (bool): If True, it doesn't log anything
        version (int): Experiment version. If version is not specified the logger inspects the save
        directory for existing versions, then automatically assigns the next available version.
        create_git_tag (bool): If True creates a git tag to save the code used in this experiment
    """
    __test__ = False
    def __init__(
            self, save_dir, name="default", description=None, debug=False,
            version=None, create_git_tag=False
    ):
        super().__init__()
        self.save_dir = save_dir
        self._name = name
        self.description = description
        self.debug = debug
        self._version = version
        self.create_git_tag = create_git_tag
        self._experiment = None
    @property
    def experiment(self):
        r"""
          Actual test-tube object. To use test-tube features do the following.
          Example::
              self.logger.experiment.some_test_tube_function()
          """
        if self._experiment is not None:
            return self._experiment
        self._experiment = Experiment(
            save_dir=self.save_dir,
            name=self._name,
            debug=self.debug,
            version=self.version,
            description=self.description,
            create_git_tag=self.create_git_tag,
            rank=self.rank,
        )
        return self._experiment
    @rank_zero_only
    def log_hyperparams(self, params):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.experiment.argparse(params)
    @rank_zero_only
    def log_metrics(self, metrics, step=None):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.experiment.log(metrics, global_step=step)
    @rank_zero_only
    def save(self):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.experiment.save()
    @rank_zero_only
    def finalize(self, status):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        self.save()
        self.close()
    @rank_zero_only
    def close(self):
        # TODO: HACK figure out where this is being set to true
        self.experiment.debug = self.debug
        if not self.debug:
            exp = self.experiment
            exp.close()
    @property
    def rank(self):
        return self._rank
    @rank.setter
    def rank(self, value):
        self._rank = value
        if self._experiment is not None:
            self.experiment.rank = value
    @property
    def name(self):
        if self._experiment is None:
            return self._name
        else:
            return self.experiment.name
    @property
    def version(self):
        if self._experiment is None:
            return self._version
        else:
            return self.experiment.version
    # Test tube experiments are not pickleable, so we need to override a few
    # methods to get DDP working. See
    # https://docs.python.org/3/library/pickle.html#handling-stateful-objects
    # for more info.
    def __getstate__(self):
        state = self.__dict__.copy()
        state["_experiment"] = self.experiment.get_meta_copy()
        return state
    def __setstate__(self, state):
        self._experiment = state["_experiment"].get_non_ddp_exp()
        del state["_experiment"]
        self.__dict__.update(state)
@@ -1,10 +0,0 @@
 """
 .. warning:: `test_tube_logger` module has been renamed to `test_tube` since v0.6.0 and will be removed in v0.8.0
 """
 import warnings
 warnings.warn("`test_tube_logger` module has been renamed to `test_tube` since v0.6.0"
              " and will be removed in v0.8.0", DeprecationWarning)
 from pytorch_lightning.logging.test_tube import TestTubeLogger  # noqa: E402
--- a/Show More
+++ b/Show More