fix for #2051, always init with -inf

The comparison `self.monitor_op == torch.gt` was always false, compare with mode instead for #2051

.circleci/config.yml

@@ -22,7 +22,7 @@ references:
      command: |
        python --version ; pip --version ; pip list
        py.test pytorch_lightning tests -v --doctest-modules --junitxml=test-reports/pytest_junit.xml
-     no_output_timeout: 30m
+     no_output_timeout: 15m
 
   examples: &examples
    run:
@@ -125,20 +125,6 @@ jobs:
       - store_artifacts:
           path: test-reports
 
-  PyTorch-v1_1:
-    docker:
-      - image: circleci/python:3.6
-    environment:
-      - TORCH_VERSION: "torch>=1.1, <1.2"
-    steps: *steps
-
-  PyTorch-v1_2:
-    docker:
-      - image: circleci/python:3.6
-    environment:
-      - TORCH_VERSION: "torch>=1.2, <1.3"
-    steps: *steps
-
   PyTorch-v1_3:
     docker:
       - image: circleci/python:3.6
@@ -189,8 +175,6 @@ workflows:
     jobs:
       - Formatting
       - Build-Docs
-      - PyTorch-v1_1
-      - PyTorch-v1_2
       - PyTorch-v1_3
       - PyTorch-v1_4
       - PyTorch-v1_5

.codecov.yml

@@ -11,6 +11,8 @@ codecov:
   notify:
     # after_n_builds: 2
     wait_for_ci: yes
+  # https://docs.codecov.io/docs/codecov-yaml#section-expired-reports
+  max_report_age: off
 
 coverage:
   precision: 0  # 2 = xx.xx%, 0 = xx%

.github/CONTRIBUTING.md

@@ -4,10 +4,10 @@ Welcome to the PyTorch Lightning community! We're building the most advanced res
 ## Main Core Value: One less thing to remember
 
 Simplify the API as much as possible from the user perspective.
- Any additions or improvements should minimize things the user needs to remember.   
+ Any additions or improvements should minimize the things the user needs to remember.   
 
 For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval().
- This avoids all sorts of subtle errors the user could make.  
+ This helps users avoid all sorts of subtle errors.
 
 ## Lightning Design Principles
 We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.   
@@ -22,17 +22,17 @@ It's useful for users to look at the code and understand very quickly what's hap
  While that's super cool, this isn't the project for that :)      
 
 #### Force User Decisions To Best Practices
-There are 1,000 ways to do something. However, something eventually becomes standard practice that everyone does.
- Thus we pick one way of doing it and force everyone to do it this way.
+There are 1,000 ways to do something. However, eventually one popular solution becomes standard practice, and everyone follows.
+ We try to find the best way to solve a particular problem, and then force our users to use it for readability and simplicity.
  A good example is accumulated gradients.
- There are many ways to implement, we just pick one and force users to use that one.
+ There are many different ways to implement it, we just pick one and force users to use it.
  A bad forced decision would be to make users use a specific library to do something.    
 
-When something becomes a best practice, we add it to the framework. This likely looks like code in utils or in the model file that everyone keeps adding over and over again across projects. When this happens, bring that code inside the trainer and add a flag for it.
+When something becomes a best practice, we add it to the framework. This is usually something like bits of code in utils or in the model file that everyone keeps adding over and over again across projects. When this happens, bring that code inside the trainer and add a flag for it.
 
 #### Simple External API
-What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others.
- Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people.
+What makes sense to you may not make sense to others. When creating an issue with an API change suggestion, please validate that it makes sense for others.
+ Treat code changes the way you treat a startup: validate that it's a needed feature, then add if it makes sense for many people.
 
 #### Backward-compatible API
 We all hate updating our deep learning packages because we don't want to refactor a bunch of stuff. In Lightning, we make sure every change we make which could break an API is backwards compatible with good deprecation warnings.
@@ -40,7 +40,7 @@ We all hate updating our deep learning packages because we don't want to refacto
 **You shouldn't be afraid to upgrade Lightning :)**
 
 #### Gain User Trust
-As a researcher you can't have any part of your code going wrong. So, make thorough tests that ensure an implementation of a new trick or subbtle change is correct.
+As a researcher you can't have any part of your code going wrong. So, make thorough tests to ensure that every implementation of a new trick or subtle change is correct.
 
 #### Interoperability
 Have a favorite feature from other libraries like fast.ai or transformers? Those should just work with lightning as well. Grab your favorite model or learning rate scheduler from your favorite library and run it in Lightning.
@@ -48,25 +48,38 @@ Have a favorite feature from other libraries like fast.ai or transformers? Those
 ---
 
 ## Contribution Types
-Currently looking for help implementing new features or adding bug fixes.
+We are currently looking for help implementing new features or adding bug fixes.
 
-A lot of good work has already been done in project mechanics (requirements.txt, setup.py, pep8, badges, ci, etc...) we're in a good state there thanks to all the early contributors (even pre-beta release)!
+A lot of good work has already been done in project mechanics (requirements.txt, setup.py, pep8, badges, ci, etc...) so we're in a good state there thanks to all the early contributors (even pre-beta release)!
 
 ### Bug Fixes:
-1. Submit a github issue - try to decried what happen so other can reproduce it too.
-2. Try to ix it or recommend a solution...
+1. Submit a github issue - try to describe what happened so others can reproduce it too (config, code samples, expected vs. actual behaviour). 
+ Note, that the sample code shall be minimal and if needed with publicly available data.
+2. Try to fix it or recommend a solution...
+ We highly recommend to use test driven approach
+   * convert your minimal code example to a unit/integration test with assert on expected results
+   * start with debugging the issue... you can run just this particular test in your IDE and draft a fix
+   * verify that your test case fails on the master branch and only passes with the fix applied
 3. Submit a PR!
 
+_**Note**, even if you do not find the solution, sending a PR with a test covering the issue is a valid contribution and we can help you or finish it with you :]_
+
 
 ### New Features:
-1. Submit a github issue - describe what is motivation of such feature (plus an use-case).
-2. Let's discuss to agree on the feature scope.
-3. Submit a PR! (with updated docs and tests 🙃).
+1. Submit a github issue - describe what is the motivation of such feature (adding the use case or an example is helpful).
+2. Let's discuss to determine the feature scope.
+3. Submit a PR! (with updated docs and tests🙃).
 
 ---
 
 ## Guidelines
 
+### Original code
+
+All added or edited code shall be the own original work of the particular contributor.
+If you use come third-party implementation, all such blocks/functions/modules shall be properly referred and if possible also agreed by code's author. For example - `This code is inpired from http://...`.
+In case you adding new dependencies, make sure that they are compatible the actual PyTorch Lightning license (ie. dependencies should be _at least_ as permissive as the PyTorch Lightning license).
+
 ### Coding Style
 
 1. Use f-strings for output formation (except logging when we stay with lazy `logging.info("Hello %s!`, name).
@@ -125,7 +138,7 @@ _Artifacts_ tab in CircleCI when you click on the task named _ci/circleci: Build
 
 ### Testing
 
-Test your work locally to speed up your work since so you can focus only in particular (failing) test-cases.
+Testing your work locally will help you speed up the process since it allows you to focus on particular (failing) test-cases.
  To setup a local development environment, install both local and test dependencies:
 ```bash
 pip install -r requirements.txt
@@ -140,21 +153,21 @@ bash .run_local_tests.sh
 
 Note: if your computer does not have multi-GPU nor TPU these tests are skipped.
 
-For convenience, you can use also your own CircleCI building which will be triggered with each commit.
-This is useful if you do not test against all required dependencies version.
+For convenience, you can also use your own CircleCI building which will be triggered with each commit.
+This is useful if you do not test against all required dependency versions.
 To do so, login to [CircleCI](https://app.circleci.com/) and enable your forked project in the dashboard. It will just work after that.
 
 ### Pull Request
 
-We welcome any useful contribution! For convinece here's a recommended workflow:
+We welcome any useful contribution! For your convenience here's a recommended workflow:
 
 0. Think about what you want to do - fix a bug, repair docs, etc. 
 1. Start your work locally (usually until you need our CI testing)
     - create a branch and prepare your changes
     - hint: do not work with your master directly, it may become complicated when you need to rebase
     - hint: give your PR a good name! it will be useful later when you may work on multiple tasks/PRs
-2. Create a "Draft PR" which is clearly marked which lets us know you don't need feedback yet.
-3. When you feel like you are ready for integrating your work, turn your PR to "Ready for review".
+2. Create a "Draft PR" which is clearly marked, to let us know you don't need feedback yet.
+3. When you feel ready for integrating your work, mark your PR "Ready for review".
 4. Use tags in PR name for following cases:
     - **[blocked by #<number>]** if you work is depending on others changes
     - **[wip]** when you start to re-edit your work, mark it so no one will accidentally merge it in meantime
@@ -163,15 +176,15 @@ We welcome any useful contribution! For convinece here's a recommended workflow:
 
 1. **How can I help/contribute?**
 
-    All help is very welcome - reporting bug, solving issues and preparing bug fixes. To solve some issues you can start with label [good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) or chose something close to your domain with label [help wanted](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22). Before you start to implement anything check that the issue description that it is clear and self-assign the task to you (if it is not possible, just comment that you take it and we assign it to you...).
+    All help is very welcome - reporting bugs, solving issues and preparing bug fixes. To solve some issues you can start with label [good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) or chose something close to your domain with label [help wanted](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22). Before you start to implement anything check that the issue description that it is clear and self-assign the task to you (if it is not possible, just comment that you take it and we assign it to you...).
 
 2. **Is there a recommendation for branch names?**
     
-    We do not rely on the name convention so far you are working with your own fork. Anyway it would be nice to follow this convention `<type>/<issue-id>_<short-name>` where the types are: `bugfix`, `feaure`, `docs`, `tests`, ...
+    We do not rely on the name convention so far you are working with your own fork. Anyway it would be nice to follow this convention `<type>/<issue-id>_<short-name>` where the types are: `bugfix`, `feature`, `docs`, `tests`, ...
 
 3. **How to rebase my PR?**
     
-    We recommend to create a PR in separate branch different from `master`, especially if you plan to submit several changes and do not want to wait until the fist one is resolved (we can work on them in parallel). Update your master with upstream (assuming you have already set [upstream](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork))
+    We recommend creating a PR from a separate branch other than `master`, especially if you plan on submitting several changes at once and do not want to wait until the first one is resolved (we can work on them in parallel). Update your master with upstream (assuming you have already set [upstream](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork))
     ```bash
     git fetch --all --prune
     git checkout master

.github/ISSUE_TEMPLATE/bug_report.md

@@ -46,8 +46,8 @@ Please copy and paste the output from our
 You can get the script and run it with:
 ```
 wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
-# For security purposes, please check the contents of collect_env.py before running it.
-python collect_env.py
+# For security purposes, please check the contents of collect_env_details.py before running it.
+python collect_env_details.py
 ```
 
  - PyTorch Version (e.g., 1.0):

.github/PULL_REQUEST_TEMPLATE.md

@@ -6,7 +6,7 @@
 - [ ] Did you write any new necessary tests?  
 - [ ] If you made a notable change (that affects users), did you update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)?
 
-<!-- For CHANGELOG separate each item in unreleased section by blank line to reduce collisions -->
+<!-- For CHANGELOG separate each item in unreleased section by a blank line to reduce collisions -->
 
 ## What does this PR do?
 Fixes # (issue).

.github/workflows/ci-testing.yml

@@ -1,15 +1,8 @@
 name: CI testing
 
 # see: https://help.github.com/en/actions/reference/events-that-trigger-workflows
-on:
-  # Trigger the workflow on push or pull request,
-  # but only for the master branch
-  push:
-    branches:
-      - master
-  pull_request:
-    branches:
-      - master
+# Trigger the workflow on push or pull request
+on: [push, pull_request]
 
 jobs:
   build:
@@ -23,7 +16,7 @@ jobs:
         python-version: [3.6, 3.7, 3.8]
         requires: ['minimal', 'latest']
         exclude:
-          # excludes node 4 on macOS
+          # excludes PT 1.3 as it is missing on pypi
           - python-version: 3.8
             requires: 'minimal'
 
@@ -52,7 +45,13 @@ jobs:
     - name: Setup Windows on Latest
       if: runner.os == 'windows' && matrix.requires == 'latest'
       run: |
-        python -c "req = open('requirements.txt').read().replace('torch>=1.1', 'torch<1.5') ; open('requirements.txt', 'w').write(req)"
+        python -c "req = open('requirements.txt').read().replace('torch>=1.3', 'torch<1.5') ; open('requirements.txt', 'w').write(req)"
+
+    # versions <= 1.3 may have issues on mac with some BLAS ops due to missing mkl (https://github.com/pytorch/pytorch/issues/18996)
+    - name: Setup MacOS Minimal
+      if: runner.os == 'macOS' && matrix.requires == 'minimal'
+      run : |
+        python -c "req = open('requirements.txt').read().replace('torch>=1.3', 'torch>=1.4') ; open('requirements.txt', 'w').write(req)"
 
     - name: Set min. dependencies
       if: matrix.requires == 'minimal'
@@ -137,4 +136,4 @@ jobs:
     - name: Statistics
       if: success()
       run: |
-         coverage report
+         coverage report

.github/workflows/docker-builds.yml

@@ -14,7 +14,7 @@ jobs:
     strategy:
       matrix:
         python_version: [3.6, 3.7, 3.8]
-        pytorch_version: [1.1, 1.2, 1.3, 1.4, 1.5]
+        pytorch_version: [1.3, 1.4, 1.5]
     steps:
       - uses: actions/checkout@v2
 

.gitignore

@@ -133,4 +133,5 @@ mnist/
 # pl tests
 ml-runs/
 *.zip
-pytorch\ lightning
+pytorch\ lightning
+test-reports/

.mergify.yml

@@ -10,7 +10,7 @@ pull_request_rules:
       # no requested chnages from any reviewer
       - "#changes-requested-reviews-by=0"
       # this serves as ALL check has to pass as we have actually 27 tests in total
-      - "#status-success>=30"
+      - "#status-success>=28"
       # this is just in case since we rely on GPU tests (note: redundand to the above)
       - status-success=continuous-integration/drone/pr
       # this is patter-like, unofrunatly serves as `any(...)` (note: redundand to the above)

.readthedocs.yml

@@ -9,15 +9,16 @@ version: 2
 # reference: https://docs.readthedocs.io/en/stable/config-file/v2.html#sphinx
 sphinx:
   configuration: docs/source/conf.py
-  # TODO: set it true and debug failing
-  fail_on_warning: false
+  fail_on_warning: true
 
 # Build documentation with MkDocs
 #mkdocs:
 #  configuration: mkdocs.yml
 
 # Optionally build your docs in additional formats such as PDF and ePub
-formats: all
+formats:
+  - htmlzip
+  - pdf
 
 # Optionally set the version of Python and requirements required to build your docs
 python:

.run_local_tests.sh

@@ -12,5 +12,8 @@ rm -rf ./tests/cometruns*
 rm -rf ./tests/wandb*
 rm -rf ./tests/tests/*
 rm -rf ./lightning_logs
-python -m coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules --flake8
+python -m coverage run --source pytorch_lightning -m py.test pytorch_lightning tests pl_examples -v --doctest-modules --flake8 --durations=0
 python -m coverage report -m
+
+# specific file
+# python -m coverage run --source pytorch_lightning -m py.test -k test_trainer.py --flake8 --durations=0

CHANGELOG.md

@@ -4,6 +4,76 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
+## [unreleased] - YYYY-MM-DD
+
+### Added
+
+- Add Metric Base Classes ([#1326](https://github.com/PyTorchLightning/pytorch-lightning/pull/1326), [#1877](https://github.com/PyTorchLightning/pytorch-lightning/pull/1877))
+
+- Added type hints in `Trainer.fit()` and `Trainer.test()` to reflect that also a list of dataloaders can be passed in ([#1723](https://github.com/PyTorchLightning/pytorch-lightning/pull/1723))
+
+- Allow dataloaders without sampler field present ([#1907](https://github.com/PyTorchLightning/pytorch-lightning/pull/1907))
+
+- Added option `save_last` to save the model at the end of every epoch in `ModelCheckpoint` [(#1908)](https://github.com/PyTorchLightning/pytorch-lightning/pull/1908)
+
+- Early stopping checks `on_validation_end` ([#1458](https://github.com/PyTorchLightning/pytorch-lightning/pull/1458))
+
+- Attribute `best_model_path` to `ModelCheckpoint` for storing and later retrieving the path to the best saved model file ([#1799](https://github.com/PyTorchLightning/pytorch-lightning/pull/1799)) 
+
+- Speed up single-core TPU training by loading data using `ParallelLoader` ([#2033](https://github.com/PyTorchLightning/pytorch-lightning/pull/2033))
+
+- Added a model hook `transfer_batch_to_device` that enables moving custom data structures to the target device ([1756](https://github.com/PyTorchLightning/pytorch-lightning/pull/1756)).
+
+### Changed
+
+- Allow user to select individual TPU core to train on ([#1729](https://github.com/PyTorchLightning/pytorch-lightning/pull/1729))
+
+- Removed non-finite values from loss in `LRFinder` ([#1862](https://github.com/PyTorchLightning/pytorch-lightning/pull/1862))
+
+- Allow passing model hyperparameters as complete kwarg list ([#1896](https://github.com/PyTorchLightning/pytorch-lightning/pull/1896)) 
+
+- Renamed `ModelCheckpoint`'s attributes `best` to `best_model_score` and `kth_best_model` to `kth_best_model_path` ([#1799](https://github.com/PyTorchLightning/pytorch-lightning/pull/1799))
+
+- Re-Enable Logger's `ImportError`s ([#1938](https://github.com/PyTorchLightning/pytorch-lightning/pull/1938))
+
+- Changed the default value of the Trainer argument `weights_summary` from `full` to `top` ([#2029](https://github.com/PyTorchLightning/pytorch-lightning/pull/2029))
+
+### Deprecated
+
+- Deprecated `ModelCheckpoint`'s attributes `best` and `kth_best_model` ([#1799](https://github.com/PyTorchLightning/pytorch-lightning/pull/1799))
+
+- Dropped official support/testing for older PyTorch versions <1.3 ([#1917](https://github.com/PyTorchLightning/pytorch-lightning/pull/1917))
+
+### Removed
+
+- Removed unintended Trainer argument `progress_bar_callback`, the callback should be passed in by `Trainer(callbacks=[...])` instead ([#1855](https://github.com/PyTorchLightning/pytorch-lightning/pull/1855))
+
+- Remove obsolete `self._device` in Trainer ([#1849](https://github.com/PyTorchLightning/pytorch-lightning/pull/1849))
+
+### Fixed
+
+- Run graceful training teardown on interpreter exit ([#1631](https://github.com/PyTorchLightning/pytorch-lightning/pull/1631))
+
+- Fixed user warning when apex was used together with learning rate schedulers ([#1873](https://github.com/PyTorchLightning/pytorch-lightning/pull/1873))
+
+- Fixed multiple calls of `EarlyStopping` callback ([#1751](https://github.com/PyTorchLightning/pytorch-lightning/issues/1751)) 
+
+- Fixed an issue with `Trainer.from_argparse_args` when passing in unknown Trainer args ([#1932](https://github.com/PyTorchLightning/pytorch-lightning/pull/1932))
+
+- Fixed bug related to logger not being reset correctly for model after tuner algorithms ([#1933](https://github.com/PyTorchLightning/pytorch-lightning/pull/1933))
+
+- Fixed root node resolution for SLURM cluster with dash in host name ([#1954](https://github.com/PyTorchLightning/pytorch-lightning/pull/1954))
+
+- Fixed `LearningRateLogger` in multi-scheduler setting ([#1944](https://github.com/PyTorchLightning/pytorch-lightning/pull/1944))
+
+- Fixed test configuration check and testing ([#1804](https://github.com/PyTorchLightning/pytorch-lightning/pull/1804))
+
+- Fixed an issue with Trainer constructor silently ignoring unknown/misspelled arguments ([#1820](https://github.com/PyTorchLightning/pytorch-lightning/pull/1820))
+
+- Fixed `save_weights_only` in ModelCheckpoint ([#1780](https://github.com/PyTorchLightning/pytorch-lightning/pull/1780))
+
+- Allow use of same `WandbLogger` instance for multiple training loops ([#2055](https://github.com/PyTorchLightning/pytorch-lightning/pull/2055))
+
 ## [0.7.6] - 2020-05-16
 
 ### Added
@@ -29,6 +99,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Don't convert `namedtuple` to `tuple` when transferring the batch to target device ([#1589](https://github.com/PyTorchLightning/pytorch-lightning/pull/1589))
 - Allow passing hparams as keyword argument to LightningModule when loading from checkpoint ([#1639](https://github.com/PyTorchLightning/pytorch-lightning/pull/1639))
 - Args should come after the last positional argument ([#1807](https://github.com/PyTorchLightning/pytorch-lightning/pull/1807))
+- Made ddp the default if no backend specified with multiple GPUs ([#1789](https://github.com/PyTorchLightning/pytorch-lightning/pull/1789))
 
 ### Deprecated
 
@@ -42,7 +113,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed sampler logic for ddp with iterable dataset ([#1734](https://github.com/PyTorchLightning/pytorch-lightning/pull/1734))
 - Fixed `_reset_eval_dataloader()` for IterableDataset ([#1560](https://github.com/PyTorchLightning/pytorch-lightning/pull/1560))
 - Fixed Horovod distributed backend to set the `root_gpu` property ([#1669](https://github.com/PyTorchLightning/pytorch-lightning/pull/1669))
-- Fixed wandb logger `global_step` affects other loggers ([#1492](https://github.com/PyTorchLightning/pytorch-lightning/issues/1485))
+- Fixed wandb logger `global_step` affects other loggers ([#1492](https://github.com/PyTorchLightning/pytorch-lightning/pull/1492))
 - Fixed disabling progress bar on non-zero ranks using Horovod backend ([#1709](https://github.com/PyTorchLightning/pytorch-lightning/pull/1709))
 - Fixed bugs that prevent lr finder to be used together with early stopping and validation dataloaders ([#1676](https://github.com/PyTorchLightning/pytorch-lightning/pull/1676))
 - Fixed a bug in Trainer that prepended the checkpoint path with `version_` when it shouldn't ([#1748](https://github.com/PyTorchLightning/pytorch-lightning/pull/1748))
@@ -51,6 +122,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed accumulation parameter and suggestion method for learning rate finder ([#1801](https://github.com/PyTorchLightning/pytorch-lightning/pull/1801))
 - Fixed num processes wasn't being set properly and auto sampler was ddp failing ([#1819](https://github.com/PyTorchLightning/pytorch-lightning/pull/1819))
 - Fixed bugs in semantic segmentation example ([#1824](https://github.com/PyTorchLightning/pytorch-lightning/pull/1824))
+- Fixed saving native AMP scaler state ([#1561](https://github.com/PyTorchLightning/pytorch-lightning/pull/1561), [#1777](https://github.com/PyTorchLightning/pytorch-lightning/pull/1777))
+- Fixed native amp + ddp ([#1788](https://github.com/PyTorchLightning/pytorch-lightning/pull/1788))
+- Fixed `hparam` logging with metrics ([#1647](https://github.com/PyTorchLightning/pytorch-lightning/pull/1647))
 
 ## [0.7.5] - 2020-04-27
 
@@ -594,16 +668,16 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 - Fixed a bug where `Experiment` object was not process safe, potentially causing logs to be overwritten
 
-## [0.3.5] - 2019-MM-DD
+## [0.3.5] - 2019-07-25
 
-## [0.3.4] - 2019-MM-DD
+## [0.3.4] - 2019-07-22
 
-## [0.3.3] - 2019-MM-DD
+## [0.3.3] - 2019-07-22
 
-## [0.3.2] - 2019-MM-DD
+## [0.3.2] - 2019-07-21
 
-## [0.3.1] - 2019-MM-DD
+## [0.3.1] - 2019-07-21
 
-## [0.2.x] - YYYY-MM-DD
+## [0.2.x] - 2019-07-09
 
-## [0.1.x] - YYYY-MM-DD
+## [0.1.x] - 2019-06-DD

README.md

@@ -15,7 +15,7 @@
 [![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=stable)](https://pytorch-lightning.readthedocs.io/en/stable/)
 [![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/pytorch-lightning/shared_invite/enQtODU5ODIyNTUzODQwLTFkMDg5Mzc1MDBmNjEzMDgxOTVmYTdhYjA1MDdmODUyOTg2OGQ1ZWZkYTQzODhhNzdhZDA3YmNhMDhlMDY4YzQ)
 [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/PytorchLightning/pytorch-lightning/blob/master/LICENSE)
-[![Next Release](https://img.shields.io/badge/Next%20Release-May%2020-<COLOR>.svg)](https://shields.io/)
+[![Next Release](https://img.shields.io/badge/Next%20Release-May%2029-<COLOR>.svg)](https://shields.io/)
 
 <!--
 removed until codecov badge isn't empy. likely a config error showing nothing on master.
@@ -27,13 +27,13 @@ removed until codecov badge isn't empy. likely a config error showing nothing on
 ## Continuous Integration
 <center>
 
-| System / PyTorch ver. | 1.1 (min. reg) | 1.2 | 1.3 | 1.4 | 1.5 (latest) |
-| :---: | :---: | :---: | :---: | :---: | :---: |
-| Linux py3.6 [CPU] | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) |
-| Linux py3.7 [GPU] | - | - | - | - | [![Build Status](http://35.192.60.23/api/badges/PyTorchLightning/pytorch-lightning/status.svg)](http://35.192.60.23/PyTorchLightning/pytorch-lightning) |
-| Linux py3.6 / py3.7 / py3.8 | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | - | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) |
-| OSX py3.6 / py3.7 / py3.8| [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | - | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) |
-| Windows py3.6 / py3.7 / py3.8 | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
+| System / PyTorch ver. | 1.3 (min. reg) | 1.4 | 1.5 (latest) |
+| :---: | :---: | :---: | :---: |
+| Linux py3.6 [CPU] | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) | [![CircleCI](https://circleci.com/gh/PyTorchLightning/pytorch-lightning.svg?style=svg)](https://circleci.com/gh/PyTorchLightning/pytorch-lightning) |
+| Linux py3.7 [GPU] | - | - | [![Build Status](http://35.192.60.23/api/badges/PyTorchLightning/pytorch-lightning/status.svg)](http://35.192.60.23/PyTorchLightning/pytorch-lightning) |
+| Linux py3.6 / py3.7 / py3.8 | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) |
+| OSX py3.6 / py3.7 / py3.8| - | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) |
+| Windows py3.6 / py3.7 / py3.8 | [![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) |[![CI testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20testing/badge.svg?event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
 
 </center>
 
@@ -287,7 +287,11 @@ trainer = Trainer(max_epochs=1, gpus=8, num_nodes=32)
 
 Or TPUs
 ```python
-trainer = Trainer(num_tpu_cores=8)
+# Distributes TPU core training
+trainer = Trainer(tpu_cores=8)
+
+# Single TPU core training
+trainer = Trainer(tpu_cores=[1])
 ```
 
 When you're done training, run the test accuracy
@@ -380,22 +384,6 @@ conda activate my_env
 pip install pytorch-lightning
 ```
 
-**Which PyTorch versions do you support?**    
-- **PyTorch 1.1.0**
-    ```bash
-    # install pytorch 1.1.0 using the official instructions
-
-    # install test-tube 0.6.7.6 which supports 1.1.0
-    pip install test-tube==0.6.7.6
-
-    # install latest Lightning version without upgrading deps
-    pip install -U --no-deps pytorch-lightning
-    ```
-- **PyTorch 1.2.0+**
-   ```python
-   pip install pytorch-lightning
-   ```
-
 ## Custom installation
 
 ### Bleeding edge
@@ -433,6 +421,7 @@ pip install https://github.com/PytorchLightning/pytorch-lightning/archive/0.X.Y.
 - Jeremy Jordan [(jeremyjordan)](https://github.com/jeremyjordan)
 - Tullie Murrell [(tullie)](https://github.com/tullie)
 - Adrian Wälchli [(awaelchli)](https://github.com/awaelchli)
+- Nicki Skafte [(skaftenicki)](https://github.com/SkafteNicki)
 
 #### Funding    
 Building open-source software with only a few part-time people is hard! We've secured funding to make sure we can

docs/source/apex.rst

@@ -10,6 +10,8 @@ Lightning offers 16-bit training for CPUs, GPUs and TPUs.
 GPU 16-bit
 -----------
 Lightning uses NVIDIA apex to handle 16-bit precision training.
+16 bit precision can cut your memory footprint by half.
+If using volta architecture GPUs it can give a dramatic training speed-up as well.
 
 To use 16-bit precision, do two things:
 
@@ -18,6 +20,7 @@ To use 16-bit precision, do two things:
 
 Install apex
 ^^^^^^^^^^^^
+
 .. code-block:: bash
 
     $ git clone https://github.com/NVIDIA/apex
@@ -58,7 +61,7 @@ TPU 16-bit
 .. testcode::
 
     # DEFAULT
-    trainer = Trainer(num_tpu_cores=8, precision=32)
+    trainer = Trainer(tpu_cores=8, precision=32)
 
     # turn on 16-bit
-    trainer = Trainer(num_tpu_cores=8, precision=16)
+    trainer = Trainer(tpu_cores=8, precision=16)

docs/source/conf.py

@@ -65,6 +65,11 @@ version = pytorch_lightning.__version__
 # The full version, including alpha/beta/rc tags
 release = pytorch_lightning.__version__
 
+# Options for the linkcode extension
+# ----------------------------------
+github_user = 'PyTorchLightning'
+github_repo = project
+
 # -- General configuration ---------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
@@ -338,12 +343,6 @@ MOCK_MANUAL_PACKAGES = [
 autodoc_mock_imports = MOCK_PACKAGES + MOCK_MANUAL_PACKAGES
 
 
-# Options for the linkcode extension
-# ----------------------------------
-github_user = 'PyTorchLightning'
-github_repo = project
-
-
 # Resolve function
 # This function is used to populate the (source) links in the API
 def linkcode_resolve(domain, info):

docs/source/early_stopping.rst

@@ -19,36 +19,63 @@ By default early stopping will be enabled if `'val_loss'`
 is found in :meth:`~pytorch_lightning.core.lightning.LightningModule.validation_epoch_end`'s
 return dict. Otherwise training will proceed with early stopping disabled.
 
-Enable Early Stopping using Callbacks on epoch end
---------------------------------------------------
-There are two ways to enable early stopping using callbacks on epoch end.
+Enable Early Stopping using the EarlyStopping Callback
+------------------------------------------------------
+The
+:class:`~pytorch_lightning.callbacks.early_stopping.EarlyStopping`
+callback can be used to monitor a validation metric and stop the training when no improvement is observed.
 
--   Set early_stop_callback to True. Will look for 'val_loss' in validation_epoch_end() return dict.
-    If it is not found an error is raised.
+There are two ways to enable the EarlyStopping callback:
+
+-   Set `early_stop_callback=True`.
+    The callback will look for 'val_loss' in the dict returned by 
+    :meth:`~pytorch_lightning.core.lightning.LightningModule.validation_epoch_end`
+    and raise an error if `val_loss` is not present.
 
     .. testcode::
 
         trainer = Trainer(early_stop_callback=True)
 
--   Or configure your own callback
+-   Create the callback object and pass it to the trainer.
+    This allows for further customization.
 
     .. testcode::
 
         early_stop_callback = EarlyStopping(
-            monitor='val_loss',
-            min_delta=0.00,
-            patience=3,
-            verbose=False,
-            mode='min'
+           monitor='val_accuracy',
+           min_delta=0.00,
+           patience=3,
+           verbose=False,
+           mode='max'
         )
         trainer = Trainer(early_stop_callback=early_stop_callback)
 
-In any case, the callback will fall back to the training metrics (returned in
-:meth:`~pytorch_lightning.core.lightning.LightningModule.training_step`,
-:meth:`~pytorch_lightning.core.lightning.LightningModule.training_step_end`)
-looking for a key to monitor if validation is disabled or
-:meth:`~pytorch_lightning.core.lightning.LightningModule.validation_epoch_end`
-is not defined.
+In case you need early stopping in a different part of training, subclass EarlyStopping
+and change where it is called:
+
+.. testcode::
+
+    class MyEarlyStopping(EarlyStopping):
+
+        def on_validation_end(self, trainer, pl_module):
+            # override this to disable early stopping at the end of val loop
+            pass
+
+        def on_train_end(self, trainer, pl_module):
+            # instead, do it at the end of training loop
+            self._run_early_stopping_check(trainer, pl_module)
+
+.. note::
+   The EarlyStopping callback runs at the end of every validation epoch,
+   which, under the default configuration, happen after every training epoch.
+   However, the frequency of validation can be modified by setting various parameters
+   on the :class:`~pytorch_lightning.trainer.trainer.Trainer`,
+   for example :paramref:`~pytorch_lightning.trainer.trainer.Trainer.check_val_every_n_epoch`
+   and :paramref:`~pytorch_lightning.trainer.trainer.Trainer.val_check_interval`.
+   It must be noted that the `patience` parameter counts the number of
+   validation epochs with no improvement, and not the number of training epochs.
+   Therefore, with parameters `check_val_every_n_epoch=10` and `patience=3`, the trainer
+   will perform at least 40 training epochs before being stopped.
 
 .. seealso::
     - :class:`~pytorch_lightning.trainer.trainer.Trainer`

docs/source/governance.rst

@@ -16,3 +16,4 @@ Core Maintainers
 - Jeremy Jordan (`jeremyjordan <https://github.com/jeremyjordan>`_)
 - Tullie Murrell (`tullie <https://github.com/tullie>`_)
 - Adrian Wälchli (`awaelchli <https://github.com/awaelchli>`_)
+- Nicki Skafte (`skaftenicki <https://github.com/SkafteNicki>`_)

docs/source/hyperparameters.rst

@@ -75,7 +75,7 @@ Now in your main trainer file, add the Trainer args, the program args, and add t
     # ie: now --gpus --num_nodes ... --fast_dev_run all work in the cli
     parser = Trainer.add_argparse_args(parser)
 
-    hparams = parser.parse_args()
+    args = parser.parse_args()
 
 Now you can call run your program like so
 
@@ -87,39 +87,50 @@ Finally, make sure to start the training like so:
 
 .. code-block:: python
 
-    # YES
-    model = LitModel(hparams)
-    trainer = Trainer.from_argparse_args(hparams, early_stopping_callback=...)
+    # init the trainer like this
+    trainer = Trainer.from_argparse_args(args, early_stopping_callback=...)
 
-    # NO
-    # model = LitModel(learning_rate=hparams.learning_rate, ...)
-    # trainer = Trainer(gpus=hparams.gpus, ...)
+    # NOT like this
+    trainer = Trainer(gpus=hparams.gpus, ...)
 
-LightningModule hparams
-^^^^^^^^^^^^^^^^^^^^^^^
+    # init the model with Namespace directly
+    model = LitModel(args)
 
-Normally, we don't hard-code the values to a model. We usually use the command line to
-modify the network and read those values in the LightningModule
+    # or init the model with all the key-value pairs
+    dict_args = vars(args)
+    model = LitModel(**dict_args)
+
+LightningModule hyperparameters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. warning:: The use of `hparams` is no longer recommended (but still supported)
+
+LightningModule is just an nn.Module, you can use it as you normally would. However, there are
+some best practices to improve readability and reproducibility.
+
+1. It's more readable to specify all the arguments that go into a module (with default values).
+This helps users of your module know everything that is required to run this.
 
 .. testcode::
 
     class LitMNIST(LightningModule):
 
-        def __init__(self, hparams):
+        def __init__(self, layer_1_dim=128, layer_2_dim=256, learning_rate=1e-4, batch_size=32, **kwargs):
             super().__init__()
+            self.layer_1_dim = layer_1_dim
+            self.layer_2_dim = layer_2_dim
+            self.learning_rate = learning_rate
+            self.batch_size = batch_size
 
-            # do this to save all arguments in any logger (tensorboard)
-            self.hparams = hparams
-
-            self.layer_1 = torch.nn.Linear(28 * 28, hparams.layer_1_dim)
-            self.layer_2 = torch.nn.Linear(hparams.layer_1_dim, hparams.layer_2_dim)
-            self.layer_3 = torch.nn.Linear(hparams.layer_2_dim, 10)
+            self.layer_1 = torch.nn.Linear(28 * 28, self.layer_1_dim)
+            self.layer_2 = torch.nn.Linear(self.layer_1_dim, self.layer_2_dim)
+            self.layer_3 = torch.nn.Linear(self.layer_2_dim, 10)
 
         def train_dataloader(self):
-            return DataLoader(mnist_train, batch_size=self.hparams.batch_size)
+            return DataLoader(mnist_train, batch_size=self.batch_size)
 
         def configure_optimizers(self):
-            return Adam(self.parameters(), lr=self.hparams.learning_rate)
+            return Adam(self.parameters(), lr=self.learning_rate)
 
         @staticmethod
         def add_model_specific_args(parent_parser):
@@ -130,20 +141,59 @@ modify the network and read those values in the LightningModule
             parser.add_argument('--learning_rate', type=float, default=0.002)
             return parser
 
-Now pass in the params when you init your model
+2. You can also pass in a dict or Namespace, but this obscures the parameters your module is looking
+for. The user would have to search the file to find what is parametrized.
+
+.. code-block:: python
+
+    # using a argparse.Namespace
+    class LitMNIST(LightningModule):
+
+        def __init__(self, hparams, *args, **kwargs):
+            super().__init__()
+            self.hparams = hparams
+
+            self.layer_1 = torch.nn.Linear(28 * 28, self.hparams.layer_1_dim)
+            self.layer_2 = torch.nn.Linear(self.hparams.layer_1_dim, self.hparams.layer_2_dim)
+            self.layer_3 = torch.nn.Linear(self.hparams.layer_2_dim, 10)
+
+        def train_dataloader(self):
+            return DataLoader(mnist_train, batch_size=self.hparams.batch_size)
+
+One way to get around this is to convert a Namespace or dict into key-value pairs using `**`
 
 .. code-block:: python
 
     parser = ArgumentParser()
     parser = LitMNIST.add_model_specific_args(parser)
-    hparams = parser.parse_args()
-    model = LitMNIST(hparams)
+    args = parser.parse_args()
+    dict_args = vars(args)
+    model = LitMNIST(**dict_args)
 
-The line `self.hparams = hparams` is very special. This line assigns your hparams to the LightningModule.
-This does two things:
+Within any LightningModule all the arguments you pass into your `__init__` will be stored in
+the checkpoint so that you know all the values that went into creating this model.
+
+We will also add all of those values to the TensorBoard hparams tab (unless it's an object which
+we won't). We also will store those values into checkpoints for you which you can use to init your
+models.
+
+.. code-block:: python
+
+    class LitMNIST(LightningModule):
+
+        def __init__(self, layer_1_dim, some_other_param):
+            super().__init__()
+            self.layer_1_dim = layer_1_dim
+            self.some_other_param = some_other_param
+
+            self.layer_1 = torch.nn.Linear(28 * 28, self.layer_1_dim)
+
+            self.layer_2 = torch.nn.Linear(self.layer_1_dim, self.some_other_param)
+            self.layer_3 = torch.nn.Linear(self.some_other_param, 10)
+
+
+    model = LitMNIST(10, 20)
 
-1.  It adds them automatically to TensorBoard logs under the hparams tab.
-2.  Lightning will save those hparams to the checkpoint and use them to restore the module correctly.
 
 Trainer args
 ^^^^^^^^^^^^
@@ -171,13 +221,13 @@ polluting the main.py file, the LightningModule lets you define arguments for ea
 
     class LitMNIST(LightningModule):
 
-        def __init__(self, hparams):
+        def __init__(self, layer_1_dim, **kwargs):
             super().__init__()
-            self.layer_1 = torch.nn.Linear(28 * 28, hparams.layer_1_dim)
+            self.layer_1 = torch.nn.Linear(28 * 28, layer_1_dim)
 
         @staticmethod
         def add_model_specific_args(parent_parser):
-            parser = ArgumentParser(parents=[parent_parser])
+            parser = ArgumentParser(parents=[parent_parser], add_help=False)
             parser.add_argument('--layer_1_dim', type=int, default=128)
             return parser
 
@@ -185,13 +235,13 @@ polluting the main.py file, the LightningModule lets you define arguments for ea
 
     class GoodGAN(LightningModule):
 
-        def __init__(self, hparams):
+        def __init__(self, encoder_layers, **kwargs):
             super().__init__()
-            self.encoder = Encoder(layers=hparams.encoder_layers)
+            self.encoder = Encoder(layers=encoder_layers)
 
         @staticmethod
         def add_model_specific_args(parent_parser):
-            parser = ArgumentParser(parents=[parent_parser])
+            parser = ArgumentParser(parents=[parent_parser], add_help=False)
             parser.add_argument('--encoder_layers', type=int, default=12)
             return parser
 
@@ -201,14 +251,14 @@ Now we can allow each model to inject the arguments it needs in the ``main.py``
 .. code-block:: python
 
     def main(args):
+        dict_args = vars(args)
 
         # pick model
         if args.model_name == 'gan':
-            model = GoodGAN(hparams=args)
+            model = GoodGAN(**dict_args)
         elif args.model_name == 'mnist':
-            model = LitMNIST(hparams=args)
+            model = LitMNIST(**dict_args)
 
-        model = LitMNIST(hparams=args)
         trainer = Trainer.from_argparse_args(args)
         trainer.fit(model)
 

docs/source/index.rst

@@ -23,6 +23,7 @@ PyTorch Lightning Documentation
    hooks
    lightning-module
    loggers
+   metrics
    trainer
 
 .. toctree::
@@ -31,6 +32,7 @@ PyTorch Lightning Documentation
    :caption: Community Examples
 
    Contextual Emotion Detection (DoubleDistilBert) <https://github.com/PyTorchLightning/emotion_transformer>
+   FasterRCNN object detection + Hydra <https://github.com/PyTorchLightning/wheat>
    Generative Adversarial Network <https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=TyYOdg8g77P0>
    Hyperparameter optimization with Optuna <https://github.com/optuna/optuna/blob/master/examples/pytorch_lightning_simple.py>
    Image Inpainting using Partial Convolutions <https://github.com/ryanwongsa/Image-Inpainting>
@@ -51,6 +53,13 @@ PyTorch Lightning Documentation
 
    From PyTorch to PyTorch Lightning <https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09>
 
+.. toctree::
+   :maxdepth: 1
+   :name: project structure
+   :caption: Recommended Lightning Project Layout
+
+   Lightning project seed <https://github.com/PyTorchLightning/pytorch-lightning-conference-seed>
+
 .. toctree::
    :maxdepth: 1
    :name: Common Use Cases
@@ -107,7 +116,8 @@ Indices and tables
    api/pytorch_lightning.core
    api/pytorch_lightning.callbacks
    api/pytorch_lightning.loggers
+   api/pytorch_lightning.metrics
    api/pytorch_lightning.overrides
    api/pytorch_lightning.profiler
    api/pytorch_lightning.trainer
-   api/pytorch_lightning.utilities
+   api/pytorch_lightning.utilities

docs/source/introduction_guide.rst

@@ -185,7 +185,7 @@ EXACTLY the same as you would a PyTorch Module.
 
  Out:
 
- .. code-block:: none
+ .. code-block:: python
 
     torch.Size([1, 10])
 
@@ -519,50 +519,8 @@ First, change the runtime to TPU (and reinstall lightning).
 
 Next, install the required xla library (adds support for PyTorch on TPUs)
 
-.. code-block:: python
-
-    import collections
-    from datetime import datetime, timedelta
-    import os
-    import requests
-    import threading
-
-    _VersionConfig = collections.namedtuple('_VersionConfig', 'wheels,server')
-    VERSION = "torch_xla==nightly"  #@param ["xrt==1.15.0", "torch_xla==nightly"]
-    CONFIG = {
-        'xrt==1.15.0': _VersionConfig('1.15', '1.15.0'),
-        'torch_xla==nightly': _VersionConfig('nightly', 'XRT-dev{}'.format(
-            (datetime.today() - timedelta(1)).strftime('%Y%m%d'))),
-    }[VERSION]
-    DIST_BUCKET = 'gs://tpu-pytorch/wheels'
-    TORCH_WHEEL = 'torch-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
-    TORCH_XLA_WHEEL = 'torch_xla-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
-    TORCHVISION_WHEEL = 'torchvision-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
-
-    # Update TPU XRT version
-    def update_server_xrt():
-      print('Updating server-side XRT to {} ...'.format(CONFIG.server))
-      url = 'http://{TPU_ADDRESS}:8475/requestversion/{XRT_VERSION}'.format(
-          TPU_ADDRESS=os.environ['COLAB_TPU_ADDR'].split(':')[0],
-          XRT_VERSION=CONFIG.server,
-      )
-      print('Done updating server-side XRT: {}'.format(requests.post(url)))
-
-    update = threading.Thread(target=update_server_xrt)
-    update.start()
-
-.. code-block::
-
-    # Install Colab TPU compat PyTorch/TPU wheels and dependencies
-    !pip uninstall -y torch torchvision
-    !gsutil cp "$DIST_BUCKET/$TORCH_WHEEL" .
-    !gsutil cp "$DIST_BUCKET/$TORCH_XLA_WHEEL" .
-    !gsutil cp "$DIST_BUCKET/$TORCHVISION_WHEEL" .
-    !pip install "$TORCH_WHEEL"
-    !pip install "$TORCH_XLA_WHEEL"
-    !pip install "$TORCHVISION_WHEEL"
-    !sudo apt-get install libomp5
-    update.join()
+    !curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
+    !python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev
 
 In distributed training (multiple GPUs and multiple TPU cores) each GPU or TPU core will run a copy
 of this program. This means that without taking any care you will download the dataset N times which
@@ -609,7 +567,7 @@ Now we can train the LightningModule on a TPU without doing anything else!
 .. code-block:: python
 
     model = LitMNIST()
-    trainer = Trainer(num_tpu_cores=8)
+    trainer = Trainer(tpu_cores=8)
     trainer.fit(model)
 
 You'll now see the TPU cores booting up.
@@ -696,7 +654,7 @@ while checking the validation set.
     from pytorch_lightning import Trainer
 
     model = LitMNIST()
-    trainer = Trainer(num_tpu_cores=8)
+    trainer = Trainer(tpu_cores=8)
     trainer.fit(model)
 
 You may have noticed the words `Validation sanity check` logged. This is because Lightning runs 5 batches
@@ -747,7 +705,7 @@ Once you train your model simply call `.test()`.
     from pytorch_lightning import Trainer
 
     model = LitMNIST()
-    trainer = Trainer(num_tpu_cores=8)
+    trainer = Trainer(tpu_cores=8)
     trainer.fit(model)
 
     # run test set
@@ -769,7 +727,7 @@ You can also run the test from a saved lightning model
 .. code-block:: python
 
     model = LitMNIST.load_from_checkpoint(PATH)
-    trainer = Trainer(num_tpu_cores=8)
+    trainer = Trainer(tpu_cores=8)
     trainer.test(model)
 
 .. note:: Lightning disables gradients, puts model in eval mode and does everything needed for testing.

docs/source/lr_finder.rst

@@ -22,32 +22,31 @@ Warnings:
 - For the moment, this feature only works with models having a single optimizer.
 - LR support for DDP is not implemented yet, it is comming soon.
 
-Using Lightnings build-in LR finder
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Using Lightning's built-in LR finder
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 In the most basic use case, this feature can be enabled during trainer construction
-with ``Trainer(auto_lr_find=True)``. When ``.fit(model)`` is called, the lr finder
+with ``Trainer(auto_lr_find=True)``. When ``.fit(model)`` is called, the LR finder
 will automatically be run before any training is done. The ``lr`` that is found
 and used will be written to the console and logged together with all other
 hyperparameters of the model.
     
 .. testcode::
         
-    # default, no automatic learning rate finder
-    trainer = Trainer(auto_lr_find=True)
+    # default: no automatic learning rate finder
+    trainer = Trainer(auto_lr_find=False)
+
+This flag sets your learning rate which can be accessed via ``self.lr`` or ``self.learning_rate``.
 
-When the ``lr`` or ``learning_rate`` key in hparams exists, this flag sets your learning_rate.
-In both cases, if the respective fields are not found, an error will be thrown.
-        
 .. testcode::
 
     class LitModel(LightningModule):
 
-        def __init__(self, hparams):
-            self.hparams = hparams
+        def __init__(self, learning_rate):
+            self.learning_rate = learning_rate
 
         def configure_optimizers(self):
-            return Adam(self.parameters(), lr=self.hparams.lr|self.hparams.learning_rate)
+            return Adam(self.parameters(), lr=(self.lr or self.learning_rate))
 
     # finds learning rate automatically
     # sets hparams.lr or hparams.learning_rate to that learning rate
@@ -97,7 +96,7 @@ of this would look like
     
     # update hparams of the model
     model.hparams.lr = new_lr
-    
+
     # Fit model
     trainer.fit(model)
     

docs/source/metrics.rst

@@ -0,0 +1,4 @@
+.. automodule:: pytorch_lightning.metrics
+   :members:
+   :noindex:
+   :exclude-members:

docs/source/multi_gpu.rst

@@ -81,9 +81,9 @@ when needed.
 
 .. note:: For iterable datasets, we don't do this automatically.
 
-Make Model Picklable
-^^^^^^^^^^^^^^^^^^^^
-It's very likely your code is already `picklable <https://docs.python.org/3/library/pickle.html>`_,
+Make model pickleable
+^^^^^^^^^^^^^^^^^^^^^
+It's very likely your code is already `pickleable <https://docs.python.org/3/library/pickle.html>`_,
 so you don't have to do anything to make this change.
 However, if you run distributed and see an error like this:
 
@@ -122,22 +122,102 @@ is usually helpful.
 ie: in the stacktrace example here, there seems to be a lambda function somewhere in the user code
 which cannot be pickled.
 
+GPU device selection
+--------------------
+
+You can select the GPU devices with ranges, a list of indices or a string containing
+a comma separated list of GPU ids:
+
+.. testsetup::
+
+    k = 1
+
+.. testcode::
+    :skipif: torch.cuda.device_count() < 2
+
+    # DEFAULT (int) specifies how many GPUs to use
+    Trainer(gpus=k)
+
+    # Above is equivalent to
+    Trainer(gpus=list(range(k)))
+
+    # Specify which GPUs to use (don't use if running on cluster)
+    Trainer(gpus=[0, 1])
+
+    # can also be a string
+    Trainer(gpus='0, 1')
+
+    # can also be -1 or '-1', this uses all available GPUs
+    # equivalent to list(range(torch.cuda.available_devices()))
+    Trainer(gpus=-1)
+
+The table below lists examples of possible input formats and how they are interpreted by Lightning.
+Note in particular the difference between `gpus=0`, `gpus=[0]` and `gpus="0"`.
+
++---------------+-----------+---------------------+---------------------------------+
+| `gpus`        | Type      | Parsed              | Meaning                         |
++===============+===========+=====================+=================================+
+| None          | NoneType  | None                | CPU                             |
++---------------+-----------+---------------------+---------------------------------+
+| 0             | int       | None                | CPU                             |
++---------------+-----------+---------------------+---------------------------------+
+| 3             | int       | [0, 1, 2]           | first 3 GPUs                    |
++---------------+-----------+---------------------+---------------------------------+
+| -1            | int       | [0, 1, 2, ...]      | all available GPUs              |
++---------------+-----------+---------------------+---------------------------------+
+| [0]           | list      | [0]                 | GPU 0                           |
++---------------+-----------+---------------------+---------------------------------+
+| [1, 3]        | list      | [1, 3]              | GPUs 1 and 3                    |
++---------------+-----------+---------------------+---------------------------------+
+| "0"           | str       | [0]                 | GPU 0                           |
++---------------+-----------+---------------------+---------------------------------+
+| "3"           | str       | [3]                 | GPU 3                           |
++---------------+-----------+---------------------+---------------------------------+
+| "1, 3"        | str       | [1, 3]              | GPUs 1 and 3                    |
++---------------+-----------+---------------------+---------------------------------+
+| "-1"          | str       | [0, 1, 2, ...]      | all available GPUs              |
++---------------+-----------+---------------------+---------------------------------+
+
+CUDA flags
+^^^^^^^^^^
+
+CUDA flags make certain GPUs visible to your script.
+Lightning sets these for you automatically, there's NO NEED to do this yourself.
+
+.. testcode::
+
+    # lightning will set according to what you give the trainer
+    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+    os.environ["CUDA_VISIBLE_DEVICES"] = "0"
+
+However, when using a cluster, Lightning will NOT set these flags (and you should not either).
+SLURM will set these for you.
+For more details see the `SLURM cluster guide <slurm.rst>`_.
+
+
 Distributed modes
 -----------------
 Lightning allows multiple ways of training
 
 - Data Parallel (`distributed_backend='dp'`) (multiple-gpus, 1 machine)
 - DistributedDataParallel (`distributed_backend='ddp'`) (multiple-gpus across many machines).
-- DistributedDataParallel2 (`distributed_backend='ddp2'`) (dp in a machine, ddp across machines).
+- DistributedDataParallel 2 (`distributed_backend='ddp2'`) (dp in a machine, ddp across machines).
 - Horovod (`distributed_backend='horovod'`) (multi-machine, multi-gpu, configured at runtime)
-- TPUs (`num_tpu_cores=8|x`) (tpu or TPU pod)
+- TPUs (`tpu_cores=8|x`) (tpu or TPU pod)
 
-.. note:: If you request multiple GPUs without setting a mode, ddp will be automatically used.
+.. note::
+    If you request multiple GPUs or nodes without setting a mode, ddp will be automatically used.
 
-Data Parallel (dp)
-^^^^^^^^^^^^^^^^^^
-`DataParallel <https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel>`_ splits a batch across k GPUs. That is, if you have a batch of 32 and use dp with 2 gpus,
-each GPU will process 16 samples, after which the root node will aggregate the results.
+For a deeper understanding of what Lightning is doing, feel free to read this
+`guide <https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565>`_.
+
+
+
+Data Parallel
+^^^^^^^^^^^^^
+`DataParallel <https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel>`_ splits a batch across k GPUs.
+That is, if you have a batch of 32 and use dp with 2 gpus, each GPU will process 16 samples,
+after which the root node will aggregate the results.
 
 .. warning:: DP use is discouraged by PyTorch and Lightning. Use ddp which is more stable and at least 3x faster
 
@@ -157,7 +237,7 @@ Distributed Data Parallel
 
 3. Each process inits the model.
 
-.. note:: Make sure  to set the random seed so that each model inits  with the same weights
+.. note:: Make sure to set the random seed so that each model initializes with the same weights.
 
 4. Each process performs a full forward and backward pass in parallel.
 
@@ -176,11 +256,11 @@ Distributed Data Parallel
 Distributed Data Parallel 2
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 In certain cases, it's advantageous to use all batches on the same machine instead of a subset.
-For instance you might want to compute a NCE loss where it pays  to have more negative samples.
+For instance you might want to compute a NCE loss where it pays to have more negative samples.
 
 In  this case, we can use ddp2 which behaves like dp in a machine and ddp across nodes. DDP2 does the following:
 
-1. Copies a subset of the  data to each node.
+1. Copies a subset of the data to each node.
 
 2. Inits a model on each node.
 
@@ -297,7 +377,7 @@ In pseudocode, the full sequence is:
     # use the full batch for something like softmax
     full out = model.training_step_end(all_results)
 
-to illustrate why this is needed, let's look at dataparallel
+to illustrate why this is needed, let's look at DataParallel
 
 .. testcode::
 
@@ -332,6 +412,30 @@ Validation and test step also have the same option when using dp
     def test_step_end(self, batch_parts_outputs):
         ...
 
+
+Distributed and 16-bit precision
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
+not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.
+
+Below are the possible configurations we support.
+
++-------+---------+----+-----+---------+------------------------------------------------------------+
+| 1 GPU | 1+ GPUs | DP | DDP | 16-bit  | command                                                    |
++=======+=========+====+=====+=========+============================================================+
+| Y     |         |    |     |         | `Trainer(gpus=1)`                                          |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+| Y     |         |    |     | Y       | `Trainer(gpus=1, use_amp=True)`                            |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+|       | Y       | Y  |     |         | `Trainer(gpus=k, distributed_backend='dp')`                |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+|       | Y       |    | Y   |         | `Trainer(gpus=k, distributed_backend='ddp')`               |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+|       | Y       |    | Y   | Y       | `Trainer(gpus=k, distributed_backend='ddp', use_amp=True)` |
++-------+---------+----+-----+---------+------------------------------------------------------------+
+
+
 Implement Your Own Distributed (DDP) training
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 If you need your own way to init PyTorch DDP you can override :meth:`pytorch_lightning.core.LightningModule.`.
@@ -388,7 +492,7 @@ Lightning supports the use of PytorchElastic to enable fault-tolerent and elasti
     Trainer(gpus=8, distributed_backend='ddp')
     
     
-Following the `PytorchElastic Quickstart documentation <https://pytorch.org/elastic/0.2.0/quickstart.html>`_, you then need to start a single-node etcd server on one of the hosts:
+Following the `PytorchElastic Quickstart documentation <https://pytorch.org/elastic/latest/quickstart.html>`_, you then need to start a single-node etcd server on one of the hosts:
 
 .. code-block:: bash
 
@@ -410,5 +514,5 @@ And then launch the elastic job with:
             YOUR_LIGHTNING_TRAINING_SCRIPT.py (--arg1 ... train script args...)
             
 
-See the official `PytorchElastic documentation <https://pytorch.org/elastic/0.2.0/index.html>`_ for details
+See the official `PytorchElastic documentation <https://pytorch.org/elastic>`_ for details
 on installation and more use cases.

docs/source/new-project.rst

@@ -189,7 +189,7 @@ However, this time you need to specifically call test (this is done so you don't
     # OPTION 2:
     # test after loading weights
     model = LitModel.load_from_checkpoint(PATH)
-    trainer = Trainer(num_tpu_cores=1)
+    trainer = Trainer(tpu_cores=1)
     trainer.test()
 
 Again, under the hood, lightning does the following in (pseudocode):
@@ -236,7 +236,7 @@ Without changing a SINGLE line of your code, you can now do the following with t
     # train on TPUs using 16 bit precision with early stopping
     # using only half the training data and checking validation every quarter of a training epoch
     trainer = Trainer(
-        nb_tpu_cores=8,
+        tpu_cores=8,
         precision=16,
         early_stop_checkpoint=True,
         train_percent_check=0.5,

docs/source/optimizers.rst

@@ -17,7 +17,7 @@ Every optimizer you use can be paired with any `LearningRateScheduler <https://p
       scheduler = ReduceLROnPlateau(optimizer, ...)
       return [optimizer], [scheduler]
 
-   # Two optimziers each with a scheduler
+   # Two optimizers each with a scheduler
    def configure_optimizers(self):
       optimizer1 = Adam(...)
       optimizer2 = SGD(...)

docs/source/slurm.rst

@@ -5,15 +5,15 @@
 Computing cluster (SLURM)
 =========================
 
-Lightning automates job the details behind  training on a SLURM powered cluster.
+Lightning automates the details behind training on a SLURM-powered cluster.
 
 .. _multi-node:
 
 Multi-node training
 -------------------
-To train a model using multiple-nodes do the following:
+To train a model using multiple nodes, do the following:
 
-1.  Design your LightningModule.
+1.  Design your :class:`~pytorch_lightning.core.LightningModule`.
 
 2.  Enable ddp in the trainer
 
@@ -22,7 +22,7 @@ To train a model using multiple-nodes do the following:
        # train on 32 GPUs across 4 nodes
        trainer = Trainer(gpus=8, num_nodes=4, distributed_backend='ddp')
 
-3.  It's a good idea to structure your train.py file like this:
+3.  It's a good idea to structure your training script like this:
 
     .. testcode::
 
@@ -47,7 +47,7 @@ To train a model using multiple-nodes do the following:
             # TRAIN
             main(hyperparams)
 
-4.  Create the appropriate SLURM job
+4.  Create the appropriate SLURM job:
 
     .. code-block:: bash
 
@@ -64,10 +64,9 @@ To train a model using multiple-nodes do the following:
         # activate conda env
         source activate $1
 
-        # -------------------------
         # debugging flags (optional)
-         export NCCL_DEBUG=INFO
-         export PYTHONFAULTHANDLER=1
+        export NCCL_DEBUG=INFO
+        export PYTHONFAULTHANDLER=1
 
         # on your cluster you might need these:
         # set the network interface
@@ -75,7 +74,6 @@ To train a model using multiple-nodes do the following:
 
         # might need the latest cuda
         # module load NCCL/2.4.7-1-cuda.10.0
-        # -------------------------
 
         # run script from above
         srun python3 train.py
@@ -92,12 +90,34 @@ To train a model using multiple-nodes do the following:
 
         sbatch submit.sh
 
-.. note:: using :class:`~torch.utils.data.distributed.DistributedSampler` is already handled by Lightning.
+.. note::
+    When running in DDP mode, any errors in your code will show up as an NCCL issue.
+    Set the `NCCL_DEBUG=INFO` flag to see the ACTUAL error.
 
-Walltime auto-resubmit
-----------------------
-When you use Lightning in a SLURM cluster, lightning automatically detects when it is about
-to run into the walltime, and it does the following:
+
+Normally now you would need to add a
+:class:`~torch.utils.data.distributed.DistributedSampler` to your dataset, however
+Lightning automates this for you. But if you still need to set a sampler set the Trainer flag
+:paramref:`~pytorch_lightning.Trainer.replace_sampler_ddp` to ``False``.
+
+Here's an example of how to add your own sampler (again, not needed with Lightning).
+
+.. testcode::
+
+    # in your LightningModule
+    def train_dataloader(self):
+        dataset = MyDataset()
+        dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+        dataloader = Dataloader(dataset, sampler=dist_sampler)
+        return dataloader
+
+    # in your training script
+    trainer = Trainer(replace_sampler_ddp=False)
+
+Wall time auto-resubmit
+-----------------------
+When you use Lightning in a SLURM cluster, it automatically detects when it is about
+to run into the wall time and does the following:
 
 1.  Saves a temporary checkpoint.
 2.  Requeues the job.
@@ -105,7 +125,76 @@ to run into the walltime, and it does the following:
 
 To get this behavior make sure to add the correct signal to your SLURM script
 
-.. code-block::
+.. code-block:: bash
 
     # 90 seconds before training ends
-    #SBATCH --signal=SIGUSR1@90
+    SBATCH --signal=SIGUSR1@90
+
+
+Building SLURM scripts
+----------------------
+
+Instead of manually building SLURM scripts, you can use the
+`SlurmCluster object <https://williamfalcon.github.io/test-tube/hpc/SlurmCluster>`_
+to do this for you. The SlurmCluster can also run a grid search if you pass
+in a `HyperOptArgumentParser
+<https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser>`_.
+
+Here is an example where you run a grid search of 9 combinations of hyperparameters.
+See also the multi-node examples
+`here <https://github.com/PyTorchLightning/pytorch-lightning/tree/master/pl_examples/basic_examples>`__.
+
+.. code-block:: python
+
+    # grid search 3 values of learning rate and 3 values of number of layers for your net
+    # this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32),
+    # (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
+    parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
+    parser.opt_list('--learning_rate', default=0.001, type=float,
+                    options=[1e-3, 1e-2, 1e-1], tunable=True)
+    parser.opt_list('--layers', default=1, type=float, options=[16, 32, 64], tunable=True)
+    hyperparams = parser.parse_args()
+
+    # Slurm cluster submits 9 jobs, each with a set of hyperparams
+    cluster = SlurmCluster(
+        hyperparam_optimizer=hyperparams,
+        log_path='/some/path/to/save',
+    )
+
+    # OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
+    # which interface your nodes use for communication
+    cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
+
+    # see output of the NCCL connection process
+    # NCCL is how the nodes talk to each other
+    cluster.add_command('export NCCL_DEBUG=INFO')
+
+    # setting a master port here is a good idea.
+    cluster.add_command('export MASTER_PORT=%r' % PORT)
+
+    # ************** DON'T FORGET THIS ***************
+    # MUST load the latest NCCL version
+    cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
+
+    # configure cluster
+    cluster.per_experiment_nb_nodes = 12
+    cluster.per_experiment_nb_gpus = 8
+
+    cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
+
+    # submit a script with 9 combinations of hyper params
+    # (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
+    cluster.optimize_parallel_cluster_gpu(
+        main,
+        nb_trials=9, # how many permutations of the grid search to run
+        job_name='name_for_squeue'
+    )
+
+
+The other option is that you generate scripts on your own via a bash command or use another library.
+
+
+Self-balancing architecture (COMING SOON)
+-----------------------------------------
+
+Here Lightning distributes parts of your module across available GPUs to optimize for speed and memory.

docs/source/test_set.rst

@@ -1,6 +1,6 @@
 Test set
 ========
-Lightning forces the user to run the test set separately to make sure it isn't evaluated by mistake
+Lightning forces the user to run the test set separately to make sure it isn't evaluated by mistake.
 
 
 Test after fit
@@ -15,6 +15,7 @@ To run the test set after training completes, use this method
     # run test set
     trainer.test()
 
+
 Test pre-trained model
 ----------------------
 To run the test set on a pre-trained model, use this method.
@@ -34,4 +35,22 @@ To run the test set on a pre-trained model, use this method.
     trainer.test(model)
 
 In this  case, the options you pass to trainer will be used when
-running the test set (ie: 16-bit, dp, ddp, etc...)
+running the test set (ie: 16-bit, dp, ddp, etc...)
+
+
+Test with additional data loaders
+---------------------------------
+You can still run inference on a test set even if the `test_dataloader` method hasn't been
+defined within your :class:`~pytorch_lightning.core.LightningModule` instance. This would be the case when your test data
+is not available at the time your model was declared.
+
+.. code-block:: python
+
+    # setup your data loader
+    test = DataLoader(...)
+
+    # test (pass in the loader)
+    trainer.test(test_dataloaders=test)
+
+You can either pass in a single dataloader or a list of them. This optional named
+parameter can be used in conjunction with any of the above use cases.

docs/source/tpu.rst

@@ -1,8 +1,8 @@
 TPU support
 ===========
 
-Lightning supports running on TPUs. At this moment, TPUs are only available
-on Google Cloud (GCP). For more information on TPUs
+Lightning supports running on TPUs. At this moment, TPUs are available
+on Google Cloud (GCP), Google Colab and Kaggle Environments. For more information on TPUs
 `watch this video <https://www.youtube.com/watch?v=kPMpmcl_Pyw>`_.
 
 ---------------
@@ -31,6 +31,7 @@ To access TPUs there are two main ways.
 
 1. Using google colab.
 2. Using Google Cloud (GCP).
+3. Using Kaggle.
 
 ---------------
 
@@ -51,50 +52,10 @@ To get a TPU on colab, follow these steps:
 4. Next, insert this code into the first cell and execute.
    This will install the xla library that interfaces between PyTorch and the TPU.
 
-   .. code-block:: python
-
-        import collections
-        from datetime import datetime, timedelta
-        import os
-        import requests
-        import threading
-
-        _VersionConfig = collections.namedtuple('_VersionConfig', 'wheels,server')
-        VERSION = "xrt==1.15.0"  #@param ["xrt==1.15.0", "torch_xla==nightly"]
-        CONFIG = {
-            'xrt==1.15.0': _VersionConfig('1.15', '1.15.0'),
-            'torch_xla==nightly': _VersionConfig('nightly', 'XRT-dev{}'.format(
-                (datetime.today() - timedelta(1)).strftime('%Y%m%d'))),
-        }[VERSION]
-        DIST_BUCKET = 'gs://tpu-pytorch/wheels'
-        TORCH_WHEEL = 'torch-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
-        TORCH_XLA_WHEEL = 'torch_xla-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
-        TORCHVISION_WHEEL = 'torchvision-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
-
-        # Update TPU XRT version
-        def update_server_xrt():
-          print('Updating server-side XRT to {} ...'.format(CONFIG.server))
-          url = 'http://{TPU_ADDRESS}:8475/requestversion/{XRT_VERSION}'.format(
-              TPU_ADDRESS=os.environ['COLAB_TPU_ADDR'].split(':')[0],
-              XRT_VERSION=CONFIG.server,
-          )
-          print('Done updating server-side XRT: {}'.format(requests.post(url)))
-
-        update = threading.Thread(target=update_server_xrt)
-        update.start()
-
    .. code-block::
 
-        # Install Colab TPU compat PyTorch/TPU wheels and dependencies
-        !pip uninstall -y torch torchvision
-        !gsutil cp "$DIST_BUCKET/$TORCH_WHEEL" .
-        !gsutil cp "$DIST_BUCKET/$TORCH_XLA_WHEEL" .
-        !gsutil cp "$DIST_BUCKET/$TORCHVISION_WHEEL" .
-        !pip install "$TORCH_WHEEL"
-        !pip install "$TORCH_XLA_WHEEL"
-        !pip install "$TORCHVISION_WHEEL"
-        !sudo apt-get install libomp5
-        update.join()
+    !curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
+    !python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev
 
 5. Once the above is done, install PyTorch Lightning (v 0.7.0+).
 
@@ -156,13 +117,23 @@ To use a full TPU pod skip to the TPU pod section.
     import pytorch_lightning as pl
 
     my_model = MyLightningModule()
-    trainer = pl.Trainer(num_tpu_cores=8)
+    trainer = pl.Trainer(tpu_cores=8)
     trainer.fit(my_model)
 
 That's it! Your model will train on all 8 TPU cores.
 
 ---------------
 
+Single TPU core training
+----------------------------
+Lightning supports training on a single TPU core. Just pass the TPU core ID [1-8] in a list.
+
+.. code-block:: python
+
+    trainer = pl.Trainer(tpu_cores=[1])
+
+---------------
+
 Distributed Backend with TPU
 ----------------------------
 The ```distributed_backend``` option used for GPUs does not apply to TPUs.
@@ -195,7 +166,7 @@ set the 16-bit flag.
     import pytorch_lightning as pl
 
     my_model = MyLightningModule()
-    trainer = pl.Trainer(num_tpu_cores=8, precision=16)
+    trainer = pl.Trainer(tpu_cores=8, precision=16)
     trainer.fit(my_model)
 
 Under the hood the xla library will use the `bfloat16 type <https://en.wikipedia.org/wiki/Bfloat16_floating-point_format>`_.

docs/source/training_tricks.rst

@@ -39,7 +39,7 @@ Auto scaling of batch size
 --------------------------
 Auto scaling of batch size may be enabled to find the largest batch size that fits into
 memory. Larger batch size often yields better estimates of gradients, but may also result in
-longer training time.
+longer training time. Inspired by https://github.com/BlackHC/toma.
 
 .. seealso:: :class:`~pytorch_lightning.trainer.trainer.Trainer`
 
@@ -67,7 +67,7 @@ a binary search.
     .. code-block:: python
         
         def train_dataloader(self):
-            return DataLoader(train_dataset, batch_size=self.hparams.batch_size)
+            return DataLoader(train_dataset, batch_size=self.batch_size)
 
 .. warning::
             

docs/source/weights_loading.rst

@@ -59,24 +59,20 @@ Or disable it by passing
    trainer = Trainer(checkpoint_callback=False)
 
 
-The Lightning checkpoint also saves the hparams (hyperparams) passed into the LightningModule init.
+The Lightning checkpoint also saves the arguments passed into the LightningModule init
+under the `module_arguments` key in the checkpoint.
 
-.. note:: hparams is a `Namespace <https://docs.python.org/2/library/argparse.html#argparse.Namespace>`_.
+.. code-block:: python
 
-.. testcode::
+    class MyLightningModule(LightningModule):
 
-   from argparse import Namespace
+       def __init__(self, learning_rate, *args, **kwargs):
+            super().__init__()
 
-   # usually these come from command line args
-   args = Namespace(learning_rate=0.001)
-
-   # define you module to have hparams as the first arg
-   # this means your checkpoint will have everything that went into making
-   # this model (in this case, learning rate)
-   class MyLightningModule(LightningModule):
-
-       def __init__(self, hparams, *args, **kwargs):
-           self.hparams = hparams
+    # all init args were saved to the checkpoint
+    checkpoint = torch.load(CKPT_PATH)
+    print(checkpoint['module_arguments'])
+    # {'learning_rate': the_value}
 
 Manual saving
 ^^^^^^^^^^^^^
@@ -92,37 +88,42 @@ You can manually save checkpoints and restore your model from the checkpointed s
 Checkpoint Loading
 ------------------
 
-To load a model along with its weights, biases and hyperparameters use following method.
+To load a model along with its weights, biases and `module_arguments` use following method.
 
 .. code-block:: python
 
     model = MyLightingModule.load_from_checkpoint(PATH)
+
+    print(model.learning_rate)
+    # prints the learning_rate you used in this checkpoint
+
     model.eval()
     y_hat = model(x)
 
-The above only works if you used `hparams` in your model definition
-
-.. testcode::
-
-    class LitModel(LightningModule):
-
-        def __init__(self, hparams):
-            self.hparams = hparams
-            self.l1 = nn.Linear(hparams.in_dim, hparams.out_dim)
-
-But if you don't and instead pass individual parameters
+But if you don't want to use the values saved in the checkpoint, pass in your own here
 
 .. testcode::
 
     class LitModel(LightningModule):
 
         def __init__(self, in_dim, out_dim):
-            self.l1 = nn.Linear(in_dim, out_dim)
+            super().__init__()
+            self.in_dim = in_dim
+            self.out_dim = out_dim
+            self.l1 = nn.Linear(self.in_dim, self.out_dim)
 
 you can restore the model like this
 
 .. code-block:: python
 
+    # if you train and save the model like this it will use these values when loading
+    # the weights. But you can overwrite this
+    LitModel(in_dim=32, out_dim=10)
+
+    # uses in_dim=32, out_dim=10
+    model = LitModel.load_from_checkpoint(PATH)
+
+    # uses in_dim=128, out_dim=10
     model = LitModel.load_from_checkpoint(PATH, in_dim=128, out_dim=10)
 
 

environment.yml

@@ -10,7 +10,7 @@ dependencies:
     - pip==20.0.2
     - tqdm>=4.35.0
     - numpy>=1.16.4
-    - pytorch>=1.1
+    - pytorch>=1.3
     - tensorboard>=1.14
     - future>=0.17.1
     - pyyaml>=3.13

pl_examples/basic_examples/cpu_template.py

@@ -10,25 +10,23 @@ import torch
 import pytorch_lightning as pl
 from pl_examples.models.lightning_template import LightningTemplateModel
 
-SEED = 2334
-torch.manual_seed(SEED)
-np.random.seed(SEED)
+pl.seed_everything(234)
 
 
-def main(hparams):
+def main(args):
     """
     Main training routine specific for this project
-    :param hparams:
+    :param args:
     """
     # ------------------------
     # 1 INIT LIGHTNING MODEL
     # ------------------------
-    model = LightningTemplateModel(hparams)
+    model = LightningTemplateModel(**vars(args))
 
     # ------------------------
     # 2 INIT TRAINER
     # ------------------------
-    trainer = pl.Trainer(max_epochs=hparams.epochs, overfit_pct=0.01, early_stop_callback=True)
+    trainer = pl.Trainer.from_argparse_args(args)
 
     # ------------------------
     # 3 START TRAINING
@@ -46,9 +44,10 @@ if __name__ == '__main__':
 
     # each LightningModule defines arguments relevant to it
     parser = LightningTemplateModel.add_model_specific_args(parent_parser, root_dir)
-    hyperparams = parser.parse_args()
+    parser = pl.Trainer.add_argparse_args(parser)
+    args = parser.parse_args()
 
     # ---------------------
     # RUN TRAINING
     # ---------------------
-    main(hyperparams)
+    main(args)

pl_examples/domain_templates/computer_vision_fine_tuning.py

@@ -148,10 +148,24 @@ class TransferLearningModel(pl.LightningModule):
         dl_path: Path where the data will be downloaded
     """
     def __init__(self,
-                 hparams: argparse.Namespace,
-                 dl_path: Union[str, Path]) -> None:
+                 dl_path: Union[str, Path],
+                 backbone: str = 'resnet50',
+                 train_bn: bool = True,
+                 milestones: tuple = (5, 10),
+                 batch_size: int = 8,
+                 lr: float = 1e-2,
+                 lr_scheduler_gamma: float = 1e-1,
+                 num_workers: int = 6, **kwargs) -> None:
         super().__init__()
-        self.hparams = hparams
+        self.dl_path = dl_path
+        self.backbone = backbone
+        self.train_bn = train_bn
+        self.milestones = milestones
+        self.batch_size = batch_size
+        self.lr = lr
+        self.lr_scheduler_gamma = lr_scheduler_gamma
+        self.num_workers = num_workers
+
         self.dl_path = dl_path
         self.__build_model()
 
@@ -159,12 +173,12 @@ class TransferLearningModel(pl.LightningModule):
         """Define model layers & loss."""
 
         # 1. Load pre-trained network:
-        model_func = getattr(models, self.hparams.backbone)
+        model_func = getattr(models, self.backbone)
         backbone = model_func(pretrained=True)
 
         _layers = list(backbone.children())[:-1]
         self.feature_extractor = torch.nn.Sequential(*_layers)
-        freeze(module=self.feature_extractor, train_bn=self.hparams.train_bn)
+        freeze(module=self.feature_extractor, train_bn=self.train_bn)
 
         # 2. Classifier:
         _fc_layers = [torch.nn.Linear(2048, 256),
@@ -194,29 +208,29 @@ class TransferLearningModel(pl.LightningModule):
         super().train(mode=mode)
 
         epoch = self.current_epoch
-        if epoch < self.hparams.milestones[0] and mode:
+        if epoch < self.milestones[0] and mode:
             # feature extractor is frozen (except for BatchNorm layers)
             freeze(module=self.feature_extractor,
-                   train_bn=self.hparams.train_bn)
+                   train_bn=self.train_bn)
 
-        elif self.hparams.milestones[0] <= epoch < self.hparams.milestones[1] and mode:
+        elif self.milestones[0] <= epoch < self.milestones[1] and mode:
             # Unfreeze last two layers of the feature extractor
             freeze(module=self.feature_extractor,
                    n=-2,
-                   train_bn=self.hparams.train_bn)
+                   train_bn=self.train_bn)
 
     def on_epoch_start(self):
         """Use `on_epoch_start` to unfreeze layers progressively."""
         optimizer = self.trainer.optimizers[0]
-        if self.current_epoch == self.hparams.milestones[0]:
+        if self.current_epoch == self.milestones[0]:
             _unfreeze_and_add_param_group(module=self.feature_extractor[-2:],
                                           optimizer=optimizer,
-                                          train_bn=self.hparams.train_bn)
+                                          train_bn=self.train_bn)
 
-        elif self.current_epoch == self.hparams.milestones[1]:
+        elif self.current_epoch == self.milestones[1]:
             _unfreeze_and_add_param_group(module=self.feature_extractor[:-2],
                                           optimizer=optimizer,
-                                          train_bn=self.hparams.train_bn)
+                                          train_bn=self.train_bn)
 
     def training_step(self, batch, batch_idx):
 
@@ -246,7 +260,7 @@ class TransferLearningModel(pl.LightningModule):
                                        for output in outputs]).mean()
         train_acc_mean = torch.stack([output['num_correct']
                                       for output in outputs]).sum().float()
-        train_acc_mean /= (len(outputs) * self.hparams.batch_size)
+        train_acc_mean /= (len(outputs) * self.batch_size)
         return {'log': {'train_loss': train_loss_mean,
                         'train_acc': train_acc_mean,
                         'step': self.current_epoch}}
@@ -273,7 +287,7 @@ class TransferLearningModel(pl.LightningModule):
                                      for output in outputs]).mean()
         val_acc_mean = torch.stack([output['num_correct']
                                     for output in outputs]).sum().float()
-        val_acc_mean /= (len(outputs) * self.hparams.batch_size)
+        val_acc_mean /= (len(outputs) * self.batch_size)
         return {'log': {'val_loss': val_loss_mean,
                         'val_acc': val_acc_mean,
                         'step': self.current_epoch}}
@@ -281,11 +295,11 @@ class TransferLearningModel(pl.LightningModule):
     def configure_optimizers(self):
         optimizer = optim.Adam(filter(lambda p: p.requires_grad,
                                       self.parameters()),
-                               lr=self.hparams.lr)
+                               lr=self.lr)
 
         scheduler = MultiStepLR(optimizer,
-                                milestones=self.hparams.milestones,
-                                gamma=self.hparams.lr_scheduler_gamma)
+                                milestones=self.milestones,
+                                gamma=self.lr_scheduler_gamma)
 
         return [optimizer], [scheduler]
 
@@ -326,8 +340,8 @@ class TransferLearningModel(pl.LightningModule):
 
         _dataset = self.train_dataset if train else self.valid_dataset
         loader = DataLoader(dataset=_dataset,
-                            batch_size=self.hparams.batch_size,
-                            num_workers=self.hparams.num_workers,
+                            batch_size=self.batch_size,
+                            num_workers=self.num_workers,
                             shuffle=True if train else False)
 
         return loader
@@ -397,28 +411,28 @@ class TransferLearningModel(pl.LightningModule):
         return parser
 
 
-def main(hparams: argparse.Namespace) -> None:
+def main(args: argparse.Namespace) -> None:
     """Train the model.
 
     Args:
-        hparams: Model hyper-parameters
+        args: Model hyper-parameters
 
     Note:
         For the sake of the example, the images dataset will be downloaded
         to a temporary directory.
     """
 
-    with TemporaryDirectory(dir=hparams.root_data_path) as tmp_dir:
+    with TemporaryDirectory(dir=args.root_data_path) as tmp_dir:
 
-        model = TransferLearningModel(hparams, dl_path=tmp_dir)
+        model = TransferLearningModel(dl_path=tmp_dir, **vars(args))
 
         trainer = pl.Trainer(
             weights_summary=None,
             show_progress_bar=True,
             num_sanity_val_steps=0,
-            gpus=hparams.gpus,
-            min_epochs=hparams.nb_epochs,
-            max_epochs=hparams.nb_epochs)
+            gpus=args.gpus,
+            min_epochs=args.nb_epochs,
+            max_epochs=args.nb_epochs)
 
         trainer.fit(model)
 
@@ -436,5 +450,4 @@ def get_args() -> argparse.Namespace:
 
 
 if __name__ == '__main__':
-
     main(get_args())

pl_examples/domain_templates/generative_adversarial_net.py

@@ -7,7 +7,7 @@ After a few epochs, launch TensorBoard to see the images being generated at ever
 tensorboard --logdir default
 """
 import os
-from argparse import ArgumentParser
+from argparse import ArgumentParser, Namespace
 from collections import OrderedDict
 
 import numpy as np
@@ -72,18 +72,26 @@ class Discriminator(nn.Module):
 
 class GAN(LightningModule):
 
-    def __init__(self, hparams):
+    def __init__(self,
+                 latent_dim: int = 100,
+                 lr: float = 0.0002,
+                 b1: float = 0.5,
+                 b2: float = 0.999,
+                 batch_size: int = 64, **kwargs):
         super().__init__()
-        self.hparams = hparams
+
+        self.latent_dim = latent_dim
+        self.lr = lr
+        self.b1 = b1
+        self.b2 = b2
+        self.batch_size = batch_size
 
         # networks
         mnist_shape = (1, 28, 28)
-        self.generator = Generator(latent_dim=hparams.latent_dim, img_shape=mnist_shape)
+        self.generator = Generator(latent_dim=self.latent_dim, img_shape=mnist_shape)
         self.discriminator = Discriminator(img_shape=mnist_shape)
 
-        # cache for generated images
-        self.generated_imgs = None
-        self.last_imgs = None
+        self.validation_z = torch.randn(8, self.latent_dim)
 
     def forward(self, z):
         return self.generator(z)
@@ -93,21 +101,21 @@ class GAN(LightningModule):
 
     def training_step(self, batch, batch_idx, optimizer_idx):
         imgs, _ = batch
-        self.last_imgs = imgs
+
+        # sample noise
+        z = torch.randn(imgs.shape[0], self.latent_dim)
+        z = z.type_as(imgs)
 
         # train generator
         if optimizer_idx == 0:
-            # sample noise
-            z = torch.randn(imgs.shape[0], self.hparams.latent_dim)
-            z = z.type_as(imgs)
 
             # generate images
             self.generated_imgs = self(z)
 
             # log sampled images
-            # sample_imgs = self.generated_imgs[:6]
-            # grid = torchvision.utils.make_grid(sample_imgs)
-            # self.logger.experiment.add_image('generated_images', grid, 0)
+            sample_imgs = self.generated_imgs[:6]
+            grid = torchvision.utils.make_grid(sample_imgs)
+            self.logger.experiment.add_image('generated_images', grid, 0)
 
             # ground truth result (ie: all fake)
             # put on GPU because we created this tensor inside training_loop
@@ -115,7 +123,7 @@ class GAN(LightningModule):
             valid = valid.type_as(imgs)
 
             # adversarial loss is binary cross-entropy
-            g_loss = self.adversarial_loss(self.discriminator(self.generated_imgs), valid)
+            g_loss = self.adversarial_loss(self.discriminator(self(z)), valid)
             tqdm_dict = {'g_loss': g_loss}
             output = OrderedDict({
                 'loss': g_loss,
@@ -136,10 +144,10 @@ class GAN(LightningModule):
 
             # how well can it label as fake?
             fake = torch.zeros(imgs.size(0), 1)
-            fake = fake.type_as(fake)
+            fake = fake.type_as(imgs)
 
             fake_loss = self.adversarial_loss(
-                self.discriminator(self.generated_imgs.detach()), fake)
+                self.discriminator(self(z).detach()), fake)
 
             # discriminator loss is the average of these
             d_loss = (real_loss + fake_loss) / 2
@@ -152,9 +160,9 @@ class GAN(LightningModule):
             return output
 
     def configure_optimizers(self):
-        lr = self.hparams.lr
-        b1 = self.hparams.b1
-        b2 = self.hparams.b2
+        lr = self.lr
+        b1 = self.b1
+        b2 = self.b2
 
         opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
         opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))
@@ -164,11 +172,10 @@ class GAN(LightningModule):
         transform = transforms.Compose([transforms.ToTensor(),
                                         transforms.Normalize([0.5], [0.5])])
         dataset = MNIST(os.getcwd(), train=True, download=True, transform=transform)
-        return DataLoader(dataset, batch_size=self.hparams.batch_size)
+        return DataLoader(dataset, batch_size=self.batch_size)
 
     def on_epoch_end(self):
-        z = torch.randn(8, self.hparams.latent_dim)
-        z = z.type_as(self.last_imgs)
+        z = self.validation_z.type_as(self.generator.model[0].weight)
 
         # log sampled images
         sample_imgs = self(z)
@@ -176,15 +183,17 @@ class GAN(LightningModule):
         self.logger.experiment.add_image('generated_images', grid, self.current_epoch)
 
 
-def main(hparams):
+def main(args: Namespace) -> None:
     # ------------------------
     # 1 INIT LIGHTNING MODEL
     # ------------------------
-    model = GAN(hparams)
+    model = GAN(**vars(args))
 
     # ------------------------
     # 2 INIT TRAINER
     # ------------------------
+    # If use distubuted training  PyTorch recommends to use DistributedDataParallel.
+    # See: https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel
     trainer = Trainer()
 
     # ------------------------

pl_examples/domain_templates/imagenet.py

@@ -1,7 +1,7 @@
 """
 This example is largely adapted from https://github.com/pytorch/examples/blob/master/imagenet/main.py
 """
-import argparse
+from argparse import ArgumentParser, Namespace
 import os
 import random
 from collections import OrderedDict
@@ -29,13 +29,26 @@ MODEL_NAMES = sorted(
 
 
 class ImageNetLightningModel(LightningModule):
-    def __init__(self, hparams):
+    def __init__(self,
+                 arch,
+                 pretrained,
+                 lr: float,
+                 momentum: float,
+                 weight_decay: int,
+                 data_path: str,
+                 batch_size: int, **kwargs):
         """
         TODO: add docstring here
         """
         super().__init__()
-        self.hparams = hparams
-        self.model = models.__dict__[self.hparams.arch](pretrained=self.hparams.pretrained)
+        self.arch = arch
+        self.pretrained = pretrained
+        self.lr = lr
+        self.momentum = momentum
+        self.weight_decay = weight_decay
+        self.data_path = data_path
+        self.batch_size = batch_size
+        self.model = models.__dict__[self.arch](pretrained=self.pretrained)
 
     def forward(self, x):
         return self.model(x)
@@ -112,9 +125,9 @@ class ImageNetLightningModel(LightningModule):
     def configure_optimizers(self):
         optimizer = optim.SGD(
             self.parameters(),
-            lr=self.hparams.lr,
-            momentum=self.hparams.momentum,
-            weight_decay=self.hparams.weight_decay
+            lr=self.lr,
+            momentum=self.momentum,
+            weight_decay=self.weight_decay
         )
         scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.1)
         return [optimizer], [scheduler]
@@ -125,7 +138,7 @@ class ImageNetLightningModel(LightningModule):
             std=[0.229, 0.224, 0.225],
         )
 
-        train_dir = os.path.join(self.hparams.data_path, 'train')
+        train_dir = os.path.join(self.data_path, 'train')
         train_dataset = datasets.ImageFolder(
             train_dir,
             transforms.Compose([
@@ -142,7 +155,7 @@ class ImageNetLightningModel(LightningModule):
 
         train_loader = torch.utils.data.DataLoader(
             dataset=train_dataset,
-            batch_size=self.hparams.batch_size,
+            batch_size=self.batch_size,
             shuffle=(train_sampler is None),
             num_workers=0,
             sampler=train_sampler
@@ -154,7 +167,7 @@ class ImageNetLightningModel(LightningModule):
             mean=[0.485, 0.456, 0.406],
             std=[0.229, 0.224, 0.225],
         )
-        val_dir = os.path.join(self.hparams.data_path, 'val')
+        val_dir = os.path.join(self.data_path, 'val')
         val_loader = torch.utils.data.DataLoader(
             datasets.ImageFolder(val_dir, transforms.Compose([
                 transforms.Resize(256),
@@ -162,7 +175,7 @@ class ImageNetLightningModel(LightningModule):
                 transforms.ToTensor(),
                 normalize,
             ])),
-            batch_size=self.hparams.batch_size,
+            batch_size=self.batch_size,
             shuffle=False,
             num_workers=0,
         )
@@ -170,7 +183,7 @@ class ImageNetLightningModel(LightningModule):
 
     @staticmethod
     def add_model_specific_args(parent_parser):  # pragma: no-cover
-        parser = argparse.ArgumentParser(parents=[parent_parser])
+        parser = ArgumentParser(parents=[parent_parser])
         parser.add_argument('-a', '--arch', metavar='ARCH', default='resnet18', choices=MODEL_NAMES,
                             help='model architecture: ' +
                                  ' | '.join(MODEL_NAMES) +
@@ -197,7 +210,7 @@ class ImageNetLightningModel(LightningModule):
 
 
 def get_args():
-    parent_parser = argparse.ArgumentParser(add_help=False)
+    parent_parser = ArgumentParser(add_help=False)
     parent_parser.add_argument('--data-path', metavar='DIR', type=str,
                                help='path to dataset')
     parent_parser.add_argument('--save-path', metavar='DIR', default=".", type=str,
@@ -215,20 +228,23 @@ def get_args():
     return parser.parse_args()
 
 
-def main(hparams):
-    model = ImageNetLightningModel(hparams)
-    if hparams.seed is not None:
-        random.seed(hparams.seed)
-        torch.manual_seed(hparams.seed)
+def main(args: Namespace) -> None:
+    model = ImageNetLightningModel(**vars(args))
+
+    if args.seed is not None:
+        random.seed(args.seed)
+        torch.manual_seed(args.seed)
         cudnn.deterministic = True
+
     trainer = pl.Trainer(
-        default_root_dir=hparams.save_path,
-        gpus=hparams.gpus,
-        max_epochs=hparams.epochs,
-        distributed_backend=hparams.distributed_backend,
-        precision=16 if hparams.use_16bit else 32,
+        default_root_dir=args.save_path,
+        gpus=args.gpus,
+        max_epochs=args.epochs,
+        distributed_backend=args.distributed_backend,
+        precision=16 if args.use_16bit else 32,
     )
-    if hparams.evaluate:
+
+    if args.evaluate:
         trainer.run_evaluation()
     else:
         trainer.fit(model)

pl_examples/domain_templates/reinforce_learn_Qnet.py

@@ -190,22 +190,41 @@ class Agent:
 class DQNLightning(pl.LightningModule):
     """ Basic DQN Model """
 
-    def __init__(self, hparams: argparse.Namespace) -> None:
+    def __init__(self,
+                 replay_size,
+                 warm_start_steps: int,
+                 gamma: float,
+                 eps_start: int,
+                 eps_end: int,
+                 eps_last_frame: int,
+                 sync_rate,
+                 lr: float,
+                 episode_length,
+                 batch_size, **kwargs) -> None:
         super().__init__()
-        self.hparams = hparams
+        self.replay_size = replay_size
+        self.warm_start_steps = warm_start_steps
+        self.gamma = gamma
+        self.eps_start = eps_start
+        self.eps_end = eps_end
+        self.eps_last_frame = eps_last_frame
+        self.sync_rate = sync_rate
+        self.lr = lr
+        self.episode_length = episode_length
+        self.batch_size = batch_size
 
-        self.env = gym.make(self.hparams.env)
+        self.env = gym.make(self.env)
         obs_size = self.env.observation_space.shape[0]
         n_actions = self.env.action_space.n
 
         self.net = DQN(obs_size, n_actions)
         self.target_net = DQN(obs_size, n_actions)
 
-        self.buffer = ReplayBuffer(self.hparams.replay_size)
+        self.buffer = ReplayBuffer(self.replay_size)
         self.agent = Agent(self.env, self.buffer)
         self.total_reward = 0
         self.episode_reward = 0
-        self.populate(self.hparams.warm_start_steps)
+        self.populate(self.warm_start_steps)
 
     def populate(self, steps: int = 1000) -> None:
         """
@@ -250,7 +269,7 @@ class DQNLightning(pl.LightningModule):
             next_state_values[dones] = 0.0
             next_state_values = next_state_values.detach()
 
-        expected_state_action_values = next_state_values * self.hparams.gamma + rewards
+        expected_state_action_values = next_state_values * self.gamma + rewards
 
         return nn.MSELoss()(state_action_values, expected_state_action_values)
 
@@ -267,8 +286,8 @@ class DQNLightning(pl.LightningModule):
             Training loss and log metrics
         """
         device = self.get_device(batch)
-        epsilon = max(self.hparams.eps_end, self.hparams.eps_start -
-                      self.global_step + 1 / self.hparams.eps_last_frame)
+        epsilon = max(self.eps_end, self.eps_start -
+                      self.global_step + 1 / self.eps_last_frame)
 
         # step through environment with agent
         reward, done = self.agent.play_step(self.net, epsilon, device)
@@ -282,7 +301,7 @@ class DQNLightning(pl.LightningModule):
             self.episode_reward = 0
 
         # Soft update of target network
-        if self.global_step % self.hparams.sync_rate == 0:
+        if self.global_step % self.sync_rate == 0:
             self.target_net.load_state_dict(self.net.state_dict())
 
         log = {'total_reward': torch.tensor(self.total_reward).to(device),
@@ -293,16 +312,17 @@ class DQNLightning(pl.LightningModule):
 
     def configure_optimizers(self) -> List[Optimizer]:
         """Initialize Adam optimizer"""
-        optimizer = optim.Adam(self.net.parameters(), lr=self.hparams.lr)
+        optimizer = optim.Adam(self.net.parameters(), lr=self.lr)
         return [optimizer]
 
     def __dataloader(self) -> DataLoader:
         """Initialize the Replay Buffer dataset used for retrieving experiences"""
-        dataset = RLDataset(self.buffer, self.hparams.episode_length)
-        dataloader = DataLoader(dataset=dataset,
-                                batch_size=self.hparams.batch_size,
-                                sampler=None
-                                )
+        dataset = RLDataset(self.buffer, self.episode_length)
+        dataloader = DataLoader(
+            dataset=dataset,
+            batch_size=self.batch_size,
+            sampler=None,
+        )
         return dataloader
 
     def train_dataloader(self) -> DataLoader:
@@ -314,8 +334,8 @@ class DQNLightning(pl.LightningModule):
         return batch[0].device.index if self.on_gpu else 'cpu'
 
 
-def main(hparams) -> None:
-    model = DQNLightning(hparams)
+def main(args) -> None:
+    model = DQNLightning(**vars(args))
 
     trainer = pl.Trainer(
         gpus=1,

pl_examples/domain_templates/semantic_segmentation.py

@@ -1,5 +1,5 @@
 import os
-from argparse import ArgumentParser
+from argparse import ArgumentParser, Namespace
 
 import numpy as np
 import torch
@@ -128,14 +128,23 @@ class SegModel(pl.LightningModule):
     Adam optimizer is used along with Cosine Annealing learning rate scheduler.
     """
 
-    def __init__(self, hparams):
+    def __init__(self,
+                 data_path: str,
+                 batch_size: int,
+                 lr: float,
+                 num_layers: int,
+                 features_start: int,
+                 bilinear: bool, **kwargs):
         super().__init__()
-        self.hparams = hparams
-        self.data_path = hparams.data_path
-        self.batch_size = hparams.batch_size
-        self.learning_rate = hparams.lr
-        self.net = UNet(num_classes=19, num_layers=hparams.num_layers,
-                        features_start=hparams.features_start, bilinear=hparams.bilinear)
+        self.data_path = data_path
+        self.batch_size = batch_size
+        self.lr = lr
+        self.num_layers = num_layers
+        self.features_start = features_start
+        self.bilinear = bilinear
+
+        self.net = UNet(num_classes=19, num_layers=self.num_layers,
+                        features_start=self.features_start, bilinear=self.bilinear)
         self.transform = transforms.Compose([
             transforms.ToTensor(),
             transforms.Normalize(mean=[0.35675976, 0.37380189, 0.3764753],
@@ -181,11 +190,11 @@ class SegModel(pl.LightningModule):
         return DataLoader(self.validset, batch_size=self.batch_size, shuffle=False)
 
 
-def main(hparams):
+def main(hparams: Namespace):
     # ------------------------
     # 1 INIT LIGHTNING MODEL
     # ------------------------
-    model = SegModel(hparams)
+    model = SegModel(**vars(hparams))
 
     # ------------------------
     # 2 SET LOGGER

pl_examples/models/lightning_template.py

@@ -3,7 +3,6 @@ Example template for defining a system.
 """
 import os
 from argparse import ArgumentParser
-from collections import OrderedDict
 
 import torch
 import torch.nn as nn
@@ -34,25 +33,38 @@ class LightningTemplateModel(LightningModule):
         ...     out_features=10,
         ...     hidden_dim=1000,
         ... )
-        >>> from argparse import Namespace
-        >>> hparams = Namespace(**params)
-        >>> model = LightningTemplateModel(hparams)
+        >>> model = LightningTemplateModel(**params)
     """
 
-    def __init__(self, hparams):
-        """
-        Pass in hyperparameters as a `argparse.Namespace` or a `dict` to the model.
-        """
+    def __init__(self,
+                 drop_prob: float = 0.2,
+                 batch_size: int = 2,
+                 in_features: int = 28 * 28,
+                 learning_rate: float = 0.001 * 8,
+                 optimizer_name: str = 'adam',
+                 data_root: str = './datasets',
+                 out_features: int = 10,
+                 hidden_dim: int = 1000,
+                 **kwargs
+                 ) -> 'LightningTemplateModel':
         # init superclass
         super().__init__()
-        self.hparams = hparams
-        self.c_d1 = nn.Linear(in_features=self.hparams.in_features,
-                              out_features=self.hparams.hidden_dim)
-        self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
-        self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)
+        self.drop_prob = drop_prob
+        self.batch_size = batch_size
+        self.in_features = in_features
+        self.learning_rate = learning_rate
+        self.optimizer_name = optimizer_name
+        self.data_root = data_root
+        self.out_features = out_features
+        self.hidden_dim = hidden_dim
 
-        self.c_d2 = nn.Linear(in_features=self.hparams.hidden_dim,
-                              out_features=self.hparams.out_features)
+        self.c_d1 = nn.Linear(in_features=self.in_features,
+                              out_features=self.hidden_dim)
+        self.c_d1_bn = nn.BatchNorm1d(self.hidden_dim)
+        self.c_d1_drop = nn.Dropout(self.drop_prob)
+
+        self.c_d2 = nn.Linear(in_features=self.hidden_dim,
+                              out_features=self.out_features)
 
     def forward(self, x):
         """
@@ -122,32 +134,32 @@ class LightningTemplateModel(LightningModule):
         Return whatever optimizers and learning rate schedulers you want here.
         At least one optimizer is required.
         """
-        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer = optim.Adam(self.parameters(), lr=self.learning_rate)
         scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)
         return [optimizer], [scheduler]
 
     def prepare_data(self):
         transform = transforms.Compose([transforms.ToTensor(),
                                         transforms.Normalize((0.5,), (1.0,))])
-        self.mnist_train = MNIST(self.hparams.data_root, train=True, download=True, transform=transform)
-        self.mnist_test = MNIST(self.hparams.data_root, train=False, download=True, transform=transform)
+        self.mnist_train = MNIST(self.data_root, train=True, download=True, transform=transform)
+        self.mnist_test = MNIST(self.data_root, train=False, download=True, transform=transform)
 
     def train_dataloader(self):
         log.info('Training data loader called.')
-        return DataLoader(self.mnist_train, batch_size=self.hparams.batch_size, num_workers=4)
+        return DataLoader(self.mnist_train, batch_size=self.batch_size, num_workers=4)
 
     def val_dataloader(self):
         log.info('Validation data loader called.')
-        return DataLoader(self.mnist_test, batch_size=self.hparams.batch_size, num_workers=4)
+        return DataLoader(self.mnist_test, batch_size=self.batch_size, num_workers=4)
 
     def test_dataloader(self):
         log.info('Test data loader called.')
-        return DataLoader(self.mnist_test, batch_size=self.hparams.batch_size, num_workers=4)
+        return DataLoader(self.mnist_test, batch_size=self.batch_size, num_workers=4)
 
     @staticmethod
     def add_model_specific_args(parent_parser, root_dir):  # pragma: no-cover
         """
-        Parameters you define here will be available to your model through `self.hparams`.
+        Define parameters that only apply to this model
         """
         parser = ArgumentParser(parents=[parent_parser])
 

pl_examples/models/unet.py

@@ -99,7 +99,10 @@ class Up(nn.Module):
         super().__init__()
         self.upsample = None
         if bilinear:
-            self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
+            self.upsample = nn.Sequential(
+                nn.Upsample(scale_factor=2, mode="bilinear", align_corners=True),
+                nn.Conv2d(in_ch, in_ch // 2, kernel_size=1),
+            )
         else:
             self.upsample = nn.ConvTranspose2d(in_ch, in_ch // 2, kernel_size=2, stride=2)
 

pytorch_lightning/__init__.py

@@ -1,6 +1,6 @@
 """Root package info."""
 
-__version__ = '0.7.6'
+__version__ = '0.8.0-dev'
 __author__ = 'William Falcon et al.'
 __author_email__ = 'waf2107@columbia.edu'
 __license__ = 'Apache-2.0'
@@ -60,8 +60,8 @@ else:
         'Trainer',
         'LightningModule',
         'Callback',
-        'data_loader'
-        'seed_everything'
+        'data_loader',
+        'seed_everything',
     ]
 
     # necessary for regular bolts imports. Skip exception since bolts is not always installed

pytorch_lightning/callbacks/early_stopping.py

@@ -2,7 +2,7 @@ r"""
 Early Stopping
 ==============
 
-Stop training when a monitored quantity has stopped improving.
+Monitor a validation metric and stop training when it stops improving.
 
 """
 
@@ -25,7 +25,7 @@ class EarlyStopping(Callback):
             to qualify as an improvement, i.e. an absolute
             change of less than `min_delta`, will count as no
             improvement. Default: ``0``.
-        patience: number of epochs with no improvement
+        patience: number of validation epochs with no improvement
             after which training will be stopped. Default: ``0``.
         verbose: verbosity mode. Default: ``False``.
         mode: one of {auto, min, max}. In `min` mode,
@@ -36,7 +36,7 @@ class EarlyStopping(Callback):
             mode, the direction is automatically inferred
             from the name of the monitored quantity. Default: ``'auto'``.
         strict: whether to crash the training if `monitor` is
-            not found in the metrics. Default: ``True``.
+            not found in the validation metrics. Default: ``True``.
 
     Example::
 
@@ -75,7 +75,7 @@ class EarlyStopping(Callback):
             if self.verbose > 0:
                 log.info(f'EarlyStopping mode set to {self.mode} for monitoring {self.monitor}.')
 
-        self.min_delta *= 1 if self.monitor_op == torch.gt else -1
+        self.min_delta *= 1 if self.mode == 'min' else -1
 
     def _validate_condition_metric(self, logs):
         """
@@ -109,7 +109,10 @@ class EarlyStopping(Callback):
         self.stopped_epoch = 0
         self.best = torch_inf if self.monitor_op == torch.lt else -torch_inf
 
-    def on_epoch_end(self, trainer, pl_module):
+    def on_validation_end(self, trainer, pl_module):
+        self._run_early_stopping_check(trainer, pl_module)
+
+    def _run_early_stopping_check(self, trainer, pl_module):
         logs = trainer.callback_metrics
         stop_training = False
         if not self._validate_condition_metric(logs):

pytorch_lightning/callbacks/lr_logger.py

@@ -10,6 +10,8 @@ Log learning rate for lr schedulers during training
 from pytorch_lightning.callbacks.base import Callback
 from pytorch_lightning.utilities.exceptions import MisconfigurationException
 
+from pytorch_lightning.utilities import rank_zero_warn
+
 
 class LearningRateLogger(Callback):
     r"""
@@ -45,21 +47,22 @@ class LearningRateLogger(Callback):
             schedulers in the case of multiple of the same type or in
             the case of multiple parameter groups
         """
-        if trainer.lr_schedulers == []:
-            raise MisconfigurationException(
-                'Cannot use LearningRateLogger callback with models that have no'
-                ' learning rate schedulers. Please see documentation for'
-                ' `configure_optimizers` method.')
-
         if not trainer.logger:
             raise MisconfigurationException(
                 'Cannot use LearningRateLogger callback with Trainer that has no logger.')
 
+        if not trainer.lr_schedulers:
+            rank_zero_warn(
+                'You are using LearningRateLogger callback with models that'
+                ' have no learning rate schedulers. Please see documentation'
+                ' for `configure_optimizers` method.', RuntimeWarning
+            )
+
         # Find names for schedulers
         names = self._find_names(trainer.lr_schedulers)
 
         # Initialize for storing values
-        self.lrs = dict.fromkeys(names, [])
+        self.lrs = {name: [] for name in names}
 
     def on_batch_start(self, trainer, pl_module):
         latest_stat = self._extract_lr(trainer, 'step')

pytorch_lightning/callbacks/model_checkpoint.py

@@ -20,7 +20,10 @@ from pytorch_lightning.utilities import rank_zero_warn, rank_zero_only
 
 class ModelCheckpoint(Callback):
     r"""
-    Save the model after every epoch.
+    Save the model after every epoch if it improves.
+
+    After training finishes, use :attr:`best_model_path` to retrieve the path to the
+    best checkpoint file and :attr:`best_model_score` to retrieve its score.
 
     Args:
         filepath: path to save the model file.
@@ -43,6 +46,7 @@ class ModelCheckpoint(Callback):
 
         monitor: quantity to monitor.
         verbose: verbosity mode. Default: ``False``.
+        save_last: always saves the model at the end of the epoch. Default: ``False``.
         save_top_k: if `save_top_k == k`,
             the best k models according to
             the quantity monitored will be saved.
@@ -80,10 +84,17 @@ class ModelCheckpoint(Callback):
         ...     filepath='my/path/sample-mnist_{epoch:02d}-{val_loss:.2f}'
         ... )
 
+        # retrieve the best checkpoint after training
+        checkpoint_callback = ModelCheckpoint(filepath='my/path/')
+        trainer = Trainer(checkpoint_callback=checkpoint_callback)
+        model = ...
+        trainer.fit(model)
+        checkpoint_callback.best_model_path
+
     """
 
     def __init__(self, filepath: Optional[str] = None, monitor: str = 'val_loss', verbose: bool = False,
-                 save_top_k: int = 1, save_weights_only: bool = False,
+                 save_last: bool = False, save_top_k: int = 1, save_weights_only: bool = False,
                  mode: str = 'auto', period: int = 1, prefix: str = ''):
         super().__init__()
         if save_top_k > 0 and filepath is not None and os.path.isdir(filepath) and len(os.listdir(filepath)) > 0:
@@ -103,6 +114,7 @@ class ModelCheckpoint(Callback):
             else:
                 self.dirpath, self.filename = os.path.split(filepath)
             os.makedirs(self.dirpath, exist_ok=True)
+        self.save_last = save_last
         self.save_top_k = save_top_k
         self.save_weights_only = save_weights_only
         self.period = period
@@ -110,8 +122,9 @@ class ModelCheckpoint(Callback):
         self.prefix = prefix
         self.best_k_models = {}
         # {filename: monitor}
-        self.kth_best_model = ''
-        self.best = 0
+        self.kth_best_model_path = ''
+        self.best_model_score = 0
+        self.best_model_path = ''
         self.save_function = None
 
         torch_inf = torch.tensor(np.Inf)
@@ -129,6 +142,18 @@ class ModelCheckpoint(Callback):
 
         self.kth_value, self.mode = mode_dict[mode]
 
+    @property
+    def best(self):
+        rank_zero_warn("Attribute `best` has been renamed to `best_model_score` since v0.8.0"
+                       " and will be removed in v0.10.0", DeprecationWarning)
+        return self.best_model_score
+
+    @property
+    def kth_best_model(self):
+        rank_zero_warn("Attribute `kth_best_model` has been renamed to `kth_best_model_path` since v0.8.0"
+                       " and will be removed in v0.10.0", DeprecationWarning)
+        return self.kth_best_model_path
+
     def _del_model(self, filepath):
         if os.path.isfile(filepath):
             os.remove(filepath)
@@ -139,7 +164,7 @@ class ModelCheckpoint(Callback):
 
         # delegate the saving to the model
         if self.save_function is not None:
-            self.save_function(filepath)
+            self.save_function(filepath, self.save_weights_only)
         else:
             raise ValueError(".save_function() not set")
 
@@ -160,7 +185,7 @@ class ModelCheckpoint(Callback):
             "max": torch.gt,
         }[self.mode]
 
-        return monitor_op(current, self.best_k_models[self.kth_best_model])
+        return monitor_op(current, self.best_k_models[self.kth_best_model_path])
 
     def format_checkpoint_name(self, epoch, metrics, ver=None):
         """Generate a filename according to the defined template.
@@ -217,6 +242,10 @@ class ModelCheckpoint(Callback):
 
         self.epoch_last_check = epoch
 
+        if self.save_last:
+            filepath = os.path.join(self.dirpath, self.prefix + 'last.ckpt')
+            self._save_model(filepath)
+
         filepath = self.format_checkpoint_name(epoch, metrics)
         version_cnt = 0
         while os.path.isfile(filepath):
@@ -252,25 +281,26 @@ class ModelCheckpoint(Callback):
 
         del_list = []
         if len(self.best_k_models) == self.save_top_k and self.save_top_k > 0:
-            delpath = self.kth_best_model
-            self.best_k_models.pop(self.kth_best_model)
+            delpath = self.kth_best_model_path
+            self.best_k_models.pop(self.kth_best_model_path)
             del_list.append(delpath)
 
         self.best_k_models[filepath] = current
         if len(self.best_k_models) == self.save_top_k:
             # monitor dict has reached k elements
             _op = max if self.mode == 'min' else min
-            self.kth_best_model = _op(self.best_k_models,
-                                      key=self.best_k_models.get)
-            self.kth_value = self.best_k_models[self.kth_best_model]
+            self.kth_best_model_path = _op(self.best_k_models,
+                                           key=self.best_k_models.get)
+            self.kth_value = self.best_k_models[self.kth_best_model_path]
 
         _op = min if self.mode == 'min' else max
-        self.best = _op(self.best_k_models.values())
+        self.best_model_path = _op(self.best_k_models, key=self.best_k_models.get)
+        self.best_model_score = self.best_k_models[self.best_model_path]
 
         if self.verbose > 0:
             log.info(
                 f'\nEpoch {epoch:05d}: {self.monitor} reached'
-                f' {current:0.5f} (best {self.best:0.5f}), saving model to'
+                f' {current:0.5f} (best {self.best_model_score:0.5f}), saving model to'
                 f' {filepath} as top {self.save_top_k}')
         self._save_model(filepath)
 

pytorch_lightning/callbacks/progress.py

@@ -323,7 +323,7 @@ class ProgressBar(ProgressBarBase):
         super().on_batch_end(trainer, pl_module)
         if self.is_enabled and self.train_batch_idx % self.refresh_rate == 0:
             self.main_progress_bar.update(self.refresh_rate)
-            self.main_progress_bar.set_postfix(**trainer.progress_bar_dict)
+            self.main_progress_bar.set_postfix(trainer.progress_bar_dict)
 
     def on_validation_start(self, trainer, pl_module):
         super().on_validation_start(trainer, pl_module)
@@ -338,7 +338,7 @@ class ProgressBar(ProgressBarBase):
 
     def on_validation_end(self, trainer, pl_module):
         super().on_validation_end(trainer, pl_module)
-        self.main_progress_bar.set_postfix(**trainer.progress_bar_dict)
+        self.main_progress_bar.set_postfix(trainer.progress_bar_dict)
         self.val_progress_bar.close()
 
     def on_train_end(self, trainer, pl_module):

pytorch_lightning/core/grads.py

@@ -1,30 +1,41 @@
 """
 Module to describe gradients
 """
-from typing import Dict
+from typing import Dict, Union
 
-from torch import nn
+import torch
 
 
-class GradInformation(nn.Module):
+class GradInformation(torch.nn.Module):
 
-    def grad_norm(self, norm_type: float) -> Dict[str, int]:
-        results = {}
-        total_norm = 0
+    def grad_norm(self, norm_type: Union[float, int, str]) -> Dict[str, float]:
+        """Compute each parameter's gradient's norm and their overall norm.
+
+        The overall norm is computed over all gradients together, as if they
+        were concatenated into a single vector.
+
+        Args:
+            norm_type: The type of the used p-norm, cast to float if necessary.
+                Can be ``'inf'`` for infinity norm.
+
+        Return:
+            norms: The dictionary of p-norms of each parameter's gradient and
+                a special entry for the total p-norm of the gradients viewed
+                as a single vector.
+        """
+        norm_type = float(norm_type)
+
+        norms, all_norms = {}, []
         for name, p in self.named_parameters():
-            if p.requires_grad:
-                try:
-                    param_norm = p.grad.data.norm(norm_type)
-                    total_norm += param_norm ** norm_type
-                    norm = param_norm ** (1 / norm_type)
+            if p.grad is None:
+                continue
 
-                    grad = round(norm.data.cpu().numpy().flatten()[0], 3)
-                    results['grad_{}_norm_{}'.format(norm_type, name)] = grad
-                except Exception:
-                    # this param had no grad
-                    pass
+            param_norm = float(p.grad.data.norm(norm_type))
+            norms[f'grad_{norm_type}_norm_{name}'] = round(param_norm, 3)
 
-        total_norm = total_norm ** (1. / norm_type)
-        grad = round(total_norm.data.cpu().numpy().flatten()[0], 3)
-        results['grad_{}_norm_total'.format(norm_type)] = grad
-        return results
+            all_norms.append(param_norm)
+
+        total_norm = float(torch.tensor(all_norms).norm(norm_type))
+        norms[f'grad_{norm_type}_norm_total'] = round(total_norm, 3)
+
+        return norms

pytorch_lightning/core/hooks.py

@@ -3,6 +3,8 @@ from typing import Any
 import torch
 from torch import Tensor
 from torch.optim.optimizer import Optimizer
+from pytorch_lightning.utilities import move_data_to_device
+
 
 try:
     from apex import amp
@@ -153,3 +155,48 @@ class ModelHooks(torch.nn.Module):
                     scaled_loss.backward()
         else:
             loss.backward()
+
+    def transfer_batch_to_device(self, batch: Any, device: torch.device) -> Any:
+        """
+        Override this hook if your :class:`~torch.utils.data.DataLoader` returns tensors
+        wrapped in a custom data structure.
+
+        The data types listed below (and any arbitrary nesting of them) are supported out of the box:
+
+        - :class:`torch.Tensor`
+        - :class:`list`
+        - :class:`dict`
+        - :class:`tuple`
+        - ``torchtext.data.Batch`` (COMING SOON)
+
+        For anything else, you need to define how the data is moved to the target device (CPU, GPU, TPU, ...).
+
+        Example::
+
+            def transfer_batch_to_device(self, batch, device)
+                if isinstance(batch, CustomBatch):
+                    # move all tensors in your custom data structure to the device
+                    batch.samples = batch.samples.to(device)
+                    batch.targets = batch.targets.to(device)
+                else:
+                    batch = super().transfer_batch_to_device(data, device)
+                return batch
+
+        Args:
+            batch: A batch of data that needs to be transferred to a new device.
+            device: The target device as defined in PyTorch.
+
+        Returns:
+            A reference to the data on the new device.
+
+        Note:
+            This hook should only transfer the data and not modify it, nor should it move the data to
+            any other device than the one passed in as argument (unless you know what you are doing).
+            The :class:`~pytorch_lightning.trainer.trainer.Trainer` already takes care of splitting the
+            batch and determines the target devices.
+
+        See Also:
+            - :func:`~pytorch_lightning.utilities.apply_func.move_data_to_device`
+            - :func:`~pytorch_lightning.utilities.apply_func.apply_to_collection`
+        """
+        return move_data_to_device(batch, device)

pytorch_lightning/core/lightning.py

@@ -17,8 +17,8 @@ from pytorch_lightning import _logger as log
 from pytorch_lightning.core.grads import GradInformation
 from pytorch_lightning.core.hooks import ModelHooks
 from pytorch_lightning.core.memory import ModelSummary
-from pytorch_lightning.core.saving import ModelIO, load_hparams_from_tags_csv, load_hparams_from_yaml, update_hparams
-from pytorch_lightning.core.properties import DeviceDtypeModuleMixin
+from pytorch_lightning.core.saving import ModelIO, load_hparams_from_tags_csv, load_hparams_from_yaml
+from pytorch_lightning.utilities.device_dtype_mixin import DeviceDtypeModuleMixin
 from pytorch_lightning.overrides.data_parallel import LightningDistributedDataParallel
 from pytorch_lightning.utilities.exceptions import MisconfigurationException
 from pytorch_lightning.utilities import rank_zero_warn
@@ -30,8 +30,10 @@ except ImportError:
 else:
     XLA_AVAILABLE = True
 
+CHECKPOINT_KEY_MODULE_ARGS = 'module_arguments'
 
-class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, ModelHooks):
+
+class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, ModelHooks, torch.nn.Module):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -53,10 +55,6 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
         self.logger = None
         self.example_input_array = None
 
-        #: True if your model is currently running on GPUs.
-        #: Useful to set flags around the LightningModule for different CPU vs GPU behavior.
-        self.on_gpu = False
-
         #: True if using dp
         self.use_dp = False
 
@@ -66,16 +64,26 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
         #: True if using ddp2
         self.use_ddp2 = False
 
+        # True if on tpu
+        self.use_tpu = False
+
         #: True if using amp
         self.use_amp = False
 
-        self.hparams = None
-
         #: Current dtype
-        self._dtype = torch.FloatTensor
+        self._dtype = torch.float
+
         #: device reference
         self._device = torch.device('cpu')
 
+    @property
+    def on_gpu(self):
+        """
+        True if your model is currently running on GPUs.
+        Useful to set flags around the LightningModule for different CPU vs GPU behavior.
+        """
+        return self.device.type == 'cuda'
+
     def print(self, *args, **kwargs) -> None:
         r"""
         Prints only from process 0. Use this in any distributed mode to log only once.
@@ -949,7 +957,7 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
                         f"is not equal to the computed world size ({world_size}). Ignored.")
 
         torch_backend = "nccl" if self.trainer.on_gpu else "gloo"
-        log.info(f"initializing proc_rank {proc_rank} world {world_size}")
+        log.info(f"initializing ddp: LOCAL_RANK: {proc_rank}/{world_size - 1} WORLD_SIZE:{world_size}")
         torch_distrib.init_process_group(torch_backend, rank=proc_rank, world_size=world_size)
 
     def configure_apex(
@@ -1153,7 +1161,7 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
                     if self.trainer.global_step < 500:
                         lr_scale = min(1., float(self.trainer.global_step + 1) / 500.)
                         for pg in optimizer.param_groups:
-                            pg['lr'] = lr_scale * self.hparams.learning_rate
+                            pg['lr'] = lr_scale * self.learning_rate
 
                     # update params
                     optimizer.step()
@@ -1307,7 +1315,7 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
                                     download=True)
                     loader = torch.utils.data.DataLoader(
                         dataset=dataset,
-                        batch_size=self.hparams.batch_size,
+                        batch_size=self.batch_size,
                         shuffle=True
                     )
                     return loader
@@ -1358,7 +1366,7 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
                                     download=True)
                     loader = torch.utils.data.DataLoader(
                         dataset=dataset,
-                        batch_size=self.hparams.batch_size,
+                        batch_size=self.batch_size,
                         shuffle=False
                     )
 
@@ -1403,7 +1411,7 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
                                     transform=transform, download=True)
                     loader = torch.utils.data.DataLoader(
                         dataset=dataset,
-                        batch_size=self.hparams.batch_size,
+                        batch_size=self.batch_size,
                         shuffle=False
                     )
 
@@ -1443,46 +1451,13 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
             map_location: Optional[Union[Dict[str, str], str, torch.device, int, Callable]] = None,
             hparams_file: Optional[str] = None,
             tags_csv: Optional[str] = None,  # backward compatible, todo: remove in v0.9.0
-            hparam_overrides: Optional[Dict] = None,
             **kwargs
     ) -> 'LightningModule':
         r"""
         Primary way of loading a model from a checkpoint. When Lightning saves a checkpoint
-        it stores the hyperparameters in the checkpoint if you initialized your :class:`LightningModule`
-        with an argument called ``hparams`` which is an object of :class:`~dict` or
-        :class:`~argparse.Namespace` (output of :meth:`~argparse.ArgumentParser.parse_args`
-        when parsing command line arguments).
-        If you want `hparams` to have a hierarchical structure, you have to define it as :class:`~dict`.
-        Any other arguments specified through \*args and \*\*kwargs will be passed to the model.
+        it stores the arguments passed to `__init__`  in the checkpoint under `module_arguments`
 
-        Example:
-            .. code-block:: python
-
-                # define hparams as Namespace
-                from argparse import Namespace
-                hparams = Namespace(**{'learning_rate': 0.1})
-
-                model = MyModel(hparams)
-
-                class MyModel(LightningModule):
-                    def __init__(self, hparams: Namespace):
-                        self.learning_rate = hparams.learning_rate
-
-                # ----------
-
-                # define hparams as dict
-                hparams = {
-                    drop_prob: 0.2,
-                    dataloader: {
-                        batch_size: 32
-                    }
-                }
-
-                model = MyModel(hparams)
-
-                class MyModel(LightningModule):
-                    def __init__(self, hparams: dict):
-                        self.learning_rate = hparams['learning_rate']
+        Any arguments specified through \*args and \*\*kwargs will override args stored in `module_arguments`.
 
         Args:
             checkpoint_path: Path to checkpoint.
@@ -1551,15 +1526,8 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
                 # override some of the params with new values
                 MyLightningModule.load_from_checkpoint(
                     PATH,
-                    hparam_overrides={'num_layers': 128, 'pretrained_ckpt_path': NEW_PATH}
-                )
-
-                # or load passing whatever args the model takes to load
-                MyLightningModule.load_from_checkpoint(
-                    'path/to/checkpoint.ckpt',
-                    learning_rate=0.1, # These arguments will be passed to the model using **kwargs
-                    layers=2,
-                    pretrained_model=some_model
+                    num_layers=128,
+                    pretrained_ckpt_path: NEW_PATH,
                 )
 
                 # predict
@@ -1589,46 +1557,23 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
             hparams['on_gpu'] = False
 
             # overwrite hparams by the given file
-            checkpoint['hparams'] = hparams
+            checkpoint[CHECKPOINT_KEY_MODULE_ARGS] = hparams
 
-        # override the hparam keys that were passed in
-        if hparam_overrides is not None:
-            update_hparams(hparams, hparam_overrides)
+        # override the module_arguments with values that were passed in
+        checkpoint[CHECKPOINT_KEY_MODULE_ARGS].update(kwargs)
 
         model = cls._load_model_state(checkpoint, *args, **kwargs)
         return model
 
     @classmethod
     def _load_model_state(cls, checkpoint: Dict[str, Any], *args, **kwargs) -> 'LightningModule':
-        cls_takes_hparams = 'hparams' in inspect.signature(cls.__init__).parameters
-        ckpt_hparams = checkpoint.get('hparams')
 
-        if cls_takes_hparams:
-            if ckpt_hparams is not None:
-                hparams_type = checkpoint.get('hparams_type', 'Namespace')
-                if hparams_type.lower() == 'dict':
-                    hparams = ckpt_hparams
-                elif hparams_type.lower() == 'namespace':
-                    hparams = Namespace(**ckpt_hparams)
-            else:
-                rank_zero_warn(
-                    f"Checkpoint does not contain hyperparameters but {cls.__name__}'s __init__"
-                    " contains argument 'hparams'. Will pass in an empty Namespace instead."
-                    " Did you forget to store your model hyperparameters in self.hparams?"
-                )
-                hparams = {}
-        else:  # The user's LightningModule does not define a hparams argument
-            if ckpt_hparams is None:
-                hparams = None
-            else:
-                raise MisconfigurationException(
-                    f"Checkpoint contains hyperparameters but {cls.__name__}'s __init__ "
-                    f"is missing the argument 'hparams'. Are you loading the correct checkpoint?"
-                )
+        # pass in the values we saved automatically
+        if CHECKPOINT_KEY_MODULE_ARGS in checkpoint:
+            model_args = checkpoint[CHECKPOINT_KEY_MODULE_ARGS]
+            kwargs.update(**model_args)
 
         # load the state_dict on the model automatically
-        if cls_takes_hparams:
-            kwargs.update(hparams=hparams)
         model = cls(*args, **kwargs)
         model.load_state_dict(checkpoint['state_dict'])
 
@@ -1752,3 +1697,53 @@ class LightningModule(ABC, DeviceDtypeModuleMixin, GradInformation, ModelIO, Mod
         rank_zero_warn("`get_tqdm_dict` was renamed to `get_progress_bar_dict` in v0.7.3"
                        " and this method will be removed in v1.0.0", DeprecationWarning)
         return self.get_progress_bar_dict()
+
+    def auto_collect_arguments(self):
+        """Collect all arguments module arguments."""
+        frame = inspect.currentframe()
+
+        frame_args = _collect_init_args(frame.f_back, [])
+        child = _get_latest_child(frame)
+
+        # set module_arguments in child
+        child._module_self_arguments = frame_args[-1]
+        child._module_parents_arguments = {}
+        for args in frame_args[:-1]:
+            child._module_parents_arguments.update(args)
+
+    @property
+    def module_arguments(self) -> dict:
+        """Aggregate this module and all parents arguments."""
+        try:
+            args = dict(self._module_parents_arguments)
+            args.update(self._module_self_arguments)
+            return args
+        except AttributeError as e:
+            rank_zero_warn('you called `module.module_arguments` without calling self.auto_collect_arguments()')
+            return {}
+
+
+def _collect_init_args(frame, path_args: list) -> list:
+    """Recursive search for all children."""
+    if '__class__' in frame.f_locals:
+        local_args = dict(frame.f_locals)
+        local_args.update(local_args.get('kwargs', {}))
+        local_args = {k: v for k, v in local_args.items()
+                      if k not in ('args', 'kwargs', 'self', '__class__', 'frame', 'frame_args')}
+        # if 'hparams' in local_args:
+        #     # back compatible hparams as single argument
+        #     hparams = local_args.get('hparams')
+        #     local_args.update(vars(hparams) if isinstance(hparams, Namespace) else hparams)
+        # recursive update
+        path_args.append(local_args)
+        return _collect_init_args(frame.f_back, path_args)
+    else:
+        return path_args
+
+
+def _get_latest_child(frame, child: object = None) -> object:
+    """Recursive search for lowest child."""
+    if 'self' in frame.f_locals:
+        return _get_latest_child(frame.f_back, frame.f_locals['self'])
+    else:
+        return child

pytorch_lightning/core/saving.py

@@ -154,6 +154,6 @@ def save_hparams_to_yaml(config_yaml, hparams: Union[dict, Namespace]) -> None:
 def convert(val: str) -> Union[int, float, bool, str]:
     try:
         return ast.literal_eval(val)
-    except (ValueError, SyntaxError) as e:
-        log.debug(e)
+    except (ValueError, SyntaxError) as err:
+        log.debug(err)
         return val

pytorch_lightning/loggers/comet.py

@@ -16,9 +16,16 @@ try:
     except ImportError:  # pragma: no-cover
         # For more information, see: https://www.comet.ml/docs/python-sdk/releases/#release-300
         from comet_ml.papi import API  # pragma: no-cover
+
+    _COMET_AVAILABLE = True
 except ImportError:  # pragma: no-cover
-    raise ImportError('You want to use `comet_ml` logger which is not installed yet,'  # pragma: no-cover
-                      ' install it with `pip install comet-ml`.')
+    CometExperiment = None
+    CometExistingExperiment = None
+    CometOfflineExperiment = None
+    CometBaseExperiment = None
+    API = None
+    _COMET_AVAILABLE = False
+
 
 import torch
 from torch import is_tensor
@@ -93,6 +100,9 @@ class CometLogger(LightningLoggerBase):
                  experiment_key: Optional[str] = None,
                  **kwargs):
 
+        if not _COMET_AVAILABLE:
+            raise ImportError('You want to use `comet_ml` logger which is not installed yet,'
+                              ' install it with `pip install comet-ml`.')
         super().__init__()
         self._experiment = None
 
@@ -125,7 +135,7 @@ class CometLogger(LightningLoggerBase):
         if experiment_name:
             try:
                 self.name = experiment_name
-            except TypeError as e:
+            except TypeError:
                 log.exception("Failed to set experiment name for comet.ml logger")
         self._kwargs = kwargs
 

pytorch_lightning/loggers/mlflow.py

@@ -10,9 +10,11 @@ from typing import Optional, Dict, Any, Union
 try:
     import mlflow
     from mlflow.tracking import MlflowClient
+    _MLFLOW_AVAILABLE = True
 except ImportError:  # pragma: no-cover
-    raise ImportError('You want to use `mlflow` logger which is not installed yet,'  # pragma: no-cover
-                      ' install it with `pip install mlflow`.')
+    mlflow = None
+    MlflowClient = None
+    _MLFLOW_AVAILABLE = False
 
 from pytorch_lightning import _logger as log
 from pytorch_lightning.loggers.base import LightningLoggerBase
@@ -54,11 +56,16 @@ class MLFlowLogger(LightningLoggerBase):
         tags: A dictionary tags for the experiment.
 
     """
+
     def __init__(self,
                  experiment_name: str = 'default',
                  tracking_uri: Optional[str] = None,
                  tags: Optional[Dict[str, Any]] = None,
                  save_dir: Optional[str] = None):
+
+        if not _MLFLOW_AVAILABLE:
+            raise ImportError('You want to use `mlflow` logger which is not installed yet,'
+                              ' install it with `pip install mlflow`.')
         super().__init__()
         if not tracking_uri and save_dir:
             tracking_uri = f'file:{os.sep * 2}{save_dir}'

pytorch_lightning/loggers/neptune.py

@@ -10,9 +10,11 @@ from PIL.Image import Image
 try:
     import neptune
     from neptune.experiments import Experiment
+    _NEPTUNE_AVAILABLE = True
 except ImportError:  # pragma: no-cover
-    raise ImportError('You want to use `neptune` logger which is not installed yet,'  # pragma: no-cover
-                      ' install it with `pip install neptune-client`.')
+    neptune = None
+    Experiment = None
+    _NEPTUNE_AVAILABLE = False
 
 import torch
 from torch import is_tensor
@@ -179,6 +181,9 @@ class NeptuneLogger(LightningLoggerBase):
                  properties: Optional[Dict[str, Any]] = None,
                  tags: Optional[List[str]] = None,
                  **kwargs):
+        if not _NEPTUNE_AVAILABLE:
+            raise ImportError('You want to use `neptune` logger which is not installed yet,'
+                              ' install it with `pip install neptune-client`.')
         super().__init__()
         self.api_key = api_key
         self.project_name = project_name

pytorch_lightning/loggers/test_tube.py

@@ -7,9 +7,10 @@ from typing import Optional, Dict, Any, Union
 
 try:
     from test_tube import Experiment
+    _TEST_TUBE_AVAILABLE = True
 except ImportError:  # pragma: no-cover
-    raise ImportError('You want to use `test_tube` logger which is not installed yet,'  # pragma: no-cover
-                      ' install it with `pip install test-tube`.')
+    Experiment = None
+    _TEST_TUBE_AVAILABLE = False
 
 from pytorch_lightning.loggers.base import LightningLoggerBase
 from pytorch_lightning.utilities.distributed import rank_zero_only
@@ -62,6 +63,10 @@ class TestTubeLogger(LightningLoggerBase):
                  debug: bool = False,
                  version: Optional[int] = None,
                  create_git_tag: bool = False):
+
+        if not _TEST_TUBE_AVAILABLE:
+            raise ImportError('You want to use `test_tube` logger which is not installed yet,'
+                              ' install it with `pip install test-tube`.')
         super().__init__()
         self.save_dir = save_dir
         self._name = name

pytorch_lightning/loggers/trains.py

@@ -14,7 +14,11 @@ from PIL.Image import Image
 try:
     import trains
     from trains import Task
+    _TRAINS_AVAILABLE = True
 except ImportError:  # pragma: no-cover
+    trains = None
+    Task = None
+    _TRAINS_AVAILABLE = False
     raise ImportError('You want to use `TRAINS` logger which is not installed yet,'  # pragma: no-cover
                       ' install it with `pip install trains`.')
 
@@ -91,6 +95,9 @@ class TrainsLogger(LightningLoggerBase):
             auto_connect_frameworks: bool = True,
             auto_resource_monitoring: bool = True
     ) -> None:
+        if not _TRAINS_AVAILABLE:
+            raise ImportError('You want to use `test_tube` logger which is not installed yet,'
+                              ' install it with `pip install test-tube`.')
         super().__init__()
         if self.bypass_mode():
             self._trains = None

pytorch_lightning/loggers/wandb.py

@@ -11,9 +11,11 @@ import torch.nn as nn
 try:
     import wandb
     from wandb.wandb_run import Run
+    _WANDB_AVAILABLE = True
 except ImportError:  # pragma: no-cover
-    raise ImportError('You want to use `wandb` logger which is not installed yet,'  # pragma: no-cover
-                      ' install it with `pip install wandb`.')
+    wandb = None
+    Run = None
+    _WANDB_AVAILABLE = False
 
 from pytorch_lightning.loggers.base import LightningLoggerBase
 from pytorch_lightning.utilities import rank_zero_only
@@ -67,6 +69,9 @@ class WandbLogger(LightningLoggerBase):
                  experiment=None,
                  entity=None,
                  group: Optional[str] = None):
+        if not _WANDB_AVAILABLE:
+            raise ImportError('You want to use `wandb` logger which is not installed yet,'  # pragma: no-cover
+                              ' install it with `pip install wandb`.')
         super().__init__()
         self._name = name
         self._save_dir = save_dir
@@ -123,7 +128,7 @@ class WandbLogger(LightningLoggerBase):
 
     @rank_zero_only
     def log_metrics(self, metrics: Dict[str, float], step: Optional[int] = None) -> None:
-        self.experiment.log(metrics, step=step)
+        self.experiment.log({'global_step': step, **metrics} if step is not None else metrics)
 
     @property
     def name(self) -> str:

pytorch_lightning/metrics/__init__.py

@@ -0,0 +1,24 @@
+"""
+Metrics
+=======
+
+Metrics are generally used to monitor model performance.
+
+The following package aims to provide the most convenient ones as well
+as a structure to implement your custom metrics for all the fancy research
+you want to do.
+
+For native PyTorch implementations of metrics, it is recommended to use
+the :class:`TensorMetric` which handles automated DDP syncing and conversions
+to tensors for all inputs and outputs.
+
+If your metrics implementation works on numpy, just use the
+:class:`NumpyMetric`, which handles the automated conversion of
+inputs to and outputs from numpy as well as automated ddp syncing.
+
+.. warning:: Employing numpy in your metric calculation might slow
+    down your training substantially, since every metric computation
+    requires a GPU sync to convert tensors to numpy.
+
+
+"""

pytorch_lightning/metrics/converters.py

@@ -0,0 +1,230 @@
+"""
+This file provides functions and decorators for automated input and output
+conversion to/from :class:`numpy.ndarray` and :class:`torch.Tensor` as well as utilities to
+sync tensors between different processes in a DDP scenario, when needed.
+"""
+
+import sys
+import numbers
+from typing import Union, Any, Callable, Optional
+
+import numpy as np
+import torch
+from torch.utils.data._utils.collate import np_str_obj_array_pattern
+
+from pytorch_lightning.utilities.apply_func import apply_to_collection
+
+
+def _apply_to_inputs(func_to_apply: Callable, *dec_args, **dec_kwargs) -> Callable:
+    """
+    Decorator function to apply a function to all inputs of a function.
+    Args:
+        func_to_apply: the function to apply to the inputs
+        *dec_args: positional arguments for the function to be applied
+        **dec_kwargs: keyword arguments for the function to be applied
+
+    Returns:
+        the decorated function
+    """
+
+    def decorator_fn(func_to_decorate):
+        # actual function applying the give function to inputs
+        def new_func(*args, **kwargs):
+            args = func_to_apply(args, *dec_args, **dec_kwargs)
+            kwargs = func_to_apply(kwargs, *dec_args, **dec_kwargs)
+            return func_to_decorate(*args, **kwargs)
+
+        return new_func
+
+    return decorator_fn
+
+
+def _apply_to_outputs(func_to_apply: Callable, *dec_args, **dec_kwargs) -> Callable:
+    """
+    Decorator function to apply a function to all outputs of a function.
+    Args:
+        func_to_apply: the function to apply to the outputs
+        *dec_args: positional arguments for the function to be applied
+        **dec_kwargs: keyword arguments for the function to be applied
+
+    Returns:
+        the decorated function
+    """
+
+    def decorator_fn(function_to_decorate):
+        # actual function applying the give function to outputs
+        def new_func(*args, **kwargs):
+            result = function_to_decorate(*args, **kwargs)
+            return func_to_apply(result, *dec_args, **dec_kwargs)
+
+        return new_func
+
+    return decorator_fn
+
+
+def _convert_to_tensor(data: Any) -> Any:
+    """
+    Maps all kind of collections and numbers to tensors.
+
+    Args:
+        data: the data to convert to tensor
+
+    Returns:
+        the converted data
+
+    """
+    if isinstance(data, numbers.Number):
+        return torch.tensor([data])
+    # is not array of object
+    elif isinstance(data, np.ndarray) and np_str_obj_array_pattern.search(data.dtype.str) is None:
+        return torch.from_numpy(data)
+    elif isinstance(data, torch.Tensor):
+        return data
+
+    raise TypeError(f"The given type ('{type(data).__name__}') cannot be converted to a tensor!")
+
+
+def _convert_to_numpy(data: Union[torch.Tensor, np.ndarray, numbers.Number]) -> np.ndarray:
+    """Convert all tensors and numpy arrays to numpy arrays.
+    Args:
+        data: the tensor or array to convert to numpy
+
+    Returns:
+        the resulting numpy array
+
+    """
+    if isinstance(data, torch.Tensor):
+        return data.cpu().detach().numpy()
+    elif isinstance(data, numbers.Number):
+        return np.array([data])
+    elif isinstance(data, np.ndarray):
+        return data
+
+    raise TypeError("The given type ('%s') cannot be converted to a numpy array!" % type(data).__name__)
+
+
+def _numpy_metric_conversion(func_to_decorate: Callable) -> Callable:
+    """
+    Decorator handling the argument conversion for metrics working on numpy.
+    All inputs of the decorated function will be converted to numpy and all
+    outputs will be converted to tensors.
+
+    Args:
+        func_to_decorate: the function whose inputs and outputs shall be converted
+
+    Returns:
+        the decorated function
+
+    """
+    # applies collection conversion from tensor to numpy to all inputs
+    # we need to include numpy arrays here, since otherwise they will also be treated as sequences
+    func_convert_inputs = _apply_to_inputs(
+        apply_to_collection, (torch.Tensor, np.ndarray, numbers.Number), _convert_to_numpy)(func_to_decorate)
+    # converts all inputs back to tensors (device doesn't matter here, since this is handled by BaseMetric)
+    func_convert_in_out = _apply_to_outputs(_convert_to_tensor)(func_convert_inputs)
+    return func_convert_in_out
+
+
+def _tensor_metric_conversion(func_to_decorate: Callable) -> Callable:
+    """
+    Decorator Handling the argument conversion for metrics working on tensors.
+    All inputs and outputs of the decorated function will be converted to tensors
+
+    Args:
+        func_to_decorate: the function whose inputs and outputs shall be converted
+
+    Returns:
+        the decorated function
+
+    """
+    # converts all inputs to tensor if possible
+    # we need to include tensors here, since otherwise they will also be treated as sequences
+    func_convert_inputs = _apply_to_inputs(
+        apply_to_collection, (torch.Tensor, np.ndarray, numbers.Number), _convert_to_tensor)(func_to_decorate)
+    # convert all outputs to tensor if possible
+    return _apply_to_outputs(_convert_to_tensor)(func_convert_inputs)
+
+
+def _sync_ddp_if_available(result: Union[torch.Tensor],
+                           group: Optional[Any] = None,
+                           reduce_op: Optional[torch.distributed.ReduceOp] = None,
+                           ) -> torch.Tensor:
+    """
+    Function to reduce the tensors from several ddp processes to one master process
+
+    Args:
+        result: the value to sync and reduce (typically tensor or number)
+        group: the process group to gather results from. Defaults to all processes (world)
+        reduce_op: the reduction operation. Defaults to sum.
+
+    Returns:
+        reduced value
+
+    """
+
+    if torch.distributed.is_available() and torch.distributed.is_initialized():
+        if group is None:
+            group = torch.distributed.group.WORLD
+
+        if reduce_op is None:
+            reduce_op = torch.distributed.ReduceOp.SUM
+
+        # sync all processes before reduction
+        torch.distributed.barrier(group=group)
+        torch.distributed.all_reduce(result, op=reduce_op, group=group,
+                                     async_op=False)
+
+    return result
+
+
+def numpy_metric(group: Optional[Any] = None,
+                 reduce_op: Optional[torch.distributed.ReduceOp] = None) -> Callable:
+    """
+    This decorator shall be used on all function metrics working on numpy arrays.
+
+    It handles the argument conversion and DDP reduction for metrics working on numpy.
+    All inputs of the decorated function will be converted to numpy and all
+    outputs will be converted to tensors.
+    In DDP Training all output tensors will be reduced according to the given rules.
+
+    Args:
+        group: the process group to gather results from. Defaults to all processes (world)
+        reduce_op: the reduction operation. Defaults to sum
+
+    Returns:
+        the decorated function
+
+    """
+
+    def decorator_fn(func_to_decorate):
+        return _apply_to_outputs(apply_to_collection, torch.Tensor, _sync_ddp_if_available,
+                                 group=group,
+                                 reduce_op=reduce_op)(_numpy_metric_conversion(func_to_decorate))
+
+    return decorator_fn
+
+
+def tensor_metric(group: Optional[Any] = None,
+                  reduce_op: Optional[torch.distributed.ReduceOp] = None) -> Callable:
+    """
+    This decorator shall be used on all function metrics working on tensors.
+
+    It handles the argument conversion and DDP reduction for metrics working on tensors.
+    All inputs and outputs of the decorated function will be converted to tensors.
+    In DDP Training all output tensors will be reduced according to the given rules.
+
+    Args:
+       group: the process group to gather results from. Defaults to all processes (world)
+       reduce_op: the reduction operation. Defaults to sum
+
+    Returns:
+       the decorated function
+
+    """
+
+    def decorator_fn(func_to_decorate):
+        return _apply_to_outputs(apply_to_collection, torch.Tensor, _sync_ddp_if_available,
+                                 group=group,
+                                 reduce_op=reduce_op)(_tensor_metric_conversion(func_to_decorate))
+
+    return decorator_fn

pytorch_lightning/metrics/metric.py

@@ -0,0 +1,103 @@
+from abc import ABC, abstractmethod
+from typing import Any, Optional, Union
+
+import torch
+import torch.distributed
+
+from pytorch_lightning.metrics.converters import tensor_metric, numpy_metric
+from pytorch_lightning.utilities.apply_func import apply_to_collection
+from pytorch_lightning.utilities.device_dtype_mixin import DeviceDtypeModuleMixin
+
+__all__ = ['Metric', 'TensorMetric', 'NumpyMetric']
+
+
+class Metric(DeviceDtypeModuleMixin, torch.nn.Module, ABC):
+    """
+    Abstract base class for metric implementation.
+
+    Should be used to implement metrics that
+    1. Return multiple Outputs
+    2. Handle their own DDP sync
+    """
+    def __init__(self, name: str):
+        """
+        Args:
+            name: the metric's name
+
+        """
+        super().__init__()
+        self.name = name
+        self._dtype = torch.get_default_dtype()
+        self._device = torch.device('cpu')
+
+    @abstractmethod
+    def forward(self, *args, **kwargs) -> torch.Tensor:
+        """
+        Implements the actual metric computation.
+
+        Returns:
+            metric value
+
+        """
+        raise NotImplementedError
+
+
+class TensorMetric(Metric):
+    """
+    Base class for metric implementation operating directly on tensors.
+    All inputs and outputs will be casted to tensors if necessary.
+    Already handles DDP sync and input/output conversions.
+    """
+    def __init__(self, name: str,
+                 reduce_group: Optional[Any] = None,
+                 reduce_op: Optional[Any] = None):
+        """
+
+        Args:
+            name: the metric's name
+            reduce_group: the process group for DDP reduces (only needed for DDP training).
+                Defaults to all processes (world)
+            reduce_op: the operation to perform during reduction within DDP (only needed for DDP training).
+                Defaults to sum.
+        """
+        super().__init__(name)
+        self._orig_call = tensor_metric(group=reduce_group,
+                                        reduce_op=reduce_op)(super().__call__)
+
+    def __call__(self, *args, **kwargs) -> torch.Tensor:
+        def _to_device_dtype(x: torch.Tensor) -> torch.Tensor:
+            return x.to(device=self.device, dtype=self.dtype, non_blocking=True)
+
+        return apply_to_collection(self._orig_call(*args, **kwargs), torch.Tensor,
+                                   _to_device_dtype)
+
+
+class NumpyMetric(Metric):
+    """
+    Base class for metric implementation operating on numpy arrays.
+    All inputs will be casted to numpy if necessary and all outputs will
+    be casted to tensors if necessary.
+    Already handles DDP sync and input/output conversions.
+    """
+    def __init__(self, name: str,
+                 reduce_group: Optional[Any] = None,
+                 reduce_op: Optional[Any] = None):
+        """
+
+        Args:
+            name: the metric's name
+            reduce_group: the process group for DDP reduces (only needed for DDP training).
+                Defaults to all processes (world)
+            reduce_op: the operation to perform during reduction within DDP (only needed for DDP training).
+                Defaults to sum.
+        """
+        super().__init__(name)
+        self._orig_call = numpy_metric(group=reduce_group,
+                                       reduce_op=reduce_op)(super().__call__)
+
+    def __call__(self, *args, **kwargs) -> torch.Tensor:
+        def _to_device_dtype(x: torch.Tensor) -> torch.Tensor:
+            return x.to(device=self.device, dtype=self.dtype, non_blocking=True)
+
+        return apply_to_collection(self._orig_call(*args, **kwargs), torch.Tensor,
+                                   _to_device_dtype)

pytorch_lightning/overrides/data_parallel.py

@@ -177,9 +177,9 @@ def parallel_apply(modules, inputs, kwargs_tup=None, devices=None):  # pragma: n
 
             with lock:
                 results[i] = output
-        except Exception as e:
+        except Exception as ex:
             with lock:
-                results[i] = e
+                results[i] = ex
 
     # TODO: fix hack (maybe not a hack)
     # make sure each module knows what training state it's in...

pytorch_lightning/profiler/__init__.py

@@ -98,8 +98,7 @@ to track and the profiler will record performance for code executed within this
     from pytorch_lightning.profiler import Profiler, PassThroughProfiler
 
     class MyModel(LightningModule):
-        def __init__(self, hparams, profiler=None):
-            self.hparams = hparams
+        def __init__(self, profiler=None):
             self.profiler = profiler or PassThroughProfiler()
 
         def custom_processing_step(self, data):
@@ -108,7 +107,7 @@ to track and the profiler will record performance for code executed within this
             return data
 
     profiler = Profiler()
-    model = MyModel(hparams, profiler)
+    model = MyModel(profiler)
     trainer = Trainer(profiler=profiler, max_epochs=1)
 
 """

pytorch_lightning/trainer/__init__.py

@@ -54,7 +54,7 @@ main.py file this way
 
         main(args)
 
-So you can run it like so:distributed_backend
+So you can run it like so:
 
 .. code-block:: bash
 
@@ -111,7 +111,7 @@ and set ``deterministic``` flag in ``Trainer``.
 
 .. code-block:: python
 
-    from pytorch-lightning import Trainer, seed_everything
+    from pytorch_lightning import Trainer, seed_everything
 
     seed_everything(42)
     # sets seeds for numpy, torch, python.random and PYTHONHASHSEED.
@@ -322,6 +322,10 @@ Example::
 
 .. note:: this option does not apply to TPU. TPUs use ```ddp``` by default (over each core)
 
+See Also:
+    - `Multi-GPU training guide <multi_gpu.rst>`_
+    - `Multi-node (SLURM) guide <slurm.rst>`_
+
 early_stop_callback
 ^^^^^^^^^^^^^^^^^^^
 
@@ -410,7 +414,8 @@ Example::
     # uses 8 gpus in total
     trainer = Trainer(gpus=2, num_nodes=4)
 
-.. note:: See the `multi-gpu computing guide <multi_gpu.rst>`_
+See Also:
+    - `Multi-GPU training guide <multi_gpu.rst>`_
 
 gradient_clip_val
 ^^^^^^^^^^^^^^^^^
@@ -598,7 +603,22 @@ nb_sanity_val_steps:
 
 num_tpu_cores
 ^^^^^^^^^^^^^
-How many TPU cores to train on (1 or 8).
+.. warning:: .. deprecated:: 0.7.6
+
+    Use `tpu_cores` instead. Will remove 0.9.0.
+
+Example::
+
+    python -m torch_xla.distributed.xla_dist
+    --tpu=$TPU_POD_NAME
+    --conda-env=torch-xla-nightly
+    --env=XLA_USE_BF16=1
+    -- python your_trainer_file.py
+
+tpu_cores
+^^^^^^^^^
+- How many TPU cores to train on (1 or 8).
+- Which TPU core to train on [1-8]
 
 A single TPU v2 or v3 has 8 cores. A TPU pod has
 up to 2048 cores. A slice of a POD means you get as many cores
@@ -615,21 +635,21 @@ Example::
     # your_trainer_file.py
 
     # default used by the Trainer (ie: train on CPU)
-    trainer = Trainer(num_tpu_cores=None)
+    trainer = Trainer(tpu_cores=None)
 
     # int: train on a single core
-    trainer = Trainer(num_tpu_cores=1)
+    trainer = Trainer(tpu_cores=1)
+
+    # list: train on a single selected core
+    trainer = Trainer(tpu_cores=[2])
 
     # int: train on all cores few cores
-    trainer = Trainer(num_tpu_cores=8)
+    trainer = Trainer(tpu_cores=8)
 
     # for 8+ cores must submit via xla script with
     # a max of 8 cores specified. The XLA script
     # will duplicate script onto each TPU in the POD
-    trainer = Trainer(num_tpu_cores=8)
-
-    # -1: train on all available TPUs
-    trainer = Trainer(num_tpu_cores=-1)
+    trainer = Trainer(tpu_cores=8)
 
 To train on more than 8 cores (ie: a POD),
 submit this script using the xla_dist script.
@@ -998,12 +1018,12 @@ Options: 'full', 'top', None.
 
 Example::
 
-    # default used by the Trainer (ie: print all weights)
-    trainer = Trainer(weights_summary='full')
-
-    # print only the top level modules
+    # default used by the Trainer (ie: print summary of top level modules)
     trainer = Trainer(weights_summary='top')
 
+    # print full summary of all modules and submodules
+    trainer = Trainer(weights_summary='full')
+
     # don't print a summary
     trainer = Trainer(weights_summary=None)
 

pytorch_lightning/trainer/callback_config.py

@@ -18,8 +18,6 @@ class TrainerCallbackConfigMixin(ABC):
     weights_save_path: str
     ckpt_path: str
     checkpoint_callback: ModelCheckpoint
-    progress_bar_refresh_rate: int
-    process_position: int
 
     @property
     @abstractmethod
@@ -109,7 +107,7 @@ class TrainerCallbackConfigMixin(ABC):
             self.early_stop_callback = early_stop_callback
             self.enable_early_stop = True
 
-    def configure_progress_bar(self):
+    def configure_progress_bar(self, refresh_rate=1, process_position=0):
         progress_bars = [c for c in self.callbacks if isinstance(c, ProgressBarBase)]
         if len(progress_bars) > 1:
             raise MisconfigurationException(
@@ -117,12 +115,14 @@ class TrainerCallbackConfigMixin(ABC):
                 ' progress bar is supported.'
             )
         elif len(progress_bars) == 1:
-            self.progress_bar_callback = progress_bars[0]
-        elif self.progress_bar_refresh_rate > 0:
-            self.progress_bar_callback = ProgressBar(
-                refresh_rate=self.progress_bar_refresh_rate,
-                process_position=self.process_position,
+            progress_bar_callback = progress_bars[0]
+        elif refresh_rate > 0:
+            progress_bar_callback = ProgressBar(
+                refresh_rate=refresh_rate,
+                process_position=process_position,
             )
-            self.callbacks.append(self.progress_bar_callback)
+            self.callbacks.append(progress_bar_callback)
         else:
-            self.progress_bar_callback = None
+            progress_bar_callback = None
+
+        return progress_bar_callback

pytorch_lightning/trainer/callback_hook.py

@@ -6,11 +6,10 @@ from pytorch_lightning.callbacks import Callback
 
 class TrainerCallbackHookMixin(ABC):
 
-    def __init__(self):
-        # this is just a summary on variables used in this abstract class,
-        # the proper values/initialisation should be done in child class
-        self.callbacks: List[Callback] = []
-        self.get_model: Callable = ...
+    # this is just a summary on variables used in this abstract class,
+    # the proper values/initialisation should be done in child class
+    callbacks: List[Callback] = []
+    get_model: Callable = ...
 
     def on_init_start(self):
         """Called when the trainer initialization begins, model has not yet been set."""

pytorch_lightning/trainer/data_loading.py

@@ -3,7 +3,7 @@ from abc import ABC, abstractmethod
 from typing import Union, List, Tuple, Callable
 
 import torch.distributed as torch_distrib
-from torch.utils.data import DataLoader, RandomSampler
+from torch.utils.data import DataLoader, RandomSampler, SequentialSampler
 from torch.utils.data.distributed import DistributedSampler
 
 from pytorch_lightning.core import LightningModule
@@ -113,39 +113,39 @@ class TrainerDataLoadingMixin(ABC):
         need_dist_sampler = (self.use_ddp or self.use_ddp2 or self.use_horovod or self.use_tpu)
 
         if self.replace_sampler_ddp and need_dist_sampler:
+            if not isinstance(dataloader.sampler, (SequentialSampler, RandomSampler)):
+                raise MisconfigurationException(
+                    'You seem to have configured a sampler in your DataLoader. This will be replaced '
+                    ' by `DistributedSampler` since `replace_sampler_ddp` is True and you are using'
+                    ' distributed training. Either remove the sampler from your DataLoader or set'
+                    ' `replace_sampler_ddp`=False if you want to use your custom sampler.')
+
             skip_keys = ['sampler', 'batch_sampler', 'dataset_kind']
 
             dl_args = {
                 k: v for k, v in dataloader.__dict__.items() if not k.startswith('_') and k not in skip_keys
             }
 
-            if self.use_tpu:
-                sampler = DistributedSampler(
-                    dataloader.dataset,
-                    num_replicas=xm.xrt_world_size(),
-                    rank=xm.get_ordinal(),
-                )
-            elif self.use_horovod:
-                sampler = DistributedSampler(dataloader.dataset,
-                                             num_replicas=hvd.size(),
-                                             rank=hvd.rank())
-            else:
-                world_size = {
-                    'ddp': self.num_nodes * self.num_processes,
-                    'ddp2': self.num_nodes,
-                    'ddp_cpu': self.num_processes * self.num_nodes
-                }
-                sampler = DistributedSampler(
-                    dataloader.dataset,
-                    num_replicas=world_size[self.distributed_backend],
-                    rank=self.proc_rank,
-                )
-
-            dl_args['sampler'] = sampler
+            dl_args['sampler'] = self._get_distributed_sampler(dataloader)
             dataloader = type(dataloader)(**dl_args)
 
         return dataloader
 
+    def _get_distributed_sampler(self, dataloader):
+        if self.use_tpu:
+            kwargs = dict(num_replicas=xm.xrt_world_size(), rank=xm.get_ordinal())
+        elif self.use_horovod:
+            kwargs = dict(num_replicas=hvd.size(), rank=hvd.rank())
+        else:
+            world_size = {
+                'ddp': self.num_nodes * self.num_processes,
+                'ddp2': self.num_nodes,
+                'ddp_cpu': self.num_processes * self.num_nodes
+            }
+            kwargs = dict(num_replicas=world_size[self.distributed_backend], rank=self.proc_rank)
+        sampler = DistributedSampler(dataloader.dataset, **kwargs)
+        return sampler
+
     def reset_train_dataloader(self, model: LightningModule) -> None:
         """Resets the train dataloader and initialises required variables
         (number of batches, when to validate, etc.).
@@ -214,7 +214,7 @@ class TrainerDataLoadingMixin(ABC):
         # shuffling in val and test set is bad practice
         for loader in dataloaders:
             if mode in ('val', 'test') and hasattr(loader, 'sampler') and isinstance(loader.sampler, RandomSampler):
-                raise MisconfigurationException(
+                rank_zero_warn(
                     f'Your {mode}_dataloader has shuffle=True, it is best practice to turn'
                     ' this off for validation and test dataloaders.')
 

pytorch_lightning/trainer/deprecated_api.py

@@ -121,7 +121,7 @@ class TrainerDeprecatedAPITillVer0_9(ABC):
         """Back compatibility, will be removed in v0.9.0"""
         rank_zero_warn("Argument `show_progress_bar` is now set by `progress_bar_refresh_rate` since v0.7.2"
                        " and this method will be removed in v0.9.0", DeprecationWarning)
-        return self.progress_bar_refresh_rate >= 1
+        return self.progress_bar_callback and self.progress_bar_callback.refresh_rate >= 1
 
     @show_progress_bar.setter
     def show_progress_bar(self, tf):
@@ -135,3 +135,9 @@ class TrainerDeprecatedAPITillVer0_9(ABC):
         rank_zero_warn("`training_tqdm_dict` was renamed to `progress_bar_dict` in v0.7.3"
                        " and this method will be removed in v0.9.0", DeprecationWarning)
         return self.progress_bar_dict
+
+    @property
+    def num_tpu_cores(self):
+        """Back compatibility, will be removed in v0.9.0"""
+        rank_zero_warn("Argument `num_tpu_cores` is now set by `tpu_cores` since v0.7.6"
+                       " and this argument will be removed in v0.9.0", DeprecationWarning)

pytorch_lightning/trainer/distrib_data_parallel.py

@@ -117,6 +117,11 @@ import os
 import re
 from abc import ABC, abstractmethod
 from typing import Union
+import subprocess
+import sys
+from time import sleep
+import numpy as np
+from os.path import abspath
 
 import torch
 from pytorch_lightning import _logger as log
@@ -277,7 +282,7 @@ class TrainerDDPMixin(ABC):
             should_fake = int(os.environ['FAKE_SLURM_MANAGING_TASKS'])
             if should_fake:
                 self.is_slurm_managing_tasks = True
-        except Exception as e:
+        except Exception:
             pass
 
         # notify user the that slurm is managing tasks
@@ -311,7 +316,7 @@ class TrainerDDPMixin(ABC):
         os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
 
         # when slurm is managing the task it sets the visible devices
-        if not is_slurm_managing_tasks:
+        if not is_slurm_managing_tasks and 'CUDA_VISIBLE_DEVICES' not in os.environ:
             if isinstance(data_parallel_device_ids, int):
                 id_str = ','.join(str(x) for x in list(range(data_parallel_device_ids)))
                 os.environ["CUDA_VISIBLE_DEVICES"] = id_str
@@ -322,7 +327,74 @@ class TrainerDDPMixin(ABC):
         # don't make this debug... this is good UX
         log.info(f'CUDA_VISIBLE_DEVICES: [{os.environ["CUDA_VISIBLE_DEVICES"]}]')
 
-    def ddp_train(self, process_idx, model):
+    def __set_random_port(self):
+        """
+        When running DDP NOT managed by SLURM, the ports might collide
+        :return:
+        """
+        try:
+            default_port = os.environ['MASTER_PORT']
+        except Exception:
+            import random
+            default_port = random.randint(10000, 19000)
+            os.environ['MASTER_PORT'] = str(default_port)
+
+    def spawn_ddp_children(self, model):
+        self.__set_random_port()
+        port = os.environ['MASTER_PORT']
+
+        master_address = '127.0.0.1' if 'MASTER_ADDR' not in os.environ else os.environ['MASTER_ADDR']
+        os.environ['MASTER_PORT'] = f'{port}'
+        os.environ['MASTER_ADDR'] = f'{master_address}'
+
+        # allow the user to pass the node rank
+        node_rank = '0'
+        if 'NODE_RANK' in os.environ:
+            node_rank = os.environ['NODE_RANK']
+        if 'GROUP_RANK' in os.environ:
+            node_rank = os.environ['GROUP_RANK']
+
+        os.environ['NODE_RANK'] = node_rank
+        os.environ['LOCAL_RANK'] = '0'
+
+        # pull out the commands used to run the script and resolve the abs file path
+        command = sys.argv
+        full_path = abspath(command[0])
+        command[0] = full_path
+        command = ['python'] + command
+
+        # since this script sets the visible devices we replace the gpus flag with a number
+        num_gpus = os.environ['CUDA_VISIBLE_DEVICES'].split(',').__len__()
+
+        # if script called without a flag, pass in a flag anyhow
+        if '--gpus' not in command:
+            arg_gpus = len(self.gpus) if isinstance(self.gpus, list) else self.gpus
+            command += ['--gpus', arg_gpus]
+
+        gpu_flag_idx = command.index('--gpus')
+        command[gpu_flag_idx + 1] = f'{num_gpus}'
+
+        os.environ['WORLD_SIZE'] = f'{num_gpus * self.num_nodes}'
+
+        self.interactive_ddp_procs = []
+        for local_rank in range(1, self.num_processes):
+            env_copy = os.environ.copy()
+            env_copy['LOCAL_RANK'] = f'{local_rank}'
+
+            # import pdb; pdb.set_trace()
+            # start process
+            proc = subprocess.Popen(command, env=env_copy)
+            self.interactive_ddp_procs.append(proc)
+
+            # starting all processes at once can cause issues
+            # with dataloaders delay between 1-10 seconds
+            delay = np.random.uniform(1, 5, 1)[0]
+            sleep(delay)
+
+        local_rank = 0
+        self.ddp_train(local_rank, model, is_master=True)
+
+    def ddp_train(self, process_idx, model, is_master=False):
         """
         Entry point into a DP thread
         :param gpu_idx:
@@ -359,8 +431,14 @@ class TrainerDDPMixin(ABC):
         # MODEL
         # copy model to each gpu
         if self.on_gpu:
-            self.root_gpu = process_idx
-            self._device = torch.device('cuda', self.root_gpu)
+            gpu_idx = process_idx
+            if is_master:
+                # source of truth is cuda for gpu idx
+                gpus = os.environ['CUDA_VISIBLE_DEVICES'].split(',')
+                local_rank = int(os.environ['LOCAL_RANK'])
+                gpu_idx = int(gpus[local_rank])
+
+            self.root_gpu = gpu_idx
             torch.cuda.set_device(self.root_gpu)
             model.cuda(self.root_gpu)
 
@@ -373,6 +451,7 @@ class TrainerDDPMixin(ABC):
         if self.use_amp and not self.use_native_amp:
             model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
             self.optimizers = optimizers
+            self.reinit_scheduler_properties(self.optimizers, self.lr_schedulers)
 
         # DDP2 uses all GPUs on the machine
         if self.distributed_backend == 'ddp':
@@ -388,9 +467,6 @@ class TrainerDDPMixin(ABC):
         # continue training routine
         self.run_pretrain_routine(model)
 
-        # when ddp ends, we save the model
-        self.save_spawn_weights(model)
-
     def save_spawn_weights(self, model):
         """
         Dump a temporary checkpoint after ddp ends to get weights out of the process
@@ -426,8 +502,8 @@ class TrainerDDPMixin(ABC):
 
     def resolve_root_node_address(self, root_node):
         if '[' in root_node:
-            name = root_node.split('[')[0]
-            number = root_node.split(',')[0]
+            name, numbers = root_node.split('[', maxsplit=1)
+            number = numbers.split(',', maxsplit=1)[0]
             if '-' in number:
                 number = number.split('-')[0]
 

pytorch_lightning/trainer/distrib_parts.py

@@ -1,339 +1,6 @@
 """
-Lightning makes multi-gpu training and 16 bit training trivial.
-
-.. note:: None of the flags below require changing anything about your lightningModel definition.
-
-Choosing a backend
-==================
-
-Lightning supports two backends. DataParallel and DistributedDataParallel.
- Both can be used for single-node multi-GPU training.
- For multi-node training you must use DistributedDataParallel.
-
-DataParallel (dp)
------------------
-
-Splits a batch across multiple GPUs on the same node. Cannot be used for multi-node training.
-
-DistributedDataParallel (ddp)
------------------------------
-
-Trains a copy of the model on each GPU and only syncs gradients. If used with DistributedSampler, each GPU trains
-on a subset of the full dataset.
-
-DistributedDataParallel-2 (ddp2)
---------------------------------
-
-Works like DDP, except each node trains a single copy of the model using ALL GPUs on that node.
- Very useful when dealing with negative samples, etc...
-
-You can toggle between each mode by setting this flag.
-
-.. code-block:: python
-
-    # DEFAULT (when using single GPU or no GPUs)
-    trainer = Trainer(distributed_backend=None)
-
-    # Change to DataParallel (gpus > 1)
-    trainer = Trainer(distributed_backend='dp')
-
-    # change to distributed data parallel (gpus > 1)
-    trainer = Trainer(distributed_backend='ddp')
-
-    # change to distributed data parallel (gpus > 1)
-    trainer = Trainer(distributed_backend='ddp2')
-
-If you request multiple nodes, the back-end will auto-switch to ddp.
- We recommend you use DistributedDataparallel even for single-node multi-GPU training.
- It is MUCH faster than DP but *may* have configuration issues depending on your cluster.
-
-For a deeper understanding of what lightning is doing, feel free to read this
- `guide <https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565>`_.
-
-Distributed and 16-bit precision
---------------------------------
-
-Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
- not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.
-
-Below are the possible configurations we support.
-
-+-------+---------+----+-----+---------+------------------------------------------------------------+
-| 1 GPU | 1+ GPUs | DP | DDP | 16-bit  | command                                                    |
-+=======+=========+====+=====+=========+============================================================+
-| Y     |         |    |     |         | `Trainer(gpus=1)`                                          |
-+-------+---------+----+-----+---------+------------------------------------------------------------+
-| Y     |         |    |     | Y       | `Trainer(gpus=1, use_amp=True)`                            |
-+-------+---------+----+-----+---------+------------------------------------------------------------+
-|       | Y       | Y  |     |         | `Trainer(gpus=k, distributed_backend='dp')`                |
-+-------+---------+----+-----+---------+------------------------------------------------------------+
-|       | Y       |    | Y   |         | `Trainer(gpus=k, distributed_backend='ddp')`               |
-+-------+---------+----+-----+---------+------------------------------------------------------------+
-|       | Y       |    | Y   | Y       | `Trainer(gpus=k, distributed_backend='ddp', use_amp=True)` |
-+-------+---------+----+-----+---------+------------------------------------------------------------+
-
-You also have the option of specifying which GPUs to use by passing a list:
-
-.. code-block:: python
-
-    # DEFAULT (int) specifies how many GPUs to use.
-    Trainer(gpus=k)
-
-    # Above is equivalent to
-    Trainer(gpus=list(range(k)))
-
-    # You specify which GPUs (don't use if running on cluster)
-    Trainer(gpus=[0, 1])
-
-    # can also be a string
-    Trainer(gpus='0, 1')
-
-    # can also be -1 or '-1', this uses all available GPUs
-    # this is equivalent to list(range(torch.cuda.available_devices()))
-    Trainer(gpus=-1)
-
-
-CUDA flags
-----------
-
-CUDA flags make certain GPUs visible to your script.
- Lightning sets these for you automatically, there's NO NEED to do this yourself.
-
-.. code-block:: python
-
-    # lightning will set according to what you give the trainer
-    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-    os.environ["CUDA_VISIBLE_DEVICES"] = "0"
-
-
-However, when using a cluster, Lightning will NOT set these flags (and you should not either).
- SLURM will set these for you.
-
-16-bit mixed precision
-----------------------
-
-16 bit precision can cut your memory footprint by half. If using volta architecture GPUs
- it can give a dramatic training speed-up as well.
- First, install apex (if install fails, look `here <https://github.com/NVIDIA/apex>`__)::
-
-    $ git clone https://github.com/NVIDIA/apex
-    $ cd apex
-
-    # ------------------------
-    # OPTIONAL: on your cluster you might need to load cuda 10 or 9
-    # depending on how you installed PyTorch
-
-    # see available modules
-    module avail
-
-    # load correct cuda before install
-    module load cuda-10.0
-    # ------------------------
-
-    # make sure you've loaded a cuda version > 4.0 and < 7.0
-    module load gcc-6.1.0
-
-    $ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
-
-
-then set this use_amp to True.::
-
-    # DEFAULT
-    trainer = Trainer(amp_level='O2', use_amp=False)
-
-
-Single-gpu
-----------
-
-Make sure you're on a GPU machine.::
-
-    # DEFAULT
-    trainer = Trainer(gpus=1)
-
-Multi-gpu
----------
-
-Make sure you're on a GPU machine. You can set as many GPUs as you want.
- In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
-
-.. code-block:: python
-
-    # to use DataParallel
-    trainer = Trainer(gpus=8, distributed_backend='dp')
-
-    # RECOMMENDED use DistributedDataParallel
-    trainer = Trainer(gpus=8, distributed_backend='ddp')
-
-Custom device selection
------------------------
-
-The number of GPUs can also be selected with a list of indices or a string containing
-a comma separated list of GPU ids.
-The table below lists examples of possible input formats and how they are interpreted by Lightning.
-Note in particular the difference between `gpus=0`, `gpus=[0]` and `gpus="0"`.
-
-+---------------+-----------+---------------------+---------------------------------+
-| `gpus`        | Type      | Parsed              | Meaning                         |
-+===============+===========+=====================+=================================+
-| None          | NoneType  | None                | CPU                             |
-+---------------+-----------+---------------------+---------------------------------+
-| 0             | int       | None                | CPU                             |
-+---------------+-----------+---------------------+---------------------------------+
-| 3             | int       | [0, 1, 2]           | first 3 GPUs                    |
-+---------------+-----------+---------------------+---------------------------------+
-| -1            | int       | [0, 1, 2, ...]      | all available GPUs              |
-+---------------+-----------+---------------------+---------------------------------+
-| [0]           | list      | [0]                 | GPU 0                           |
-+---------------+-----------+---------------------+---------------------------------+
-| [1, 3]        | list      | [1, 3]              | GPUs 1 and 3                    |
-+---------------+-----------+---------------------+---------------------------------+
-| "0"           | str       | [0]                 | GPU 0                           |
-+---------------+-----------+---------------------+---------------------------------+
-| "3"           | str       | [3]                 | GPU 3                           |
-+---------------+-----------+---------------------+---------------------------------+
-| "1, 3"        | str       | [1, 3]              | GPUs 1 and 3                    |
-+---------------+-----------+---------------------+---------------------------------+
-| "-1"          | str       | [0, 1, 2, ...]      | all available GPUs              |
-+---------------+-----------+---------------------+---------------------------------+
-
-
-Multi-node
-----------
-
-Multi-node training is easily done by specifying these flags.
-
-.. code-block:: python
-
-    # train on 12*8 GPUs
-    trainer = Trainer(gpus=8, num_nodes=12, distributed_backend='ddp')
-
-
-You must configure your job submission script correctly for the trainer to work.
- Here is an example script for the above trainer configuration.
-
-.. code-block:: bash
-
-    #!/bin/bash -l
-
-    # SLURM SUBMIT SCRIPT
-    #SBATCH --nodes=12
-    #SBATCH --gres=gpu:8
-    #SBATCH --ntasks-per-node=8
-    #SBATCH --mem=0
-    #SBATCH --time=0-02:00:00
-
-    # activate conda env
-    conda activate my_env
-
-    # -------------------------
-    # OPTIONAL
-    # -------------------------
-    # debugging flags (optional)
-    # export NCCL_DEBUG=INFO
-    # export PYTHONFAULTHANDLER=1
-
-    # PyTorch comes with prebuilt NCCL support... but if you have issues with it
-    # you might need to load the latest version from your  modules
-    # module load NCCL/2.4.7-1-cuda.10.0
-
-    # on your cluster you might need these:
-    # set the network interface
-    # export NCCL_SOCKET_IFNAME=^docker0,lo
-    # -------------------------
-
-    # random port between 12k and 20k
-    export MASTER_PORT=$((12000 + RANDOM % 20000))
-
-    # run script from above
-    python my_main_file.py
-
-.. note:: When running in DDP mode, any errors in your code will show up as an NCCL issue.
- Set the `NCCL_DEBUG=INFO` flag to see the ACTUAL error.
-
-Normally now you would need to add a distributed sampler to your dataset, however
-Lightning automates this for you. But if you still need to set a sampler Lightning will
-not interfere nor automate it.
-
-Here's an example of how to add your own sampler (again no need with Lightning).
-
-.. code-block:: python
-
-    # ie: this:
-    dataset = myDataset()
-    dataloader = Dataloader(dataset)
-
-    # becomes:
-    dataset = myDataset()
-    dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-    dataloader = Dataloader(dataset, sampler=dist_sampler)
-
-
-Auto-slurm-job-submission
--------------------------
-
-Instead of manually building SLURM scripts, you can use the
-`SlurmCluster object <https://williamfalcon.github.io/test-tube/hpc/SlurmCluster>`_
-to do this for you. The SlurmCluster can also run a grid search if you pass
-in a `HyperOptArgumentParser
-<https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser>`_.
-
-Here is an example where you run a grid search of 9 combinations of hyperparams.
-The full examples are
-`here <https://github.com/PyTorchLightning/pytorch-lightning/tree/master/pl_examples/multi_node_examples>`__.
-
-.. code-block:: python
-
-    # grid search 3 values of learning rate and 3 values of number of layers for your net
-    # this generates 9 experiments (lr=1e-3, layers=16), (lr=1e-3, layers=32),
-    #  (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
-    parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
-    parser.opt_list('--learning_rate', default=0.001, type=float,
-                    options=[1e-3, 1e-2, 1e-1], tunable=True)
-    parser.opt_list('--layers', default=1, type=float, options=[16, 32, 64], tunable=True)
-    hyperparams = parser.parse_args()
-
-    # Slurm cluster submits 9 jobs, each with a set of hyperparams
-    cluster = SlurmCluster(
-        hyperparam_optimizer=hyperparams,
-        log_path='/some/path/to/save',
-    )
-
-    # OPTIONAL FLAGS WHICH MAY BE CLUSTER DEPENDENT
-    # which interface your nodes use for communication
-    cluster.add_command('export NCCL_SOCKET_IFNAME=^docker0,lo')
-
-    # see output of the NCCL connection process
-    # NCCL is how the nodes talk to each other
-    cluster.add_command('export NCCL_DEBUG=INFO')
-
-    # setting a master port here is a good idea.
-    cluster.add_command('export MASTER_PORT=%r' % PORT)
-
-    # ************** DON'T FORGET THIS ***************
-    # MUST load the latest NCCL version
-    cluster.load_modules(['NCCL/2.4.7-1-cuda.10.0'])
-
-    # configure cluster
-    cluster.per_experiment_nb_nodes = 12
-    cluster.per_experiment_nb_gpus = 8
-
-    cluster.add_slurm_cmd(cmd='ntasks-per-node', value=8, comment='1 task per gpu')
-
-    # submit a script with 9 combinations of hyper params
-    # (lr=1e-3, layers=16), (lr=1e-3, layers=32), (lr=1e-3, layers=64), ... (lr=1e-1, layers=64)
-    cluster.optimize_parallel_cluster_gpu(
-        main,
-        nb_trials=9, # how many permutations of the grid search to run
-        job_name='name_for_squeue'
-    )
-
-
-The other option is that you generate scripts on your own via a bash command or use another library...
-
-Self-balancing architecture
----------------------------
-
-Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.
+Root module for all distributed operations in Lightning.
+Currently supports training on CPU, GPU (dp, ddp, ddp2, horovod) and TPU.
 
 """
 
@@ -343,7 +10,7 @@ from abc import ABC, abstractmethod
 import time
 import random
 import torch
-from typing import Union
+from typing import Union, Callable, Any, List, Optional
 
 from pytorch_lightning import _logger as log
 from pytorch_lightning.loggers import LightningLoggerBase
@@ -351,6 +18,7 @@ from pytorch_lightning.overrides.data_parallel import (
     LightningDistributedDataParallel,
     LightningDataParallel,
 )
+from pytorch_lightning.utilities import move_data_to_device
 from pytorch_lightning.utilities.exceptions import MisconfigurationException
 from pytorch_lightning.utilities.distributed import rank_zero_only
 
@@ -389,7 +57,6 @@ class TrainerDPMixin(ABC):
     root_gpu: ...
     amp_level: str
     precision: ...
-    current_tpu_idx: ...
     proc_rank: int
     tpu_local_core_rank: int
     tpu_global_core_rank: int
@@ -398,6 +65,7 @@ class TrainerDPMixin(ABC):
     data_parallel_device_ids: ...
     logger: Union[LightningLoggerBase, bool]
     progress_bar_callback: ...
+    tpu_id: int
 
     @property
     @abstractmethod
@@ -422,7 +90,6 @@ class TrainerDPMixin(ABC):
 
         for m in [model, ref_model]:
             m.trainer = self
-            m.on_gpu = self.on_gpu
             m.use_dp = self.use_dp
             m.use_ddp2 = self.use_ddp2
             m.use_ddp = self.use_ddp
@@ -432,63 +99,54 @@ class TrainerDPMixin(ABC):
             m.use_tpu = self.use_tpu
             m.tpu_local_core_rank = self.tpu_local_core_rank
             m.tpu_global_core_rank = self.tpu_global_core_rank
-            m._device = self._device
 
-    def transfer_batch_to_tpu(self, batch):
-        return self.__transfer_data_to_device(batch, device='tpu')
+    def transfer_batch_to_tpu(self, batch: Any, tpu_id: Optional[int] = None):
+        """
+        Transfers the data to the TPU.
 
-    def transfer_batch_to_gpu(self, batch, gpu_id):
-        return self.__transfer_data_to_device(batch, device='gpu', gpu_id=gpu_id)
+        Args:
+            batch: A tensor or collection of tensors.
+            tpu_id: The id of the TPU core. If omitted, the first available core is chosen.
 
-    def __transfer_data_to_device(self, batch, device, gpu_id=None):
-        if device == 'tpu' and XLA_AVAILABLE:
-            # base case: object can be directly moved using `to`
-            if callable(getattr(batch, 'to', None)):
-                return batch.to(xm.xla_device())
+        Return:
+            the tensor on the TPU device.
 
-        if device == 'gpu':
-            # base case: object can be directly moved using `cuda` or `to`
-            if callable(getattr(batch, 'cuda', None)):
-                # non_blocking will be ignored if tensor is not pinned.
-                # so we can always set it to True
-                return batch.cuda(gpu_id, non_blocking=True)
+        See Also:
+            - :func:`~pytorch_lightning.utilities.apply_func.move_data_to_device`
+        """
+        if not XLA_AVAILABLE:
+            raise MisconfigurationException(
+                'Requested to transfer batch to TPU but XLA is not available.'
+                ' Are you sure this machine has TPUs?'
+            )
+        device = xm.xla_device(tpu_id)
+        return self.__transfer_batch_to_device(batch, device)
 
-            if callable(getattr(batch, 'to', None)):
-                # non_blocking will be ignored if tensor is not pinned.
-                # so we can always set it to True
-                return batch.to(torch.device('cuda', gpu_id), non_blocking=True)
+    def transfer_batch_to_gpu(self, batch: Any, gpu_id: Optional[int] = None):
+        """
+        Transfers the data to the GPU.
 
-        # when list
-        if isinstance(batch, list):
-            for i, x in enumerate(batch):
-                batch[i] = self.__transfer_data_to_device(x, device, gpu_id)
-            return batch
+        Args:
+            batch: A tensor or collection of tensors.
+            gpu_id: The id of the GPU device. If omitted, the first available GPU is chosen.
 
-        # when tuple
-        if isinstance(batch, tuple):
-            # when namedtuple
-            if hasattr(batch, '_fields'):
-                elem_type = type(batch)
-                return elem_type(*(self.__transfer_data_to_device(x, device, gpu_id) for x in batch))
-            else:
-                batch = list(batch)
-                for i, x in enumerate(batch):
-                    batch[i] = self.__transfer_data_to_device(x, device, gpu_id)
-                return tuple(batch)
+        Return:
+            the tensor on the GPU device.
 
-        # when dict
-        if isinstance(batch, dict):
-            for k, v in batch.items():
-                batch[k] = self.__transfer_data_to_device(v, device, gpu_id)
+        See Also:
+            - :func:`~pytorch_lightning.utilities.apply_func.move_data_to_device`
+        """
+        device = torch.device('cuda', gpu_id)
+        return self.__transfer_batch_to_device(batch, device)
 
-            return batch
-
-        # nothing matches, return the value as is without transform
-        return batch
+    def __transfer_batch_to_device(self, batch: Any, device: torch.device):
+        model = self.get_model()
+        if model is not None:
+            return model.transfer_batch_to_device(batch, device)
+        return move_data_to_device(batch, device)
 
     def single_gpu_train(self, model):
         model.cuda(self.root_gpu)
-        self._device = torch.device('cuda', self.root_gpu)
 
         # CHOOSE OPTIMIZER
         # allow for lr schedulers as well
@@ -499,13 +157,14 @@ class TrainerDPMixin(ABC):
             # An example
             model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
             self.optimizers = optimizers
+            self.reinit_scheduler_properties(self.optimizers, self.lr_schedulers)
 
         self.run_pretrain_routine(model)
 
     def tpu_train(self, tpu_core_idx, model):
         # put model on tpu
-        model.to(xm.xla_device())
-        self._device = xm.xla_device()
+        self._device = xm.xla_device(self.tpu_id) if self.tpu_id is not None else xm.xla_device()
+        model.to(self._device)
 
         # get the appropriate tpu ranks
         self.tpu_local_core_rank = xm.get_local_ordinal()
@@ -515,8 +174,6 @@ class TrainerDPMixin(ABC):
         if self.tpu_global_core_rank != 0 and self.progress_bar_callback is not None:
             self.progress_bar_callback.disable()
 
-        # track current tpu
-        self.current_tpu_idx = tpu_core_idx
         self.proc_rank = self.tpu_local_core_rank
         rank_zero_only.rank = self.proc_rank
 
@@ -545,7 +202,6 @@ class TrainerDPMixin(ABC):
         self.optimizers, self.lr_schedulers, self.optimizer_frequencies = self.init_optimizers(model)
 
         model.cuda(self.root_gpu)
-        self._device = torch.device('cuda', self.root_gpu)
 
         # hack forward to do autocast for the user
         model_autocast_original_forward = model.forward
@@ -564,6 +220,7 @@ class TrainerDPMixin(ABC):
                     f' We recommend you switch to ddp if you want to use amp')
             else:
                 model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
+                self.reinit_scheduler_properties(optimizers, self.lr_schedulers)
 
         # create list of device ids
         device_ids = self.data_parallel_device_ids
@@ -585,7 +242,6 @@ class TrainerDPMixin(ABC):
             assert self.root_gpu == hvd.local_rank()
             torch.cuda.set_device(self.root_gpu)
             model.cuda(self.root_gpu)
-            self._device = torch.device('cuda', self.root_gpu)
 
         # avoid duplicating progress bar
         if hvd.rank() != 0 and self.progress_bar_callback is not None:
@@ -605,6 +261,7 @@ class TrainerDPMixin(ABC):
             # An example
             model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
             self.optimizers = optimizers
+            self.reinit_scheduler_properties(self.optimizers, self.lr_schedulers)
 
         # Horovod: broadcast parameters & optimizer state to ensure consistent initialization
         hvd.broadcast_parameters(model.state_dict(), root_rank=0)
@@ -647,26 +304,27 @@ def normalize_parse_gpu_string_input(s):
         return s
 
 
-def get_all_available_gpus():
+def get_all_available_gpus() -> List[int]:
     """
-    :return: a list of all available gpus
+    Returns:
+         a list of all available gpus
     """
     return list(range(torch.cuda.device_count()))
 
 
-def check_gpus_data_type(gpus):
-    """
-    :param gpus: gpus parameter as passed to the Trainer
-        Function checks that it is one of: None, Int, String or List
-        Throws otherwise
-    :return: return unmodified gpus variable
+def check_gpus_data_type(gpus: Any) -> None:
     """
+    Checks that the gpus argument is one of: None, Int, String or List.
+    Raises a MisconfigurationException otherwise.
 
+    Args:
+        gpus: parameter as passed to the Trainer
+    """
     if gpus is not None and (not isinstance(gpus, (int, str, list)) or isinstance(gpus, bool)):
         raise MisconfigurationException("GPUs must be int, string or list of ints or None.")
 
 
-def normalize_parse_gpu_input_to_list(gpus):
+def normalize_parse_gpu_input_to_list(gpus: Union[int, List[int]]) -> Optional[List[int]]:
     assert gpus is not None
     if isinstance(gpus, list):
         return gpus
@@ -680,16 +338,30 @@ def normalize_parse_gpu_input_to_list(gpus):
     return list(range(gpus))
 
 
-def sanitize_gpu_ids(gpus):
+def sanitize_gpu_ids(gpus: List[int]) -> List[int]:
     """
-    :param gpus: list of ints corresponding to GPU indices
-        Checks that each of the GPUs in the list is actually available.
-        Throws if any of the GPUs is not available.
-    :return: unmodified gpus variable
+    Checks that each of the GPUs in the list is actually available.
+    Raises a MisconfigurationException if any of the GPUs is not available.
+
+    Args:
+        gpus: list of ints corresponding to GPU indices
+
+    Returns:
+        unmodified gpus variable
     """
     all_available_gpus = get_all_available_gpus()
+    misconfig = False
     for gpu in gpus:
         if gpu not in all_available_gpus:
+            misconfig = True
+
+    if misconfig:
+        # sometimes auto ddp might have different flags
+        # but this is not what the user intended
+        # correct for the user
+        if len(gpus) == len(all_available_gpus):
+            gpus = all_available_gpus
+        else:
             raise MisconfigurationException(f"""
                 You requested GPUs: {gpus}
                 But your machine only has: {all_available_gpus}
@@ -697,18 +369,23 @@ def sanitize_gpu_ids(gpus):
     return gpus
 
 
-def parse_gpu_ids(gpus):
+def parse_gpu_ids(gpus: Union[int, str, List]) -> Optional[List[int]]:
     """
-    :param gpus: Int, string or list
-        An int -1 or string '-1' indicate that all available GPUs should be used.
-        A list of ints or a string containing list of comma separated integers
-        indicates specific GPUs to use
-        An int 0 means that no GPUs should be used
-        Any int N > 0 indicates that GPUs [0..N) should be used.
-    :return: List of gpus to be used
+    Parses the GPU ids given in the format as accepted by the
+    :class:`~pytorch_lightning.trainer.Trainer`.
 
-        If no GPUs are available but the value of gpus variable indicates request for GPUs
-        then a misconfiguration exception is raised.
+    Args:
+        gpus: An int -1 or string '-1' indicate that all available GPUs should be used.
+            A list of ints or a string containing list of comma separated integers
+            indicates specific GPUs to use.
+            An int 0 means that no GPUs should be used.
+            Any int N > 0 indicates that GPUs [0..N) should be used.
+
+    Returns:
+        a list of gpus to be used or ``None`` if no GPUs were requested
+
+    If no GPUs are available but the value of gpus variable indicates request for GPUs
+    then a MisconfigurationException is raised.
     """
 
     # nothing was passed into the GPUs argument
@@ -734,10 +411,13 @@ def parse_gpu_ids(gpus):
     return gpus
 
 
-def determine_root_gpu_device(gpus):
+def determine_root_gpu_device(gpus: List[int]) -> Optional[int]:
     """
-    :param gpus: non empty list of ints representing which gpus to use
-    :return: designated root GPU device
+    Args:
+        gpus: non-empty list of ints representing which gpus to use
+
+    Returns:
+        designated root GPU device id
     """
     if gpus is None:
         return None
@@ -751,26 +431,33 @@ def determine_root_gpu_device(gpus):
     return root_gpu
 
 
-def retry_jittered_backoff(f, num_retries=5):
-    # Based on:
-    # https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
-    cap = 1.0                  # max sleep time is 1s
-    base = 0.01                # initial sleep time is 10ms
-    sleep = base               # initial sleep time is 10ms
+def retry_jittered_backoff(func: Callable, num_retries: int = 5, cap_delay: float = 1.0, base_delay: float = 0.01):
+    """Retry jittered backoff.
+
+    Based on:
+    https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
+
+    Args:
+        func: tested function
+        num_retries: number of tries
+        cap_delay: max sleep time
+        base_delay: initial sleep time is 10ms
+    """
+    sleep_delay = base_delay         # initial sleep time is 10ms
 
     for i in range(num_retries):
         try:
-            return f()
-        except RuntimeError as e:
+            return func()
+        except RuntimeError as err:
             if i == num_retries - 1:
-                raise e
+                raise err
             else:
                 continue
-        time.sleep(sleep)
-        sleep = min(cap, random.uniform(base, sleep * 3))
+        time.sleep(sleep_delay)
+        sleep_delay = min(cap_delay, random.uniform(base_delay, sleep_delay * 3))
 
 
-def pick_single_gpu(exclude_gpus=[]):
+def pick_single_gpu(exclude_gpus: list):
     for i in range(torch.cuda.device_count()):
         if i in exclude_gpus:
             continue
@@ -784,9 +471,9 @@ def pick_single_gpu(exclude_gpus=[]):
     raise RuntimeError("No GPUs available.")
 
 
-def pick_multiple_gpus(n):
+def pick_multiple_gpus(nb):
     picked = []
-    for _ in range(n):
+    for _ in range(nb):
         picked.append(pick_single_gpu(exclude_gpus=picked))
 
     return picked

pytorch_lightning/trainer/evaluation_loop.py

@@ -174,6 +174,7 @@ class TrainerEvaluationLoopMixin(ABC):
     val_dataloaders: DataLoader
     use_tpu: bool
     reload_dataloaders_every_epoch: ...
+    tpu_id: int
 
     # Callback system
     on_validation_batch_start: Callable
@@ -249,7 +250,7 @@ class TrainerEvaluationLoopMixin(ABC):
 
             # on TPU we have to wrap it under the ParallelLoader
             if self.use_tpu:
-                device = xm.xla_device()
+                device = xm.xla_device(self.tpu_id)
                 dataloader = xla_pl.ParallelLoader(dataloader, [device])
                 dataloader = dataloader.per_device_loader(device)
 
@@ -334,20 +335,13 @@ class TrainerEvaluationLoopMixin(ABC):
         return eval_results
 
     def run_evaluation(self, test_mode: bool = False):
-        # when testing make sure user defined a test step
-        if test_mode and not self.is_overridden('test_step'):
-            raise MisconfigurationException(
-                "You called `.test()` without defining model's `.test_step()`."
-                " Please define and try again")
-
         # hook
         model = self.get_model()
         model.on_pre_performance_check()
 
         # select dataloaders
         if test_mode:
-            if self.test_dataloaders is None:
-                self.reset_test_dataloader(model)
+            self.reset_test_dataloader(model)
 
             dataloaders = self.test_dataloaders
             max_batches = self.num_test_batches
@@ -440,7 +434,7 @@ class TrainerEvaluationLoopMixin(ABC):
 
         # TPU data  transfer
         if self.use_tpu:
-            batch = self.transfer_batch_to_tpu(batch)
+            batch = self.transfer_batch_to_tpu(batch, self.tpu_id)
             args[0] = batch
 
         # CPU, TPU or gpu step

pytorch_lightning/trainer/lr_finder.py

@@ -2,7 +2,7 @@
 Trainer Learning Rate Finder
 """
 from abc import ABC, abstractmethod
-from typing import Optional
+from typing import Optional, Sequence
 
 import numpy as np
 import torch
@@ -20,6 +20,8 @@ from pytorch_lightning.utilities import rank_zero_warn
 
 
 class TrainerLRFinderMixin(ABC):
+    default_root_dir: str
+
     @abstractmethod
     def save_checkpoint(self, *args):
         """Warning: this is just empty shell for code implemented in other class."""
@@ -35,17 +37,17 @@ class TrainerLRFinderMixin(ABC):
         # TODO: log lr.results to self.logger
         if isinstance(self.auto_lr_find, str):
             # Try to find requested field, may be nested
-            if _nested_hasattr(model.hparams, self.auto_lr_find):
-                _nested_setattr(model.hparams, self.auto_lr_find, lr)
+            if _nested_hasattr(model, self.auto_lr_find):
+                _nested_setattr(model, self.auto_lr_find, lr)
             else:
                 raise MisconfigurationException(
                     f'`auto_lr_find` was set to {self.auto_lr_find}, however'
                     ' could not find this as a field in `model.hparams`.')
         else:
-            if hasattr(model.hparams, 'lr'):
-                model.hparams.lr = lr
-            elif hasattr(model.hparams, 'learning_rate'):
-                model.hparams.learning_rate = lr
+            if hasattr(model, 'lr'):
+                model.lr = lr
+            elif hasattr(model, 'learning_rate'):
+                model.learning_rate = lr
             else:
                 raise MisconfigurationException(
                     'When auto_lr_find is set to True, expects that hparams'
@@ -196,11 +198,9 @@ class TrainerLRFinderMixin(ABC):
             'callbacks': self.callbacks,
             'logger': self.logger,
             'max_steps': self.max_steps,
-            'progress_bar_refresh_rate': self.progress_bar_refresh_rate,
             'checkpoint_callback': self.checkpoint_callback,
             'early_stop_callback': self.early_stop_callback,
             'enable_early_stop': self.enable_early_stop,
-            'progress_bar_callback': self.progress_bar_callback,
             'configure_optimizers': model.configure_optimizers,
         }
 
@@ -209,11 +209,9 @@ class TrainerLRFinderMixin(ABC):
         self.logger = self.__dumped_params['logger']
         self.callbacks = self.__dumped_params['callbacks']
         self.max_steps = self.__dumped_params['max_steps']
-        self.progress_bar_refresh_rate = self.__dumped_params['progress_bar_refresh_rate']
         self.checkpoint_callback = self.__dumped_params['checkpoint_callback']
         self.early_stop_callback = self.__dumped_params['early_stop_callback']
         self.enable_early_stop = self.__dumped_params['enable_early_stop']
-        self.progress_bar_callback = self.__dumped_params['progress_bar_callback']
         model.configure_optimizers = self.__dumped_params['configure_optimizers']
         del self.__dumped_params
 
@@ -321,8 +319,9 @@ class _LRFinder(object):
 
         """
         try:
-            loss = self.results["loss"][skip_begin:-skip_end]
-            min_grad = (np.gradient(np.array(loss))).argmin()
+            loss = np.array(self.results["loss"][skip_begin:-skip_end])
+            loss = loss[np.isfinite(loss)]
+            min_grad = np.gradient(loss).argmin()
             self._optimal_idx = min_grad + skip_begin
             return self.results["lr"][self._optimal_idx]
         except Exception:
@@ -349,7 +348,7 @@ class _LRCallback(Callback):
     """
     def __init__(self, num_training: int,
                  early_stop_threshold: float = 4.0,
-                 progress_bar_refresh_rate: bool = False,
+                 progress_bar_refresh_rate: int = 0,
                  beta: float = 0.98):
         self.num_training = num_training
         self.early_stop_threshold = early_stop_threshold
@@ -413,6 +412,8 @@ class _LinearLR(_LRScheduler):
 
         last_epoch: the index of last epoch. Default: -1.
     """
+    last_epoch: int
+    base_lrs: Sequence
 
     def __init__(self,
                  optimizer: torch.optim.Optimizer,
@@ -453,6 +454,8 @@ class _ExponentialLR(_LRScheduler):
 
         last_epoch: the index of last epoch. Default: -1.
     """
+    last_epoch: int
+    base_lrs: Sequence
 
     def __init__(self,
                  optimizer: torch.optim.Optimizer,

pytorch_lightning/trainer/optimizers.py

@@ -108,6 +108,19 @@ class TrainerOptimizersMixin(ABC):
                                  'is a invalid input.')
         return lr_schedulers
 
+    def reinit_scheduler_properties(self, optimizers: list, schedulers: list):
+        # Reinitialize optimizer.step properties added by schedulers
+        for scheduler in schedulers:
+            for optimizer in optimizers:
+                scheduler = scheduler['scheduler']
+                # check that we dont mix users optimizers and schedulers
+                if scheduler.optimizer == optimizer:
+                    # Find the mro belonging to the base lr scheduler class
+                    for i, mro in enumerate(scheduler.__class__.__mro__):
+                        if mro == optim.lr_scheduler._LRScheduler:
+                            idx = i
+                    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
+
 
 class _MockOptimizer(Optimizer):
     """The `_MockOptimizer` will be used inplace of an optimizer in the event that `None`

pytorch_lightning/trainer/trainer.py

@@ -35,7 +35,6 @@ from pytorch_lightning.trainer.lr_finder import TrainerLRFinderMixin
 from pytorch_lightning.utilities.exceptions import MisconfigurationException
 from pytorch_lightning.utilities import rank_zero_warn, parsing
 
-
 try:
     from apex import amp
 except ImportError:
@@ -82,7 +81,7 @@ class Trainer(
         'gradient_clip', 'nb_gpu_nodes', 'max_nb_epochs', 'min_nb_epochs',
         'add_row_log_interval', 'nb_sanity_val_steps', 'tng_tqdm_dic',
     )
-    DEPRECATED_IN_0_9 = ('use_amp', 'show_progress_bar', 'training_tqdm_dict')
+    DEPRECATED_IN_0_9 = ('use_amp', 'show_progress_bar', 'training_tqdm_dict', 'num_tpu_cores')
 
     def __init__(
             self,
@@ -97,11 +96,11 @@ class Trainer(
             num_processes: int = 1,
             gpus: Optional[Union[List[int], str, int]] = None,
             auto_select_gpus: bool = False,
-            num_tpu_cores: Optional[int] = None,
+            tpu_cores: Optional[Union[List[int], int]] = None,
             log_gpu_memory: Optional[str] = None,
             progress_bar_refresh_rate: int = 1,
             overfit_pct: float = 0.0,
-            track_grad_norm: int = -1,
+            track_grad_norm: Union[int, float, str] = -1,
             check_val_every_n_epoch: int = 1,
             fast_dev_run: bool = False,
             accumulate_grad_batches: Union[int, Dict[int, int], List[list]] = 1,
@@ -119,7 +118,7 @@ class Trainer(
             distributed_backend: Optional[str] = None,
             precision: int = 32,
             print_nan_grads: bool = False,  # backward compatible, todo: remove in v0.9.0
-            weights_summary: Optional[str] = 'full',
+            weights_summary: Optional[str] = 'top',
             weights_save_path: Optional[str] = None,
             num_sanity_val_steps: int = 2,
             truncated_bptt_steps: Optional[int] = None,
@@ -130,9 +129,9 @@ class Trainer(
             reload_dataloaders_every_epoch: bool = False,
             auto_lr_find: Union[bool, str] = False,
             replace_sampler_ddp: bool = True,
-            progress_bar_callback: Optional[Union[ProgressBarBase, bool]] = True,
             terminate_on_nan: bool = False,
             auto_scale_batch_size: Union[str, bool] = False,
+            num_tpu_cores: Optional[int] = None,  # backward compatible, todo: remove in v0.9.0
             amp_level: str = 'O1',  # backward compatible, todo: remove in v0.8.0
             default_save_path=None,  # backward compatible, todo: remove in v0.8.0
             gradient_clip=None,  # backward compatible, todo: remove in v0.8.0
@@ -142,7 +141,6 @@ class Trainer(
             use_amp=None,  # backward compatible, todo: remove in v0.9.0
             show_progress_bar=None,  # backward compatible, todo: remove in v0.9.0
             nb_sanity_val_steps=None,  # backward compatible, todo: remove in v0.8.0
-            **kwargs
     ):
         r"""
 
@@ -189,7 +187,10 @@ class Trainer(
                 GPUs are configured to be in "exclusive mode", such
                 that only one process at a time can access them.
 
-            num_tpu_cores: How many TPU cores to train on (1 or 8).
+            tpu_cores: How many TPU cores to train on (1 or 8) / Single TPU to train on [1]
+
+            num_tpu_cores: How many TPU cores to train on (1 or 8)
+                .. warning:: .. deprecated:: 0.7.6. Will remove 0.9.0.
 
             log_gpu_memory: None, 'min_max', 'all'. Might slow performance
 
@@ -203,7 +204,7 @@ class Trainer(
 
             overfit_pct: How much of training-, validation-, and test dataset to check.
 
-            track_grad_norm: -1 no tracking. Otherwise tracks that norm
+            track_grad_norm: -1 no tracking. Otherwise tracks that p-norm. May be set to 'inf' infinity-norm.
 
             check_val_every_n_epoch: Check val every n train epochs.
 
@@ -286,7 +287,7 @@ class Trainer(
 
             auto_lr_find: If set to True, will `initially` run a learning rate finder,
                 trying to optimize initial learning for faster convergence. Sets learning
-                rate in self.hparams.lr | self.hparams.learning_rate in the lightning module.
+                rate in self.lr or self.learning_rate in the LightningModule.
                 To use a different key, set a string instead of True with the key name.
 
             replace_sampler_ddp: Explicitly enables or disables sampler replacement.
@@ -301,10 +302,11 @@ class Trainer(
 
             auto_scale_batch_size: If set to True, will `initially` run a batch size
                 finder trying to find the largest batch size that fits into memory.
-                The result will be stored in self.hparams.batch_size in the LightningModule.
+                The result will be stored in self.batch_size in the LightningModule.
                 Additionally, can be set to either `power` that estimates the batch size through
                 a power search or `binsearch` that estimates the batch size through a binary search.
         """
+        super().__init__()
 
         self.deterministic = deterministic
         torch.backends.cudnn.deterministic = self.deterministic
@@ -338,19 +340,33 @@ class Trainer(
             self.gradient_clip = gradient_clip
 
         self.check_val_every_n_epoch = check_val_every_n_epoch
-        self.track_grad_norm = track_grad_norm
+
+        if not isinstance(track_grad_norm, (int, float)) and track_grad_norm != 'inf':
+            raise MisconfigurationException(
+                "track_grad_norm can be an int, a float or 'inf' (infinity norm).")
+        self.track_grad_norm = float(track_grad_norm)
+
         self.on_gpu = True if (gpus and torch.cuda.is_available()) else False
 
         # tpu config
-        self.on_tpu = num_tpu_cores is not None
-        self.num_tpu_cores = num_tpu_cores
-        assert num_tpu_cores in [1, 8, None], 'num_tpu_cores can only be 1 or 8'
+        if num_tpu_cores is not None:
+            rank_zero_warn("Argument `num_tpu_cores` is now set by `tpu_cores` since v0.7.6"
+                           " and this argument will be removed in v0.9.0", DeprecationWarning)
+
+        if tpu_cores is None:
+            tpu_cores = num_tpu_cores
+        self.on_tpu = tpu_cores is not None
+        self.tpu_cores = tpu_cores
+        assert self.tpu_cores in (1, 8, None) or (
+            isinstance(self.tpu_cores, (list, tuple, set)) and len(self.tpu_cores) == 1
+        ), '`tpu_cores` can only be 1, 8 or [<1-8>]'
+
+        self.tpu_id = tpu_cores[0] if isinstance(tpu_cores, list) else None
 
         if num_processes != 1 and distributed_backend != "ddp_cpu":
             rank_zero_warn("num_processes is only used for distributed_backend=\"ddp_cpu\". Ignoring it.")
         self.num_processes = num_processes
 
-        self.process_position = process_position
         self.weights_summary = weights_summary
 
         self.max_epochs = max_epochs
@@ -387,6 +403,7 @@ class Trainer(
 
         self.auto_lr_find = auto_lr_find
         self.auto_scale_batch_size = auto_scale_batch_size
+        self._is_data_prepared = False
         self.replace_sampler_ddp = replace_sampler_ddp
 
         self.truncated_bptt_steps = truncated_bptt_steps
@@ -473,16 +490,15 @@ class Trainer(
         # distributed backend choice
         self.distributed_backend = distributed_backend
         self.set_distributed_mode(distributed_backend)
-        self._device = torch.device('cpu')
 
         # override dist backend when using tpus
         if self.on_tpu:
             self.init_tpu()
-            self.current_tpu_idx = None
 
         # init flags for SLURM+ddp to work
         self.proc_rank = 0
         self.world_size = 1
+        self.interactive_ddp_procs = []
         self.configure_slurm_ddp(self.num_nodes)
         self.node_rank = self.determine_ddp_node_rank()
 
@@ -493,9 +509,7 @@ class Trainer(
         if show_progress_bar is not None:
             self.show_progress_bar = show_progress_bar
 
-        self.progress_bar_refresh_rate = progress_bar_refresh_rate
-        self.progress_bar_callback = progress_bar_callback
-        self.configure_progress_bar()
+        self._progress_bar_callback = self.configure_progress_bar(progress_bar_refresh_rate, process_position)
 
         # logging
         self.log_save_interval = log_save_interval
@@ -648,7 +662,6 @@ class Trainer(
              'min_steps': None,
              ...
              'profiler': None,
-             'progress_bar_callback': True,
              'progress_bar_refresh_rate': 1,
              ...}
 
@@ -717,20 +730,32 @@ class Trainer(
 
     @classmethod
     def from_argparse_args(cls, args: Union[Namespace, ArgumentParser], **kwargs) -> 'Trainer':
-        """create an instance from CLI arguments
+        """
+        Create an instance from CLI arguments.
+
+        Args:
+            args: The parser or namespace to take arguments from. Only known arguments will be
+                parsed and passed to the :class:`Trainer`.
+            **kwargs: Additional keyword arguments that may override ones in the parser or namespace.
+                These must be valid Trainer arguments.
 
         Example:
             >>> parser = ArgumentParser(add_help=False)
             >>> parser = Trainer.add_argparse_args(parser)
+            >>> parser.add_argument('--my_custom_arg', default='something')  # doctest: +SKIP
             >>> args = Trainer.parse_argparser(parser.parse_args(""))
-            >>> trainer = Trainer.from_argparse_args(args)
+            >>> trainer = Trainer.from_argparse_args(args, logger=False)
         """
         if isinstance(args, ArgumentParser):
-            args = Trainer.parse_argparser(args)
+            args = cls.parse_argparser(args)
         params = vars(args)
-        params.update(**kwargs)
 
-        return cls(**params)
+        # we only want to pass in valid Trainer args, the rest may be user specific
+        valid_kwargs = inspect.signature(cls.__init__).parameters
+        trainer_kwargs = dict((name, params[name]) for name in valid_kwargs if name in params)
+        trainer_kwargs.update(**kwargs)
+
+        return cls(**trainer_kwargs)
 
     @property
     def num_gpus(self) -> int:
@@ -743,6 +768,10 @@ class Trainer(
     def data_parallel(self) -> bool:
         return self.use_dp or self.use_ddp or self.use_ddp2
 
+    @property
+    def progress_bar_callback(self):
+        return self._progress_bar_callback
+
     @property
     def progress_bar_dict(self) -> dict:
         """ Read-only for progress bar metrics. """
@@ -811,41 +840,48 @@ class Trainer(
         # download the data and do whatever transforms we need
         # do before any spawn calls so that the model can assign properties
         # only on proc 0 because no spawn has happened yet
-        model.prepare_data()
+        if not self._is_data_prepared:
+            model.prepare_data()
+            self._is_data_prepared = True
 
         # Run auto batch size scaling
         if self.auto_scale_batch_size:
             if isinstance(self.auto_scale_batch_size, bool):
                 self.auto_scale_batch_size = 'power'
             self.scale_batch_size(model, mode=self.auto_scale_batch_size)
+            model.logger = self.logger  # reset logger binding
 
         # Run learning rate finder:
         if self.auto_lr_find:
             self._run_lr_finder_internally(model)
+            model.logger = self.logger  # reset logger binding
 
         # route to appropriate start method
         # when using multi-node or DDP within a node start each module in a separate process
         if self.use_ddp2:
-            task = int(os.environ['SLURM_LOCALID'])
+            if self.is_slurm_managing_tasks:
+                task = int(os.environ['SLURM_LOCALID'])
+
+            # torchelastic or general non_slurm ddp2
+            elif 'WORLD_SIZE' in os.environ and ('GROUP_RANK' in os.environ or 'NODE_RANK' in os.environ):
+                task = int(os.environ['LOCAL_RANK'])
             self.ddp_train(task, model)
         elif self.use_ddp:
             if self.is_slurm_managing_tasks:
                 task = int(os.environ['SLURM_LOCALID'])
                 self.ddp_train(task, model)
-            # torchelastic
-            elif 'WORLD_SIZE' in os.environ and 'GROUP_RANK' in os.environ:
+
+            # torchelastic or general non_slurm ddp
+            elif 'WORLD_SIZE' in os.environ and ('GROUP_RANK' in os.environ or 'NODE_RANK' in os.environ):
                 task = int(os.environ['LOCAL_RANK'])
                 self.ddp_train(task, model)
-            else:
-                self.__set_random_port()
-                # track for predict
+
+            elif self.distributed_backend == 'cpu_ddp':
                 self.model = model
-                # train
                 mp.spawn(self.ddp_train, nprocs=self.num_processes, args=(model,))
-                # load weights if not interrupted
-                if self.on_colab_kaggle:
-                    self.load_spawn_weights(model)
-                    self.model = model
+
+            elif self.distributed_backend == 'ddp':
+                self.spawn_ddp_children(model)
 
         # 1 gpu or dp option triggers training using DP module
         # easier to avoid NCCL issues
@@ -859,7 +895,7 @@ class Trainer(
             self.single_gpu_train(model)
 
         elif self.use_tpu:  # pragma: no-cover
-            log.info(f'training on {self.num_tpu_cores} TPU cores')
+            log.info(f'training on {self.tpu_cores} TPU cores')
 
             #  COLAB_GPU is an env var available by default in Colab environments.
             start_method = 'fork' if self.on_colab_kaggle else 'spawn'
@@ -868,7 +904,10 @@ class Trainer(
             self.model = model
 
             # train
-            xmp.spawn(self.tpu_train, args=(model,), nprocs=self.num_tpu_cores, start_method=start_method)
+            if self.tpu_id is not None:
+                self.tpu_train(self.tpu_id, model)
+            else:
+                xmp.spawn(self.tpu_train, args=(model,), nprocs=self.tpu_cores, start_method=start_method)
 
             # load weights if not interrupted
             self.load_spawn_weights(model)
@@ -890,18 +929,6 @@ class Trainer(
         # used for testing or when we need to know that training succeeded
         return 1
 
-    def __set_random_port(self):
-        """
-        When running DDP NOT managed by SLURM, the ports might collide
-        :return:
-        """
-        try:
-            default_port = os.environ['MASTER_PORT']
-        except Exception:
-            import random
-            default_port = random.randint(10000, 19000)
-            os.environ['MASTER_PORT'] = str(default_port)
-
     def __attach_dataloaders(self, model, train_dataloader=None, val_dataloaders=None, test_dataloaders=None):
         # when dataloader is passed via fit, patch the train_dataloader
         # functions to overwrite with these implementations
@@ -937,8 +964,7 @@ class Trainer(
         # log hyper-parameters
         if self.logger is not None:
             # save exp to get started
-            if hasattr(ref_model, "hparams"):
-                self.logger.log_hyperparams(ref_model.hparams)
+            self.logger.log_hyperparams(ref_model.module_arguments)
 
             self.logger.save()
 
@@ -1009,7 +1035,10 @@ class Trainer(
 
         # clear cache before training
         if self.on_gpu:
-            torch.cuda.empty_cache()
+            # use context because of:
+            # https://discuss.pytorch.org/t/out-of-memory-when-i-use-torch-cuda-empty-cache/57898
+            with torch.cuda.device(f'cuda:{self.root_gpu}'):
+                torch.cuda.empty_cache()
 
         # CORE TRAINING LOOP
         self.train()
@@ -1056,13 +1085,13 @@ class Trainer(
             else:
                 self.__attach_dataloaders(self.model, test_dataloaders=test_dataloaders)
 
-        # give proper warnings if user only passed in loader without hooks
-        self.check_testing_model_configuration(model if model else self.model)
-
         if model is not None:
             self.model = model
             self.fit(model)
-        elif self.use_ddp or self.use_tpu:  # pragma: no-cover
+
+        # on tpu, .spawn means we don't have a trained model
+        # TODO: remove TPU spawn
+        elif self.use_tpu:  # pragma: no-cover
             # attempt to load weights from a spawn
             path = os.path.join(self.default_root_dir, '__temp_weight_ddp_end.ckpt')
             test_model = self.model
@@ -1077,44 +1106,45 @@ class Trainer(
 
     def check_model_configuration(self, model: LightningModule):
         r"""
-        Checks that the model is configured correctly before training is started.
+        Checks that the model is configured correctly before training or testing is started.
 
         Args:
-            model: The model to test.
+            model: The model to check the configuration.
 
         """
         # Check training_step, train_dataloader, configure_optimizer methods
-        if not self.is_overridden('training_step', model):
-            raise MisconfigurationException(
-                'No `training_step()` method defined. Lightning `Trainer` expects as minimum a'
-                ' `training_step()`, `train_dataloader()` and `configure_optimizers()` to be defined.')
+        if not self.testing:
+            if not self.is_overridden('training_step', model):
+                raise MisconfigurationException(
+                    'No `training_step()` method defined. Lightning `Trainer` expects as minimum a'
+                    ' `training_step()`, `training_dataloader()` and `configure_optimizers()` to be defined.')
 
-        if not self.is_overridden('train_dataloader', model):
-            raise MisconfigurationException(
-                'No `train_dataloader()` method defined. Lightning `Trainer` expects as minimum a'
-                ' `training_step()`, `train_dataloader()` and `configure_optimizers()` to be defined.')
+            if not self.is_overridden('train_dataloader', model):
+                raise MisconfigurationException(
+                    'No `train_dataloader()` method defined. Lightning `Trainer` expects as minimum a'
+                    ' `training_step()`, `training_dataloader()` and `configure_optimizers()` to be defined.')
 
-        if not self.is_overridden('configure_optimizers', model):
-            raise MisconfigurationException(
-                'No `configure_optimizers()` method defined. Lightning `Trainer` expects as minimum a'
-                ' `training_step()`, `train_dataloader()` and `configure_optimizers()` to be defined.')
+            if not self.is_overridden('configure_optimizers', model):
+                raise MisconfigurationException(
+                    'No `configure_optimizers()` method defined. Lightning `Trainer` expects as minimum a'
+                    ' `training_step()`, `training_dataloader()` and `configure_optimizers()` to be defined.')
 
-        # Check val_dataloader, validation_step and validation_epoch_end
-        if self.is_overridden('val_dataloader', model):
-            if not self.is_overridden('validation_step', model):
-                raise MisconfigurationException('You have passed in a `val_dataloader()`'
-                                                ' but have not defined `validation_step()`.')
+            # Check val_dataloader, validation_step and validation_epoch_end
+            if self.is_overridden('val_dataloader', model):
+                if not self.is_overridden('validation_step', model):
+                    raise MisconfigurationException('You have passed in a `val_dataloader()`'
+                                                    ' but have not defined `validation_step()`.')
+                else:
+                    if not self.is_overridden('validation_epoch_end', model):
+                        rank_zero_warn(
+                            'You have defined a `val_dataloader()` and have defined a `validation_step()`,'
+                            ' you may also want to define `validation_epoch_end()` for accumulating stats.',
+                            RuntimeWarning
+                        )
             else:
-                if not self.is_overridden('validation_epoch_end', model):
-                    rank_zero_warn(
-                        'You have defined a `val_dataloader()` and have defined a `validation_step()`,'
-                        ' you may also want to define `validation_epoch_end()` for accumulating stats.',
-                        RuntimeWarning
-                    )
-        else:
-            if self.is_overridden('validation_step', model):
-                raise MisconfigurationException('You have defined `validation_step()`,'
-                                                ' but have not passed in a val_dataloader().')
+                if self.is_overridden('validation_step', model):
+                    raise MisconfigurationException('You have defined `validation_step()`,'
+                                                    ' but have not passed in a `val_dataloader()`.')
 
         # Check test_dataloader, test_step and test_epoch_end
         if self.is_overridden('test_dataloader', model):
@@ -1127,25 +1157,10 @@ class Trainer(
                         'You have defined a `test_dataloader()` and have defined a `test_step()`, you may also want to'
                         ' define `test_epoch_end()` for accumulating stats.', RuntimeWarning
                     )
-
-    def check_testing_model_configuration(self, model: LightningModule):
-
-        has_test_step = self.is_overridden('test_step', model)
-        has_test_epoch_end = self.is_overridden('test_epoch_end', model)
-        gave_test_loader = self.is_overridden('test_dataloader', model)
-
-        if gave_test_loader and not has_test_step:
-            raise MisconfigurationException('You passed in a `test_dataloader` but did not implement `test_step()`')
-
-        if has_test_step and not gave_test_loader:
-            raise MisconfigurationException('You defined `test_step()` but did not implement'
-                                            ' `test_dataloader` nor passed in `.fit(test_dataloaders`.')
-
-        if has_test_step and gave_test_loader and not has_test_epoch_end:
-            rank_zero_warn(
-                'You passed  in a `test_dataloader` and have defined a `test_step()`, you may also want to'
-                ' define `test_epoch_end()` for accumulating stats.', RuntimeWarning
-            )
+        else:
+            if self.testing and self.is_overridden('test_step', model):
+                raise MisconfigurationException('You have defined `test_step()` but did not'
+                                                ' implement `test_dataloader` nor passed in `.test(test_dataloader)`.')
 
 
 class _PatchDataLoader(object):

pytorch_lightning/trainer/training_io.py

@@ -95,7 +95,7 @@ import torch
 import torch.distributed as torch_distrib
 
 from pytorch_lightning import _logger as log
-from pytorch_lightning.core.lightning import LightningModule
+from pytorch_lightning.core.lightning import LightningModule, CHECKPOINT_KEY_MODULE_ARGS
 from pytorch_lightning.loggers import LightningLoggerBase
 from pytorch_lightning.overrides.data_parallel import (
     LightningDistributedDataParallel,
@@ -119,6 +119,12 @@ except ImportError:
 else:
     HOROVOD_AVAILABLE = True
 
+PRIMITIVE_TYPES = (
+    bool, int, float, str,
+    list, tuple, set, dict,
+    Namespace,  # for back compatibility
+)
+
 
 class TrainerIOMixin(ABC):
 
@@ -141,6 +147,9 @@ class TrainerIOMixin(ABC):
     on_tpu: bool
     num_training_batches: int
     accumulate_grad_batches: int
+    use_amp: bool
+    use_native_amp: bool
+    scaler: ...
 
     def get_model(self):
         is_dp_module = isinstance(self.model, (LightningDistributedDataParallel,
@@ -201,7 +210,7 @@ class TrainerIOMixin(ABC):
             job_name = os.environ['SLURM_JOB_NAME']
             if job_name != 'bash':
                 on_slurm = True
-        except Exception as e:
+        except Exception:
             pass
 
         if on_slurm:
@@ -256,19 +265,18 @@ class TrainerIOMixin(ABC):
         torch.save(checkpoint, tmp_path)
         os.replace(tmp_path, filepath)
 
-    def save_checkpoint(self, filepath):
-        checkpoint = self.dump_checkpoint()
+    def save_checkpoint(self, filepath, weights_only: bool = False):
+        checkpoint = self.dump_checkpoint(weights_only)
 
         if self.proc_rank == 0:
             # do the actual save
             try:
                 self._atomic_save(checkpoint, filepath)
-            except AttributeError as e:
-                if 'hparams' in checkpoint:
-                    del checkpoint['hparams']
-                rank_zero_warn('warning, `hparams` dropped from checkpoint.'
-                               f' An attribute is not picklable {e}')
-
+            except AttributeError as err:
+                if CHECKPOINT_KEY_MODULE_ARGS in checkpoint:
+                    del checkpoint[CHECKPOINT_KEY_MODULE_ARGS]
+                rank_zero_warn('Warning, `module_arguments` dropped from checkpoint.'
+                               f' An attribute is not picklable {err}')
                 self._atomic_save(checkpoint, filepath)
 
     def restore(self, checkpoint_path: str, on_gpu: bool):
@@ -306,58 +314,56 @@ class TrainerIOMixin(ABC):
         # load training state (affects trainer only)
         self.restore_training_state(checkpoint)
 
-    def dump_checkpoint(self):
+    def dump_checkpoint(self, weights_only: bool = False) -> dict:
+        """Creating model checkpoint.
+
+        Args:
+            weights_only: saving model weights only
+
+        Return:
+             structured dictionary
+        """
         checkpoint = {
             'epoch': self.current_epoch + 1,
             'global_step': self.global_step + 1,
         }
 
-        if self.checkpoint_callback is not None and self.checkpoint_callback is not False:
-            checkpoint['checkpoint_callback_best'] = self.checkpoint_callback.best
+        if not weights_only:
+            if self.checkpoint_callback:
+                checkpoint['checkpoint_callback_best_model_score'] = self.checkpoint_callback.best_model_score
+                checkpoint['checkpoint_callback_best_model_path'] = self.checkpoint_callback.best_model_path
 
-        if self.early_stop_callback is not None and self.checkpoint_callback is not False:
-            checkpoint['early_stop_callback_wait'] = self.early_stop_callback.wait
-            checkpoint['early_stop_callback_patience'] = self.early_stop_callback.patience
+            if self.early_stop_callback:
+                checkpoint['early_stop_callback_wait'] = self.early_stop_callback.wait
+                checkpoint['early_stop_callback_patience'] = self.early_stop_callback.patience
 
-        # save optimizers
-        optimizer_states = []
-        for i, optimizer in enumerate(self.optimizers):
-            optimizer_states.append(optimizer.state_dict())
+            # save optimizers
+            optimizer_states = []
+            for i, optimizer in enumerate(self.optimizers):
+                optimizer_states.append(optimizer.state_dict())
 
-        checkpoint['optimizer_states'] = optimizer_states
+            checkpoint['optimizer_states'] = optimizer_states
 
-        # save lr schedulers
-        lr_schedulers = []
-        for scheduler in self.lr_schedulers:
-            lr_schedulers.append(scheduler['scheduler'].state_dict())
+            # save lr schedulers
+            lr_schedulers = []
+            for scheduler in self.lr_schedulers:
+                lr_schedulers.append(scheduler['scheduler'].state_dict())
 
-        checkpoint['lr_schedulers'] = lr_schedulers
+            checkpoint['lr_schedulers'] = lr_schedulers
 
-        # add the hparams and state_dict from the model
+            # save native amp scaling
+            if self.use_amp and self.use_native_amp:
+                checkpoint['native_amp_scaling_state'] = self.scaler.state_dict()
+
+        # add the module_arguments and state_dict from the model
         model = self.get_model()
 
         checkpoint['state_dict'] = model.state_dict()
 
-        # save native amp scaling
-        if self.use_amp and self.use_native_amp:
-            checkpoint['native_amp_scaling_state'] = self.scaler.state_dict()
-
-        if hasattr(model, "hparams") and model.hparams is not None:
-            parsing.clean_namespace(model.hparams)
-            checkpoint['hparams_type'] = model.hparams.__class__.__name__
-            if checkpoint['hparams_type'] == 'dict':
-                checkpoint['hparams'] = model.hparams
-            elif checkpoint['hparams_type'] == 'Namespace':
-                checkpoint['hparams'] = vars(model.hparams)
-            else:
-                raise ValueError(
-                    'The acceptable hparams type is dict or argparse.Namespace,',
-                    f' not {checkpoint["hparams_type"]}'
-                )
-        else:
-            rank_zero_warn(
-                "Did not find hyperparameters at model hparams. Saving checkpoint without hyperparameters."
-            )
+        if hasattr(model, CHECKPOINT_KEY_MODULE_ARGS) and model.module_arguments:
+            # add arguments to the checkpoint
+            checkpoint[CHECKPOINT_KEY_MODULE_ARGS] = {k: v for k, v in model.module_arguments.items()
+                                                      if isinstance(v, PRIMITIVE_TYPES)}
 
         # give the model a chance to add a few things
         model.on_save_checkpoint(checkpoint)
@@ -390,10 +396,25 @@ class TrainerIOMixin(ABC):
         :param checkpoint:
         :return:
         """
-        if self.checkpoint_callback is not None and self.checkpoint_callback is not False:
-            self.checkpoint_callback.best = checkpoint['checkpoint_callback_best']
+        if 'optimizer_states' not in checkpoint or 'lr_schedulers' not in checkpoint:
+            raise KeyError(
+                'Trying to restore training state but checkpoint contains only the model.'
+                ' This is probably due to `ModelCheckpoint.save_weights_only` being set to `True`.'
+            )
 
-        if self.early_stop_callback is not None and self.early_stop_callback is not False:
+        if self.checkpoint_callback:
+            if 'checkpoint_callback_best_model_score' in checkpoint:
+                self.checkpoint_callback.best_model_score = checkpoint['checkpoint_callback_best_model_score']
+            else:
+                # Old naming until version 0.7.6
+                rank_zero_warn(
+                    'Loading a checkpoint created with an old version of Lightning; '
+                    'this will not be supported in the future.'
+                )
+                self.checkpoint_callback.best_model_score = checkpoint['checkpoint_callback_best']
+            self.checkpoint_callback.best_model_path = checkpoint['checkpoint_callback_best_model_path']
+
+        if self.early_stop_callback:
             self.early_stop_callback.wait = checkpoint['early_stop_callback_wait']
             self.early_stop_callback.patience = checkpoint['early_stop_callback_patience']
 
@@ -455,12 +476,11 @@ class TrainerIOMixin(ABC):
         # TODO: fix for anything with multiprocess DP, DDP, DDP2
         try:
             self._atomic_save(checkpoint, filepath)
-        except AttributeError as e:
-            if 'hparams' in checkpoint:
-                del checkpoint['hparams']
-            rank_zero_warn('warning, `hparams` dropped from checkpoint.'
-                           f' An attribute is not picklable {e}')
-
+        except AttributeError as err:
+            if CHECKPOINT_KEY_MODULE_ARGS in checkpoint:
+                del checkpoint[CHECKPOINT_KEY_MODULE_ARGS]
+            rank_zero_warn('warning, `module_arguments` dropped from checkpoint.'
+                           f' An attribute is not picklable {err}')
             self._atomic_save(checkpoint, filepath)
 
         return filepath

pytorch_lightning/trainer/training_loop.py

@@ -141,21 +141,24 @@ in your model.
 
 """
 
+import atexit
+import signal
 from abc import ABC, abstractmethod
 from typing import Callable
 from typing import Union, List
 
 import numpy as np
-from torch.utils.data import DataLoader
 import torch
+from torch.utils.data import DataLoader
 
 from pytorch_lightning import _logger as log
 from pytorch_lightning.callbacks.base import Callback
 from pytorch_lightning.core.lightning import LightningModule
 from pytorch_lightning.loggers import LightningLoggerBase
-from pytorch_lightning.utilities.exceptions import MisconfigurationException
 from pytorch_lightning.trainer.supporters import TensorRunningAccum
 from pytorch_lightning.utilities import rank_zero_warn
+from pytorch_lightning.utilities.exceptions import MisconfigurationException
+import subprocess
 
 try:
     from apex import amp
@@ -179,9 +182,11 @@ except ImportError:
 else:
     HOROVOD_AVAILABLE = True
 
+# constant which signals should be catched for graceful trainer shutdown
+SIGNAL_TERMINATE = ('SIGTERM', 'SIGSEGV', 'SIGINT')
+
 
 class TrainerTrainLoopMixin(ABC):
-
     # this is just a summary on variables used in this abstract class,
     #  the proper values/initialisation should be done in child class
     max_epochs: int
@@ -231,6 +236,7 @@ class TrainerTrainLoopMixin(ABC):
     total_batch_idx: int
     checkpoint_callback: ...
     terminate_on_nan: bool
+    tpu_id: int
 
     # Callback system
     callbacks: List[Callback]
@@ -299,6 +305,15 @@ class TrainerTrainLoopMixin(ABC):
         """Warning: this is just empty shell for code implemented in other class."""
 
     def train(self):
+        # add signal handlers for process kills
+        # def _signal_kill_handler(*args):
+        #     return TrainerTrainLoopMixin.run_training_teardown(self)
+        #
+        # orig_signal_handlers = {}
+        # for sig_name in SIGNAL_TERMINATE:
+        #     orig_signal_handlers[sig_name] = signal.signal(getattr(signal, sig_name),
+        #                                                    _signal_kill_handler)
+
         # get model
         model = self.get_model()
 
@@ -326,6 +341,7 @@ class TrainerTrainLoopMixin(ABC):
                     self.reset_train_dataloader(model)
                 # set seed for distributed sampler (enables shuffling for each epoch)
                 if (self.use_ddp or self.use_horovod) \
+                        and hasattr(self.train_dataloader, 'sampler') \
                         and hasattr(self.train_dataloader.sampler, 'set_epoch'):
                     self.train_dataloader.sampler.set_epoch(epoch)
 
@@ -346,13 +362,13 @@ class TrainerTrainLoopMixin(ABC):
                 # -----------------
                 self.run_training_epoch()
 
-                # update LR schedulers
-                self.update_learning_rates(interval='epoch')
-
                 if self.max_steps and self.max_steps == self.global_step:
                     self.run_training_teardown()
                     return
 
+                # update LR schedulers
+                self.update_learning_rates(interval='epoch')
+
                 # early stopping
                 met_min_epochs = epoch >= self.min_epochs - 1
                 met_min_steps = self.global_step >= self.min_steps if self.min_steps else True
@@ -360,7 +376,7 @@ class TrainerTrainLoopMixin(ABC):
                 # TODO wrap this logic into the callback
                 if self.enable_early_stop:
                     if (met_min_epochs and met_min_steps) or self.fast_dev_run:
-                        should_stop = self.early_stop_callback.on_epoch_end(self, self.get_model())
+                        should_stop = self.early_stop_callback.on_validation_end(self, self.get_model())
                         # stop training
                         stop = should_stop and met_min_epochs
                         if stop:
@@ -370,10 +386,16 @@ class TrainerTrainLoopMixin(ABC):
             self.run_training_teardown()
 
         except KeyboardInterrupt:
-            if self.proc_rank == 0:
-                log.info('Detected KeyboardInterrupt, attempting graceful shutdown...')
-            self.interrupted = True
-            self.run_training_teardown()
+            rank_zero_warn('Detected KeyboardInterrupt, attempting graceful shutdown...')
+
+            # user could press ctrl+c many times... only shutdown once
+            if not self.interrupted:
+                self.interrupted = True
+
+                for proc in self.interactive_ddp_procs:
+                    subprocess.Popen.kill(proc)
+
+                self.run_training_teardown()
 
     def run_training_epoch(self):
 
@@ -394,7 +416,7 @@ class TrainerTrainLoopMixin(ABC):
 
         # on TPU we have to wrap it under the ParallelLoader
         if self.use_tpu:
-            device = xm.xla_device()
+            device = xm.xla_device(self.tpu_id)
             train_dataloader = xla_pl.ParallelLoader(train_dataloader, [device])
             train_dataloader = train_dataloader.per_device_loader(device)
 
@@ -403,7 +425,7 @@ class TrainerTrainLoopMixin(ABC):
 
         # run epoch
         for batch_idx, (batch, is_last_batch) in self.profiler.profile_iterable(
-            enumerate(_with_is_last(train_dataloader)), "get_train_batch"
+                enumerate(_with_is_last(train_dataloader)), "get_train_batch"
         ):
             # stop epoch if we limited the number of training batches
             if batch_idx >= self.num_training_batches:
@@ -450,7 +472,6 @@ class TrainerTrainLoopMixin(ABC):
             if self.fast_dev_run or should_check_val:
                 self.run_evaluation(test_mode=self.testing)
                 self.call_checkpoint_callback()
-                self.call_early_stop_callback()
 
             # when logs should be saved
             should_save_log = (batch_idx + 1) % self.log_save_interval == 0 or early_stop_epoch
@@ -496,7 +517,6 @@ class TrainerTrainLoopMixin(ABC):
         # when no val loop is present or fast-dev-run still need to call checkpoints
         if not self.is_overridden('validation_step') and not (self.fast_dev_run or should_check_val):
             self.call_checkpoint_callback()
-            self.call_early_stop_callback()
 
         # Epoch end events
         with self.profiler.profile('on_epoch_end'):
@@ -608,7 +628,7 @@ class TrainerTrainLoopMixin(ABC):
 
                     # track gradient norms when requested
                     if batch_idx % self.row_log_interval == 0:
-                        if self.track_grad_norm > 0:
+                        if float(self.track_grad_norm) > 0:
                             model = self.get_model()
                             grad_norm_dic = model.grad_norm(
                                 self.track_grad_norm)
@@ -661,7 +681,10 @@ class TrainerTrainLoopMixin(ABC):
         opt_idx = np.argmax(optimizer_freq_cumsum > current_place_in_loop)
         return [(opt_idx, self.optimizers[opt_idx])]
 
+    # @atexit.register
     def run_training_teardown(self):
+        if hasattr(self, '_teardown_already_run') and self._teardown_already_run:
+            return
         # Train end events
         with self.profiler.profile('on_train_end'):
             # callbacks
@@ -676,6 +699,8 @@ class TrainerTrainLoopMixin(ABC):
         # summarize profile results
         self.profiler.describe()
 
+        self._teardown_already_run = True
+
     def training_forward(self, batch, batch_idx, opt_idx, hiddens):
         """
         Handle forward for each training case (distributed, single gpu, etc...)
@@ -728,7 +753,7 @@ class TrainerTrainLoopMixin(ABC):
 
         # TPU support
         elif self.use_tpu:
-            batch = self.transfer_batch_to_tpu(batch)
+            batch = self.transfer_batch_to_tpu(batch, self.tpu_id)
             args[0] = batch
             output = self.model.training_step(*args)
 
@@ -789,10 +814,6 @@ class TrainerTrainLoopMixin(ABC):
         if self.checkpoint_callback is not None:
             self.checkpoint_callback.on_validation_end(self, self.get_model())
 
-    def call_early_stop_callback(self):
-        if self.early_stop_callback:
-            self.early_stop_callback.on_epoch_end(self, self.get_model())
-
 
 def _with_is_last(iterable):
     """Pass through values from the given iterable with an added boolean indicating if this is the last item.

pytorch_lightning/trainer/training_tricks.py

@@ -25,7 +25,9 @@ class TrainerTrainingTricksMixin(ABC):
     # this is just a summary on variables used in this abstract class,
     #  the proper values/initialisation should be done in child class
     gradient_clip_val: ...
-    precision: ...
+    precision: int
+    default_root_dir: str
+    progress_bar_callback: ...
     on_gpu: bool
 
     @abstractmethod
@@ -133,7 +135,7 @@ class TrainerTrainingTricksMixin(ABC):
                algorithm is terminated
 
         """
-        if not hasattr(model.hparams, batch_arg_name):
+        if not hasattr(model, batch_arg_name):
             raise MisconfigurationException(f'Field {batch_arg_name} not found in `model.hparams`')
 
         if hasattr(model.train_dataloader, 'patch_loader_code'):
@@ -243,9 +245,9 @@ def _adjust_batch_size(trainer,
 
     """
     model = trainer.get_model()
-    batch_size = getattr(model.hparams, batch_arg_name)
+    batch_size = getattr(model, batch_arg_name)
     if value:
-        setattr(model.hparams, batch_arg_name, value)
+        setattr(model, batch_arg_name, value)
         new_size = value
         if desc:
             log.info(f'Batch size {batch_size} {desc}, trying batch size {new_size}')
@@ -253,7 +255,7 @@ def _adjust_batch_size(trainer,
         new_size = int(batch_size * factor)
         if desc:
             log.info(f'Batch size {batch_size} {desc}, trying batch size {new_size}')
-        setattr(model.hparams, batch_arg_name, new_size)
+        setattr(model, batch_arg_name, new_size)
     return new_size
 
 

pytorch_lightning/utilities/__init__.py

@@ -1,3 +1,4 @@
 """General utilities"""
 
 from pytorch_lightning.utilities.distributed import rank_zero_only, rank_zero_warn
+from pytorch_lightning.utilities.apply_func import move_data_to_device

pytorch_lightning/utilities/apply_func.py

@@ -0,0 +1,59 @@
+from collections import Mapping, Sequence
+from typing import Any, Callable, Union
+
+import torch
+
+
+def apply_to_collection(data: Any, dtype: Union[type, tuple], function: Callable, *args, **kwargs) -> Any:
+    """
+    Recursively applies a function to all elements of a certain dtype.
+
+    Args:
+        data: the collection to apply the function to
+        dtype: the given function will be applied to all elements of this dtype
+        function: the function to apply
+        *args: positional arguments (will be forwarded to calls of ``function``)
+        **kwargs: keyword arguments (will be forwarded to calls of ``function``)
+
+    Returns:
+        the resulting collection
+
+    """
+    elem_type = type(data)
+
+    # Breaking condition
+    if isinstance(data, dtype):
+        return function(data, *args, **kwargs)
+
+    # Recursively apply to collection items
+    elif isinstance(data, Mapping):
+        return elem_type({k: apply_to_collection(v, dtype, function, *args, **kwargs)
+                          for k, v in data.items()})
+    elif isinstance(data, tuple) and hasattr(data, '_fields'):  # named tuple
+        return elem_type(*(apply_to_collection(d, dtype, function, *args, **kwargs) for d in data))
+    elif isinstance(data, Sequence) and not isinstance(data, str):
+        return elem_type([apply_to_collection(d, dtype, function, *args, **kwargs) for d in data])
+
+    # data is neither of dtype, nor a collection
+    return data
+
+
+def move_data_to_device(batch: Any, device: torch.device):
+    """
+    Transfers a collection of tensors to the given device.
+
+    Args:
+        batch: A tensor or collection of tensors. See :func:`apply_to_collection`
+            for a list of supported collection types.
+        device: The device to which tensors should be moved
+
+    Return:
+        the same collection but with all contained tensors residing on the new device.
+
+    See Also:
+        - :meth:`torch.Tensor.to`
+        - :class:`torch.device`
+    """
+    def to(tensor):
+        return tensor.to(device, non_blocking=True)
+    return apply_to_collection(batch, dtype=torch.Tensor, function=to)

pytorch_lightning/utilities/memory.py

@@ -32,24 +32,29 @@ def is_oom_error(exception):
         or is_out_of_cpu_memory(exception)
 
 
+# based on https://github.com/BlackHC/toma/blob/master/toma/torch_cuda_memory.py
 def is_cuda_out_of_memory(exception):
     return isinstance(exception, RuntimeError) \
         and len(exception.args) == 1 \
         and "CUDA out of memory." in exception.args[0]
 
 
+# based on https://github.com/BlackHC/toma/blob/master/toma/torch_cuda_memory.py
 def is_cudnn_snafu(exception):
+    # For/because of https://github.com/pytorch/pytorch/issues/4107
     return isinstance(exception, RuntimeError) \
         and len(exception.args) == 1 \
         and "cuDNN error: CUDNN_STATUS_NOT_SUPPORTED." in exception.args[0]
 
 
+# based on https://github.com/BlackHC/toma/blob/master/toma/cpu_memory.py
 def is_out_of_cpu_memory(exception):
     return isinstance(exception, RuntimeError) \
         and len(exception.args) == 1 \
         and "DefaultCPUAllocator: can't allocate memory" in exception.args[0]
 
 
+# based on https://github.com/BlackHC/toma/blob/master/toma/torch_cuda_memory.py
 def garbage_collection_cuda():
     """Garbage collection Torch (CUDA) memory."""
     gc.collect()

requirements-extra.txt

@@ -8,4 +8,5 @@ wandb>=0.8.21
 trains>=0.14.1
 matplotlib>=3.1.1
 # no need to install with [pytorch] as pytorch is already installed and torchvision is required only for Horovod examples
-horovod>=0.19.1
+horovod>=0.19.1
+omegaconf==2.0.0

requirements.txt

@@ -2,7 +2,7 @@
 
 tqdm>=4.41.0
 numpy>=1.16.4
-torch>=1.1
+torch>=1.3
 tensorboard>=1.14
 future>=0.17.1  # required for builtins in setup.py
 pyyaml>=3.13

tests/README.md

@@ -9,6 +9,9 @@ run on a 2-GPU machine to validate the full test-suite.
 
 
 To run all tests do the following:
+
+Install [Open MPI](https://www.open-mpi.org/) or another MPI implementation. Learn how to install Open MPI [on this page](https://www.open-mpi.org/faq/?category=building#easy-build>).
+
 ```bash
 git clone https://github.com/PyTorchLightning/pytorch-lightning
 cd pytorch-lightning

tests/base/model_optimizers.py

@@ -4,12 +4,13 @@ from torch import optim
 
 
 class ConfigureOptimizersPool(ABC):
+
     def configure_optimizers(self):
         """
         return whatever optimizers we want here.
         :return: list of optimizers
         """
-        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer = optim.Adam(self.parameters(), lr=self.learning_rate)
         return optimizer
 
     def configure_optimizers__empty(self):
@@ -20,7 +21,7 @@ class ConfigureOptimizersPool(ABC):
         return whatever optimizers we want here.
         :return: list of optimizers
         """
-        optimizer = optim.LBFGS(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer = optim.LBFGS(self.parameters(), lr=self.learning_rate)
         return optimizer
 
     def configure_optimizers__multiple_optimizers(self):
@@ -29,26 +30,26 @@ class ConfigureOptimizersPool(ABC):
         :return: list of optimizers
         """
         # try no scheduler for this model (testing purposes)
-        optimizer1 = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
-        optimizer2 = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer1 = optim.Adam(self.parameters(), lr=self.learning_rate)
+        optimizer2 = optim.Adam(self.parameters(), lr=self.learning_rate)
         return optimizer1, optimizer2
 
     def configure_optimizers__single_scheduler(self):
-        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer = optim.Adam(self.parameters(), lr=self.learning_rate)
         lr_scheduler = optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.1)
         return [optimizer], [lr_scheduler]
 
     def configure_optimizers__multiple_schedulers(self):
-        optimizer1 = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
-        optimizer2 = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer1 = optim.Adam(self.parameters(), lr=self.learning_rate)
+        optimizer2 = optim.Adam(self.parameters(), lr=self.learning_rate)
         lr_scheduler1 = optim.lr_scheduler.StepLR(optimizer1, 1, gamma=0.1)
         lr_scheduler2 = optim.lr_scheduler.StepLR(optimizer2, 1, gamma=0.1)
 
         return [optimizer1, optimizer2], [lr_scheduler1, lr_scheduler2]
 
     def configure_optimizers__mixed_scheduling(self):
-        optimizer1 = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
-        optimizer2 = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer1 = optim.Adam(self.parameters(), lr=self.learning_rate)
+        optimizer2 = optim.Adam(self.parameters(), lr=self.learning_rate)
         lr_scheduler1 = optim.lr_scheduler.StepLR(optimizer1, 4, gamma=0.1)
         lr_scheduler2 = optim.lr_scheduler.StepLR(optimizer2, 1, gamma=0.1)
 
@@ -56,14 +57,14 @@ class ConfigureOptimizersPool(ABC):
             [{'scheduler': lr_scheduler1, 'interval': 'step'}, lr_scheduler2]
 
     def configure_optimizers__reduce_lr_on_plateau(self):
-        optimizer = optim.Adam(self.parameters(), lr=self.hparams.learning_rate)
+        optimizer = optim.Adam(self.parameters(), lr=self.learning_rate)
         lr_scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer)
         return [optimizer], [lr_scheduler]
 
     def configure_optimizers__param_groups(self):
         param_groups = [
-            {'params': list(self.parameters())[:2], 'lr': self.hparams.learning_rate * 0.1},
-            {'params': list(self.parameters())[2:], 'lr': self.hparams.learning_rate}
+            {'params': list(self.parameters())[:2], 'lr': self.learning_rate * 0.1},
+            {'params': list(self.parameters())[2:], 'lr': self.learning_rate}
         ]
 
         optimizer = optim.Adam(param_groups)

tests/base/model_template.py

@@ -37,16 +37,38 @@ class EvalModelTemplate(
 
     >>> model = EvalModelTemplate()
     """
-    def __init__(self, hparams: object = None) -> object:
-        """Pass in parsed HyperOptArgumentParser to the model."""
-        if hparams is None:
-            hparams = EvalModelTemplate.get_default_hparams()
+
+    def __init__(self,
+                 *args,
+                 drop_prob: float = 0.2,
+                 batch_size: int = 32,
+                 in_features: int = 28 * 28,
+                 learning_rate: float = 0.001 * 8,
+                 optimizer_name: str = 'adam',
+                 data_root: str = PATH_DATASETS,
+                 out_features: int = 10,
+                 hidden_dim: int = 1000,
+                 b1: float = 0.5,
+                 b2: float = 0.999,
+                 **kwargs) -> object:
         # init superclass
         super().__init__()
-        self.hparams = Namespace(**hparams) if isinstance(hparams, dict) else hparams
+        self.auto_collect_arguments()
+
+        self.drop_prob = drop_prob
+        self.batch_size = batch_size
+        self.in_features = in_features
+        self.learning_rate = learning_rate
+        self.optimizer_name = optimizer_name
+        self.data_root = data_root
+        self.out_features = out_features
+        self.hidden_dim = hidden_dim
+        self.b1 = b1
+        self.b2 = b2
 
         # if you specify an example input, the summary will show input/output for each layer
-        self.example_input_array = torch.rand(5, 28 * 28)
+        # TODO: to be fixed in #1773
+        # self.example_input_array = torch.rand(5, 28 * 28)
 
         # build model
         self.__build_model()
@@ -57,15 +79,15 @@ class EvalModelTemplate(
         :return:
         """
         self.c_d1 = nn.Linear(
-            in_features=self.hparams.in_features,
-            out_features=self.hparams.hidden_dim
+            in_features=self.in_features,
+            out_features=self.hidden_dim
         )
-        self.c_d1_bn = nn.BatchNorm1d(self.hparams.hidden_dim)
-        self.c_d1_drop = nn.Dropout(self.hparams.drop_prob)
+        self.c_d1_bn = nn.BatchNorm1d(self.hidden_dim)
+        self.c_d1_drop = nn.Dropout(self.drop_prob)
 
         self.c_d2 = nn.Linear(
-            in_features=self.hparams.hidden_dim,
-            out_features=self.hparams.out_features
+            in_features=self.hidden_dim,
+            out_features=self.out_features
         )
 
     def forward(self, x):
@@ -84,10 +106,10 @@ class EvalModelTemplate(
         return nll
 
     def prepare_data(self):
-        _ = TrialMNIST(root=self.hparams.data_root, train=True, download=True)
+        _ = TrialMNIST(root=self.data_root, train=True, download=True)
 
     @staticmethod
-    def get_default_hparams(continue_training: bool = False, hpc_exp_number: int = 0) -> Namespace:
+    def get_default_hparams(continue_training: bool = False, hpc_exp_number: int = 0) -> dict:
         args = dict(
             drop_prob=0.2,
             batch_size=32,
@@ -107,5 +129,4 @@ class EvalModelTemplate(
                 hpc_exp_number=hpc_exp_number,
             )
 
-        hparams = Namespace(**args)
-        return hparams
+        return args

tests/base/model_train_steps.py

@@ -23,16 +23,12 @@ class TrainingStepVariations(ABC):
         loss_val = self.loss(y, y_hat)
 
         # alternate possible outputs to test
-        if self.trainer.batch_idx % 1 == 0:
-            output = OrderedDict({
-                'loss': loss_val,
-                'progress_bar': {'some_val': loss_val * loss_val},
-                'log': {'train_some_val': loss_val * loss_val},
-            })
-            return output
-
-        if self.trainer.batch_idx % 2 == 0:
-            return loss_val
+        output = OrderedDict({
+            'loss': loss_val,
+            'progress_bar': {'some_val': loss_val * loss_val},
+            'log': {'train_some_val': loss_val * loss_val},
+        })
+        return output
 
     def training_step__inf_loss(self, batch, batch_idx, optimizer_idx=None):
         output = self.training_step(batch, batch_idx, optimizer_idx)

tests/base/model_utilities.py

@@ -7,12 +7,12 @@ class ModelTemplateData:
     hparams: ...
 
     def dataloader(self, train):
-        dataset = TrialMNIST(root=self.hparams.data_root, train=train, download=True)
+        dataset = TrialMNIST(root=self.data_root, train=train, download=True)
 
         loader = DataLoader(
             dataset=dataset,
-            batch_size=self.hparams.batch_size,
-            # test and valid shall not be shuffled
+            batch_size=self.batch_size,
+            num_workers=3,
             shuffle=train,
         )
         return loader

tests/base/model_valid_epoch_ends.py

@@ -20,28 +20,35 @@ class ValidationEpochEndVariations(ABC):
             # recursive mean for multilevel dicts
             return torch.stack([x[key] if isinstance(x, dict) else _mean(x, key) for x in res]).mean()
 
-        # return torch.stack(outputs).mean()
         val_loss_mean = _mean(outputs, 'val_loss')
         val_acc_mean = _mean(outputs, 'val_acc')
-        for output in outputs:
-            val_loss = self.get_output_metric(output, 'val_loss')
-
-            # reduce manually when using dp
-            if self.trainer.use_dp or self.trainer.use_ddp2:
-                val_loss = torch.mean(val_loss)
-            val_loss_mean += val_loss
-
-            # reduce manually when using dp
-            val_acc = self.get_output_metric(output, 'val_acc')
-            if self.trainer.use_dp or self.trainer.use_ddp2:
-                val_acc = torch.mean(val_acc)
-
-            val_acc_mean += val_acc
-
-        if outputs:  # skip zero divisions
-            val_loss_mean /= len(outputs)
-            val_acc_mean /= len(outputs)
 
         metrics_dict = {'val_loss': val_loss_mean.item(), 'val_acc': val_acc_mean.item()}
         results = {'progress_bar': metrics_dict, 'log': metrics_dict}
         return results
+
+    def validation_epoch_end_multiple_dataloaders(self, outputs):
+        """
+        Called at the end of validation to aggregate outputs
+
+        Args:
+            outputs: list of individual outputs of each validation step
+        """
+
+        # if returned a scalar from validation_step, outputs is a list of tensor scalars
+        # we return just the average in this case (if we want)
+        def _mean(res, key):
+            return torch.stack([x[key] for x in res]).mean()
+
+        pbar = {}
+        logs = {}
+        for dl_output_list in outputs:
+            output_keys = dl_output_list[0].keys()
+            output_keys = [x for x in output_keys if 'val_' in x]
+            for key in output_keys:
+                metric_out = _mean(dl_output_list, key)
+                pbar[key] = metric_out
+                logs[key] = metric_out
+
+        results = {'progress_bar': pbar, 'log': logs}
+        return results

tests/base/model_valid_steps.py

@@ -23,33 +23,14 @@ class ValidationStepVariations(ABC):
         # acc
         labels_hat = torch.argmax(y_hat, dim=1)
         val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-        val_acc = torch.tensor(val_acc)
+        val_acc = torch.tensor(val_acc).type_as(x)
 
-        if self.on_gpu:
-            val_acc = val_acc.cuda(loss_val.device.index)
-
-        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp:
-            loss_val = loss_val.unsqueeze(0)
-            val_acc = val_acc.unsqueeze(0)
-
-        # alternate possible outputs to test
-        if batch_idx % 1 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-            })
-            return output
-        if batch_idx % 2 == 0:
-            return val_acc
-
-        if batch_idx % 3 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-                'test_dic': {'val_loss_a': loss_val}
-            })
-            return output
+        output = OrderedDict({
+            'val_loss': loss_val,
+            'val_acc': val_acc,
+            'test_dic': {'val_loss_a': loss_val}
+        })
+        return output
 
     def validation_step__multiple_dataloaders(self, batch, batch_idx, dataloader_idx, **kwargs):
         """
@@ -66,36 +47,10 @@ class ValidationStepVariations(ABC):
         # acc
         labels_hat = torch.argmax(y_hat, dim=1)
         val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)
-        val_acc = torch.tensor(val_acc)
+        val_acc = torch.tensor(val_acc).type_as(x)
 
-        if self.on_gpu:
-            val_acc = val_acc.cuda(loss_val.device.index)
-
-        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
-        if self.trainer.use_dp:
-            loss_val = loss_val.unsqueeze(0)
-            val_acc = val_acc.unsqueeze(0)
-
-        # alternate possible outputs to test
-        if batch_idx % 1 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-            })
-            return output
-        if batch_idx % 2 == 0:
-            return val_acc
-
-        if batch_idx % 3 == 0:
-            output = OrderedDict({
-                'val_loss': loss_val,
-                'val_acc': val_acc,
-                'test_dic': {'val_loss_a': loss_val}
-            })
-            return output
-        if batch_idx % 5 == 0:
-            output = OrderedDict({
-                f'val_loss_{dataloader_idx}': loss_val,
-                f'val_acc_{dataloader_idx}': val_acc,
-            })
-            return output
+        output = OrderedDict({
+            f'val_loss_{dataloader_idx}': loss_val,
+            f'val_acc_{dataloader_idx}': val_acc,
+        })
+        return output

tests/base/models.py

@@ -18,7 +18,7 @@ from pytorch_lightning.core.lightning import LightningModule
 
 
 class Generator(nn.Module):
-    def __init__(self, latent_dim, img_shape):
+    def __init__(self, latent_dim: tuple, img_shape: tuple):
         super().__init__()
         self.img_shape = img_shape
 
@@ -45,7 +45,7 @@ class Generator(nn.Module):
 
 
 class Discriminator(nn.Module):
-    def __init__(self, img_shape):
+    def __init__(self, img_shape: tuple):
         super().__init__()
 
         self.model = nn.Sequential(
@@ -67,13 +67,16 @@ class Discriminator(nn.Module):
 class TestGAN(LightningModule):
     """Implements a basic GAN for the purpose of illustrating multiple optimizers."""
 
-    def __init__(self, hparams):
+    def __init__(self, hidden_dim, learning_rate, b1, b2, **kwargs):
         super().__init__()
-        self.hparams = hparams
+        self.hidden_dim = hidden_dim
+        self.learning_rate = learning_rate
+        self.b1 = b1
+        self.b2 = b2
 
         # networks
         mnist_shape = (1, 28, 28)
-        self.generator = Generator(latent_dim=hparams.hidden_dim, img_shape=mnist_shape)
+        self.generator = Generator(latent_dim=self.hidden_dim, img_shape=mnist_shape)
         self.discriminator = Discriminator(img_shape=mnist_shape)
 
         # cache for generated images
@@ -93,7 +96,7 @@ class TestGAN(LightningModule):
         # train generator
         if optimizer_idx == 0:
             # sample noise
-            z = torch.randn(imgs.shape[0], self.hparams.hidden_dim)
+            z = torch.randn(imgs.shape[0], self.hidden_dim)
             z = z.type_as(imgs)
 
             # generate images
@@ -128,8 +131,7 @@ class TestGAN(LightningModule):
             fake = torch.zeros(imgs.size(0), 1)
             fake = fake.type_as(fake)
 
-            fake_loss = self.adversarial_loss(
-                self.discriminator(self.generated_imgs.detach()), fake)
+            fake_loss = self.adversarial_loss(self.discriminator(self.generated_imgs.detach()), fake)
 
             # discriminator loss is the average of these
             d_loss = (real_loss + fake_loss) / 2
@@ -142,9 +144,9 @@ class TestGAN(LightningModule):
             return output
 
     def configure_optimizers(self):
-        lr = self.hparams.learning_rate
-        b1 = self.hparams.b1
-        b2 = self.hparams.b2
+        lr = self.learning_rate
+        b1 = self.b1
+        b2 = self.b2
 
         opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
         opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))

tests/base/utils.py

@@ -25,7 +25,7 @@ def assert_speed_parity(pl_times, pt_times, num_epochs):
         f"lightning was slower than PT (threshold {max_diff_per_epoch})"
 
 
-def run_model_test_without_loggers(trainer_options, model, min_acc=0.50):
+def run_model_test_without_loggers(trainer_options, model, min_acc=0.30):
     reset_seed()
 
     # fit model
@@ -155,7 +155,7 @@ def load_model_from_checkpoint(root_weights_dir, module_class=EvalModelTemplate)
     return trained_model
 
 
-def run_prediction(dataloader, trained_model, dp=False, min_acc=0.5):
+def run_prediction(dataloader, trained_model, dp=False, min_acc=0.3):
     # run prediction on 1 batch
     for batch in dataloader:
         break

tests/callbacks/test_callbacks.py

@@ -1,19 +1,20 @@
+from pathlib import Path
+
 import pytest
 
 import tests.base.utils as tutils
 from pytorch_lightning import Callback
 from pytorch_lightning import Trainer, LightningModule
-from pytorch_lightning.callbacks import EarlyStopping, LearningRateLogger, ModelCheckpoint
+from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 from pytorch_lightning.loggers import TensorBoardLogger
 from tests.base import EvalModelTemplate
-from pathlib import Path
 
 
 def test_trainer_callback_system(tmpdir):
     """Test the callback system."""
 
     hparams = EvalModelTemplate.get_default_hparams()
-    model = EvalModelTemplate(hparams)
+    model = EvalModelTemplate(**hparams)
 
     def _check_args(trainer, pl_module):
         assert isinstance(trainer, Trainer)
@@ -219,7 +220,7 @@ def test_early_stopping_no_val_step(tmpdir):
         default_root_dir=tmpdir,
         early_stop_callback=stopping,
         overfit_pct=0.20,
-        max_epochs=5,
+        max_epochs=2,
     )
     result = trainer.fit(model)
 
@@ -253,7 +254,7 @@ def test_model_checkpoint_with_non_string_input(tmpdir, save_top_k):
     trainer = Trainer(default_root_dir=tmpdir,
                       checkpoint_callback=checkpoint,
                       overfit_pct=0.20,
-                      max_epochs=5
+                      max_epochs=2
                       )
     trainer.fit(model)
 
@@ -274,84 +275,10 @@ def test_model_checkpoint_path(tmpdir, logger_version, expected):
     trainer = Trainer(
         default_root_dir=tmpdir,
         overfit_pct=0.2,
-        max_epochs=5,
+        max_epochs=2,
         logger=logger
     )
     trainer.fit(model)
 
     ckpt_version = Path(trainer.ckpt_path).parent.name
     assert ckpt_version == expected
-
-
-def test_lr_logger_single_lr(tmpdir):
-    """ Test that learning rates are extracted and logged for single lr scheduler"""
-    tutils.reset_seed()
-
-    model = EvalModelTemplate()
-    model.configure_optimizers = model.configure_optimizers__single_scheduler
-
-    lr_logger = LearningRateLogger()
-    trainer = Trainer(
-        default_root_dir=tmpdir,
-        max_epochs=5,
-        val_percent_check=0.1,
-        train_percent_check=0.5,
-        callbacks=[lr_logger]
-    )
-    results = trainer.fit(model)
-
-    assert results == 1
-    assert lr_logger.lrs, 'No learning rates logged'
-    assert len(lr_logger.lrs) == len(trainer.lr_schedulers), \
-        'Number of learning rates logged does not match number of lr schedulers'
-    assert all([k in ['lr-Adam'] for k in lr_logger.lrs.keys()]), \
-        'Names of learning rates not set correctly'
-
-
-def test_lr_logger_multi_lrs(tmpdir):
-    """ Test that learning rates are extracted and logged for multi lr schedulers """
-    tutils.reset_seed()
-
-    model = EvalModelTemplate()
-    model.configure_optimizers = model.configure_optimizers__multiple_schedulers
-
-    lr_logger = LearningRateLogger()
-    trainer = Trainer(
-        default_root_dir=tmpdir,
-        max_epochs=1,
-        val_percent_check=0.1,
-        train_percent_check=0.5,
-        callbacks=[lr_logger]
-    )
-    results = trainer.fit(model)
-
-    assert results == 1
-    assert lr_logger.lrs, 'No learning rates logged'
-    assert len(lr_logger.lrs) == len(trainer.lr_schedulers), \
-        'Number of learning rates logged does not match number of lr schedulers'
-    assert all([k in ['lr-Adam', 'lr-Adam-1'] for k in lr_logger.lrs.keys()]), \
-        'Names of learning rates not set correctly'
-
-
-def test_lr_logger_param_groups(tmpdir):
-    """ Test that learning rates are extracted and logged for single lr scheduler"""
-    tutils.reset_seed()
-
-    model = EvalModelTemplate()
-    model.configure_optimizers = model.configure_optimizers__param_groups
-
-    lr_logger = LearningRateLogger()
-    trainer = Trainer(
-        default_root_dir=tmpdir,
-        max_epochs=5,
-        val_percent_check=0.1,
-        train_percent_check=0.5,
-        callbacks=[lr_logger]
-    )
-    results = trainer.fit(model)
-
-    assert lr_logger.lrs, 'No learning rates logged'
-    assert len(lr_logger.lrs) == 2 * len(trainer.lr_schedulers), \
-        'Number of learning rates logged does not match number of param groups'
-    assert all([k in ['lr-Adam/pg1', 'lr-Adam/pg2'] for k in lr_logger.lrs.keys()]), \
-        'Names of learning rates not set correctly'

tests/callbacks/test_lr.py

@@ -0,0 +1,102 @@
+import pytest
+
+import tests.base.utils as tutils
+from pytorch_lightning import Trainer
+from pytorch_lightning.callbacks import LearningRateLogger
+from tests.base import EvalModelTemplate
+
+
+def test_lr_logger_single_lr(tmpdir):
+    """ Test that learning rates are extracted and logged for single lr scheduler. """
+    tutils.reset_seed()
+
+    model = EvalModelTemplate()
+    model.configure_optimizers = model.configure_optimizers__single_scheduler
+
+    lr_logger = LearningRateLogger()
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        max_epochs=2,
+        val_percent_check=0.1,
+        train_percent_check=0.5,
+        callbacks=[lr_logger]
+    )
+    result = trainer.fit(model)
+    assert result
+
+    assert lr_logger.lrs, 'No learning rates logged'
+    assert len(lr_logger.lrs) == len(trainer.lr_schedulers), \
+        'Number of learning rates logged does not match number of lr schedulers'
+    assert all([k in ['lr-Adam'] for k in lr_logger.lrs.keys()]), \
+        'Names of learning rates not set correctly'
+
+
+def test_lr_logger_no_lr(tmpdir):
+    tutils.reset_seed()
+
+    model = EvalModelTemplate()
+
+    lr_logger = LearningRateLogger()
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        max_epochs=2,
+        val_percent_check=0.1,
+        train_percent_check=0.5,
+        callbacks=[lr_logger]
+    )
+
+    with pytest.warns(RuntimeWarning):
+        result = trainer.fit(model)
+        assert result
+
+
+def test_lr_logger_multi_lrs(tmpdir):
+    """ Test that learning rates are extracted and logged for multi lr schedulers. """
+    tutils.reset_seed()
+
+    model = EvalModelTemplate()
+    model.configure_optimizers = model.configure_optimizers__multiple_schedulers
+
+    lr_logger = LearningRateLogger()
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        max_epochs=2,
+        val_percent_check=0.1,
+        train_percent_check=0.5,
+        callbacks=[lr_logger]
+    )
+    result = trainer.fit(model)
+    assert result
+
+    assert lr_logger.lrs, 'No learning rates logged'
+    assert len(lr_logger.lrs) == len(trainer.lr_schedulers), \
+        'Number of learning rates logged does not match number of lr schedulers'
+    assert all([k in ['lr-Adam', 'lr-Adam-1'] for k in lr_logger.lrs.keys()]), \
+        'Names of learning rates not set correctly'
+    assert all(len(lr) == trainer.max_epochs for k, lr in lr_logger.lrs.items()), \
+        'Length of logged learning rates exceeds the number of epochs'
+
+
+def test_lr_logger_param_groups(tmpdir):
+    """ Test that learning rates are extracted and logged for single lr scheduler. """
+    tutils.reset_seed()
+
+    model = EvalModelTemplate()
+    model.configure_optimizers = model.configure_optimizers__param_groups
+
+    lr_logger = LearningRateLogger()
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        max_epochs=2,
+        val_percent_check=0.1,
+        train_percent_check=0.5,
+        callbacks=[lr_logger]
+    )
+    result = trainer.fit(model)
+    assert result
+
+    assert lr_logger.lrs, 'No learning rates logged'
+    assert len(lr_logger.lrs) == 2 * len(trainer.lr_schedulers), \
+        'Number of learning rates logged does not match number of param groups'
+    assert all([k in ['lr-Adam/pg1', 'lr-Adam/pg2'] for k in lr_logger.lrs.keys()]), \
+        'Names of learning rates not set correctly'

tests/callbacks/test_progress_bar.py

@@ -179,7 +179,7 @@ def test_progress_bar_progress_refresh(refresh_rate):
         num_sanity_val_steps=2,
         max_epochs=3,
     )
-    assert trainer.progress_bar_callback.refresh_rate == refresh_rate != trainer.progress_bar_refresh_rate
+    assert trainer.progress_bar_callback.refresh_rate == refresh_rate
 
     trainer.fit(model)
     assert progress_bar.train_batches_seen == 3 * progress_bar.total_train_batches

tests/loggers/test_all.py

@@ -96,3 +96,28 @@ def test_loggers_pickle(tmpdir, monkeypatch, logger_class):
 
     trainer2 = pickle.loads(pkl_bytes)
     trainer2.logger.log_metrics({'acc': 1.0})
+
+
+@pytest.mark.parametrize("extra_params", [
+    pytest.param(dict(max_epochs=1, auto_scale_batch_size=True), id='Batch-size-Finder'),
+    pytest.param(dict(max_epochs=3, auto_lr_find=True), id='LR-Finder'),
+])
+def test_logger_reset_correctly(tmpdir, extra_params):
+    """ Test that the tuners do not alter the logger reference """
+    tutils.reset_seed()
+
+    model = EvalModelTemplate()
+
+    trainer = Trainer(
+        default_save_path=tmpdir,
+        **extra_params
+    )
+    logger1 = trainer.logger
+    trainer.fit(model)
+    logger2 = trainer.logger
+    logger3 = model.logger
+
+    assert logger1 == logger2, \
+        'Finder altered the logger of trainer'
+    assert logger2 == logger3, \
+        'Finder altered the logger of model'

tests/loggers/test_base.py

@@ -61,7 +61,7 @@ class CustomLogger(LightningLoggerBase):
 
 def test_custom_logger(tmpdir):
     hparams = EvalModelTemplate.get_default_hparams()
-    model = EvalModelTemplate(hparams)
+    model = EvalModelTemplate(**hparams)
 
     logger = CustomLogger()
 
@@ -80,7 +80,7 @@ def test_custom_logger(tmpdir):
 
 def test_multiple_loggers(tmpdir):
     hparams = EvalModelTemplate.get_default_hparams()
-    model = EvalModelTemplate(hparams)
+    model = EvalModelTemplate(**hparams)
 
     logger1 = CustomLogger()
     logger2 = CustomLogger()
@@ -143,7 +143,7 @@ def test_adding_step_key(tmpdir):
     model.validation_epoch_end = _validation_epoch_end
     model.training_epoch_end = _training_epoch_end
     trainer = Trainer(
-        max_epochs=4,
+        max_epochs=3,
         default_root_dir=tmpdir,
         train_percent_check=0.001,
         val_percent_check=0.01,

tests/loggers/test_wandb.py

@@ -13,11 +13,11 @@ def test_wandb_logger(wandb):
     logger = WandbLogger(anonymous=True, offline=True)
 
     logger.log_metrics({'acc': 1.0})
-    wandb.init().log.assert_called_once_with({'acc': 1.0}, step=None)
+    wandb.init().log.assert_called_once_with({'acc': 1.0})
 
     wandb.init().log.reset_mock()
     logger.log_metrics({'acc': 1.0}, step=3)
-    wandb.init().log.assert_called_once_with({'acc': 1.0}, step=3)
+    wandb.init().log.assert_called_once_with({'global_step': 3, 'acc': 1.0})
 
     logger.log_hyperparams({'test': None})
     wandb.init().config.update.assert_called_once_with({'test': None}, allow_val_change=True)

tests/metrics/test_converters.py

@@ -0,0 +1,214 @@
+import numpy as np
+import pytest
+import torch
+import torch.distributed as dist
+import torch.multiprocessing as mp
+
+import tests.base.utils as tutils
+from pytorch_lightning.metrics.converters import (
+    _apply_to_inputs, _apply_to_outputs, _convert_to_tensor, _convert_to_numpy,
+    _numpy_metric_conversion, _tensor_metric_conversion, _sync_ddp_if_available, tensor_metric, numpy_metric)
+
+
+@pytest.mark.parametrize(['args', 'kwargs'],
+                         [pytest.param([], {}),
+                          pytest.param([1., 2.], {}),
+                          pytest.param([], {'a': 1., 'b': 2.}),
+                          pytest.param([1., 2.], {'a': 1., 'b': 2.})])
+def test_apply_to_inputs(args, kwargs):
+    def apply_fn(inputs, factor):
+        if isinstance(inputs, (float, int)):
+            return inputs * factor
+        elif isinstance(inputs, dict):
+            return {k: apply_fn(v, factor) for k, v in inputs.items()}
+        elif isinstance(inputs, (tuple, list)):
+            return [apply_fn(x, factor) for x in inputs]
+
+    @_apply_to_inputs(apply_fn, factor=2.)
+    def test_fn(*func_args, **func_kwargs):
+        return func_args, func_kwargs
+
+    result_args, result_kwargs = test_fn(*args, **kwargs)
+    assert isinstance(result_args, (list, tuple))
+    assert isinstance(result_kwargs, dict)
+    assert len(result_args) == len(args)
+    assert len(result_kwargs) == len(kwargs)
+    assert all([k in result_kwargs for k in kwargs.keys()])
+    for arg, result_arg in zip(args, result_args):
+        assert arg * 2. == result_arg
+
+    for key in kwargs.keys():
+        arg = kwargs[key]
+        result_arg = result_kwargs[key]
+        assert arg * 2. == result_arg
+
+
+def test_apply_to_outputs():
+    def apply_fn(inputs, additional_str):
+        return str(inputs) + additional_str
+
+    @_apply_to_outputs(apply_fn, additional_str='_str')
+    def test_fn(*args, **kwargs):
+        return 'dummy'
+
+    assert test_fn() == 'dummy_str'
+
+
+def test_convert_to_tensor():
+    for test_item in [1., np.array([1.])]:
+        result_tensor = _convert_to_tensor(test_item)
+        assert isinstance(result_tensor, torch.Tensor)
+        assert result_tensor.item() == 1.
+
+
+def test_convert_to_numpy():
+    for test_item in [1., torch.tensor([1.])]:
+        result = _convert_to_numpy(test_item)
+        assert isinstance(result, np.ndarray)
+        assert result.item() == 1.
+
+
+def test_numpy_metric_conversion():
+    @_numpy_metric_conversion
+    def numpy_test_metric(*args, **kwargs):
+        for arg in args:
+            assert isinstance(arg, np.ndarray)
+
+        for v in kwargs.values():
+            assert isinstance(v, np.ndarray)
+
+        return 5.
+
+    result = numpy_test_metric(torch.tensor([1.]), dummy_kwarg=2.)
+    assert isinstance(result, torch.Tensor)
+    assert result.item() == 5.
+
+
+def test_tensor_metric_conversion():
+    @_tensor_metric_conversion
+    def tensor_test_metric(*args, **kwargs):
+        for arg in args:
+            assert isinstance(arg, torch.Tensor)
+
+        for v in kwargs.values():
+            assert isinstance(v, torch.Tensor)
+
+        return 5.
+
+    result = tensor_test_metric(np.array([1.]), dummy_kwarg=2.)
+    assert isinstance(result, torch.Tensor)
+    assert result.item() == 5.
+
+
+def setup_ddp(rank, worldsize, ):
+    import os
+
+    os.environ['MASTER_ADDR'] = 'localhost'
+
+    # initialize the process group
+    dist.init_process_group("gloo", rank=rank, world_size=worldsize)
+
+
+def ddp_test_fn(rank, worldsize):
+    setup_ddp(rank, worldsize)
+    tensor = torch.tensor([1.], device='cuda:0')
+
+    reduced_tensor = _sync_ddp_if_available(tensor)
+
+    assert reduced_tensor.item() == dist.get_world_size(), \
+        'Sync-Reduce does not work properly with DDP and Tensors'
+
+
+@pytest.mark.skipif(torch.cuda.device_count() < 2, reason="test requires multi-GPU machine")
+def test_sync_reduce_ddp():
+    """Make sure sync-reduce works with DDP"""
+    tutils.reset_seed()
+    tutils.set_random_master_port()
+
+    worldsize = 2
+    mp.spawn(ddp_test_fn, args=(worldsize,), nprocs=worldsize)
+
+
+def test_sync_reduce_simple():
+    """Make sure sync-reduce works without DDP"""
+    tensor = torch.tensor([1.], device='cpu')
+
+    reduced_tensor = _sync_ddp_if_available(tensor)
+
+    assert torch.allclose(tensor, reduced_tensor), \
+        'Sync-Reduce does not work properly without DDP and Tensors'
+
+
+def _test_tensor_metric(is_ddp: bool):
+    @tensor_metric()
+    def tensor_test_metric(*args, **kwargs):
+        for arg in args:
+            assert isinstance(arg, torch.Tensor)
+
+        for v in kwargs.values():
+            assert isinstance(v, torch.Tensor)
+
+        return 5.
+
+    if is_ddp:
+        factor = dist.get_world_size()
+    else:
+        factor = 1.
+
+    result = tensor_test_metric(np.array([1.]), dummy_kwarg=2.)
+    assert isinstance(result, torch.Tensor)
+    assert result.item() == 5. * factor
+
+
+def _ddp_test_tensor_metric(rank, worldsize):
+    setup_ddp(rank, worldsize)
+    _test_tensor_metric(True)
+
+
+def test_tensor_metric_ddp():
+    tutils.reset_seed()
+    tutils.set_random_master_port()
+
+    world_size = 2
+    mp.spawn(_ddp_test_tensor_metric, args=(world_size,), nprocs=world_size)
+
+
+def test_tensor_metric_simple():
+    _test_tensor_metric(False)
+
+
+def _test_numpy_metric(is_ddp: bool):
+    @numpy_metric()
+    def numpy_test_metric(*args, **kwargs):
+        for arg in args:
+            assert isinstance(arg, np.ndarray)
+
+        for v in kwargs.values():
+            assert isinstance(v, np.ndarray)
+
+        return 5.
+
+    if is_ddp:
+        factor = dist.get_world_size()
+    else:
+        factor = 1.
+
+    result = numpy_test_metric(torch.tensor([1.]), dummy_kwarg=2.)
+    assert isinstance(result, torch.Tensor)
+    assert result.item() == 5. * factor
+
+
+def _ddp_test_numpy_metric(rank, worldsize):
+    setup_ddp(rank, worldsize)
+    _test_numpy_metric(True)
+
+
+def test_numpy_metric_ddp():
+    tutils.reset_seed()
+    tutils.set_random_master_port()
+    world_size = 2
+    mp.spawn(_ddp_test_numpy_metric, args=(world_size,), nprocs=world_size)
+
+
+def test_numpy_metric_simple():
+    _test_tensor_metric(False)

tests/metrics/test_metrics.py

@@ -0,0 +1,85 @@
+import numpy as np
+import torch
+
+from pytorch_lightning.metrics.metric import Metric, TensorMetric, NumpyMetric
+
+
+class DummyTensorMetric(TensorMetric):
+    def __init__(self):
+        super().__init__('dummy')
+
+    def forward(self, input1, input2):
+        assert isinstance(input1, torch.Tensor)
+        assert isinstance(input2, torch.Tensor)
+        return 1.
+
+
+class DummyNumpyMetric(NumpyMetric):
+    def __init__(self):
+        super().__init__('dummy')
+
+    def forward(self, input1, input2):
+        assert isinstance(input1, np.ndarray)
+        assert isinstance(input2, np.ndarray)
+        return 1.
+
+
+def _test_metric(metric: Metric):
+    input1, input2 = torch.tensor([1.]), torch.tensor([2.])
+
+    def change_and_check_device_dtype(device, dtype):
+        metric.to(device=device, dtype=dtype)
+
+        metric_val = metric(input1, input2)
+        assert isinstance(metric_val, torch.Tensor)
+
+        if device is not None:
+            assert metric.device in [device, torch.device(device)]
+            assert metric_val.device in [device, torch.device(device)]
+
+        if dtype is not None:
+            assert metric.dtype == dtype
+            assert metric_val.dtype == dtype
+
+    devices = [None, 'cpu']
+    if torch.cuda.is_available():
+        devices += ['cuda:0']
+
+    for device in devices:
+        for dtype in [None, torch.float32, torch.float64]:
+            change_and_check_device_dtype(device=device, dtype=dtype)
+
+    if torch.cuda.is_available():
+        metric.cuda(0)
+        assert metric.device == torch.device('cuda', index=0)
+        assert metric(input1, input2).device == torch.device('cuda', index=0)
+
+    metric.cpu()
+    assert metric.device == torch.device('cpu')
+    assert metric(input1, input2).device == torch.device('cpu')
+
+    metric.type(torch.int8)
+    assert metric.dtype == torch.int8
+    assert metric(input1, input2).dtype == torch.int8
+
+    metric.float()
+    assert metric.dtype == torch.float32
+    assert metric(input1, input2).dtype == torch.float32
+
+    metric.double()
+    assert metric.dtype == torch.float64
+    assert metric(input1, input2).dtype == torch.float64
+
+    if torch.cuda.is_available():
+        metric.cuda()
+        metric.half()
+        assert metric.dtype == torch.float16
+        assert metric(input1, input2).dtype == torch.float16
+
+
+def test_tensor_metric():
+    _test_metric(DummyTensorMetric())
+
+
+def test_numpy_metric():
+    _test_metric(DummyNumpyMetric())

tests/models/test_cpu.py

@@ -1,3 +1,4 @@
+import os
 import platform
 from collections import namedtuple
 
@@ -9,6 +10,77 @@ import tests.base.utils as tutils
 from pytorch_lightning import Trainer
 from pytorch_lightning.callbacks import EarlyStopping
 from tests.base import EvalModelTemplate
+from pytorch_lightning.callbacks import ModelCheckpoint
+
+
+def test_cpu_slurm_save_load(tmpdir):
+    """Verify model save/load/checkpoint on CPU."""
+    hparams = EvalModelTemplate.get_default_hparams()
+    model = EvalModelTemplate(**hparams)
+
+    # logger file to get meta
+    logger = tutils.get_default_logger(tmpdir)
+    version = logger.version
+
+    # fit model
+    trainer = Trainer(
+        max_epochs=1,
+        logger=logger,
+        train_percent_check=0.2,
+        val_percent_check=0.2,
+        checkpoint_callback=ModelCheckpoint(tmpdir)
+    )
+    result = trainer.fit(model)
+    real_global_step = trainer.global_step
+
+    # traning complete
+    assert result == 1, 'cpu model failed to complete'
+
+    # predict with trained model before saving
+    # make a prediction
+    dataloaders = model.test_dataloader()
+    if not isinstance(dataloaders, list):
+        dataloaders = [dataloaders]
+
+    for dataloader in dataloaders:
+        for batch in dataloader:
+            break
+
+    x, y = batch
+    x = x.view(x.size(0), -1)
+
+    model.eval()
+    pred_before_saving = model(x)
+
+    # test HPC saving
+    # simulate snapshot on slurm
+    saved_filepath = trainer.hpc_save(tmpdir, logger)
+    assert os.path.exists(saved_filepath)
+
+    # new logger file to get meta
+    logger = tutils.get_default_logger(tmpdir, version=version)
+
+    trainer = Trainer(
+        max_epochs=1,
+        logger=logger,
+        checkpoint_callback=ModelCheckpoint(tmpdir),
+    )
+    model = EvalModelTemplate(**hparams)
+
+    # set the epoch start hook so we can predict before the model does the full training
+    def assert_pred_same():
+        assert trainer.global_step == real_global_step and trainer.global_step > 0
+
+        # predict with loaded model to make sure answers are the same
+        trainer.model.eval()
+        new_pred = trainer.model(x)
+        assert torch.all(torch.eq(pred_before_saving, new_pred)).item() == 1
+
+    model.on_epoch_start = assert_pred_same
+
+    # by calling fit again, we trigger training, loading weights from the cluster
+    # and our hook to predict using current model before any more weight updates
+    trainer.fit(model)
 
 
 def test_early_stopping_cpu_model(tmpdir):
@@ -17,6 +89,7 @@ def test_early_stopping_cpu_model(tmpdir):
     trainer_options = dict(
         default_root_dir=tmpdir,
         early_stop_callback=stopping,
+        max_epochs=2,
         gradient_clip_val=1.0,
         overfit_pct=0.20,
         track_grad_norm=2,
@@ -39,6 +112,7 @@ def test_early_stopping_cpu_model(tmpdir):
                      version_parse(torch.__version__) < version_parse("1.3.0")),
                     reason="Distributed training is not supported on MacOS before Torch 1.3.0")
 def test_multi_cpu_model_ddp(tmpdir):
+    print('in ddp test')
     """Make sure DDP works."""
     tutils.set_random_master_port()
 
@@ -61,19 +135,19 @@ def test_lbfgs_cpu_model(tmpdir):
     """Test each of the trainer options."""
     trainer_options = dict(
         default_root_dir=tmpdir,
-        max_epochs=2,
+        max_epochs=1,
         progress_bar_refresh_rate=0,
         weights_summary='top',
-        train_percent_check=1.0,
+        train_percent_check=0.2,
         val_percent_check=0.2,
     )
 
     hparams = EvalModelTemplate.get_default_hparams()
-    setattr(hparams, 'optimizer_name', 'lbfgs')
-    setattr(hparams, 'learning_rate', 0.002)
-    model = EvalModelTemplate(hparams)
+    hparams.update(optimizer_name='lbfgs',
+                   learning_rate=0.004)
+    model = EvalModelTemplate(**hparams)
     model.configure_optimizers = model.configure_optimizers__lbfgs
-    tutils.run_model_test_without_loggers(trainer_options, model, min_acc=0.5)
+    tutils.run_model_test_without_loggers(trainer_options, model, min_acc=0.25)
 
 
 def test_default_logger_callbacks_cpu_model(tmpdir):
@@ -110,7 +184,7 @@ def test_running_test_after_fitting(tmpdir):
     trainer = Trainer(
         default_root_dir=tmpdir,
         progress_bar_refresh_rate=0,
-        max_epochs=8,
+        max_epochs=2,
         train_percent_check=0.4,
         val_percent_check=0.2,
         test_percent_check=0.2,
@@ -272,8 +346,8 @@ def test_tbptt_cpu_model(tmpdir):
             return 1
 
     class BpttTestModel(EvalModelTemplate):
-        def __init__(self, hparams):
-            super().__init__(hparams)
+        def __init__(self, *args, **kwargs):
+            super().__init__(*args, **kwargs)
             self.test_hidden = None
 
         def training_step(self, batch, batch_idx, hiddens):
@@ -303,12 +377,14 @@ def test_tbptt_cpu_model(tmpdir):
             )
 
     hparams = EvalModelTemplate.get_default_hparams()
-    hparams.batch_size = batch_size
-    hparams.in_features = truncated_bptt_steps
-    hparams.hidden_dim = truncated_bptt_steps
-    hparams.out_features = truncated_bptt_steps
+    hparams.update(
+        batch_size=batch_size,
+        in_features=truncated_bptt_steps,
+        hidden_dim=truncated_bptt_steps,
+        out_features=truncated_bptt_steps
+    )
 
-    model = BpttTestModel(hparams)
+    model = BpttTestModel(**hparams)
 
     # fit model
     trainer = Trainer(
@@ -322,19 +398,3 @@ def test_tbptt_cpu_model(tmpdir):
     result = trainer.fit(model)
 
     assert result == 1, 'training failed to complete'
-
-
-@pytest.mark.skipif(not torch.cuda.is_available(), reason="test requires GPU machine")
-def test_single_gpu_model(tmpdir):
-    """Make sure single GPU works (DP mode)."""
-    trainer_options = dict(
-        default_root_dir=tmpdir,
-        progress_bar_refresh_rate=0,
-        max_epochs=1,
-        train_percent_check=0.1,
-        val_percent_check=0.1,
-        gpus=1
-    )
-
-    model = EvalModelTemplate()
-    tutils.run_model_test(trainer_options, model)

tests/models/test_gpu.py

@@ -5,7 +5,6 @@ import torch
 
 import tests.base.utils as tutils
 from pytorch_lightning import Trainer
-from pytorch_lightning.callbacks import ModelCheckpoint
 from pytorch_lightning.core import memory
 from pytorch_lightning.trainer.distrib_parts import parse_gpu_ids, determine_root_gpu_device
 from pytorch_lightning.utilities.exceptions import MisconfigurationException
@@ -14,6 +13,23 @@ from tests.base import EvalModelTemplate
 PRETEND_N_OF_GPUS = 16
 
 
+@pytest.mark.skipif(not torch.cuda.is_available(), reason="test requires GPU machine")
+@pytest.mark.parametrize('gpus', [1, [0], [1]])
+def test_single_gpu_model(tmpdir, gpus):
+    """Make sure single GPU works (DP mode)."""
+    trainer_options = dict(
+        default_root_dir=tmpdir,
+        progress_bar_refresh_rate=0,
+        max_epochs=1,
+        train_percent_check=0.1,
+        val_percent_check=0.1,
+        gpus=gpus
+    )
+
+    model = EvalModelTemplate()
+    tutils.run_model_test(trainer_options, model)
+
+
 @pytest.mark.spawn
 @pytest.mark.parametrize("backend", ['dp', 'ddp', 'ddp2'])
 @pytest.mark.skipif(torch.cuda.device_count() < 2, reason="test requires multi-GPU machine")
@@ -40,6 +56,7 @@ def test_multi_gpu_model(tmpdir, backend):
     memory.get_memory_profile('min_max')
 
 
+@pytest.mark.spawn
 @pytest.mark.skipif(torch.cuda.device_count() < 2, reason="test requires multi-GPU machine")
 def test_ddp_all_dataloaders_passed_to_fit(tmpdir):
     """Make sure DDP works with dataloaders passed to fit()"""
@@ -48,8 +65,8 @@ def test_ddp_all_dataloaders_passed_to_fit(tmpdir):
     trainer_options = dict(default_root_dir=tmpdir,
                            progress_bar_refresh_rate=0,
                            max_epochs=1,
-                           train_percent_check=0.4,
-                           val_percent_check=0.2,
+                           train_percent_check=0.1,
+                           val_percent_check=0.1,
                            gpus=[0, 1],
                            distributed_backend='ddp')
 
@@ -62,74 +79,6 @@ def test_ddp_all_dataloaders_passed_to_fit(tmpdir):
     assert result == 1, "DDP doesn't work with dataloaders passed to fit()."
 
 
-def test_cpu_slurm_save_load(tmpdir):
-    """Verify model save/load/checkpoint on CPU."""
-    hparams = EvalModelTemplate.get_default_hparams()
-    model = EvalModelTemplate(hparams)
-
-    # logger file to get meta
-    logger = tutils.get_default_logger(tmpdir)
-    version = logger.version
-
-    # fit model
-    trainer = Trainer(
-        max_epochs=1,
-        logger=logger,
-        checkpoint_callback=ModelCheckpoint(tmpdir)
-    )
-    result = trainer.fit(model)
-    real_global_step = trainer.global_step
-
-    # traning complete
-    assert result == 1, 'cpu model failed to complete'
-
-    # predict with trained model before saving
-    # make a prediction
-    dataloaders = model.test_dataloader()
-    if not isinstance(dataloaders, list):
-        dataloaders = [dataloaders]
-
-    for dataloader in dataloaders:
-        for batch in dataloader:
-            break
-
-    x, y = batch
-    x = x.view(x.size(0), -1)
-
-    model.eval()
-    pred_before_saving = model(x)
-
-    # test HPC saving
-    # simulate snapshot on slurm
-    saved_filepath = trainer.hpc_save(tmpdir, logger)
-    assert os.path.exists(saved_filepath)
-
-    # new logger file to get meta
-    logger = tutils.get_default_logger(tmpdir, version=version)
-
-    trainer = Trainer(
-        max_epochs=1,
-        logger=logger,
-        checkpoint_callback=ModelCheckpoint(tmpdir),
-    )
-    model = EvalModelTemplate(hparams)
-
-    # set the epoch start hook so we can predict before the model does the full training
-    def assert_pred_same():
-        assert trainer.global_step == real_global_step and trainer.global_step > 0
-
-        # predict with loaded model to make sure answers are the same
-        trainer.model.eval()
-        new_pred = trainer.model(x)
-        assert torch.all(torch.eq(pred_before_saving, new_pred)).item() == 1
-
-    model.on_epoch_start = assert_pred_same
-
-    # by calling fit again, we trigger training, loading weights from the cluster
-    # and our hook to predict using current model before any more weight updates
-    trainer.fit(model)
-
-
 @pytest.mark.spawn
 @pytest.mark.skipif(torch.cuda.device_count() < 2, reason="test requires multi-GPU machine")
 def test_multi_gpu_none_backend(tmpdir):

tests/models/test_grad_norm.py

@@ -0,0 +1,106 @@
+import torch
+import pytest
+import numpy as np
+
+from pytorch_lightning import Trainer, seed_everything
+
+from pytorch_lightning.loggers import LightningLoggerBase
+from pytorch_lightning.utilities import rank_zero_only
+
+from tests.base import EvalModelTemplate
+from tests.base.utils import reset_seed
+
+
+class OnlyMetricsListLogger(LightningLoggerBase):
+    def __init__(self):
+        super().__init__()
+        self.metrics = []
+
+    @rank_zero_only
+    def log_metrics(self, metrics, step):
+        self.metrics.append(metrics)
+
+    @property
+    def experiment(self):
+        return 'test'
+
+    @rank_zero_only
+    def log_hyperparams(self, params):
+        pass
+
+    @rank_zero_only
+    def finalize(self, status):
+        pass
+
+    @property
+    def name(self):
+        return 'name'
+
+    @property
+    def version(self):
+        return '1'
+
+
+class ModelWithManualGradTracker(EvalModelTemplate):
+    def __init__(self, norm_type, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.stored_grad_norms, self.norm_type = [], float(norm_type)
+
+    # validation spoils logger's metrics with `val_loss` records
+    validation_step = None
+    val_dataloader = None
+
+    def training_step(self, batch, batch_idx, optimizer_idx=None):
+        # just return a loss, no log or progress bar meta
+        x, y = batch
+        loss_val = self.loss(y, self(x.flatten(1, -1)))
+        return {'loss': loss_val}
+
+    def on_after_backward(self):
+        out, norms = {}, []
+        prefix = f'grad_{self.norm_type}_norm_'
+        for name, p in self.named_parameters():
+            if p.grad is None:
+                continue
+
+            # `np.linalg.norm` implementation likely uses fp64 intermediates
+            flat = p.grad.data.cpu().numpy().ravel()
+            norm = np.linalg.norm(flat, self.norm_type)
+            norms.append(norm)
+
+            out[prefix + name] = round(norm, 3)
+
+        # handle total norm
+        norm = np.linalg.norm(norms, self.norm_type)
+        out[prefix + 'total'] = round(norm, 3)
+        self.stored_grad_norms.append(out)
+
+
+@pytest.mark.parametrize("norm_type", [1., 1.25, 1.5, 2, 3, 5, 10, 'inf'])
+def test_grad_tracking(tmpdir, norm_type, rtol=5e-3):
+    # rtol=5e-3 respects the 3 decmials rounding in `.grad_norms` and above
+
+    reset_seed()
+
+    # use a custom grad tracking module and a list logger
+    model = ModelWithManualGradTracker(norm_type)
+    logger = OnlyMetricsListLogger()
+
+    trainer = Trainer(
+        max_epochs=3,
+        logger=logger,
+        track_grad_norm=norm_type,
+        row_log_interval=1,  # request grad_norms every batch
+    )
+    result = trainer.fit(model)
+
+    assert result == 1, "Training failed"
+    assert len(logger.metrics) == len(model.stored_grad_norms)
+
+    # compare the logged metrics against tracked norms on `.backward`
+    for mod, log in zip(model.stored_grad_norms, logger.metrics):
+        common = mod.keys() & log.keys()
+
+        log, mod = [log[k] for k in common], [mod[k] for k in common]
+
+        assert np.allclose(log, mod, rtol=rtol)