indents

This will import the ipywidgets version of tqdm if available. This works nicely in notebooks by not filling up the log.

In the terminal it will use the same old tqdm.

We might also want to consider passing in the tqdm we want as an argument since there may be some edge cases where ipywidgets is available but the interface doesn't support it (e.g. vscode?) or isn't working. In which case people will get a warning message, but may want to configure it themselves.

.gitignore

@@ -6,6 +6,7 @@ app/models/
 pip-wheel-metadata/
 lightning_logs/
 
+
 # Test-tube
 test_tube_logs/
 test_tube_data/
@@ -14,7 +15,7 @@ test_tube_exp/
 # Documentations
 docs/source/pl_examples*.rst
 docs/source/pytorch_lightning*.rst
-/docs/source/*.md
+tests/tests/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -22,6 +23,7 @@ __pycache__/
 *$py.class
 timit_data/
 
+
 # C extensions
 *.so
 

README.md

@@ -1,6 +1,6 @@
 <div align="center">
 
-![Logo](docs/source/_static/images/lightning_logo_small.png)
+<img src="docs/source/_static/images/lightning_logo.png" width="50" height="50">
 
 # PyTorch Lightning
 
@@ -14,10 +14,10 @@
 [![Coverage](docs/source/_static/images/coverage.svg)](https://github.com/PytorchLightning/pytorch-lightning/tree/master/tests#running-coverage)
 [![CodeFactor](https://www.codefactor.io/repository/github/borda/pytorch-lightning/badge)](https://www.codefactor.io/repository/github/borda/pytorch-lightning)    
 
-[![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=latest)](https://pytorch-lightning.readthedocs.io/en/latest)
+[![ReadTheDocs](https://readthedocs.org/projects/pytorch-lightning/badge/?version=0.6.0)](https://pytorch-lightning.readthedocs.io/en/0.6.0/)
 [![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/pytorch-lightning/shared_invite/enQtODU5ODIyNTUzODQwLTFkMDg5Mzc1MDBmNjEzMDgxOTVmYTdhYjA1MDdmODUyOTg2OGQ1ZWZkYTQzODhhNzdhZDA3YmNhMDhlMDY4YzQ)
 [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/PytorchLightning/pytorch-lightning/blob/master/LICENSE)
-[![Next Release](https://img.shields.io/badge/Next%20Release-Feb%206-<COLOR>.svg)](https://shields.io/)
+[![Next Release](https://img.shields.io/badge/Next%20Release-Mar%2021-<COLOR>.svg)](https://shields.io/)
 
 <!-- 
 removed until codecov badge isn't empy. likely a config error showing nothing on master.
@@ -32,12 +32,9 @@ pip install pytorch-lightning
 ```
 
 ## Docs   
-[jan 20, 2020]   
-
-**[Old docs (some links might be broken)](https://pytorch-lightning.readthedocs.io/en/stable)
-###### As a temporary hack, when you get the 404, replace williamfalcon.github.io with pytorchlightning.github.io.   
-
-**[New docs, CURRENTLY DEBUGING](https://pytorch-lightning.rtfd.io/en/latest)**   
+- [master](https://pytorch-lightning.readthedocs.io/en/latest)   
+- [0.6.0](https://pytorch-lightning.readthedocs.io/en/0.6.0/)
+- [0.5.3.2](https://pytorch-lightning.readthedocs.io/en/0.5.3.2/)
 
 
 ## Demo  

docs/requirements.txt

@@ -6,4 +6,5 @@ pandoc
 docutils
 git+https://github.com/PytorchLightning/lightning_sphinx_theme.git
 sphinxcontrib-fulltoc
-sphinxcontrib-mockautodoc
+sphinxcontrib-mockautodoc
+pip_shims

docs/source/BECOMING_A_CORE_CONTRIBUTOR.md

@@ -0,0 +1,59 @@
+# How to become a core contributor
+
+Thanks for your interest in joining the Lightning team! We’re a rapidly growing project which is poised to become the go-to framework for DL researchers! 
+We're currently recruiting for a team of 5 core maintainers. 
+
+As a core maintainer you will have a strong say in the direction of the project. Big changes will require a majority of maintainers to agree.
+
+### Code of conduct  
+First and foremost, you'll be evaluated against [these core values](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md). Any code we commit or feature we add needs to align with those core values.
+
+### The bar for joining the team   
+Lightning is being used to solve really hard problems at the top AI labs in the world. As such, the bar for adding team members is extremely high. Candidates must have solid engineering skills, have a good eye for user experience, and must be a power user of Lightning and PyTorch.
+
+With that said, the Lightning team will be diverse and a reflection of an inclusive AI community. You don't have to be an engineer to conntribute! Scientists with great usability intuition and PyTorch ninja skills are welcomed!
+
+### Responsibilities:
+The responsibilities mainly revolve around 3 things.
+
+#### Github issues
+- Here we want to help users have an amazing experience. These range from questions from new people getting into DL to questions from researchers about doing something esoteric with Lightning
+Often, these issues require some sort of bug fix, document clarification or new functionality to be scoped out.
+
+- To become a core member you must resolve at least 10 Github issues which align with the API design goals for Lightning. By the end of these 10 issues I should feel comfortable in the way you answer user questions
+Pleasant/helpful tone.
+
+- Can abstract from that issue or bug into functionality that might solve other related issues or makes the platform more flexible.
+
+- Don’t make users feel like they don’t know what they’re doing. We’re here to help and to make everyone’s experience delightful.
+
+#### Pull requests
+
+- Here we need to ensure the code that enters Lightning is high quality. For each PR we need to:
+- Make sure code coverage does not decrease
+- Documents are updated
+- Code is elegant and simple
+- Code is NOT overly engineered or hard to read
+- Ask yourself, could a non-engineer understand what’s happening here?
+- Make sure new tests are written
+- Is this NECESSARY for Lightning? There are some PRs which are just purely about adding engineering complexity which have no place in Lightning. 
+Guidance
+- Some other PRs are for people who are wanting to get involved and add something unnecessary. We do want their help though! So don’t approve the PR, but direct them to a Github issue that they might be interested in helping with instead!
+- To be considered for core contributor, please review 10 PRs and help the authors land it on master. Once you've finished the review, ping me 
+for a sanity check. At the end of 10 PRs if your PR reviews are inline with expectations described above, then you can merge PRs on your own going forward,
+otherwise we'll do a few more until we're both comfortable :)
+
+#### Project directions
+There are some big decisions which the project must make. For these I expect core contributors to have something meaningful to add if it’s their area of expertise.
+
+#### Diversity
+Lightning should reflect the broader community it serves. As such we should have scientists/researchers from 
+different fields contributing!   
+
+The first 5 core contributors will fit this profile. Thus if you overlap strongly with experiences and expertise as someone else on the team, you might have to wait until the next set of contributors are added.
+
+#### Summary: Requirements to apply
+- Solve 10 Github issues. The goal is to be inline with expectations for solving issues by the last one so you can do them on your own. If not, I might ask you to solve a few more specific ones.
+- Do 10 PR reviews. The goal is to be inline with expectations for solving issues by the last one so you can do them on your own. If not, I might ask you to solve a few more specific ones.
+
+If you want to be considered, ping me on gitter and start [tracking your progress here](https://docs.google.com/spreadsheets/d/15D58gp8DvI0Z6qbbYVRuaWioiwzafcP58-UlbuO_CMU/edit?usp=sharing).

docs/source/CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at waf2107@columbia.edu. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

docs/source/CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing    
+Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!   
+
+## Main Core Value: One less thing to remember    
+Simplify the API as much as possible from the user perspective. Any additions or improvements should minimize things the user needs to remember.   
+
+For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval(). This avoids all sorts of subtle errors the user could make.  
+
+## Lightning Design Principles   
+We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.    
+#### No PyTorch Interference   
+We don't want to add any abstractions on top of pure PyTorch. This gives researchers all the control they need without having to learn yet another framework.    
+
+#### Simple Internal Code    
+It's useful for users to look at the code and understand very quickly what's happening. Many users won't be engineers. Thus we need to value clear, simple code over condensed ninja moves. While that's super cool, this isn't the project for that :)      
+
+#### Force User Decisions To Best Practices    
+There are 1,000 ways to do something. However, something eventually becomes standard practice that everyone does. Thus we pick one way of doing it and force everyone to do it this way. A good example is accumulated gradients. There are many ways to implement, we just pick one and force users to use that one. A bad forced decision would be to make users use a specific library to do something.    
+
+When something becomes a best practice, we add it to the framework. This likely looks like code in utils or in the model file that everyone keeps adding over and over again across projects. When this happens, bring that code inside the trainer and add a flag for it.
+
+#### Simple External API    
+What makes sense to you may not make sense to others. Create an issue with an API change suggestion and validate that it makes sense for others. Treat code changes how you treat a startup: validate that it's a needed feature, then add if it makes sense for many people. 
+
+#### Backward-compatible API   
+We all hate updating our deep learning packages because we don't want to refactor a bunch of stuff. In Lightning, we make sure every change we make which could break an API is backwards compatible with good deprecation warnings.
+
+You shouldn't be afraid to upgrade Lightning :)
+
+#### Gain User Trust    
+As a researcher you can't have any part of your code going wrong. So, make thorough tests that ensure an implementation of a new trick or subbtle change is correct.    
+
+#### Interoperability  
+Have a favorite feature from other libraries like fast.ai or transformers? Those should just work with lightning as well. Grab your favorite model or learning rate scheduler from your favorite library and run it in Lightning.
+
+## Contribution Types    
+Currently looking for help implementing new features or adding bug fixes. 
+
+A lot of good work has already been done in project mechanics (requirements.txt, setup.py, pep8, badges, ci, etc...) we're in a good state there thanks to all the early contributors (even pre-beta release)!   
+
+## Bug Fixes:  
+1. Submit a github issue.   
+2. Fix it.  
+3. Submit a PR! 
+
+## New Features:  
+1. Submit a github issue.   
+2. We'll agree on the feature scope.     
+3. Submit a PR! (with updated docs and tests 🙃).   
+
+## Coding Styleguide
+1. Test the code with flake8.
+2. Use f-strings.

docs/source/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,16 @@
+# Before submitting
+
+- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
+- [ ] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md)?
+- [ ] Did you make sure to update the docs?   
+- [ ] Did you write any new necessary tests?  
+
+## What does this PR do?
+Fixes # (issue).
+
+## PR review    
+Anyone in the community is free to review the PR once the tests have passed.     
+If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
+
+## Did you have fun?
+Make sure you had fun coding 🙃

docs/source/_templates/theme_variables.jinja

@@ -2,6 +2,7 @@
   'github': 'https://github.com/PytorchLightning/pytorch-lightning',
   'github_issues': 'https://github.com/PytorchLightning/pytorch-lightning/issues',
   'contributing': 'https://github.com/PytorchLightning/pytorch-lightning/blob/master/CONTRIBUTING.md',
+  'governance': 'https://github.com/PytorchLightning/pytorch-lightning/blob/master/governance.md',
   'docs': 'https://pytorch-lightning.rtfd.io/en/latest',
   'twitter': 'https://twitter.com/PyTorchLightnin',
   'discuss': 'https://discuss.pytorch.org',

docs/source/callbacks.rst

@@ -0,0 +1,14 @@
+.. role:: hidden
+    :class: hidden-section
+
+Callbacks
+===========
+.. automodule:: pytorch_lightning.callbacks
+   :exclude-members:
+        _del_model,
+        _save_model,
+        on_epoch_end,
+        on_train_end,
+        on_epoch_begin,
+        check_monitor_top_k,
+        on_train_begin,

docs/source/common-cases.rst

@@ -0,0 +1,21 @@
+Multi-gpu (same node) training
+==============================
+
+Multi-node training
+====================
+
+16-bit precision
+=================
+
+gradient clipping
+=================
+
+modifying training via hooks
+=============================
+
+
+
+.. toctree::
+   :maxdepth: 3
+
+   pl_examples

docs/source/governance.md

@@ -0,0 +1,8 @@
+# Pytorch Lightning Governance | Persons of interest
+
+### Maintainers  
+- William Falcon ([williamFalcon](https://github.com/williamFalcon))   
+- Jirka Borovek ([Borda](https://github.com/Borda))   
+- Nick Eggert ([neggert](https://github.com/neggert))   
+- Jeff Ling ([jeffling](https://github.com/jeffling))   
+- Tullie Murrell ([tullie](https://github.com/tullie))

docs/source/index.rst

@@ -44,22 +44,20 @@ PyTorch-Lightning Documentation
 
    common-cases
 
-
 .. toctree::
    :maxdepth: 1
    :name: community
    :caption: Community
 
+
    CODE_OF_CONDUCT.md
    CONTRIBUTING.md
    BECOMING_A_CORE_CONTRIBUTOR.md
    governance.md
 
-
 Indices and tables
 ------------------
 
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
-

docs/source/lightning-module.rst

@@ -0,0 +1,10 @@
+.. role:: hidden
+    :class: hidden-section
+
+LightningModule
+===========
+.. automodule:: pytorch_lightning.core
+   :exclude-members:
+        _abc_impl,
+        summarize,
+

docs/source/logging.rst

@@ -0,0 +1,12 @@
+.. role:: hidden
+    :class: hidden-section
+
+Logging
+===========
+.. automodule:: pytorch_lightning.logging
+   :exclude-members:
+        _abc_impl,
+        _save_model,
+        on_epoch_end,
+        on_train_end,
+        on_epoch_begin,

docs/source/modules.rst

@@ -0,0 +1,7 @@
+pl_examples
+===========
+
+.. toctree::
+   :maxdepth: 4
+
+   pl_examples

docs/source/trainer.rst

@@ -0,0 +1,21 @@
+.. role:: hidden
+    :class: hidden-section
+
+Trainer
+===========
+.. automodule:: pytorch_lightning.trainer
+   :members: fit, test
+   :exclude-members: 
+        run_pretrain_routine, 
+        _abc_impl, 
+        _Trainer__set_root_gpu, 
+        _Trainer__init_optimizers,
+        _Trainer__parse_gpu_ids,
+        _Trainer__configure_schedulers,
+        data_parallel,
+        num_gpus,
+        slurm_job_id,
+        tng_tqdm_dic,
+        training_tqdm_dict,
+        init_optimizers,
+        configure_schedulers

docs/source/tutorials.rst

@@ -0,0 +1,20 @@
+Refactoring PyTorch into Lightning
+==================================
+`Tutorial <https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538>`_
+
+Start a research project
+=========================
+`Research seed <https://github.com/PytorchLightning/pytorch-lightning-conference-seed>`_
+
+Basic Lightning use
+====================
+`Tutorial <https://towardsdatascience.com/supercharge-your-ai-research-with-pytorch-lightning-337948a99eec>`_
+
+9 key Lightning  tricks
+========================
+`Tutorial <9 key speed features in Pytorch-Lightning>`_
+
+Multi-node training on SLURM
+=============================
+`Tutorial <https://towardsdatascience.com/trivial-multi-node-training-with-pytorch-lightning-ff75dfb809bd>`_
+

pl_examples/full_examples/imagenet/imagenet_example.py

@@ -34,9 +34,12 @@ class ImageNetLightningModel(pl.LightningModule):
         self.hparams = hparams
         self.model = models.__dict__[self.hparams.arch](pretrained=self.hparams.pretrained)
 
+    def forward(self, x):
+        return self.model(x)
+
     def training_step(self, batch, batch_idx):
         images, target = batch
-        output = self.model(images)
+        output = self.forward(images)
         loss_val = F.cross_entropy(output, target)
         acc1, acc5 = self.__accuracy(output, target, topk=(1, 5))
 
@@ -59,7 +62,7 @@ class ImageNetLightningModel(pl.LightningModule):
 
     def validation_step(self, batch, batch_idx):
         images, target = batch
-        output = self.model(images)
+        output = self.forward(images)
         loss_val = F.cross_entropy(output, target)
         acc1, acc5 = self.__accuracy(output, target, topk=(1, 5))
 
@@ -132,7 +135,7 @@ class ImageNetLightningModel(pl.LightningModule):
             std=[0.229, 0.224, 0.225],
         )
 
-        train_dir = os.path.join(self.hparams.data, 'train')
+        train_dir = os.path.join(self.hparams.data_path, 'train')
         train_dataset = datasets.ImageFolder(
             train_dir,
             transforms.Compose([
@@ -162,7 +165,7 @@ class ImageNetLightningModel(pl.LightningModule):
             mean=[0.485, 0.456, 0.406],
             std=[0.229, 0.224, 0.225],
         )
-        val_dir = os.path.join(self.hparams.data, 'val')
+        val_dir = os.path.join(self.hparams.data_path, 'val')
         val_loader = torch.utils.data.DataLoader(
             datasets.ImageFolder(val_dir, transforms.Compose([
                 transforms.Resize(256),
@@ -185,7 +188,7 @@ class ImageNetLightningModel(pl.LightningModule):
                                  ' (default: resnet18)')
         parser.add_argument('--epochs', default=90, type=int, metavar='N',
                             help='number of total epochs to run')
-        parser.add_argument('--seed', type=int, default=None,
+        parser.add_argument('--seed', type=int, default=42,
                             help='seed for initializing training. ')
         parser.add_argument('-b', '--batch-size', default=256, type=int,
                             metavar='N',
@@ -214,7 +217,7 @@ def get_args():
                                help='how many gpus')
     parent_parser.add_argument('--distributed-backend', type=str, default='dp', choices=('dp', 'ddp', 'ddp2'),
                                help='supports three options dp, ddp, ddp2')
-    parent_parser.add_argument('--use-16bit', dest='use-16bit', action='store_true',
+    parent_parser.add_argument('--use-16bit', dest='use_16bit', action='store_true',
                                help='if true uses 16 bit precision')
     parent_parser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true',
                                help='evaluate model on validation set')

pytorch_lightning/callbacks/pt_callbacks.py

@@ -1,15 +1,13 @@
 """
 Callbacks
-=========
-
+====================================
 Callbacks supported by Lightning
 """
 
-import logging
 import os
 import shutil
+import logging
 import warnings
-
 import numpy as np
 
 from pytorch_lightning.overrides.data_parallel import LightningDistributedDataParallel
@@ -73,21 +71,23 @@ class EarlyStopping(Callback):
     Stop training when a monitored quantity has stopped improving.
 
     Args:
-        monitor (str): quantity to be monitored.
+        monitor (str): quantity to be monitored. Default: ``'val_loss'``.
         min_delta (float): minimum change in the monitored quantity
             to qualify as an improvement, i.e. an absolute
-            change of less than min_delta, will count as no
-            improvement.
+            change of less than `min_delta`, will count as no
+            improvement. Default: ``0``.
         patience (int): number of epochs with no improvement
-            after which training will be stopped.
-        verbose (bool): verbosity mode.
+            after which training will be stopped. Default: ``0``.
+        verbose (bool): verbosity mode. Default: ``0``.
         mode (str): one of {auto, min, max}. In `min` mode,
             training will stop when the quantity
             monitored has stopped decreasing; in `max`
             mode it will stop when the quantity
             monitored has stopped increasing; in `auto`
             mode, the direction is automatically inferred
-            from the name of the monitored quantity.
+            from the name of the monitored quantity. Default: ``'auto'``.
+        strict (bool): whether to crash the training if `monitor` is
+            not found in the metrics. Default: ``True``.
 
     Example::
 
@@ -99,18 +99,20 @@ class EarlyStopping(Callback):
     """
 
     def __init__(self, monitor='val_loss',
-                 min_delta=0.0, patience=0, verbose=0, mode='auto'):
+                 min_delta=0.0, patience=0, verbose=0, mode='auto', strict=True):
         super(EarlyStopping, self).__init__()
 
         self.monitor = monitor
         self.patience = patience
         self.verbose = verbose
+        self.strict = strict
         self.min_delta = min_delta
         self.wait = 0
         self.stopped_epoch = 0
 
         if mode not in ['auto', 'min', 'max']:
-            logging.info(f'EarlyStopping mode {mode} is unknown, fallback to auto mode.')
+            if self.verbose > 0:
+                logging.info(f'EarlyStopping mode {mode} is unknown, fallback to auto mode.')
             mode = 'auto'
 
         if mode == 'min':
@@ -130,6 +132,22 @@ class EarlyStopping(Callback):
 
         self.on_train_begin()
 
+    def check_metrics(self, logs):
+        monitor_val = logs.get(self.monitor)
+        error_msg = (f'Early stopping conditioned on metric `{self.monitor}`'
+                     f' which is not available. Available metrics are:'
+                     f' `{"`, `".join(list(logs.keys()))}`')
+
+        if monitor_val is None:
+            if self.strict:
+                raise RuntimeError(error_msg)
+            elif self.verbose > 0:
+                warnings.warn(error_msg, RuntimeWarning)
+
+            return False
+
+        return True
+
     def on_train_begin(self, logs=None):
         # Allow instances to be re-used
         self.wait = 0
@@ -137,16 +155,11 @@ class EarlyStopping(Callback):
         self.best = np.Inf if self.monitor_op == np.less else -np.Inf
 
     def on_epoch_end(self, epoch, logs=None):
-        current = logs.get(self.monitor)
         stop_training = False
-        if current is None:
-            warnings.warn(
-                f'Early stopping conditioned on metric `{self.monitor}`'
-                f' which is not available. Available metrics are: {",".join(list(logs.keys()))}',
-                RuntimeWarning)
-            stop_training = True
+        if not self.check_metrics(logs):
             return stop_training
 
+        current = logs.get(self.monitor)
         if self.monitor_op(current - self.min_delta, self.best):
             self.best = current
             self.wait = 0
@@ -165,7 +178,9 @@ class EarlyStopping(Callback):
 
 
 class ModelCheckpoint(Callback):
-    r"""Save the model after every epoch.
+    r"""
+
+    Save the model after every epoch.
 
     Args:
         filepath (str): path to save the model file.

pytorch_lightning/core/hooks.py

@@ -124,12 +124,13 @@ class ModelHooks(torch.nn.Module):
         """
         pass
 
-    def backward(self, use_amp, loss, optimizer):
+    def backward(self, use_amp, loss, optimizer, optimizer_idx):
         """Override backward with your own implementation if you need to
 
         :param use_amp: Whether amp was requested or not
         :param loss: Loss is already scaled by accumulated grads
         :param optimizer: Current optimizer being used
+        :param optimizer_idx: Index of the current optimizer being used
         :return:
 
         Called to perform backward step.

pytorch_lightning/core/lightning.py

@@ -5,15 +5,16 @@ import warnings
 from abc import ABC, abstractmethod
 from argparse import Namespace
 
+
 import pandas as pd
 import torch
 import torch.distributed as dist
-
+#
 from pytorch_lightning.core.decorators import data_loader
 from pytorch_lightning.core.grads import GradInformation
 from pytorch_lightning.core.hooks import ModelHooks
-from pytorch_lightning.core.memory import ModelSummary
 from pytorch_lightning.core.saving import ModelIO
+from pytorch_lightning.core.memory import ModelSummary
 from pytorch_lightning.overrides.data_parallel import LightningDistributedDataParallel
 
 
@@ -679,7 +680,6 @@ class LightningModule(ABC, GradInformation, ModelIO, ModelHooks):
     @abstractmethod
     def configure_optimizers(self):
         r"""
-
         This is where you choose what optimizers and learning-rate schedulers to use in your optimization.
         Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple.
 
@@ -1015,7 +1015,6 @@ class LightningModule(ABC, GradInformation, ModelIO, ModelHooks):
 
             map_location (dict): A dictionary mapping saved weight GPU devices to new
                 GPU devices (example: {'cuda:1':'cuda:0'})
-
         Return:
             LightningModule with loaded weights
 
@@ -1136,7 +1135,6 @@ class LightningModule(ABC, GradInformation, ModelIO, ModelHooks):
 
     def freeze(self):
         r"""
-
         Freeze all params for inference
 
         Example
@@ -1168,7 +1166,6 @@ class LightningModule(ABC, GradInformation, ModelIO, ModelHooks):
 
     def on_load_checkpoint(self, checkpoint):
         r"""
-
         Called by lightning to restore your model.
         If you saved something with **on_save_checkpoint** this is your chance to restore this.
 

pytorch_lightning/logging/__init__.py

@@ -71,47 +71,44 @@ Call the logger anywhere from your LightningModule by doing:
 Supported Loggers
 -----------------
 """
-
 from os import environ
 
 from .base import LightningLoggerBase, rank_zero_only
 from .tensorboard import TensorBoardLogger
 
-all = []
+loggers = ['TensorBoardLogger']
 
 try:
     # needed to prevent ImportError and duplicated logs.
     environ["COMET_DISABLE_AUTO_LOGGING"] = "1"
 
     from .comet import CometLogger
-    all.append('CometLogger')
+    loggers.append('CometLogger')
 except ImportError:
     del environ["COMET_DISABLE_AUTO_LOGGING"]
 
 try:
     from .mlflow import MLFlowLogger
-    all.append('MLFlowLogger')
+    loggers.append('MLFlowLogger')
 except ImportError:
     pass
 
 try:
     from .neptune import NeptuneLogger
-    all.append('NeptuneLogger')
+    loggers.append('NeptuneLogger')
 except ImportError:
     pass
 
-all.append('TensorBoardLogger')
-
 try:
     from .test_tube import TestTubeLogger
-    all.append('TestTubeLogger')
+    loggers.append('TestTubeLogger')
 except ImportError:
     pass
 
 try:
     from .wandb import WandbLogger
-    all.append('WandbLogger')
+    loggers.append('WandbLogger')
 except ImportError:
     pass
 
-__all__ = all
+__all__ = loggers

pytorch_lightning/root_module/__init__.py

@@ -8,4 +8,5 @@ warnings.warn("`root_module` package has been renamed to `core` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.core import (  # noqa: E402
-    decorators, grads, hooks, root_module, memory, model_saving)
+    decorators, grads, hooks, root_module, memory, model_saving
+)

pytorch_lightning/trainer/__init__.py

@@ -26,5 +26,4 @@ This is the basic use of the trainer:
 """
 
 from .trainer import Trainer
-
 __all__ = ['Trainer']

pytorch_lightning/trainer/callback_config.py

@@ -55,10 +55,20 @@ class TrainerCallbackConfigMixin(ABC):
             self.early_stop_callback = EarlyStopping(
                 monitor='val_loss',
                 patience=3,
+                strict=True,
                 verbose=True,
                 mode='min'
             )
             self.enable_early_stop = True
+        elif early_stop_callback is None:
+            self.early_stop_callback = EarlyStopping(
+                monitor='val_loss',
+                patience=3,
+                strict=False,
+                verbose=False,
+                mode='min'
+            )
+            self.enable_early_stop = True
         elif not early_stop_callback:
             self.early_stop_callback = None
             self.enable_early_stop = False

pytorch_lightning/trainer/evaluation_loop.py

@@ -127,7 +127,7 @@ import sys
 from abc import ABC, abstractmethod
 
 import torch
-import tqdm
+from tqdm.auto import tqdm
 
 from pytorch_lightning.utilities.debugging import MisconfigurationException
 
@@ -293,9 +293,9 @@ class TrainerEvaluationLoopMixin(ABC):
         # main progress bar will already be closed when testing so initial position is free
         position = 2 * self.process_position + (not test)
         desc = 'Testing' if test else 'Validating'
-        pbar = tqdm.tqdm(desc=desc, total=max_batches, leave=test, position=position,
-                         disable=not self.show_progress_bar, dynamic_ncols=True,
-                         unit='batch', file=sys.stdout)
+        pbar = tqdm(desc=desc, total=max_batches, leave=test, position=position,
+                    disable=not self.show_progress_bar, dynamic_ncols=True,
+                    unit='batch', file=sys.stdout)
         setattr(self, f'{"test" if test else "val"}_progress_bar', pbar)
 
         # run evaluation

pytorch_lightning/trainer/trainer.py

@@ -1,14 +1,13 @@
-
-
-import logging
 import os
 import sys
 import warnings
+import logging
+
 
 import torch
 import torch.distributed as dist
 import torch.multiprocessing as mp
-import tqdm
+from tqdm.auto import tqdm
 from torch.optim.optimizer import Optimizer
 
 from pytorch_lightning.trainer.auto_mix_precision import TrainerAMPMixin
@@ -20,6 +19,7 @@ from pytorch_lightning.trainer.distrib_parts import (
     parse_gpu_ids,
     determine_root_gpu_device
 )
+
 from pytorch_lightning.trainer.evaluation_loop import TrainerEvaluationLoopMixin
 from pytorch_lightning.trainer.logging import TrainerLoggingMixin
 from pytorch_lightning.trainer.model_hooks import TrainerModelHooksMixin
@@ -52,7 +52,7 @@ class Trainer(TrainerIOMixin,
             self,
             logger=True,
             checkpoint_callback=True,
-            early_stop_callback=True,
+            early_stop_callback=None,
             default_save_path=None,
             gradient_clip_val=0,
             gradient_clip=None,  # backward compatible, todo: remove in v0.8.0
@@ -121,7 +121,13 @@ class Trainer(TrainerIOMixin,
                     )
 
                     trainer = Trainer(checkpoint_callback=checkpoint_callback)
-            early_stop_callback (:class:`.EarlyStopping`): Callback for early stopping
+            early_stop_callback (:class:`.EarlyStopping`): Callback for early stopping. If
+                set to ``True``, then the default callback monitoring ``'val_loss'`` is created.
+                Will raise an error if ``'val_loss'`` is not found.
+                If set to ``False``, then early stopping will be disabled.
+                If set to ``None``, then the default callback monitoring ``'val_loss'`` is created.
+                If ``'val_loss'`` is not found will work as if early stopping is disabled.
+                Default: ``None``.
                 Example::
                     from pytorch_lightning.callbacks import EarlyStopping
 
@@ -129,7 +135,8 @@ class Trainer(TrainerIOMixin,
                     early_stop_callback = EarlyStopping(
                         monitor='val_loss',
                         patience=3,
-                        verbose=True,
+                        strict=False,
+                        verbose=False,
                         mode='min'
                     )
 
@@ -801,24 +808,29 @@ class Trainer(TrainerIOMixin,
         ref_model.on_train_start()
         if not self.disable_validation and self.num_sanity_val_steps > 0:
             # init progress bars for validation sanity check
-            pbar = tqdm.tqdm(desc='Validation sanity check',
+            pbar = tqdm(desc='Validation sanity check',
                              total=self.num_sanity_val_steps * len(self.get_val_dataloaders()),
                              leave=False, position=2 * self.process_position,
                              disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch')
             self.main_progress_bar = pbar
             # dummy validation progress bar
-            self.val_progress_bar = tqdm.tqdm(disable=True)
+            self.val_progress_bar = tqdm(disable=True)
 
-            self.evaluate(model, self.get_val_dataloaders(), self.num_sanity_val_steps, self.testing)
+            eval_results = self.evaluate(model, self.get_val_dataloaders(),
+                                         self.num_sanity_val_steps, False)
+            _, _, _, callback_metrics, _ = self.process_output(eval_results)
 
             # close progress bars
             self.main_progress_bar.close()
             self.val_progress_bar.close()
 
+            if self.enable_early_stop:
+                self.early_stop_callback.check_metrics(callback_metrics)
+
         # init progress bar
-        pbar = tqdm.tqdm(leave=True, position=2 * self.process_position,
-                         disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch',
-                         file=sys.stdout)
+        pbar = tqdm(leave=True, position=2 * self.process_position,
+                    disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch',
+                    file=sys.stdout)
         self.main_progress_bar = pbar
 
         # clear cache before training
@@ -850,7 +862,6 @@ class Trainer(TrainerIOMixin,
             # run test from a loaded model
             model = LightningModule.load_from_checkpoint('path/to/checkpoint.ckpt')
             trainer = Trainer()
-
             trainer.test(model)
         """
         self.testing = True

pytorch_lightning/trainer/training_io.py

@@ -98,7 +98,6 @@ from abc import ABC
 from subprocess import call
 from argparse import Namespace
 
-import pandas as pd
 import torch
 import torch.distributed as dist
 

pytorch_lightning/trainer/training_loop.py

@@ -296,6 +296,7 @@ class TrainerTrainLoopMixin(ABC):
             self.current_epoch = epoch
 
             total_val_batches = 0
+            is_val_epoch = False
             if not self.disable_validation:
                 # val can be checked multiple times in epoch
                 is_val_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
@@ -346,13 +347,15 @@ class TrainerTrainLoopMixin(ABC):
 
             # early stopping
             met_min_epochs = epoch >= self.min_epochs - 1
-            if self.enable_early_stop and (met_min_epochs or self.fast_dev_run):
+            if (self.enable_early_stop and not self.disable_validation and is_val_epoch and
+                    (met_min_epochs or self.fast_dev_run)):
                 should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch,
                                                                     logs=self.callback_metrics)
                 # stop training
                 stop = should_stop and met_min_epochs
                 if stop:
                     self.main_progress_bar.close()
+                    model.on_train_end()
                     return
 
         self.main_progress_bar.close()
@@ -400,6 +403,9 @@ class TrainerTrainLoopMixin(ABC):
             if self.fast_dev_run or should_check_val:
                 self.run_evaluation(test=self.testing)
 
+                if self.enable_early_stop:
+                    self.early_stop_callback.check_metrics(self.callback_metrics)
+
             # when logs should be saved
             should_save_log = (batch_idx + 1) % self.log_save_interval == 0 or early_stop_epoch
             if should_save_log or self.fast_dev_run:
@@ -485,13 +491,14 @@ class TrainerTrainLoopMixin(ABC):
 
                     # backward pass
                     model_ref = self.get_model()
-                    model_ref.backward(self.use_amp, closure_loss, optimizer)
+                    model_ref.backward(self.use_amp, closure_loss, optimizer, opt_idx)
 
                     # track metrics for callbacks
                     all_callback_metrics.append(callback_metrics)
 
                     # track progress bar metrics
                     self.add_tqdm_metrics(progress_bar_metrics)
+                    self.add_tqdm_metrics(progress_bar_metrics)
                     all_log_metrics.append(log_metrics)
 
                     # insert after step hook

tests/test_cpu_models.py

@@ -140,7 +140,8 @@ def test_running_test_without_val(tmpdir):
         val_percent_check=0.2,
         test_percent_check=0.2,
         checkpoint_callback=checkpoint,
-        logger=logger
+        logger=logger,
+        early_stop_callback=False
     )
 
     # fit model
@@ -318,6 +319,7 @@ def test_tbptt_cpu_model(tmpdir):
         truncated_bptt_steps=truncated_bptt_steps,
         val_percent_check=0,
         weights_summary=None,
+        early_stop_callback=False
     )
 
     hparams = tutils.get_hparams()

tests/test_trainer.py

@@ -392,7 +392,7 @@ def test_multiple_test_dataloader(tmpdir):
         default_save_path=tmpdir,
         max_epochs=1,
         val_percent_check=0.1,
-        train_percent_check=0.2,
+        train_percent_check=0.2
     )
 
     # fit model