Merge branch 'master' into tb

* fix dangling gradients

make sure only the gradients of the current optimizer's paramaters are calculated in the training step.

* add note about multiple optimizer gradient update

* Update training_loop.py

.travis.yml

@@ -33,13 +33,14 @@ matrix:
       python: 3.7
       env: TOXENV=py37
     - os: osx
-      osx_image: xcode9.4
+      # https://blog.travis-ci.com/2019-08-07-extensive-python-testing-on-travis-ci
+      osx_image: xcode10.3
       language: generic
-      env: TOXENV=py36
-      addons:
-        homebrew:
-          # update: true
-          packages: python3.6
+      env: TOXENV=py37
+      #addons:
+      #  homebrew:
+      #    # update: true
+      #    packages: python3.7
       before_install:
         - pip3 install virtualenv
         - virtualenv -p python3 ~/venv

README.md

@@ -38,10 +38,22 @@ pip install pytorch-lightning
 [Copy and run this COLAB!](https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=HOk9c4_35FKg)
 
 ## What is it?  
-Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts.
+Lightning is a very lightweight wrapper on PyTorch that decouples the science code from the engineering code. It's more of a style-guide than a framework. By refactoring your code, we can automate most of the non-research code.  
+
+To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format (the science) and Lightning will automate the rest (the engineering). Lightning guarantees tested, correct, modern best practices for the automated parts.
+
+- If you are a researcher, Lightning is infinitely flexible, you can modify everything down to the way .backward is called or distributed is set up. 
+- If you are a scientist or production team, lightning is very simple to use with best practice defaults.
+
+## What does lightning control for me?   
+
+Everything in Blue!   
+This is how lightning separates the science (red) from the engineering (blue).
+
+![Overview](docs/source/_static/images/pl.gif)
 
 ## How much effort is it to convert?
-You're probably tired of switching frameworks at this point. But it is a very quick process to refactor into the Lightning format. [Check out this tutorial](https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538)
+You're probably tired of switching frameworks at this point. But it is a very quick process to refactor into the Lightning format (ie: hours). [Check out this tutorial](https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538)
 
 ## Starting a new project?   
 [Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed)     
@@ -179,29 +191,6 @@ When you're all done you can even run the test set separately.
 trainer.test()
 ```
 
-## What does lightning control for me?   
-
-Everything in gray!    
-You define the blue parts using the LightningModule interface:  
-
-![Overview](docs/source/_static/images/overview_flat.jpg)
-
-```python
-# what to do in the training loop
-def training_step(self, batch, batch_idx):
-
-# what to do in the validation loop
-def validation_step(self, batch, batch_idx):
-
-# how to aggregate validation_step outputs
-def validation_end(self, outputs):
-
-# and your dataloaders
-def train_dataloader():
-def val_dataloader():
-def test_dataloader():
-```
-
 **Could be as complex as seq-2-seq + attention**    
 
 ```python

pytorch_lightning/__init__.py

@@ -1,6 +1,6 @@
 """Package info"""
 
-__version__ = '0.5.3.2'
+__version__ = '0.6.0'
 __author__ = 'William Falcon et al.'
 __author_email__ = 'waf2107@columbia.edu'
 __license__ = 'Apache-2.0'

pytorch_lightning/core/__init__.py

@@ -48,8 +48,8 @@ Minimal example
 
         def validation_end(self, outputs):
             # OPTIONAL
-            avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
-            return {'avg_val_loss': avg_loss}
+            val_loss_mean = torch.stack([x['val_loss'] for x in outputs]).mean()
+            return {'val_loss': val_loss_mean}
 
         def test_step(self, batch, batch_idx):
             # OPTIONAL
@@ -59,8 +59,8 @@ Minimal example
 
         def test_end(self, outputs):
             # OPTIONAL
-            avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
-            return {'avg_test_loss': avg_loss}
+            test_loss_mean = torch.stack([x['test_loss'] for x in outputs]).mean()
+            return {'test_loss': test_loss_mean}
 
         def configure_optimizers(self):
             # REQUIRED

pytorch_lightning/core/lightning.py

@@ -664,6 +664,8 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
         .. note:: If you use multiple optimizers, training_step will have an additional `optimizer_idx` parameter.
 
         .. note:: If you use LBFGS lightning handles the closure function automatically for you.
+        
+        .. note:: If you use multiple optimizers, gradients will be calculated only for the parameters of current optimizer at each training step.
 
         Example
         -------
@@ -930,7 +932,7 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
         return None
 
     @classmethod
-    def load_from_metrics(cls, weights_path, tags_csv):
+    def load_from_metrics(cls, weights_path, tags_csv, map_location=None):
         """Primary way of loading model from csv weights path.
 
         :param str weights_path: Path to a PyTorch checkpoint
@@ -975,9 +977,10 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
         hparams = load_hparams_from_tags_csv(tags_csv)
         hparams.__setattr__('on_gpu', False)
 
-        # load on CPU only to avoid OOM issues
-        # then its up to user to put back on GPUs
-        checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)
+        if map_location is not None:
+            checkpoint = torch.load(weights_path, map_location=map_location)
+        else:
+            checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)
 
         # load the state_dict on the model automatically
         model = cls(hparams)
@@ -989,7 +992,7 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
         return model
 
     @classmethod
-    def load_from_checkpoint(cls, checkpoint_path):
+    def load_from_checkpoint(cls, checkpoint_path, map_location=None):
         """
         Primary way of loading model from a checkpoint
         :param checkpoint_path:
@@ -997,9 +1000,11 @@ class LightningModule(GradInformation, ModelIO, ModelHooks):
         :return:
         """
 
-        # load on CPU only to avoid OOM issues
-        # then its up to user to put back on GPUs
-        checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
+        if map_location is not None:
+            checkpoint = torch.load(checkpoint_path, map_location=map_location)
+        else:
+            checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
+
         try:
             ckpt_hparams = checkpoint['hparams']
         except KeyError:

pytorch_lightning/core/model_saving.py

@@ -1,10 +1,10 @@
 """
-.. warning:: `model_saving` module has been renamed to `saving` since v0.5.3 and will be removed in v0.8.0
+.. warning:: `model_saving` module has been renamed to `saving` since v0.6.0 and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`model_saving` module has been renamed to `saving` since v0.5.3"
+warnings.warn("`model_saving` module has been renamed to `saving` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.core.saving import ModelIO  # noqa: E402

pytorch_lightning/core/root_module.py

@@ -1,10 +1,10 @@
 """
-.. warning:: `root_module` module has been renamed to `lightning` since v0.5.3 and will be removed in v0.8.0
+.. warning:: `root_module` module has been renamed to `lightning` since v0.6.0 and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`root_module` module has been renamed to `lightning` since v0.5.3"
+warnings.warn("`root_module` module has been renamed to `lightning` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.core.lightning import LightningModule  # noqa: E402

pytorch_lightning/logging/comet_logger.py

@@ -1,10 +1,10 @@
 """
-.. warning:: `comet_logger` module has been renamed to `comet` since v0.5.3 and will be removed in v0.8.0
+.. warning:: `comet_logger` module has been renamed to `comet` since v0.6.0 and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`comet_logger` module has been renamed to `comet` since v0.5.3"
+warnings.warn("`comet_logger` module has been renamed to `comet` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.logging.comet import CometLogger  # noqa: E402

pytorch_lightning/logging/mlflow.py

@@ -59,7 +59,7 @@ class MLFlowLogger(LightningLoggerBase):
         if expt:
             self._expt_id = expt.experiment_id
         else:
-            logger.warning(f"Experiment with name f{self.experiment_name} not found. Creating it.")
+            logger.warning(f"Experiment with name {self.experiment_name} not found. Creating it.")
             self._expt_id = self._mlflow_client.create_experiment(name=self.experiment_name)
 
         run = self._mlflow_client.create_run(experiment_id=self._expt_id, tags=self.tags)

pytorch_lightning/logging/mlflow_logger.py

@@ -1,10 +1,10 @@
 """
-.. warning:: `mlflow_logger` module has been renamed to `mlflow` since v0.5.3 and will be removed in v0.8.0
+.. warning:: `mlflow_logger` module has been renamed to `mlflow` since v0.6.0 and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`mlflow_logger` module has been renamed to `mlflow` since v0.5.3"
+warnings.warn("`mlflow_logger` module has been renamed to `mlflow` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.logging.mlflow import MLFlowLogger  # noqa: E402

pytorch_lightning/logging/test_tube_logger.py

@@ -1,10 +1,10 @@
 """
-.. warning:: `test_tube_logger` module has been renamed to `test_tube` since v0.5.3 and will be removed in v0.8.0
+.. warning:: `test_tube_logger` module has been renamed to `test_tube` since v0.6.0 and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`test_tube_logger` module has been renamed to `test_tube` since v0.5.3"
+warnings.warn("`test_tube_logger` module has been renamed to `test_tube` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.logging.test_tube import TestTubeLogger  # noqa: E402

pytorch_lightning/overrides/override_data_parallel.py

@@ -1,11 +1,11 @@
 """
-.. warning:: `override_data_parallel` module has been renamed to `data_parallel` since v0.5.3
+.. warning:: `override_data_parallel` module has been renamed to `data_parallel` since v0.6.0
  and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`override_data_parallel` module has been renamed to `data_parallel` since v0.5.3"
+warnings.warn("`override_data_parallel` module has been renamed to `data_parallel` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.overrides.data_parallel import (  # noqa: E402

pytorch_lightning/pt_overrides/__init__.py

@@ -1,10 +1,10 @@
 """
-.. warning:: `pt_overrides` package has been renamed to `overrides` since v0.5.3 and will be removed in v0.8.0
+.. warning:: `pt_overrides` package has been renamed to `overrides` since v0.6.0 and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`pt_overrides` package has been renamed to `overrides` since v0.5.3"
+warnings.warn("`pt_overrides` package has been renamed to `overrides` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.overrides import override_data_parallel  # noqa: E402

pytorch_lightning/root_module/__init__.py

@@ -1,10 +1,10 @@
 """
-.. warning:: `root_module` package has been renamed to `core` since v0.5.3 and will be removed in v0.8.0
+.. warning:: `root_module` package has been renamed to `core` since v0.6.0 and will be removed in v0.8.0
 """
 
 import warnings
 
-warnings.warn("`root_module` package has been renamed to `core` since v0.5.3"
+warnings.warn("`root_module` package has been renamed to `core` since v0.6.0"
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.core import (  # noqa: E402

pytorch_lightning/trainer/callback_config.py

@@ -2,7 +2,7 @@ import os
 from abc import ABC
 
 from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
-from pytorch_lightning.logging import TestTubeLogger
+from pytorch_lightning.logging import TensorboardLogger
 
 
 class TrainerCallbackConfigMixin(ABC):
@@ -69,7 +69,7 @@ class TrainerCallbackConfigMixin(ABC):
         # configure logger
         if logger is True:
             # default logger
-            self.logger = TestTubeLogger(
+            self.logger = TensorboardLogger(
                 save_dir=self.default_save_path,
                 version=self.slurm_job_id,
                 name='lightning_logs'

pytorch_lightning/trainer/data_loading.py

@@ -36,6 +36,15 @@ class TrainerDataLoadingMixin(ABC):
         self.shown_warnings = None
         self.val_check_interval = None
 
+    def _percent_range_check(self, name):
+        value = getattr(self, name)
+        msg = f"`{name}` must lie in the range [0.0, 1.0], but got {value:.3f}."
+        if name == "val_check_interval":
+            msg += " If you want to disable validation set `val_percent_check` to 0.0 instead."
+
+        if not 0. <= value <= 1.:
+            raise ValueError(msg)
+
     def init_train_dataloader(self, model):
         """
         Dataloaders are provided by the model
@@ -48,6 +57,8 @@ class TrainerDataLoadingMixin(ABC):
         if EXIST_ITER_DATASET and isinstance(self.get_train_dataloader().dataset, IterableDataset):
             self.num_training_batches = float('inf')
         else:
+            self._percent_range_check('train_percent_check')
+
             self.num_training_batches = len(self.get_train_dataloader())
             self.num_training_batches = int(self.num_training_batches * self.train_percent_check)
 
@@ -56,7 +67,14 @@ class TrainerDataLoadingMixin(ABC):
         # otherwise, it checks in [0, 1.0] % range of a training epoch
         if isinstance(self.val_check_interval, int):
             self.val_check_batch = self.val_check_interval
+            if self.val_check_batch > self.num_training_batches:
+                raise ValueError(
+                    f"`val_check_interval` ({self.val_check_interval}) must be less than or equal "
+                    f"to the number of the training batches ({self.num_training_batches}). "
+                    f"If you want to disable validation set `val_percent_check` to 0.0 instead.")
         else:
+            self._percent_range_check('val_check_interval')
+
             self.val_check_batch = int(self.num_training_batches * self.val_check_interval)
             self.val_check_batch = max(1, self.val_check_batch)
 
@@ -89,13 +107,15 @@ class TrainerDataLoadingMixin(ABC):
         :return:
         """
         self.get_val_dataloaders = model.val_dataloader
+        self.num_val_batches = 0
 
         # determine number of validation batches
         # val datasets could be none, 1 or 2+
         if self.get_val_dataloaders() is not None:
+            self._percent_range_check('val_percent_check')
+
             self.num_val_batches = sum(len(dataloader) for dataloader in self.get_val_dataloaders())
             self.num_val_batches = int(self.num_val_batches * self.val_percent_check)
-            self.num_val_batches = max(1, self.num_val_batches)
 
         on_ddp = self.use_ddp or self.use_ddp2
         if on_ddp and self.get_val_dataloaders() is not None:
@@ -134,10 +154,11 @@ class TrainerDataLoadingMixin(ABC):
 
         # determine number of test batches
         if self.get_test_dataloaders() is not None:
+            self._percent_range_check('test_percent_check')
+
             len_sum = sum(len(dataloader) for dataloader in self.get_test_dataloaders())
             self.num_test_batches = len_sum
             self.num_test_batches = int(self.num_test_batches * self.test_percent_check)
-            self.num_test_batches = max(1, self.num_test_batches)
 
         on_ddp = self.use_ddp or self.use_ddp2
         if on_ddp and self.get_test_dataloaders() is not None:
@@ -208,6 +229,10 @@ class TrainerDataLoadingMixin(ABC):
         self.val_percent_check = val_percent_check
         self.test_percent_check = test_percent_check
         if overfit_pct > 0:
+            if overfit_pct > 1:
+                raise ValueError(f"`overfit_pct` must be not greater than 1.0, but got "
+                                 f"{overfit_pct:.3f}.")
+
             self.train_percent_check = overfit_pct
             self.val_percent_check = overfit_pct
             self.test_percent_check = overfit_pct

pytorch_lightning/trainer/trainer.py

@@ -139,41 +139,53 @@ class Trainer(TrainerIOMixin,
 
         """
         # Transfer params
-        if nb_gpu_nodes is not None:  # Backward compatibility
+        # Backward compatibility
+        if nb_gpu_nodes is not None:
             warnings.warn("`nb_gpu_nodes` has renamed to `num_nodes` since v0.5.0"
                           " and will be removed in v0.8.0", DeprecationWarning)
             if not num_nodes:  # in case you did not set the proper value
                 num_nodes = nb_gpu_nodes
         self.num_gpu_nodes = num_nodes
+
         self.log_gpu_memory = log_gpu_memory
-        if gradient_clip is not None:  # Backward compatibility
+
+        # Backward compatibility
+        if gradient_clip is not None:
             warnings.warn("`gradient_clip` has renamed to `gradient_clip_val` since v0.5.0"
                           " and will be removed in v0.8.0", DeprecationWarning)
             if not gradient_clip_val:  # in case you did not set the proper value
                 gradient_clip_val = gradient_clip
         self.gradient_clip_val = gradient_clip_val
+
         self.check_val_every_n_epoch = check_val_every_n_epoch
         self.track_grad_norm = track_grad_norm
         self.on_gpu = True if (gpus and torch.cuda.is_available()) else False
         self.process_position = process_position
         self.weights_summary = weights_summary
-        if max_nb_epochs is not None:  # Backward compatibility
+
+        # Backward compatibility
+        if max_nb_epochs is not None:
             warnings.warn("`max_nb_epochs` has renamed to `max_epochs` since v0.5.0"
                           " and will be removed in v0.8.0", DeprecationWarning)
             if not max_epochs:  # in case you did not set the proper value
                 max_epochs = max_nb_epochs
         self.max_epochs = max_epochs
-        if min_nb_epochs is not None:  # Backward compatibility
+
+        # Backward compatibility
+        if min_nb_epochs is not None:
             warnings.warn("`min_nb_epochs` has renamed to `min_epochs` since v0.5.0"
                           " and will be removed in v0.8.0", DeprecationWarning)
             if not min_epochs:  # in case you did not set the proper value
                 min_epochs = min_nb_epochs
         self.min_epochs = min_epochs
-        if nb_sanity_val_steps is not None:  # Backward compatibility
+
+        # Backward compatibility
+        if nb_sanity_val_steps is not None:
             warnings.warn("`nb_sanity_val_steps` has renamed to `num_sanity_val_steps` since v0.5.0"
                           " and will be removed in v0.8.0", DeprecationWarning)
             if not num_sanity_val_steps:  # in case you did not set the proper value
                 num_sanity_val_steps = nb_sanity_val_steps
+
         self.num_sanity_val_steps = num_sanity_val_steps
         self.print_nan_grads = print_nan_grads
         self.truncated_bptt_steps = truncated_bptt_steps
@@ -261,8 +273,9 @@ class Trainer(TrainerIOMixin,
         # logging
         self.log_save_interval = log_save_interval
         self.val_check_interval = val_check_interval
+
+        # backward compatibility
         if add_row_log_interval is not None:
-            # backward compatibility
             warnings.warn("`add_row_log_interval` has renamed to `row_log_interval` since v0.5.0"
                           " and will be removed in v0.8.0", DeprecationWarning)
             if not row_log_interval:  # in case you did not set the proper value

pytorch_lightning/trainer/training_loop.py

@@ -367,6 +367,10 @@ class TrainerTrainLoopMixin(ABC):
 
         # run epoch
         for batch_idx, batch in enumerate(self.get_train_dataloader()):
+            # stop epoch if we limited the number of training batches
+            if batch_idx >= self.num_training_batches:
+                break
+
             self.batch_idx = batch_idx
 
             model = self.get_model()
@@ -413,11 +417,6 @@ class TrainerTrainLoopMixin(ABC):
             if early_stop_epoch or self.fast_dev_run:
                 break
 
-            # stop epoch if we limited the number of training batches
-            met_batch_limit = batch_idx >= self.num_training_batches
-            if met_batch_limit:
-                break
-
         # epoch end hook
         if self.is_function_implemented('on_epoch_end'):
             model = self.get_model()
@@ -455,6 +454,13 @@ class TrainerTrainLoopMixin(ABC):
 
             # call training_step once per optimizer
             for opt_idx, optimizer in enumerate(self.optimizers):
+                # make sure only the gradients of the current optimizer's paramaters are calculated 
+                # in the training step to prevent dangling gradients in multiple-optimizer setup.
+                for param in self.get_model().parameters():
+                    param.requires_grad = False
+                for group in optimizer.param_groups:
+                    for param in group['params']:
+                        param.requires_grad = True
 
                 # wrap the forward step in a closure so second order methods work
                 def optimizer_closure():

requirements.txt

@@ -5,4 +5,4 @@ torch>=1.1
 torchvision>=0.4.0
 pandas>=0.24  # lower version do not support py3.7
 test-tube>=0.7.5
-future>=0.17.1  # required for buildins in setup.py
+future>=0.17.1  # required for builtins in setup.py

tests/conftest.py

@@ -0,0 +1,22 @@
+import pytest
+
+import torch.multiprocessing as mp
+
+
+def pytest_configure(config):
+    config.addinivalue_line("markers", "spawn: spawn test in a separate process using torch.multiprocessing.spawn")
+
+
+def wrap(i, fn, args):
+    return fn(*args)
+
+
+@pytest.mark.tryfirst
+def pytest_pyfunc_call(pyfuncitem):
+    if pyfuncitem.get_closest_marker("spawn"):
+        testfunction = pyfuncitem.obj
+        funcargs = pyfuncitem.funcargs
+        testargs = tuple([funcargs[arg] for arg in pyfuncitem._fixtureinfo.argnames])
+
+        mp.spawn(wrap, (testfunction, testargs))
+        return True

tests/requirements.txt

@@ -8,4 +8,5 @@ check-manifest
 # test_tube  # already installed in main req.
 mlflow
 comet_ml
-twine==1.13.0
+twine==1.13.0
+pillow<7.0.0

tests/test_amp.py

@@ -32,6 +32,7 @@ def test_amp_single_gpu(tmpdir):
     tutils.run_model_test(trainer_options, model)
 
 
+@pytest.mark.spawn
 def test_no_amp_single_gpu(tmpdir):
     """Make sure DDP + AMP work."""
     tutils.reset_seed()
@@ -51,8 +52,10 @@ def test_no_amp_single_gpu(tmpdir):
         use_amp=True
     )
 
-    with pytest.raises((MisconfigurationException, ModuleNotFoundError)):
-        tutils.run_model_test(trainer_options, model)
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    assert result == 1
 
 
 def test_amp_gpu_ddp(tmpdir):
@@ -78,6 +81,7 @@ def test_amp_gpu_ddp(tmpdir):
     tutils.run_model_test(trainer_options, model)
 
 
+@pytest.mark.spawn
 def test_amp_gpu_ddp_slurm_managed(tmpdir):
     """Make sure DDP + AMP work."""
     if not tutils.can_run_gpu_test():
@@ -124,26 +128,6 @@ def test_amp_gpu_ddp_slurm_managed(tmpdir):
     assert trainer.resolve_root_node_address('abc[23-24]') == 'abc23'
     assert trainer.resolve_root_node_address('abc[23-24, 45-40, 40]') == 'abc23'
 
-    # test model loading with a map_location
-    pretrained_model = tutils.load_model(logger.experiment, trainer.checkpoint_callback.filepath)
-
-    # test model preds
-    for dataloader in trainer.get_test_dataloaders():
-        tutils.run_prediction(dataloader, pretrained_model)
-
-    if trainer.use_ddp:
-        # on hpc this would work fine... but need to hack it for the purpose of the test
-        trainer.model = pretrained_model
-        trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers()
-
-    # test HPC loading / saving
-    trainer.hpc_save(tmpdir, logger)
-    trainer.hpc_load(tmpdir, on_gpu=True)
-
-    # test freeze on gpu
-    model.freeze()
-    model.unfreeze()
-
 
 def test_cpu_model_with_amp(tmpdir):
     """Make sure model trains on CPU."""
@@ -165,6 +149,7 @@ def test_cpu_model_with_amp(tmpdir):
         tutils.run_model_test(trainer_options, model, on_gpu=False)
 
 
+@pytest.mark.spawn
 def test_amp_gpu_dp(tmpdir):
     """Make sure DP + AMP work."""
     tutils.reset_seed()
@@ -180,8 +165,11 @@ def test_amp_gpu_dp(tmpdir):
         distributed_backend='dp',
         use_amp=True
     )
-    with pytest.raises(MisconfigurationException):
-        tutils.run_model_test(trainer_options, model, hparams)
+
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    assert result == 1
 
 
 if __name__ == '__main__':

tests/test_cpu_models.py

@@ -97,7 +97,7 @@ def test_running_test_after_fitting(tmpdir):
     trainer_options = dict(
         default_save_path=tmpdir,
         show_progress_bar=False,
-        max_epochs=1,
+        max_epochs=4,
         train_percent_check=0.4,
         val_percent_check=0.2,
         test_percent_check=0.2,

tests/test_logging.py

@@ -309,7 +309,7 @@ def test_custom_logger(tmpdir):
 
     trainer_options = dict(
         max_epochs=1,
-        train_percent_check=0.01,
+        train_percent_check=0.05,
         logger=logger,
         default_save_path=tmpdir
     )

tests/test_restore_models.py

@@ -153,7 +153,7 @@ def test_running_test_pretrained_model_dp(tmpdir):
 
     trainer_options = dict(
         show_progress_bar=True,
-        max_epochs=1,
+        max_epochs=4,
         train_percent_check=0.4,
         val_percent_check=0.2,
         checkpoint_callback=checkpoint,
@@ -269,12 +269,12 @@ def test_cpu_restore_training(tmpdir):
     logger = tutils.get_test_tube_logger(tmpdir, False, version=test_logger_version)
 
     trainer_options = dict(
-        max_epochs=2,
+        max_epochs=8,
         val_check_interval=0.50,
         val_percent_check=0.2,
         train_percent_check=0.2,
         logger=logger,
-        checkpoint_callback=ModelCheckpoint(tmpdir)
+        checkpoint_callback=ModelCheckpoint(tmpdir, save_top_k=-1)
     )
 
     # fit model

tests/test_trainer.py

@@ -393,7 +393,7 @@ def test_multiple_test_dataloader(tmpdir):
         default_save_path=tmpdir,
         max_epochs=1,
         val_percent_check=0.1,
-        train_percent_check=0.1,
+        train_percent_check=0.2,
     )
 
     # fit model