diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index d98076fb..6ae828ac 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -11,32 +11,52 @@ assignees: '' 1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/williamFalcon/pytorch-lightning/issues/79). 2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/williamFalcon/pytorch-lightning#faq) -### Describe the bug -A clear and concise description of what the bug is. +## 🐛 Bug + + + +### To Reproduce -#### To Reproduce Steps to reproduce the behavior: + 1. Go to '...' 2. Run '....' 3. Scroll down to '....' 4. See error + + + #### Code sample -Ideally attach a minimal code sample to reproduce the decried issue. -[Minimal means having the shortest code but still preserving the bug] + -#### Expected behavior -A clear and concise description of what you expected to happen. +### Expected behavior -#### Screenshots -If applicable, add screenshots to help explain your problem. + -### Environment information +### Environment -Desktop (please complete the following information): - - OS: [e.g. iOS, Linux, Win] - - Packaging [e.g. pip, conda] - - Version [e.g. 0.5.2.1] +Please copy and paste the output from our +[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py) +(or fill out the checklist below manually). + +You can get the script and run it with: +``` +wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py +# For security purposes, please check the contents of collect_env.py before running it. +python collect_env.py +``` + + - PyTorch Version (e.g., 1.0): + - OS (e.g., Linux): + - How you installed PyTorch (`conda`, `pip`, source): + - Build command you used (if compiling from source): + - Python version: + - CUDA/cuDNN version: + - GPU models and configuration: + - Any other relevant information: ### Additional context -Add any other context about the problem here. + + diff --git a/.github/ISSUE_TEMPLATE/typos-and-doc-fixes.md b/.github/ISSUE_TEMPLATE/documentation.md similarity index 89% rename from .github/ISSUE_TEMPLATE/typos-and-doc-fixes.md rename to .github/ISSUE_TEMPLATE/documentation.md index b9b61ef4..1162def1 100644 --- a/.github/ISSUE_TEMPLATE/typos-and-doc-fixes.md +++ b/.github/ISSUE_TEMPLATE/documentation.md @@ -7,7 +7,7 @@ assignees: '' --- -## Typos and docs fixes +## 📚 Documentation For typos and doc fixes, please go ahead and: diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index dcb4a88a..3350917a 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -7,14 +7,21 @@ assignees: '' --- -**Is your feature request related to a problem? Please describe.** -A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] +## 🚀 Feature + -**Describe the solution you'd like** -A clear and concise description of what you want to happen. +### Motivation -**Describe alternatives you've considered** -A clear and concise description of any alternative solutions or features you've considered. + -**Additional context** -Add any other context or screenshots about the feature request here. +### Pitch + + + +### Alternatives + + + +### Additional context + + diff --git a/.github/ISSUE_TEMPLATE/how-to-question.md b/.github/ISSUE_TEMPLATE/how-to-question.md index f18f0a6a..87e0f3ec 100644 --- a/.github/ISSUE_TEMPLATE/how-to-question.md +++ b/.github/ISSUE_TEMPLATE/how-to-question.md @@ -7,20 +7,24 @@ assignees: '' --- +## ❓ Questions and Help + ### Before asking: 1. search the issues. 2. search the docs. -If you still can't find what you need: -#### What is your question? + -#### Code -Please paste a code snippet if your question requires it! +#### What is your question? -#### What have you tried? +#### Code -#### What's your environment? -- conda version (no venv) -- PyTorch version -- Lightning version -- Test-tube version + + +#### What have you tried? + +#### What's your environment? + + - OS: [e.g. iOS, Linux, Win] + - Packaging [e.g. pip, conda] + - Version [e.g. 0.5.2.1] diff --git a/.travis.yml b/.travis.yml index 84fb911d..aea3b67d 100644 --- a/.travis.yml +++ b/.travis.yml @@ -33,13 +33,14 @@ matrix: python: 3.7 env: TOXENV=py37 - os: osx - osx_image: xcode9.4 + # https://blog.travis-ci.com/2019-08-07-extensive-python-testing-on-travis-ci + osx_image: xcode10.3 language: generic - env: TOXENV=py36 - addons: - homebrew: - # update: true - packages: python3.6 + env: TOXENV=py37 + #addons: + # homebrew: + # # update: true + # packages: python3.7 before_install: - pip3 install virtualenv - virtualenv -p python3 ~/venv diff --git a/README.md b/README.md index 16cd3993..4be0bfad 100644 --- a/README.md +++ b/README.md @@ -31,16 +31,29 @@ Simple installation from PyPI pip install pytorch-lightning ``` -[LIVE COLAB DEMO](https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=HOk9c4_35FKg) - ## Docs **[View the docs here](https://williamfalcon.github.io/pytorch-lightning/)** +## Demo +[Copy and run this COLAB!](https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=HOk9c4_35FKg) + ## What is it? -Lightning is a very lightweight wrapper on PyTorch. This means you don't have to learn a new library. To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format and Lightning will automate the rest. Lightning guarantees tested, correct, modern best practices for the automated parts. +Lightning is a very lightweight wrapper on PyTorch that decouples the science code from the engineering code. It's more of a style-guide than a framework. By refactoring your code, we can automate most of the non-research code. + +To use Lightning, simply refactor your research code into the [LightningModule](https://github.com/williamFalcon/pytorch-lightning#how-do-i-do-use-it) format (the science) and Lightning will automate the rest (the engineering). Lightning guarantees tested, correct, modern best practices for the automated parts. + +- If you are a researcher, Lightning is infinitely flexible, you can modify everything down to the way .backward is called or distributed is set up. +- If you are a scientist or production team, lightning is very simple to use with best practice defaults. + +## What does lightning control for me? + +Everything in Blue! +This is how lightning separates the science (red) from the engineering (blue). + +![Overview](docs/source/_static/images/pl.gif) ## How much effort is it to convert? -You're probably tired of switching frameworks at this point. But it is a very quick process to refactor into the Lightning format. [Check out this tutorial](https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538) +You're probably tired of switching frameworks at this point. But it is a very quick process to refactor into the Lightning format (ie: hours). [Check out this tutorial](https://towardsdatascience.com/how-to-refactor-your-pytorch-code-to-get-these-42-benefits-of-pytorch-lighting-6fdd0dc97538) ## Starting a new project? [Use our seed-project aimed at reproducibility!](https://github.com/williamFalcon/pytorch-lightning-conference-seed) @@ -178,29 +191,6 @@ When you're all done you can even run the test set separately. trainer.test() ``` -## What does lightning control for me? - -Everything in gray! -You define the blue parts using the LightningModule interface: - -![Overview](docs/source/_static/images/overview_flat.jpg) - -```python -# what to do in the training loop -def training_step(self, batch, batch_idx): - -# what to do in the validation loop -def validation_step(self, batch, batch_idx): - -# how to aggregate validation_step outputs -def validation_end(self, outputs): - -# and your dataloaders -def train_dataloader(): -def val_dataloader(): -def test_dataloader(): -``` - **Could be as complex as seq-2-seq + attention** ```python diff --git a/docs/source/_static/images/pl.gif b/docs/source/_static/images/pl.gif new file mode 100644 index 00000000..b2c37444 Binary files /dev/null and b/docs/source/_static/images/pl.gif differ diff --git a/pytorch_lightning/__init__.py b/pytorch_lightning/__init__.py index 6252e343..3ef4f4a1 100644 --- a/pytorch_lightning/__init__.py +++ b/pytorch_lightning/__init__.py @@ -1,6 +1,6 @@ """Package info""" -__version__ = '0.5.3.2' +__version__ = '0.6.0' __author__ = 'William Falcon et al.' __author_email__ = 'waf2107@columbia.edu' __license__ = 'Apache-2.0' diff --git a/pytorch_lightning/core/__init__.py b/pytorch_lightning/core/__init__.py index 0bb2323f..c2694eab 100644 --- a/pytorch_lightning/core/__init__.py +++ b/pytorch_lightning/core/__init__.py @@ -48,8 +48,8 @@ Minimal example def validation_end(self, outputs): # OPTIONAL - avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean() - return {'avg_val_loss': avg_loss} + val_loss_mean = torch.stack([x['val_loss'] for x in outputs]).mean() + return {'val_loss': val_loss_mean} def test_step(self, batch, batch_idx): # OPTIONAL @@ -59,8 +59,8 @@ Minimal example def test_end(self, outputs): # OPTIONAL - avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean() - return {'avg_test_loss': avg_loss} + test_loss_mean = torch.stack([x['test_loss'] for x in outputs]).mean() + return {'test_loss': test_loss_mean} def configure_optimizers(self): # REQUIRED diff --git a/pytorch_lightning/core/lightning.py b/pytorch_lightning/core/lightning.py index ef2e4d4c..2e62e03b 100644 --- a/pytorch_lightning/core/lightning.py +++ b/pytorch_lightning/core/lightning.py @@ -664,6 +664,8 @@ class LightningModule(GradInformation, ModelIO, ModelHooks): .. note:: If you use multiple optimizers, training_step will have an additional `optimizer_idx` parameter. .. note:: If you use LBFGS lightning handles the closure function automatically for you. + + .. note:: If you use multiple optimizers, gradients will be calculated only for the parameters of current optimizer at each training step. Example ------- @@ -930,7 +932,7 @@ class LightningModule(GradInformation, ModelIO, ModelHooks): return None @classmethod - def load_from_metrics(cls, weights_path, tags_csv): + def load_from_metrics(cls, weights_path, tags_csv, map_location=None): """Primary way of loading model from csv weights path. :param str weights_path: Path to a PyTorch checkpoint @@ -975,9 +977,10 @@ class LightningModule(GradInformation, ModelIO, ModelHooks): hparams = load_hparams_from_tags_csv(tags_csv) hparams.__setattr__('on_gpu', False) - # load on CPU only to avoid OOM issues - # then its up to user to put back on GPUs - checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage) + if map_location is not None: + checkpoint = torch.load(weights_path, map_location=map_location) + else: + checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage) # load the state_dict on the model automatically model = cls(hparams) @@ -989,7 +992,7 @@ class LightningModule(GradInformation, ModelIO, ModelHooks): return model @classmethod - def load_from_checkpoint(cls, checkpoint_path): + def load_from_checkpoint(cls, checkpoint_path, map_location=None): """ Primary way of loading model from a checkpoint :param checkpoint_path: @@ -997,9 +1000,11 @@ class LightningModule(GradInformation, ModelIO, ModelHooks): :return: """ - # load on CPU only to avoid OOM issues - # then its up to user to put back on GPUs - checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage) + if map_location is not None: + checkpoint = torch.load(checkpoint_path, map_location=map_location) + else: + checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage) + try: ckpt_hparams = checkpoint['hparams'] except KeyError: diff --git a/pytorch_lightning/core/memory.py b/pytorch_lightning/core/memory.py index b01451f8..1abc349e 100644 --- a/pytorch_lightning/core/memory.py +++ b/pytorch_lightning/core/memory.py @@ -50,20 +50,31 @@ class ModelSummary(object): input_ = self.model.example_input_array if self.model.on_gpu: - input_ = input_.cuda(0) + device = next(self.model.parameters()).get_device() + # test if input is a list or a tuple + if isinstance(input_, (list, tuple)): + input_ = [input_i.cuda(device) if torch.is_tensor(input_i) else input_i + for input_i in input_] + else: + input_ = input_.cuda(device) if self.model.trainer.use_amp: - input_ = input_.half() + # test if it is not a list or a tuple + if isinstance(input_, (list, tuple)): + input_ = [input_i.half() if torch.is_tensor(input_i) else input_i + for input_i in input_] + else: + input_ = input_.half() with torch.no_grad(): for _, m in mods: - if type(input_) is list or type(input_) is tuple: # pragma: no cover + if isinstance(input_, (list, tuple)): # pragma: no cover out = m(*input_) else: out = m(input_) - if type(input_) is tuple or type(input_) is list: # pragma: no cover + if isinstance(input_, (list, tuple)): # pragma: no cover in_size = [] for x in input_: if type(x) is list: @@ -75,7 +86,7 @@ class ModelSummary(object): in_sizes.append(in_size) - if type(out) is tuple or type(out) is list: # pragma: no cover + if isinstance(out, (list, tuple)): # pragma: no cover out_size = np.asarray([x.size() for x in out]) else: out_size = np.array(out.size()) diff --git a/pytorch_lightning/core/model_saving.py b/pytorch_lightning/core/model_saving.py index a7bf1e6b..d607422c 100644 --- a/pytorch_lightning/core/model_saving.py +++ b/pytorch_lightning/core/model_saving.py @@ -1,10 +1,10 @@ """ -.. warning:: `model_saving` module has been renamed to `saving` since v0.5.3 and will be removed in v0.8.0 +.. warning:: `model_saving` module has been renamed to `saving` since v0.6.0 and will be removed in v0.8.0 """ import warnings -warnings.warn("`model_saving` module has been renamed to `saving` since v0.5.3" +warnings.warn("`model_saving` module has been renamed to `saving` since v0.6.0" " and will be removed in v0.8.0", DeprecationWarning) from pytorch_lightning.core.saving import ModelIO # noqa: E402 diff --git a/pytorch_lightning/core/root_module.py b/pytorch_lightning/core/root_module.py index 43eb8276..9f8612bd 100644 --- a/pytorch_lightning/core/root_module.py +++ b/pytorch_lightning/core/root_module.py @@ -1,10 +1,10 @@ """ -.. warning:: `root_module` module has been renamed to `lightning` since v0.5.3 and will be removed in v0.8.0 +.. warning:: `root_module` module has been renamed to `lightning` since v0.6.0 and will be removed in v0.8.0 """ import warnings -warnings.warn("`root_module` module has been renamed to `lightning` since v0.5.3" +warnings.warn("`root_module` module has been renamed to `lightning` since v0.6.0" " and will be removed in v0.8.0", DeprecationWarning) from pytorch_lightning.core.lightning import LightningModule # noqa: E402 diff --git a/pytorch_lightning/logging/__init__.py b/pytorch_lightning/logging/__init__.py index 8a6c3d03..fca3be61 100644 --- a/pytorch_lightning/logging/__init__.py +++ b/pytorch_lightning/logging/__init__.py @@ -168,7 +168,7 @@ Every k batches, lightning will write the new logs to disk from os import environ from .base import LightningLoggerBase, rank_zero_only -from .tensorboard import TensorboardLogger +from .tensorboard import TensorBoardLogger try: from .test_tube import TestTubeLogger diff --git a/pytorch_lightning/logging/comet.py b/pytorch_lightning/logging/comet.py index b7476066..3fe25418 100644 --- a/pytorch_lightning/logging/comet.py +++ b/pytorch_lightning/logging/comet.py @@ -52,7 +52,11 @@ from logging import getLogger try: from comet_ml import Experiment as CometExperiment from comet_ml import OfflineExperiment as CometOfflineExperiment - from comet_ml.papi import API + try: + from comet_ml.api import API + except ImportError: + # For more information, see: https://www.comet.ml/docs/python-sdk/releases/#release-300 + from comet_ml.papi import API except ImportError: raise ImportError('Missing comet_ml package.') diff --git a/pytorch_lightning/logging/comet_logger.py b/pytorch_lightning/logging/comet_logger.py index 93cf32b0..4ea0c071 100644 --- a/pytorch_lightning/logging/comet_logger.py +++ b/pytorch_lightning/logging/comet_logger.py @@ -1,10 +1,10 @@ """ -.. warning:: `comet_logger` module has been renamed to `comet` since v0.5.3 and will be removed in v0.8.0 +.. warning:: `comet_logger` module has been renamed to `comet` since v0.6.0 and will be removed in v0.8.0 """ import warnings -warnings.warn("`comet_logger` module has been renamed to `comet` since v0.5.3" +warnings.warn("`comet_logger` module has been renamed to `comet` since v0.6.0" " and will be removed in v0.8.0", DeprecationWarning) from pytorch_lightning.logging.comet import CometLogger # noqa: E402 diff --git a/pytorch_lightning/logging/mlflow.py b/pytorch_lightning/logging/mlflow.py index fde2945b..5769693d 100644 --- a/pytorch_lightning/logging/mlflow.py +++ b/pytorch_lightning/logging/mlflow.py @@ -59,7 +59,7 @@ class MLFlowLogger(LightningLoggerBase): if expt: self._expt_id = expt.experiment_id else: - logger.warning(f"Experiment with name f{self.experiment_name} not found. Creating it.") + logger.warning(f"Experiment with name {self.experiment_name} not found. Creating it.") self._expt_id = self._mlflow_client.create_experiment(name=self.experiment_name) run = self._mlflow_client.create_run(experiment_id=self._expt_id, tags=self.tags) diff --git a/pytorch_lightning/logging/mlflow_logger.py b/pytorch_lightning/logging/mlflow_logger.py index a2d0d17e..aa56f5ff 100644 --- a/pytorch_lightning/logging/mlflow_logger.py +++ b/pytorch_lightning/logging/mlflow_logger.py @@ -1,10 +1,10 @@ """ -.. warning:: `mlflow_logger` module has been renamed to `mlflow` since v0.5.3 and will be removed in v0.8.0 +.. warning:: `mlflow_logger` module has been renamed to `mlflow` since v0.6.0 and will be removed in v0.8.0 """ import warnings -warnings.warn("`mlflow_logger` module has been renamed to `mlflow` since v0.5.3" +warnings.warn("`mlflow_logger` module has been renamed to `mlflow` since v0.6.0" " and will be removed in v0.8.0", DeprecationWarning) from pytorch_lightning.logging.mlflow import MLFlowLogger # noqa: E402 diff --git a/pytorch_lightning/logging/tensorboard.py b/pytorch_lightning/logging/tensorboard.py index 338b02cc..26d626d1 100644 --- a/pytorch_lightning/logging/tensorboard.py +++ b/pytorch_lightning/logging/tensorboard.py @@ -8,8 +8,8 @@ from torch.utils.tensorboard import SummaryWriter from .base import LightningLoggerBase, rank_zero_only -class TensorboardLogger(LightningLoggerBase): - r"""Log to local file system in Tensorboard format +class TensorBoardLogger(LightningLoggerBase): + r"""Log to local file system in TensorBoard format Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to `os.path.join(save_dir, name, version)` @@ -18,7 +18,7 @@ class TensorboardLogger(LightningLoggerBase): .. code-block:: python - logger = TensorboardLogger("tb_logs", name="my_model") + logger = TensorBoardLogger("tb_logs", name="my_model") trainer = Trainer(logger=logger) trainer.train(model) @@ -35,7 +35,7 @@ class TensorboardLogger(LightningLoggerBase): super().__init__() self.save_dir = save_dir self._name = name - self._version = version if version is not None else None + self._version = version self._experiment = None self.kwargs = kwargs @@ -59,23 +59,35 @@ class TensorboardLogger(LightningLoggerBase): def log_hyperparams(self, params): if parse_version(torch.__version__) < parse_version("1.3.0"): warn( - f"Hyperparameter logging is not available for Torch version {torch.__version__}. " - "Skipping log_hyperparams. Upgrade to Torch 1.3.0 or above to enable " - "hyperparameter logging" + f"Hyperparameter logging is not available for Torch version {torch.__version__}." + " Skipping log_hyperparams. Upgrade to Torch 1.3.0 or above to enable" + " hyperparameter logging." ) + # TODO: some alternative should be added return - self.experiment.add_hparams(hparam_dict=vars(params)) + try: + # in case converting from namespace, todo: rather test if it is namespace + params = vars(params) + except TypeError: + pass + if params is not None: + # `add_hparams` requires both - hparams and metric + self.experiment.add_hparams(hparam_dict=dict(params), metric_dict={}) @rank_zero_only - def log_metrics(self, metrics, step_idx=None): + def log_metrics(self, metrics, step=None): for k, v in metrics.items(): if isinstance(v, torch.Tensor): v = v.item() - self.experiment.add_scalar(k, v, step_idx) + self.experiment.add_scalar(k, v, step) @rank_zero_only def save(self): - self.experiment.flush() + try: + self.experiment.flush() + except AttributeError: + # you are using PT version ( self.num_training_batches: + raise ValueError( + f"`val_check_interval` ({self.val_check_interval}) must be less than or equal " + f"to the number of the training batches ({self.num_training_batches}). " + f"If you want to disable validation set `val_percent_check` to 0.0 instead.") else: + self._percent_range_check('val_check_interval') + self.val_check_batch = int(self.num_training_batches * self.val_check_interval) self.val_check_batch = max(1, self.val_check_batch) @@ -89,13 +107,15 @@ class TrainerDataLoadingMixin(ABC): :return: """ self.get_val_dataloaders = model.val_dataloader + self.num_val_batches = 0 # determine number of validation batches # val datasets could be none, 1 or 2+ if self.get_val_dataloaders() is not None: + self._percent_range_check('val_percent_check') + self.num_val_batches = sum(len(dataloader) for dataloader in self.get_val_dataloaders()) self.num_val_batches = int(self.num_val_batches * self.val_percent_check) - self.num_val_batches = max(1, self.num_val_batches) on_ddp = self.use_ddp or self.use_ddp2 if on_ddp and self.get_val_dataloaders() is not None: @@ -134,10 +154,11 @@ class TrainerDataLoadingMixin(ABC): # determine number of test batches if self.get_test_dataloaders() is not None: + self._percent_range_check('test_percent_check') + len_sum = sum(len(dataloader) for dataloader in self.get_test_dataloaders()) self.num_test_batches = len_sum self.num_test_batches = int(self.num_test_batches * self.test_percent_check) - self.num_test_batches = max(1, self.num_test_batches) on_ddp = self.use_ddp or self.use_ddp2 if on_ddp and self.get_test_dataloaders() is not None: @@ -208,6 +229,10 @@ class TrainerDataLoadingMixin(ABC): self.val_percent_check = val_percent_check self.test_percent_check = test_percent_check if overfit_pct > 0: + if overfit_pct > 1: + raise ValueError(f"`overfit_pct` must be not greater than 1.0, but got " + f"{overfit_pct:.3f}.") + self.train_percent_check = overfit_pct self.val_percent_check = overfit_pct self.test_percent_check = overfit_pct diff --git a/pytorch_lightning/trainer/evaluation_loop.py b/pytorch_lightning/trainer/evaluation_loop.py index 4e507bdd..03824777 100644 --- a/pytorch_lightning/trainer/evaluation_loop.py +++ b/pytorch_lightning/trainer/evaluation_loop.py @@ -145,8 +145,8 @@ class TrainerEvaluationLoopMixin(ABC): self.single_gpu = None self.data_parallel_device_ids = None self.model = None - self.nb_test_batches = None - self.nb_val_batches = None + self.num_test_batches = None + self.num_val_batches = None self.fast_dev_run = None self.process_position = None self.show_progress_bar = None diff --git a/pytorch_lightning/trainer/logging.py b/pytorch_lightning/trainer/logging.py index 8a178917..7c445c07 100644 --- a/pytorch_lightning/trainer/logging.py +++ b/pytorch_lightning/trainer/logging.py @@ -21,7 +21,7 @@ class TrainerLoggingMixin(ABC): self.use_ddp2 = None self.num_gpus = None - def log_metrics(self, metrics, grad_norm_dic): + def log_metrics(self, metrics, grad_norm_dic, step=None): """Logs the metric dict passed in. :param metrics: @@ -41,9 +41,10 @@ class TrainerLoggingMixin(ABC): # turn all tensors to scalars scalar_metrics = self.metrics_to_scalars(metrics) + step = step if step is not None else self.global_step # log actual metrics if self.proc_rank == 0 and self.logger is not None: - self.logger.log_metrics(scalar_metrics, step=self.global_step) + self.logger.log_metrics(scalar_metrics, step=step) self.logger.save() def add_tqdm_metrics(self, metrics): @@ -176,7 +177,7 @@ class TrainerLoggingMixin(ABC): elif isinstance(output[k], torch.Tensor) and output[k].dim() == 0: pass - # reduce only metrics that have the same nb of gpus + # reduce only metrics that have the same number of gpus elif output[k].size(0) == num_gpus: reduced = torch.mean(output[k]) output[k] = reduced diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py index 2e07b17c..ee264342 100644 --- a/pytorch_lightning/trainer/trainer.py +++ b/pytorch_lightning/trainer/trainer.py @@ -359,7 +359,7 @@ class Trainer(TrainerIOMixin, tqdm_dict['split_idx'] = self.split_idx if self.logger is not None and self.logger.version is not None: - tqdm_dict['v_nb'] = self.logger.version + tqdm_dict['v_num'] = self.logger.version tqdm_dict.update(self.tqdm_metrics) diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py index 2431f88c..cc9e6e21 100644 --- a/pytorch_lightning/trainer/training_loop.py +++ b/pytorch_lightning/trainer/training_loop.py @@ -50,6 +50,9 @@ To modify this behavior, pass in your own EarlyStopping callback. # pass in your own to override the default callback trainer = Trainer(early_stop_callback=early_stop_callback) + # pass in min_epochs to enable the callback after min_epochs have run + trainer = Trainer(early_stop_callback=early_stop_callback, min_epochs=5) + # pass in None to disable it trainer = Trainer(early_stop_callback=None) @@ -151,6 +154,7 @@ When this flag is enabled each batch is split into sequences of size truncated_b import inspect from abc import ABC, abstractmethod +import warnings import numpy as np @@ -169,22 +173,22 @@ class TrainerTrainLoopMixin(ABC): def __init__(self): # this is just a summary on variables used in this abstract class, # the proper values/initialisation should be done in child class - self.max_nb_epochs = None + self.max_epochs = None + self.min_epochs = None self.use_ddp = None self.use_dp = None self.use_ddp2 = None self.single_gpu = None self.data_parallel_device_ids = None self.check_val_every_n_epoch = None - self.nb_training_batches = None + self.num_training_batches = None self.val_check_batch = None - self.nb_val_batches = None + self.num_val_batches = None self.fast_dev_run = None self.is_iterable_train_dataloader = None self.main_progress_bar = None self.accumulation_scheduler = None self.lr_schedulers = None - self.min_nb_epochs = None self.enable_early_stop = None self.early_stop_callback = None self.callback_metrics = None @@ -194,7 +198,7 @@ class TrainerTrainLoopMixin(ABC): self.log_save_interval = None self.proc_rank = None self.row_log_interval = None - self.total_batch_nb = None + self.total_batches = None self.truncated_bptt_steps = None self.optimizers = None self.accumulate_grad_batches = None @@ -207,6 +211,24 @@ class TrainerTrainLoopMixin(ABC): self.get_train_dataloader = None self.reduce_lr_on_plateau_scheduler = None + @property + def max_nb_epochs(self): + """ + .. warning:: `max_nb_epochs` is deprecated and will be removed in v0.8.0, use `max_epochs` instead. + """ + warnings.warn("`max_nb_epochs` is deprecated and will be removed in " + "v0.8.0, use `max_epochs` instead.", DeprecationWarning) + return self.max_epochs + + @property + def min_nb_epochs(self): + """ + .. warning:: `min_nb_epochs` is deprecated and will be removed in v0.8.0, use `min_epochs` instead. + """ + warnings.warn("`min_nb_epochs` is deprecated and will be removed in " + "v0.8.0, use `min_epochs` instead.", DeprecationWarning) + return self.min_epochs + @abstractmethod def get_model(self): # this is just empty shell for code from other class @@ -320,7 +342,7 @@ class TrainerTrainLoopMixin(ABC): self.reduce_lr_on_plateau_scheduler.step(val_loss, epoch=self.current_epoch) # early stopping - met_min_epochs = epoch > self.min_epochs + met_min_epochs = epoch >= self.min_epochs - 1 if self.enable_early_stop and (met_min_epochs or self.fast_dev_run): should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch, logs=self.callback_metrics) @@ -345,6 +367,10 @@ class TrainerTrainLoopMixin(ABC): # run epoch for batch_idx, batch in enumerate(self.get_train_dataloader()): + # stop epoch if we limited the number of training batches + if batch_idx >= self.num_training_batches: + break + self.batch_idx = batch_idx model = self.get_model() @@ -391,11 +417,6 @@ class TrainerTrainLoopMixin(ABC): if early_stop_epoch or self.fast_dev_run: break - # stop epoch if we limited nb batches - met_batch_limit = batch_idx >= self.num_training_batches - if met_batch_limit: - break - # epoch end hook if self.is_function_implemented('on_epoch_end'): model = self.get_model() @@ -433,6 +454,13 @@ class TrainerTrainLoopMixin(ABC): # call training_step once per optimizer for opt_idx, optimizer in enumerate(self.optimizers): + # make sure only the gradients of the current optimizer's paramaters are calculated + # in the training step to prevent dangling gradients in multiple-optimizer setup. + for param in self.get_model().parameters(): + param.requires_grad = False + for group in optimizer.param_groups: + for param in group['params']: + param.requires_grad = True # wrap the forward step in a closure so second order methods work def optimizer_closure(): diff --git a/requirements.txt b/requirements.txt index c6dc2c75..26d9084a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,4 +4,5 @@ numpy>=1.16.4 torch>=1.1 torchvision>=0.4.0 pandas>=0.24 # lower version do not support py3.7 -future>=0.17.1 # required for buildins in setup.py +test-tube>=0.7.5 +future>=0.17.1 # required for builtins in setup.py \ No newline at end of file diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 00000000..bfb2b0d5 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,22 @@ +import pytest + +import torch.multiprocessing as mp + + +def pytest_configure(config): + config.addinivalue_line("markers", "spawn: spawn test in a separate process using torch.multiprocessing.spawn") + + +def wrap(i, fn, args): + return fn(*args) + + +@pytest.mark.tryfirst +def pytest_pyfunc_call(pyfuncitem): + if pyfuncitem.get_closest_marker("spawn"): + testfunction = pyfuncitem.obj + funcargs = pyfuncitem.funcargs + testargs = tuple([funcargs[arg] for arg in pyfuncitem._fixtureinfo.argnames]) + + mp.spawn(wrap, (testfunction, testargs)) + return True diff --git a/tests/requirements.txt b/tests/requirements.txt index 1ff186d8..bc3fcc8f 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -8,4 +8,5 @@ check-manifest # test_tube # already installed in main req. mlflow comet_ml -twine==1.13.0 \ No newline at end of file +twine==1.13.0 +pillow<7.0.0 \ No newline at end of file diff --git a/tests/test_amp.py b/tests/test_amp.py index 85025a56..4d4ae22d 100644 --- a/tests/test_amp.py +++ b/tests/test_amp.py @@ -32,6 +32,7 @@ def test_amp_single_gpu(tmpdir): tutils.run_model_test(trainer_options, model) +@pytest.mark.spawn def test_no_amp_single_gpu(tmpdir): """Make sure DDP + AMP work.""" tutils.reset_seed() @@ -51,8 +52,10 @@ def test_no_amp_single_gpu(tmpdir): use_amp=True ) - with pytest.raises((MisconfigurationException, ModuleNotFoundError)): - tutils.run_model_test(trainer_options, model) + trainer = Trainer(**trainer_options) + result = trainer.fit(model) + + assert result == 1 def test_amp_gpu_ddp(tmpdir): @@ -78,6 +81,7 @@ def test_amp_gpu_ddp(tmpdir): tutils.run_model_test(trainer_options, model) +@pytest.mark.spawn def test_amp_gpu_ddp_slurm_managed(tmpdir): """Make sure DDP + AMP work.""" if not tutils.can_run_gpu_test(): @@ -124,26 +128,6 @@ def test_amp_gpu_ddp_slurm_managed(tmpdir): assert trainer.resolve_root_node_address('abc[23-24]') == 'abc23' assert trainer.resolve_root_node_address('abc[23-24, 45-40, 40]') == 'abc23' - # test model loading with a map_location - pretrained_model = tutils.load_model(logger.experiment, trainer.checkpoint_callback.filepath) - - # test model preds - for dataloader in trainer.get_test_dataloaders(): - tutils.run_prediction(dataloader, pretrained_model) - - if trainer.use_ddp: - # on hpc this would work fine... but need to hack it for the purpose of the test - trainer.model = pretrained_model - trainer.optimizers, trainer.lr_schedulers = pretrained_model.configure_optimizers() - - # test HPC loading / saving - trainer.hpc_save(tmpdir, logger) - trainer.hpc_load(tmpdir, on_gpu=True) - - # test freeze on gpu - model.freeze() - model.unfreeze() - def test_cpu_model_with_amp(tmpdir): """Make sure model trains on CPU.""" @@ -165,6 +149,7 @@ def test_cpu_model_with_amp(tmpdir): tutils.run_model_test(trainer_options, model, on_gpu=False) +@pytest.mark.spawn def test_amp_gpu_dp(tmpdir): """Make sure DP + AMP work.""" tutils.reset_seed() @@ -180,8 +165,11 @@ def test_amp_gpu_dp(tmpdir): distributed_backend='dp', use_amp=True ) - with pytest.raises(MisconfigurationException): - tutils.run_model_test(trainer_options, model, hparams) + + trainer = Trainer(**trainer_options) + result = trainer.fit(model) + + assert result == 1 if __name__ == '__main__': diff --git a/tests/test_cpu_models.py b/tests/test_cpu_models.py index f2cae173..1e71b8cd 100644 --- a/tests/test_cpu_models.py +++ b/tests/test_cpu_models.py @@ -97,7 +97,7 @@ def test_running_test_after_fitting(tmpdir): trainer_options = dict( default_save_path=tmpdir, show_progress_bar=False, - max_epochs=1, + max_epochs=4, train_percent_check=0.4, val_percent_check=0.2, test_percent_check=0.2, diff --git a/tests/test_logging.py b/tests/test_logging.py index 24d0c918..f5215824 100644 --- a/tests/test_logging.py +++ b/tests/test_logging.py @@ -9,7 +9,7 @@ from pytorch_lightning import Trainer from pytorch_lightning.logging import ( LightningLoggerBase, rank_zero_only, - TensorboardLogger, + TensorBoardLogger, ) from pytorch_lightning.testing import LightningTestModel @@ -169,8 +169,8 @@ def test_comet_pickle(tmpdir, monkeypatch): except ModuleNotFoundError: return - hparams = tutils.get_hparams() - model = LightningTestModel(hparams) + # hparams = tutils.get_hparams() + # model = LightningTestModel(hparams) comet_dir = os.path.join(tmpdir, "cometruns") @@ -199,9 +199,9 @@ def test_tensorboard_logger(tmpdir): hparams = tutils.get_hparams() model = LightningTestModel(hparams) - logger = TensorboardLogger(save_dir=tmpdir, name="tensorboard_logger_test") + logger = TensorBoardLogger(save_dir=tmpdir, name="tensorboard_logger_test") - trainer_options = dict(max_num_epochs=1, train_percent_check=0.01, logger=logger) + trainer_options = dict(max_epochs=1, train_percent_check=0.01, logger=logger) trainer = Trainer(**trainer_options) result = trainer.fit(model) @@ -213,14 +213,12 @@ def test_tensorboard_logger(tmpdir): def test_tensorboard_pickle(tmpdir): """Verify that pickling trainer with Tensorboard logger works.""" - hparams = tutils.get_hparams() - model = LightningTestModel(hparams) + # hparams = tutils.get_hparams() + # model = LightningTestModel(hparams) - comet_dir = os.path.join(tmpdir, "cometruns") + logger = TensorBoardLogger(save_dir=tmpdir, name="tensorboard_pickle_test") - logger = TensorboardLogger(save_dir=tmpdir, name="tensorboard_pickle_test") - - trainer_options = dict(max_num_epochs=1, logger=logger) + trainer_options = dict(max_epochs=1, logger=logger) trainer = Trainer(**trainer_options) pkl_bytes = pickle.dumps(trainer) @@ -235,7 +233,7 @@ def test_tensorboard_automatic_versioning(tmpdir): root_dir.mkdir("0") root_dir.mkdir("1") - logger = TensorboardLogger(save_dir=tmpdir, name="tb_versioning") + logger = TensorBoardLogger(save_dir=tmpdir, name="tb_versioning") assert logger.version == 2 @@ -248,14 +246,14 @@ def test_tensorboard_manual_versioning(tmpdir): root_dir.mkdir("1") root_dir.mkdir("2") - logger = TensorboardLogger(save_dir=tmpdir, name="tb_versioning", version=1) + logger = TensorBoardLogger(save_dir=tmpdir, name="tb_versioning", version=1) assert logger.version == 1 @pytest.mark.parametrize("step_idx", [10, None]) def test_tensorboard_log_metrics(tmpdir, step_idx): - logger = TensorboardLogger(tmpdir) + logger = TensorBoardLogger(tmpdir) metrics = { "float": 0.3, "int": 1, @@ -266,7 +264,7 @@ def test_tensorboard_log_metrics(tmpdir, step_idx): def test_tensorboard_log_hyperparams(tmpdir): - logger = TensorboardLogger(tmpdir) + logger = TensorBoardLogger(tmpdir) hparams = { "float": 0.3, "int": 1, @@ -311,7 +309,7 @@ def test_custom_logger(tmpdir): trainer_options = dict( max_epochs=1, - train_percent_check=0.01, + train_percent_check=0.05, logger=logger, default_save_path=tmpdir ) diff --git a/tests/test_restore_models.py b/tests/test_restore_models.py index b04bb07d..fe6f3adf 100644 --- a/tests/test_restore_models.py +++ b/tests/test_restore_models.py @@ -153,7 +153,7 @@ def test_running_test_pretrained_model_dp(tmpdir): trainer_options = dict( show_progress_bar=True, - max_epochs=1, + max_epochs=4, train_percent_check=0.4, val_percent_check=0.2, checkpoint_callback=checkpoint, @@ -269,12 +269,12 @@ def test_cpu_restore_training(tmpdir): logger = tutils.get_test_tube_logger(tmpdir, False, version=test_logger_version) trainer_options = dict( - max_epochs=2, + max_epochs=8, val_check_interval=0.50, val_percent_check=0.2, train_percent_check=0.2, logger=logger, - checkpoint_callback=ModelCheckpoint(tmpdir) + checkpoint_callback=ModelCheckpoint(tmpdir, save_top_k=-1) ) # fit model diff --git a/tests/test_trainer.py b/tests/test_trainer.py index 65d5e4be..3da7f279 100644 --- a/tests/test_trainer.py +++ b/tests/test_trainer.py @@ -393,7 +393,7 @@ def test_multiple_test_dataloader(tmpdir): default_save_path=tmpdir, max_epochs=1, val_percent_check=0.1, - train_percent_check=0.1, + train_percent_check=0.2, ) # fit model