26 Commits
Author SHA1 Message Date
Fabio Natanael KeplerandJirka Borovec 8b9b923ca8 Keep track of the best model's path saved by ModelCheckpoint (#1799)
* Add an additional attribute to ModelCheckpoint to keep track of the best model's path

Currently, only the best metric value is directly tracked. This new attribute will help in uses cases where the trained model needs to be used or tracked right after training.

* Add small description and usage example to docs

* Fix PEP8 issues

* Fix doctest example

* Fix expected output in doctest

* Apply suggestions from code review

* Show example as code block instead of doctest

* Apply suggestions from code review

* Update CHANGELOG.md

* Rename `ModelCheckpoint.best` to `ModelCheckpoint.best_model_score`

Also rename `ModelCheckpoint.best_model` (added in this PR) to `ModelCheckpoint.best_model_path`, for consistency, and `kth_best_model` to `kth_best_model_path`.

* Update pytorch_lightning/trainer/training_io.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Add warning when loading checkpoint from an old version

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-05-31 08:47:13 -04:00
Lucas VazquezandJeremy Jordan 112dd5c4f6 Adds the option of saving the last model on checkpoint (#1908)
* saves model every epoch

* implement test for save_last

* Update CHANGELOG.md

* Update CHANGELOG.md

* changes test description

Co-authored-by: Jeremy Jordan <13970565+jeremyjordan@users.noreply.github.com>

Co-authored-by: Jeremy Jordan <13970565+jeremyjordan@users.noreply.github.com>
2020-05-25 07:47:44 -04:00
Fabio Natanael KeplerandJirka Borovec 8c4c7b105e Fix save_weights_only flag in ModelCheckpoint (#1780)
* Add flag to `dump_checkpoint` for only including weights

`ModelCheckpoint` then passes `self.save_weights_only` to the save function.

* Fix tests and add changelog entry

* Add check and descriptive message when training state is restored from a weights only checkpoint

Also add a test for making sure `ModelCheckpoint.save_weights_only` works as expected.

* Fix weights-only test to properly match expected exception

* Apply suggestions from code review

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-05-17 09:24:17 -04:00
William Falcon 53d9316a56 fixes ddp bugs (#1819)
* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug
2020-05-13 19:17:04 -04:00
Peter Yuandmergify[bot] 42d5cfc3b0 None check for filepath in ModelCheckpoint (#1654)
Check if the optional filepath is None before checking if it exists

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-04-29 16:36:28 +00:00
Wojciech Jabłoński 735520b03e Fix ModelCheckpoint not being fixable. 2020-04-27 11:02:33 +02:00
58a467dd68 model checkpint on rank_zero_only & global rank state (#1408)
* try delete in async or DDP us0-ecase

* changelog

* add model chekpoint rank

* simple delete

* flake8

* use global rank

* chnagelog

* fix review

* fix import

* proposal

* proposal

* proposal

* improve proposal (fix problems with method call self)

* cleaning

Co-authored-by: Adrian Wälchli <adrian.waelchli@students.unibe.ch>
Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-04-24 17:21:00 -04:00
Nicki SkafteandNicki Skafte e977d1cde5 Default value for ModelCheckpoint filepath (#1548)
* allow determine of filepath at runtime

* typing

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
2020-04-23 11:50:58 -04:00
William Falcon ae2e14e3ed fixed memory leak from opt return (#1528)
* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return

* fixed memory leak from opt return
2020-04-19 16:41:54 -04:00
9b31272cf0 feat: save checkpoint before deleting old ones (#1453)
* feat: save checkpoint before deleting old ones

* fix: make sure that the new model is not deleted

* changelog

Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-04-16 16:40:51 +00:00
Jirka Borovec 17f58d2e11 add rank warning (#1428)
* add rank warning

* changelog

* use rank_zero_warn

* user trainer_init

* replace warnings

* fix test

* flake8

* docs

* changelog

* bug lol
2020-04-09 14:05:46 -04:00
f1e11d8b38 model_checkpoint to save all models (#1359)
* model_checkpoint to save all models

* changelog

* rise if

Co-authored-by: jamesjjcondon <jamesjjcondon@gmail.com>
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
2020-04-05 15:56:26 +02:00
Adrian WälchliandJirka Borovec 1f2da71069 Improved docs for callbacks (#1370)
* improved docs for callbacks

* class references

* make doctest pass

* doctests

* fix lines too long

* fix line too long

* fix permission error in doctest

* Apply suggestions from code review

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* fix doctest

* fix default

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-04-05 09:38:52 +00:00
09167efdb5 Checkpointing interval (#1272)
* formatting

* formatting

* fix interval

* fix train loop

* fix test

* parametrize test

* Apply suggestions from code review

Co-Authored-By: Adrian Wälchli <adrian.waelchli@students.unibe.ch>

* fix calling

* flake8

* add types

Co-authored-by: Adrian Wälchli <adrian.waelchli@students.unibe.ch>
Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-03-30 18:37:02 -04:00
Jacob Zhong 1a73fa0b03 change default logger to dedicated one (#1064)
Fix test


Fix format

Update pytorch_lightning/__init__.py
Separate imports
2020-03-17 18:44:00 -04:00
Jirka Borovec 514d182b7f cleaning imports (#1032) 2020-03-12 12:41:37 -04:00
Xing Zhao LEEandxingzhaolee b4d4e489bf Run on_validation_end only on main process in DDP (#1125)
Co-authored-by: xingzhaolee <xingzhaolee@users.noreply.github.com>
2020-03-12 15:50:00 +01:00
Jeremy Jordan c5b81d04c7 add docs title (#1063) 2020-03-05 22:25:05 -05:00
William FalconandJirka Borovec bcb45d906d proper checkpoint implementation (#1043)
* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* enabled early stopping/checkpooiunt even  without val step

* name formatting

* version

* testing

* add test

* fix test

* Update model_checkpoint.py

* doctests

* pylint

* tests

* debug

* debug

* enabled early stopping/checkpooiunt even  without val step

* fix MNIST download (#1044)

* fix MNIST download

* simple

* name formatting

* version

* testing

* add test

* fix test

* doctests

* tests

* debug

* debug

* rebased 1041

* rebased 1041

* tests

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

* rebased 1041

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-03-04 23:02:19 -05:00
Jirka BorovecandWilliam Falcon e586ed4767 hparams as dict [blocked by 1041] (#1029)
* hparams as dict

* hparams as dict

* fixing

* fixing

* fixing

* fixing

* typing

* typing

* chnagelog

* update set hparams

* use setter

* simplify

* chnagelog

* imports

* pylint

* typing

* Update training_io.py

* Update training_io.py

* Update lightning.py

* Update test_trainer.py

* Update __init__.py

* Update base.py

* Update utils.py

* Update test_trainer.py

* Update training_io.py

* Update test_trainer.py

* Update test_trainer.py

* Update test_trainer.py

* Update test_trainer.py

* Update callback_config.py

* Update callback_config.py

* Update test_trainer.py

Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-03-04 09:33:39 -05:00
William Falcon 6a39573267 lint 2020-03-03 16:57:49 -05:00
Jirka BorovecandWilliam Falcon 64de57b09e update checkpoint docs (#1016)
* update checkpoint docs

* fix tests

* fix tests

* formatting

* typing

* filename

* fix tests

* fixing tests

* fixing tests

* fixing tests

* unique name

* fixing

* fixing

* Update model_checkpoint.py

Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-03-03 15:16:57 -05:00
William Falcon 1d11f61c36 Docs2 (#1028)
* added community examples

* added community examples
2020-03-03 11:39:43 -05:00
Jeremy Jordan 705e576417 consolidate callbacks and hooks (#950)
* consolidate callbacks and hooks

* ensure callbacks recieve proper arg types

* remove model from init callback events

* clean up early stopping event

* update changelog

* remove on_fit_start and on_fit_end

* fix args for on_init_start and on_init_end

* handle case where early stopping is not used

* show all callback methods

* wrap checkpoint callback logic into proper class

* fix check for main process in checkpoint callback

* move callbacks test to separate file

* refactor arg checks

* get model and call hook on same line

* define trainer_options dict in one call

* add more asserts to callback test
2020-03-02 23:51:32 -05:00
Hadrien Mary be244560b2 Callbacks [wip] (#889)
* Add callback system + associated test

* Add trainer and pl_module args to callback methods

* typing

* typo in docstring

* Switch to on_.*_start()

* fix on_test_start

* fix the mess after rebasing
2020-02-25 23:17:27 -05:00
Hadrien MaryandJirka Borovec 89d5772f55 Split callbacks (#849)
* add .vscode in .gitignore

* Split callbacks in individual files + add a  property to Callback for easy trainer instance access

* formatting

* Add a conda env file for quick and easy env setup to develop on PL

* Adress comments

* add fix to kth_best_model

* add some typing to callbacks

* fix typo

* add autopep8 config to pyproject.toml

* format again

* format

* fix toml

* fix toml again

* consistent max line length in all config files

* remove conda env file

* Update pytorch_lightning/callbacks/early_stopping.py

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* Update pytorch_lightning/callbacks/model_checkpoint.py

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* docstring

* Update pytorch_lightning/callbacks/model_checkpoint.py

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* Update pytorch_lightning/callbacks/model_checkpoint.py

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* fix logic error

* format

* simplify if/else

* format

* fix linting issue in changelog

* edit changelog about new callback mechanism

* fix remaining formating issue on CHANGELOG

* remove lambda function because it's compatible with pickle (used during ddp)

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-02-22 21:45:34 -05:00