d0c9472cb3
Add SLURM check in ddp_train() and init_ddp_connection() ( #1387 )
...
* slurm check in ddp_train and init_ddp_connection
* Remove code example in init_ddp_connection
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* remove blank line
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* improve for test coverage
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* update changelog
* Default values and warnings for DDP env variables
* fix merge artifacts
* update localhost value
* change to NODE_RANK
Co-authored-by: Alexander Reshytko <areshytko@Alexanders-MacBook-Pro.local >
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
Co-authored-by: William Falcon <waf2107@columbia.edu >
2020-04-19 17:08:19 -04:00
Justus Schock
c71bd73acb
DDP sampler ( #1513 )
...
* Add explicit flag for ddp sampler replacement
* Add flag for sampler replacement in ddp
* Update data_loading.py
* Update CHANGELOG.md
* pep8 fixes
* pep8
2020-04-19 16:58:57 -04:00
William Falcon
ae2e14e3ed
fixed memory leak from opt return ( #1528 )
...
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
* fixed memory leak from opt return
2020-04-19 16:41:54 -04:00
Jonas-Jaeger
e02146943d
Removed redundant computations in clip_gradients that slowed down the gradient clipping. ( #1523 )
...
Fixes #1522
2020-04-18 23:07:15 -04:00
Krishna Penukonda and Adrian Wälchli
a22a8142ac
Allow Trainer's gpus arg type to be subclass of currently accepted types ( #1423 )
...
* Fixed Trainer `gpus` arg type issue
Fixes #1388
* Disallow boolean gpus parameter
Co-Authored-By: Adrian Wälchli <aedu.waelchli@gmail.com >
* Fixed missing paranthesis
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com >
2020-04-17 18:18:29 -04:00
e3001a0929
Add ddp_cpu backend for testing ddp without GPUs ( #1158 )
...
* Add tests for distributed backend config
* Refactor set_distributed_mode
* Use gloo backend on cpu
* Use 127.0.0.1 instead of 127.0.0.2
Not totally clear on why this is necessary, but it seemt to work
* Update LightningDDP so that it works with CPU
* Add ddp_cpu backend and num_processes Trainer arg
* PEP8
* Fix test skipping. Inequalities are hard :/
* Skip ddp_cpu test on Windows
* Make a few more cases fall back to ddp_cpu
* New function name
* Flake8
* Don't test distributed on MacOS with torch < 1.3
Support for distributed in MacOS was added in Torch 1.3.0
* Add ddp_cpu and num_processes to docs
* Parametrize trainer config tests
* Tweak warning
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* Remove redundant test
* Replace pass branches with comments
* Add missing warnings import
* save_path -> root_dir
* Use new rank_zero_warn
* Whitespace
* Apply suggestions from code review
* formatting
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-15 23:17:31 -04:00
William Falcon and J. Borovec
3431c62d41
Remove error when test dataloader used in test ( #1495 )
...
* remove error when test dataloader used in test
* remove error when test dataloader used in test
* remove error when test dataloader used in test
* remove error when test dataloader used in test
* remove error when test dataloader used in test
* remove error when test dataloader used in test
* fix lost model reference
* remove error when test dataloader used in test
* fix lost model reference
* moved optimizer types
* moved optimizer types
* moved optimizer types
* moved optimizer types
* moved optimizer types
* moved optimizer types
* moved optimizer types
* moved optimizer types
* added tests for warning
* fix lost model reference
* fix lost model reference
* added tests for warning
* added tests for warning
* refactoring
* refactoring
* fix imports
* refactoring
* fix imports
* refactoring
* fix tests
* fix mnist
* flake8
* review
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-15 22:16:40 -04:00
Jirka Borovec
b3fe17ddeb
fix flushing loggers ( #1459 )
...
* flushing loggers
* flushing loggers
* flushing loggers
* flushing loggers
* changelog
* typo
* fix trains
* optimize imports
* add logger test all
* add logger test pickle
* flake8
* fix benchmark
* hanging loggers
* try
* del
* all
* cleaning
2020-04-14 20:32:33 -04:00
William Falcon and Jirka Borovec
c96c6a6b33
attempting to remove some speed issues ( #1482 )
...
* removed some .items
* added speed tests
* added speed tests
* Update benchmarks/test_rnn_parity.py
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* Update benchmarks/test_trainer_parity.py
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* fix lost model reference
* added speed tests
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
2020-04-14 20:23:36 -04:00
Ethan Harris
8544b334e4
Replace automatic nan check with optional flag ( #1475 )
...
* Replace automatic nan check with optional flag
* Update CHANGELOG.md
2020-04-13 14:06:25 -04:00
3f09b32df3
Learning Rate finder ( #1347 )
...
* initial structure
* rebase
* incorporate suggestions
* update CHANGELOG.md
* initial docs
* fixes based on reviews
* added trainer arg
* update docs
* added saving/restore of model state
* initial tests
* fix styling
* added more tests
* fix docs, backward compatility and progressbar
* fix styling
* docs update
* updates based on review
* changed saving to standard functions
* consistent naming
* fix formatting
* improve docs, added support for nested fields, improve codecov
* update CHANGELOG.md
* Update lr_finder.rst
* Update pytorch_lightning/trainer/trainer.py
* Update trainer.py
* Update CHANGELOG.md
* Update path
* restoring
* test
* attribs
* docs
* doc typo
Co-authored-by: Nicki Skafte <nugginea@gmail.com >
Co-authored-by: William Falcon <waf2107@columbia.edu >
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-10 14:34:23 -04:00
Jirka Borovec
d05ac813dc
fix deprecated default_save_path ( #1449 )
2020-04-10 14:32:56 -04:00
William Falcon
b78c3d4da8
Fix weights path ( #1445 )
...
* renamed default path to actual root_dir
* added default weights path
* added default weights path
* added default weights path
2020-04-10 12:02:59 -04:00
Allard Hendriksen
7ac1580a31
Add automatic GPU choice to trainer ( #1426 )
...
* Add automatic GPU choice to trainer
This commit adds the `gpu_choice` parameter to Trainer. By default,
this parameter is set to 'manual' which causes no observable
difference in behavior.
When `gpu_choice` is set to "auto" and `gpus` is an int, then the
trainer will automatically allocate the first available GPU.
This is especially useful when GPUs are configured to be in "exclusive
mode", which means that only one process at a time can use them.
* Rename gpu_choice -> auto_select_gpus
2020-04-10 11:45:29 -04:00
Rohit Gupta
e79ae18cae
Add test_dataloaders to test method ( #1434 )
...
* Add test_dataloaders to test method
* Remove test_dataloaders from .fit()
* Fix code comment
* Fix tests
* Add test_dataloaders to test method (#1393 )
* Fix failing tests
* Update docs (#1393 )
2020-04-10 11:44:03 -04:00
Alexey Karnachev
4c34d16a34
Fixed configure optimizer from dict without "scheduler" key ( #1443 )
...
* `configure_optimizer` from dict with only "optimizer" key. bug fixed
* autopep8
* pep8speaks suggested fixes
* CHANGELOG.md upd
2020-04-10 11:43:06 -04:00
William Falcon and J. Borovec
1f685c2882
fix pretty print ( #1441 )
...
* grid sample
* grid sample
* grid sample
* grid sample
* grid sample
* changelog
* version
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-10 08:43:22 -04:00
Alex Sergeev
8dd9b80d7a
Fix gradient clipping ( #1438 )
...
* Fix gradient clipping
* Relax accuracy constraint
2020-04-09 21:08:28 -04:00
Jirka Borovec
b2707c9b2e
fix retruning returns ( #1431 )
...
* returns
* changelog
2020-04-09 15:01:08 -04:00
William Falcon
2dec93f588
workers warning not on windows ( #1433 )
2020-04-09 14:46:51 -04:00
Jirka Borovec
17f58d2e11
add rank warning ( #1428 )
...
* add rank warning
* changelog
* use rank_zero_warn
* user trainer_init
* replace warnings
* fix test
* flake8
* docs
* changelog
* bug lol
2020-04-09 14:05:46 -04:00
William Falcon
b4eb3884cf
workers warning not on windows ( #1430 )
2020-04-09 12:42:30 -04:00
William Falcon
21a1972921
fixed default sampler ( #1425 )
2020-04-09 08:52:15 -04:00
2ae2bd2b46
Print test results only if prog_bar_metrics is not empty ( #1411 )
...
* Print test results only if prog_bar_metrics is not empty
* Update evaluation_loop.py
Co-authored-by: vitor-guizilini <vitor.guizilini@tri.global >
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
2020-04-08 11:51:52 -04:00
7d0c2c7db8
Loader docs ( #1416 )
...
* added multiple loader docs
* added multiple loader docs
* added multiple loader docs
* added multiple loader docs
* added multiple loader docs
* Apply suggestions from code review
* added multiple loader docs
* added build docs script
* typo
* added build docs script
* added build docs script
* added build docs script
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-08 11:38:12 -04:00
ddbf7de6dc
Added accumulation of loggers' metrics for the same steps ( #1278 )
...
* `add_argparse_args` method fixed (argument types added)
* autopep8 fixes
* --gpus=0 removed from test (for ci tests)
* Update pytorch_lightning/trainer/trainer.py
Co-Authored-By: Joe Davison <joe@huggingface.co >
* test_with_accumulate_grad_batches added
* agg_and_log_metrics logic added to the base logger class
* small format fix
* agg metrics strategies removed (not to complicate stuff)
* agg metrics: handle zero step
* autopep8
* changelog upd
* flake fix
* metrics aggregators factored out, metrics_agg.py added + tests
* metrics agg default value added
* Update pytorch_lightning/loggers/metrics_agg.py
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* metrics aggregators factored out, metrics_agg.py added + tests
* metrics agg default value added
* Update pytorch_lightning/loggers/metrics_agg.py
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* remove .item which causes sync issues (#1254 )
* remove .item which causes sync issues
* fixed gradient acc sched
* fixed gradient acc sched
* test_metrics_agg.py removed (all tested in doctrings), agg metrics refactored
* test_metrics_agg.py removed (all tested in doctrings), agg metrics refactored
* autopep8
* loggers base.py types fixed
* test
* test
* metrics aggregation for loggers: each key now has a specific function (or default one)
* metrics aggregation for loggers: each key now has a specific function (or default one)
* docstrings upd
* manual typehints removed from docstrings
* batch_size decreased for test `test_with_accumulate_grad_batches`
* extend running accum
* refactor
* fix tests
* fix tests
* allowed_types generator scoped
* trainer.py distutils was imported twice, fixed
* TensorRunningAccum refactored
* TensorRunningAccum added to change log (Changed)
* change log pull link added
Co-authored-by: Joe Davison <joe@huggingface.co >
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
Co-authored-by: William Falcon <waf2107@columbia.edu >
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-08 08:35:47 -04:00
Asaf Manor
09668df726
Update optimizers.py ( #1383 )
2020-04-07 09:09:23 -04:00
Paweł Rzepiński
b8ff9bc1d2
Fix unimplemented type() on TPU ( #1396 )
...
* Fix unimplemented type() on TPU
* Add changelog entry
* Add quotation marks
2020-04-06 20:29:55 -04:00
areshytko and Alexander Reshytko
9754c5da55
load_spawn_weights only in proc rank 0 ( #1385 )
...
Co-authored-by: Alexander Reshytko <areshytko@Alexanders-MacBook-Pro.local >
2020-04-06 10:17:16 -04:00
4ed3027309
Set precision=16 when use_amp is passed as True ( #1145 )
...
* Set precision=16 when use_amp is passed as True
* Update CHANGELOG.md
* add use_amp to deprecated API
* Update trainer.py
* Update trainer.py
* move the use_amp attribute to deprecated API
* move use_amp deprecation back to Trainer's __init__
* drop unsed
* drop deprecated
* reorder imports
* typing
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
Co-authored-by: William Falcon <waf2107@columbia.edu >
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-06 08:13:24 -04:00
Jeremy Jordan
91c9b29d47
add trainer attribute to denote if interrupted ( #1368 )
...
* add trainer attribute to denote if interrupted
* bugfix and formatting
2020-04-05 11:12:41 -04:00
Ethan Harris and Jirka Borovec
b18accc64c
Add warning for few workers ( #1378 )
...
* Add warning for few workers
* Fix style issue
* Update CHANGELOG.md
* Update test
* formatting
* formatting
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
2020-04-05 11:07:16 -04:00
William Falcon and Shubham Agarwal
16f4cc9ff0
Shubhamagarwal92 master ( #1349 )
...
* SA: for #958 : set torch cuda device when finding root
* SA: for #958 : removing root gpu hack in trainer/evaluation_loop
* SA: setting torch cuda device
* comment line too long
* check if root gpu exists or available
* Incorporating suggestions on #1094
* since root gpu returns none instead of -1 for cpu
* undo changes
* fixed dp memory thing
Co-authored-by: Shubham Agarwal <shubhamagarwal92@gmail.com >
2020-04-03 17:56:19 -04:00
Justus Schock and J. Borovec
f6a86e8551
generalize reinstantiation of dataloader ( #1346 )
...
* generalize reinstantiation of dataloader
* fix condition
* add test
* update changelog
* fix changelog
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-03 17:55:08 -04:00
Jean-Baptiste SCHIRATTI
e570d2e1ca
Doc fixes ( #1362 )
...
* Doc fixes from #1357 (awaelchli's comments) + changelog.
* Fix indentation.
* Add blank line to fix doc build?
2020-04-03 15:02:20 -04:00
Tullie Murrell
38e89dd890
Fix fast_dev_run running validation twice ( #1365 )
2020-04-03 15:00:26 -04:00
Adrian Wälchli
ebd9fc9530
Fix for incorrect run on the validation set with overwritten validation_epoch_end and test_end ( #1353 )
...
* reorder if clauses
* fix wrong method overload in test
* fix formatting
* update change_log
* fix line too long
2020-04-03 09:25:32 -04:00
Jean-Baptiste SCHIRATTI and Jean-Baptiste SCHIRATTI
868b172f05
Make training_epoch_end behave like validation_epoch_end ( #1357 )
...
* Make training_epoch_end behave like validation_epoch_end + minor fixes in docstrings.
* Minor fixes (Borda's comments).
* Detach tensors in batch_output (to avoid possible memory leak) + doc fix.
Co-authored-by: Jean-Baptiste SCHIRATTI <jean-baptisteschiratti@MacBook-Pro-de-Jean-Baptiste.local >
2020-04-03 14:43:26 +02:00
William Falcon
2eca8a9ef2
quick patch __code__ ( #1352 )
...
* quick patch
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
* testing fix
2020-04-03 08:40:02 -04:00
Santiago Castro
1576ad9963
Fix docs typo ( #1355 )
...
* Fix typo
* Fix typo
2020-04-03 07:35:09 +02:00
f33b5a8d99
Simplify progress bar args ( #1108 )
...
* show progress bar dependent on refresh_rate
* test progress_bar_refresh control show bar
* remove show_progress_bar from other tests
* borda fixes
* flake8 fix
* changelog update prog bar refresh rate
* move show_progress_bar to deprecated 0.9 api
* rm show_progress_bar references, test deprecated
* Update pytorch_lightning/trainer/__init__.py
* fix test
* changelog
* minor CHANGELOG.md format
* Update pytorch_lightning/trainer/__init__.py
* Update pytorch_lightning/trainer/trainer.py
Co-authored-by: Gerard Bentley <gbkh2015@mymail.pomona.edu >
Co-authored-by: William Falcon <waf2107@columbia.edu >
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz >
2020-04-03 00:53:00 +02:00
2912239fe6
Add useful errors when model is not configured correctly ( #1199 )
...
* add check_model_configuration method
* trying to fix errors
* trying to fix tests
* added test_epoch_end to lightning template
* fix tests
* fix new test after rebase
* fix spelling
* added more checks
* updated formating
* added tests
* fixed CHANGELOG
* Apply suggestions from code review
* move test to new module
* change check on configure_optimizers
Co-authored-by: Nicki Skafte <nugginea@gmail.com >
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
2020-04-02 11:53:37 -04:00
Ethan Harris and Jirka Borovec
28242f02d1
Remove default optimizer, add None optimizer option ( #1279 )
...
* Add warning when using default optimizer
* Refactor optimizer tests to test_optimizers
* Remove default optimizer, add option to use no optimizer
* Update CHANGELOG.md
* Update pytorch_lightning/trainer/optimizers.py
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* Fix style
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
2020-04-02 11:48:53 -04:00
04935ea718
fixed extra dataloader bug ( #1196 )
...
* fixed extra dataloader bug
* Update pytorch_lightning/trainer/training_loop.py
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com >
* updated CHANGELOG
* Small non-repetition change
self.get_model() => model as it was already defined
* Update CHANGELOG.md
* changed argument name to reload_train_dataloader_every_epoch
* fixed doc underline too short
* reverted to `reload_dataloaders_every_epoch`
* fixed val and test reloading
* fixed val and test reloading
Co-authored-by: TevenLeScao <teven.lescao@gmail.com >
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
2020-04-02 11:41:56 +02:00
William Falcon
e48422df38
Sampler ( #1328 )
...
* sampler
* check for dataloader type
* check for dataloader type
* fixed sampler cases
2020-04-01 12:57:36 -04:00
William Falcon
7de51f78ac
Sampler ( #1318 )
...
* sampler
* sampler
* sampler
* check for dataloader type
* check for dataloader type
2020-03-31 18:22:45 -04:00
Asaf Manor and Jirka Borovec
aca8c7e6f3
Optimizer Frequencies logic, and new configure_optimizers ( #1269 )
...
* init_optimizers accepts Dict, Sequence[Dict]
and returns optimizer_frequencies.
optimizer_frequencies was added as a member of Trainer.
* Optimizer frequencies logic implemented in training_loop.
Description added to configure_optimizers in LightningModule
* optimizer frequencies tests added to test_gpu
* Fixed formatting for merging PR #1269
* Apply suggestions from code review
* Apply suggestions from code review
Co-Authored-By: Asaf Manor <32155911+asafmanor@users.noreply.github.com >
* Update trainer.py
* Moving get_optimizers_iterable() outside.
* Update note
* Apply suggestions from code review
* formatting
* formatting
* Update CHANGELOG.md
* formatting
* Update CHANGELOG.md
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com >
2020-03-31 16:41:24 +00:00
Jirka Borovec
ee68d5ba8e
reorder deprecated args ( #1230 )
2020-03-31 09:01:47 -04:00
Jirka Borovec
6ddb03922a
Profiler summary ( #1259 )
...
* refactor and add types
* add Prorfiler summary
* fix imports
* Revert "refactor and add types"
This reverts commit b4c552fa
* changelog
* revert rename
* fix test
* mute verbose
2020-03-31 08:57:48 -04:00
Adrian Wälchli and mergify[bot]
4dcb9d3e30
fixed type hint for weights_summary arg ( #1313 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-03-31 13:42:29 +02:00
