Commit Graph
330 Commits
Author SHA1 Message Date
Alexander KreuzerandAlexander Kreuzer ee17c7c9c8 Fixed error message and test docstring (#1698)
training_dataloader -> train_dataloader

Co-authored-by: Alexander Kreuzer <alexander.kreuzer@sap.com>
2020-05-10 13:16:16 -04:00
4970927ec8 Feature: auto scale batch size (#1638)
* auto batch finder

* fix styling

* add description

* add different modes

* fix copy paste error

* better organised code

* fix styling

* add tests

* fix

* fix

* add some documentation

* added CHANGELOG.md

* some documentation

* update based on review

* Update trainer.py

* Update docs/source/training_tricks.rst

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

* Update tests/trainer/test_trainer_tricks.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Update tests/trainer/test_trainer_tricks.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

* use EvalModelTemplate

* param tests

* rename

* wrap params

* rename function

* rename

* rename param

* fix

* abs

* rename

* refactor code

* add docs

* try

* arg

* loop

* exept

* loop

* drop bool

* docs

* docs

* added check and test for passing dataloader to fit

* styling fix

* update based on review

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
Co-authored-by: William Falcon <waf2107@columbia.edu>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
2020-05-09 08:28:36 -04:00
Adrian WälchliandJirka 25bbd059df Also update progress_bar in training_epoch_end (#1724)
* update prog. bar metrics on train epoch end

* changelog

* wip test

* more thorough testing

* comments

* update docs

* move test

Co-authored-by: Jirka <jirka.borovec@seznam.cz>
2020-05-08 23:31:56 -04:00
Yuri Brovmanandybrovman 3a642601e8 added warning for None dataloader (#1745)
* added warning for None dataloader

* fixed variable style

* updated warning message

* remove unused import

Co-authored-by: ybrovman <ybrovman@ebay.com>
2020-05-07 09:26:41 -04:00
Shunta Komatsu f656882942 Fix typo (#1750) 2020-05-07 09:25:54 -04:00
Pavel Grunt b9364f96b1 lr_finder: Fix typo in docstring (#1746) 2020-05-06 12:39:22 -04:00
Peter Yu 851866333c Attach version_ to checkpoint path only if version is int (#1748) 2020-05-06 12:38:32 -04:00
Yuri Brovmanandybrovman 35bbe178bd fix _reset_eval_dataloader() for IterableDataset (#1560)
* removed if dl from _reset_eval_dataloader()

* changed to if dl != None to be more safe

* hints from pep8speaks

Co-authored-by: ybrovman <ybrovman@ebay.com>
2020-05-05 14:09:48 -04:00
Tian Wang d6a0375974 Fixing logic (#1734) 2020-05-05 14:07:26 -04:00
Jirka BorovecandWilliam Falcon 2a2f303ae9 Tests: refactor trainer dataloaders (#1690)
* refactor default model

* drop redundant seeds

* refactor dataloaders tests

* fix multiple

* fix conf

* flake8

* Apply suggestions from code review

Co-authored-by: William Falcon <waf2107@columbia.edu>

Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-05-05 12:31:15 -04:00
Travis Addair f90afa29b8 Fix disabling progress bar on non-zero ranks using Horovod backend (#1709)
* Fix Horovod backend to disable progress bar on all ranks except 0

* Add join barriers

* Added changelog

* Make protected and add verbosity

* Refactor to disable progress bar callback in train

* Removed vebose setting

* Add cache check for Horovod

* Test run again

* Updated comment

* Always skip cache for Horovod

* Only reinstall when necessary

* Added separate step

* Fixed spacing

* Skip Python 3.8
2020-05-04 13:02:57 -04:00
Nicki SkafteandNicki Skafte e865b046b1 Bugfix/lr finder (#1676)
* fix early stopping bug

* allow val dataloader

* update CHANGELOG.md

* fix early stopping bug

* allow val dataloader

* update CHANGELOG.md

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
2020-05-04 11:38:51 -04:00
Adrian Wälchli d28b145393 Update type hints for multiple dataloaders in .fit() and .test() (#1723)
* update typehints

* change log
2020-05-04 08:24:34 -04:00
Adrian WälchliandJ. Borovec e6b34ef90d [WIP] Reduction when batch size < num gpus (#1609)
* reduce if <= num_gpus

* add test with explanation

* chlog

* fix changelog

Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
2020-05-02 11:01:44 -04:00
Travis AddairandJirka 2950f66983 Fix Horovod distributed backend to set the root_gpu property (#1669)
* params

* drop acc

* Fix Horovod distributed backend to set the root_gpu

* Fixed test

* Fixed tests

* Fixed lint

* Set root_gpu during initialization

* chlog

Co-authored-by: Jirka <jirka.borovec@seznam.cz>
2020-05-01 14:13:35 -04:00
Nathan Breitsch 3eac6cfd4f Don't convert namedtuple to tuple (#1589)
* Don't convert namedtuple to tuple

* Test namedtuples sent to device correctly
2020-04-30 08:04:50 -04:00
William FalconandJ. Borovec d40425d257 added warning to crash (#1625)
* added warning to crash

* formatting

Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
2020-04-30 08:04:18 -04:00
Jacob ZhongandJirka Borovec f9c9e39ab8 Add log output for slurm (#1657)
* add log output for slurm

* change log levels

* formatting

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-04-30 07:58:03 -04:00
Peter Yu 8d564b5e38 call on_load_checkpoint() when resuming from checkpoint (#1666) 2020-04-30 07:57:24 -04:00
Dan Campbell 981758fd04 Only call load_spawn_weights if COLAB_GPU or KAGGLE_URL_BASE
environment variables are set
2020-04-27 19:46:29 -04:00
William Falcon 4c4d0e6bc6 Merge pull request #1635 from PyTorchLightning/pkl
Fixes CPU DDP breaking change and DDP change
2020-04-27 07:42:42 -04:00
William Falcon cebc74f7bc ddp pickle 2020-04-27 07:19:12 -04:00
William Falcon 2181ad1bc7 ddp pickle 2020-04-27 07:12:04 -04:00
William Falcon d2989f76e6 ddp pickle 2020-04-27 07:08:34 -04:00
William Falcon d5dff384eb ddp pickle 2020-04-27 07:07:03 -04:00
William Falcon 4aac5568a6 ddp pickle 2020-04-27 07:05:29 -04:00
William Falcon 6e86b59d21 ddp pickle 2020-04-27 06:51:42 -04:00
Justus Schock f91b131ba2 Remove warning 2020-04-27 11:41:03 +02:00
William Falcon 879d879985 fix hparams issue 2020-04-26 17:27:45 -04:00
4755ded863 Clean up Argparse interface with trainer (#1606)
* fixed distutil parsing

* fixed distutil parsing

* Apply suggestions from code review

* log

* fixed distutil parsing

* fixed distutil parsing

* fixed distutil parsing

* fixed distutil parsing

* doctest

* fixed hparams section

* fixed hparams section

* fixed hparams section

* formatting

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
2020-04-26 09:20:06 -04:00
William FalconandJ. Borovec b620d86c54 diable val and test shuffling (#1600)
* diable val and test shuffling

* diable val and test shuffling

* diable val and test shuffling

* diable val and test shuffling

* log

* condition

* shuffle

* refactor

Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
2020-04-25 16:45:20 -04:00
William Falcon 791ba91dec slurm job id (#1605) 2020-04-25 16:01:15 -04:00
58a467dd68 model checkpint on rank_zero_only & global rank state (#1408)
* try delete in async or DDP us0-ecase

* changelog

* add model chekpoint rank

* simple delete

* flake8

* use global rank

* chnagelog

* fix review

* fix import

* proposal

* proposal

* proposal

* improve proposal (fix problems with method call self)

* cleaning

Co-authored-by: Adrian Wälchli <adrian.waelchli@students.unibe.ch>
Co-authored-by: William Falcon <waf2107@columbia.edu>
2020-04-24 17:21:00 -04:00
William Falcon d0faf97893 fixed dataset stuff + docs (#1599)
* Fixed dataset docs and disabled auto-sampler for iterable dataset
2020-04-24 16:51:26 -04:00
Jirka Borovec 570b2c7aeb fix depreated call (#1596)
* fix parity

* update deprecated call
2020-04-24 14:45:43 -04:00
William Falcon cd15bfc3ce fixed new amp bugs (#1593) 2020-04-24 09:29:39 -04:00
William Falcon 890458fdbd Fixes automatic parser bug (#1585)
* fixes gpu parsing

* fixes gpu parsing
2020-04-23 21:00:41 -04:00
3e8f2d99a9 Progress bar callback (#1450)
* squash and rebase

sanity check hooks


sanity check callback hook finish


moved core progress bar functionality into callback


wip


remove duplicate merge


clean up


imports


docs


sanity check progress bar main


sanity


move callback calls


init progrss bar callback


configuration and docs


changelog


rate decorator


pass process_position


disable on rank > 0


position index


is_enabled


remove decorator


refactor init tqdm bars


callback method ordering 


cannot reset when disabled


sequence -> list


default values


fix has no attr _time() 


move on_val_end to proper place


fix the pickle issue


update warning


properties


check for None


remove old comment


switch order


pull out non-tqdm functionality into base class


documentation for the base class


docs


fix refresh rate issue in validation


restrict type hint of trainer arg


more docs


update trainer docs


rst docs


fix lines too long


fix test


add missing type hints


fix typo


move docstring to __init__ solves doctest failures


remove doctest :(( can't fix the pickle error


fix example


simplify by saving trainer reference


fix docs errors


move docstring


initial value


multiple val checks per epoch


simpler handling of inf dataset sizes


update inf docs


renamed training_tqdm_dict


rename get_tqdm_dict


rename occurences of tqdm 


update changelog


fix doctest


fix formatting errors


added callback tests


progress bar on off test


more tests for progress bar


weird test fix?


add ignored property


disable default progress bar in LR finder


change enable/disable behavior


trying doctest in CI again


undo doctest pickle error


undo doctest pickle error :((


remove progress_bar_callback Trainer arg and fix tests


restore progress bar after auto lr find


update docs


fix rebase


fix wrong negation

* fix fast dev run total

* more thorough testing

* remove old args

* fix merge

* fix merge

* separate tests

* type hint total batches

* reduce if

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* is_disabled

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* is_enabled

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* rename enabled/disabled

* move deprecated api

* remove duplicated test from merge

* fix rename is_disabled

* newline

* test also testprogress for fast dev run

Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-04-23 20:46:18 -04:00
Guy Davidson fe2b6666e0 Fixing a small issue in trainer logging (#1563)
* The epoch was being logged to metrics, which isn't read, rather than to current_metrics.

* Updated the tests to account for the epoch arriving at the logger.
2020-04-23 17:52:41 -04:00
Jirka Borovec 7989ca844c test deprecation warnings (#1470)
* check deprecation warnings

* extend warning test

* try

* unimport modules

* update
2020-04-23 17:34:47 -04:00
William Falcon 47629536e2 Amp2 (#1580)
* fixed new amp bugs

* fixed new amp bugs
2020-04-23 15:24:02 -04:00
William Falcon 68ca577919 why copy? (#1579) 2020-04-23 15:03:39 -04:00
29ebe92208 support for native amp (#1561)
* adding native amp suppport

* adding native amp suppport

* adding native amp suppport

* adding native amp suppport

* autocast

* autocast

* autocast

* autocast

* autocast

* autocast

* removed comments

* removed comments

* added state saving

* added state saving

* try install amp again

* added state saving

* drop Apex reinstall

Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
2020-04-23 14:47:08 -04:00
karlinjfandJosh Karlin 41b6cbb3ca Don't copy the batch when training on a single gpu (#1576)
* fix

* whitespace

Co-authored-by: Josh Karlin <karlinjf@gmail.com>
2020-04-23 14:28:20 -04:00
Nicki SkafteandNicki Skafte e977d1cde5 Default value for ModelCheckpoint filepath (#1548)
* allow determine of filepath at runtime

* typing

Co-authored-by: Nicki Skafte <nugginea@gmail.com>
2020-04-23 11:50:58 -04:00
Ferdinand Schlatt 545b38ec5f fix boolean argparse (#1571)
* fix boolean argparse #1570

* update change log
2020-04-23 11:44:18 -04:00
Lezwon Castelino 831842972f check for kaggle env variable (#1568)
* check for kaggle env variable

* added changelog
2020-04-23 07:12:54 -04:00
7024177f7d Added Horovod distributed backend (#1529)
* Initial commit of Horovod distributed backend implementation

* Update distrib_data_parallel.py

* Update distrib_data_parallel.py

* Update tests/models/test_horovod.py

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* Update tests/models/test_horovod.py

Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>

* Fixed tests

* Added six

* tests

* Install tox for GitHub CI

* Retry tests

* Catch all exceptions

* Skip cache

* Remove tox

* Restore pip cache

* Remove the cache

* Restore pip cache

* Remove AMP

Co-authored-by: William Falcon <waf2107@columbia.edu>
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: J. Borovec <jirka.borovec@seznam.cz>
2020-04-22 17:39:08 -04:00
Jirka Borovec c1c6e3b6c9 default test logger (#1478)
* default test logger

* fix tests

* spawn

* try

* simplify tests

* simplify tests

* formatting

* loggers

* loggers

* revert to TestTube

* default

* default

* wraps

* world size

* optim imports
2020-04-21 20:33:10 -04:00
Kevin Chen bafdeca42f Replace GPU device idx with current process index (#1541) 2020-04-21 14:29:15 -04:00