Commit Graph
664 Commits
Author SHA1 Message Date
Jirka Borovec f18aee30a5 Minor imports cleaning (#402)
* code cleaning

* drop unused imports

* optimize imports
2019-10-22 11:32:40 +03:00
William Falcon 792ad00ff9 Fixed val interval (#405)
* added fixed frequency val batch check

* added fixed frequency val batch check

* Finished IterableDataset support

* flake8

* flake8

* flake8
2019-10-22 05:10:00 +03:00
Cristobal Eyzaguirre ab6794406e Logger consistency (#397)
* added comet logger

* bug fix in cases where comet was not imported before torch

* fixed mlflow logger to be consistent with docs, updated cometLogger and cometLoggers docs + flake 8 compliance
2019-10-22 04:51:17 +03:00
William Falcon 1424157731 Refactor (#407)
* moved dp, ddp outside of trainer

* added main mixins

* finished major mixin refactor

* flake8

* finished major mixin refactor

* finished major mixin refactor

* finished major mixin refactor

* finished major mixin refactor

* finished major mixin refactor

* finished major mixin refactor

* finished major mixin refactor
2019-10-22 04:16:51 +03:00
tamyiuchau 4103a5ca73 Provide backward compatibility for #124 (#400)
* Provide backward compatibility for e681253

* typo fix
2019-10-21 08:16:55 +02:00
William Falcon 6111edaf82 Test fx (#390)
* changes to test fx

* changes to test fx

* changes to test fx

* changes to test fx

* changes to test fx

* changes to test fx

* changes to test fx

* changes to test fx

* changes to test fx

* changes to test fx
2019-10-19 00:39:30 +02:00
William Falcon 699bd2cb50 removed mlflow and custom logger tests (#389)
* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests

* changes to seed for tests
2019-10-18 23:03:28 +02:00
William Falcon 416187fd56 Fixes #380 2019-10-18 15:33:00 +02:00
William Falcon 3dfcef6994 Loss keys (#387)
* any key in logs or progress bar is a candidate for callback metric

* any key in logs or progress bar is a candidate for callback metric
2019-10-18 15:28:13 +02:00
Hiroyuki Vincent Yamazaki 0fac2d64cf Fix off-by-one epoch length (#377) 2019-10-18 10:18:05 +02:00
William Falcon 82223275ba Revert "add package info (#359)" (#384)
This reverts commit a7f26a67ac.
2019-10-18 10:05:03 +02:00
Jirka Borovec a7f26a67ac add package info (#359)
* add package info #358

* Update __init__.py

* Update __init__.py
2019-10-18 10:01:32 +02:00
William Falcon e5050700ce docs 2019-10-18 00:17:27 +02:00
William Falcon 2044126821 fixing tests (#372)
* fixing tests

* fixing tests

* fixing tests

* fixing tests

* fixing tests

* fixing tests

* fixing tests

* fixed tests

* fixed tests

* fixed tests

* fixed tests

* fixed tests

* fixed tests

* fixed tests

* fixed tests

* fixed tests
2019-10-16 07:28:47 -04:00
William Falcon e2cabb03ba fix val logging (#362)
* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* no warnings always

* no warnings always

* no warnings always

* no warnings always
2019-10-15 12:44:20 -04:00
Nic Eggert 19c2b8fc9e Allow disabling default logger, checkpoint_callback, and early_stop_callback (#360)
* Allow disabling logger, early stopping, and checkpoints

* Typo

* Get tests passing

* Update trainer.py
2019-10-12 06:00:24 -04:00
Yasser Souri 792ba59b78 Pad experiment version with zero for easier listing (#355) 2019-10-10 19:39:26 -04:00
William Falcon 426bb19846 Update trainer.py 2019-10-10 18:17:26 -04:00
William Falcon 46322b906b fixed ckpt tests (#352)
* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests

* fixed ckpt tests
2019-10-10 15:16:19 -04:00
William Falcon 96c2a2de50 fixes Flake8 2019-10-09 17:49:29 -04:00
festeh 0eab1e42b2 add tags argument to MLFlowLogger (#349) 2019-10-09 17:47:17 -04:00
William Falcon 453568179b Logger default (#351)
* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder

* ckpt callback in pretrain routine so exp already has version

* ckpt callback in pretrain routine so exp already has version

* ckpt callback in pretrain routine so exp already has version
2019-10-09 17:46:27 -04:00
William Falcon d95e693598 Logger default (#350)
* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder

* weights go into default logger folder
2019-10-09 16:25:04 -04:00
William Falcon 6e0a562ecb fixed callback metrics ddp bug 2019-10-09 12:53:33 -04:00
William Falcon 5f1f3f6acc removed pdb 2019-10-09 10:45:06 -04:00
William Falcon ec10119e97 Fixed tests (#340)
* removed hparam calls

* removed hparam calls

* removed hparam calls

* removed hparam calls

* removed hparam calls

* Update test_models.py
2019-10-09 10:37:10 -04:00
William Falcon 608a90a490 fixes non python type callback metrics and fast_dev_run (#345)
* fixes non python type callback metrics

* fixed fast dev run

* fixed fast dev run

* fixed fast dev run

* fixed fast dev run

* fixed fast dev run

* fixed fast dev run

* fixed fast dev run
2019-10-09 10:23:08 -04:00
Nic Eggert 8088052825 Finalize logger (#337)
* Ensure logger.finalize is called

* Call logger.finalize

* Update mlflow_logger.py

* Update test_logging.py

* Update trainer.py
2019-10-08 17:33:33 -04:00
William Falcon 49e04de5ac Ports (#338)
* remove os.exit from early stopping

* remove os.exit from early stopping

* fixed weight summary

* fixed weight summary

* fixed weight summary

* fixed weight summary

* fixed weight summary

* fixed weight summary

* fixed weight summary
2019-10-08 17:11:47 -04:00
William Falcon dcaba55251 Early stopping (#332)
* callbacks use all other keys in return dict

* callbacks use all other keys in return dict

* callbacks use all other keys in return dict

* callbacks use all other keys in return dict

* remove os.exit from early stopping
2019-10-08 16:21:00 -04:00
Adrian Wälchli 6e3e740a7f Param printing (#336)
* print thousands as K, M, B, T, ...

* add option to print top-level modules only

* added doc string and added spacing

* do not print summary if neither "full" nor "top"

* updated docs showing summary print options

* fix line length for travis
2019-10-08 15:30:06 -04:00
William Falcon ff2a21a08a default to O1 (#334) 2019-10-08 09:09:57 -04:00
Jon Tamir 1cf2e228ba fix CONTRIBUTING link and silence checkpoint callback message (#325) 2019-10-08 07:40:14 -04:00
William Falcon ac6d0154c2 Fixes lack of logging in logger (#319)
* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* changed rank 0

* models wait to restore weights

* models wait to restore weights
2019-10-06 17:57:23 -04:00
William Falcon 491100abdd Docs (#315)
* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up docs

* cleaned up test_tube logger

* cleaned up test_tube logger

* cleaned up test_tube logger
2019-10-05 23:52:32 -04:00
William Falcon ef98931d18 flake8 2019-10-05 16:56:24 -04:00
William Falcon 07c5d22ae3 cleaning up demos (#313)
* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos

* cleaning up demos
2019-10-05 16:39:05 -04:00
William Falcon cdfcb01073 Fixes #234 (#311)
* Fixes #234

* default logger version is now slurm job id

* default logger version is now slurm job id
2019-10-05 14:45:37 -04:00
William Falcon 6cc3f1757f decouple returns from each step (#307)
* decoupled training metrics from logging metrics

* decoupled validation metrics from log metrics

* updated docs

* updated docs

* updated docs

* Fixed test

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master

* merged master
2019-10-05 13:35:20 -04:00
William Falcon 8f5a06bfb8 Gpu mem (#308)
* Fixes #289

* Fixes #289

* added lbfgs support

* Fixes #280 (#309)

* added test seeds (#306)

* added test seeds

* added test seeds

* updated docs

* added lbfgs support (#310)

* added lbfgs support

* added lbfgs support

* added lbfgs support

* Fixes #280 (#309)

* added test seeds (#306)

* added test seeds

* added test seeds

* updated docs

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* Fixes #289

* Fixes #289

* merged master

* merged master
2019-10-05 11:29:34 -04:00
William Falcon 75fd89106f added lbfgs support (#310)
* added lbfgs support

* added lbfgs support

* added lbfgs support

* Fixes #280 (#309)

* added test seeds (#306)

* added test seeds

* added test seeds

* updated docs

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support

* added lbfgs support
2019-10-05 11:10:21 -04:00
William Falcon 2ac9f1aea7 Fixes #280 (#309) 2019-10-05 10:55:50 -04:00
William Falcon 967957e55c added lbfgs support 2019-10-05 10:47:18 -04:00
William Falcon bf09060fef Fixes #292 (#303)
* early stopping callback is not default

* added a default logger

* added default checkpoint callback

* added default checkpoint/loggers

* added default checkpoint/loggers

* updated docs

* cleaned demos

* cleaned demos

* cleaned demos

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers

* clean up docs around loggers
2019-10-04 19:48:57 -04:00
William Falcon a578de511d clean up docs around loggers (#304) 2019-10-04 18:53:38 -04:00
William Falcon a60a24d11b disable auto gpu loading when restoring weights to avoid OOM (#242)
* Update root_module.py

* Update root_module.py

* Update root_module.py

* tests fix

* tests fix
2019-10-04 16:18:43 -04:00
William Falcon 73a7cf3c99 Mem crash (#299)
* fixes memory crash

* fixes memory crash
2019-10-04 15:53:44 -04:00
Hendrik Schröter 36f0b5bbd0 Use getter instead of python property for the dataloaders (#275)
* Use getter instead of python property for the dataloaders

* Fix lint

* Update trainer.py
2019-10-04 15:35:02 -04:00
William Falcon 32e74b8f36 Ddp2 (#261)
* adds ddp2 option where on each node a single  process  uses all gpus

* added ddp2  test

* added ddp2 docs

* Update Distributed training.md

* delete ref to old update_training_log_metrics

* delete ref to old update_training_log_metrics

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* banana pancakes

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* cheesecake
2019-10-04 15:07:54 -04:00
Hendrik Schröter 42764d18c7 Better error message if no loss was returned from model.training_step() (#294) 2019-10-04 07:15:19 -04:00