kvhooreb
41236c7bbb
WIP: Moved grad_norm tracking code to __run_tng_batch ( #278 )
...
* Moved grad_norm tracking code to __run_tng_batch + added norms to tqdm_metrics
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
2019-10-02 11:11:08 -04:00
Nic Eggert
614cb3c03b
Initialize loggers only once ( #270 )
...
* Create underlying loggers lazily
This avoids creating duplicate experiments or run in multi-node DDP.
* Save hyperparameters automatically
* Update docs for snapshotting hyperparams
* Fix test tube
* Fix test tube pickling
2019-10-02 11:10:40 -04:00
Anton Bakhtin
222d7d2d5d
Hacky fix for mlflow logger ( #277 )
...
* Hacky fix for mlflow logger
It dies when "created_at" is logged
* Log warning
2019-10-01 21:32:52 -04:00
Hendrik Schröter
8a2472269a
Make test_tube optional ( #274 )
2019-10-01 05:19:47 -04:00
Nic Eggert
480eed5cb6
Enable any ML experiment tracking framework ( #223 )
...
* Implement generic loggers for experiment tracking
* Add tests for loggers
* Get model tests passing
* Test and fix logger pickling
* Expand pickle test and fix bug
* Missed exp -> logger conversion
* Remove commented code
* Add docstrings
* Update logging docs
* Add mlflow to test requirements
* Make linter happy
* Fix mlflow timestamp
* Update Logging.md
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update properties.md
* Fix tests
* Line length
2019-09-27 12:05:29 -04:00
William Falcon
481aa24974
always calls the lr scheduler with epoch nb. Fixes #98 ( #252 )
...
* always calls the lr scheduler with epoch nb
* added docs for cluster grid search
* added docs for cluster grid search
* undo test changes
* undo test changes
2019-09-26 16:36:41 -04:00
William Falcon
25d2f93256
enables samplers which don't need set epoch (or when ppl don't need a sampler) ( #254 )
...
* enables samplers which dont need set epoch
* added docs for single gpu ddp
* added docs for single gpu ddp
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
2019-09-26 14:39:04 -04:00
William Falcon
8b2a2aeda3
Dim 0 warning ( #256 )
...
* added ignore warnings module
* added ignore warnings module
* Fixes #249
* Update ignored_warnings.py
2019-09-26 13:20:54 -04:00
Alok Singh
b0a0a47a0b
Rename variables ( #124 )
...
- data_batch → batch
- batch_i → batch_idx
- dataloader_i → dataloader_idx
- tng → training
- training_dataloader → train_dataloader
- add_log_row_interval → row_log_interval
- gradient_clip → gradient_clip_val
- prog → progress
- tqdm_dic → tqdm_dict
2019-09-25 19:05:06 -04:00
William Falcon
87708157bc
Update trainer.py ( #233 )
2019-09-19 08:23:48 -04:00
William Falcon
55e7322747
Metrics load ( #228 )
...
* load from metrics defaults to CPU
* load from metrics defaults to CPU
* load from metrics defaults to CPU
2019-09-16 10:47:19 -04:00
Ananya Harsh Jha
c0f3b6b035
added set_epoch for distributed sampler, fix for #224 ( #225 )
2019-09-16 10:21:00 -04:00
William Falcon
9576dd28b2
added load on CPU first ( #221 )
...
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added print logs
* added print logs
* changed close order
* changed close order
2019-09-11 07:52:36 -04:00
William Falcon
30b25c8146
Sai prasanna master ( #219 )
...
* Fix incorrect warning for DistributedSampler.
Check whether `dataloader.sampler` is an instance of DistributedSampler instead of checking the `dataloader`.
* Update trainer.py
* merged
2019-09-09 11:36:24 -04:00
William Falcon
506d5da68b
enable single gpu per node ( #218 )
...
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
2019-09-09 07:37:20 -04:00
William Falcon
10d190e045
Simplified gpu api. No NVIDIA flag managing by lightning for cluster ( #213 )
...
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added simple cluster template
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
2019-09-08 15:36:58 -04:00
Alok Singh
81df2259ef
Make print_nan_grads print grad ( #208 )
...
This seems more useful for debugging.
2019-09-07 01:08:09 -04:00
William Falcon
0c7fbc7178
Weights path ( #211 )
...
* added docs. removed options. added weights_save option
* removed old restore
* cleaned up save path
* cleaned up save path
* flake8
2019-09-06 17:01:03 -04:00
William Falcon
7099f8dbfb
split trainer mixins ( #209 )
...
* split trainer mixins
* Update multi_node_cluster_template.py
* Update single_cpu_template.py
* Update single_gpu_node_16bit_template.py
* Update single_gpu_node_ddp_template.py
* Update single_gpu_node_dp_template.py
* Update trainer_cpu_template.py
* Update trainer_io.py
* split trainer mixins
* Update multi_node_cluster_template.py
* deconflicted
* deconflicted
* deconflicted
2019-09-06 14:11:07 -04:00
William Falcon
60633eaa32
Moves hpc auto-resubmit to trainer from test-tube ( #207 )
...
* added slurm signal handler
* added restore weight functions
* set slurm signal handling inside process
* added resubmit docs
* added resubmit docs
* fixed missing param
* Update trainer.py
* fixed missing param
* fixed missing param
* debugging tests
* debugging tests
* debugging tests
* debugging tests
* debugging tests
* debugging tests
* debugging tests
2019-09-06 11:54:51 -04:00
Nic Eggert
1733dba735
Pass outputs from all dataloaders to test_end and validation_end ( #203 )
...
* Pass outputs from all dataloaders to test_end and validation_end
* Update tests
* Update docs
* Update trainer.py
* Update test_models.py
2019-09-06 07:37:25 -04:00
William Falcon
7e0ac3149c
refactored init ( #206 )
2019-09-06 00:29:38 -04:00
Anton Konstantinov
34b824a9d3
Implement correct transfer to GPU for batches ( #200 )
2019-09-05 07:13:06 -04:00
Thomas J Fan
62252cee58
STY Minor flake8 fix ( #197 )
2019-09-04 17:46:56 -04:00
Max Horn
dac41030d4
Allow to deactivate GPU memory logging in Trainer ( #190 )
...
* Allow to deactivate GPU memory logging in Trainer
Adds the flag `log_gpu_memory` to Trainer to deactivate logging of GPU
memory utilization. On some servers logging the GPU memory usage can
significantly slow down training.
* Update Logging.md
* Update trainer.py
2019-09-04 10:43:46 -04:00
Nic Eggert
64688e1e15
Refactor test modules ( #180 )
...
* Expectopatronum implement #89 (#182 )
* rename validate -> evaluate; implement test logic; allow multiple test_loaders
* add test_step and test_end to LightningModule
* add in_test_mode to pretraining to implement case 2 (test pretrained model)
* fix code style issues
* LightningTestModel: add optional second test set, implement test_step and test_end
* implemented test for multiple test_dataloaders; fixed typo
* add two test cases for #89
* add documentation for test_step, test_end; fix computation of loss in validation_step example
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Added proper dp ddp routing calls for test mode
* Update trainer.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update override_data_parallel.py
* Update test_models.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update test_models.py
* Update test_models.py
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* Update trainer.py
* Update override_data_parallel.py
* Update debug.py
* Update lm_test_module.py
* Update test_models.py
* release v0.4.8
* Update README.md
* add training loop docs
* testing loop docs
* testing loop docs
* Convert __dataloader to _dataloader
This will let inherited classes use it
* Factor common test model setup into base class
* Specialized test modules inherit from LightningTestModelBase
* Fix __is_overriden so that it works with more complicated inheritance
* Use mixins to add functionality to test models
* Fix test with no val_dataloader
* Remove unused imports
* Get rid of wild card import
* Update trainer.py
* Update lm_test_module.py
2019-09-02 15:46:16 -04:00
Verena Haunschmid
25d5b25792
Expectopatronum implement #89 ( #182 )
...
* rename validate -> evaluate; implement test logic; allow multiple test_loaders
* add test_step and test_end to LightningModule
* add in_test_mode to pretraining to implement case 2 (test pretrained model)
* fix code style issues
* LightningTestModel: add optional second test set, implement test_step and test_end
* implemented test for multiple test_dataloaders; fixed typo
* add two test cases for #89
* add documentation for test_step, test_end; fix computation of loss in validation_step example
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Added proper dp ddp routing calls for test mode
* Update trainer.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update override_data_parallel.py
* Update test_models.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update test_models.py
* Update test_models.py
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* Update trainer.py
* Update override_data_parallel.py
* Update debug.py
* Update lm_test_module.py
* Update test_models.py
2019-09-02 07:15:27 -04:00
Stanislav
73cf47112e
Gradient accumulation callback ( #150 )
...
* Gradient accumulation callback
* little test case
* typo
* import fix
* method name fix
* fix epochs indexing from 1
* better code style
* code style fix v2 :/
* change interface
* fix Trainre new api in tests
* trainer api bug fix
* new raising error, new update method
* extentions tests
* a little better tests
* typo fix
* flack8 better
* using scheduler for int and dict
* typo
* firs epoch bug fix
* test update
* empty dict exception
* floats check
* codestyle fix
* grad counting test
* someday, i will install normal linter
* add more checks
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
2019-08-30 10:56:14 -04:00
Ir1dXD
c2247350bb
feat(val_sanity): enable skipping validation sanity ( #176 )
...
* feat(val_sanity): enable skipping validation sanity when self.nb_sanity_val_steps is 0
* docs: elaborate on skipping
2019-08-28 06:41:31 -04:00
William Falcon
4104a0fc47
cleaned up progbar ( #165 )
...
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* flake 8
2019-08-23 21:23:27 -04:00
Sebastian Præsius
b31539f62e
Guard against AttributeError in dataloaders. ( #161 )
...
A solution for https://github.com/williamFalcon/pytorch-lightning/issues/142 .
Since hasattr "calls getattr(object, name) and to see whether it raises an AttributeError or not", I replaced it with a single call to getattr.
See also https://stackoverflow.com/questions/24971061/python-hasattr-vs-getattr
2019-08-23 08:21:39 -04:00
William Falcon
cbb9821d9b
Cleaned up val/tng/test nb batches ( #163 )
...
Set all to be 0 instead of None.
Cleaned up val batch
2019-08-23 07:42:17 -04:00
eqs
4a0b56755c
bug fix for #157 ( #158 )
...
* Separate condition list/tuple case into separated cases
* Add test for tuple of tensor list and list of tensor dict
* Update test_models.py
2019-08-21 10:22:51 -04:00
William Falcon
55a804b7cf
fixes #154 ( #155 )
...
* fixes #154
* Update trainer.py
* Update trainer.py
2019-08-20 16:59:26 -04:00
Ananya Harsh Jha
5b694c7e0e
bug fix for #138 ( #143 )
...
* bug fix for #138
* split if for readability
2019-08-19 15:03:04 -04:00
sebftw
4bdb976284
Set val_check_interval default to 1.0. ( #145 )
...
See discussion in https://github.com/williamFalcon/pytorch-lightning/issues/139 .
2019-08-19 10:42:08 -04:00
sebftw
23a4421595
Removed redundant line. ( #140 )
2019-08-18 18:16:30 -04:00
sebftw
26d3f0dbea
Error if dataset size = 1 batch. ( #141 )
...
Fix for the bug mentioned in https://github.com/williamFalcon/pytorch-lightning/issues/139
2019-08-18 18:15:58 -04:00
Maxim Andreev
e646d745da
use val_percent_check in validation step ( #135 )
2019-08-18 11:02:28 -04:00
William Falcon
308239cef0
allow loss to be used for early stopping ( #127 )
2019-08-16 11:58:44 -04:00
William Falcon
a27fb5d54c
enhanced optimizer return options ( #120 )
...
* added smarter optimizer options
* added smarter optimizer options
* added smarter optimizer options tests
* added smarter optimizer options tests
* added smarter optimizer options tests
* added smarter optimizer options tests
* added smarter optimizer options tests
* added smarter optimizer options tests
* added smarter optimizer options tests
* added single gpu data transfer recursive
* added single gpu data transfer recursive
* added single gpu data transfer recursive
* added single gpu data transfer recursive
* added single gpu data transfer recursive
2019-08-15 11:31:56 -04:00
William Falcon
db9254acbe
enable recursive parsing for single gpu inputs ( #121 )
...
* added tests
* added single gpu data transfer recursive
* added single gpu data transfer recursive
* added single gpu data transfer recursive
* added single gpu data transfer recursive
* added single gpu data transfer recursive
* added single gpu data transfer recursive
2019-08-15 09:39:09 -04:00
William Falcon
2f984c9971
enable returning only opt list ( #114 )
2019-08-14 09:02:11 -04:00
William Falcon
7f53e7bfb3
Val idx optional in validation_step ( #108 )
...
* made dataset_i only available with multiple datasets
* updated interface signature
* updated tests
2019-08-13 11:37:37 -04:00
William Falcon
905a2e5a12
allow user to control optimizer step for every optimizer
...
* added custom hook for user defined optimizer step
* refactored to allow multiple optimizers different training_step
* refactored to allow multiple optimizers different training_step
* refactored to allow multiple optimizers different training_step
* refactored to allow multiple optimizers different training_step
* refactored to allow multiple optimizers different training_step
* pep8
2019-08-13 09:32:45 -04:00
William Falcon
5d5968033f
LR scheduler + train refactor ( #103 )
...
* split __train up for clarity
* split __train up for clarity
* added lr scheduler after epoch completes
2019-08-12 16:07:42 -04:00
Sidhanth Holalkere
511f7ecb9a
Support for multiple val_dataloaders ( #97 )
...
* Added support for multiple validation dataloaders
* Fix typo in README.md
* Update trainer.py
* Add support for multiple dataloaders
* Rename dataloader_index to dataloader_i
* Added warning to check val_dataloaders
Added a warning to ensure that all val_dataloaders were DistributedSamplers if ddp is enabled
* Updated DistributedSampler warning
* Fixed typo
* Added multiple val_dataloaders
* Multiple val_dataloader test
* Update lightning_module_template.py
Added dataloader_i to validation_step parameters
* Update trainer.py
* Reverted template changes
* Create multi_val_module.py
* Update no_val_end_module.py
* New MultiValModel
* Rename MultiValModel to MultiValTestModel
* Revert to LightningTestModel
* Update test_models.py
* Update trainer.py
* Update test_models.py
* multiple val_dataloaders in test template
* Fixed flake8 warnings
* Update trainer.py
* Fix flake errors
* Fixed Flake8 errors
* Update lm_test_module.py
keep this test model with a single dataset for val
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update RequiredTrainerInterface.md
* Update RequiredTrainerInterface.md
* Update test_models.py
* Update trainer.py
dont need the else clause, val_dataloader is either a list or none because of get_dataloaders()
* Update trainer.py
fixed flake errors
* Update trainer.py
2019-08-12 15:23:11 -04:00
William Falcon
e5805bf8ff
val and test are optional now ( #95 )
...
* made validation step optional
* added no val model
* val_step can be implemented but not validation_end
* added no val end model
* added tests
* added tests
* remove class
* remove class
* remove class
* remove class
* remove class
* remove class
* remove class
* remove class
* remove class
* remove class
* remove class
* updated docs
* updated docs
* updated test
* updated test
* updated test
* updated test
* updated test
* updated test
* updated test
* updated test
* updated test
* fix pep8
2019-08-11 10:01:57 -04:00
Nic Eggert
996b1f9a6d
When running DDP without DistributedSampler, throw warning instead of exception ( #91 )
2019-08-10 15:58:12 -04:00
William Falcon
73d08557ba
fix accumulated grad norm fixes #87 ( #88 )
...
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
2019-08-10 08:32:45 -04:00
kvhooreb