William Falcon
cdfcb01073
Fixes #234 ( #311 )
...
* Fixes #234
* default logger version is now slurm job id
* default logger version is now slurm job id
2019-10-05 14:45:37 -04:00
William Falcon
6cc3f1757f
decouple returns from each step ( #307 )
...
* decoupled training metrics from logging metrics
* decoupled validation metrics from log metrics
* updated docs
* updated docs
* updated docs
* Fixed test
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
* merged master
2019-10-05 13:35:20 -04:00
William Falcon
8f5a06bfb8
Gpu mem ( #308 )
...
* Fixes #289
* Fixes #289
* added lbfgs support
* Fixes #280 (#309 )
* added test seeds (#306 )
* added test seeds
* added test seeds
* updated docs
* added lbfgs support (#310 )
* added lbfgs support
* added lbfgs support
* added lbfgs support
* Fixes #280 (#309 )
* added test seeds (#306 )
* added test seeds
* added test seeds
* updated docs
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* Fixes #289
* Fixes #289
* merged master
* merged master
2019-10-05 11:29:34 -04:00
William Falcon
75fd89106f
added lbfgs support ( #310 )
...
* added lbfgs support
* added lbfgs support
* added lbfgs support
* Fixes #280 (#309 )
* added test seeds (#306 )
* added test seeds
* added test seeds
* updated docs
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
* added lbfgs support
2019-10-05 11:10:21 -04:00
William Falcon
2ac9f1aea7
Fixes #280 ( #309 )
2019-10-05 10:55:50 -04:00
William Falcon
967957e55c
added lbfgs support
2019-10-05 10:47:18 -04:00
William Falcon
bf09060fef
Fixes #292 ( #303 )
...
* early stopping callback is not default
* added a default logger
* added default checkpoint callback
* added default checkpoint/loggers
* added default checkpoint/loggers
* updated docs
* cleaned demos
* cleaned demos
* cleaned demos
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
* clean up docs around loggers
2019-10-04 19:48:57 -04:00
William Falcon
a578de511d
clean up docs around loggers ( #304 )
2019-10-04 18:53:38 -04:00
William Falcon
a60a24d11b
disable auto gpu loading when restoring weights to avoid OOM ( #242 )
...
* Update root_module.py
* Update root_module.py
* Update root_module.py
* tests fix
* tests fix
2019-10-04 16:18:43 -04:00
William Falcon
73a7cf3c99
Mem crash ( #299 )
...
* fixes memory crash
* fixes memory crash
2019-10-04 15:53:44 -04:00
Hendrik Schröter
36f0b5bbd0
Use getter instead of python property for the dataloaders ( #275 )
...
* Use getter instead of python property for the dataloaders
* Fix lint
* Update trainer.py
2019-10-04 15:35:02 -04:00
William Falcon
32e74b8f36
Ddp2 ( #261 )
...
* adds ddp2 option where on each node a single process uses all gpus
* added ddp2 test
* added ddp2 docs
* Update Distributed training.md
* delete ref to old update_training_log_metrics
* delete ref to old update_training_log_metrics
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* banana pancakes
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* cheesecake
2019-10-04 15:07:54 -04:00
Hendrik Schröter
42764d18c7
Better error message if no loss was returned from model.training_step() ( #294 )
2019-10-04 07:15:19 -04:00
kvhooreb
41236c7bbb
WIP: Moved grad_norm tracking code to __run_tng_batch ( #278 )
...
* Moved grad_norm tracking code to __run_tng_batch + added norms to tqdm_metrics
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
2019-10-02 11:11:08 -04:00
Nic Eggert
614cb3c03b
Initialize loggers only once ( #270 )
...
* Create underlying loggers lazily
This avoids creating duplicate experiments or run in multi-node DDP.
* Save hyperparameters automatically
* Update docs for snapshotting hyperparams
* Fix test tube
* Fix test tube pickling
2019-10-02 11:10:40 -04:00
Anton Bakhtin
222d7d2d5d
Hacky fix for mlflow logger ( #277 )
...
* Hacky fix for mlflow logger
It dies when "created_at" is logged
* Log warning
2019-10-01 21:32:52 -04:00
Hendrik Schröter
8a2472269a
Make test_tube optional ( #274 )
2019-10-01 05:19:47 -04:00
Nic Eggert
480eed5cb6
Enable any ML experiment tracking framework ( #223 )
...
* Implement generic loggers for experiment tracking
* Add tests for loggers
* Get model tests passing
* Test and fix logger pickling
* Expand pickle test and fix bug
* Missed exp -> logger conversion
* Remove commented code
* Add docstrings
* Update logging docs
* Add mlflow to test requirements
* Make linter happy
* Fix mlflow timestamp
* Update Logging.md
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update properties.md
* Fix tests
* Line length
2019-09-27 12:05:29 -04:00
William Falcon
481aa24974
always calls the lr scheduler with epoch nb. Fixes #98 ( #252 )
...
* always calls the lr scheduler with epoch nb
* added docs for cluster grid search
* added docs for cluster grid search
* undo test changes
* undo test changes
2019-09-26 16:36:41 -04:00
William Falcon
25d2f93256
enables samplers which don't need set epoch (or when ppl don't need a sampler) ( #254 )
...
* enables samplers which dont need set epoch
* added docs for single gpu ddp
* added docs for single gpu ddp
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
* added docs for cluster grid search
2019-09-26 14:39:04 -04:00
William Falcon
8b2a2aeda3
Dim 0 warning ( #256 )
...
* added ignore warnings module
* added ignore warnings module
* Fixes #249
* Update ignored_warnings.py
2019-09-26 13:20:54 -04:00
Alok Singh
b0a0a47a0b
Rename variables ( #124 )
...
- data_batch → batch
- batch_i → batch_idx
- dataloader_i → dataloader_idx
- tng → training
- training_dataloader → train_dataloader
- add_log_row_interval → row_log_interval
- gradient_clip → gradient_clip_val
- prog → progress
- tqdm_dic → tqdm_dict
2019-09-25 19:05:06 -04:00
William Falcon
87708157bc
Update trainer.py ( #233 )
2019-09-19 08:23:48 -04:00
William Falcon
55e7322747
Metrics load ( #228 )
...
* load from metrics defaults to CPU
* load from metrics defaults to CPU
* load from metrics defaults to CPU
2019-09-16 10:47:19 -04:00
Ananya Harsh Jha
c0f3b6b035
added set_epoch for distributed sampler, fix for #224 ( #225 )
2019-09-16 10:21:00 -04:00
William Falcon
9576dd28b2
added load on CPU first ( #221 )
...
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added load on CPU first
* added print logs
* added print logs
* changed close order
* changed close order
2019-09-11 07:52:36 -04:00
William Falcon
30b25c8146
Sai prasanna master ( #219 )
...
* Fix incorrect warning for DistributedSampler.
Check whether `dataloader.sampler` is an instance of DistributedSampler instead of checking the `dataloader`.
* Update trainer.py
* merged
2019-09-09 11:36:24 -04:00
William Falcon
506d5da68b
enable single gpu per node ( #218 )
...
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
* enable single gpu per node
2019-09-09 07:37:20 -04:00
William Falcon
10d190e045
Simplified gpu api. No NVIDIA flag managing by lightning for cluster ( #213 )
...
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added nvidia flag set
* added simple cluster template
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
* sets correct backend for possible combinations of gpu inputs
2019-09-08 15:36:58 -04:00
Alok Singh
81df2259ef
Make print_nan_grads print grad ( #208 )
...
This seems more useful for debugging.
2019-09-07 01:08:09 -04:00
William Falcon
0c7fbc7178
Weights path ( #211 )
...
* added docs. removed options. added weights_save option
* removed old restore
* cleaned up save path
* cleaned up save path
* flake8
2019-09-06 17:01:03 -04:00
William Falcon
7099f8dbfb
split trainer mixins ( #209 )
...
* split trainer mixins
* Update multi_node_cluster_template.py
* Update single_cpu_template.py
* Update single_gpu_node_16bit_template.py
* Update single_gpu_node_ddp_template.py
* Update single_gpu_node_dp_template.py
* Update trainer_cpu_template.py
* Update trainer_io.py
* split trainer mixins
* Update multi_node_cluster_template.py
* deconflicted
* deconflicted
* deconflicted
2019-09-06 14:11:07 -04:00
William Falcon
60633eaa32
Moves hpc auto-resubmit to trainer from test-tube ( #207 )
...
* added slurm signal handler
* added restore weight functions
* set slurm signal handling inside process
* added resubmit docs
* added resubmit docs
* fixed missing param
* Update trainer.py
* fixed missing param
* fixed missing param
* debugging tests
* debugging tests
* debugging tests
* debugging tests
* debugging tests
* debugging tests
* debugging tests
2019-09-06 11:54:51 -04:00
Nic Eggert
1733dba735
Pass outputs from all dataloaders to test_end and validation_end ( #203 )
...
* Pass outputs from all dataloaders to test_end and validation_end
* Update tests
* Update docs
* Update trainer.py
* Update test_models.py
2019-09-06 07:37:25 -04:00
William Falcon
7e0ac3149c
refactored init ( #206 )
2019-09-06 00:29:38 -04:00
Anton Konstantinov
34b824a9d3
Implement correct transfer to GPU for batches ( #200 )
2019-09-05 07:13:06 -04:00
Thomas J Fan
62252cee58
STY Minor flake8 fix ( #197 )
2019-09-04 17:46:56 -04:00
Max Horn
dac41030d4
Allow to deactivate GPU memory logging in Trainer ( #190 )
...
* Allow to deactivate GPU memory logging in Trainer
Adds the flag `log_gpu_memory` to Trainer to deactivate logging of GPU
memory utilization. On some servers logging the GPU memory usage can
significantly slow down training.
* Update Logging.md
* Update trainer.py
2019-09-04 10:43:46 -04:00
Nic Eggert
64688e1e15
Refactor test modules ( #180 )
...
* Expectopatronum implement #89 (#182 )
* rename validate -> evaluate; implement test logic; allow multiple test_loaders
* add test_step and test_end to LightningModule
* add in_test_mode to pretraining to implement case 2 (test pretrained model)
* fix code style issues
* LightningTestModel: add optional second test set, implement test_step and test_end
* implemented test for multiple test_dataloaders; fixed typo
* add two test cases for #89
* add documentation for test_step, test_end; fix computation of loss in validation_step example
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Added proper dp ddp routing calls for test mode
* Update trainer.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update override_data_parallel.py
* Update test_models.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update test_models.py
* Update test_models.py
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* Update trainer.py
* Update override_data_parallel.py
* Update debug.py
* Update lm_test_module.py
* Update test_models.py
* release v0.4.8
* Update README.md
* add training loop docs
* testing loop docs
* testing loop docs
* Convert __dataloader to _dataloader
This will let inherited classes use it
* Factor common test model setup into base class
* Specialized test modules inherit from LightningTestModelBase
* Fix __is_overriden so that it works with more complicated inheritance
* Use mixins to add functionality to test models
* Fix test with no val_dataloader
* Remove unused imports
* Get rid of wild card import
* Update trainer.py
* Update lm_test_module.py
2019-09-02 15:46:16 -04:00
Verena Haunschmid
25d5b25792
Expectopatronum implement #89 ( #182 )
...
* rename validate -> evaluate; implement test logic; allow multiple test_loaders
* add test_step and test_end to LightningModule
* add in_test_mode to pretraining to implement case 2 (test pretrained model)
* fix code style issues
* LightningTestModel: add optional second test set, implement test_step and test_end
* implemented test for multiple test_dataloaders; fixed typo
* add two test cases for #89
* add documentation for test_step, test_end; fix computation of loss in validation_step example
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Added proper dp ddp routing calls for test mode
* Update trainer.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update override_data_parallel.py
* Update test_models.py
* Update test_models.py
* Update trainer.py
* Update trainer.py
* Update trainer.py
* Update test_models.py
* Update test_models.py
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* debug
* Update trainer.py
* Update override_data_parallel.py
* Update debug.py
* Update lm_test_module.py
* Update test_models.py
2019-09-02 07:15:27 -04:00
Stanislav
73cf47112e
Gradient accumulation callback ( #150 )
...
* Gradient accumulation callback
* little test case
* typo
* import fix
* method name fix
* fix epochs indexing from 1
* better code style
* code style fix v2 :/
* change interface
* fix Trainre new api in tests
* trainer api bug fix
* new raising error, new update method
* extentions tests
* a little better tests
* typo fix
* flack8 better
* using scheduler for int and dict
* typo
* firs epoch bug fix
* test update
* empty dict exception
* floats check
* codestyle fix
* grad counting test
* someday, i will install normal linter
* add more checks
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
* Update test_models.py
2019-08-30 10:56:14 -04:00
Ir1dXD
c2247350bb
feat(val_sanity): enable skipping validation sanity ( #176 )
...
* feat(val_sanity): enable skipping validation sanity when self.nb_sanity_val_steps is 0
* docs: elaborate on skipping
2019-08-28 06:41:31 -04:00
William Falcon
4104a0fc47
cleaned up progbar ( #165 )
...
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* cleaned up progbar
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* updated base files
* flake 8
2019-08-23 21:23:27 -04:00
Sebastian Præsius
b31539f62e
Guard against AttributeError in dataloaders. ( #161 )
...
A solution for https://github.com/williamFalcon/pytorch-lightning/issues/142 .
Since hasattr "calls getattr(object, name) and to see whether it raises an AttributeError or not", I replaced it with a single call to getattr.
See also https://stackoverflow.com/questions/24971061/python-hasattr-vs-getattr
2019-08-23 08:21:39 -04:00
William Falcon
cbb9821d9b
Cleaned up val/tng/test nb batches ( #163 )
...
Set all to be 0 instead of None.
Cleaned up val batch
2019-08-23 07:42:17 -04:00
eqs
4a0b56755c
bug fix for #157 ( #158 )
...
* Separate condition list/tuple case into separated cases
* Add test for tuple of tensor list and list of tensor dict
* Update test_models.py
2019-08-21 10:22:51 -04:00
William Falcon
55a804b7cf
fixes #154 ( #155 )
...
* fixes #154
* Update trainer.py
* Update trainer.py
2019-08-20 16:59:26 -04:00
Ananya Harsh Jha
5b694c7e0e
bug fix for #138 ( #143 )
...
* bug fix for #138
* split if for readability
2019-08-19 15:03:04 -04:00
sebftw
4bdb976284
Set val_check_interval default to 1.0. ( #145 )
...
See discussion in https://github.com/williamFalcon/pytorch-lightning/issues/139 .
2019-08-19 10:42:08 -04:00
sebftw
23a4421595
Removed redundant line. ( #140 )
2019-08-18 18:16:30 -04:00
William Falcon