+ + + + + + Examples + Template model definition In 99% of cases you want to just copy this template to start a new lightningModule and change the core of what your model is actually trying to do. # get a copy of the module template @@ -328,54 +728,79 @@ hyperparams.optimize_parallel_gpu( # run cluster hyperparameter search optimize_on_cluster(hyperparams) + + + -
+ + + + + Lightning Module interface [Github Code] A lightning module is a strict superclass of nn.Module, it provides a standard interface for the trainer to interact with the model. @@ -596,54 +1265,79 @@ def add_model_specific_args(parent_parser, root_dir): parser.opt_list('--optimizer_name', default='adam', type=str, options=['adam'], tunable=False) return parser + + + -
+ + + + + + Methods + Lightning modules are strict superclasses of torch.nn.Module. A LightningModule offers the following in addition to that API. freeze @@ -230,54 +595,79 @@ y_hat = pretrained_model(x) model = MyLightningModule(...) model.unfreeze() + + + -
+ + + + + + Properties + A LightningModule has the following properties which you can access at any time current_epoch @@ -199,54 +597,79 @@ Trainer Last resort access to any state the trainer has. Changing certain properties here could affect your training run. + + + -
+ + + + + + Checkpointing + Lightning can automate saving and loading checkpoints. Model saving @@ -181,54 +524,79 @@ checkpoint_callback = ModelCheckpoint( trainer = Trainer(checkpoint_callback=checkpoint_callback) + + + -
+ + + + + + Distributed training + Lightning makes multi-gpu training and 16 bit training trivial. Note: None of the flags below require changing anything about your lightningModel definition. @@ -224,54 +611,79 @@ trainer = Trainer(gpus=[0,1,2,3,4,5,6,7]) Self-balancing architecture Here lightning distributes parts of your module across available GPUs to optimize for speed and memory. COMING SOON. + + + -
+ + + + + + Logging + Lighting offers a few options for logging information about model, gpu usage, etc (via test-tube). It also offers printing options for training monitoring. Display metrics in progress bar @@ -231,54 +629,79 @@ Trainer(experiment=exp) # DEFAULT (ie: save a .csv log file every 100 batches) trainer = Trainer(log_save_interval=100) + + + -
+ + + + + + SLURM Managed Cluster + Lightning supports model training on a cluster managed by SLURM in the following cases: Training on single or multi-cpus only. @@ -259,54 +613,79 @@ With this feature lightning will: resubmit a continuation job. load the checkpoint and trainer session in the new model + + + -
+ + + + + + Training Loop + The lightning training loop handles everything except the actual computations of your model. To decide what will happen in your training loop, define the training_step function. Below are all the things lightning automates for you in the training loop. @@ -242,54 +651,79 @@ trainer = Trainer(train_percent_check=1.0) # check 10% only trainer = Trainer(train_percent_check=0.1) + + + -
+ + + + + + Validation loop + The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the validation_step function. Below are all the things lightning automates for you in the validation loop. Note @@ -224,54 +611,79 @@ trainer = Trainer(val_check_interval=0.25) # DEFAULT trainer = Trainer(nb_sanity_val_steps=5) + + + -
+ + + + + + Debugging + These flags are useful to help debug a model. Fast dev run @@ -222,54 +620,79 @@ trainer = Trainer(print_nan_grads=False) Log GPU usage Lightning automatically logs gpu usage to the test tube logs. It'll only do it at the metric logging interval, so it doesn't slow down training. + + + -
+ + + + + Trainer [Github Code] The lightning trainer abstracts best practices for running a training, val, test routine. It calls parts of your model when it wants to hand over full control and otherwise makes training assumptions which are now standard practice in AI research. @@ -232,54 +534,79 @@ trainer.fit(model) Set validation check frequency within 1 training epoch Set the number of validation sanity steps + + + -
+ + + + + PYTORCH-LIGHTNING DOCUMENTATION New project Quick Start To start a new project define these two files. @@ -263,55 +714,65 @@ Set validation check frequency within 1 training epoch Set the number of validation sanity steps + + + -