

Home

New project Quick Start

To start a new project you define two files, a LightningModule and a Trainer file. A separate trainer file allows to run many LightningModules. Each LightningModule has the core logic to a particular research project. For example, one lightningModule could be an image classifier, the other one could be a seq-2-seq model, both (optionally) ran by the same trainer file.

1. MNIST LightningModule
2. Trainer
   • Basic CPU Trainer Template
   • Multi-GPU Trainer Template
   • GPU cluster Trainer Template

Docs shortcuts

• LightningModule
• Trainer

Quick start examples

• CPU example
• Hyperparameter search on single GPU
• Hyperparameter search on multiple GPUs on same node
• Hyperparameter search on a SLURM HPC cluster

Checkpointing

• Model saving
• Model loading

Computing cluster (SLURM)

• Running grid search on a cluster
• Walltime auto-resubmit

Debugging

• Fast dev run
• Inspect gradient norms
• Log GPU usage
• Make model overfit on subset of data
• Print the parameter count by layer
• Pring which gradients are nan

Distributed training

• 16-bit mixed precision
• Multi-GPU
• Multi-node
• Single GPU
• Self-balancing architecture

Experiment Logging

• Display metrics in progress bar
• Log metric row every k batches
• Process position
• Tensorboard support
• Save a snapshot of all hyperparameters
• Snapshot code for a training run
• Write logs file to csv every k batches

Training loop

• Accumulate gradients
• Anneal Learning rate
• Force training for min or max epochs
• Force disable early stop
• Gradient Clipping
• Hooks
• Use multiple optimizers (like GANs)
• Set how much of the training set to check (1-100%)

Validation loop

• Check validation every n epochs
• Hooks
• Set how much of the validation set to check
• Set how much of the test set to check
• Set validation check frequency within 1 training epoch
• Set the number of validation sanity steps