pytorch-lightning/docs/Trainer/index.md

Trainer

[Github Code]

The lightning trainer abstracts best practices for running a training, val, test routine. It calls parts of your model when it wants to hand over full control and otherwise makes training assumptions which are now standard practice in AI research.

This is the basic use of the trainer:

from pytorch_lightning import Trainer

model = LightningTemplate()

trainer = Trainer()
trainer.fit(model)

But of course the fun is in all the advanced things it can do:

Training loop

• Accumulate gradients
• Anneal Learning rate
• Check GPU usage
• Check which gradients are nan
• Display metrics in progress bar
• Display the parameter count by layer
• Fast dev run
• Force training for min or max epochs
• Force disable early stop
• Inspect gradient norms
• Make model overfit on subset of data
• Use multiple optimizers (like GANs)
• Process position
• Set how much of the training set to check (1-100%)

Validation loop

• Check validation every n epochs
• Set how much of the validation set to check
• Set how much of the test set to check
• Set validation check frequency within 1 training epoch
• Set the number of validation sanity steps
• Check validation every n epochs

Distributed training

• Single-gpu
• Multi-gpu
• Multi-node
• 16-bit mixed precision

Checkpointing

• Model saving
• Model loading

Computing cluster (SLURM)

• Automatic checkpointing
• Automatic saving, loading
• Running grid search on a cluster
• Walltime auto-resubmit