diff --git a/docs/source/apex.rst b/docs/source/apex.rst new file mode 100644 index 00000000..d4dbc4cd --- /dev/null +++ b/docs/source/apex.rst @@ -0,0 +1,42 @@ +16-bit training +================= +Lightning uses NVIDIA apex to handle 16-bit precision training. + +To use 16-bit precision, do two things: +1. Install Apex +2. Set the amp trainer flag. + +Install apex +---------------------------------------------- +.. code-block:: bash + + $ git clone https://github.com/NVIDIA/apex + $ cd apex + + # ------------------------ + # OPTIONAL: on your cluster you might need to load cuda 10 or 9 + # depending on how you installed PyTorch + + # see available modules + module avail + + # load correct cuda before install + module load cuda-10.0 + # ------------------------ + + # make sure you've loaded a cuda version > 4.0 and < 7.0 + module load gcc-6.1.0 + + $ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./ + + +Enable 16-bit +-------------- + +.. code-block:: python + + # DEFAULT + trainer = Trainer(amp_level='O1', use_amp=False) + +If you need to configure the apex init for your particular use case or want to use a different way of doing +16-bit training, override :meth:`pytorch_lightning.core.LightningModule.configure_apex`. \ No newline at end of file diff --git a/docs/source/checkpointing.rst b/docs/source/checkpointing.rst new file mode 100644 index 00000000..318d95dd --- /dev/null +++ b/docs/source/checkpointing.rst @@ -0,0 +1,80 @@ +Checkpointing +============== + +.. _model-saving: + +Model saving +------------------- +To save a LightningModule, provide a :meth:`pytorch_lightning.callbacks.ModelCheckpoint` callback. + +The Lightning checkpoint also saves the hparams (hyperparams) passed into the LightningModule init. + +.. note:: hparams is a `Namespace `_ or dictionary. + +.. code-block:: python + :emphasize-lines: 8 + + from argparse import Namespace + + # usually these come from command line args + args = Namespace(**{'learning_rate':0.001}) + + # define you module to have hparams as the first arg + # this means your checkpoint will have everything that went into making + # this model (in this case, learning rate) + class MyLightningModule(pl.LightningModule): + + def __init__(self, hparams, ...): + self.hparams = hparams + + my_model = MyLightningModule(args) + + # auto-saves checkpoint + checkpoint_callback = ModelCheckpoint(filepath='my_path') + Trainer(checkpoint_callback=checkpoint_callback) + + +Model loading +----------------------------------- + +To load a model, use :meth:`pytorch_lightning.core.LightningModule.load_from_checkpoint` + +.. note:: If lightning created your checkpoint, your model will receive all the hyperparameters used + to create the checkpoint. (See: :ref:`model-saving`). + +.. code-block:: python + + # load weights without mapping + MyLightningModule.load_from_checkpoint('path/to/checkpoint.ckpt') + + # load weights mapping all weights from GPU 1 to GPU 0 + map_location = {'cuda:1':'cuda:0'} + MyLightningModule.load_from_checkpoint('path/to/checkpoint.ckpt', map_location=map_location) + +Restoring training session +----------------------------------- + +If you want to pick up training from where you left off, you have a few options. + +1. Pass in a logger with the same experiment version to continue training. + +.. code-block:: python + + # train the first time and set the version number + logger = TensorboardLogger(version=10) + trainer = Trainer(logger=logger) + trainer.fit(model) + + # when you init another logger with that same version, the model + # will continue where it left off + logger = TensorboardLogger(version=10) + trainer = Trainer(logger=logger) + trainer.fit(model) + +2. A second option is to pass in a path to a checkpoint (see: :ref:`pytorch_lightning.trainer`). + +.. code-block:: python + + # train the first time and set the version number + trainer = Trainer(resume_from_checkpoint='some/path/to/my_checkpoint.ckpt') + trainer.fit(model) \ No newline at end of file diff --git a/docs/source/common-cases.rst b/docs/source/common-cases.rst deleted file mode 100644 index cc4ca362..00000000 --- a/docs/source/common-cases.rst +++ /dev/null @@ -1,27 +0,0 @@ -Multi-gpu (same node) training -============================== - -Multi-node training -==================== - -16-bit precision -================= - -gradient clipping -================= - -modifying training via hooks -============================= - -.. toctree:: - :maxdepth: 3 - - pl_examples - - -profiling a training run -======================== -.. toctree:: - :maxdepth: 1 - - profiler \ No newline at end of file diff --git a/docs/source/debugging.rst b/docs/source/debugging.rst new file mode 100644 index 00000000..b4f15202 --- /dev/null +++ b/docs/source/debugging.rst @@ -0,0 +1,75 @@ +Debugging +========== +The following are flags that make debugging much easier. + +Fast dev run +------------------- +This flag runs a "unit test" by running 1 training batch and 1 validation batch. +The point is to detect any bugs in the training/validation loop without having to wait for +a full epoch to crash. + +.. code-block:: python + + trainer = pl.Trainer(fast_dev_run=True) + +Inspect gradient norms +----------------------------------- +Logs (to a logger), the norm of each weight matrix. + +.. code-block:: python + + # the 2-norm + trainer = pl.Trainer(track_grad_norm=2) + +Log GPU usage +----------------------------------- +Logs (to a logger) the GPU usage for each GPU on the master machine. + +(See: :ref:`trainer`) + +.. code-block:: python + + trainer = pl.Trainer(log_gpu_memory=True) + +Make model overfit on subset of data +----------------------------------- + +A good debugging technique is to take a tiny portion of your data (say 2 samples per class), +and try to get your model to overfit. If it can't, it's a sign it won't work with large datasets. + +(See: :ref:`trainer`) + +.. code-block:: python + + trainer = pl.Trainer(overfit_pct=0.01) + +Print the parameter count by layer +----------------------------------- +Whenever the .fit() function gets called, the Trainer will print the weights summary for the lightningModule. +To disable this behavior, turn off this flag: + +(See: :ref:`trainer.weights_summary`) + +.. code-block:: python + + trainer = pl.Trainer(weights_summary=None) + +Print which gradients are nan +------------------------------ +Prints the tensors with nan gradients. + +(See: :meth:`trainer.print_nan_grads`) + +.. code-block:: python + + trainer = pl.Trainer(print_nan_grads=False) + +Set the number of validation sanity steps +------------------------------------- +Lightning runs a few steps of validation in the beginning of training. +This avoids crashing in the validation loop sometime deep into a lengthy training loop. + +.. code-block:: python + + # DEFAULT + trainer = Trainer(nb_sanity_val_steps=5) \ No newline at end of file diff --git a/docs/source/early_stopping.rst b/docs/source/early_stopping.rst new file mode 100644 index 00000000..a7eb2226 --- /dev/null +++ b/docs/source/early_stopping.rst @@ -0,0 +1,35 @@ +Early stopping +================== + + +Enable Early Stopping +---------------------- +There are two ways to enable early stopping. + +.. note:: See: :ref:`trainer` + +.. code-block:: python + + # A) Looks for val_loss in validation_step return dict + trainer = Trainer(early_stop_callback=True) + + # B) Or configure your own callback + early_stop_callback = EarlyStopping( + monitor='val_loss', + min_delta=0.00, + patience=3, + verbose=False, + mode='min' + ) + trainer = Trainer(early_stop_callback=early_stop_callback) + +Force disable early stop +------------------------------------- +To disable early stopping pass None to the early_stop_callback + +.. note:: See: :ref:`trainer` + +.. code-block:: python + + # DEFAULT + trainer = Trainer(early_stop_callback=None) \ No newline at end of file diff --git a/docs/source/experiment_logging.rst b/docs/source/experiment_logging.rst new file mode 100644 index 00000000..e3f5f83b --- /dev/null +++ b/docs/source/experiment_logging.rst @@ -0,0 +1,139 @@ +Experiment Logging +=================== + +Comet.ml +^^^^^^^^^^ + +`Comet.ml `_ is a third-party logger. +To use CometLogger as your logger do the following. + +.. note:: See: :ref:`comet` docs. + +.. code-block:: python + + from pytorch_lightning.loggers import TestTubeLogger + + comet_logger = CometLogger( + api_key=os.environ["COMET_KEY"], + workspace=os.environ["COMET_WORKSPACE"], # Optional + project_name="default_project", # Optional + rest_api_key=os.environ["COMET_REST_KEY"], # Optional + experiment_name="default" # Optional + ) + trainer = Trainer(logger=comet_logger) + +The CometLogger is available anywhere in your LightningModule + +.. code-block:: python + + class MyModule(pl.LightningModule): + + def __init__(self, ...): + some_img = fake_image() + self.logger.experiment.add_image('generated_images', some_img, 0) + +Neptune.ai +^^^^^^^^^^ + +`Neptune.ai `_ is a third-party logger. +To use Neptune.ai as your logger do the following. + +.. note:: See: :ref:`neptune` docs. + +.. code-block:: python + + from pytorch_lightning.loggers import NeptuneLogger + + neptune_logger = NeptuneLogger( + project_name="USER_NAME/PROJECT_NAME", + experiment_name="default", # Optional, + params={"max_epochs": 10}, # Optional, + tags=["pytorch-lightning","mlp"] # Optional, + ) + trainer = Trainer(logger=neptune_logger) + +The Neptune.ai is available anywhere in your LightningModule + +.. code-block:: python + + class MyModule(pl.LightningModule): + + def __init__(self, ...): + some_img = fake_image() + self.logger.experiment.add_image('generated_images', some_img, 0) + +Tensorboard +^^^^^^^^^^^^^ + +To use `Tensorboard `_ as your logger do the following. + +.. note:: See: TensorBoardLogger :ref:`tf-logger` + +.. code-block:: python + + from pytorch_lightning.loggers import TensorBoardLogger + + logger = TensorBoardLogger("tb_logs", name="my_model") + trainer = Trainer(logger=logger) + +The TensorBoardLogger is available anywhere in your LightningModule + +.. code-block:: python + + class MyModule(pl.LightningModule): + + def __init__(self, ...): + some_img = fake_image() + self.logger.experiment.add_image('generated_images', some_img, 0) + + +Test Tube +^^^^^^^^^^^^^ + +`Test Tube `_ is a tensorboard logger but with nicer file structure. +To use TestTube as your logger do the following. + +.. note:: See: TestTube :ref:`testTube` + +.. code-block:: python + + from pytorch_lightning.loggers import TestTubeLogger + + logger = TestTubeLogger("tb_logs", name="my_model") + trainer = Trainer(logger=logger) + +The TestTubeLogger is available anywhere in your LightningModule + +.. code-block:: python + + class MyModule(pl.LightningModule): + + def __init__(self, ...): + some_img = fake_image() + self.logger.experiment.add_image('generated_images', some_img, 0) + +Wandb +^^^^^^^^^^^^^ + +`Wandb `_ is a third-party logger. +To use Wandb as your logger do the following. + +.. note:: See: :ref:`wandb` docs + +.. code-block:: python + + from pytorch_lightning.loggers import WandbLogger + + wandb_logger = WandbLogger() + trainer = Trainer(logger=wandb_logger) + +The Wandb logger is available anywhere in your LightningModule + +.. code-block:: python + + class MyModule(pl.LightningModule): + + def __init__(self, ...): + some_img = fake_image() + self.logger.experiment.add_image('generated_images', some_img, 0) + diff --git a/docs/source/experiment_reporting.rst b/docs/source/experiment_reporting.rst new file mode 100644 index 00000000..ce3a2784 --- /dev/null +++ b/docs/source/experiment_reporting.rst @@ -0,0 +1,133 @@ +Experiment Reporting +===================== + +Lightning supports many different experiment loggers. These loggers allow you to monitor losses, images, text, etc... +as training progresses. They usually provide a GUI to visualize and can sometimes even snapshot hyperparameters +used in each experiment. + + +Control logging frequency +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It may slow training down to log every single batch. Trainer has an option to log every k batches instead. + +.. code-block:: python + + # k = 10 + Trainer(row_log_interval=10) + +Control log writing frequency +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Writing to a logger can be expensive. In Lightning you can set the interval at which you +want to log using this trainer flag. + +.. note:: See: :ref:`trainer` + +.. code-block:: python + + k = 100 + Trainer(log_save_interval=k) + +Log metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To plot metrics into whatever logger you passed in (tensorboard, comet, neptune, etc...) + +1. Training_end, validation_end, test_end will all log anything in the "log" key of the return dict. + +.. code-block:: python + + def training_end(self, batch, batch_idx): + loss = some_loss() + ... + + logs = {'train_loss': loss} + results = {'log': logs} + return results + + def validation_end(self, batch, batch_idx): + loss = some_loss() + ... + + logs = {'val_loss': loss} + results = {'log': logs} + return results + + def test_end(self, batch, batch_idx): + loss = some_loss() + ... + + logs = {'test_loss': loss} + results = {'log': logs} + return results + +2. Most of the time, you only need training_step and not training_end. You can also return logs from here: + +.. code-block:: python + + def training_step(self, batch, batch_idx): + loss = some_loss() + ... + + logs = {'train_loss': loss} + results = {'log': logs} + return results + +3. In addition, you can also use any arbitrary functionality from a particular logger from within your LightningModule. +For instance, here we log images using tensorboard. + +.. code-block:: python + + def training_step(self, batch, batch_idx): + self.generated_imgs = self.decoder.generate() + + sample_imgs = self.generated_imgs[:6] + grid = torchvision.utils.make_grid(sample_imgs) + self.logger.experiment.add_image('generated_images', grid, 0) + + ... + return results + +Modify progress bar +^^^^^^^^^^^^^^^^^^^^^^ + +Each return dict from the training_end, validation_end, testing_end and training_step also has +a key called "progress_bar". + +Here we show the validation loss in the progress bar + +.. code-block:: python + + def validation_end(self, batch, batch_idx): + loss = some_loss() + ... + + logs = {'val_loss': loss} + results = {'progress_bar': logs} + return results + +Snapshot hyperparameters +^^^^^^^^^^^^^^^^^^^^^^^^^^ +When training a model, it's useful to know what hyperparams went into that model. +When Lightning creates a checkpoint, it stores a key "hparams" with the hyperparams. + +.. code-block:: python + + lightning_checkpoint = torch.load(filepath, map_location=lambda storage, loc: storage) + hyperparams = lightning_checkpoint['hparams'] + +Some loggers also allow logging the hyperparams used in the experiment. For instance, +when using the TestTubeLogger or the TensorBoardLogger, all hyperparams will show +in the `hparams tab `_. + +Snapshot code +^^^^^^^^^^^^^^^^^^^^^^^^^^ +Loggers also allow you to snapshot a copy of the code used in this experiment. +For example, TestTubeLogger does this with a flag: + +.. code-block:: python + + from pytorch_lightning.loggers import TestTubeLogger + + logger = TestTubeLogger(create_git_tag=True) diff --git a/docs/source/fast_training.rst b/docs/source/fast_training.rst new file mode 100644 index 00000000..b0d909e3 --- /dev/null +++ b/docs/source/fast_training.rst @@ -0,0 +1,82 @@ +Fast Training +================ +There are multiple options to speed up different parts of the training by choosing to train +on a subset of data. This could be done for speed or debugging purposes. + +Check validation every n epochs +------------------------------------- +If you have a small dataset you might want to check validation every n epochs + +.. code-block:: python + + # DEFAULT + trainer = Trainer(check_val_every_n_epoch=1) + +Force training for min or max epochs +------------------------------------- +It can be useful to force training for a minimum number of epochs or limit to a max number. + +.. note:: See: :ref:`trainer` + +.. code-block:: python + + # DEFAULT + trainer = Trainer(min_nb_epochs=1, max_nb_epochs=1000) + + +Set validation check frequency within 1 training epoch +------------------------------------------------------- +For large datasets it's often desirable to check validation multiple times within a training loop. +Pass in a float to check that often within 1 training epoch. Pass in an int k to check every k training batches. +Must use an int if using an IterableDataset. + +.. code-block:: python + + # DEFAULT + trainer = Trainer(val_check_interval=0.95) + + # check every .25 of an epoch + trainer = Trainer(val_check_interval=0.25) + + # check every 100 train batches (ie: for IterableDatasets or fixed frequency) + trainer = Trainer(val_check_interval=100) + +Use training data subset +---------------------------------- +If you don't want to check 100% of the training set (for debugging or if it's huge), set this flag. + +.. code-block:: python + + # DEFAULT + trainer = Trainer(train_percent_check=1.0) + + # check 10% only + trainer = Trainer(train_percent_check=0.1) + +.. note:: train_percent_check will be overwritten by overfit_pct if overfit_pct > 0 + +Use test data subset +------------------------------------- +If you don't want to check 100% of the test set (for debugging or if it's huge), set this flag +test_percent_check will be overwritten by overfit_pct if overfit_pct > 0. + +.. code-block:: python + + # DEFAULT + trainer = Trainer(test_percent_check=1.0) + + # check 10% only + trainer = Trainer(test_percent_check=0.1) + +Use validation data subset +-------------------------------------------- +If you don't want to check 100% of the validation set (for debugging or if it's huge), set this flag +val_percent_check will be overwritten by overfit_pct if overfit_pct > 0 + +.. code-block:: python + + # DEFAULT + trainer = Trainer(val_percent_check=1.0) + + # check 10% only + trainer = Trainer(val_percent_check=0.1) \ No newline at end of file diff --git a/docs/source/hooks.rst b/docs/source/hooks.rst new file mode 100644 index 00000000..fee74ea2 --- /dev/null +++ b/docs/source/hooks.rst @@ -0,0 +1,53 @@ +Hooks +======= +This is the order in which lightning calls the hooks. You can override each for custom behavior. + +Training set-up +-------------------- +- init_ddp_connection +- init_optimizers +- configure_apex +- configure_ddp +- get_train_dataloader +- get_test_dataloaders +- get_val_dataloaders +- summarize +- restore_weights + +Training loop +-------------------- + +- on_epoch_start +- on_batch_start +- tbptt_split_batch +- training_step +- training_end (optional) +- backward +- on_after_backward +- optimizer.step() +- on_batch_end +- on_epoch_end + +Validation loop +-------------------- + +- model.zero_grad() +- model.eval() +- torch.set_grad_enabled(False) +- validation_step +- validation_end +- model.train() +- torch.set_grad_enabled(True) +- on_post_performance_check + +Test loop +------------ + +- model.zero_grad() +- model.eval() +- torch.set_grad_enabled(False) +- test_step +- test_end +- model.train() +- torch.set_grad_enabled(True) +- on_post_performance_check \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 33d3c34f..ebd0fb9c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -20,7 +20,7 @@ PyTorch-Lightning Documentation callbacks lightning-module - logging + loggers trainer .. toctree:: @@ -42,7 +42,22 @@ PyTorch-Lightning Documentation :name: Common Use Cases :caption: Common Use Cases - common-cases + apex + checkpointing + slurm + debugging + experiment_logging + experiment_reporting + early_stopping + fast_training + hooks + multi_gpu + single_gpu + sequences + training_tricks + test_set + optimizers + profiler .. toctree:: :maxdepth: 1 diff --git a/docs/source/logging.rst b/docs/source/loggers.rst similarity index 78% rename from docs/source/logging.rst rename to docs/source/loggers.rst index 24f49f0a..c030e653 100644 --- a/docs/source/logging.rst +++ b/docs/source/loggers.rst @@ -1,9 +1,9 @@ .. role:: hidden :class: hidden-section -Logging +Loggers =========== -.. automodule:: pytorch_lightning.logging +.. automodule:: pytorch_lightning.loggers :exclude-members: _abc_impl, _save_model, diff --git a/docs/source/modules.rst b/docs/source/modules.rst new file mode 100644 index 00000000..e4c51218 --- /dev/null +++ b/docs/source/modules.rst @@ -0,0 +1,7 @@ +pl_examples +=========== + +.. toctree:: + :maxdepth: 4 + + pl_examples diff --git a/docs/source/multi_gpu.rst b/docs/source/multi_gpu.rst new file mode 100644 index 00000000..f44cb3f9 --- /dev/null +++ b/docs/source/multi_gpu.rst @@ -0,0 +1,69 @@ +Multi-GPU training +===================== + +Lightning supports multiple ways of doing distributed training. + +Data Parallel (dp) +------------------- +`DataParallel `_ splits a batch across k GPUs. That is, if you have a batch of 32 and use dp with 2 gpus, +each GPU will process 16 samples, after which the root node will aggregate the results. + +.. code-block:: python + + # train on 1 GPU (using dp mode) + trainer = pl.Trainer(gpus=2, distributed_backend='dp') + +Distributed Data Parallel +--------------------------- +`DistributedDataParallel `_ works as follows. + +1. Each GPU across every node gets its own process. + +2. Each GPU gets visibility into a subset of the overall dataset. It will only ever see that subset. + +3. Each process inits the model. + +.. note:: Make sure to set the random seed so that each model inits with the same weights + +4. Each process performs a full forward and backward pass in parallel. + +5. The gradients are synced and averaged across all processes. + +6. Each process updates its optimizer. + +.. code-block:: python + + # train on 8 GPUs (same machine (ie: node)) + trainer = pl.Trainer(gpus=8, distributed_backend='ddp') + + # train on 32 GPUs (4 nodes) + trainer = pl.Trainer(gpus=8, distributed_backend='ddp', num_nodes=4) + +Distributed Data Parallel 2 +----------------------------- +In certain cases, it's advantageous to use all batches on the same machine instead of a subset. +For instance you might want to compute a NCE loss where it pays to have more negative samples. + +In this case, we can use ddp2 which behaves like dp in a machine and ddp across nodes. DDP2 does the following: + +1. Copies a subset of the data to each node. + +2. Inits a model on each node. + +3. Runs a forward and backward pass using DP. + +4. Syncs gradients across nodes. + +5. Applies the optimizer updates. + +.. code-block:: python + + # train on 32 GPUs (4 nodes) + trainer = pl.Trainer(gpus=8, distributed_backend='ddp2', num_nodes=4) + + +Implement Your Own Distributed (DDP) training +---------------------------------------------- +If you need your own way to init PyTorch DDP you can override :meth:`pytorch_lightning.core.LightningModule.init_ddp_connection`. + +If you also need to use your own DDP implementation, override: :meth:`pytorch_lightning.core.LightningModule.configure_ddp`. diff --git a/docs/source/optimizers.rst b/docs/source/optimizers.rst new file mode 100644 index 00000000..2dd4c631 --- /dev/null +++ b/docs/source/optimizers.rst @@ -0,0 +1,99 @@ +Optimization +=============== + +Learning rate scheduling +------------------------------------- +Every optimizer you use can be paired with any `LearningRateScheduler `_. + +.. code-block:: python + + # no LR scheduler + def configure_optimizers(self): + return Adam(...) + + # Adam + LR scheduler + def configure_optimizers(self): + return [Adam(...)], [ReduceLROnPlateau()] + + # Two optimziers each with a scheduler + def configure_optimizers(self): + return [Adam(...), SGD(...)], [ReduceLROnPlateau(), LambdaLR()] + + +Use multiple optimizers (like GANs) +------------------------------------- +To use multiple optimizers return > 1 optimizers from :meth:`pytorch_lightning.core.LightningModule.configure_optimizers` + +.. code-block:: python + + # one optimizer + def configure_optimizers(self): + return Adam(...) + + # two optimizers, no schedulers + def configure_optimizers(self): + return Adam(...), SGD(...) + + # Two optimizers, one scheduler for adam only + def configure_optimizers(self): + return [Adam(...), SGD(...)], [ReduceLROnPlateau()] + +Lightning will call each optimizer sequentially: + +.. code-block:: python + + for epoch in epochs: + for batch in data: + for opt in optimizers: + train_step(opt) + opt.step() + + for scheduler in scheduler: + scheduler.step() + + +Step optimizers at arbitrary intervals +------------------------------------- +To do more interesting things with your optimizers such as learning rate warm-up or odd scheduling, +override the :meth:`optimizer_step' function. + +For example, here step optimizer A every 2 batches and optimizer B every 4 batches + +.. code-block:: python + + def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None): + optimizer.step() + optimizer.zero_grad() + + # Alternating schedule for optimizer steps (ie: GANs) + def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None): + # update generator opt every 2 steps + if optimizer_i == 0: + if batch_nb % 2 == 0 : + optimizer.step() + optimizer.zero_grad() + + # update discriminator opt every 4 steps + if optimizer_i == 1: + if batch_nb % 4 == 0 : + optimizer.step() + optimizer.zero_grad() + + # ... + # add as many optimizers as you want + +Here we add a learning-rate warm up + +.. code-block:: python + + # learning rate warm-up + def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None): + # warm up lr + if self.trainer.global_step < 500: + lr_scale = min(1., float(self.trainer.global_step + 1) / 500.) + for pg in optimizer.param_groups: + pg['lr'] = lr_scale * self.hparams.learning_rate + + # update params + optimizer.step() + optimizer.zero_grad() \ No newline at end of file diff --git a/docs/source/profiler.rst b/docs/source/profiler.rst index 6443e7dd..605a472f 100644 --- a/docs/source/profiler.rst +++ b/docs/source/profiler.rst @@ -2,7 +2,7 @@ :class: hidden-section -Profiling performance during training +Performance and Bottleneck Profiler =========== .. automodule:: pytorch_lightning.profiler :exclude-members: diff --git a/docs/source/sequences.rst b/docs/source/sequences.rst new file mode 100644 index 00000000..3e0e064b --- /dev/null +++ b/docs/source/sequences.rst @@ -0,0 +1,45 @@ +Sequential Data +================ +Lightning has built in support for dealing with sequential data. + + +Packed sequences as inputs +---------------------------- +When using PackedSequence, do 2 things: + +1. return either a padded tensor in dataset or a list of variable length tensors in the dataloader collate_fn (example above shows the list implementation). +2. Pack the sequence in forward or training and validation steps depending on use case. + +.. code-block:: python + + # For use in dataloader + def collate_fn(batch): + x = [item[0] for item in batch] + y = [item[1] for item in batch] + return x, y + + # In module + def training_step(self, batch, batch_nb): + x = rnn.pack_sequence(batch[0], enforce_sorted=False) + y = rnn.pack_sequence(batch[1], enforce_sorted=False) + +Truncated Backpropagation Through Time +--------------------------------------- +There are times when multiple backwards passes are needed for each batch. +For example, it may save memory to use Truncated Backpropagation Through Time when training RNNs. + +Lightning can handle TBTT automatically via this flag. + +.. code-block:: python + + # DEFAULT (single backwards pass per batch) + trainer = Trainer(truncated_bptt_steps=None) + + # (split batch into sequences of size 2) + trainer = Trainer(truncated_bptt_steps=2) + +.. note:: If you need to modify how the batch is split, + override :meth:`pytorch_lightning.core.LightningModule.tbptt_split_batch`. + +.. note:: Using this feature requires updating your LightningModule's :meth:`pytorch_lightning.core.LightningModule.training_step` to include + a `hiddens` arg. \ No newline at end of file diff --git a/docs/source/single_gpu.rst b/docs/source/single_gpu.rst new file mode 100644 index 00000000..73908489 --- /dev/null +++ b/docs/source/single_gpu.rst @@ -0,0 +1,9 @@ +Single GPU Training +==================== +Make sure you are running on a machine that has at least one GPU. Lightning handles all the NVIDIA flags for you, +there's no need to set them yourself. + +.. code-block:: python + + # train on 1 GPU (using dp mode) + trainer = pl.Trainer(gpus=1) \ No newline at end of file diff --git a/docs/source/slurm.rst b/docs/source/slurm.rst new file mode 100644 index 00000000..57d2a6b3 --- /dev/null +++ b/docs/source/slurm.rst @@ -0,0 +1,91 @@ +Computing cluster (SLURM) +========================== + +Lightning automates job the details behind training on a SLURM powered cluster. + +.. _multi-node: + +Multi-node training +-------------------- +To train a model using multiple-nodes do the following: + +1. Design your LightningModule. + +2. Add `torch.DistributedSampler `_ + which enables access to a subset of your full dataset to each GPU. + +3. Enable ddp in the trainer + +.. code-block:: python + + # train on 32 GPUs across 4 nodes + trainer = Trainer(gpus=8, num_nodes=4, distributed_backend='ddp') + +4. It's a good idea to structure your train.py file like this: + +.. code-block:: python + + # train.py + def main(hparams): + model = LightningTemplateModel(hparams) + + trainer = pl.Trainer( + gpus=8, + num_nodes=4, + distributed_backend='ddp' + ) + + trainer.fit(model) + + + if __name__ == '__main__': + root_dir = os.path.dirname(os.path.realpath(__file__)) + parent_parser = ArgumentParser(add_help=False) + hyperparams = parser.parse_args() + + # TRAIN + main(hyperparams) + +4. Submit the appropriate SLURM job + +.. code-block:: bash + + #!/bin/bash -l + + # SLURM SUBMIT SCRIPT + #SBATCH --nodes=4 + #SBATCH --gres=gpu:8 + #SBATCH --ntasks-per-node=8 + #SBATCH --mem=0 + #SBATCH --time=0-02:00:00 + + # activate conda env + source activate $1 + + # ------------------------- + # debugging flags (optional) + export NCCL_DEBUG=INFO + export PYTHONFAULTHANDLER=1 + + # on your cluster you might need these: + # set the network interface + # export NCCL_SOCKET_IFNAME=^docker0,lo + + # might need the latest cuda + # module load NCCL/2.4.7-1-cuda.10.0 + # ------------------------- + + # run script from above + srun python3 train.py + + +Walltime auto-resubmit +----------------------------------- +When you use Lightning in a SLURM cluster, lightning automatically detects when it is about +to run into the walltime, and it does the following: + +1. Saves a temporary checkpoint. +2. Requeues the job. +3. When the job starts, it loads the temporary checkpoint. + +.. note:: To get this behavior you have to do nothing. diff --git a/docs/source/test_set.rst b/docs/source/test_set.rst new file mode 100644 index 00000000..abbf48fd --- /dev/null +++ b/docs/source/test_set.rst @@ -0,0 +1,39 @@ +Test set +========== +Lightning forces the user to run the test set separately to make sure it isn't evaluated by mistake + + +Test after fit +---------------- +To run the test set after training completes, use this method + +.. code-block:: python + + # run full training + trainer.fit(model) + + # run test set + trainer.test() + + +Test pre-trained model +----------------- +To run the test set on a pretrained model, use this method. + +.. code-block:: python + + model = MyLightningModule.load_from_metrics( + weights_path='/path/to/pytorch_checkpoint.ckpt', + tags_csv='/path/to/test_tube/experiment/version/meta_tags.csv', + on_gpu=True, + map_location=None + ) + + # init trainer with whatever options + trainer = Trainer(...) + + # test (pass in the model) + trainer.test(model) + +In this case, the options you pass to trainer will be used when +running the test set (ie: 16-bit, dp, ddp, etc... \ No newline at end of file diff --git a/docs/source/training_tricks.rst b/docs/source/training_tricks.rst new file mode 100644 index 00000000..416fcdb8 --- /dev/null +++ b/docs/source/training_tricks.rst @@ -0,0 +1,31 @@ +Training Tricks +================ +Lightning implements various tricks to help during training + +Accumulate gradients +------------------------------------- +Accumulated gradients runs K small batches of size N before doing a backwards pass. +The effect is a large effective batch size of size KxN. + +.. note:: See: :ref:`trainer` + +.. code-block:: python + + # DEFAULT (ie: no accumulated grads) + trainer = Trainer(accumulate_grad_batches=1) + + +Gradient Clipping +------------------------------------- +Gradient clipping may be enabled to avoid exploding gradients. Specifically, this will `clip the gradient +norm `_ computed over all model parameters together. + +.. note:: See: :ref:`trainer` + +.. code-block:: python + + # DEFAULT (ie: don't clip) + trainer = Trainer(gradient_clip_val=0) + + # clip gradients with norm above 0.5 + trainer = Trainer(gradient_clip_val=0.5) diff --git a/pytorch_lightning/core/lightning.py b/pytorch_lightning/core/lightning.py index 837fea0b..a0df8dc1 100644 --- a/pytorch_lightning/core/lightning.py +++ b/pytorch_lightning/core/lightning.py @@ -110,10 +110,17 @@ class LightningModule(ABC, GradInformation, ModelIO, ModelHooks): @abstractmethod def training_step(self, *args, **kwargs): - """return loss, dict with metrics for tqdm + r"""return loss, dict with metrics for tqdm + + Args: + batch (torch.nn.Tensor | (Tensor, Tensor) | [Tensor, Tensor]): The output of your dataloader. + A tensor, tuple or list + batch_idx (int): Integer displaying index of this batch + optimizer_idx (int): If using multiple optimizers, this argument will also be present. + hiddens(:`Tensor `_): Passed in if truncated_bptt_steps > 0. + + :param - :param batch: The output of your dataloader. A tensor, tuple or list - :param int batch_idx: Integer displaying which batch this is :return: dict with loss key and optional log, progress keys if implementing training_step, return whatever you need in that step: diff --git a/pytorch_lightning/loggers/comet.py b/pytorch_lightning/loggers/comet.py index 1b5950c1..b3d59eb1 100644 --- a/pytorch_lightning/loggers/comet.py +++ b/pytorch_lightning/loggers/comet.py @@ -1,3 +1,11 @@ +r""" + +.. _comet: + +CometLogger +------------- +""" + from logging import getLogger try: @@ -20,19 +28,19 @@ logger = getLogger(__name__) class CometLogger(LightningLoggerBase): + r""" + Log using `comet.ml `_. + """ def __init__(self, api_key=None, save_dir=None, workspace=None, rest_api_key=None, project_name=None, experiment_name=None, **kwargs): r""" - Log using `comet `_. - Requires either an API Key (online mode) or a local directory path (offline mode) .. code-block:: python # ONLINE MODE from pytorch_lightning.loggers import CometLogger - # arguments made to CometLogger are passed on to the comet_ml.Experiment class comet_logger = CometLogger( api_key=os.environ["COMET_KEY"], @@ -43,12 +51,10 @@ class CometLogger(LightningLoggerBase): ) trainer = Trainer(logger=comet_logger) - .. code-block:: python # OFFLINE MODE from pytorch_lightning.loggers import CometLogger - # arguments made to CometLogger are passed on to the comet_ml.Experiment class comet_logger = CometLogger( save_dir=".", diff --git a/pytorch_lightning/loggers/neptune.py b/pytorch_lightning/loggers/neptune.py index 066ad915..6659bcff 100644 --- a/pytorch_lightning/loggers/neptune.py +++ b/pytorch_lightning/loggers/neptune.py @@ -1,41 +1,10 @@ """ -Log using `neptune `_ - -Neptune logger can be used in the online mode or offline (silent) mode. -To log experiment data in online mode, NeptuneLogger requries an API key: - -.. code-block:: python - - from pytorch_lightning.loggers import NeptuneLogger - # arguments made to NeptuneLogger are passed on to the neptune.experiments.Experiment class - - neptune_logger = NeptuneLogger( - api_key=os.environ["NEPTUNE_API_TOKEN"], - project_name="USER_NAME/PROJECT_NAME", - experiment_name="default", # Optional, - params={"max_epochs": 10}, # Optional, - tags=["pytorch-lightning","mlp"] # Optional, - ) - trainer = Trainer(max_epochs=10, logger=neptune_logger) - -Use the logger anywhere in you LightningModule as follows: - -.. code-block:: python - - def train_step(...): - # example - self.logger.experiment.log_metric("acc_train", acc_train) # log metrics - self.logger.experiment.log_image("worse_predictions", prediction_image) # log images - self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint - self.logger.experiment.whatever_neptune_supports(...) - - def any_lightning_module_function_or_hook(...): - self.logger.experiment.log_metric("acc_train", acc_train) # log metrics - self.logger.experiment.log_image("worse_predictions", prediction_image) # log images - self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint - self.logger.experiment.whatever_neptune_supports(...) +Log using `neptune-logger `_ +.. _neptune: +NeptuneLogger +-------------- """ from logging import getLogger @@ -54,6 +23,10 @@ logger = getLogger(__name__) class NeptuneLogger(LightningLoggerBase): + r""" + Neptune logger can be used in the online mode or offline (silent) mode. + To log experiment data in online mode, NeptuneLogger requries an API key: + """ def __init__(self, api_key=None, project_name=None, offline_mode=False, experiment_name=None, upload_source_files=None, params=None, properties=None, tags=None, **kwargs): @@ -92,6 +65,23 @@ class NeptuneLogger(LightningLoggerBase): ) trainer = Trainer(max_epochs=10, logger=neptune_logger) + Use the logger anywhere in you LightningModule as follows: + + .. code-block:: python + + def train_step(...): + # example + self.logger.experiment.log_metric("acc_train", acc_train) # log metrics + self.logger.experiment.log_image("worse_predictions", prediction_image) # log images + self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint + self.logger.experiment.whatever_neptune_supports(...) + + def any_lightning_module_function_or_hook(...): + self.logger.experiment.log_metric("acc_train", acc_train) # log metrics + self.logger.experiment.log_image("worse_predictions", prediction_image) # log images + self.logger.experiment.log_artifact("model_checkpoint.pt", prediction_image) # log model checkpoint + self.logger.experiment.whatever_neptune_supports(...) + Args: api_key (str | None): Required in online mode. Neputne API token, found on https://neptune.ml. Read how to get your API key diff --git a/pytorch_lightning/loggers/tensorboard.py b/pytorch_lightning/loggers/tensorboard.py index e22a5c21..d7222ee8 100644 --- a/pytorch_lightning/loggers/tensorboard.py +++ b/pytorch_lightning/loggers/tensorboard.py @@ -18,8 +18,10 @@ class TensorBoardLogger(LightningLoggerBase): Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to `os.path.join(save_dir, name, version)` + .. _tf-logger: + Example - -------- + ------------------ .. code-block:: python diff --git a/pytorch_lightning/loggers/test_tube.py b/pytorch_lightning/loggers/test_tube.py index 30509c7d..9247efbc 100644 --- a/pytorch_lightning/loggers/test_tube.py +++ b/pytorch_lightning/loggers/test_tube.py @@ -1,34 +1,3 @@ -""" -Log using `test tube '_. Test tube logger is -a strict subclass of `PyTorch SummaryWriter `_, refer to their -documentation for all supported operations. The TestTubeLogger adds a nicer folder structure -to manage experiments and snapshots all hyperparameters you pass to a LightningModule. - -.. code-block:: python - - from pytorch_lightning.loggers import TestTubeLogger - tt_logger = TestTubeLogger( - save_dir=".", - name="default", - debug=False, - create_git_tag=False - ) - trainer = Trainer(logger=tt_logger) - - -Use the logger anywhere in you LightningModule as follows: - -.. code-block:: python - - def train_step(...): - # example - self.logger.experiment.whatever_method_summary_writer_supports(...) - - def any_lightning_module_function_or_hook(...): - self.logger.experiment.add_histogram(...) - -""" - try: from test_tube import Experiment except ImportError: @@ -39,30 +8,8 @@ from .base import LightningLoggerBase, rank_zero_only class TestTubeLogger(LightningLoggerBase): r""" - Log to local file system in TensorBoard format but using a nicer folder structure. - - Implemented using :class:`torch.utils.tensorboard.SummaryWriter`. Logs are saved to - `os.path.join(save_dir, name, version)` - - Example - ------- - - .. code-block:: python - - logger = TestTubeLogger("tt_logs", name="my_exp_name") - trainer = Trainer(logger=logger) - trainer.train(model) - - Args: - save_dir (str): Save directory - name (str): Experiment name. Defaults to "default". - description (str): A short snippet about this experiment - debug (bool): If True, it doesn't log anything - version (int): Experiment version. If version is not specified the logger inspects the save - directory for existing versions, then automatically assigns the next available version. - create_git_tag (bool): If True creates a git tag to save the code used in this experiment - + (see `full docs `_). """ __test__ = False @@ -71,6 +18,40 @@ class TestTubeLogger(LightningLoggerBase): self, save_dir, name="default", description=None, debug=False, version=None, create_git_tag=False ): + r""" + + .. _testTube: + + Example + ---------- + + .. code-block:: python + + logger = TestTubeLogger("tt_logs", name="my_exp_name") + trainer = Trainer(logger=logger) + trainer.train(model) + + Use the logger anywhere in you LightningModule as follows: + + .. code-block:: python + + def train_step(...): + # example + self.logger.experiment.whatever_method_summary_writer_supports(...) + + def any_lightning_module_function_or_hook(...): + self.logger.experiment.add_histogram(...) + + Args: + save_dir (str): Save directory + name (str): Experiment name. Defaults to "default". + description (str): A short snippet about this experiment + debug (bool): If True, it doesn't log anything + version (int): Experiment version. If version is not specified the logger inspects the save + directory for existing versions, then automatically assigns the next available version. + create_git_tag (bool): If True creates a git tag to save the code used in this experiment + + """ super().__init__() self.save_dir = save_dir self._name = name diff --git a/pytorch_lightning/loggers/wandb.py b/pytorch_lightning/loggers/wandb.py index e728d554..c55f1e6c 100644 --- a/pytorch_lightning/loggers/wandb.py +++ b/pytorch_lightning/loggers/wandb.py @@ -1,3 +1,11 @@ +r""" + +.. _wandb: + +WandbLogger +------------- +""" + import os try: @@ -10,7 +18,7 @@ from .base import LightningLoggerBase, rank_zero_only class WandbLogger(LightningLoggerBase): """ - Logger for W&B. + Logger for `W&B `_. Args: name (str): display name for the run. diff --git a/pytorch_lightning/profiler/__init__.py b/pytorch_lightning/profiler/__init__.py index a69e3ccf..0341f07f 100644 --- a/pytorch_lightning/profiler/__init__.py +++ b/pytorch_lightning/profiler/__init__.py @@ -1,6 +1,10 @@ """ Profiling your training run can help you understand if there are any bottlenecks in your code. + +Built-in checks +---------------- + PyTorch Lightning supports profiling standard actions in the training loop out of the box, including: - on_epoch_start @@ -15,6 +19,9 @@ PyTorch Lightning supports profiling standard actions in the training loop out o - training_end - on_training_end +Enable simple profiling +------------------------- + If you only wish to profile the standard actions, you can set `profiler=True` when constructing your `Trainer` object. @@ -42,6 +49,9 @@ The profiler's results will be printed at the completion of a training `fit()`. on_train_end | 5.449e-06 | 5.449e-06 +Advanced Profiling +-------------------- + If you want more information on the functions called during each event, you can use the `AdvancedProfiler`. This option uses Python's cProfiler_ to provide a report of time spent on *each* function called within your code. @@ -67,10 +77,10 @@ of logging it to the output in your terminal. The output below shows the profili List reduced from 76 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 3752/1876 0.011 0.000 18.887 0.010 {built-in method builtins.next} - 1876 0.008 0.000 18.877 0.010 dataloader.py:344(__next__) - 1876 0.074 0.000 18.869 0.010 dataloader.py:383(_next_data) - 1875 0.012 0.000 18.721 0.010 fetch.py:42(fetch) - 1875 0.084 0.000 18.290 0.010 fetch.py:44() + 1876 0.008 0.000 18.877 0.010 dataloader.py:344(__next__) + 1876 0.074 0.000 18.869 0.010 dataloader.py:383(_next_data) + 1875 0.012 0.000 18.721 0.010 fetch.py:42(fetch) + 1875 0.084 0.000 18.290 0.010 fetch.py:44() 60000 1.759 0.000 18.206 0.000 mnist.py:80(__getitem__) 60000 0.267 0.000 13.022 0.000 transforms.py:68(__call__) 60000 0.182 0.000 7.020 0.000 transforms.py:93(__call__) diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py index 9402d925..a92c4d4e 100644 --- a/pytorch_lightning/trainer/trainer.py +++ b/pytorch_lightning/trainer/trainer.py @@ -452,9 +452,14 @@ class Trainer(TrainerIOMixin, # backprop every 5 steps in a batch trainer = Trainer(truncated_bptt_steps=5) - Using this feature requires updating your LightningModule's `training_step()` to include - a `hiddens` arg. + Lightning takes care to split your batch along the time-dimension. + + .. note:: If you need to modify how the batch is split, + override :meth:`pytorch_lightning.core.LightningModule.tbptt_split_batch`. + + .. note:: Using this feature requires updating your LightningModule's + :meth:`pytorch_lightning.core.LightningModule.training_step` to include a `hiddens` arg. resume_from_checkpoint (str): To resume training from a specific checkpoint pass in the path here.k Example::