Docs (#813)

* added outline of all features * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated common use cases doc * updated docs
2026-09-12 12:40:20 +08:00 · 2020-02-10 23:55:22 -05:00
parent af44583050
commit 4c6c3d04ce
28 changed files with 1163 additions and 137 deletions
@@ -0,0 +1,31 @@
+Training Tricks
+================
+Lightning implements various tricks to help during training
+
+Accumulate gradients
+-------------------------------------
+Accumulated gradients runs K small batches of size N before doing a backwards pass.
+The effect is a large effective batch size of size KxN.
+
+.. note:: See: :ref:`trainer`
+
+.. code-block:: python
+
+    # DEFAULT (ie: no accumulated grads)
+    trainer = Trainer(accumulate_grad_batches=1)
+
+
+Gradient Clipping
+-------------------------------------
+Gradient clipping may be enabled to avoid exploding gradients. Specifically, this will `clip the gradient
+norm <https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_>`_ computed over all model parameters together.
+
+.. note:: See: :ref:`trainer`
+
+.. code-block:: python
+
+    # DEFAULT (ie: don't clip)
+    trainer = Trainer(gradient_clip_val=0)
+
+    # clip gradients with norm above 0.5
+    trainer = Trainer(gradient_clip_val=0.5)