From 6ea3cc326f6503be039bf1f795fadc4922660421 Mon Sep 17 00:00:00 2001
From: William Falcon <falc@fb.com>
Date: Thu, 27 Jun 2019 14:22:00 -0400
Subject: [PATCH] debugging and gpu guide

---
 docs/Pytorch-Lightning/LightningModule.md |  2 +-
 docs/Trainer/Distributed training.md      | 55 +++++++++++++++++++++++
 docs/Trainer/Logging.md                   | 35 +++++++++------
 docs/Trainer/index.md                     | 18 ++++----
 4 files changed, 88 insertions(+), 22 deletions(-)

diff --git a/docs/Pytorch-Lightning/LightningModule.md b/docs/Pytorch-Lightning/LightningModule.md
index e0c80526..6ce33343 100644
--- a/docs/Pytorch-Lightning/LightningModule.md
+++ b/docs/Pytorch-Lightning/LightningModule.md
@@ -28,7 +28,7 @@ Otherwise, to Define a Lightning Module, implement the following methods:
 
 ---
 
-### training_step
+### training_ste**p
 
 ``` {.python}
 def training_step(self, data_batch, batch_nb)
diff --git a/docs/Trainer/Distributed training.md b/docs/Trainer/Distributed training.md
index e69de29b..dcd8a422 100644
--- a/docs/Trainer/Distributed training.md	
+++ b/docs/Trainer/Distributed training.md	
@@ -0,0 +1,55 @@
+Lightning makes multi-gpu training and 16 bit training trivial.
+
+*Note:*   
+None of the flags below require changing anything about your lightningModel definition. 
+
+---
+#### 16-bit mixed precision
+16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.    
+First, install apex (if install fails, look [here](https://github.com/NVIDIA/apex)):
+```bash
+$ git clone https://github.com/NVIDIA/apex
+$ cd apex
+$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
+```
+
+then set this use_amp to True.
+``` {.python}
+# DEFAULT
+trainer = Trainer(amp_level='O2', use_amp=False)
+```
+
+---
+#### Single-gpu
+Make sure you're on a GPU machine. 
+```python
+# set these flags
+os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+os.environ["CUDA_VISIBLE_DEVICES"] = "0"
+
+# DEFAULT
+trainer = Trainer(gpus=[0])
+```
+
+---
+#### multi-gpu 
+Make sure you're on a GPU machine. You can set as many GPUs as you want.
+In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.
+```python
+# set these flags
+os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7"
+
+# DEFAULT
+trainer = Trainer(gpus=[0,1,2,3,4,5,6,7])
+```
+
+---
+#### Multi-node
+COMING SOON.
+
+---
+#### Self-balancing architecture
+Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.   
+
+COMING SOON.
diff --git a/docs/Trainer/Logging.md b/docs/Trainer/Logging.md
index 1f57439a..1596251f 100644
--- a/docs/Trainer/Logging.md
+++ b/docs/Trainer/Logging.md
@@ -1,4 +1,4 @@
-
+Lighting offers a few options for logging information about model, gpu usage, etc (via test-tube). It also offers printing options for training monitoring.
 
 
 ---
@@ -8,6 +8,13 @@
 trainer = Trainer(progress_bar=True)
 ```
 
+---
+#### Log metric row every k batches 
+Every k batches lightning will make an entry in the metrics log
+``` {.python}
+# DEFAULT (ie: save a .csv log file every 10 batches)
+trainer = Trainer(add_log_row_interval=10)
+```
 
 ---
 #### Process position
@@ -21,27 +28,29 @@ trainer = Trainer(process_position=0)
 trainer = Trainer(process_position=1)
 ```
 
----
-#### Print which gradients are nan 
-This option prints a list of tensors with nan gradients.
-``` {.python}
-# DEFAULT
-trainer = Trainer(print_nan_grads=False)
-```
-
 ---
 #### Save a snapshot of all hyperparameters 
 Whenever you call .save() on the test-tube experiment it logs all the hyperparameters in current use.
 Give lightning a test-tube Experiment object to automate this for you.
+``` {.python}
+from test-tube import Experiment
+
+exp = Experiment(...)
+Trainer(experiment=exp)
+```
 
 ---
-#### Log metric row every k batches 
-Every k batches lightning will make an entry in the metrics log
+#### Snapshot code for a training run
+Whenever you call .save() on the test-tube experiment it snapshows all code and pushes to a git tag.
+Give lightning a test-tube Experiment object to automate this for you.
 ``` {.python}
-# DEFAULT (ie: save a .csv log file every 100 batches)
-trainer = Trainer(add_log_row_interval=10)
+from test-tube import Experiment
+
+exp = Experiment(create_git_tag=True)
+Trainer(experiment=exp)
 ```
 
+
 ---
 #### Write logs file to csv every k batches 
 Every k batches, lightning will write the new logs to disk
diff --git a/docs/Trainer/index.md b/docs/Trainer/index.md
index f30684db..38143149 100644
--- a/docs/Trainer/index.md
+++ b/docs/Trainer/index.md
@@ -41,20 +41,22 @@ But of course the fun is in all the advanced things it can do:
 
 **Distributed training**    
 
-- 16-bit mixed precision
-- Single-gpu      
-- Multi-gpu      
-- Multi-node   
+- [16-bit mixed precision](Distributed%20training/#16-bit-mixed-precision)
+- [Multi-GPU](Distributed%20training/#Multi-GPU)
+- [Multi-node](Distributed%20training/#Multi-node)
+- [Single GPU](Distributed%20training/#single-gpu)
+- [Self-balancing architecture](Distributed%20training/#self-balancing-architecture)
+
 
 **Experiment Logging**   
 
 - [Display metrics in progress bar](Logging/#display-metrics-in-progress-bar)
 - Log arbitrary metrics
-- [Process position](Logging/#process-position)
-- [Write logs file to csv every k batches](Logging/#write-logs-file-to-csv-every-k-batches)
 - [Log metric row every k batches](Logging/#log-metric-row-every-k-batches)
-- Save a snapshot of all hyperparameters
-- Save a snapshot of the code for a particular model run
+- [Process position](Logging/#process-position)
+- [Save a snapshot of all hyperparameters](Logging/#save-a-snapshot-of-all-hyperparameters) 
+- [Snapshot code for a training run](Logging/#snapshot-code-for-a-training-run) 
+- [Write logs file to csv every k batches](Logging/#write-logs-file-to-csv-every-k-batches)
 
 **Training loop**