Deployed 6ea3cc3 with MkDocs version: 1.0.4

Pytorch-Lightning/LightningModule/index.html

@@ -66,7 +66,7 @@
     
         <ul>
         
-            <li><a class="toctree-l4" href="#training_step">training_step</a></li>
+            <li><a class="toctree-l4" href="#training_step">training_ste**p</a></li>
         
             <li><a class="toctree-l4" href="#validation_step">validation_step</a></li>
         
@@ -209,7 +209,7 @@
 <li><a href="./#add_model_specific_args">add_model_specific_args</a></li>
 </ul>
 <hr />
-<h3 id="training_step">training_step</h3>
+<h3 id="training_step">training_ste**p</h3>
 <pre><code class="python">def training_step(self, data_batch, batch_nb)
 </code></pre>
 

Trainer/Distributed training/index.html

@@ -81,6 +81,21 @@
     <a class="current" href="./">Distributed training</a>
     <ul class="subnav">
             
+    <li class="toctree-l3"><a href="#16-bit-mixed-precision">16-bit mixed precision</a></li>
+    
+
+    <li class="toctree-l3"><a href="#single-gpu">Single-gpu</a></li>
+    
+
+    <li class="toctree-l3"><a href="#multi-gpu">multi-gpu</a></li>
+    
+
+    <li class="toctree-l3"><a href="#multi-node">Multi-node</a></li>
+    
+
+    <li class="toctree-l3"><a href="#self-balancing-architecture">Self-balancing architecture</a></li>
+    
+
     </ul>
                 </li>
                 <li class="">
@@ -149,7 +164,53 @@
           <div role="main">
             <div class="section">
               
-                
+                <p>Lightning makes multi-gpu training and 16 bit training trivial.</p>
+<p><em>Note:</em> <br />
+None of the flags below require changing anything about your lightningModel definition. </p>
+<hr />
+<h4 id="16-bit-mixed-precision">16-bit mixed precision</h4>
+<p>16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.  <br />
+First, install apex (if install fails, look <a href="https://github.com/NVIDIA/apex">here</a>):</p>
+<pre><code class="bash">$ git clone https://github.com/NVIDIA/apex
+$ cd apex
+$ pip install -v --no-cache-dir --global-option=&quot;--cpp_ext&quot; --global-option=&quot;--cuda_ext&quot; ./
+</code></pre>
+
+<p>then set this use_amp to True.</p>
+<pre><code class="python"># DEFAULT
+trainer = Trainer(amp_level='O2', use_amp=False)
+</code></pre>
+
+<hr />
+<h4 id="single-gpu">Single-gpu</h4>
+<p>Make sure you're on a GPU machine. </p>
+<pre><code class="python"># set these flags
+os.environ[&quot;CUDA_DEVICE_ORDER&quot;] = &quot;PCI_BUS_ID&quot;
+os.environ[&quot;CUDA_VISIBLE_DEVICES&quot;] = &quot;0&quot;
+
+# DEFAULT
+trainer = Trainer(gpus=[0])
+</code></pre>
+
+<hr />
+<h4 id="multi-gpu">multi-gpu</h4>
+<p>Make sure you're on a GPU machine. You can set as many GPUs as you want.
+In this setting, the model will run on all 8 GPUs at once using DataParallel under the hood.</p>
+<pre><code class="python"># set these flags
+os.environ[&quot;CUDA_DEVICE_ORDER&quot;] = &quot;PCI_BUS_ID&quot;
+os.environ[&quot;CUDA_VISIBLE_DEVICES&quot;] = &quot;0,1,2,3,4,5,6,7&quot;
+
+# DEFAULT
+trainer = Trainer(gpus=[0,1,2,3,4,5,6,7])
+</code></pre>
+
+<hr />
+<h4 id="multi-node">Multi-node</h4>
+<p>COMING SOON.</p>
+<hr />
+<h4 id="self-balancing-architecture">Self-balancing architecture</h4>
+<p>Here lightning distributes parts of your module across available GPUs to optimize for speed and memory.   </p>
+<p>COMING SOON.</p>
               
             </div>
           </div>

Trainer/Logging/index.html

@@ -88,12 +88,21 @@
     <li class="toctree-l3"><a href="#display-metrics-in-progress-bar">Display metrics in progress bar</a></li>
     
 
-    <li class="toctree-l3"><a href="#print-which-gradients-are-nan">Print which gradients are nan</a></li>
+    <li class="toctree-l3"><a href="#log-metric-row-every-k-batches">Log metric row every k batches</a></li>
     
 
     <li class="toctree-l3"><a href="#process-position">Process position</a></li>
     
 
+    <li class="toctree-l3"><a href="#save-a-snapshot-of-all-hyperparameters">Save a snapshot of all hyperparameters</a></li>
+    
+
+    <li class="toctree-l3"><a href="#snapshot-code-for-a-training-run">Snapshot code for a training run</a></li>
+    
+
+    <li class="toctree-l3"><a href="#write-logs-file-to-csv-every-k-batches">Write logs file to csv every k batches</a></li>
+    
+
     </ul>
                 </li>
                 <li class="">
@@ -158,17 +167,18 @@
           <div role="main">
             <div class="section">
               
-                <hr />
+                <p>Lighting offers a few options for logging information about model, gpu usage, etc (via test-tube). It also offers printing options for training monitoring.</p>
+<hr />
 <h4 id="display-metrics-in-progress-bar">Display metrics in progress bar</h4>
 <pre><code class="python"># DEFAULT
 trainer = Trainer(progress_bar=True)
 </code></pre>
 
 <hr />
-<h4 id="print-which-gradients-are-nan">Print which gradients are nan</h4>
-<p>This option prints a list of tensors with nan gradients.</p>
-<pre><code class="python"># DEFAULT
-trainer = Trainer(print_nan_grads=False)
+<h4 id="log-metric-row-every-k-batches">Log metric row every k batches</h4>
+<p>Every k batches lightning will make an entry in the metrics log</p>
+<pre><code class="python"># DEFAULT (ie: save a .csv log file every 10 batches)
+trainer = Trainer(add_log_row_interval=10)
 </code></pre>
 
 <hr />
@@ -180,6 +190,33 @@ trainer = Trainer(process_position=0)
 
 # if this is the second model on the node, show the second progress bar below
 trainer = Trainer(process_position=1)
+</code></pre>
+
+<hr />
+<h4 id="save-a-snapshot-of-all-hyperparameters">Save a snapshot of all hyperparameters</h4>
+<p>Whenever you call .save() on the test-tube experiment it logs all the hyperparameters in current use.
+Give lightning a test-tube Experiment object to automate this for you.</p>
+<pre><code class="python">from test-tube import Experiment
+
+exp = Experiment(...)
+Trainer(experiment=exp)
+</code></pre>
+
+<hr />
+<h4 id="snapshot-code-for-a-training-run">Snapshot code for a training run</h4>
+<p>Whenever you call .save() on the test-tube experiment it snapshows all code and pushes to a git tag.
+Give lightning a test-tube Experiment object to automate this for you.</p>
+<pre><code class="python">from test-tube import Experiment
+
+exp = Experiment(create_git_tag=True)
+Trainer(experiment=exp)
+</code></pre>
+
+<hr />
+<h4 id="write-logs-file-to-csv-every-k-batches">Write logs file to csv every k batches</h4>
+<p>Every k batches, lightning will write the new logs to disk</p>
+<pre><code class="python"># DEFAULT (ie: save a .csv log file every 100 batches)
+trainer = Trainer(log_save_interval=100)
 </code></pre>
               
             </div>

Trainer/index.html

@@ -165,6 +165,45 @@ trainer.fit(model)
 </code></pre>
 
 <p>But of course the fun is in all the advanced things it can do:</p>
+<p><strong>Checkpointing</strong>    </p>
+<ul>
+<li>Model saving</li>
+<li>Model loading </li>
+</ul>
+<p><strong>Computing cluster (SLURM)</strong>    </p>
+<ul>
+<li>Automatic checkpointing   </li>
+<li>Automatic saving, loading  </li>
+<li>Running grid search on a cluster </li>
+<li>Walltime auto-resubmit   </li>
+</ul>
+<p><strong>Debugging</strong>  </p>
+<ul>
+<li><a href="Debugging/#fast-dev-run">Fast dev run</a></li>
+<li><a href="Debugging/#inspect-gradient-norms">Inspect gradient norms</a></li>
+<li><a href="Debugging/#Log-gpu-usage">Log GPU usage</a></li>
+<li><a href="Debugging/#make-model-overfit-on-subset-of-data">Make model overfit on subset of data</a></li>
+<li><a href="Debugging/#print-the-parameter-count-by-layer">Print the parameter count by layer</a></li>
+<li><a href="Debugging/#print-which-gradients-are-nan">Pring which gradients are nan</a></li>
+</ul>
+<p><strong>Distributed training</strong>    </p>
+<ul>
+<li><a href="Distributed%20training/#16-bit-mixed-precision">16-bit mixed precision</a></li>
+<li><a href="Distributed%20training/#Multi-GPU">Multi-GPU</a></li>
+<li><a href="Distributed%20training/#Multi-node">Multi-node</a></li>
+<li><a href="Distributed%20training/#single-gpu">Single GPU</a></li>
+<li><a href="Distributed%20training/#self-balancing-architecture">Self-balancing architecture</a></li>
+</ul>
+<p><strong>Experiment Logging</strong>   </p>
+<ul>
+<li><a href="Logging/#display-metrics-in-progress-bar">Display metrics in progress bar</a></li>
+<li>Log arbitrary metrics</li>
+<li><a href="Logging/#log-metric-row-every-k-batches">Log metric row every k batches</a></li>
+<li><a href="Logging/#process-position">Process position</a></li>
+<li><a href="Logging/#save-a-snapshot-of-all-hyperparameters">Save a snapshot of all hyperparameters</a> </li>
+<li><a href="Logging/#snapshot-code-for-a-training-run">Snapshot code for a training run</a> </li>
+<li><a href="Logging/#write-logs-file-to-csv-every-k-batches">Write logs file to csv every k batches</a></li>
+</ul>
 <p><strong>Training loop</strong>    </p>
 <ul>
 <li><a href="Training%20Loop/#accumulated-gradients">Accumulate gradients</a></li>
@@ -181,42 +220,6 @@ trainer.fit(model)
 <li><a href="Validation%20Loop/#set-how-much-of-the-test-set-to-check">Set how much of the test set to check</a></li>
 <li><a href="Validation%20Loop/#set-validation-check-frequency-within-1-training-epoch">Set validation check frequency within 1 training epoch</a></li>
 <li><a href="Validation%20Loop/#set-the-number-of-validation-sanity-steps">Set the number of validation sanity steps</a></li>
-</ul>
-<p><strong>Debugging</strong>  </p>
-<ul>
-<li><a href="Debugging/#fast-dev-run">Fast dev run</a></li>
-<li><a href="Debugging/#inspect-gradient-norms">Inspect gradient norms</a></li>
-<li><a href="Debugging/#Log-gpu-usage">Log GPU usage</a></li>
-<li><a href="Debugging/#make-model-overfit-on-subset-of-data">Make model overfit on subset of data</a></li>
-<li><a href="Debugging/#print-the-parameter-count-by-layer">Print the parameter count by layer</a></li>
-<li><a href="Debugging/#print-which-gradients-are-nan">Pring which gradients are nan</a></li>
-</ul>
-<p><strong>Experiment Logging</strong>   </p>
-<ul>
-<li><a href="Logging/#display-metrics-in-progress-bar">Display metrics in progress bar</a></li>
-<li>Log arbitrary metrics</li>
-<li><a href="Logging/#process-position">Process position</a></li>
-<li>Save a snapshot of all hyperparameters</li>
-<li>Save a snapshot of the code for a particular model run</li>
-</ul>
-<p><strong>Distributed training</strong>    </p>
-<ul>
-<li>16-bit mixed precision</li>
-<li>Single-gpu      </li>
-<li>Multi-gpu      </li>
-<li>Multi-node   </li>
-</ul>
-<p><strong>Checkpointing</strong>    </p>
-<ul>
-<li>Model saving</li>
-<li>Model loading </li>
-</ul>
-<p><strong>Computing cluster (SLURM)</strong>    </p>
-<ul>
-<li>Automatic checkpointing   </li>
-<li>Automatic saving, loading  </li>
-<li>Running grid search on a cluster </li>
-<li>Walltime auto-resubmit   </li>
 </ul>
               
             </div>

index.html

@@ -273,5 +273,5 @@
 
 <!--
 MkDocs version : 1.0.4
-Build Date UTC : 2019-06-27 17:47:30
+Build Date UTC : 2019-06-27 18:22:11
 -->