Deployed cca6d2c with MkDocs version: 1.0.4

Trainer/Distributed training/index.html

@@ -360,6 +360,13 @@
     Choosing a backend
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
+    Distributed and 16-bit precision.
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -551,6 +558,13 @@
     Choosing a backend
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#distributed-and-16-bit-precision" title="Distributed and 16-bit precision." class="md-nav__link">
+    Distributed and 16-bit precision.
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -636,6 +650,56 @@ We recommend you use DistributedDataparallel even for single-node multi-GPU trai
 have configuration issues depending on your cluster.</p>
 <p>For a deeper understanding of what lightning is doing, feel free to read <a href="https://medium.com/@_willfalcon/9-tips-for-training-lightning-fast-neural-networks-in-pytorch-8e63a502f565">this guide</a>.   </p>
 <hr />
+<h4 id="distributed-and-16-bit-precision">Distributed and 16-bit precision.</h4>
+<p>Due to an issue with apex and DistributedDataParallel (PyTorch and NVIDIA issue), Lightning does
+not allow 16-bit and DP training. We tried to get this to work, but it's an issue on their end.   </p>
+<table>
+<thead>
+<tr>
+<th>1 GPU</th>
+<th>1+ GPUs</th>
+<th>DP</th>
+<th>DDP</th>
+<th>16-bit</th>
+<th>command</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Y</td>
+<td></td>
+<td></td>
+<td></td>
+<td>Y</td>
+<td><code>Trainer(gpus=[0])</code></td>
+</tr>
+<tr>
+<td></td>
+<td>Y</td>
+<td>Y</td>
+<td></td>
+<td></td>
+<td><code>Trainer(gpus=[0, ...])</code></td>
+</tr>
+<tr>
+<td></td>
+<td>Y</td>
+<td></td>
+<td>Y</td>
+<td></td>
+<td><code>Trainer(gpus=[0, ...], distributed_backend='ddp')</code></td>
+</tr>
+<tr>
+<td></td>
+<td>Y</td>
+<td></td>
+<td>Y</td>
+<td>Y</td>
+<td><code>Trainer(gpus=[0, ...], distributed_backend='ddp', use_amp=True)</code></td>
+</tr>
+</tbody>
+</table>
+<hr />
 <h4 id="cuda-flags">CUDA flags</h4>
 <p>CUDA flags make certain GPUs visible to your script. 
 Lightning sets these for you automatically, there's NO NEED to do this yourself.</p>