Deployed 974afba with MkDocs version: 1.0.4

LightningModule/RequiredTrainerInterface/index.html

@@ -1370,8 +1370,8 @@ the <a href="https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#op
 </td></tr></table>
 
 <p><strong>OPTIONAL</strong>  <br />
-If you don't need to validate you don't need to implement this method.    </p>
-<p>In this step you'd normally generate examples or calculate anything of interest such as accuracy. </p>
+If you don't need to validate you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy. </p>
+<p>When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.</p>
 <p>The dict you return here will be available in the <code>validation_end</code> method. </p>
 <p><strong>Params</strong>   </p>
 <table>
@@ -1487,7 +1487,7 @@ If you don't need to validate you don't need to implement this method.    </p>
 </td></tr></table>
 
 <p>If you didn't define a validation_step, this won't be called.       </p>
-<p>Called at the end of the validation loop with the output of each validation_step.  Called once per validation dataset.   </p>
+<p>Called at the end of the validation loop with the outputs of validation_step.</p>
 <p>The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    </p>
 <p><strong>Params</strong>    </p>
 <table>
@@ -1500,7 +1500,7 @@ If you don't need to validate you don't need to implement this method.    </p>
 <tbody>
 <tr>
 <td>outputs</td>
-<td>List of outputs you defined in validation_step</td>
+<td>List of outputs you defined in validation_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader</td>
 </tr>
 </tbody>
 </table>
@@ -1522,6 +1522,7 @@ If you don't need to validate you don't need to implement this method.    </p>
 </tbody>
 </table>
 <p><strong>Example</strong></p>
+<p>With a single dataloader</p>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
  2
  3
@@ -1556,6 +1557,49 @@ If you don't need to validate you don't need to implement this method.    </p>
 </pre></div>
 </td></tr></table>
 
+<p>With multiple dataloaders, <code>outputs</code> will be a list of lists. The outer list contains
+one entry per dataloader, while the inner list contains the individual outputs of 
+each validation step for that dataloader.</p>
+<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">validation_end</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">outputs</span><span class="p">):</span>
+    <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Called at the end of validation to aggregate outputs</span>
+<span class="sd">    :param outputs: list of list of individual outputs of each validation step</span>
+<span class="sd">    :return:</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+    <span class="n">val_loss_mean</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="n">val_acc_mean</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="k">for</span> <span class="n">dataloader_outputs</span> <span class="ow">in</span> <span class="n">outputs</span><span class="p">:</span>
+        <span class="k">for</span> <span class="n">output</span> <span class="ow">in</span> <span class="n">dataloader_outputs</span><span class="p">:</span>
+            <span class="n">val_loss_mean</span> <span class="o">+=</span> <span class="n">output</span><span class="p">[</span><span class="s1">&#39;val_loss&#39;</span><span class="p">]</span>
+            <span class="n">val_acc_mean</span> <span class="o">+=</span> <span class="n">output</span><span class="p">[</span><span class="s1">&#39;val_acc&#39;</span><span class="p">]</span>
+            <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
+
+    <span class="n">val_loss_mean</span> <span class="o">/=</span> <span class="n">i</span>
+    <span class="n">val_acc_mean</span> <span class="o">/=</span> <span class="n">i</span>
+    <span class="n">tqdm_dic</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;val_loss&#39;</span><span class="p">:</span> <span class="n">val_loss_mean</span><span class="o">.</span><span class="n">item</span><span class="p">(),</span> <span class="s1">&#39;val_acc&#39;</span><span class="p">:</span> <span class="n">val_acc_mean</span><span class="o">.</span><span class="n">item</span><span class="p">()}</span>
+    <span class="k">return</span> <span class="n">tqdm_dic</span>
+</pre></div>
+</td></tr></table>
+
 <h3 id="test_step">test_step</h3>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
 2
@@ -1570,8 +1614,8 @@ If you don't need to validate you don't need to implement this method.    </p>
 </td></tr></table>
 
 <p><strong>OPTIONAL</strong>  <br />
-If you don't need to test you don't need to implement this method.    </p>
-<p>In this step you'd normally generate examples or calculate anything of interest such as accuracy. </p>
+If you don't need to test you don't need to implement this method. In this step you'd normally generate examples or calculate anything of interest such as accuracy.   </p>
+<p>When the validation_step is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, model goes back to training mode and gradients are enabled.</p>
 <p>The dict you return here will be available in the <code>test_end</code> method. </p>
 <p>This function is used when you execute <code>trainer.test()</code>.</p>
 <p><strong>Params</strong>   </p>
@@ -1676,7 +1720,7 @@ If you don't need to test you don't need to implement this method.    </p>
 </td></tr></table>
 
 <p>If you didn't define a test_step, this won't be called.       </p>
-<p>Called at the end of the test step with the output of each test_step.  Called once per test dataset.   </p>
+<p>Called at the end of the test step with the output of each test_step.</p>
 <p>The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything.    </p>
 <p><strong>Params</strong>    </p>
 <table>
@@ -1689,7 +1733,7 @@ If you don't need to test you don't need to implement this method.    </p>
 <tbody>
 <tr>
 <td>outputs</td>
-<td>List of outputs you defined test_step</td>
+<td>List of outputs you defined in test_step, or if there are multiple dataloaders, a list containing a list of outputs for each dataloader</td>
 </tr>
 </tbody>
 </table>
@@ -1745,6 +1789,49 @@ If you don't need to test you don't need to implement this method.    </p>
 </pre></div>
 </td></tr></table>
 
+<p>With multiple dataloaders, <code>outputs</code> will be a list of lists. The outer list contains
+one entry per dataloader, while the inner list contains the individual outputs of 
+each validation step for that dataloader.</p>
+<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">test_end</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">outputs</span><span class="p">):</span>
+    <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Called at the end of test to aggregate outputs</span>
+<span class="sd">    :param outputs: list of individual outputs of each test step</span>
+<span class="sd">    :return:</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+    <span class="n">test_loss_mean</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="n">test_acc_mean</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="k">for</span> <span class="n">dataloader_outputs</span> <span class="ow">in</span> <span class="n">outputs</span><span class="p">:</span>
+        <span class="k">for</span> <span class="n">output</span> <span class="ow">in</span> <span class="n">dataloader_outputs</span><span class="p">:</span>
+            <span class="n">test_loss_mean</span> <span class="o">+=</span> <span class="n">output</span><span class="p">[</span><span class="s1">&#39;test_loss&#39;</span><span class="p">]</span>
+            <span class="n">test_acc_mean</span> <span class="o">+=</span> <span class="n">output</span><span class="p">[</span><span class="s1">&#39;test_acc&#39;</span><span class="p">]</span>
+            <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
+
+    <span class="n">test_loss_mean</span> <span class="o">/=</span> <span class="n">i</span> 
+    <span class="n">test_acc_mean</span> <span class="o">/=</span> <span class="n">i</span>
+    <span class="n">tqdm_dic</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;test_loss&#39;</span><span class="p">:</span> <span class="n">test_loss_mean</span><span class="o">.</span><span class="n">item</span><span class="p">(),</span> <span class="s1">&#39;test_acc&#39;</span><span class="p">:</span> <span class="n">test_acc_mean</span><span class="o">.</span><span class="n">item</span><span class="p">()}</span>
+    <span class="k">return</span> <span class="n">tqdm_dic</span>
+</pre></div>
+</td></tr></table>
+
 <hr />
 <h3 id="on_save_checkpoint">on_save_checkpoint</h3>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">on_save_checkpoint</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">checkpoint</span><span class="p">)</span>

Trainer/Distributed training/index.html

@@ -654,10 +654,16 @@ For multi-node training you must use DistributedDataParallel.   </p>
 2
 3
 4
-5</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT uses DataParallel</span>
+5
+6
+7
+8</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (when using single GPU or no GPUs)</span>
+<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">distributed_backend</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
+
+<span class="c1"># Change to DataParallel (gpus &gt; 1)</span>
 <span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;dp&#39;</span><span class="p">)</span>
 
-<span class="c1"># change to distributed data parallel</span>
+<span class="c1"># change to distributed data parallel (gpus &gt; 1)</span>
 <span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;ddp&#39;</span><span class="p">)</span>
 </pre></div>
 </td></tr></table>
@@ -689,7 +695,7 @@ not allow 16-bit and DP training. We tried to get this to work, but it's an issu
 <td></td>
 <td></td>
 <td></td>
-<td><code>Trainer(gpus=[0])</code></td>
+<td><code>Trainer(gpus=1)</code></td>
 </tr>
 <tr>
 <td>Y</td>
@@ -697,7 +703,7 @@ not allow 16-bit and DP training. We tried to get this to work, but it's an issu
 <td></td>
 <td></td>
 <td>Y</td>
-<td><code>Trainer(gpus=[0], use_amp=True)</code></td>
+<td><code>Trainer(gpus=1, use_amp=True)</code></td>
 </tr>
 <tr>
 <td></td>
@@ -705,7 +711,7 @@ not allow 16-bit and DP training. We tried to get this to work, but it's an issu
 <td>Y</td>
 <td></td>
 <td></td>
-<td><code>Trainer(gpus=[0, ...])</code></td>
+<td><code>Trainer(gpus=k)</code></td>
 </tr>
 <tr>
 <td></td>
@@ -713,7 +719,7 @@ not allow 16-bit and DP training. We tried to get this to work, but it's an issu
 <td></td>
 <td>Y</td>
 <td></td>
-<td><code>Trainer(gpus=[0, ...], distributed_backend='ddp')</code></td>
+<td><code>Trainer(gpus=k, distributed_backend='ddp')</code></td>
 </tr>
 <tr>
 <td></td>
@@ -721,10 +727,29 @@ not allow 16-bit and DP training. We tried to get this to work, but it's an issu
 <td></td>
 <td>Y</td>
 <td>Y</td>
-<td><code>Trainer(gpus=[0, ...], distributed_backend='ddp', use_amp=True)</code></td>
+<td><code>Trainer(gpus=k, distributed_backend='ddp', use_amp=True)</code></td>
 </tr>
 </tbody>
 </table>
+<p>You also have the option of specifying which GPUs to use by passing a list:   </p>
+<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
+2
+3
+4
+5
+6
+7
+8</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT (int)</span>
+<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="n">k</span><span class="p">)</span>  
+
+<span class="c1"># You specify which GPUs (don&#39;t use if running on cluster)  </span>
+<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>  
+
+<span class="c1"># can also be a string</span>
+<span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="s1">&#39;0, 1&#39;</span><span class="p">)</span>
+</pre></div>
+</td></tr></table>
+
 <hr />
 <h4 id="cuda-flags">CUDA flags</h4>
 <p>CUDA flags make certain GPUs visible to your script. 
@@ -737,6 +762,8 @@ Lightning sets these for you automatically, there's NO NEED to do this yourself.
 </pre></div>
 </td></tr></table>
 
+<p>However, when using a cluster, Lightning will NOT set these flags (and you should not either). 
+SLURM will set these for you.   </p>
 <hr />
 <h4 id="16-bit-mixed-precision">16-bit mixed precision</h4>
 <p>16 bit precision can cut your memory footprint by half. If using volta architecture GPUs it can give a dramatic training speed-up as well.  <br />
@@ -761,7 +788,7 @@ $ pip install -v --no-cache-dir --global-option<span class="o">=</span><span cla
 <p>Make sure you're on a GPU machine. </p>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
 2</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT</span>
-<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
+<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
 </pre></div>
 </td></tr></table>
 
@@ -773,11 +800,11 @@ In this setting, the model will run on all 8 GPUs at once using DataParallel und
 2
 3
 4
-5</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># to use DataParallel (default)</span>
-<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">7</span><span class="p">],</span> <span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;dp&#39;</span><span class="p">)</span>
+5</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># to use DataParallel</span>
+<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;dp&#39;</span><span class="p">)</span>
 
 <span class="c1"># RECOMMENDED use DistributedDataParallel</span>
-<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">7</span><span class="p">],</span> <span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;ddp&#39;</span><span class="p">)</span>
+<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">distributed_backend</span><span class="o">=</span><span class="s1">&#39;ddp&#39;</span><span class="p">)</span>
 </pre></div>
 </td></tr></table>
 
@@ -786,7 +813,7 @@ In this setting, the model will run on all 8 GPUs at once using DataParallel und
 <p>Multi-node training is easily done by specifying these flags.</p>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
 2</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># train on 12*8 GPUs</span>
-<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">7</span><span class="p">],</span> <span class="n">nb_gpu_nodes</span><span class="o">=</span><span class="mi">12</span><span class="p">)</span>
+<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">gpus</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">nb_gpu_nodes</span><span class="o">=</span><span class="mi">12</span><span class="p">)</span>
 </pre></div>
 </td></tr></table>
 

Trainer/Logging/index.html

@@ -379,6 +379,13 @@
     Log metric row every k batches
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#log-metric-row-every-k-batches_1" title="Log metric row every k batches" class="md-nav__link">
+    Log metric row every k batches
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -576,6 +583,13 @@
     Log metric row every k batches
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#log-metric-row-every-k-batches_1" title="Log metric row every k batches" class="md-nav__link">
+    Log metric row every k batches
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -658,6 +672,15 @@
 </pre></div>
 </td></tr></table>
 
+<hr />
+<h4 id="log-metric-row-every-k-batches_1">Log metric row every k batches</h4>
+<p>Logs GPU memory when metrics are logged.   </p>
+<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
+2</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># DEFAULT</span>
+<span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">log_gpu_memory</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+</pre></div>
+</td></tr></table>
+
 <hr />
 <h4 id="process-position">Process position</h4>
 <p>When running multiple models on the same machine we want to decide which progress bar to use.

Trainer/SLURM Managed Cluster/index.html

@@ -560,10 +560,11 @@
                 
                 <p>Lightning supports model training on a cluster managed by SLURM in the following cases:    </p>
 <ol>
-<li>Training on single or multi-cpus only.</li>
-<li>Training on single or multi-gpus on the same node.</li>
-<li>Coming SOON: Training across multiple nodes.</li>
+<li>Training on a single cpu or single GPU.</li>
+<li>Train on multiple GPUs on the same node using DataParallel or DistributedDataParallel</li>
+<li>Training across multiple GPUs on multiple different nodes via DistributedDataParallel.</li>
 </ol>
+<p><strong>Note: A node means a machine with multiple GPUs</strong></p>
 <hr />
 <h4 id="running-grid-search-on-a-cluster">Running grid search on a cluster</h4>
 <p>To use lightning to run a hyperparameter search (grid-search or random-search) on a cluster do 4 things:   </p>
@@ -645,7 +646,8 @@
 </pre></div>
 </td></tr></table>
 
-<p>(3). Give trainer the cluster_manager in your main function:    </p>
+<p>(3). Make a main function with your model and trainer. Each job will call this function with a particular
+hparams configuration.    </p>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
  2
  3
@@ -663,12 +665,12 @@
     <span class="n">my_model</span> <span class="o">=</span> <span class="n">MyLightningModel</span><span class="p">()</span>
 
     <span class="c1"># give the trainer the cluster object</span>
-    <span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">cluster</span><span class="o">=</span><span class="n">cluster_manager</span><span class="p">)</span>
+    <span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">()</span>
     <span class="n">trainer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">my_model</span><span class="p">)</span>
 </pre></div>
 </td></tr></table>
 
-<p>(4). Start the grid search     </p>
+<p>(3). Start the grid/random search     </p>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
 2
 3
@@ -683,25 +685,22 @@
 </pre></div>
 </td></tr></table>
 
-<p>That's it! The SlurmCluster object will automatically checkpoint the lightning model and resubmit if it runs into the walltime!</p>
 <hr />
 <h4 id="walltime-auto-resubmit">Walltime auto-resubmit</h4>
-<p>Lightning automatically resubmits jobs when they reach the walltime. You get this behavior for free if you give lightning
-a slurm cluster object.</p>
+<p>Lightning automatically resubmits jobs when they reach the walltime. Make sure to set the SIGUSR1 signal in 
+your SLURM script.   </p>
 <table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1
-2</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">my_main_fx</span><span class="p">(</span><span class="n">hparams</span><span class="p">,</span> <span class="n">slurm_manager</span><span class="p">,</span> <span class="n">_</span><span class="p">):</span>
-    <span class="n">trainer</span> <span class="o">=</span> <span class="n">Trainer</span><span class="p">(</span><span class="n">cluster</span><span class="o">=</span><span class="n">slurm_manager</span><span class="p">)</span>
+2</pre></div></td><td class="code"><div class="codehilite"><pre><span></span><span class="c1"># 90 seconds before training ends</span>
+<span class="c1">#SBATCH --signal=SIGUSR1@90</span>
 </pre></div>
 </td></tr></table>
 
-<p>(See the grid search example above for cluster configuration).
-With this feature lightning will:    </p>
-<ol>
-<li>automatically checkpoint the model</li>
-<li>checkpoint the trainer session</li>
-<li>resubmit a continuation job.</li>
-<li>load the checkpoint and trainer session in the new model</li>
-</ol>
+<p>When lightning receives the SIGUSR1 signal it will:
+1. save a checkpoint with 'hpc_ckpt' in the name.
+2. resubmit the job using the SLURM_JOB_ID  </p>
+<p>When the script starts again, Lightning will:
+1. search for a 'hpc_ckpt' checkpoint. 
+2. restore the model, optimizers, schedulers, epoch, etc...   </p>
                 
                   
                 

sitemap.xml

@@ -2,77 +2,77 @@
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
     <url>
      <loc>None</loc>
-     <lastmod>2019-08-31</lastmod>
+     <lastmod>2019-09-16</lastmod>
      <changefreq>daily</changefreq>
     </url>
 </urlset>