wassname/pytorch-lightning
1
0
Fork 0
mirror of https://github.com/wassname/pytorch-lightning.git synced 2026-09-09 11:32:07 +08:00
Code Issues Packages Projects Releases Wiki Activity

Deployed a78ee48 with MkDocs version: 1.0.4

Browse Source
This commit is contained in:
William Falcon
2019-08-12 15:13:26 -05:00
parent c2ee2951c6
commit 71d2fcb7cf
4 changed files with 226 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LightningModule/RequiredTrainerInterface/index.html
+211 -144
View File
@@ -290,29 +290,23 @@
</li>
<li class="md-nav__item">
<a href="#required-methods" title="Required Methods" class="md-nav__link">
Required Methods
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#training_step" title="training_step" class="md-nav__link">
training_step
</a>
</li>
<li class="md-nav__item">
<a href="#validation_step" title="validation_step" class="md-nav__link">
validation_step
</a>
</li>
<li class="md-nav__item">
<a href="#validation_end" title="validation_end" class="md-nav__link">
validation_end
</a>
</li>
<li class="md-nav__item">
<a href="#configure_optimizers" title="configure_optimizers" class="md-nav__link">
configure_optimizers
<li class="md-nav__item">
<a href="#tng_dataloader" title="tng_dataloader" class="md-nav__link">
tng_dataloader
</a>
<nav class="md-nav">
@@ -329,10 +323,10 @@
</nav>
</li>
<li class="md-nav__item">
<a href="#on_save_checkpoint" title="on_save_checkpoint" class="md-nav__link">
on_save_checkpoint
<li class="md-nav__item">
<a href="#configure_optimizers" title="configure_optimizers" class="md-nav__link">
configure_optimizers
</a>
<nav class="md-nav">
@@ -348,11 +342,38 @@
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#on_load_checkpoint" title="on_load_checkpoint" class="md-nav__link">
on_load_checkpoint
<a href="#optional-methods" title="Optional Methods" class="md-nav__link">
Optional Methods
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#validation_step" title="validation_step" class="md-nav__link">
validation_step
</a>
</li>
<li class="md-nav__item">
<a href="#validation_end" title="validation_end" class="md-nav__link">
validation_end
</a>
</li>
<li class="md-nav__item">
<a href="#on_save_checkpoint" title="on_save_checkpoint" class="md-nav__link">
on_save_checkpoint
</a>
<nav class="md-nav">
@@ -369,10 +390,10 @@
</nav>
</li>
<li class="md-nav__item">
<a href="#tng_dataloader" title="tng_dataloader" class="md-nav__link">
tng_dataloader
<li class="md-nav__item">
<a href="#on_load_checkpoint" title="on_load_checkpoint" class="md-nav__link">
on_load_checkpoint
</a>
<nav class="md-nav">
@@ -389,8 +410,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#val_dataloader" title="val_dataloader" class="md-nav__link">
val_dataloader
</a>
@@ -409,8 +430,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#test_dataloader" title="test_dataloader" class="md-nav__link">
test_dataloader
</a>
@@ -429,8 +450,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#update_tng_log_metrics" title="update_tng_log_metrics" class="md-nav__link">
update_tng_log_metrics
</a>
@@ -449,8 +470,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#add_model_specific_args" title="add_model_specific_args" class="md-nav__link">
add_model_specific_args
</a>
@@ -469,6 +490,11 @@
</nav>
</li>
</ul>
</nav>
</li>
@@ -713,29 +739,23 @@
</li>
<li class="md-nav__item">
<a href="#required-methods" title="Required Methods" class="md-nav__link">
Required Methods
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#training_step" title="training_step" class="md-nav__link">
training_step
</a>
</li>
<li class="md-nav__item">
<a href="#validation_step" title="validation_step" class="md-nav__link">
validation_step
</a>
</li>
<li class="md-nav__item">
<a href="#validation_end" title="validation_end" class="md-nav__link">
validation_end
</a>
</li>
<li class="md-nav__item">
<a href="#configure_optimizers" title="configure_optimizers" class="md-nav__link">
configure_optimizers
<li class="md-nav__item">
<a href="#tng_dataloader" title="tng_dataloader" class="md-nav__link">
tng_dataloader
</a>
<nav class="md-nav">
@@ -752,10 +772,10 @@
</nav>
</li>
<li class="md-nav__item">
<a href="#on_save_checkpoint" title="on_save_checkpoint" class="md-nav__link">
on_save_checkpoint
<li class="md-nav__item">
<a href="#configure_optimizers" title="configure_optimizers" class="md-nav__link">
configure_optimizers
</a>
<nav class="md-nav">
@@ -771,11 +791,38 @@
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#on_load_checkpoint" title="on_load_checkpoint" class="md-nav__link">
on_load_checkpoint
<a href="#optional-methods" title="Optional Methods" class="md-nav__link">
Optional Methods
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#validation_step" title="validation_step" class="md-nav__link">
validation_step
</a>
</li>
<li class="md-nav__item">
<a href="#validation_end" title="validation_end" class="md-nav__link">
validation_end
</a>
</li>
<li class="md-nav__item">
<a href="#on_save_checkpoint" title="on_save_checkpoint" class="md-nav__link">
on_save_checkpoint
</a>
<nav class="md-nav">
@@ -792,10 +839,10 @@
</nav>
</li>
<li class="md-nav__item">
<a href="#tng_dataloader" title="tng_dataloader" class="md-nav__link">
tng_dataloader
<li class="md-nav__item">
<a href="#on_load_checkpoint" title="on_load_checkpoint" class="md-nav__link">
on_load_checkpoint
</a>
<nav class="md-nav">
@@ -812,8 +859,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#val_dataloader" title="val_dataloader" class="md-nav__link">
val_dataloader
</a>
@@ -832,8 +879,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#test_dataloader" title="test_dataloader" class="md-nav__link">
test_dataloader
</a>
@@ -852,8 +899,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#update_tng_log_metrics" title="update_tng_log_metrics" class="md-nav__link">
update_tng_log_metrics
</a>
@@ -872,8 +919,8 @@
</nav>
</li>
<li class="md-nav__item">
<li class="md-nav__item">
<a href="#add_model_specific_args" title="add_model_specific_args" class="md-nav__link">
add_model_specific_args
</a>
@@ -892,6 +939,11 @@
</nav>
</li>
</ul>
</nav>
</li>
@@ -920,20 +972,14 @@
<p><strong>Required</strong>: </p>
<ul>
<li><a href="./#training_step">training_step</a> </li>
<li><a href="./#validation_step">validation_step</a></li>
<li>
<p><a href="./#validation_end">validation_end</a></p>
</li>
<li>
<p><a href="./#configure_optimizers">configure_optimizers</a></p>
</li>
<li>
<p><a href="./#tng_dataloader">tng_dataloader</a></p>
</li>
<li><a href="./#tng_dataloader">tng_dataloader</a></li>
<li><a href="./#test_dataloader">test_dataloader</a></li>
<li><a href="./#configure_optimizers">configure_optimizers</a></li>
</ul>
<p><strong>Optional</strong>: </p>
<p><strong>Optional</strong>: <br />
- <a href="./#validation_step">validation_step</a>
- <a href="./#validation_end">validation_end</a>
- <a href="./#val_dataloader">val_dataloader</a>
- <a href="./#test_dataloader">test_dataloader</a></p>
<ul>
<li><a href="./#on_save_checkpoint">on_save_checkpoint</a></li>
<li><a href="./#on_load_checkpoint">on_load_checkpoint</a></li>
@@ -961,24 +1007,25 @@ class CoolModel(pl.LightningModule):
def forward(self, x):
return torch.relu(self.l1(x.view(x.size(0), -1)))
def my_loss(self, y_hat, y):
return F.cross_entropy(y_hat, y)
def training_step(self, batch, batch_nb):
# REQUIRED
x, y = batch
y_hat = self.forward(x)
return {'loss': self.my_loss(y_hat, y)}
return {'loss': F.cross_entropy(y_hat, y)(y_hat, y)}
def validation_step(self, batch, batch_nb):
# OPTIONAL
x, y = batch
y_hat = self.forward(x)
return {'val_loss': self.my_loss(y_hat, y)}
return {'val_loss': F.cross_entropy(y_hat, y)(y_hat, y)}
def validation_end(self, outputs):
# OPTIONAL
avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
return {'avg_val_loss': avg_loss}
def configure_optimizers(self):
# REQUIRED
return [torch.optim.Adam(self.parameters(), lr=0.02)]
@pl.data_loader
@@ -987,10 +1034,13 @@ class CoolModel(pl.LightningModule):
@pl.data_loader
def val_dataloader(self):
# OPTIONAL
# can also return a list of val dataloaders
return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
@pl.data_loader
def test_dataloader(self):
# OPTIONAL
return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
</code></pre>
@@ -1003,7 +1053,7 @@ class CoolModel(pl.LightningModule):
</a>
</p>
<hr />
<h2 id="required-methods">Required Methods</h2>
<h3 id="training_step">training_step</h3>
<pre><code class="python">def training_step(self, data_batch, batch_nb)
</code></pre>
@@ -1069,12 +1119,59 @@ class CoolModel(pl.LightningModule):
</code></pre>
<hr />
<h3 id="validation_step">validation_step</h3>
<pre><code class="python">def validation_step(self, data_batch, batch_nb)
<h3 id="tng_dataloader">tng_dataloader</h3>
<pre><code class="python">@pl.data_loader
def tng_dataloader(self)
</code></pre>
<p>In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something specific to your model.
This is most likely the same as your training_step. But unlike training step, the outputs from here will go to validation_end for collation.</p>
<p>Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.</p>
<h5 id="return">Return</h5>
<p>PyTorch DataLoader</p>
<p><strong>Example</strong></p>
<pre><code class="python">@pl.data_loader
def tng_dataloader(self):
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
loader = torch.utils.data.DataLoader(
dataset=dataset,
batch_size=self.hparams.batch_size,
shuffle=True
)
return loader
</code></pre>
<hr />
<h3 id="configure_optimizers">configure_optimizers</h3>
<pre><code class="python">def configure_optimizers(self)
</code></pre>
<p>Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple.
Lightning will call .backward() and .step() on each one in every epoch. If you use 16 bit precision it will also handle that.</p>
<h5 id="return_1">Return</h5>
<p>List or Tuple - List of optimizers with an optional second list of learning-rate schedulers</p>
<p><strong>Example</strong></p>
<pre><code class="python"># most cases
def configure_optimizers(self):
opt = Adam(self.parameters(), lr=0.01)
return [opt]
# gan example, with scheduler for discriminator
def configure_optimizers(self):
generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
return [generator_opt, disriminator_opt], [discriminator_sched]
</code></pre>
<h2 id="optional-methods">Optional Methods</h2>
<h3 id="validation_step">validation_step</h3>
<pre><code class="python">def validation_step(self, data_batch, batch_nb, dataloader_i)
</code></pre>
<p><strong>OPTIONAL</strong> <br />
If you don't need to validate you don't need to implement this method. </p>
<p>In this step you'd normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes, calculate accuracy, or save example outputs (using self.experiment or whatever you want). Really, anything you want. </p>
<p>This is most likely the same as your training_step. But unlike training step, the outputs from here will go to validation_end for collation.</p>
<p><strong>Params</strong> </p>
<table>
<thead>
@@ -1092,6 +1189,10 @@ This is most likely the same as your training_step. But unlike training step, th
<td>batch_nb</td>
<td>Integer displaying which batch this is</td>
</tr>
<tr>
<td>dataloader_i</td>
<td>Integer displaying which dataloader this is</td>
</tr>
</tbody>
</table>
<p><strong>Return</strong> </p>
@@ -1139,7 +1240,9 @@ This is most likely the same as your training_step. But unlike training step, th
<pre><code class="python">def validation_end(self, outputs)
</code></pre>
<p>Called at the end of the validation loop with the output of each validation_step.</p>
<p>If you didn't define a validation_step, this won't be called. </p>
<p>Called at the end of the validation loop with the output of each validation_step. Called once per validation dataset. </p>
<p>The outputs here are strictly for the progress bar. If you don't need to display anything, don't return anything. </p>
<p><strong>Params</strong> </p>
<table>
<thead>
@@ -1191,29 +1294,6 @@ This is most likely the same as your training_step. But unlike training step, th
return tqdm_dic
</code></pre>
<hr />
<h3 id="configure_optimizers">configure_optimizers</h3>
<pre><code class="python">def configure_optimizers(self)
</code></pre>
<p>Set up as many optimizers and (optionally) learning rate schedulers as you need. Normally you'd need one. But in the case of GANs or something more esoteric you might have multiple.
Lightning will call .backward() and .step() on each one in every epoch. If you use 16 bit precision it will also handle that.</p>
<h5 id="return">Return</h5>
<p>List or Tuple - List of optimizers with an optional second list of learning-rate schedulers</p>
<p><strong>Example</strong></p>
<pre><code class="python"># most cases
def configure_optimizers(self):
opt = Adam(self.parameters(), lr=0.01)
return [opt]
# gan example, with scheduler for discriminator
def configure_optimizers(self):
generator_opt = Adam(self.model_gen.parameters(), lr=0.01)
disriminator_opt = Adam(self.model_disc.parameters(), lr=0.02)
discriminator_sched = CosineAnnealing(discriminator_opt, T_max=10)
return [generator_opt, disriminator_opt], [discriminator_sched]
</code></pre>
<hr />
<h3 id="on_save_checkpoint">on_save_checkpoint</h3>
<pre><code class="python">def on_save_checkpoint(self, checkpoint)
@@ -1222,7 +1302,7 @@ def configure_optimizers(self):
<p>Called by lightning to checkpoint your model. Lightning saves the training state (current epoch, global_step, etc)
and also saves the model state_dict. If you want to save anything else, use this method to add your own
key-value pair.</p>
<h5 id="return_1">Return</h5>
<h5 id="return_2">Return</h5>
<p>Nothing</p>
<p><strong>Example</strong></p>
<pre><code class="python">def on_save_checkpoint(self, checkpoint):
@@ -1238,7 +1318,7 @@ key-value pair.</p>
<p>Called by lightning to restore your model. Lighting auto-restores global step, epoch, etc...
It also restores the model state_dict.
If you saved something with <strong>on_save_checkpoint</strong> this is your chance to restore this.</p>
<h5 id="return_2">Return</h5>
<h5 id="return_3">Return</h5>
<p>Nothing </p>
<p><strong>Example</strong></p>
<pre><code class="python">def on_load_checkpoint(self, checkpoint):
@@ -1246,37 +1326,17 @@ If you saved something with <strong>on_save_checkpoint</strong> this is your cha
self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']
</code></pre>
<hr />
<h3 id="tng_dataloader">tng_dataloader</h3>
<pre><code class="python">@pl.data_loader
def tng_dataloader(self)
</code></pre>
<p>Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.</p>
<h5 id="return_3">Return</h5>
<p>PyTorch DataLoader</p>
<p><strong>Example</strong></p>
<pre><code class="python">@pl.data_loader
def tng_dataloader(self):
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
dataset = MNIST(root='/path/to/mnist/', train=True, transform=transform, download=True)
loader = torch.utils.data.DataLoader(
dataset=dataset,
batch_size=self.hparams.batch_size,
shuffle=True
)
return loader
</code></pre>
<hr />
<h3 id="val_dataloader">val_dataloader</h3>
<pre><code class="python">@pl.data_loader
def tng_dataloader(self)
</code></pre>
<p>Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.</p>
<p><strong>OPTIONAL</strong> <br />
If you don't need a validation dataset and a validation_step, you don't need to implement this method. </p>
<p>Called by lightning during validation loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed. </p>
<h5 id="return_4">Return</h5>
<p>PyTorch DataLoader</p>
<p>PyTorch DataLoader or list of PyTorch Dataloaders. </p>
<p><strong>Example</strong></p>
<pre><code class="python">@pl.data_loader
def val_dataloader(self):
@@ -1289,6 +1349,11 @@ def val_dataloader(self):
)
return loader
# can also return multiple dataloaders
@pl.data_loader
def val_dataloader(self):
return [loader_a, loader_b, ..., loader_n]
</code></pre>
<hr />
@@ -1297,6 +1362,8 @@ def val_dataloader(self):
def test_dataloader(self)
</code></pre>
<p><strong>OPTIONAL</strong> <br />
If you don't need a test dataset and a test_step, you don't need to implement this method. </p>
<p>Called by lightning during test loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.</p>
<h5 id="return_5">Return</h5>
<p>PyTorch DataLoader</p>