packed sequence clarification in train_dataloader (#443)

* packed sequence clarification in train_dataloader

* moved changes to training loop

* removed changes from required interface

* added index entry
This commit is contained in:
s-rog
2019-11-03 05:26:27 -05:00
committed by William Falcon
parent 1865de1ff8
commit 4e9fd95f79
3 changed files with 21 additions and 1 deletions
@@ -186,7 +186,7 @@ break out of the current training epoch early.
def train_dataloader(self)
```
Called by lightning during training loop. Make sure to use the @pl.data_loader decorator, this ensures not calling this function until the data are needed.
If you want to change the data during every epoch DON'T use the data_loader decorator.
If you want to change the data during every epoch DON'T use the data_loader decorator.
##### Return
PyTorch DataLoader
+19
View File
@@ -92,6 +92,25 @@ trainer = Trainer(train_percent_check=1.0)
trainer = Trainer(train_percent_check=0.1)
```
---
#### Packed sequences as inputs
When using PackedSequence, do 2 things:
1. return either a padded tensor in dataset or a list of variable length tensors in the dataloader collate_fn (example above shows the list implementation).
2. Pack the sequence in forward or training and validation steps depending on use case.
``` {.python}
# For use in dataloader
def collate_fn(batch):
x = [item[0] for item in batch]
y = [item[1] for item in batch]
return x, y
# In module
def training_step(self, batch, batch_nb):
x = rnn.pack_sequence(batch[0], enforce_sorted=False)
y = rnn.pack_sequence(batch[1], enforce_sorted=False)
```
---
#### Truncated Back Propagation Through Time
There are times when multiple backwards passes are needed for each batch. For example, it may save memory to use Truncated Back Propagation Through Time when training RNNs.
+1
View File
@@ -71,6 +71,7 @@ But of course the fun is in all the advanced things it can do:
- [Use multiple optimizers (like GANs)](https://williamfalcon.github.io/pytorch-lightning/LightningModule/RequiredTrainerInterface/#configure_optimizers)
- [Set how much of the training set to check (1-100%)](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#set-how-much-of-the-training-set-to-check)
- [Step optimizers at arbitrary intervals](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#optimizer_step)
- [Packed sequences](https://williamfalcon.github.io/pytorch-lightning/Trainer/Training%20Loop/#packed-sequences-as-inputs)
- [Truncated Back Propagation Through Time](https://williamfalcon.github.io/pytorch-lightning//Training%20Loop/#truncated-back-propation-through-time)
**Validation loop**