mirror of
https://github.com/wassname/Castor.git
synced 2026-09-09 11:13:20 +08:00
Ashutosh-Adhikari
91ed6261db
* Add Regularization Modules for LSTM * Update Reuters Trainer and Evalueator for regularization * Remove unnecessary comments * Comply with PEP8 * Comply import order with PEP8 * Fix typos in README.md * Comply with PEP8 * Add BSD 3-Clause Licence * Remove deprecated call to Variable for PyTorch 0.4 * Update dataset selection in main * Remove block comments
43 lines
1.5 KiB
Markdown
43 lines
1.5 KiB
Markdown
# lstm_baseline with Regularization
|
|
|
|
Implementation of a standard LSTM using PyTorch and Torchtext for text classification baseline measurements with Regularization.
|
|
|
|
## Model Type
|
|
|
|
- rand: All words are randomly initialized and then modified during training.
|
|
- static: A model with pre-trained vectors from [word2vec](https://code.google.com/archive/p/word2vec/). All words -- including the unknown ones that are initialized with zero -- are kept static and only the other parameters of the model are learned.
|
|
- non-static: Same as above but the pretrained vectors are fine-tuned for each task.
|
|
|
|
## Quick Start
|
|
|
|
To run the model on Reuters dataset on static, just run the following from the Castor working directory.
|
|
|
|
```
|
|
python -m lstm_baseline --mode static
|
|
```
|
|
|
|
## Dataset
|
|
|
|
We experiment the model on the following datasets.
|
|
|
|
- Reuters dataset - ModApte splits
|
|
|
|
## Settings
|
|
|
|
Adam is used for training with an option of temporal averaging.
|
|
|
|
## TODO
|
|
- Support ONNX export. Currently throws a ONNX export failed (Couldn't export Python operator forward_flattened_wrapper) exception.
|
|
- Add dataset results with different hyperparameters
|
|
- Parameters tuning
|
|
|
|
## Regularization Module
|
|
|
|
- Regularization methods like Embedding dropout, Weight Dropped LSTM and Temporal Activation Regularization are implemented.
|
|
- Temporal Averaging is also an additional module
|
|
|
|
## Acknowledgement
|
|
- The additional modules have been heavily inspired by two open source repositories:
|
|
- https://github.com/salesforce/awd-lstm-lm.git
|
|
- https://github.com/AMLab-Amsterdam/L0_regularization.git
|