Castor/lstm_regularization

lstm_baseline with Regularization

Implementation of a standard LSTM using PyTorch and Torchtext for text classification baseline measurements with Regularization.

Model Type

• rand: All words are randomly initialized and then modified during training.
• static: A model with pre-trained vectors from word2vec. All words -- including the unknown ones that are initialized with zero -- are kept static and only the other parameters of the model are learned.
• non-static: Same as above but the pretrained vectors are fine-tuned for each task.

Quick Start

To run the model on Reuters dataset on static, just run the following from the Castor working directory.

python -m lstm_baseline --mode static

Dataset

We experiment the model on the following datasets.

• Reuters dataset - ModApte splits

Settings

Adam is used for training with an option of temporal averaging.

TODO

• Support ONNX export. Currently throws a ONNX export failed (Couldn't export Python operator forward_flattened_wrapper) exception.
• Add dataset results with different hyperparameters
• Parameters tuning

Regularization Module

• Regularization methods like Embedding dropout, Weight Dropped LSTM and Temporal Activation Regularization are implemented.
• Temporal Averaging is also an additional module

Acknowledgement

• The additional modules have been heavily inspired by two open source repositories:
  • https://github.com/salesforce/awd-lstm-lm.git
  • https://github.com/AMLab-Amsterdam/L0_regularization.git