rand: All words are randomly initialized and then modified during training.
static: A model with pre-trained vectors from word2vec. All words -- including the unknown ones that are initialized with zero -- are kept static and only the other parameters of the model are learned.
non-static: Same as above but the pretrained vectors are fine-tuned for each task.

Quick Start

To run the model on Reuters dataset on static, just run the following from the Castor working directory.

python -m han --dataset Reuters

The file will be saved in

han/saves/best_model.pt

To test the model, you can use the following command.

python -m han --trained_model han/saves/Reuters/static_best_model.pt

Dataset

We experiment the model on the following datasets.

Reuters-21578: Split the data into sentences for the sentence level attention model and split the sentences into words for the word level attention. The word2vec pretrained embeddings were used for the task.

Adam is used for training.

For training time, when

torch.backends.cudnn.deterministic = True

is specified, the training will be ~10 min. Reuters-21578 is a relatively small dataset and the implementation is a vectorized one, hence the speed.

a combined hyperparameter tuning on a few of the datasets and report results with the hyperparameters