mirror of
https://github.com/wassname/Castor.git
synced 2026-09-09 11:13:20 +08:00
Michael Tu
f7a0167b81
* Refactor main README * Update Anserini Dependency docs * Update idf baseline and Kim CNN docs to use Castor-data * Update remaining READMEs to reference Castor-data * Change default path from data to Castor-data * Fix wrong order of embeddings path
99 lines
3.5 KiB
Markdown
99 lines
3.5 KiB
Markdown
# kim_cnn
|
|
|
|
Implementation for Convolutional Neural Networks for Sentence Classification of [Kim (2014)](https://arxiv.org/abs/1408.5882) with PyTorch and Torchtext.
|
|
|
|
## Model Type
|
|
|
|
- rand: All words are randomly initialized and then modified during training.
|
|
- static: A model with pre-trained vectors from [word2vec](https://code.google.com/archive/p/word2vec/). All words -- including the unknown ones that are initialized with zero -- are kept static and only the other parameters of the model are learned.
|
|
- non-static: Same as above but the pretrained vectors are fine-tuned for each task.
|
|
- multichannel: A model with two sets of word vectors. Each set of vectors is treated as a 'channel' and each filter is applied to both channels, but gradients are back-propagated only through one of the channels. Hence the model is able to fine-tune one set of vectors while keeping the other static. Both channels are initialized with word2vec.# text-classification-cnn
|
|
Implementation for Convolutional Neural Networks for Sentence Classification of [Kim (2014)](https://arxiv.org/abs/1408.5882) with PyTorch.
|
|
|
|
## Requirement
|
|
|
|
Assuming you already have PyTorch, just install torchtext (`pip install torchtext==0.2.0`)
|
|
|
|
## Quick Start
|
|
|
|
To get the dataset, you can run this.
|
|
```
|
|
cd kim_cnn
|
|
bash get_data.sh
|
|
```
|
|
|
|
To run the model on SST-1 dataset on multichannel, just run the following code.
|
|
|
|
```
|
|
python train.py --mode multichannel
|
|
```
|
|
|
|
The file will be saved in
|
|
|
|
```
|
|
saves/best_model.pt
|
|
```
|
|
|
|
To test the model, you can use the following command.
|
|
|
|
```
|
|
python main.py --trained_model saves/best_model.pt --mode multichannel
|
|
```
|
|
|
|
|
|
|
|
## Dataset and Embeddings
|
|
|
|
We experiment the model on the following three datasets.
|
|
|
|
- SST-1: Keep the original splits and train with phrase level dataset and test on sentence level dataset.
|
|
|
|
**word2vec.sst-1.pt** is a subset of word2vector. We just select the word appearing in the SST-1 dataset and generate this file with the **vector_preprocess.py**(you will get this after you run get_data.sh or you can download [here](https://raw.githubusercontent.com/Impavidity/kim_cnn/master/vector_preprocess.py)) You can select these from any kind of word embedding text file and generate in following format.
|
|
```
|
|
word vector_in_one_line
|
|
```
|
|
and then run
|
|
```
|
|
python vector_preprocess.py file_in embed.pt
|
|
```
|
|
Here you can get *embed.pt* for the embedding file. Remember change the argument in *args.py* file with your own embedding.
|
|
|
|
## Settings
|
|
Adadelta is used for training.
|
|
|
|
## Training Time
|
|
|
|
For training time, when
|
|
|
|
```
|
|
torch.backends.cudnn.deterministic = True
|
|
```
|
|
|
|
is specified, the training will be ~3h because deterministic cnn algorithm is used (accuracy v.s. speed).
|
|
|
|
Other option is that
|
|
|
|
```
|
|
torch.backends.cudnn.enabled = False
|
|
```
|
|
but this will take ~6-7x training time.
|
|
|
|
## Results
|
|
|
|
Deterministic Algorithm for CNN.
|
|
|
|
| Dev Accuracy on SST-1 | rand | static | non-static | multichannel |
|
|
|:--------------------------:|:-----------:|:-----------:|:-------------:|:---------------:|
|
|
| My-Implementation | 42.597639| 48.773842| 48.864668 | 49.046322 |
|
|
|
|
| Test Accuracy on SST-1| rand | static | non-static | multichannel |
|
|
|:--------------------------:|:-----------:|:-----------:|:-------------:|:---------------:|
|
|
| Kim-Implementation | 45.0 | 45.5 | 48.0 | 47.4 |
|
|
| My- Implementation | 39.683258 | 45.972851| 48.914027| 47.330317 |
|
|
|
|
## TODO
|
|
|
|
- More experiments on SST-2 and subjectivity
|
|
- Parameters tuning
|
|
|