MP-CNN PyTorch Implementation
This is a PyTorch implementation of the following paper
- Hua He, Kevin Gimpel, and Jimmy Lin. Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 1576-1586.
Please ensure you have followed instructions in the main README doc before running any further commands in this doc.
SICK Dataset
To run MP-CNN on the SICK dataset, use the following command. --dropout 0 is for mimicking the original paper, although adding dropout can improve performance. If you have any problems running it check the Troubleshooting section below.
python -m mp_cnn mpcnn.sick.model.castor --dataset sick --epochs 19 --epsilon 1e-7 --dropout 0
| Implementation and config | Pearson's r | Spearman's p |
|---|---|---|
| Paper | 0.8686 | 0.8047 |
| PyTorch using above config | 0.8684 | 0.8083 |
MSRVID Dataset
To run MP-CNN on the MSRVID dataset, use the following command:
python -m mp_cnn mpcnn.msrvid.model.castor --dataset msrvid --batch-size 16 --epsilon 1e-7 --epochs 32 --dropout 0 --regularization 0.0025
| Implementation and config | Pearson's r |
|---|---|
| Paper | 0.9090 |
| PyTorch using above config | 0.8911 |
TrecQA Dataset
To run MP-CNN on (Raw) TrecQA, you first need to run ./get_trec_eval.sh in utils under the repo root while inside the utils directory. This will download and compile the official trec_eval tool used for evaluation.
Then, you can run:
python -m mp_cnn mpcnn.trecqa.model --dataset trecqa --epochs 5 --regularization 0.0005 --dropout 0.5 --eps 0.1
| Implementation and config | map | mrr |
|---|---|---|
| Paper | 0.762 | 0.830 |
| PyTorch using above config | 0.7904 | 0.8223 |
The paper results are reported in Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks.
WikiQA Dataset
You also need trec_eval for this dataset, similar to TrecQA.
Then, you can run:
python -m mp_cnn mpcnn.wikiqa.model --epochs 10 --dataset wikiqa --batch-size 64 --lr 0.0004 --regularization 0.02
| Implementation and config | map | mrr |
|---|---|---|
| Paper | 0.693 | 0.709 |
| PyTorch using above config | 0.693 | 0.7091 |
The paper results are reported in Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks.
These are not the optimal hyperparameters but they are decent. This README will be updated with more optimal hyperparameters and results in the future.
To see all options available, use
python -m mp_cnn --help
Troubleshooting
ModuleNotFoundError: datasets
Traceback (most recent call last):
File "__main__.py", line 9, in <module>
from common.dataset import DatasetFactory
File "/u/z3tu/castorini/Castor/common/dataset.py", line 12, in <module>
from datasets.sick import SICK
ModuleNotFoundError: No module named 'datasets'
You need to make sure the repository root is in your PYTHONPATH environment variable. One way to do this is while you are in the repo root (Castor) as your current working directory, run export PYTHONPATH=$(pwd).
Optional Dependencies
To optionally visualize the learning curve during training, we make use of https://github.com/lanpa/tensorboard-pytorch to connect to TensorBoard. These projects require TensorFlow as a dependency, so you need to install TensorFlow before running the commands below. After these are installed, just add --tensorboard when running main.py and open TensorBoard in the browser.
pip install tensorboardX
pip install tensorflow-tensorboard