wassname/Castor
1
0
Fork 0
mirror of https://github.com/wassname/Castor.git synced 2026-08-20 12:00:37 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update VDPWI README with new results (#131)

Browse Source 
- Update with results on SICK, TrecQA, and WikiQA datasets
This commit is contained in:
Linqing Liu
2018-07-03 09:37:18 -04:00
committed by Ralph Tang
parent a4e847725c
commit fae229eba4
1 changed files with 19 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vdpwi/README.md
+19 -2
View File
@@ -14,6 +14,10 @@ To run VDPWI on the SICK dataset, use the following command. If you have any pro
```
python -m vdpwi vdpwi.sick.model.castor --dataset sick --lr 5e-4 --optimizer rmsprop --epochs 15 --batch-size 8 --rnn-hidden-dim 256 --epsilon 1e-7
```
| Implementation and config | Pearson'r | Spearman's p | MSE
| ------------- | ------------- | ------------- | ------------- |
| Paper | 0.8784 | 0.8199 | 0.2329|
| PyTorch Implementation | 0.8710 | 0.8092 | 0.2501
## MSRVID Dataset
@@ -27,10 +31,17 @@ python -m vdpwi vdpwi.msrvid.model.castor --dataset msrvid --batch-size 16 --epo
To run VDPWI on (Raw) TrecQA, you first need to run `./get_trec_eval.sh` in `utils` under the repo root while inside the `utils` directory. This will download and compile the official `trec_eval` tool used for evaluation.
Then, you can run:
```
python -m vdpwi vdpwi.trecqa.model --dataset trecqa --epochs 5 --regularization 0.0005 --eps 0.1
python -m vdpwi vdpwi.trecqa.model --dataset trecqa --epochs 5 --regularization 0.0005 --eps 0.1 --optimizer rmsprop --lr 0.0005 --batch-size 8 --rnn-hidden-dim 256
```
| Implementation and config | MAP | MRR |
| ------------- | ------------- | ------------- |
| Paper | 0.7588 |0.8219
| PyTorch Implementation | 0.7581 | 0.8172|
The paper results are reported in [Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks](https://dl.acm.org/citation.cfm?id=2983872).
## WikiQA Dataset
@@ -38,10 +49,16 @@ The paper results are reported in [Noise-Contrastive Estimation for Answer Selec
You also need `trec_eval` for this dataset, similar to TrecQA.
Then, you can run:
```
python -m vdpwi vdpwi.wikiqa.model --epochs 10 --dataset wikiqa --batch-size 64 --lr 0.0004 --regularization 0.02
python -m vdpwi vdpwi.wikiqa.model --dataset wikiqa --epochs 10 --batch-size 64 --lr 0.0005 --regularization 0.02 --optimizer rmsprop --rnn-hidden-dim 256
```
| Implementation and config | MAP | MRR |
| ------------- | ------------- | ------------- |
| Paper | 0.7090 |0.7234
| PyTorch Implementation | 0.7184 | 0.7286 |
To see all options available, use
```
python -m vdpwi --help