diff --git a/README.md b/README.md index 94342ea..597288e 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,51 @@ # Castor -PyTorch deep learning models. +Deep learning for information retrieval with PyTorch. -1. [SM model](./sm_cnn/): Similarity between question and candidate answers. +## Models +### Baselines + +1. [IDF Baseline](./idf_baseline/): IDF overlap between question and candidate answers + +### Deep Learning Models + +1. [SM-CNN](./sm_cnn/): Ranking short text pairs with Convolutional Neural Networks +2. [Kim CNN](./kim_cnn/): Sentence classification using Convolutional Neural Networks +3. [MP-CNN](./mp_cnn/): Sentence pair modelling with Multi-Perspective Convolutional Neural Networks +4. [NCE](./nce/): Noise-Contrastive Estimation for answer selection applied on SM-CNN and MP-CNN +5. [conv-RNN](./conv_rnn): Convolutional RNN for text modelling ## Setting up PyTorch -You need Python 3.6 to use the models in this repository. +Copy and run the command at https://pytorch.org/ for your environment. PyTorch recommends the Anaconda environment, which we use in our lab. -As per [pytorch.org](pytorch.org), -> "[Anaconda](https://www.continuum.io/downloads) is our recommended package manager" +The typical installation command is -```conda install pytorch torchvision -c soumith``` +```bash +conda install pytorch torchvision -c pytorch +``` -Other pytorch installation modalities (e.g. via ```pip```) can be seen at [pytorch.org](pytorch.org). +## Data and Pre-Trained Models -We also recommend [gensim](https://radimrehurek.com/gensim/). We use some gensim modules to cache word embeddings. +Data associated for use with this repository can be found at: https://git.uwaterloo.ca/jimmylin/Castor-data.git. -```conda install gensim``` +Pre-trained models can be found at: https://github.com/castorini/models.git. +Your directory structure should look like +``` +. +├── Castor +├── Castor-data +└── models +``` -PyTorch has good support for GPU computations. -CUDA installation guide for linux can be found [here](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/) +For example (if you use HTTPS instead of SSH): -**NOTE**: Install CUDA libraries **before** installing conda and pytorch. +```bash +git clone https://github.com/castorini/Castor.git +git clone https://git.uwaterloo.ca/jimmylin/Castor-data.git +git clone https://github.com/castorini/models.git +``` - -## data for models - -Sourcing and pre-processing of input data for each model is described in respective ```model/README.md```'s - -## Baselines - -1. [IDF Baseline](./idf_baseline/): IDF overlap between question and candidate answers. - -## Tutorials - -SM Model tutorial: [sm_cnn/tutorial.ipynb](sm_cnn/tutorial.ipynb) - notebook that walks through SM CNN model, good for beginnners. +Sourcing and pre-processing of input data for each model is described in the respective ```model/README.md```'s. diff --git a/anserini_dependency/README.md b/anserini_dependency/README.md index 0b51ecf..f2575e3 100644 --- a/anserini_dependency/README.md +++ b/anserini_dependency/README.md @@ -1,11 +1,8 @@ ## Setup Retrieve Sentences and end2end QA pipeline -#### 1. Clone [Anserini](https://github.com/castorini/Anserini.git), [Castor](https://github.com/castorini/Castor.git), [data](https://github.com/castorini/data.git), and [models](https://github.com/castorini/models.git): +#### 1. Assuming you've already followed the main [README](../README.md) instructions, just clone [Anserini](https://github.com/castorini/Anserini.git): ```bash git clone https://github.com/castorini/Anserini.git -git clone https://github.com/castorini/Castor.git -git clone https://github.com/castorini/data.git -git clone https://github.com/castorini/models.git ``` Your directory structure should look like @@ -13,7 +10,7 @@ Your directory structure should look like . ├── Anserini ├── Castor -├── data +├── Castor-data └── models ``` @@ -34,22 +31,13 @@ Install the dependency packages: ``` cd Castor -pip3 install -r requirements.txt +pip install -r requirements.txt ``` -Make sure that you have PyTorch installed. For more help, follow [these](https://github.com/castorini/Castor) steps. #### 3. Download Dependencies - Download the TrecQA lucene index - Download the Google word2vec file from [here](https://drive.google.com/drive/folders/0B2u_nClt6NbzNWJkWExmaklYNTA?usp=sharing) -#### 4. Additional files for pipeline: -As some of the files are too large to be uploaded onto GitHub, please download the following files from -[here](https://drive.google.com/drive/folders/0B2u_nClt6NbzNm1LdjlwUFdzQVE?usp=sharing) and place them -in the appropriate locations: - -- copy the contents of `word2vec` directory to `data/word2vec` -- copy `word2dfs.p` to `data/TrecQA/` - ### To run RetrieveSentences: ```bash diff --git a/anserini_dependency/api.py b/anserini_dependency/api.py index b56eb3a..28cc61e 100755 --- a/anserini_dependency/api.py +++ b/anserini_dependency/api.py @@ -51,7 +51,7 @@ def get_answers(question, num_hits, k): parser.add_argument("--scorer", help="passage scores", default="Idf") parser.add_argument("--k", help="top-k passages to be retrieved", default=1) parser.add_argument('--model', help="the path to the saved model file") - parser.add_argument('--dataset', help="the QA dataset folder {TrecQA|WikiQA}", default='../../data/TrecQA/') + parser.add_argument('--dataset', help="the QA dataset folder {TrecQA|WikiQA}", default='../../Castor-data/TrecQA/') parser.add_argument('--no_cuda', action='store_false', help='do not use cuda', dest='cuda') parser.add_argument('--gpu', type=int, default=0) # Use -1 for CPU parser.add_argument('--seed', type=int, default=3435) @@ -109,7 +109,7 @@ if __name__ == "__main__": parser.add_argument('--no_cuda', action='store_false', help='do not use cuda', dest='cuda') parser.add_argument('--gpu', type=int, default=0) # Use -1 for CPU parser.add_argument('--seed', type=int, default=3435) - parser.add_argument('--dataset', help="the QA dataset folder {TrecQA|WikiQA}", default='../../data/TrecQA/') + parser.add_argument('--dataset', help="the QA dataset folder {TrecQA|WikiQA}", default='../../Castor-data/TrecQA/') args = parser.parse_args() if not args.cuda: diff --git a/idf_baseline/README.md b/idf_baseline/README.md index 09fe9f9..f3fc534 100644 --- a/idf_baseline/README.md +++ b/idf_baseline/README.md @@ -4,11 +4,11 @@ Implements IDF baselines for QA datasets. ### Getting the data -Git clone [castorini/data](https://github.com/castorini/data) to get TrecQA and WikiQA datasets. +Assuming you followed instructions in the main [README](../README.md) instructions to clone Castor-data. Follow instructions in ``TrecQA/README.txt`` and ``WikiQA/README.txt`` to process the data into a _standard_ format. -After running the respectve scripts, you should have the following directories structure in ``castorini/data/TrecQA`` +After running the respective scripts, you should have the following directories structure in ``castorini/Castor-data/TrecQA`` ``` ├── raw-dev ├── raw-test @@ -16,7 +16,7 @@ After running the respectve scripts, you should have the following directories s └── train-all ``` -and, the following directories in ``castorini/data/WikiQA``. +and, the following directories in ``castorini/Castor-data/WikiQA``. ``` ├── dev ├── test @@ -138,25 +138,25 @@ eval/trec_eval.9.0/trec_eval -m map -m recip_rank For the WikiQA dataset ``` -../../Anserini/eval/trec_eval.9.0/trec_eval -m map ../../data/WikiQA/WikiQACorpus/WikiQA-$set.ref WikiQA.$set.idfsim +../../Anserini/eval/trec_eval.9.0/trec_eval -m map ../../Castor-data/WikiQA/WikiQACorpus/WikiQA-$set.ref WikiQA.$set.idfsim ``` For the TrecQA dataset ``` -../../Anserini/eval/trec_eval.9.0/trec_eval -m map ../../data/TrecQA/$set.qrel TrecQA.$set.idfsim +../../Anserini/eval/trec_eval.9.0/trec_eval -m map ../../Castor-data/TrecQA/$set.qrel TrecQA.$set.idfsim ``` #### 3. IDF sum similarity using only the QA dataset to compute IDF of terms ``` -python qa-data-idf-only.py ../../data/TrecQA TrecQA -python qa-data-only-idf.py ../../data/WikiQA WikiQA +python qa-data-idf-only.py ../../Castor-data/TrecQA TrecQA +python qa-data-only-idf.py ../../Castor-data/WikiQA WikiQA ``` Evaluate these using step 2. -The same script can now also be used to comput idf sum similarity based on corpus idf statistics +The same script can now also be used to compute idf sum similarity based on corpus idf statistics ``` -python qa-data-only-idf.py ../../data/TrecQA TrecQA --index-for-corpusIDF ../../data/indices/index.qadata.pos.docvectors.keepstopwords/ +python qa-data-only-idf.py ../../Castor-data/TrecQA TrecQA --index-for-corpusIDF ../../Castor-data/indices/index.qadata.pos.docvectors.keepstopwords/ ``` ### Baseline results diff --git a/idf_baseline/experimental_settings.py b/idf_baseline/experimental_settings.py index 3b88acb..cd3c8ba 100644 --- a/idf_baseline/experimental_settings.py +++ b/idf_baseline/experimental_settings.py @@ -122,7 +122,7 @@ if __name__ == "__main__": ap.add_argument("--runall", help="runs all experiments in order", action="store_true") ap.add_argument("index_path", help="required for some combination of experiments") ap.add_argument('qa_data', help="path to the QA dataset", - choices=['../../data/TrecQA', '../../data/WikiQA']) + choices=['../../Castor-data/TrecQA', '../../Castor-data/WikiQA']) args = ap.parse_args() diff --git a/idf_baseline/qa-data-only-idf.py b/idf_baseline/qa-data-only-idf.py index b49a7ee..ce96536 100644 --- a/idf_baseline/qa-data-only-idf.py +++ b/idf_baseline/qa-data-only-idf.py @@ -125,7 +125,7 @@ if __name__ == "__main__": ap = argparse.ArgumentParser(description="uses idf weights from the question-answer pairs only,\ and not from the whole corpus") ap.add_argument('qa_data', help="path to the QA dataset", - choices=['../../data/TrecQA', '../../data/WikiQA']) + choices=['../../Castor-data/TrecQA', '../../Castor-data/WikiQA']) ap.add_argument('outfile_prefix', help="output file prefix") ap.add_argument('--ignore-test', help="does not consider test data when computing IDF of terms", action="store_true") diff --git a/kim_cnn/README.md b/kim_cnn/README.md index 47c12c8..d95d8e2 100644 --- a/kim_cnn/README.md +++ b/kim_cnn/README.md @@ -16,31 +16,12 @@ Assuming you already have PyTorch, just install torchtext (`pip install torchtex ## Quick Start -Clone and create the dataset. -``` -git clone https://github.com/castorini/Castor.git -``` - -``` -. -├── Castor - ├── README.md - ├── baseline_results.tsv - ├── idf_baseline - ├── kim_cnn - ├── mp_cnn - ├── setup.py - ├── sm_cnn - └── sm_modified_cnn -``` - To get the dataset, you can run this. ``` cd kim_cnn bash get_data.sh ``` - To run the model on SST-1 dataset on multichannel, just run the following code. ``` diff --git a/mp_cnn/README.md b/mp_cnn/README.md index cbc598a..d6abd15 100644 --- a/mp_cnn/README.md +++ b/mp_cnn/README.md @@ -4,21 +4,7 @@ This is a PyTorch implementation of the following paper * Hua He, Kevin Gimpel, and Jimmy Lin. [Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks](http://aclweb.org/anthology/D/D15/D15-1181.pdf). *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)*, pages 1576-1586. -The SICK and MSRVID datasets are available in https://github.com/castorini/data, as well as the GloVe word embeddings. - -Directory layout should be like this: -``` -├── Castor -│ ├── README.md -│ ├── ... -│ └── mp_cnn/ -├── data -│ ├── README.md -│ ├── ... -│ ├── msrvid/ -│ ├── sick/ -│ └── GloVe/ -``` +Please ensure you have followed instructions in the main [README](../README.md) doc before running any further commands in this doc. ## SICK Dataset diff --git a/mp_cnn/dataset.py b/mp_cnn/dataset.py index 0ffb2a0..67d9b0c 100644 --- a/mp_cnn/dataset.py +++ b/mp_cnn/dataset.py @@ -31,14 +31,14 @@ class MPCNNDatasetFactory(object): @staticmethod def get_dataset(dataset_name, word_vectors_dir, word_vectors_file, batch_size, device, castor_dir="../", utils_trecqa="utils/trec_eval-9.0.5/trec_eval"): if dataset_name == 'sick': - dataset_root = os.path.join(os.pardir, castor_dir, 'data', 'sick/') + dataset_root = os.path.join(castor_dir, os.pardir, 'Castor-data', 'sick/') train_loader, dev_loader, test_loader = SICK.iters(dataset_root, word_vectors_file, word_vectors_dir, batch_size, device=device, unk_init=UnknownWordVecCache.unk) embedding_dim = SICK.TEXT_FIELD.vocab.vectors.size() embedding = nn.Embedding(embedding_dim[0], embedding_dim[1]) embedding.weight = nn.Parameter(SICK.TEXT_FIELD.vocab.vectors) return SICK, embedding, train_loader, test_loader, dev_loader elif dataset_name == 'msrvid': - dataset_root = os.path.join(os.pardir, castor_dir, 'data', 'msrvid/') + dataset_root = os.path.join(castor_dir, os.pardir, 'Castor-data', 'msrvid/') dev_loader = None train_loader, test_loader = MSRVID.iters(dataset_root, word_vectors_file, word_vectors_dir, batch_size, device=device, unk_init=UnknownWordVecCache.unk) embedding_dim = MSRVID.TEXT_FIELD.vocab.vectors.size() @@ -48,7 +48,7 @@ class MPCNNDatasetFactory(object): elif dataset_name == 'trecqa': if not os.path.exists(os.path.join(castor_dir, utils_trecqa)): raise FileNotFoundError('TrecQA requires the trec_eval tool to run. Please run get_trec_eval.sh inside Castor/utils (as working directory) before continuing.') - dataset_root = os.path.join(os.pardir, castor_dir, 'data', 'TrecQA/') + dataset_root = os.path.join(castor_dir, os.pardir, 'Castor-data', 'TrecQA/') train_loader, dev_loader, test_loader = TRECQA.iters(dataset_root, word_vectors_file, word_vectors_dir, batch_size, device=device, unk_init=UnknownWordVecCache.unk) embedding_dim = TRECQA.TEXT_FIELD.vocab.vectors.size() embedding = nn.Embedding(embedding_dim[0], embedding_dim[1]) @@ -57,7 +57,7 @@ class MPCNNDatasetFactory(object): elif dataset_name == 'wikiqa': if not os.path.exists(os.path.join(castor_dir, utils_trecqa)): raise FileNotFoundError('TrecQA requires the trec_eval tool to run. Please run get_trec_eval.sh inside Castor/utils (as working directory) before continuing.') - dataset_root = os.path.join(os.pardir, castor_dir, 'data', 'WikiQA/') + dataset_root = os.path.join(castor_dir, os.pardir, 'Castor-data', 'WikiQA/') train_loader, dev_loader, test_loader = WikiQA.iters(dataset_root, word_vectors_file, word_vectors_dir, batch_size, device=device, unk_init=UnknownWordVecCache.unk) embedding_dim = WikiQA.TEXT_FIELD.vocab.vectors.size() embedding = nn.Embedding(embedding_dim[0], embedding_dim[1]) diff --git a/mp_cnn/main.py b/mp_cnn/main.py index a91a300..c89fe80 100644 --- a/mp_cnn/main.py +++ b/mp_cnn/main.py @@ -18,7 +18,7 @@ if __name__ == '__main__': parser = argparse.ArgumentParser(description='PyTorch implementation of Multi-Perspective CNN') parser.add_argument('model_outfile', help='file to save final model') parser.add_argument('--dataset', help='dataset to use, one of [sick, msrvid, trecqa, wikiqa]', default='sick') - parser.add_argument('--word-vectors-dir', help='word vectors directory', default=os.path.join(os.pardir, os.pardir, 'data', 'GloVe')) + parser.add_argument('--word-vectors-dir', help='word vectors directory', default=os.path.join(os.pardir, os.pardir, 'Castor-data', 'embeddings', 'GloVe')) parser.add_argument('--word-vectors-file', help='word vectors filename', default='glove.840B.300d.txt') parser.add_argument('--skip-training', help='will load pre-trained model', action='store_true') parser.add_argument('--device', type=int, default=0, help='GPU device, -1 for CPU (default: 0)') diff --git a/nce/nce_pairwise_mp/README.md b/nce/nce_pairwise_mp/README.md index f9dbc0a..90c3ffd 100644 --- a/nce/nce_pairwise_mp/README.md +++ b/nce/nce_pairwise_mp/README.md @@ -5,22 +5,7 @@ This is a PyTorch implementation of the following paper * Hua He, Kevin Gimpel, and Jimmy Lin. [Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks](http://aclweb.org/anthology/D/D15/D15-1181.pdf). *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)*, pages 1576-1586. * Jinfeng Rao, Hua He, and Jimmy Lin. [Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks.](http://dl.acm.org/citation.cfm?id=2983872) *Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (CIKM 2016)*, pages 1913-1916. - -The SICK and MSRVID datasets are available in https://github.com/castorini/data, as well as the GloVe word embeddings. - -Directory layout should be like this: -``` -├── Castor -│ ├── README.md -│ ├── ... -│ └── mp_cnn/ -├── data -│ ├── README.md -│ ├── ... -│ ├── msrvid/ -│ ├── sick/ -│ └── GloVe/ -``` +Please ensure you have followed instructions in the main [README](../README.md) doc before running any further commands in this doc. ## TrecQA Dataset diff --git a/nce/nce_pairwise_mp/main.py b/nce/nce_pairwise_mp/main.py index ee57d56..32b39f8 100644 --- a/nce/nce_pairwise_mp/main.py +++ b/nce/nce_pairwise_mp/main.py @@ -18,7 +18,7 @@ if __name__ == '__main__': parser = argparse.ArgumentParser(description='PyTorch implementation of Multi-Perspective CNN') parser.add_argument('model_outfile', help='file to save final model') parser.add_argument('--dataset', help='dataset to use, one of [sick, msrvid, trecqa, wikiqa]', default='sick') - parser.add_argument('--word-vectors-dir', help='word vectors directory', default=os.path.join(os.pardir, os.pardir, os.pardir, 'data', 'GloVe')) + parser.add_argument('--word-vectors-dir', help='word vectors directory', default=os.path.join(os.pardir, os.pardir, os.pardir, 'Castor-data', 'embeddings', 'GloVe')) parser.add_argument('--word-vectors-file', help='word vectors filename', default='glove.840B.300d.txt') parser.add_argument('--skip-training', help='will load pre-trained model', action='store_true') parser.add_argument('--device', type=int, default=0, help='GPU device, -1 for CPU (default: 0)') diff --git a/nce/nce_pairwise_sm/args.py b/nce/nce_pairwise_sm/args.py index 1f03e7d..cc97ab4 100644 --- a/nce/nce_pairwise_sm/args.py +++ b/nce/nce_pairwise_sm/args.py @@ -20,7 +20,7 @@ def get_args(): parser.add_argument('--words_dim', type=int, default=50) parser.add_argument('--dropout', type=float, default=0.5) parser.add_argument('--epoch_decay', type=int, default=15) - parser.add_argument('--wordvec_dir', type=str, default='../../../data/word2vec/') + parser.add_argument('--wordvec_dir', type=str, default='../../../Castor-data/embeddings/word2vec/') parser.add_argument('--vector_cache', type=str, default='word2vec.trecqa.pt') parser.add_argument('--trained_model', type=str, default="") parser.add_argument('--weight_decay',type=float, default=1e-5) diff --git a/nce/nce_pairwise_sm/main.py b/nce/nce_pairwise_sm/main.py index 9862535..6061386 100644 --- a/nce/nce_pairwise_sm/main.py +++ b/nce/nce_pairwise_sm/main.py @@ -38,10 +38,10 @@ if torch.cuda.is_available() and not args.cuda: if args.dataset == "trec": dataset_cls = TRECQA - dataset_root = os.path.join(os.pardir, os.pardir, os.pardir, 'data', 'TrecQA/') + dataset_root = os.path.join(os.pardir, os.pardir, os.pardir, 'Castor-data', 'embeddings', 'TrecQA/') elif args.dataset == "wiki": dataset_cls = WikiQA - dataset_root = os.path.join(os.pardir, os.pardir, os.pardir, 'data', 'WikiQA/') + dataset_root = os.path.join(os.pardir, os.pardir, os.pardir, 'Castor-data', 'embeddings', 'WikiQA/') else: logger.info("Unsupported dataset") exit() diff --git a/nce/nce_pairwise_sm/overlap_features.py b/nce/nce_pairwise_sm/overlap_features.py index 9996bac..118ba4e 100644 --- a/nce/nce_pairwise_sm/overlap_features.py +++ b/nce/nce_pairwise_sm/overlap_features.py @@ -102,10 +102,10 @@ def compute_dfs(docs): if __name__ == '__main__': parser = ArgumentParser(description='create TrecQA/WikiQA dataset') - parser.add_argument('--dir', help='path to the TrecQA|WikiQA data directory', default="../../data/TrecQA") + parser.add_argument('--dir', help='path to the TrecQA|WikiQA data directory', default="../../Castor-data/TrecQA") args = parser.parse_args() - stoplist = set([line.strip() for line in open('../../data/TrecQA/stopwords.txt', encoding='utf-8')]) + stoplist = set([line.strip() for line in open('../../Castor-data/TrecQA/stopwords.txt', encoding='utf-8')]) punct = set(string.punctuation) stoplist.update(punct) diff --git a/sm_cnn/README.md b/sm_cnn/README.md index 987aa66..1acf41a 100644 --- a/sm_cnn/README.md +++ b/sm_cnn/README.md @@ -5,54 +5,7 @@ Networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '15). ACM, New York, NY, USA, 373-382. DOI: http://dx.doi.org/10.1145/2766462.2767738 - -### Setup -Clone and create the dataset: -```bash -git clone https://github.com/castorini/data.git -git clone https://github.com/castorini/Castor.git -``` - -You should you see the following tree: -``` -. -├── Castor -│   ├── README.md -│   ├── baseline_results.tsv -│   ├── idf_baseline -│   ├── kim_cnn -│   ├── mp_cnn -│   ├── setup.py -│   ├── sm_cnn -└── data - ├── GloVe - ├── ParagramEmbeddings - ├── README.md - ├── SimpleQuestions_v2 - ├── TrecQA - ├── WikiQA - ├── msrvid - ├── requirements.txt - ├── sick - ├── twitterPPDB - ├── utils - └── word2vec -``` - -Parse the TrecQA datset: -```bash -cd ../../data/TrecQA/ -python parse.py -cd - -``` - -Parse the WikiQA datset: -```bash -cd ../../data/WikiQA/ -unzip WikiQACorpus.zip -python create-train-dev-test-data.py -cd - -``` +Please ensure you have followed instructions in the main [README](../README.md) doc before running any further commands in this doc. Your repository root should be in your `PYTHONPATH` environment variable: ```bash @@ -65,7 +18,6 @@ cd Castor/sm_cnn/ ./create_dataset.sh ``` - We use `trec_eval` for evaluation: ```bash @@ -144,7 +96,7 @@ NB: The results on WikiQA are based on the SM model hyperparameters. to the `data/` folder ```bash -python $PYTHONPATH/utils/build_w2v.py --input data/aquaint+wiki.txt.gz.ndim=50.bin +python $PYTHONPATH/utils/build_w2v.py --input ../../Castor-data/embeddings/word2vec/aquaint+wiki.txt.gz.ndim=50.bin ``` -Note that `$PYTHONPATH` holds the location of the repository root. \ No newline at end of file +Note that `$PYTHONPATH` holds the location of the repository root. diff --git a/sm_cnn/bridge.py b/sm_cnn/bridge.py index 33de3cd..3b0e3a1 100644 --- a/sm_cnn/bridge.py +++ b/sm_cnn/bridge.py @@ -110,7 +110,7 @@ if __name__ == "__main__": parser = argparse.ArgumentParser(description="Bridge Demo. Produces scores in trec_eval format", formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('--model', help="the path to the saved model file") - parser.add_argument('--dataset', help="the QA dataset folder {TrecQA|WikiQA}", default='../../data/TrecQA/') + parser.add_argument('--dataset', help="the QA dataset folder {TrecQA|WikiQA}", default='../../Castor-data/TrecQA/') parser.add_argument("--index", help="Lucene index", required=True) parser.add_argument("--embeddings", help="Path of the word2vec index", default="") parser.add_argument("--topics", help="topics file", default="") diff --git a/sm_cnn/create_dataset.sh b/sm_cnn/create_dataset.sh index e8d7903..f786fed 100755 --- a/sm_cnn/create_dataset.sh +++ b/sm_cnn/create_dataset.sh @@ -1,16 +1,16 @@ #!/bin/sh mkdir -p data -python overlap_features.py --dir ../../data/TrecQA/ +python overlap_features.py --dir ../../Castor-data/TrecQA/ CURRENT_DIR=$(pwd) -cd ../../data/TrecQA +cd ../../Castor-data/TrecQA cd raw-dev/; paste id.txt sim.txt a.toks b.toks overlap_feats.txt > $CURRENT_DIR/data/trecqa.dev.tsv; cd .. cd raw-test/; paste id.txt sim.txt a.toks b.toks overlap_feats.txt > $CURRENT_DIR/data/trecqa.test.tsv; cd .. cd train-all/; paste id.txt sim.txt a.toks b.toks overlap_feats.txt > $CURRENT_DIR/data/trecqa.train.tsv; cd .. cd $CURRENT_DIR -python overlap_features.py --dir ../../data/WikiQA/ -cd ../../data/WikiQA +python overlap_features.py --dir ../../Castor-data/WikiQA/ +cd ../../Castor-data/WikiQA cd dev/; paste id.txt sim.txt a.toks b.toks overlap_feats.txt > $CURRENT_DIR/data/wikiqa.dev.tsv; cd .. cd test/; paste id.txt sim.txt a.toks b.toks overlap_feats.txt > $CURRENT_DIR/data/wikiqa.test.tsv; cd .. cd train/; paste id.txt sim.txt a.toks b.toks overlap_feats.txt > $CURRENT_DIR/data/wikiqa.train.tsv; cd .. diff --git a/sm_cnn/overlap_features.py b/sm_cnn/overlap_features.py index 9a799e3..5937157 100644 --- a/sm_cnn/overlap_features.py +++ b/sm_cnn/overlap_features.py @@ -102,10 +102,10 @@ def compute_dfs(docs): if __name__ == '__main__': parser = ArgumentParser(description='create TrecQA/WikiQA dataset') - parser.add_argument('--dir', help='path to the TrecQA|WikiQA data directory', default="../../data/TrecQA") + parser.add_argument('--dir', help='path to the TrecQA|WikiQA data directory', default="../../Castor-data/TrecQA") args = parser.parse_args() - stoplist = set([line.strip() for line in open('../../data/TrecQA/stopwords.txt', encoding='utf-8')]) + stoplist = set([line.strip() for line in open('../../Castor-data/TrecQA/stopwords.txt', encoding='utf-8')]) punct = set(string.punctuation) stoplist.update(punct)