wassname/Castor
1
0
Fork 0
mirror of https://github.com/wassname/Castor.git synced 2026-09-10 11:40:44 +08:00
Code Issues Packages Projects Releases Wiki Activity

Replication of STOA for Reuters Dataset (#152)

Browse Source 
* Add ReutersTrainer, ReutersEvaluator options in Factory classes

* Add Reuters to Kim-CNN command line arguments

* Fix SST dataset path according to changes in Kim-CNN args

The dataset path in args.py was made to point at the dataset folder rather than dataset/SST folder. Hence SST folder was added to paths in the SST dataset class

* Add Reuters dataset class, and support in __main__

* Add Reuters dataset trainers and evaluators

* Remove debug print statement in reuters_evaluator

* Fix rounding bug in reuters_trainer and reuters_evaluator

* Add LSTM for baseline text classification measurements

* Add eval metrics for lstm_baseline

* Set batch_first param in lstm_baseline

* Remove onnx args from lstm_baseline

* Pack padded sequences in LSTM_baseline

* Add TensorBoardX support for Reuters trainer

* Add Arxiv Academic Paper Dataset (AAPD)

* Add Hidden Bottleneck Layer to BiLSTM

* Fix packing of padded tensors in Reuters

* Add cmdline args for Hidden Bottleneck Layer for BiLSTM

* Include pre-padding lengths in AAPD dataset

* Remove duplication of preprocessing code in AAPD

* Remove batch_size condition in ReutersTrainer
This commit is contained in:
Achyudh Ram
2018-10-26 19:10:19 -04:00
committed by Peng Shi
parent 650882fb6e
commit 6daa5a128f
10 changed files with 121 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common/dataset.py
+6
View File
@@ -12,6 +12,7 @@ from datasets.snli import SNLI
from datasets.sts2014 import STS2014
from datasets.quora import Quora
from datasets.reuters import Reuters
from datasets.aapd import AAPD
class UnknownWordVecCache(object):
"""
@@ -92,6 +93,11 @@ class DatasetFactory(object):
train_loader, dev_loader, test_loader = Reuters.iters(dataset_root, word_vectors_file, word_vectors_dir, batch_size, device=device, unk_init=UnknownWordVecCache.unk)
embedding = nn.Embedding.from_pretrained(Reuters.TEXT_FIELD.vocab.vectors)
return Reuters, embedding, train_loader, test_loader, dev_loader
elif dataset_name == 'aapd':
dataset_root = os.path.join(castor_dir, os.pardir, 'Castor-data', 'datasets', 'AAPD/')
train_loader, dev_loader, test_loader = AAPD.iters(dataset_root, word_vectors_file, word_vectors_dir, batch_size, device=device, unk_init=UnknownWordVecCache.unk)
embedding = nn.Embedding.from_pretrained(AAPD.TEXT_FIELD.vocab.vectors)
return AAPD, embedding, train_loader, test_loader, dev_loader
else:
raise ValueError('{} is not a valid dataset.'.format(dataset_name))