diff --git a/kim_cnn/README.md b/kim_cnn/README.md index 61916a2..36848f1 100644 --- a/kim_cnn/README.md +++ b/kim_cnn/README.md @@ -1,130 +1,58 @@ -# text-classification-cnn -Implementation for Convolutional Neural Networks for Sentence Classification of [Kim (2014)](https://arxiv.org/abs/1408.5882) with PyTorch. +# kim_cnn -## Project Structure - -``` -kim-cnn - ├── config - │ └── classification.cfg - ├── etc - │ ├── kmeans.py - │ └── utils.py - ├── model - │ └── cnnText - │ └── cnntext.py - ├── network - │ └── cnnTextNetwork.py - ├── data - ├── saves - ├── README.md - ├── getData.sh - ├── bucket.py - ├── configurable.py - ├── dataset.py - ├── example.py - ├── vocab.py - └── main.py - -``` +Implementation for Convolutional Neural Networks for Sentence Classification of [Kim (2014)](https://arxiv.org/abs/1408.5882) with PyTorch and Torchtext. ## Model Type - rand: All words are randomly initialized and then modified during training. -- static: A model with pre-trained vectors from [word2vec](https://code.google.com/archive/p/word2vec/). All words -- including the unknown ones that are randomly initialized -- are kept static and only the other parameters of the model are learned. +- static: A model with pre-trained vectors from [word2vec](https://code.google.com/archive/p/word2vec/). All words -- including the unknown ones that are initialized with zero -- are kept static and only the other parameters of the model are learned. - non-static: Same as above but the pretrained vectors are fine-tuned for each task. -- multichannel: A model with two sets of word vectors. Each set of vectors is treated as a 'channel' and each filter is applied to both channels, but gradients are back-propagated only through one of the channels. Hence the model is able to fine-tune one set of vectors while keeping the other static. Both channels are initialized with word2vec. - +- multichannel: A model with two sets of word vectors. Each set of vectors is treated as a 'channel' and each filter is applied to both channels, but gradients are back-propagated only through one of the channels. Hence the model is able to fine-tune one set of vectors while keeping the other static. Both channels are initialized with word2vec.# text-classification-cnn +Implementation for Convolutional Neural Networks for Sentence Classification of [Kim (2014)](https://arxiv.org/abs/1408.5882) with PyTorch. ## Quick Start -Run + +To run the model on [SST-1] dataset on [multichannel](Model Type), just run the following code. ``` -bash getData.sh -``` - -to get dataset. - -To run the model on [TREC](http://cogcomp.cs.illinois.edu/Data/QA/QC/) dataset on [rand](Model Type), just run the following code. - -``` -python main.py --config_file config/trec.cfg --model_type CNNText --train +python train.py --mode multichannel ``` The file will be saved in ``` -saves/model_file +saves/best_model.pt ``` -You can modify these parameters under these [instructions](Configurable File) To test the model, you can use the following command. ``` -python main.py --config_file config/trec.cfg --model_type CNNText --test --restore_from saves/trec/model_file +python main.py --trained_model saves/best_model.pt --mode multichannel ``` -You need to specify the config file and the path to model file here. Note: The path need to be the same as the declaration in the config file. - -## Configurable File - -- model_type: **CNNText** in this case. -- mode: **rand**, **static**, **non-static** and **multichannel** which are specified [here](Model Type) -- save_dir: the path you want to save the model parameters -- word_file: all words appearing in the dataset -- target_file: all labels appearing in the dataset -- data_dir: the path of dataset -- train_file: the name of the training file -- valid_file: the name of the validation file -- test_file: the name of the test file -- save_model_file: the name you want to use for the model parameters -- restore_from: this option will be used we you want to restore file for validation and testing -- embed_file: embedding file -- use_gpu: use GPU or not -- words_dim: the dimension of the words. This is same with pre-trained word embedding. -- n_bkts: the bucket number for training dataset. Group the sentence according to the lengths -- n_valid_bkts: the bucket number for testing dataset. -- dataset_type: the dataset you used. **TREC**, **SST-1** and **SST-2** in this case. -- min_occur_count: set the word as *UNK* according to its frequency. -- learning_rate: leanring rate -- epoch_decay: decay the learning rate 0.75 for every *epoch_dacay* epoch -- valid_interval: validate for every *valid_interval* iterations -- train_batch_size: the token number for each batch in training -- test_batch_size: the token number for each batch in testing ## Dataset and Embeddings We experiment the model on the following three datasets. -- TREC: We use the 1-5000 for training, 5001-5452 for validation, and original test dataset for testing. - SST-1: Keep the original splits and train with phrase level dataset and test on sentence level dataset. -- SST-2: Same as above. - -Furthermore, we filter the word embeddings to fit specific dataset. These file can be found in dir *data*. - -For self-defined data, please keep the format as - -``` -label1 sentence -label2 sentence -``` - -And you can use [this](http://ocp59jkku.bkt.clouddn.com/filterVec.py) script to filter the pre-trained word embeddings. - -Or you can modify the *reading_dataset* in *dataset.py*, *add_train_file* in *vocab.py* and *example.py* to fit your own dataset. ## Results - +### best dev |dataset|rand|static|non-static|multichannel| |---|---|---|---|---| -|TREC|91.98|90.32|92.62|93.36| -|SST-1|42.59|46.33|44.32|47.32| -|SST-2|82.20|86.42|85.43|84.39| +|SST-1|43.142598|48.773842|49.137148|49.318801| + + +### test +|dataset|rand|static|non-static|multichannel| +|---|---|---|---|---| +|SST-1|39.909502|46.380090|45.294118|48.416290| We do not tune the parameters for each dataset. And the implementation is simplified from the original version on regularization. diff --git a/kim_cnn/SST1.py b/kim_cnn/SST1.py new file mode 100644 index 0000000..e8da083 --- /dev/null +++ b/kim_cnn/SST1.py @@ -0,0 +1,15 @@ +from torchtext import data +import os + + +class SST1Dataset(data.TabularDataset): + dirname = 'data' + @classmethod + def splits(cls, text_field, label_field, + train='phrases.train.tsv', validation='dev.tsv', test='test.tsv'): + prefix_name = 'stsa.fine.' + path = './data' + return super(SST1Dataset, cls).splits( + os.path.join(path, prefix_name), train, validation, test, + format='TSV', fields=[('label', label_field), ('text', text_field)] + ) \ No newline at end of file diff --git a/kim_cnn/args.py b/kim_cnn/args.py new file mode 100644 index 0000000..6e12269 --- /dev/null +++ b/kim_cnn/args.py @@ -0,0 +1,31 @@ +import os + +from argparse import ArgumentParser + +def get_args(): + parser = ArgumentParser(description="Kim CNN") + parser.add_argument('--no_cuda', action='store_false', help='do not use cuda', dest='cuda') + parser.add_argument('--gpu', type=int, default=0) # Use -1 for CPU + parser.add_argument('--epochs', type=int, default=30) + parser.add_argument('--batch_size', type=int, default=1000) + parser.add_argument('--mode', type=str, default='multichannel') + parser.add_argument('--lr', type=float, default=1.0) + parser.add_argument('--seed', type=int, default=3435) + parser.add_argument('--dataset', type=str, default='SST-1') + parser.add_argument('--resume_snapshot', type=str, default=None) + parser.add_argument('--dev_every', type=int, default=30) + parser.add_argument('--log_every', type=int, default=10) + parser.add_argument('--patience', type=int, default=50) + parser.add_argument('--save_path', type=str, default='saves') + parser.add_argument('--output_channel', type=int, default=100) + parser.add_argument('--words_dim', type=int, default=300) + parser.add_argument('--embed_dim', type=int, default=300) + parser.add_argument('--dropout', type=float, default=0.5) + parser.add_argument('--epoch_decay', type=int, default=15) + parser.add_argument('--vector_cache', type=str, default="data/word2vec.sst-1.pt") + parser.add_argument('--trained_model', type=str, default="") + parser.add_argument('--weight_decay',type=float, default=0) + + + args = parser.parse_args() + return args diff --git a/kim_cnn/bucket.py b/kim_cnn/bucket.py deleted file mode 100644 index 9587a86..0000000 --- a/kim_cnn/bucket.py +++ /dev/null @@ -1,87 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - - -from configurable import Configurable -import numpy as np - - -class Bucket(Configurable): - """ - """ - def __init__(self, *args, **kwargs): - """ - - :param args: - :param kwargs: - """ - super(Bucket, self).__init__(*args, **kwargs) - self._size = None - self._data = None - self._sents = None - self._target = None - - def set_size(self, size): - self._size = size - self._data = [] - self._sents = [] - self._target = [] - - def add(self, example): - # TODO: After finalize, we can not add data anymore - if example.length > self._size: #and self._size != -1: - # TODO: we may support size = -1 in the future - raise ValueError("Bucket of size %d received sequence of len %d" % (self._size, example.length)) - self._data.append(example.data['words']) - self._sents.append(example.sent['words']) - self._target.append(example.data['targets']) - return len(self._data)-1 - - def finalize(self): - if self._data is None: - raise ValueError("You need to set size before finalize it") - if len(self._data) > 0: - shape = (len(self._data), self._size, len(self._data[-1][-1])) - data = np.zeros(shape, dtype=np.int64) - for i, datum in enumerate(self._data): - try: - datum = np.array(datum) - data[i,0:len(datum)] = datum - except: - print("sentence %d has Error with data :"%(i+1)) - print(datum) - exit() - self._data = data - self._sents = np.array(self._sents) - self._target = np.array(self._target) - - - else: - print("Finalize Error in bucket") - exit() - print("Bucket %s is %d x %d" % ((self._name,) + self._data.shape[0:2])) - - - def __len__(self): - return len(self._data) - - @property - def size(self): - return self._size - @property - def data(self): - return self._data - @property - def sents(self): - return self._sents - @property - def target(self): - return self._target - - - - - - - - diff --git a/kim_cnn/config/sst-1.cfg b/kim_cnn/config/sst-1.cfg deleted file mode 100644 index 95fbdfd..0000000 --- a/kim_cnn/config/sst-1.cfg +++ /dev/null @@ -1,34 +0,0 @@ -[OS] -model_type = CNNText -mode = multichannel -save_dir = saves/sst-1 -word_file = %(save_dir)s/words.txt -target_file = %(save_dir)s/targets.txt -data_dir = data -train_file = %(data_dir)s/stsa.fine.phrases.train -valid_file = %(data_dir)s/stsa.fine.dev -test_file = %(data_dir)s/stsa.fine.test -save_model_file = %(save_dir)s/model_file -restore_from = %(save_dir)s/model_file -embed_file = %(data_dir)s/word2vec.sst-1 -use_gpu = False - -[Sizes] -words_dim = 300 - -[Dataset] -n_bkts = 10 -n_valid_bkts = 3 -dataset_type = SST-1 -min_occur_count = 2 - -[Learning rate] -learning_rate = 1e-3 -epoch_decay = 30 -dropout = 0.5 - -[Training] -log_interval = 10 -valid_interval = 100 -train_batch_size = 2000 -test_batch_size = 2000 \ No newline at end of file diff --git a/kim_cnn/config/sst-2.cfg b/kim_cnn/config/sst-2.cfg deleted file mode 100644 index 78663a1..0000000 --- a/kim_cnn/config/sst-2.cfg +++ /dev/null @@ -1,34 +0,0 @@ -[OS] -model_type = CNNText -mode = multichannel -save_dir = saves/sst-2 -word_file = %(save_dir)s/words.txt -target_file = %(save_dir)s/targets.txt -data_dir = data -train_file = %(data_dir)s/stsa.binary.phrases.train -valid_file = %(data_dir)s/stsa.binary.dev -test_file = %(data_dir)s/stsa.binary.test -save_model_file = %(save_dir)s/model_file -restore_from = %(save_dir)s/model_file -embed_file = %(data_dir)s/word2vec.sst-2 -use_gpu = False - -[Sizes] -words_dim = 300 - -[Dataset] -n_bkts = 10 -n_valid_bkts = 3 -dataset_type = SST-2 -min_occur_count = 2 - -[Learning rate] -learning_rate = 2e-3 -epoch_decay = 30 -dropout = 0.5 - -[Training] -log_interval = 10 -valid_interval = 100 -train_batch_size = 2000 -test_batch_size = 2000 \ No newline at end of file diff --git a/kim_cnn/config/trec.cfg b/kim_cnn/config/trec.cfg deleted file mode 100644 index c0758a2..0000000 --- a/kim_cnn/config/trec.cfg +++ /dev/null @@ -1,34 +0,0 @@ -[OS] -model_type = CNNText -mode = rand -save_dir = saves/trec -word_file = %(save_dir)s/words.txt -target_file = %(save_dir)s/targets.txt -data_dir = data -train_file = %(data_dir)s/train.trec -valid_file = %(data_dir)s/validate.trec -test_file = %(data_dir)s/test.trec -save_model_file = %(save_dir)s/model_file -restore_from = %(save_dir)s/model_file -embed_file = %(data_dir)s/word2vec.trec -use_gpu = False - -[Sizes] -words_dim = 300 - -[Dataset] -n_bkts = 10 -n_valid_bkts = 3 -dataset_type = TREC -min_occur_count = 2 - -[Learning rate] -learning_rate = 2e-3 -epoch_decay = 30 -dropout = 0.5 - -[Training] -log_interval = 10 -valid_interval = 100 -train_batch_size = 2000 -test_batch_size = 2000 \ No newline at end of file diff --git a/kim_cnn/configurable.py b/kim_cnn/configurable.py deleted file mode 100644 index 28d5f01..0000000 --- a/kim_cnn/configurable.py +++ /dev/null @@ -1,166 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function - -import os -import argparse - -from configparser import SafeConfigParser - - -class Configurable(object): - """ - Configuration processing for the network - """ - def __init__(self, *args, **kwargs): - self._name = kwargs.pop("name", "Unknown") - if args and kwargs: - raise TypeError('Configurable must take either a config parser or keyword args') - if len(args) > 1: - raise TypeError('Configurable takes at most one argument') - if args: - self._config = args[0] - else: - self._config = self._configure(**kwargs) - return - - @property - def name(self): - return self._name - - - def _configure(self, **kwargs): - config = SafeConfigParser() - config_file = kwargs.pop("config_file", "") - config.read(config_file) - # Override the config setting if the (k,v) specified in command line - for option, value in kwargs.items(): - assigned = False - for section in config.sections(): - if option in config.options(section): - config.set(section, option, str(value)) - assigned = True - break - if not assigned: - raise ValueError("%s is not a valid option" % option) - return config - - argparser = argparse.ArgumentParser() - argparser.add_argument('--config_file') - - # ====== - # [OS] - @property - def model_type(self): - return self._config.get('OS', 'model_type') - argparser.add_argument('--model_type') - @property - def mode(self): - return self._config.get('OS', 'mode') - argparser.add_argument('--mode') - @property - def save_dir(self): - return self._config.get('OS', 'save_dir') - argparser.add_argument('--save_dir') - @property - def word_file(self): - return self._config.get('OS', 'word_file') - argparser.add_argument('--word_file') - @property - def target_file(self): - return self._config.get('OS', 'target_file') - argparser.add_argument('--target_file') - @property - def train_file(self): - return self._config.get('OS', 'train_file') - argparser.add_argument('--train_file') - @property - def valid_file(self): - return self._config.get('OS', 'valid_file') - argparser.add_argument('--valid_file') - @property - def test_file(self): - return self._config.get('OS', 'test_file') - argparser.add_argument('--test_file') - @property - def save_model_file(self): - return self._config.get('OS', 'save_model_file') - argparser.add_argument('--save_model_file') - @property - def restore_from(self): - return self._config.get('OS', 'restore_from') - argparser.add_argument('--restore_from') - @property - def embed_file(self): - return self._config.get('OS', 'embed_file') - argparser.add_argument('--embed_file') - @property - def use_gpu(self): - return self._config.getboolean('OS', 'use_gpu') - argparser.add_argument('--use_gpu') - - - # [Dataset] - @property - def n_bkts(self): - return self._config.getint('Dataset', 'n_bkts') - argparser.add_argument('--n_bkts') - @property - def n_valid_bkts(self): - return self._config.getint('Dataset', 'n_valid_bkts') - argparser.add_argument('--n_valid_bkts') - @property - def dataset_type(self): - return self._config.get('Dataset', 'dataset_type') - argparser.add_argument('--dataset_type') - @property - def min_occur_count(self): - return self._config.getint('Dataset', 'min_occur_count') - argparser.add_argument('--min_occur_count') - - - - # [Learning rate] - @property - def learning_rate(self): - return self._config.getfloat('Learning rate', 'learning_rate') - argparser.add_argument('--learning_rate') - @property - def epoch_decay(self): - return self._config.getint('Learning rate', 'epoch_decay') - argparser.add_argument('--epoch_decay') - @property - def dropout(self): - return self._config.getfloat('Learning rate', 'dropout') - argparser.add_argument('--dropout') - - # [Sizes] - @property - def words_dim(self): - return self._config.getint('Sizes', 'words_dim') - argparser.add_argument('--words_dim') - - - # [Training] - @property - def log_interval(self): - return self._config.getint('Training', 'log_interval') - argparser.add_argument('--log_interval') - @property - def valid_interval(self): - return self._config.getint('Training', 'valid_interval') - argparser.add_argument('--valid_interval') - @property - def train_batch_size(self): - return self._config.getint('Training', 'train_batch_size') - argparser.add_argument('--train_batch_size') - @property - def test_batch_size(self): - return self._config.getint('Training', 'test_batch_size') - argparser.add_argument('--test_batch_size') - - - diff --git a/kim_cnn/dataset.py b/kim_cnn/dataset.py deleted file mode 100644 index 6d1f4a6..0000000 --- a/kim_cnn/dataset.py +++ /dev/null @@ -1,145 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -from configurable import Configurable -from bucket import Bucket -from example import Example -from collections import Counter -from etc.kmeans import KMeans -import numpy as np -from etc.utils import clean_str, clean_str_sst - -class Dataset(Configurable): - """ - Dataset Class: - - Store Data - - Generate Minibatch - - Padding - """ - def __init__(self, filename, vocabs, *args, **kwargs): - - super(Dataset, self).__init__(*args, **kwargs) - self._train = (filename == self.train_file) - self.vocabs = vocabs - self.buckets = [Bucket(self._config, name='Sents-%d' % i) for i in range(self.n_bkts)] - self.id2position = [] - self.len2bkts = {} - self.vocabs = vocabs - self.reading_dataset(filename) - self._finalize() - - def _finalize(self): - for bucket in self.buckets: - bucket.finalize() - - - - - @property - def n_bkts(self): - if self._train: - return super(Dataset, self).n_bkts - else: - return super(Dataset, self).n_valid_bkts - - - def reading_dataset(self, filename): - """ - :param filename: - :return: - """ - if self.dataset_type == 'SST-1' or self.dataset_type == 'SST-2': - with open(filename) as f: - buff = [] - for line_num, line in enumerate(f): - line = clean_str_sst(line).split() - if len(line) > 1: - buff.append(line) - self._process_buff(buff) - else: - with open(filename) as f: - buff = [] - for line_num, line in enumerate(f): - line = clean_str(line).split() - if line: - buff.append(line) - self._process_buff(buff) - return - - def _process_buff(self, buff): - """ - :param buff: - :return: - """ - len_cntr = Counter() - for sent in buff: - len_cntr[len(sent)] += 1 - bkts_splits = KMeans(self.n_bkts, len_cntr).splits - # Count the sents length - # Use k-means to splits the sents into n_bkts parts - - # reset bucket size - # map the lenth to bkts id - prev_size = -1 - for bkt_idx, size in enumerate(bkts_splits): - self.buckets[bkt_idx].set_size(size) - self.len2bkts.update(zip(range(prev_size+1, size+1), [bkt_idx] * (size-prev_size))) - prev_size = size - # map all length from min to max to bkts id - # some of lengths do not appear in the data set - for sent in buff: - # Add the sent to the specific bucket according to their length - # Construct the sent into example first - # And then push them into buckets - bkt_idx = self.len2bkts[len(sent)] - example = Example(sent, self._config) - example.convert(self.vocabs) - # save to bucket - idx = self.buckets[bkt_idx].add(example) - self.id2position.append((bkt_idx, idx)) - - - - - - - - def minibatch(self, batch_size, input_idx, target_idx, shuffle=True): - minibatches = [] - for bkt_idx, bucket in enumerate(self.buckets): - if batch_size == 0: - print("Please Specify the batch size") - exit() - else: - n_tokens = len(bucket) * bucket.size - n_splits = max(n_tokens // batch_size, 1) - - if shuffle: - range_func = np.random.permutation - else: - range_func = np.arange - arr_sp = np.array_split(range_func(len(bucket)), n_splits) - for bkt_mb in arr_sp: - minibatches.append((bkt_idx, bkt_mb)) - if shuffle: - np.random.shuffle(minibatches) - - for bkt_idx, bkt_mb in minibatches: - data = self.buckets[bkt_idx].data[bkt_mb] - sents = self.buckets[bkt_idx].sents[bkt_mb] - target = self.buckets[bkt_idx].target[bkt_mb] - maxlen = np.max(np.sum(np.greater(data[:,:,0], 0), axis=1)) - # Do not use dynamic index like conll_index - # For word, set 0 data = [(fea1, fea2, fea3), (fea1, fea2, fea3), ...] - # For target, target = [(target1,), (target2,), ...] - feed_dict = { - 'text' : data[:,:maxlen, input_idx], - 'label' : target[:, target_idx], - 'batch_size' : len(target) - } - yield feed_dict - - @property - def sentsNum(self): - return len(self.id2position) - diff --git a/kim_cnn/etc/__init__.py b/kim_cnn/etc/__init__.py deleted file mode 100644 index b9e3d2e..0000000 --- a/kim_cnn/etc/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -from etc.kmeans import KMeans -from etc.utils import clean_str, clean_str_sst \ No newline at end of file diff --git a/kim_cnn/etc/kmeans.py b/kim_cnn/etc/kmeans.py deleted file mode 100644 index cb80a80..0000000 --- a/kim_cnn/etc/kmeans.py +++ /dev/null @@ -1,180 +0,0 @@ -#!/usr/bin/env python -# -*- coding: UTF-8 -*- - -""" -Adapted from Bi-Affine Parser code -""" - -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function - -from collections import Counter - -import numpy as np - - -# *************************************************************** -class KMeans(object): - """""" - - # ============================================================= - def __init__(self, k, len_cntr): - """""" - - # Error checking - if len(len_cntr) < k: - raise ValueError('Trying to sort %d data points into %d buckets' % (len(len_cntr), k)) - - # Initialize variables - self._k = k - self._len_cntr = len_cntr - self._lengths = sorted(self._len_cntr.keys()) - self._splits = [] - self._split2len_idx = {} - self._len2split_idx = {} - self._split_cntr = Counter() - - # Initialize the splits evenly - lengths = [] - for length, count in self._len_cntr.items(): - lengths.extend([length] * count) - lengths.sort() - self._splits = [np.max(split) for split in np.array_split(lengths, self._k)] - - i = len(self._splits) - 1 - while i > 0: - while self._splits[i - 1] >= self._splits[i] or self._splits[i - 1] not in self._len_cntr: - self._splits[i - 1] -= 1 - i -= 1 - - i = 1 - while i < len(self._splits) - 1: - while self._splits[i] <= self._splits[i - 1] or self._splits[i] not in self._len_cntr: - self._splits[i] += 1 - i += 1 - - # Reindex everything - split_idx = 0 - split = self._splits[split_idx] - for len_idx, length in enumerate(self._lengths): - count = self._len_cntr[length] - self._split_cntr[split] += count - if length == split: - self._split2len_idx[split] = len_idx - split_idx += 1 - if split_idx < len(self._splits): - split = self._splits[split_idx] - self._split_cntr[split] = 0 - elif length > split: - raise IndexError() - - # Iterate - old_splits = None - # print('0) Initial splits: %s; Initial mass: %d' % (self._splits, self.get_mass())) - i = 0 - while self._splits != old_splits: - old_splits = list(self._splits) - self.recenter() - i += 1 - # print('%d) Final splits: %s; Final mass: %d' % (i, self._splits, self.get_mass())) - - self.reindex() - return - - # ============================================================= - def recenter(self): - """""" - - for split_idx in range(len(self._splits)): - split = self._splits[split_idx] - len_idx = self._split2len_idx[split] - if split == self._splits[-1]: - continue - right_split = self._splits[split_idx + 1] - - # Try shifting the centroid to the left - if len_idx > 0 and self._lengths[len_idx - 1] not in self._split_cntr: - new_split = self._lengths[len_idx - 1] - left_delta = self._len_cntr[split] * (right_split - new_split) - self._split_cntr[split] * (split - new_split) - if left_delta < 0: - self._splits[split_idx] = new_split - self._split2len_idx[new_split] = len_idx - 1 - del self._split2len_idx[split] - self._split_cntr[split] -= self._len_cntr[split] - self._split_cntr[right_split] += self._len_cntr[split] - self._split_cntr[new_split] = self._split_cntr[split] - del self._split_cntr[split] - - # Try shifting the centroid to the right - elif len_idx < len(self._lengths) - 2 and self._lengths[len_idx + 1] not in self._split_cntr: - new_split = self._lengths[len_idx + 1] - right_delta = self._split_cntr[split] * (new_split - split) - self._len_cntr[split] * (new_split - split) - if right_delta <= 0: - self._splits[split_idx] = new_split - self._split2len_idx[new_split] = len_idx + 1 - del self._split2len_idx[split] - self._split_cntr[split] += self._len_cntr[split] - self._split_cntr[right_split] -= self._len_cntr[split] - self._split_cntr[new_split] = self._split_cntr[split] - del self._split_cntr[split] - return - - # ============================================================= - - def get_mass(self): - """""" - - mass = 0 - split_idx = 0 - split = self._splits[split_idx] - for len_idx, length in enumerate(self._lengths): - count = self._len_cntr[length] - mass += split * count - if length == split: - split_idx += 1 - if split_idx < len(self._splits): - split = self._splits[split_idx] - return mass - - # ============================================================= - def reindex(self): - """""" - - self._len2split_idx = {} - last_split = -1 - for split_idx, split in enumerate(self._splits): - self._len2split_idx.update(dict(zip(range(last_split + 1, split), [split_idx] * (split - (last_split + 1))))) - return - - # ============================================================= - - def __len__(self): - return self._k - - def __iter__(self): - return (split for split in self.splits) - - def __getitem__(self, key): - return self._splits[key] - - # ============================================================= - @property - def splits(self): - return self._splits - - @property - def len2split_idx(self): - return self._len2split_idx - - -# *************************************************************** -if __name__ == '__main__': - """""" - - len_cntr = Counter() - for i in range(10000): - len_cntr[1 + int(10 ** (1 + np.random.randn()))] += 1 - print(len_cntr) - kmeans = KMeans(10, len_cntr) - print(kmeans.splits) \ No newline at end of file diff --git a/kim_cnn/example.py b/kim_cnn/example.py deleted file mode 100644 index 3e640f1..0000000 --- a/kim_cnn/example.py +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - - - -from configurable import Configurable -class Example(Configurable): - """ - - """ - def __init__(self, sent, *args, **kwargs): - super(Example, self).__init__(*args, **kwargs) - self.length = len(sent) - self.sent = None - self.data = None - # original word in this setting "TREC" - # TODO: for different dataset, the original data will have different format - # TODO: this data format is related to output. Leave for future work - # self.feature = None - # # Convert each of the features to one-hot representation: (n_word, n_feature) - # self.target = None - # # Convert each of the targets to one-hot representation: (n_word, n_target) - if self.dataset_type == "TREC": - self.data = {} - self.sent = {} - self.sent["words"] = sent[2:] - self.sent["targets"] = sent[0] - else: - self.data = {} - self.sent = {} - self.sent["words"] = sent[1:] - self.sent["targets"] = sent[0] - - def convert(self, vocabs): - words, target = vocabs - self.data["words"] = [] - self.data["targets"] = target[self.sent["targets"]] - for word in self.sent["words"]: - self.data["words"].append(words[word]) - - - diff --git a/kim_cnn/getData.sh b/kim_cnn/getData.sh deleted file mode 100644 index 73e9a89..0000000 --- a/kim_cnn/getData.sh +++ /dev/null @@ -1,8 +0,0 @@ -mkdir data -mkdir saves -wget http://ocp59jkku.bkt.clouddn.com/sst-1.zip -P data/ -wget http://ocp59jkku.bkt.clouddn.com/sst-2.zip -P data/ -wget http://ocp59jkku.bkt.clouddn.com/trec.zip -P data/ -unzip data/sst-1.zip -d data/ -unzip data/sst-2.zip -d data/ -unzip data/trec.zip -d data/ diff --git a/kim_cnn/main.py b/kim_cnn/main.py index 0e88410..c0f695f 100644 --- a/kim_cnn/main.py +++ b/kim_cnn/main.py @@ -1,85 +1,74 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -import model -from network import cnnTextNetwork -from configurable import Configurable -import torch +import sys +import random import numpy as np -import os - -if __name__=='__main__': - - import argparse +import torch +from torchtext import data +from args import get_args +from SST1 import SST1Dataset +from utils import clean_str_sst - argparser = argparse.ArgumentParser() - argparser.add_argument('--train', action='store_true') - argparser.add_argument('--validate', action='store_true') - argparser.add_argument('--test', action='store_true') - argparser.add_argument('--load', action='store_true') - argparser.add_argument('--seed', help='Random seed', type=int, default=3435) - argparser.add_argument('--num_threads', help='The number of threads to use', type=int, default=4) - - args, extra_args = argparser.parse_known_args() - # args.train = True/False ... - # extra_args['--some': "xxxx"] - cargs = {k: v for (k, v) in vars(Configurable.argparser.parse_args(extra_args)).items() if v is not None} - - torch.manual_seed(args.seed) - np.random.seed(args.seed) - if torch.cuda.is_available(): +args = get_args() +torch.manual_seed(args.seed) +if not args.cuda: + args.gpu = -1 +if torch.cuda.is_available() and args.cuda: + print("Note: You are using GPU for training") + torch.cuda.set_device(args.gpu) torch.cuda.manual_seed(args.seed) - torch.set_num_threads(args.num_threads) +if torch.cuda.is_available() and not args.cuda: + print("Warning: You have Cuda but do not use it. You are using CPU for training") +np.random.seed(args.seed) +random.seed(args.seed) - if 'model_type' not in cargs: - print("You need to specify the model_type") - exit() - print('*** '+cargs['model_type']+" ***") +if not args.trained_model: + print("Error: You need to provide a option 'trained_model' to load the model") + sys.exit(1) - if args.load and 'restore_from' in cargs: - print("Loading model from [%s]..." % (cargs['restore_from'])) - try: - m = torch.load(cargs['restore_from']) - cargs.pop(cargs['restore_from'], "") - except: - print("The model doesn't exist") - exit() - elif args.validate or args.test: - print("Loading model from [%s]..." % (cargs['restore_from'])) - try: - m = torch.load(cargs['restore_from']) - cargs.pop(cargs['restore_from'], "") - except: - print("The model doesn't exist") - exit() - else: - m = getattr(model, cargs['model_type']) +if args.dataset == 'SST-1': + TEXT = data.Field(batch_first=True, lower=True, tokenize=clean_str_sst) + LABEL = data.Field(sequential=False) + train, dev, test = SST1Dataset.splits(TEXT, LABEL) + +TEXT.build_vocab(train, min_freq=2) +LABEL.build_vocab(train) + +train_iter = data.Iterator(train, batch_size=args.batch_size, device=args.gpu, train=True, repeat=False, + sort=False, shuffle=True) +dev_iter = data.Iterator(dev, batch_size=args.batch_size, device=args.gpu, train=False, repeat=False, + sort=False, shuffle=False) +test_iter = data.Iterator(test, batch_size=args.batch_size, device=args.gpu, train=False, repeat=False, + sort=False, shuffle=False) + +config = args +config.target_class = len(LABEL.vocab) +config.words_num = len(TEXT.vocab) +config.embed_num = len(TEXT.vocab) + +print("Label dict:", LABEL.vocab.itos) + +if args.cuda: + model = torch.load(args.trained_model, map_location=lambda storage, location: storage.cuda(args.gpu)) +else: + model = torch.load(args.trained_model, map_location=lambda storage,location: storage) - network = None +def predict(dataset_iter, dataset, dataset_name): + print("Dataset: {}".format(dataset_name)) + model.eval() + dataset_iter.init_epoch() - if cargs['model_type'] == "CNNText": - cargs.pop("model_type", "") - network = cnnTextNetwork(args, m, **cargs) - else: - print("The model type is not supported") - exit() + n_correct = 0 + for data_batch_idx, data_batch in enumerate(dataset_iter): + scores = model(data_batch) + n_correct += (torch.max(scores, 1)[1].view(data_batch.label.size()).data == data_batch.label.data).sum() - if not os.path.exists(network.save_dir): - os.mkdir(network.save_dir) + print("no. correct {} out of {}".format(n_correct, len(dataset))) + accuracy = 100. * n_correct / len(dataset) + print("{} accuracy: {:8.6f}%".format(dataset_name, accuracy)) +# Run the model on the dev set +predict(dataset_iter=dev_iter, dataset=dev, dataset_name="valid") - # if torch.cuda.is_available(): - # torch.cuda.manual_seed_all(1) - - - - if args.train: - network.train() - elif args.validate: - print("### The accuracy for validate set: ") - print(network.test(validate=True)) - elif args.test: - print("### The accuracy for test set: ") - print(network.test(validate=False)) +# Run the model on the test set +predict(dataset_iter=test_iter, dataset=test, dataset_name="test") \ No newline at end of file diff --git a/kim_cnn/model.py b/kim_cnn/model.py new file mode 100644 index 0000000..d66359d --- /dev/null +++ b/kim_cnn/model.py @@ -0,0 +1,59 @@ +import torch +import torch.nn as nn + +import torch.nn.functional as F + +class KimCNN(nn.Module): + def __init__(self, config): + super(KimCNN, self).__init__() + output_channel = config.output_channel + target_class = config.target_class + words_num = config.words_num + words_dim = config.words_dim + embed_num = config.embed_num + embed_dim = config.embed_dim + self.mode = config.mode + Ks = 3 # There are three conv net here + if config.mode == 'multichannel': + input_channel = 2 + else: + input_channel = 1 + self.embed = nn.Embedding(words_num, words_dim) + self.static_embed = nn.Embedding(embed_num, embed_dim) + self.non_static_embed = nn.Embedding(embed_num, embed_dim) + self.static_embed.weight.requires_grad = False + + self.conv1 = nn.Conv2d(input_channel, output_channel, (3, words_dim), padding=(2,0)) + self.conv2 = nn.Conv2d(input_channel, output_channel, (4, words_dim), padding=(3,0)) + self.conv3 = nn.Conv2d(input_channel, output_channel, (5, words_dim), padding=(4,0)) + + self.dropout = nn.Dropout(config.dropout) + self.fc1 = nn.Linear(Ks * output_channel, target_class) + + + def forward(self, x): + x = x.text + if self.mode == 'rand': + word_input = self.embed(x) # (batch, sent_len, embed_dim) + x = word_input.unsqueeze(1) # (batch, channel_input, sent_len, embed_dim) + elif self.mode == 'static': + static_input = self.static_embed(x) + x = static_input.unsqueeze(1) # (batch, channel_input, sent_len, embed_dim) + elif self.mode == 'non-static': + non_static_input = self.non_static_embed(x) + x = non_static_input.unsqueeze(1) # (batch, channel_input, sent_len, embed_dim) + elif self.mode == 'multichannel': + non_static_input = self.non_static_embed(x) + static_input = self.static_embed(x) + x = torch.stack([non_static_input, static_input], dim=1) # (batch, channel_input=2, sent_len, embed_dim) + else: + print("Unsupported Mode") + exit() + x = [F.relu(self.conv1(x)).squeeze(3), F.relu(self.conv2(x)).squeeze(3), F.relu(self.conv3(x)).squeeze(3)] + # (batch, channel_output, ~=sent_len) * Ks + x = [F.max_pool1d(i, i.size(2)).squeeze(2) for i in x] # max-over-time pooling + # (batch, channel_output) * Ks + x = torch.cat(x, 1) # (batch, channel_output * Ks) + x = self.dropout(x) + logit = self.fc1(x) # (batch, target_size) + return logit diff --git a/kim_cnn/model/__init__.py b/kim_cnn/model/__init__.py deleted file mode 100644 index 88b6b9d..0000000 --- a/kim_cnn/model/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from model.cnnText import * \ No newline at end of file diff --git a/kim_cnn/model/cnnText/__init__.py b/kim_cnn/model/cnnText/__init__.py deleted file mode 100644 index f57fbc7..0000000 --- a/kim_cnn/model/cnnText/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from model.cnnText.cnntext import CNNText \ No newline at end of file diff --git a/kim_cnn/model/cnnText/cnntext.py b/kim_cnn/model/cnnText/cnntext.py deleted file mode 100644 index 7714618..0000000 --- a/kim_cnn/model/cnnText/cnntext.py +++ /dev/null @@ -1,86 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -import torch -import torch.nn as nn - -import torch.nn.functional as F - - -class CNNText(nn.Module): - """ - Model class for the computational graph - """ - def __init__(self, args): - super(CNNText, self).__init__() - - - #input_channel = args['input_channels'] - output_channel = args['output_channels'] - target_class = args['target_class'] - words_num = args['words_num'] - words_dim = args['words_dim'] - embeds_num = args['embeds_num'] - embeds_dim = args['embeds_dim'] - Ks = args['kernel_sizes'] - self.mode = args['mode'] - if self.mode == 'multichannel': - input_channel = 2 - else: - input_channel = 1 - self.use_gpu = args['use_gpu'] - self.embed = nn.Embedding(words_num, words_dim) - self.static_embed = nn.Embedding(embeds_num, embeds_dim) - self.static_embed.weight.data.copy_(torch.from_numpy(args['embeds'])) - self.non_static_embed = nn.Embedding(embeds_num, embeds_dim) - self.non_static_embed.weight.data.copy_(torch.from_numpy(args['embeds'])) - self.static_embed.weight.requires_grad = False - - self.conv1 = nn.Conv2d(input_channel, output_channel, (3, words_dim), padding=(2, 0)) - self.conv2 = nn.Conv2d(input_channel, output_channel, (4, words_dim), padding=(3, 0)) - self.conv3 = nn.Conv2d(input_channel, output_channel, (5, words_dim), padding=(4, 0)) - #self.convs1 = [nn.Conv2d(input_channel, output_channel, (K, words_dim), padding=(K-1, 0)) for K in Ks] - - self.dropout = nn.Dropout(args['dropout']) - self.fc1 = nn.Linear(len(Ks) * output_channel, target_class) - - def conv_and_pool(self, x, conv): - x = F.relu(conv(x)).squeeze(3) # (batch_size, output_channel, feature_map_dim) - x = F.max_pool1d(x, x.size(2)).squeeze(2) - return x - - def forward(self, x): - #if self.use_gpu: - # self.conv1s = [model.cuda() for model in self.convs1] - if self.mode == 'rand': - words = x[:,:,0] - word_input = self.embed(words) # (batch, sent_len, embed_dim) - x = word_input.unsqueeze(1) # (batch, channel_input, sent_len, embed_dim) - elif self.mode == 'static': - static_words = x[:,:,1] - static_input = self.static_embed(static_words) - x = static_input.unsqueeze(1) # (batch, channel_input, sent_len, embed_dim) - elif self.mode == 'non-static': - non_static_words = x[:, :, 1] - non_static_input = self.non_static_embed(non_static_words) - x = non_static_input.unsqueeze(1) # (batch, channel_input, sent_len, embed_dim) - elif self.mode == 'multichannel': - words = x[:, :, 1] - word_input = self.non_static_embed(words) # (batch, sent_len, embed_dim) - static_words = x[:, :, 1] - static_input = self.static_embed(static_words) - x = torch.stack([word_input, static_input], dim=1)# (batch, channel_input, sent_len, embed_dim) - else: - print("Unsupported Mode") - exit() - #x = word_input.unsqueeze(1) # (batch, channel_input, sent_len, embed_dim) - #x = [F.relu(conv(x)).squeeze(3) for conv in self.convs1] - x = [F.relu(self.conv1(x)).squeeze(3), F.relu(self.conv2(x)).squeeze(3), F.relu(self.conv3(x)).squeeze(3)] - # (batch, channel_output, ~=(sent_len)) * len(Ks) - x = [F.max_pool1d(i, i.size(2)).squeeze(2) for i in x] # max-over-time pooling - # (batch, channel_output) * len(Ks) - x = torch.cat(x, 1) # (batch, channel_output * len(Ks)) - x = self.dropout(x) - logit = self.fc1(x) # (batch, target_size) - return logit - diff --git a/kim_cnn/network/__init__.py b/kim_cnn/network/__init__.py deleted file mode 100644 index 1fef405..0000000 --- a/kim_cnn/network/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from network.cnnTextNetwork import cnnTextNetwork \ No newline at end of file diff --git a/kim_cnn/network/cnnTextNetwork.py b/kim_cnn/network/cnnTextNetwork.py deleted file mode 100644 index f0e8d90..0000000 --- a/kim_cnn/network/cnnTextNetwork.py +++ /dev/null @@ -1,226 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -from configurable import Configurable -from vocab import Vocab -from dataset import Dataset -import os -import sys -import torch -import torch.nn.functional as F -from torch.autograd import Variable - - -class cnnTextNetwork(Configurable): - """ - Network class - - build the vocabulary - - build the dataset - - control the training - - control the validation and testing - - save the model and store the best result - """ - - def __init__(self, option, model, *args, **cargs): - - '''check args?''' - super(cnnTextNetwork, self).__init__(*args, **cargs) - if not os.path.isdir(self.save_dir): - os.mkdir(self.save_dir) - - with open(os.path.join(self.save_dir, 'config_file'), 'w') as f: - self._config.write(f) - - - self._vocabs = [] - vocab_file = [(self.word_file, 'Words'), - (self.target_file, "Targets")] - - for i, (vocab_file, name) in enumerate(vocab_file): - vocab = Vocab(vocab_file, self._config, - name = name, - load_embed_file = (not i), - lower_case = (not i) - ) - self._vocabs.append(vocab) - - print("################## Data ##################") - print("There are %d words in training set" % (len(self.words) - 2)) - print("There are %d targets in training set" % (len(self.targets) - 2)) - print("Loading training set ...") - self._trainset = Dataset(self.train_file, self._vocabs, self._config, name="Trainset") - print("There are %d sentences in training set" % (self._trainset.sentsNum)) - print("Loading validation set ...") - self._validset = Dataset(self.valid_file, self._vocabs, self._config, name="Validset") - print("There are %d sentences in validation set" % (self._validset.sentsNum)) - print("Loading testing set ...") - self._testset = Dataset(self.test_file, self._vocabs, self._config, name="Testset") - print("There are %d sentences in testing set" % (self._testset.sentsNum)) - - self.args = {#'input_channels':2, - 'kernel_sizes':[3,4,5], - 'words_num': len(self.words), - 'words_dim': self.words_dim, - 'target_class': len(self.targets), - 'output_channels': 100, - 'dropout': self.dropout, - 'embeds_num' : self.words.embeds_size, - 'embeds_dim' : self.words_dim, # Embedding size must be the same with words size - 'embeds':self.words.pretrained_embeddings, - 'use_gpu': self.use_gpu, - 'mode': self.mode} - - self.model = model - return - - - def train_minibatch(self): - return self._trainset.minibatch(self.train_batch_size, self.input_idx, self.target_idx, shuffle=True) - - def valid_minibatch(self): - return self._validset.minibatch(self.test_batch_size, self.input_idx, self.target_idx, shuffle=False) - - def test_minibatch(self): - return self._testset.minibatch(self.test_batch_size, self.input_idx, self.target_idx, shuffle=False) - - def train(self): - # if torch.cuda.is_available(): # and use_cuda - # self.model.cuda() - if self.use_gpu: - self.model = self.model(self.args).cuda() - else: - self.model = self.model(self.args) - parameter = filter(lambda p: p.requires_grad, self.model.parameters()) - optimizer = torch.optim.Adam(parameter, lr=self.learning_rate) - # The optimizer doesn't have adaptive learning rate - - step = 0 - best_score = 0 - valid_accuracy = 0 - test_accuracy = 0 - - acc_corrects = 0 # count the corrects for one log_interval - acc_sents = 0 # count sents number for one log_interval - - epoch = 0 - best_accuracy = 0 - best_model = 0 - - while True: - for batch in self.train_minibatch(): - self.model.train() - feature, target = batch['text'], batch['label'] - # Sanity check - # for sent in feature: - # for word in sent: - # word_str = self.words._idx2str[word[0]] - # embed_str = self.words._embed2str[word[1]] - # if word_str != embed_str: - # print(word_str, embed_str) - ## - if self.use_gpu: - feature = Variable(torch.from_numpy(feature).cuda()) - target = Variable(torch.from_numpy(target).cuda())[:, 0] - else: - feature = Variable(torch.from_numpy(feature)) - target = Variable(torch.from_numpy(target))[:, 0] - - # if torch.cuda.is_available(): - # feature, target = feature.cuda(), target.cuda() - optimizer.zero_grad() # Clears the gradients of all optimized Variable - logit = self.model(feature) - loss = F.cross_entropy(logit, target) - loss.backward() - optimizer.step() - step += 1 - preds = torch.max(logit, 1)[1].view(target.size()) # get the index - acc_corrects += (preds.cpu().data == target.cpu().data).sum() - acc_sents += batch['batch_size'] - # if step % self.log_interval == 0: - # accuracy = float(acc_corrects) / float(acc_sents) * 100.0 - # print("## [Batch %d] Accuracy : %5.2f" % (step, accuracy)) - # acc_corrects = 0 - # acc_sents = 0 - - if step == 1 or step % self.valid_interval == 0: - accuracy = self.test(validate=True) - print("## Validation: %5.2f" % (accuracy)) - if accuracy > best_score: - best_score = accuracy - valid_accuracy = accuracy - print("## Update Model ##") - torch.save(self.model, self.save_model_file) - - print("## Currently the best validation: Accucacy %5.2f" % (valid_accuracy)) - - epoch += 1 - accuracy = float(acc_corrects) / float(acc_sents) * 100 - - # if the new accuracy is better than the old accuracy by 0.1% - if accuracy - best_accuracy > 0.1: - best_accuracy = accuracy - best_model = epoch - - print("[EPOCH] %d Accuracy: %5.2f" % (epoch, accuracy)) - - # stop training if the accuracy remains the same over 5 epochs - if (epoch - best_model) >= 5: - print('No improvement since the last {} epochs. Stopping training'.format(epoch - best_model)) - break - - acc_corrects = 0 - acc_sents = 0 - if (epoch % self.epoch_decay == 0): - lr = self.learning_rate * (0.75 ** (epoch // self.epoch_decay)) - for param_group in optimizer.param_groups: - param_group['lr'] = lr - - - def test(self, validate=False): - self.model.eval() - if validate: - dataset = self._validset - minibatch = self.valid_minibatch - else: - dataset = self._testset - minibatch = self.test_minibatch - - test_corrects = 0 - test_sents = 0 - for batch in minibatch(): - # TODO: Prediton to Text - feature, target = batch['text'], batch['label'] - if self.use_gpu: - feature = Variable(torch.from_numpy(feature).cuda()) - else: - feature = Variable(torch.from_numpy(feature)) - target = Variable(torch.from_numpy(target))[:,0] - # if torch.cuda.is_available(): - # feature, target = feature.cuda(), target.cuda() - - logit = self.model(feature) - preds = torch.max(logit, 1)[1].view(target.size()) # get the index - test_corrects += (preds.cpu().data == target.data).sum() - test_sents += batch['batch_size'] - return float(test_corrects) / float(test_sents) * 100.0 - - @property - def words(self): - return self._vocabs[0] - - @property - def targets(self): - return self._vocabs[1] - - - @property - def input_idx(self): - return (0, 1) - - - @property - def target_idx(self): - return (0,) - - - diff --git a/kim_cnn/train.py b/kim_cnn/train.py new file mode 100644 index 0000000..2638bdb --- /dev/null +++ b/kim_cnn/train.py @@ -0,0 +1,191 @@ +import time +import os +import random +import torch +import torch.nn as nn +import numpy as np +from torchtext import data +from args import get_args +from model import KimCNN +from SST1 import SST1Dataset +from utils import clean_str_sst + +# Set default configuration in : args.py +args = get_args() + +# Set random seed for reproducibility + +torch.manual_seed(args.seed) +if not args.cuda: + args.gpu = -1 +if torch.cuda.is_available() and args.cuda: + print("Note: You are using GPU for training") + torch.cuda.set_device(args.gpu) + torch.cuda.manual_seed(args.seed) +if torch.cuda.is_available() and not args.cuda: + print("Warning: You have Cuda but not use it. You are using CPU for training.") +np.random.seed(args.seed) +random.seed(args.seed) + +# Set up the data for training +# SST-1 +if args.dataset == 'SST-1': + TEXT = data.Field(batch_first=True, tokenize=clean_str_sst) + LABEL = data.Field(sequential=False) + train, dev, test = SST1Dataset.splits(TEXT, LABEL) + +TEXT.build_vocab(train, min_freq=2) +LABEL.build_vocab(train) + +if os.path.isfile(args.vector_cache): + stoi, vectors, dim = torch.load(args.vector_cache) + TEXT.vocab.vectors = torch.Tensor(len(TEXT.vocab), dim) + for i, token in enumerate(TEXT.vocab.itos): + wv_index = stoi.get(token, None) + if wv_index is not None: + TEXT.vocab.vectors[i] = vectors[wv_index] + else: + TEXT.vocab.vectors[i] = torch.Tensor.zero_(TEXT.vocab.vectors[i]) +else: + print("Error: Need word embedding pt file") + exit(1) + +#print('len(TEXT.vocab)', len(TEXT.vocab)) +#print('TEXT.vocab.vectors.size()', TEXT.vocab.vectors.size()) + +train_iter = data.Iterator(train, batch_size=args.batch_size, device=args.gpu, train=True, repeat=False, + sort=False, shuffle=True) +dev_iter = data.Iterator(dev, batch_size=args.batch_size, device=args.gpu, train=False, repeat=False, + sort=False, shuffle=False) +test_iter = data.Iterator(test, batch_size=args.batch_size, device=args.gpu, train=False, repeat=False, + sort=False, shuffle=False) + +config = args +config.target_class = len(LABEL.vocab) +config.words_num = len(TEXT.vocab) +config.embed_num = len(TEXT.vocab) + + +#print(config) +print("Dataset {} Mode {}".format(args.dataset, args.mode)) +print("VOCAB num",len(TEXT.vocab)) +print("LABEL.target_class:", len(LABEL.vocab)) +print("LABELS:",LABEL.vocab.itos) +print("Train instance", len(train)) +print("Dev instance", len(dev)) +print("Test instance", len(test)) + + +if args.resume_snapshot: + if args.cuda: + model = torch.load(args.resume_snapshot, map_location=lambda storage, location: storage.cuda(args.gpu)) + else: + model = torch.load(args.resume_snapshot, map_location=lambda storage, location: storage) +else: + model = KimCNN(config) + model.static_embed.weight.data.copy_(TEXT.vocab.vectors) + model.non_static_embed.weight.data.copy_(TEXT.vocab.vectors) + if args.cuda: + model.cuda() + print("Shift model to GPU") + + +parameter = filter(lambda p: p.requires_grad, model.parameters()) +#for idx, p in enumerate(parameter): +# print(idx, p) +optimizer = torch.optim.Adadelta(parameter, lr=args.lr, weight_decay=args.weight_decay) +criterion = nn.CrossEntropyLoss() +early_stop = False +best_dev_acc = 0 +iterations = 0 +iters_not_improved = 0 +epoch = 0 +start = time.time() +header = ' Time Epoch Iteration Progress (%Epoch) Loss Dev/Loss Accuracy Dev/Accuracy' +dev_log_template = ' '.join('{:>6.0f},{:>5.0f},{:>9.0f},{:>5.0f}/{:<5.0f} {:>7.0f}%,{:>8.6f},{:8.6f},{:12.4f},{:12.4f}'.split(',')) +log_template = ' '.join('{:>6.0f},{:>5.0f},{:>9.0f},{:>5.0f}/{:<5.0f} {:>7.0f}%,{:>8.6f},{},{:12.4f},{}'.split(',')) +os.makedirs(args.save_path, exist_ok=True) +os.makedirs(os.path.join(args.save_path, args.dataset), exist_ok=True) +print(header) + + +while True: + if early_stop: + print("Early Stopping. Epoch: {}, Best Dev Acc: {}".format(epoch, best_dev_acc)) + break + epoch += 1 + train_iter.init_epoch() + n_correct, n_total = 0, 0 + + for batch_idx, batch in enumerate(train_iter): + # Batch size : (Sentence Length, Batch_size) + iterations += 1 + model.train(); optimizer.zero_grad() + #print("Text Size:", batch.text.size()) + #print("Label Size:", batch.label.size()) + scores = model(batch) + n_correct += (torch.max(scores, 1)[1].view(batch.label.size()).data == batch.label.data).sum() + n_total += batch.batch_size + train_acc = 100. * n_correct / n_total + + loss = criterion(scores, batch.label) + loss.backward() + + optimizer.step() + + + # Evaluate performance on validation set + if iterations % args.dev_every == 1: + # switch model into evalutaion mode + model.eval(); dev_iter.init_epoch() + n_dev_correct = 0 + dev_losses = [] + for dev_batch_idx, dev_batch in enumerate(dev_iter): + scores = model(dev_batch) + n_dev_correct += (torch.max(scores, 1)[1].view(dev_batch.label.size()).data == dev_batch.label.data).sum() + dev_loss = criterion(scores, dev_batch.label) + dev_losses.append(dev_loss.data[0]) + dev_acc = 100. * n_dev_correct / len(dev) + print(dev_log_template.format(time.time() - start, + epoch, iterations, 1 + batch_idx, len(train_iter), + 100. * (1 + batch_idx) / len(train_iter), loss.data[0], + sum(dev_losses) / len(dev_losses), train_acc, dev_acc)) + + # Update validation results + if dev_acc > best_dev_acc: + iters_not_improved = 0 + best_dev_acc = dev_acc + snapshot_path = os.path.join(args.save_path, args.dataset, args.mode+'_best_model.pt') + torch.save(model, snapshot_path) + else: + iters_not_improved += 1 + if iters_not_improved >= args.patience: + early_stop = True + break + + if iterations % args.log_every == 1: + # print progress message + print(log_template.format(time.time() - start, + epoch, iterations, 1 + batch_idx, len(train_iter), + 100. * (1 + batch_idx) / len(train_iter), loss.data[0], ' ' * 8, + n_correct / n_total * 100, ' ' * 12)) + + + + + + + + + + + + + + + + + + + + diff --git a/kim_cnn/etc/utils.py b/kim_cnn/utils.py similarity index 91% rename from kim_cnn/etc/utils.py rename to kim_cnn/utils.py index bbcf4b9..12ae678 100644 --- a/kim_cnn/etc/utils.py +++ b/kim_cnn/utils.py @@ -18,7 +18,7 @@ def clean_str(string): string = re.sub(r"\)", " ) ", string) string = re.sub(r"\?", " ? ", string) string = re.sub(r"\s{2,}", " ", string) - return string.strip() + return string.lower().strip().split() def clean_str_sst(string): @@ -27,4 +27,4 @@ def clean_str_sst(string): """ string = re.sub(r"[^A-Za-z0-9(),!?\'\`]", " ", string) string = re.sub(r"\s{2,}", " ", string) - return string.strip() \ No newline at end of file + return string.lower().strip().split() \ No newline at end of file diff --git a/kim_cnn/vocab.py b/kim_cnn/vocab.py deleted file mode 100644 index 56b855a..0000000 --- a/kim_cnn/vocab.py +++ /dev/null @@ -1,165 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -from configurable import Configurable -from collections import Counter -import numpy as np - -from etc.utils import clean_str, clean_str_sst - -class Vocab(Configurable): - """ - Vocab for - - id-str converting - - word embedding / lookup - """ - def __init__(self, vocab_file, *args, **kwargs): - """ - vocab_file: the name of the vocab_file - args: it should be self._config - kwargs: some options for Vocab, before super, they should be pop out - because these are not options for the global settings - """ - self.vocab_file = vocab_file - load_embed_file = kwargs.pop('load_embed_file', False) - self.lower_case = kwargs.pop('lower_case', False) - super(Vocab, self).__init__(*args, **kwargs) - - self.SPECIAL_TOKENS = ('', '') - self.START_IDX = len(self.SPECIAL_TOKENS) - self.PAD, self.UNK = range(self.START_IDX) - self.pretrained_embeddings = None - self._count = Counter() # Count the number of vocab - - self._str2idx = dict(zip(self.SPECIAL_TOKENS, range(self.START_IDX))) - self._idx2str = dict(zip(range(self.START_IDX), self.SPECIAL_TOKENS)) - - self._str2embed = {} - self._embed2str = {} - - self.add_train_file() - self.save_vocab_file() - if load_embed_file: - self.load_embed_file() - - - - def add_train_file(self): - if self.dataset_type == 'TREC': - with open(self.train_file) as f: - for line_num, line in enumerate(f): - line = clean_str(line).split() - if line: - if self.name == 'Targets': - self.add(line[0]) - if self.name == 'Words': - for word in line[2:]: - self.add(word) - else: - with open(self.train_file) as f: - for line_num, line in enumerate(f): - line = clean_str(line).split() - if line: - if self.name == 'Targets': - self.add(line[0]) - if self.name == 'Words': - for word in line[1:]: - self.add(word) - - self.index_vocab() - - def add(self, item): - if self.lower_case: - item = item.lower() - - self._count[item] += 1 - return - - def index_vocab(self): - """ - Sorted the vocabs by frequency and assign id to them - Process: - - Get all the words with same frequency from the Counter - - Sort those words - - Assign ID to them - - Go back to first step - """ - cur_idx = self.START_IDX - buff = [] - for word_and_count in self._count.most_common(): - if (not buff) or (buff[-1][1]==word_and_count[1]): - buff.append(word_and_count) - else: - buff.sort() - for word, count in buff: - if count >= self.min_occur_count and (word not in self._str2idx): - self._str2idx[word] = cur_idx - self._idx2str[cur_idx] = word - cur_idx += 1 - buff = [word_and_count] - buff.sort() - for word, count in buff: - if count >= self.min_occur_count and word not in self._str2idx: - self._str2idx[word] = cur_idx - self._idx2str[cur_idx] = word - cur_idx += 1 - return - - - def save_vocab_file(self): - """ - save the words on the file - """ - with open(self.vocab_file, "w") as f: - for word_and_count in self._count.most_common(): - f.write('%s\t%d\n' %(word_and_count)) - return - - def load_embed_file(self): - self._str2embed = dict(zip(self.SPECIAL_TOKENS, range(self.START_IDX))) - self._embed2str = dict(zip(range(self.START_IDX), self.SPECIAL_TOKENS)) - embeds = [[0] * self.words_dim, [0] * self.words_dim] - with open(self.embed_file) as f: - cur_idx = self.START_IDX - for line_num, line in enumerate(f): - line = line.strip().split() - if line: - try: - if self.dataset_type != 'SST-1' or self.dataset_type != 'SST-2': - self._str2embed[clean_str(line[0])] = cur_idx - self._embed2str[cur_idx] = clean_str(line[0]) - else: - self._str2embed[clean_str_sst(line[0])] = cur_idx - self._embed2str[cur_idx] = clean_str_sst(line[0]) - embeds.append(line[1:]) - cur_idx += 1 - except: - raise ValueError('The embedding file is misformatted at line %d' % (line_num+1)) - # Randomly initialize the pre-trained vector for those words not in pre-train-file - for word in self._str2idx.keys(): - if word not in self._str2embed.keys(): - self._str2embed[word] = cur_idx - self._embed2str[cur_idx] = word - embeds.append(list(np.random.uniform(-1, 1, self.words_dim))) - cur_idx += 1 - self.pretrained_embeddings = np.array(embeds, dtype=np.float64) - del embeds - return - - - def __getitem__(self, key): - if isinstance(key, str): - # Convert the lower case - if self.lower_case: - key = key.lower() - if self.pretrained_embeddings is not None: - return (self._str2idx.get(key, self.UNK), self._str2embed.get(key, self.UNK)) - else: - return (self._str2idx.get(key, self.UNK),) - - def __len__(self): - return len(self._str2idx) - - @property - def embeds_size(self): - return len(self._embed2str)