Merge branch 'master' of https://github.com/codekansas/keras-language-modeling

README.md

@@ -5,7 +5,7 @@ Some code for doing language modeling with Keras, in particular for question-ans
 ### Stuff that might be of interest
 
  - `attention_lstm.py`: Attentional LSTM, based on one of the papers referenced in the blog post and others. One application used it for [image captioning](http://arxiv.org/pdf/1502.03044.pdf). It is initialized with an attention vector which provides the attention component for the neural network.
- - `insurance_qa_eval.py`: Evaluation framework for the InsuranceQA dataset. To get this working, clone the [data repository](https://github.com/codekansas/insurance_qa_python) and change the `data_path` to the cloned repository. Changing `config` will adjust how the model is trained.
+ - `insurance_qa_eval.py`: Evaluation framework for the InsuranceQA dataset. To get this working, clone the [data repository](https://github.com/codekansas/insurance_qa_python) and set the `DATA_PATH` environment variable to the cloned repository. Changing `config` will adjust how the model is trained.
  - `keras-language-model.py`: The `LanguageModel` class uses the `config` settings to generate a training model and a testing model. The model can be trained by passing a question vector, a ground truth answer vector, and a bad answer vector to `fit`. Then `predict` calculates the similarity between a question and answer. Override the `build` method with whatever language model you want to get a trainable model. Examples are provided at the bottom, including the `EmbeddingModel`, `ConvolutionModel`, and `RecurrentModel`.
  - `word_embeddings.py`: A Word2Vec layer that uses the embeddings generated by Gensim's word2vec model to provide vectors in place of the Keras `Embedding` layer, which could help improve convergence, since fewer parameters need to be learned. Note that this requires generating a separate file with the word2vec weights, so it doesn't fit in very nicely with the Keras architecture.
 

insurance_qa_embeddings.py

@@ -1,6 +1,7 @@
 from __future__ import print_function
 
 import os
+import sys
 import random
 from time import strftime, gmtime
 
@@ -21,7 +22,12 @@ def revert(vocab, indices):
     return [vocab.get(i, 'X') for i in indices]
 
 if __name__ == '__main__':
-    data_path = '/media/moloch/HHD/MachineLearning/data/insuranceQA/pyenc'
+    try:
+        data_path = os.environ['DATA_PATH'] 
+    except KeyError:
+        print("DATA_PATH is not set.  Set it to your clone of https://github.com/codekansas/insurance_qa_python")
+        sys.exit(1)
+        
     vocab = load(data_path, 'vocab')
 
     sentences = list()

insurance_qa_eval.py

@@ -1,6 +1,7 @@
 from __future__ import print_function
 
 import os
+import sys
 import random
 from time import strftime, gmtime
 
@@ -199,7 +200,11 @@ class Evaluator:
         return top1s, mrrs
 
 if __name__ == '__main__':
-    data_path = '/media/moloch/HHD/MachineLearning/data/insuranceQA/pyenc'
+    try:
+        data_path = os.environ['DATA_PATH'] 
+    except KeyError:
+        print("DATA_PATH is not set.  Set it to your clone of https://github.com/codekansas/insurance_qa_python")
+        sys.exit(1)
 
     conf = {
         'question_len': 20,