keras-language-modeling

Some code for doing language modeling with Keras, in particular for question-answering tasks. The original paper, which provided inspiriation for this project, can be found here.

Stuff that might be of interest

Most of this stuff is just a project for a class of mine, but in the process of writing it I wrote a few files that might be of interest to other people.

• attention_lstm.py: Attentional LSTM, based on the referenced paper, and others. One application used it for image captioning.
• keras_attention_model.py: Implementation of an attentional LSTM architecture for question-answer matching, which closely mimics one of the architectures described in the referenced paper. In particular, the model uses the attentional LSTM implementation in attention_lstm.py to generate a sentence embedding for each answer given a particular question, and fits the answer embedding to have a high cosine similarity with the question embedding.
• word_embeddings.py: A Word2Vec layer that uses the embeddings generated by Gensim's word2vec model to provide vectors in place of the Keras Embedding layer, which could help improve convergence, since fewer parameters need to be learned. Note that this requires generating a separate file with the word2vec weights, so it doesn't fit in very nicely with the Keras architecture.

Data

• L6 from Yahoo Webscope
• InsuranceQA data