Single-layer bi-LSTM with max pooling, 40 words per sentence, loss margin of 0.2
 - MRR ~0.17
 - Seemed to converge after about 20 epochs, with randomization between epochs
 - Using the pure embedding layer worked better than using the Word2Vec model (gave MRR ~0.09)

CNN + ALSTM model seems to have good loss
 - In general, the model predicts the topics well, but doesn't necessarily match it well with the question. This might be due to the masking issue.
 - MRR: 0.30957

2 ALSTM + MaxPooling
 - MRR: 0.0147 (didn't converge)

Pure CNN Model:
 - 4000 filters
 - Maxpooling
 - After 120 epochs, InsuranceQA MRR = 0.530 and Top-1 Precision = 0.401
   - Decreased slightly with further training due to overfitting
 - Mixing together different filter lengths improved it to MRR = 0.56 and Top-1 Precision = 0.44
 - Validation loss around 6e-4 (lower than maxpooling) underperformed compared to maxpooling

Embedding + MaxPooling:
 - I can't believe this model performed so well. It blew the other ones out of the water, and trains ridiculously quickly.
 - Test 1: Top-1 Precision = 0.4922, MRR = 0.6239
 - Test 2: Top-1 Precision = 0.4817, MRR = 0.6110
 - Dev: Top-1 Precision = 0.4950, MRR = 0.6244
 - With 100 dimensions, Top-1 Precision = 0.281, MRR = 0.417 on test 1
 - Adding more embedding dimensions (beyond 1000) didn't lead to an improvement
 - Converted after about 20 epochs
 - Validation loss was around 7e-4 (with margin of 0.009)
 - Configuration:
    conf = {
        'question_len': 100,
        'answer_len': 100,
        'n_words': 22353, # len(vocabulary) + 1
        'margin': 0.009,

        'training_params': {
            'save_every': 1,
            'eval_every': 20,
            'batch_size': 128,
            'nb_epoch': 1000,
            'validation_split': 0.2,
            'optimizer': 'adam',
            # 'n_eval': 20,
        },

        'model_params': {
            'n_embed_dims': 1000,
            'n_hidden': 200,

            # convolution
            'nb_filters': 1000,
            'conv_activation': 'relu',

            # recurrent
            'n_lstm_dims': 300,
        },

        'similarity_params': {
            'mode': 'cosine',
            'gamma': 1,
            'c': 1,
            'd': 2,
        }
    }


Model described in paper (Dense + CNN):
 - Top 1 precision:
   - 0.183 on test 1
   - 0.178 on test 2
   - 0.204 on dev
 - MRR:
   - 0.328 on test 1
   - 0.319 on test 2
   - 0.343 on dev
 - Why won't this train better :(

 - Best validation loss was about 0.0011 (for margin of 0.009)
 - After increasing the number of parameters/embedding size the model improved a bit
 - Top 1 precision:
   - 0.327 on test 1
   - 0.302 on test 2
 - MRR:
   - 0.459 on test 1
   - 0.433 on test 2

 - Pre-trained the embedding layer via the EmbeddingModel, re-ran
 - Top 1 precision:
   - 0.493 on test 1
   - 0.427 on test 2
   - 0.470 on dev
 - MRR:
   - 0.620 on test 1
   - 0.557 on test 2
   - 0.596 on dev

1000 embed dims + 2000 CNN filters (total over 4 lengths):
 - Loss ~6e-4 for margin of 0.009
 - Test 1: 0.409 Top-1 Precision, 0.543 MRR
 - Test 2: 0.376 Top-1 Precision, 0.507 MRR