From ab6438f65c6ee0ec8af89e656c9035a997d4b67b Mon Sep 17 00:00:00 2001 From: Mike Clark Date: Fri, 6 Mar 2020 23:32:05 +0000 Subject: [PATCH 1/2] papers in readme --- readme.md | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/readme.md b/readme.md index adf79a0..4bc2193 100644 --- a/readme.md +++ b/readme.md @@ -206,14 +206,26 @@ I'm very grateful for all these authors for sharing their work. It was a pleasur Neural process papers: -- [2019, Attentive Neural Processes](https://arxiv.org/abs/1910.09323) (using attention to prevent underfitting) -- [2019, Functional Neural Processes](https://arxiv.org/abs/1906.08324) -- [2019, Recurrent Neural Processes](https://arxiv.org/abs/1906.05915) (2d and 3d over time) -- [2019, Spatiotemporal Modeling using Recurrent -Neural Processes](https://www.ri.cmu.edu/wp-content/uploads/2019/08/msr_thesis_document.pdf) (infilling spatial information, using a RNN for time information, no code) -- [2018, Conditional Neural Processes](https://arxiv.org/abs/1807.01613) [code](https://github.com/deepmind/neural-processes) -- [2018, Neural Processes](https://arxiv.org/abs/1807.01622) + +Neural process papers: + +- [2019-12-12, "Probing Uncertainty Estimates of Neural Processes"](http://bayesiandeeplearning.org/2019/papers/125.pdf) +- [2019-10-17, "Recurrent Attentive Neural Process for Sequential Data"](https://arxiv.org/abs/1910.09323) - LSTM on X before encoder, no code +- [2019-10-29, "Convolutional Conditional Neural Processes"](https://arxiv.org/abs/1910.13556). [code](https://github.com/cambridge-mlg/convcnp) +- [2019-10-01, "Wasserstein Neural Processes"](https://arxiv.org/abs/1910.00668) would be helpfull if the output dist never converges for your problem +- [2019-08-08, "Spatiotemporal Modeling using Recurrent Neural Processes"](https://www.ri.cmu.edu/wp-content/uploads/2019/08/msr_thesis_document.pdf) (infilling spatial information, using a RNN for time information, no code) +- [2019-06-13, "Recurrent Neural Processes"](https://arxiv.org/abs/1906.05915) (2d and 3d over time, using LSTM in encoder/decoder, no code) +- [2019-06-19, "The Functional Neural Processes"](https://arxiv.org/abs/1906.08324) +- [2018-12-03, "Empirical Evaluation of Neural Process Objectives"](http://bayesiandeeplearning.org/2018/papers/92.pdf) +- [2019-01-17, "Attentive Neural Processes"](https://arxiv.org/abs/1901.05761) (using attention to prevent underfitting) [code](https://github.com/deepmind/neural-processes) +- [2018-07-04, "Conditional Neural Processes"](https://arxiv.org/abs/1807.01613) [code](https://github.com/deepmind/neural-processes) +- [2019-06-24, "Sequential Neural Processes"](https://arxiv.org/abs/1906.10264) [code](https://github.com/singhgautam/snp) modelling a 1 or 2d process evolving over time +- [2018-07-04, "Neural Processes"](https://arxiv.org/abs/1807.01622) Blogposts: -- [2018, Neural Processes as distributions over functions +- [2018-08-10, "Neural Processes as distributions over functions" ](https://kasparmartens.rbind.io/post/np/) + +# Citing + +If you like our work and end up using this code for your reseach give us a shout-out by citing or acknowledging From e2e96cfe857561e5b3d53d24927c9dbaf655ae42 Mon Sep 17 00:00:00 2001 From: Mike Clark Date: Fri, 6 Mar 2020 23:36:12 +0000 Subject: [PATCH 2/2] Update readme.md --- readme.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/readme.md b/readme.md index 4bc2193..46ed4a0 100644 --- a/readme.md +++ b/readme.md @@ -192,6 +192,23 @@ Changes for stability: ![](docs/anp-rnn-mcdropout.png) +## Tips + +- Make you normalise all data, ideally the output two, this seems to be very important +- Batchnorm, lvar, dropout: these seem ok but it's unclear to me how to make these help reliably. Attention dropout or lstm dropout can be especially unreliable. +- sometimes you need quite a large hidden space to model a process. Making the network deep seems to stop it learning effectivly. It would be helpfull to try differen't activations, initializations and make sure the gradient flows effectivly to deeper networks, +- The deterministic path had unclear value, I found it best to leave it out +- The absolute size and comparitive size of the context and target is important for performance. + - If the context is too long and complex the model cannot summarize it + - If the target is too long and complex the model cannot fit it well + - If the context is in the target, the model may collapse to just fitting this. To fix + - make it small + - or make the loss on this part downweighted, this seems like the best approach since x_context->y_context may still be a usefull secondary task + - or do not include context in target + - however including the target in the context may sometimes be helpfull +- This repo compares models, but the biggest difference in this situation would be from additional data sources, but that is outside the scope of these experiments + + ## See also: A list of projects I used as reference or modified to make this one: @@ -203,10 +220,6 @@ A list of projects I used as reference or modified to make this one: I'm very grateful for all these authors for sharing their work. It was a pleasure to dive deep into these models compare the different implementations. - -Neural process papers: - - Neural process papers: - [2019-12-12, "Probing Uncertainty Estimates of Neural Processes"](http://bayesiandeeplearning.org/2019/papers/125.pdf)