diff --git a/readme.md b/readme.md index 4bc2193..46ed4a0 100644 --- a/readme.md +++ b/readme.md @@ -192,6 +192,23 @@ Changes for stability: ![](docs/anp-rnn-mcdropout.png) +## Tips + +- Make you normalise all data, ideally the output two, this seems to be very important +- Batchnorm, lvar, dropout: these seem ok but it's unclear to me how to make these help reliably. Attention dropout or lstm dropout can be especially unreliable. +- sometimes you need quite a large hidden space to model a process. Making the network deep seems to stop it learning effectivly. It would be helpfull to try differen't activations, initializations and make sure the gradient flows effectivly to deeper networks, +- The deterministic path had unclear value, I found it best to leave it out +- The absolute size and comparitive size of the context and target is important for performance. + - If the context is too long and complex the model cannot summarize it + - If the target is too long and complex the model cannot fit it well + - If the context is in the target, the model may collapse to just fitting this. To fix + - make it small + - or make the loss on this part downweighted, this seems like the best approach since x_context->y_context may still be a usefull secondary task + - or do not include context in target + - however including the target in the context may sometimes be helpfull +- This repo compares models, but the biggest difference in this situation would be from additional data sources, but that is outside the scope of these experiments + + ## See also: A list of projects I used as reference or modified to make this one: @@ -203,10 +220,6 @@ A list of projects I used as reference or modified to make this one: I'm very grateful for all these authors for sharing their work. It was a pleasure to dive deep into these models compare the different implementations. - -Neural process papers: - - Neural process papers: - [2019-12-12, "Probing Uncertainty Estimates of Neural Processes"](http://bayesiandeeplearning.org/2019/papers/125.pdf)