A really minor change. Should we input logits instead of probs into `Categorical`? It will do less converting from logits and back, leading to increased stability during training (theoretically).
Not sure if your interested in adding this, it just supports saving and loading SharedReplay. For example if you want to load previous DDPG experience.
I tracked some NaN's I was getting down to here. It happens when std is a small number, then var is even smaller, and log_density=inf. There are some problems where the agent will learn to use small standard deviations because jittering movements have a high cost, this makes those more stable.
I think you intended this to be outside the softplus. The reason is that it should be applied just before the log to avoid `log(0)=inf`.e.g.
- `log(softplus(-1000+1e-5))=log(0)=inf`.
- `log(softplus(-1000)+1e-5)=log(1e-5)!=inf`.
Also this fixes a NaN I had.
Mike Clark