diff --git a/docs/videos/beales_CyclicLR_20171117_04-51-12_2d.gif b/docs/videos/beales_CyclicLR_20171117_04-51-12_2d.gif new file mode 100644 index 0000000..75f7180 Binary files /dev/null and b/docs/videos/beales_CyclicLR_20171117_04-51-12_2d.gif differ diff --git a/docs/videos/beales_CyclicLR_20171117_04-51-12_loss.gif b/docs/videos/beales_CyclicLR_20171117_04-51-12_loss.gif new file mode 100644 index 0000000..de59e2b Binary files /dev/null and b/docs/videos/beales_CyclicLR_20171117_04-51-12_loss.gif differ diff --git a/readme.md b/readme.md index 9c3d799..c8e4065 100644 --- a/readme.md +++ b/readme.md @@ -1,17 +1,16 @@ This project generates animations of pytorch optimizers solving toy problems. Examples Below. -[Some nice animations](http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html) were posted a few years ago by Alex Radford but didn't include the Adam optimizer or landscapes with noise. [Louis Tiao](http://louistiao.me/notes/visualizing-and-animating-optimization-algorithms-with-matplotlib/) blogged about how to make the visualizations. The [pytorch unit tests](https://github.com/pytorch/pytorch/blob/master/test/test_optim.py) show how to run the optimizers on test functions. I pulled these together and shared the result at https://github.com/wassname/viz_torch_optim. Please make some better animations and share them. +[Some nice animations](http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html) were posted a few years ago by Alex Radford but didn't include the Adam optimizer or landscapes with noise. [Louis Tiao](http://louistiao.me/notes/visualizing-and-animating-optimization-algorithms-with-matplotlib/) blogged about how to make the visualizations. The [pytorch unit tests](https://github.com/pytorch/pytorch/blob/master/test/test_optim.py) show how to run the optimizers on test functions. I pulled these together and shared the result at https://github.com/wassname/viz_torch_optim. # Examples -Please note each optimizer has a differen't learning rate. This is because simpler optimizers perform better on low dimensional problems. So, with a constant learning rate, the simpler SGD optimizer races while Adam crawls along. In that case we would be able to see SGD's path or Adam's movement. So I used differen't learning rates for each optimizer in order to show them on the same video. +Please note that each optimizer has a differen't learning rate, so they are not directly comparable. This is because simpler optimizers perform better on low dimensional problems. So, with a constant learning rate, the simpler SGD optimizer races to the finish while Adam crawls along. In this case we SGD is to fast to see and Adam is too bording to watch. So I used differen't learning rates for each optimizer so I could show them on the same video. ## With cyclic annealing: -![](docs/videos/beales_CyclicLR_20171117_04-01-14_2d.gif) +![](docs/videos/beales_CyclicLR_20171117_04-51-12_2d.gif) ![](docs/videos/beales_CyclicLR_20171117_04-51-12_loss.gif) - ## Constant learning rate ### Beales function