viz_torch_optim/readme.md

This project generates animations of pytorch optimizers solving toy problems. Examples Below.

Some nice animations were posted a few years ago by Alex Radford but didn't include the Adam optimizer or landscapes with noise. Louis Tiao blogged about how to make the visualizations. The pytorch unit tests show how to run the optimizers on test functions. I pulled these together and shared the result at https://github.com/wassname/viz_torch_optim. Please make some better animations and share them.

Examples

Please note that these don't have the same learning rate as simple optimizers perform better on low dimensional problems. So with a constant learning rate SGD gets to the minima too fast to see while Adam crawls along. So I used differen't learning rate per optimizer to show them in the same video.

With cyclic annealing:

Constant learning rate

Beales function

Six humped camel function

Madsen function

Rosenbrock function

Usage:

• git clone https://github.com/wassname/viz_torch_optim
• jupyter notebook
• open main.ipynb
• install any missing dependencies with pip