From abf3347a1f6962c7a1ee3bd718fc6f12dbe13d42 Mon Sep 17 00:00:00 2001 From: Less Wright Date: Mon, 12 Apr 2021 18:47:24 -0700 Subject: [PATCH 1/7] Update README.md --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index ea8a05d..3f3a89e 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,16 @@ Ranger, with Radam + Lookahead core, is now 1.5 years old. In the interim, a nu Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective of some of the most impressive papers this past year. The focus for Ranger21 is that these internals will be parameterized, and where possible, automated, so that you can easily test and leverage some of the newest concepts in AI training, to optimize the optimizer on your respective dataset. ### Ranger21 Status:
+ April 12 - positive negative momentum added, madgrad core checked in Testing over the weekend showed that positive negative momentum works really well, and even better with GC. +Code is a bit messy atm b/c also tested Adaiw, but did not do that well so removed and added pos negative momentum. +Pos Neg momentum is a new technique to add parameter based, anisotropic noise to the gradient which helps it settle into flatter minima and also escape saddle points. In other words, better results. +Link to their excellent paper: +https://arxiv.org/abs/2103.17182 + +You can toggle between madgrad or not with the use_madgrad = True/False flag: +![ranger21_use_madgrad_toggle](https://user-images.githubusercontent.com/46302957/114484623-6c1f9500-9bbf-11eb-84f0-830859556856.JPG) + + April 10 - madgrad core engine integrated Madgrad has been added in a way that you will be able to select to use MadGrad or Adam as the core 'engine' for the optimizer. Thus, you'll be able to simply toggle which opt engine to use, as well as the various enhancements (warmup, stable weight decay, gradient_centralization) and thus quickly find the best optimization setup for your specific dataset. From b24bb9d06f82888799c88c1fe1f38180c1c67efc Mon Sep 17 00:00:00 2001 From: Less Wright Date: Mon, 12 Apr 2021 18:48:15 -0700 Subject: [PATCH 2/7] Update README.md --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3f3a89e..fc14cfb 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,9 @@ Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective ### Ranger21 Status:
April 12 - positive negative momentum added, madgrad core checked in Testing over the weekend showed that positive negative momentum works really well, and even better with GC. Code is a bit messy atm b/c also tested Adaiw, but did not do that well so removed and added pos negative momentum. -Pos Neg momentum is a new technique to add parameter based, anisotropic noise to the gradient which helps it settle into flatter minima and also escape saddle points. In other words, better results. +Pos Neg momentum is a new technique to add parameter based, anisotropic noise to the gradient which helps it settle into flatter minima and also escape saddle points. +In other words, better results. +
Link to their excellent paper: https://arxiv.org/abs/2103.17182 From b16922f9f7d2c4684fe02376a2be9cc4a763d8ec Mon Sep 17 00:00:00 2001 From: Less Wright Date: Sat, 17 Apr 2021 14:58:14 -0700 Subject: [PATCH 3/7] Update README.md --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index fc14cfb..f4bc82c 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,15 @@ Ranger, with Radam + Lookahead core, is now 1.5 years old. In the interim, a nu Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective of some of the most impressive papers this past year. The focus for Ranger21 is that these internals will be parameterized, and where possible, automated, so that you can easily test and leverage some of the newest concepts in AI training, to optimize the optimizer on your respective dataset. ### Ranger21 Status:
+ April 17 - building benchmark dataset(s) As a cost effective way of testing Ranger21 and it's various options, currently taking a subset of ImageNet categories and building out at the high level an "ImageSubNet50" and also a few sub category datasets. These are similar in spirit to ImageNette and ImageWoof, but hope to make a few relative improvements including pre-sizing to 224x224 for speed of training/testing. +First sub-dataset in progress in ImageBirds, which includes: +n01614925 bald eagle +n01616318 vulture +n01622779 grey owl +n01806143 peacock +n01833805 hummingbird +This is a medium-fine classification problem and will use as first tests for this type of benchmarking. Ideally, will make a seperate repo for the ImageBirds shortly to make it available for people to use though hosting the dataset poses a cost problem... + April 12 - positive negative momentum added, madgrad core checked in Testing over the weekend showed that positive negative momentum works really well, and even better with GC. Code is a bit messy atm b/c also tested Adaiw, but did not do that well so removed and added pos negative momentum. Pos Neg momentum is a new technique to add parameter based, anisotropic noise to the gradient which helps it settle into flatter minima and also escape saddle points. From 59ea0f582f5bfd5975269203acb32198abfc1747 Mon Sep 17 00:00:00 2001 From: Less Wright Date: Sat, 17 Apr 2021 14:59:53 -0700 Subject: [PATCH 4/7] Update README.md --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index f4bc82c..a660b0c 100644 --- a/README.md +++ b/README.md @@ -7,12 +7,13 @@ Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective ### Ranger21 Status:
April 17 - building benchmark dataset(s) As a cost effective way of testing Ranger21 and it's various options, currently taking a subset of ImageNet categories and building out at the high level an "ImageSubNet50" and also a few sub category datasets. These are similar in spirit to ImageNette and ImageWoof, but hope to make a few relative improvements including pre-sizing to 224x224 for speed of training/testing. -First sub-dataset in progress in ImageBirds, which includes: -n01614925 bald eagle -n01616318 vulture -n01622779 grey owl -n01806143 peacock -n01833805 hummingbird +First sub-dataset in progress in ImageBirds, which includes:
+n01614925 bald eagle
+n01616318 vulture
+n01622779 grey owl
+n01806143 peacock
+n01833805 hummingbird
+
This is a medium-fine classification problem and will use as first tests for this type of benchmarking. Ideally, will make a seperate repo for the ImageBirds shortly to make it available for people to use though hosting the dataset poses a cost problem... April 12 - positive negative momentum added, madgrad core checked in Testing over the weekend showed that positive negative momentum works really well, and even better with GC. From bc78e88fde68c7b19a77981be87ab10996ced95e Mon Sep 17 00:00:00 2001 From: Less Wright Date: Sun, 18 Apr 2021 12:28:56 -0700 Subject: [PATCH 5/7] Update README.md --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index a660b0c..2c55def 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,12 @@ Ranger, with Radam + Lookahead core, is now 1.5 years old. In the interim, a nu Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective of some of the most impressive papers this past year. The focus for Ranger21 is that these internals will be parameterized, and where possible, automated, so that you can easily test and leverage some of the newest concepts in AI training, to optimize the optimizer on your respective dataset. ### Ranger21 Status:
+ April 18 AM - chebyshev fractals added, cosine warmdown (cosine decay) added
+Chebyshev performed reasonably well, but still needs more work before recommending so it's defaulting to off atm. +There are two papers providing support for using Chebyshev, one of which is: +https://arxiv.org/abs/2010.13335v1
+Cosine warmdown has been added so that the default lr schedule for Ranger21 is linear warmup, flat run at provided lr, and then cosine decay of lr starting at the X% passed in. (Default is .65). + April 17 - building benchmark dataset(s) As a cost effective way of testing Ranger21 and it's various options, currently taking a subset of ImageNet categories and building out at the high level an "ImageSubNet50" and also a few sub category datasets. These are similar in spirit to ImageNette and ImageWoof, but hope to make a few relative improvements including pre-sizing to 224x224 for speed of training/testing. First sub-dataset in progress in ImageBirds, which includes:
n01614925 bald eagle
From 237522ee236e2a349e98c19dafab3042b734cef8 Mon Sep 17 00:00:00 2001 From: Less Wright Date: Sun, 18 Apr 2021 13:53:44 -0700 Subject: [PATCH 6/7] Update README.md --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 2c55def..26d605d 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,12 @@ Ranger, with Radam + Lookahead core, is now 1.5 years old. In the interim, a nu Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective of some of the most impressive papers this past year. The focus for Ranger21 is that these internals will be parameterized, and where possible, automated, so that you can easily test and leverage some of the newest concepts in AI training, to optimize the optimizer on your respective dataset. ### Ranger21 Status:
+ April 18 PM - Adaptive gradient clipping added, thanks for suggestion and code from @kayuksel. AGC is used in NFNets to replace BN. For our use case here, it's to have a smarter gradient clipping algo vs the usual hard clipping, and ideally better stabilize training. + +Here's how the Ranger21 settings output looks atm: +![ranger21_settings](https://user-images.githubusercontent.com/46302957/115160522-7a513380-a04d-11eb-80a9-871f99da798e.JPG) + + April 18 AM - chebyshev fractals added, cosine warmdown (cosine decay) added
Chebyshev performed reasonably well, but still needs more work before recommending so it's defaulting to off atm. There are two papers providing support for using Chebyshev, one of which is: From d56b006927f00eb14e6a5dacf4765dbaaaa13eb4 Mon Sep 17 00:00:00 2001 From: Less Wright Date: Sun, 18 Apr 2021 13:54:06 -0700 Subject: [PATCH 7/7] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 26d605d..d1af581 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Ranger, with Radam + Lookahead core, is now 1.5 years old. In the interim, a nu Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective of some of the most impressive papers this past year. The focus for Ranger21 is that these internals will be parameterized, and where possible, automated, so that you can easily test and leverage some of the newest concepts in AI training, to optimize the optimizer on your respective dataset. ### Ranger21 Status:
- April 18 PM - Adaptive gradient clipping added, thanks for suggestion and code from @kayuksel. AGC is used in NFNets to replace BN. For our use case here, it's to have a smarter gradient clipping algo vs the usual hard clipping, and ideally better stabilize training. + April 18 PM - Adaptive gradient clipping added, thanks for suggestion and code from @kayuksel. AGC is used in NFNets to replace BN. For our use case here, it's to have a smarter gradient clipping algo vs the usual hard clipping, and ideally better stabilize training. Here's how the Ranger21 settings output looks atm: ![ranger21_settings](https://user-images.githubusercontent.com/46302957/115160522-7a513380-a04d-11eb-80a9-871f99da798e.JPG)