From b24bb9d06f82888799c88c1fe1f38180c1c67efc Mon Sep 17 00:00:00 2001 From: Less Wright Date: Mon, 12 Apr 2021 18:48:15 -0700 Subject: [PATCH] Update README.md --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3f3a89e..fc14cfb 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,9 @@ Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective ### Ranger21 Status:
April 12 - positive negative momentum added, madgrad core checked in Testing over the weekend showed that positive negative momentum works really well, and even better with GC. Code is a bit messy atm b/c also tested Adaiw, but did not do that well so removed and added pos negative momentum. -Pos Neg momentum is a new technique to add parameter based, anisotropic noise to the gradient which helps it settle into flatter minima and also escape saddle points. In other words, better results. +Pos Neg momentum is a new technique to add parameter based, anisotropic noise to the gradient which helps it settle into flatter minima and also escape saddle points. +In other words, better results. +
Link to their excellent paper: https://arxiv.org/abs/2103.17182