From 5bc90c35017a72632a8dc375a4fdda65c134def5 Mon Sep 17 00:00:00 2001 From: Less Wright Date: Mon, 5 Apr 2021 20:25:50 -0700 Subject: [PATCH] Update README.md --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 6b67b8d..fb8209d 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,9 @@ Ranger with Radam + Lookahead core is now 1.5 years old. In the interim, a numb Thus, Ranger21 (as in 2021) is a rewrite with the following components planned to be added. The idea is that these internals will be paramaterized so that you can mix and match to optimize the optimizer on your respective dataset. +Satus: +4/5 - stable weight decay added. Quick testing shows nice results with 1e-4 weight decay on subset of ImageNet. + Current feature set planned: 1 - Linear and Exponential warmup in place of RAdam. This is based on the findings of https://arxiv.org/abs/1910.04209v3