From 3c1cc095cd9c324a4d4c432f8ce336a6874d45d7 Mon Sep 17 00:00:00 2001 From: Less Wright Date: Tue, 27 Apr 2021 22:01:06 -0700 Subject: [PATCH] Ranger21 now training on ImageNet --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index b1e5303..ea2dccc 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,8 @@ Net results: 14.15% greater accuracy with Ranger21 vs Adam, same training epochs ### Ranger21 Status:
+ April 27 PM - Ranger21 now training on ImageNet! Starting work on benchmarking Ranger21 on ImageNet. Due to cost, will train to 40 epochs on ImageNet and compare with same setup with 40 epochs using Adam to have a basic "gold standard" comparison. Training is underway now, hope to have results end of this week. + April 26 PM - added smarter auto warmup based on Dickson Neoh report (tested with only 5 epochs), and first pip install setup thanks to @BrianPugh!
The warmup structure for Ranger21 is based on the paper by Ma/Yarats which uses the beta2 param to compute the default warmup. However, that also assumes we have a longer training run. @DNH on the fastai forums tested with 5 epochs which meant it never got past warmup phase. Thus have added a check for the % warmup relative to the total training time and will auto fall back to 30% (settable via warmup_pct_default) in order to account for shorter training runs.