From 2f4269eef302fdbfeb8ee69be6d16b8139380739 Mon Sep 17 00:00:00 2001 From: Less Wright Date: Sun, 25 Apr 2021 13:48:50 -0700 Subject: [PATCH] Update README.md --- README.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/README.md b/README.md index 33d0f38..5187479 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,28 @@ Ranger, with Radam + Lookahead core, is now 1.5 years old. In the interim, a nu Thus, Ranger21 (as in 2021) is a rewrite with multiple new additions reflective of some of the most impressive papers this past year. The focus for Ranger21 is that these internals will be parameterized, and where possible, automated, so that you can easily test and leverage some of the newest concepts in AI training, to optimize the optimizer on your respective dataset. +#### Latest Simple Benchmark comparison (Image classification, dog breed subset of ImageNet, ResNet-18):
+Ranger 21:
+Accuracy: 74.02% Validation Loss: 15.00
+ +Adam:
+Accuracy: 64.84% Validation Loss: 17.19
+ +Net results: 14.15% greater accuracy with Ranger21 vs Adam, same training epochs.
+ + ### Ranger21 Status:
+ April 25 - Fixed warmdown calculation error, moved to Linear warmdown, new high in benchmark: Found that there was an error in the warmdown calculations. Fixed and also moved to linear warmdown. This resulted in another new high for the simple benchmark, with results now moved to above so they don't get lost in the updates section. +Note that the warmdown now calculates based on the decay between the full lr, to the minimal lr (defaults to 3e-5), rather than previously declining to 0. +
+Note that you can display the lr curves directly by simply using: +~~~ +lr_curve = optimizer.tracking_lr +plt.plot(lr_curve) +~~~ +Ranger21 internally tracks the lr per epoch for this type of review. +Additional updates include adding a 'clear_cache' to reset the cached lookahead params, and also moved the lookahead procesing to it's own function and cleaned up some naming conventions. Will use item_active=True/False rather than the prior using_item=True/False to keep the code simpler as now item properties are alpha grouped vs being cluttered into the using_item layout. +
April 24 - New record on benchmark with NormLoss, Lookahead, PosNeg momo, Stable decay etc. all combined NormLoss and Lookahead integrated into Ranger21 set a new high on our simple benchmark (ResNet 18, subset of ImageWoof).
Best Accuracy = 73.41 Best Val Loss = 15.06