wassname/Ranger21
1
0
Fork 0
mirror of https://github.com/wassname/Ranger21.git synced 2026-09-09 11:15:43 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' of https://github.com/lessw2020/Ranger21 into main

Browse Source
This commit is contained in:
Less Wright
2021-04-05 20:34:03 -07:00
parent 75ab25f72c 5bc90c3501
commit 22c7afd27d
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3
View File
@@ -5,6 +5,9 @@ Ranger with Radam + Lookahead core is now 1.5 years old. In the interim, a numb
Thus, Ranger21 (as in 2021) is a rewrite with the following components planned to be added. The idea is that these internals will be paramaterized so that you can mix and match to optimize the optimizer on your respective dataset.
Satus:
4/5 - stable weight decay added. Quick testing shows nice results with 1e-4 weight decay on subset of ImageNet.
Current feature set planned:
1 - Linear and Exponential warmup in place of RAdam. This is based on the findings of https://arxiv.org/abs/1910.04209v3