wassname/Ranger21
1
0
Fork 0
mirror of https://github.com/wassname/Ranger21.git synced 2026-09-09 11:15:43 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Less Wright
2021-04-06 20:45:37 -07:00
committed by GitHub
parent 5fd5ad731c
commit b27ec7e9ea
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -2
View File
@@ -1,4 +1,4 @@
# Ranger21
# Ranger21 - integrating the latest deep learning components into a single optimizer
Ranger deep learning optimizer rewrite to use newest components
Ranger, with Radam + Lookahead core, is now 1.5 years old. In the interim, a number of new developments have happened including the rise of Transformers for Vision.
@@ -17,7 +17,7 @@ April 5 - stable weight decay added. Quick testing shows nice results with 1e-4
Current feature set planned:</br>
1 - <b>feature complete</b> - Linear and Exponential warmup in place of RAdam. This is based on the findings of https://arxiv.org/abs/1910.04209v3
1 - <b>feature complete</b> - automated, Linear and Exponential warmup in place of RAdam. This is based on the findings of https://arxiv.org/abs/1910.04209v3
2 - MadGrad core engine in place of Adam internals. This is based on my own testing with Vision Transformers as well as the compelling MadGrad paper: https://arxiv.org/abs/2101.11075v1