wassname/Ranger21
1
0
Fork 0
mirror of https://github.com/wassname/Ranger21.git synced 2026-09-11 11:52:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Less Wright
2021-04-10 21:06:56 -07:00
committed by GitHub
parent cf730dd483
commit 8d9834017a
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3
View File
@@ -12,6 +12,9 @@ Thus, you'll be able to simply toggle which opt engine to use, as well as the va
Still testing things and then will update code here...
Gradient centralization good for both - first findings are gradient centralization definitely improves MadGrad (just like it does with Adam core) so will have GC on as default for both engines.
![madgrad_added_ranger21](https://user-images.githubusercontent.com/46302957/114292041-aca4d480-9a40-11eb-92b3-4243fd6d4390.JPG)
### LR selection is very different between MadGrad and Adam core engine:
One item - the starting lr for madgrad is very different (much higher) than with Adam....have done some testing with automated LR scheduling (HyperExplorer and ABEL), but that will be added later if it's successful. But if you simply plug your usual Adam LR's into Madgrad you won't be impressed :)