wassname c792ad3e5f Add LoRA-XS variant: train only r×r core R between frozen SVD factors ...

Bałazy et al. 2024 (arxiv 2405.17604). A=diag(Sr)Vhr, B=Ur frozen from
top-r SVD of W (W left intact); only the r×r R is trained, init normal(0,1e-5)
so the adapter ~ identity at t=0. ~25k params at r=32 (24 down_proj targets).
justfile: alpha=r (scale=1) and lr=4e-3, matching the ref LLaMA math config.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>

2026-06-18 19:48:40 +08:00

..

test_metamath_gsm8k_benchmark.py

tyro and benchmark

2026-04-27 06:23:30 +08:00

test_metamath_smoke.py

Add LoRA-XS variant: train only r×r core R between frozen SVD factors

2026-06-18 19:48:40 +08:00