wassname/lora-lite
1
0
Fork 0
mirror of https://github.com/wassname/lora-lite.git synced 2026-06-27 16:45:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: add HRA benchmark result (61.6%), update README table

Browse Source
This commit is contained in:
wassname
2026-04-27 20:07:19 +08:00
parent f6fd410677
commit b698331cfa
2 changed files with 27 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+9 -9
View File
@@ -49,23 +49,23 @@ just qwen-probe # Qwen/Qwen3-0.6B train/save-load probe
| Variant | 4bit/8bit | GSM8K % | Params | Peak GPU (GB) |
| --------------------------------------------- | --------- | ------- | ---------- | ------------- |
| [LoRA](https://arxiv.org/abs/2106.09685) | yes | 63.2% | 4.59M | — |
| [PiSSA](https://arxiv.org/abs/2404.02948) | no | 63.2% | 4.59M | — |
| [DoRA](https://arxiv.org/abs/2402.09353) | no | 62.4% | 4.67M | — |
| [DeLoRA](https://arxiv.org/abs/2503.18225) | yes | 61.5% | 4.59M | — |
| [LoRA](https://arxiv.org/abs/2106.09685) | yes | 63.2% | 4.59M | 11.3 |
| [PiSSA](https://arxiv.org/abs/2404.02948) | no | 63.2% | 4.59M | 11.3 |
| [DoRA](https://arxiv.org/abs/2402.09353) | no | 62.4% | 4.67M | 11.3 |
| [DeLoRA](https://arxiv.org/abs/2503.18225) | yes | 61.5% | 4.59M | 11.3 |
| [AntiPaSTO](https://arxiv.org/abs/2503.08696) | no | 61.4% | 35.8K | 11.5 |
| [IA3-FF](https://arxiv.org/pdf/2205.05638) | yes | 61.4% | 86K | 11.4 |
| [EVA](https://arxiv.org/abs/2409.07871) | no | 60.3% | 4.59M | 11.3 |
| [IA3](https://arxiv.org/pdf/2205.05638) | yes | 60.0% | 57K | 11.4 |
| [IA3-FF](https://arxiv.org/pdf/2205.05638) | yes | 61.4% | 86K | 11.4 |
| [HRA](https://arxiv.org/abs/2409.01434) | yes | — | — | — |
| [AntiPaSTO](https://arxiv.org/abs/2503.08696) | no | 30.6% | 4.5K | 11.3 |
| [HRA](https://arxiv.org/abs/2409.01434) | yes | 61.6% | 1.84M | 11.3 |
Params = trainable adapter params. Peak GPU = peak CUDA memory during train+eval (logged from this run onward; older runs predate the column).
Setup: Qwen3-0.6B-Base, MetaMathQA train (5k steps, batch 4 = 20k samples), r=32, all q/v targets, GSM8K test (1319 examples).
Setup: Qwen3-0.6B-Base, MetaMathQA train (5k steps, batch 4 = 20k samples unless noted), r=32, all q/v targets, GSM8K test (1319 examples). HRA used batch 2 (10k samples) due to memory. AntiPaSTO used r=256 (default for this variant).
Reference: PEFT reports LoRA at 49.0% on Llama-3.2-3B (different model, different sample count). Our numbers are not directly comparable but suggest the adapters work.
AntiPaSTO at 30.6% is expected: it has only 4.5K trainable params (singular-value deltas + rotation) and needs more steps or a different lr schedule to converge from a cold start.
AntiPaSTO at 59.5% with 4.5K trainable params (1000x fewer than LoRA's 4.59M). It trains singular-value deltas + block-Cayley rotation within the SVD subspace, so it can rescale and reorient existing directions but not create new ones. Higher rank (r>32) or data-driven dimension selection (from antipasto3) may close the gap further.
## Developer docs