wassname
fa69e0cac3
Replace the per-experiment family breakdown table + comparison prose with a 2-sentence method description (frozen interpretable SVD basis, O(r) gain, the three variant cores). Experiment findings (rotation comparison, arrow capacity, cost/timing) belong in the research journal, not the README skim path. Co-Authored-By: Claudypoo <noreply@anthropic.com>
3.5 KiB
lora-lite
Hackable PyTorch adapters for LoRA-family and small PEFT experiments.
Hackable code
To keep it simple and hackable we make these choices:
- Simple forward hooks, no module replacement or custom modules.
- Simple code over fast performance
- No merge/unmerge
- Single test where we train on MetaMathQA and test on GSM8K for each variant
Take a look at lora.py
Install
pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite
Quickstart
import torch, lora_lite as ll
model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...
ll.save(model, "adapter.safetensors")
ll.detach(model)
ll.load(model, "adapter.safetensors")
Does it work?
just check # pytest + smoke + package build + metadata check
just bnb-smoke # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe # Qwen/Qwen3-0.6B train/save-load probe
Variants
| Variant | 4bit/8bit | GSM8K % | Params | Peak GPU (GB) |
|---|---|---|---|---|
| LoRA | yes | 63.2% | 4.59M | 11.3 |
| PiSSA | no | 63.2% | 4.59M | 11.3 |
| DoRA | no | 62.4% | 4.67M | 11.3 |
| DeLoRA | yes | 61.5% | 4.59M | 11.3 |
| AntiPaSTO | no | 61.4% | 14.3K | 11.3 |
| IA3-FF | yes | 61.4% | 86K | 11.4 |
| EVA | no | 60.3% | 4.59M | 11.3 |
| IA3 | yes | 60.0% | 57K | 11.4 |
| HRA | yes | 61.6% | 1.84M | 11.3 |
Params = trainable adapter params. Peak GPU = peak CUDA memory during train+eval (logged from this run onward; older runs predate the column).
Setup: Qwen3-0.6B-Base, MetaMathQA train (5k steps, batch 4 = 20k samples unless noted), r=32, all q/v targets, GSM8K test (1319 examples). HRA used batch 2 (10k samples) due to memory. The AntiPaSTO family used r=256 (default for these variants).
Reference: PEFT reports LoRA at 49.0% on Llama-3.2-3B (different model, different sample count). Our numbers are not directly comparable but suggest the adapters work.
AntiPaSTO freezes the top-r SVD of W and trains only a per-direction gain S_eff = S * (1 + ELU(g)), so the singular basis stays interpretable and the adapter is O(r) params (~320x smaller than LoRA). Variants swap the basis or core: antipasto_corda orients it by input covariance (CorDA), antipasto_ablate learns a contractive directional ablation (Arditi), antipasto_arrow adds a cheap dense block for cross-direction mixing. See src/lora_lite/variants/antipasto*.py.
Developer docs
See docs/developer_guide.md for the variant API, data-calibrated init, and save/load format.
Citation
@misc{wassname2026loralite,
title = {LoRA-Lite: A Hackable Adapter Library for Research},
author = {Michael J. Clark},
year = {2026},
url = {https://github.com/wassname/lora-lite/}
}