wassname
7df786e80b
- _base_weight_fingerprint was PiSSA-only defensive check that cluttered every save with per-target SHA256. If you load onto wrong base, you get wrong weights -- that's user error, not a library bug. - test_lora_lite.py deleted. All coverage lives in test_metamath_smoke.py which runs the real benchmark pipeline per variant.
lora-lite
Hackable PyTorch adapters for LoRA-family and small PEFT experiments.
Hackable code
To keep it simple and hackable we make these choices:
- Simple forward hooks, no module replacement or custom modules.
- Simple code over fast performance
- No merge/unmerge
- Single test where we train on MetaMathQA and test on GSM8K for each variant
Take a look at lora.py
Install
pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite
Quickstart
import torch, lora_lite as ll
model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...
ll.save(model, "adapter.safetensors")
ll.detach(model)
ll.load(model, "adapter.safetensors")
Does it work?
just check # pytest + smoke + package build + metadata check
just bnb-smoke # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe # Qwen/Qwen3-0.6B train/save-load probe
Variants
| Variant | 4bit/8bit | GSM8K % | Params | Peak GPU (GB) |
|---|---|---|---|---|
| LoRA | yes | 63.2% | 4.59M | — |
| PiSSA | no | 63.2% | 4.59M | — |
| DoRA | no | 62.4% | 4.67M | — |
| DeLoRA | yes | 61.5% | 4.59M | — |
| EVA | no | 60.3% | 4.59M | 11.3 |
| IA3 | yes | 60.0% | 57K | 11.4 |
| IA3-FF | yes | 61.4% | 86K | 11.4 |
| HRA | yes | — | — | — |
| AntiPaSTO | no | 30.6% | 4.5K | 11.3 |
Params = trainable adapter params. Peak GPU = peak CUDA memory during train+eval (logged from this run onward; older runs predate the column).
Setup: Qwen3-0.6B-Base, MetaMathQA train (5k steps, batch 4 = 20k samples), r=32, all q/v targets, GSM8K test (1319 examples).
Reference: PEFT reports LoRA at 49.0% on Llama-3.2-3B (different model, different sample count). Our numbers are not directly comparable but suggest the adapters work.
AntiPaSTO at 30.6% is expected: it has only 4.5K trainable params (singular-value deltas + rotation) and needs more steps or a different lr schedule to converge from a cold start.
Developer docs
See docs/developer_guide.md for the variant API, data-calibrated init, and save/load format.
Citation
@misc{wassname2026loralite,
title = {LoRA-Lite: A Hackable Adapter Library for Research},
author = {Michael J. Clark},
year = {2026},
url = {https://github.com/wassname/lora-lite/}
}