wassname
9d027752ad
antipasto_arrow -> antipasto_dplr. The arrowhead's dense b x b block is the wrong
shape: b^2 params, mixes only the top-b, and sits on the S-scaled coords so its
perturbation is amplified by the largest singular values (block=128 collapsed to
45.7% at the gain's lr). Replace it with LoRA's lesson -- a low-rank core inside
the frozen basis, ADDED to the gain:
DeltaW = U [diag(S_eff) + coeff * B A] Vh, A:(k,r) B:(r,k), B=0 at init
The low-rank part mixes the whole top-r subspace for 2*r*k params (k=LoRA's rank),
and being additive (not * diag(S)) it is S-independent -- the amplification edge is
gone by construction. Diagonal gain unchanged; identity at init from B=0 and g=0.
Wired through benchmark (antipasto_lora_rank, run_id __k suffix), justfile, cost_report,
smoke (green, dplr attaches/trains/round-trips). Arrow code removed; its run results
stay on disk for comparison.
Co-Authored-By: Claudypoo <noreply@anthropic.com>
lora-lite
Hackable PyTorch adapters for LoRA-family and small PEFT experiments.
Hackable code
To keep it simple and hackable we make these choices:
- Simple forward hooks, no module replacement or custom modules.
- Simple code over fast performance
- No merge/unmerge
- Single test where we train on MetaMathQA and test on GSM8K for each variant
Take a look at lora.py
Install
pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite
Quickstart
import torch, lora_lite as ll
model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...
ll.save(model, "adapter.safetensors")
ll.detach(model)
ll.load(model, "adapter.safetensors")
Does it work?
just check # pytest + smoke + package build + metadata check
just bnb-smoke # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe # Qwen/Qwen3-0.6B train/save-load probe
Variants
| Variant | 4bit/8bit | GSM8K % | Params | Peak GPU (GB) |
|---|---|---|---|---|
| LoRA | yes | 63.2% | 4.59M | 11.3 |
| PiSSA | no | 63.2% | 4.59M | 11.3 |
| DoRA | no | 62.4% | 4.67M | 11.3 |
| DeLoRA | yes | 61.5% | 4.59M | 11.3 |
| AntiPaSTO | no | 61.4% | 14.3K | 11.3 |
| AntiPaSTO-CorDA | no | 61.9% | 14.3K | 11.3 |
| AntiPaSTO-ablate | no | 61.0% | 14.4K | 11.3 |
| AntiPaSTO-arrow | no | 60.5% | 17.5K | 11.3 |
| IA3-FF | yes | 61.4% | 86K | 11.4 |
| EVA | no | 60.3% | 4.59M | 11.3 |
| IA3 | yes | 60.0% | 57K | 11.4 |
| HRA | yes | 61.6% | 1.84M | 11.3 |
Params = trainable adapter params. Peak GPU = peak CUDA memory during train+eval (logged from this run onward; older runs predate the column).
We validate our adapters the same way PEFT does. If acting properly they train on a 5k-step MetaMathQA subset and test 49%+ on GSM8K; as you can see, all adapters pass this mark. See this file for more details.
AntiPaSTO is the novel row here: instead of adding trainable directions like LoRA, it freezes W's own top-r SVD and learns only a bounded per-direction gain S_eff = S * (1 + ELU(g)). The singular basis stays fixed and interpretable, and the adapter is O(r) params (~320x smaller than LoRA). The variants change only the basis or core: CorDA orients it by input covariance (Yang+ 2024), ablate learns a contractive directional ablation (Arditi+ 2024), arrow adds a small dense block for cross-direction mixing.
Developer docs
See docs/developer_guide.md for the variant API, data-calibrated init, and save/load format.
Citation
@misc{wassname2026loralite,
title = {LoRA-Lite: A Hackable Adapter Library for Research},
author = {Michael J. Clark},
year = {2026},
url = {https://github.com/wassname/lora-lite/}
}