wassname
d4ec550dd8
The benchmark only passed calibration_data to eva, so antipasto_corda's group_init hit `if calibration_data is None: return` and every corda run was actually plain SVD. The covariance orientation never executed -- all prior corda-vs-antipasto comparisons are void. - antipasto_corda.group_init: raise on None instead of silently degrading (orientation is the variant's whole identity; fail loud). - benchmark: feed ~256 MetaMath calibration samples (IPM, per PEFT/CorDA) to corda and to cov_orient ablate; run_id now carries an __lr tag. - adapter.save/load: a data-driven group_init rewrites the frozen base residual W_res into a form init() cannot reproduce at load (it only knows the plain top-r crop). Persist those residuals in the adapter and restore them. Fixes a reload-logits mismatch that was masked while group_init never ran. - probe check: compare every saved tensor (lora_ buffers AND base residuals) against the reloaded model state. - justfile: bench-variant gains an lr_override (the core wants a tamer lr than the gain's 5e-3). Co-Authored-By: Claudypoo <noreply@anthropic.com>
lora-lite
Hackable PyTorch adapters for LoRA-family and small PEFT experiments.
Hackable code
To keep it simple and hackable we make these choices:
- Simple forward hooks, no module replacement or custom modules.
- Simple code over fast performance
- No merge/unmerge
- Single test where we train on MetaMathQA and test on GSM8K for each variant
Take a look at lora.py
Install
pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite
Quickstart
import torch, lora_lite as ll
model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...
ll.save(model, "adapter.safetensors")
ll.detach(model)
ll.load(model, "adapter.safetensors")
Does it work?
just check # pytest + smoke + package build + metadata check
just bnb-smoke # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe # Qwen/Qwen3-0.6B train/save-load probe
Variants
| Variant | 4bit/8bit | GSM8K % | Params | Peak GPU (GB) |
|---|---|---|---|---|
| LoRA | yes | 63.2% | 4.59M | 11.3 |
| PiSSA | no | 63.2% | 4.59M | 11.3 |
| DoRA | no | 62.4% | 4.67M | 11.3 |
| DeLoRA | yes | 61.5% | 4.59M | 11.3 |
| AntiPaSTO | no | 61.4% | 14.3K | 11.3 |
| AntiPaSTO-CorDA | no | 61.9% | 14.3K | 11.3 |
| AntiPaSTO-ablate | no | 61.0% | 14.4K | 11.3 |
| AntiPaSTO-arrow | no | 60.5% | 17.5K | 11.3 |
| IA3-FF | yes | 61.4% | 86K | 11.4 |
| EVA | no | 60.3% | 4.59M | 11.3 |
| IA3 | yes | 60.0% | 57K | 11.4 |
| HRA | yes | 61.6% | 1.84M | 11.3 |
Params = trainable adapter params. Peak GPU = peak CUDA memory during train+eval (logged from this run onward; older runs predate the column).
We validate our adapters the same way PEFT does. If acting properly they train on a 5k-step MetaMathQA subset and test 49%+ on GSM8K; as you can see, all adapters pass this mark. See this file for more details.
AntiPaSTO is the novel row here: instead of adding trainable directions like LoRA, it freezes W's own top-r SVD and learns only a bounded per-direction gain S_eff = S * (1 + ELU(g)). The singular basis stays fixed and interpretable, and the adapter is O(r) params (~320x smaller than LoRA). The variants change only the basis or core: CorDA orients it by input covariance (Yang+ 2024), ablate learns a contractive directional ablation (Arditi+ 2024), arrow adds a small dense block for cross-direction mixing.
Developer docs
See docs/developer_guide.md for the variant API, data-calibrated init, and save/load format.
Citation
@misc{wassname2026loralite,
title = {LoRA-Lite: A Hackable Adapter Library for Research},
author = {Michael J. Clark},
year = {2026},
url = {https://github.com/wassname/lora-lite/}
}