lora-lite/README.md

lora-lite

Hackable PyTorch adapters for LoRA-family and small PEFT experiments.

Hackable code

To keep it simple and hackable we make these choices:

• Simple forward hooks, no module replacement or custom modules.
• Simple code over fast performance
• No merge/unmerge
• Single test where we train on MetaMathQA and test on GSM8K for each variant

Take a look at lora.py

Install

pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite

Quickstart

import torch, lora_lite as ll

model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)

opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...

ll.save(model, "adapter.safetensors")
ll.detach(model)
ll.load(model, "adapter.safetensors")

Does it work?

just check       # pytest + smoke + package build + metadata check
just bnb-smoke   # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe  # Qwen/Qwen3-0.6B train/save-load probe

Variants

Trained on a MetaMathQA subset, tested on GSM8K, all on Qwen/Qwen3.5-0.8B-Base targeting down_proj in all 24 layers (2500 steps, effective batch 8 = 20k samples). Standard adapters use r=32; the AntiPaSTO family uses r=256 (it tunes only S-space gain, so it needs the rank).

Variant	test %	valid %	Params	+MACs/tok	fwd/bwd (ms)	init (s)
DoRA	60.2	68.0	3.56M	3.54M	161 / 556	0.16
LoRA	59.8	68.0	3.54M	3.54M	173 / 573	0.02
PiSSA	59.8	76.0	3.54M	3.54M	146 / 549	2.04
HRA	59.2	70.0	2.75M	2.75M	225 / 948	0.04
EVA	59.3	74.0	3.54M	3.54M	151 / 660	28.3
IA3-FF	56.3	62.0	0.086M	0M	140 / 510	0.01
DeLoRA	56.2	62.0	3.54M	3.54M	169 / 593	0.21
AntiPaSTO	56.0	62.0	0.0061M	28.3M	166 / 571	2.5
AntiPaSTO-rot	57.2	60.0	0.0154M	28.3M	165 / 596	2.0
AntiPaSTO-ablate	56.0	68.0	0.0062M	28.3M	166 / 580	2.2
AntiPaSTO-dplr	56.0	64.0	0.1044M	28.4M	153 / 582	3.6
AntiPaSTO-ASVD (diag C)	55.6	64.0	0.0061M	28.3M	150 / 533	34
AntiPaSTO-CorDA (full C)	54.7	58.0	0.0061M	28.3M	146 / 576	120
IA3	52.3	62.0	0.0061M	0M	161 / 515	0.01

test/valid % = GSM8K exact-match accuracy. Params = trainable adapter params. +MACs/tok = added forward MACs per token (analytic, hardware-independent). fwd/bwd = median ms over one batch. init = one-time calibration (CorDA's d_in x d_in covariance eigh; ~0 for the rest). Peak CUDA memory is ~9.8 GB for every row. Empty rows fill in as the sweep lands.

We validate our adapters the same way PEFT does: train on a MetaMathQA subset and check meaningful GSM8K accuracy. See this file for details.

AntiPaSTO is the novel row here: instead of adding trainable directions like LoRA, it freezes W's own top-r SVD and learns only a bounded per-direction gain S_eff = S * (1 + ELU(g)). The singular basis stays fixed and interpretable, and the adapter is O(r) params (the 6.1K gain is ~580x smaller than LoRA's 3.54M). The variants change only the basis or core: rot learns a small block-rotation of the frozen basis, CorDA/ASVD orient it by the input second moment (full covariance vs diagonal-only, Yang+ 2024 / Yuan+ 2023), ablate learns a contractive directional ablation (Arditi+ 2024), dplr adds a small low-rank core for cross-direction mixing.

CorDA (full C) and ASVD (diag C) are a metric-axis ablation against plain AntiPaSTO (C=I): does covariance orientation earn its d_in x d_in eigh over the cheap diagonal or no calibration at all? On GSM8K/down_proj the answer is no: C=I 56.0, diag C 55.6, full C 54.7 (single seed). The off-diagonal orientation is the slowest arm (120 s init vs 2.5 s) and lands slightly below no calibration, so plain top-r SVD is the right default for this bounded-gain adapter here.

AntiPaSTO-rot tunes that basis instead of the metric: a block-diagonal Cayley rotation of the input (V, the table row), output (U), or both. Across two seeds the basis choice is within noise: seed0 ranks V 57.2 > U 56.5 > both 55.6, but seed1 flips it to U 57.5 > both 56.9 > V 56.2, and the 2-seed test means (U 57.0, V 56.7, both 56.3) span 0.7pp -- inside the ~1pp standard error of a 2-seed mean at n=1319. So no rotation basis is reliably best here; the single-seed V>U>both ordering was seed variance. rot(V) is the default as a fine, cheapest representative (15K params, ~230x under LoRA's 3.54M), not a measured winner.

Developer docs

See docs/developer_guide.md for the variant API, data-calibrated init, and save/load format.

Citation

@misc{wassname2026loralite,
  title = {LoRA-Lite: A Hackable Adapter Library for Research},
  author = {Michael J. Clark},
  year = {2026},
  url = {https://github.com/wassname/lora-lite/}
}