mirror of https://github.com/wassname/lora-lite.git synced 2026-06-27 16:45:56 +08:00

Files

T

wassname 6ab1dfff0e README: antipasto variants as table rows; real PEFT reference

- Fold the family into the main Variants table as rows (CorDA/ablate/arrow)
  instead of a separate table.
- Lead with the point (freeze W's SVD, learn only a bounded gain -> interpretable,
  O(r) params) before any numbers.
- Replace the unsourced 'PEFT reports 49.0%' line (wrong; LoRA is ~48%) with a
  real link to PEFT's method_comparison/MetaMathQA and a pointer to the benchmark
  script for hyperparameters. Link CorDA/Arditi papers inline.

Co-Authored-By: Claudypoo <noreply@anthropic.com>

2026-06-15 18:18:09 +08:00

3.9 KiB

Raw Blame History

lora-lite

Hackable PyTorch adapters for LoRA-family and small PEFT experiments.

Hackable code

To keep it simple and hackable we make these choices:

Simple forward hooks, no module replacement or custom modules.
Simple code over fast performance
No merge/unmerge
Single test where we train on MetaMathQA and test on GSM8K for each variant

Take a look at lora.py

Install

pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite

Quickstart

import torch, lora_lite as ll

model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)

opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...

ll.save(model, "adapter.safetensors")
ll.detach(model)
ll.load(model, "adapter.safetensors")

Does it work?

just check       # pytest + smoke + package build + metadata check
just bnb-smoke   # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe  # Qwen/Qwen3-0.6B train/save-load probe

Variants

Variant	4bit/8bit	GSM8K %	Params	Peak GPU (GB)
LoRA	yes	63.2%	4.59M	11.3
PiSSA	no	63.2%	4.59M	11.3
DoRA	no	62.4%	4.67M	11.3
DeLoRA	yes	61.5%	4.59M	11.3
AntiPaSTO	no	61.4%	14.3K	11.3
AntiPaSTO-CorDA	no	61.9%	14.3K	11.3
AntiPaSTO-ablate	no	61.0%	14.4K	11.3
AntiPaSTO-arrow	no	60.5%	17.5K	11.3
IA3-FF	yes	61.4%	86K	11.4
EVA	no	60.3%	4.59M	11.3
IA3	yes	60.0%	57K	11.4
HRA	yes	61.6%	1.84M	11.3

Params = trainable adapter params. Peak GPU = peak CUDA memory during train+eval (logged from this run onward; older runs predate the column).

Setup follows PEFT's method comparison: train on a MetaMathQA subset, test on GSM8K. We swap their Llama-3.2-3B (where LoRA gets ~48%) for the smaller Qwen3-0.6B-Base, so these track method rank, not cross-setup absolutes. Hyperparameters are in the benchmark script (r=32 q/v; the AntiPaSTO family uses r=256).

AntiPaSTO is the novel row here: instead of adding trainable directions like LoRA, it freezes W's own top-r SVD and learns only a bounded per-direction gain S_eff = S * (1 + ELU(g)). The singular basis stays fixed and interpretable, and the adapter is O(r) params (~320x smaller than LoRA). The variants change only the basis or core: CorDA orients it by input covariance (Yang+ 2024), ablate learns a contractive directional ablation (Arditi+ 2024), arrow adds a small dense block for cross-direction mixing.

Developer docs

See docs/developer_guide.md for the variant API, data-calibrated init, and save/load format.

Citation

@misc{wassname2026loralite,
  title = {LoRA-Lite: A Hackable Adapter Library for Research},
  author = {Michael J. Clark},
  year = {2026},
  url = {https://github.com/wassname/lora-lite/}
}

3.9 KiB Raw Blame History