lora-lite/README.md

lora-lite

Hackable PyTorch adapters for LoRA-family and small PEFT experiments.

Hackable code

To keep it simple and hackable we make these choices:

• Simple forward hooks, no module replacement or custom modules.
• Simple code over fast performance
• No merge/unmerge
• Single test where we train on MetaMathQA and test on GSM8K for each variant

Take a look at lora.py

Install

pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite

Quickstart

import torch, lora_lite as ll

model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)

opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...

ll.save(model, "adapter.safetensors")
ll.detach(model)
ll.load(model, "adapter.safetensors")

Does it work?

just check       # pytest + smoke + package build + metadata check
just bnb-smoke   # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe  # Qwen/Qwen3-0.6B train/save-load probe

Variants

Variant	4bit/8bit	GSM8K %
LoRA	yes	63.2%
PiSSA	no	—
DeLoRA	yes	—
IA3	yes	—
DoRA	no	—
HRA	yes	—
EVA	no	—
AntiPaSTO	no	—

Our test setup: We take Qwen3-0.6B-Base and train one MetaMathQA for 5000 steps. We use a rank of 32, and itnervene on all linear layer then test on GSM8K.

Is this a good accuracy? TODO we need a like-for-like comparison against PEFT LoRA in the same setup before drawing conclusions. But the PEFT library reports LoRA at 49.0% on Llama-3.2-3B (different model and sample count).

Developer docs

See docs/developer_guide.md for the variant API, data-calibrated init, and save/load format.

Citation

@misc{wassname2026loralite,
  title = {LoRA-Lite: A Hackable Adapter Library for Research},
  author = {Michael J. Clark},
  year = {2026},
  url = {https://github.com/wassname/lora-lite/}
}