lora-lite uses forward hooks instead of module replacement. Adapter parameters are plain nn.Parameters on the target layer, e.g. model.layers[5].self_attn.q_proj.lora_A.

Install

pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite

Quickstart

import torch, lora_lite as ll

model = MyTransformer()
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)

opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...

ll.save(model, "adapter.pt")
ll.detach(model)
ll.load(model, "adapter.pt")

Does it work?

just check       # pytest + smoke + package build + metadata check
just bnb-smoke   # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe  # Qwen/Qwen3-0.6B train/save-load probe

See docs/spec/20260426_lora_lite_plan.md for verification history and exact results.

Variants

Variant	Support	Notes
LoRA	yes	additive low-rank adapter
PiSSA	yes, fp only	mutates `weight` into `W_res`; quantized PiSSA intentionally fails
DeLoRA	yes	normalized additive adapter with learned scalar
IA3	yes	output gate (`ia3`) or input gate (`ia3_ff`); init to ones
DoRA	yes, fp only	reads dense `weight` for column-norm; quantized DoRA fails loudly
HRA	yes	input-side Householder product via pre-hook; works on bnb
EVA	yes, fp only	LoRA forward; `lora_A` init from PCA on calibration activations
AntiPaSTO	yes, fp only	top-r weight SVD with learnable singular-value deltas + Cayley rotation
SSVD / OFT / ROAD	no	planned

Targeting

By default, lora-lite targets linear-like modules with in_features, out_features, and weight, excluding lm_head and embed_tokens.

Useful AdapterConfig fields (shared across all variants; subclasses add variant-specific knobs like lambda0 on DeLoRAConfig):

target_roles: subset of ("reader", "writer", "inner"); () means all.
target_names: regex includes.
exclude_names: regex excludes.
layers: layer indices, matching .layers.<idx>. in module names.

This structural targeting is why LoRA, DeLoRA, and IA3 can run on bnb-style Linear4bit/Linear8bitLt modules. PiSSA is different because it edits the base weight.

Save format

Adapters are just:

torch.save({"cfg": cfg.to_dict(), "state": lora_state_dict}, "adapter.pt")

lora_state_dict contains full-path keys with "lora_" in the name. Missing or unexpected adapter keys fail on load.

Developer docs

See docs/developer_guide.md for the variant API, data-calibrated init, and adapter roadmap.

Citation

@misc{wassname2026loralite,
  title = {LoRA-Lite: A Hackable Adapter Library for Research},
  author = {Michael J. Clark},
  year = {2026},
  url = {https://github.com/wassname/lora-lite/}
}