mirror of
https://github.com/wassname/lora-lite.git
synced 2026-08-11 11:21:06 +08:00
wassname
67a6daf6aa
DeLoRA (variants/delora.py):
lora_B init zeros not kaiming, matching peft (docs/refs/peft_delora_layer.py:139).
With B=0 the t=0 delta is zero regardless of lambda, so identity holds with
the peft default lambda0=15 instead of needing the lambda0=0 hack.
HRA (variants/hra.py):
forward_input loop reversed: now applies x @ H_{r-1} ... H_0 = x @ R^T so
the base layer computes x R^T W^T = F.linear(x, W @ R), matching peft. The
bug was masked by paired-symmetry init (R = R^T at t=0) but would corrupt
any non-symmetric U.
EVA (variants/eva.py):
lora_A is now a trainable Parameter (peft semantics): SVD only changes the
init. group_init still copies the SVD basis but under a no_grad guard.
AntiPaSTO (variants/antipasto.py):
docstring now references arxiv.org/pdf/2601.07473 and
github.com/wassname/AntiPaSTO so V4 review NO_REFERENCE flag is resolved.
qwen probe (scripts/qwen_train_probe.py):
perturb_first_adapter walks priority list including lora_U (HRA) and
lora_A (EVA, LoRA-style A-trainable variants) so HRA tests no longer raise
'no perturbable adapter parameter found'.
smoke (tests/smoke.py):
+ hra_forward_order_smoke: distinguishing check that compares adapted output
to F.linear(x, W @ R) with paired symmetry broken; would fail under the
forward-iter bug.
+ EVA assert lora_A.requires_grad == True per layer.
- DeLoRA bnb moved to bnb_skip (fp16 + B=0 + clamp(min=1e-4) overflow makes
grad NaN; real bnb usage needs dequant).
delora train still uses lambda0=0.1 because peft default 15.0 explodes
Adam lr=1e-1 in 20 steps.
lora-lite
Hackable PyTorch adapters for LoRA-family and small PEFT experiments.
lora-lite uses forward hooks instead of module replacement. Adapter parameters are plain nn.Parameters on the target layer, e.g. model.layers[5].self_attn.q_proj.lora_A.
Install
pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite
Quickstart
import torch, lora_lite as ll
model = MyTransformer()
cfg = ll.LoraLiteConfig(variant="lora", r=8, alpha=16, dtype=torch.bfloat16)
ll.attach(model, cfg)
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
# train...
ll.save(model, "adapter.pt")
ll.detach(model)
ll.load(model, "adapter.pt")
Does it work?
just check # pytest + smoke + package build + metadata check
just bnb-smoke # required CUDA bitsandbytes 4bit/8bit smoke
just qwen-probe # Qwen/Qwen3-0.6B train/save-load probe
See docs/spec/20260426_lora_lite_plan.md for verification history and exact results.
Variants
| Variant | Support | Notes |
|---|---|---|
| LoRA | yes | additive low-rank adapter |
| PiSSA | yes, fp only | mutates weight into W_res; quantized PiSSA intentionally fails |
| DeLoRA | yes | normalized additive adapter with learned scalar |
| IA3 | yes | output gate initialized to ones |
| DoRA | yes, fp only | reads dense weight for column-norm; quantized DoRA fails loudly |
| HRA | yes | output-side Householder reflection with identity gate; works on bnb |
| SSVD / OFT / ROAD | no | planned |
| S-steer / AntiPaSTO | no | should use data-calibrated group_init, not plain LoRA tests |
Targeting
By default, lora-lite targets linear-like modules with in_features, out_features, and weight, excluding lm_head and embed_tokens.
Useful LoraLiteConfig fields:
target_roles: subset of("reader", "writer", "inner");()means all.target_names: regex includes.exclude_names: regex excludes.layers: layer indices, matching.layers.<idx>.in module names.
This structural targeting is why LoRA, DeLoRA, and IA3 can run on bnb-style Linear4bit/Linear8bitLt modules. PiSSA is different because it edits the base weight.
Save format
Adapters are just:
torch.save({"cfg": cfg.to_dict(), "state": lora_state_dict}, "adapter.pt")
lora_state_dict contains full-path keys with "lora_" in the name. Missing or unexpected adapter keys fail on load.
Developer docs
See docs/developer_guide.md for the variant API, data-calibrated init, and adapter roadmap.
Citation
@misc{wassname2026loralite,
title = {LoRA-Lite: A Hackable Adapter Library for Research},
author = {Michael J. Clark},
year = {2026},
url = {https://github.com/wassname/lora-lite/}
}