mirror of
https://github.com/wassname/lora-lite.git
synced 2026-06-27 19:31:26 +08:00
wassname
91 lines
2.9 KiB
Markdown
91 lines
2.9 KiB
Markdown
# lora-lite
|
|
|
|
Hackable PyTorch adapters for LoRA-family and small PEFT experiments.
|
|
|
|
`lora-lite` uses forward hooks instead of module replacement. Adapter parameters are plain `nn.Parameter`s on the target layer, e.g. `model.layers[5].self_attn.q_proj.lora_A`.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite
|
|
```
|
|
|
|
## Quickstart
|
|
|
|
```python
|
|
import torch, lora_lite as ll
|
|
|
|
model = MyTransformer()
|
|
cfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)
|
|
ll.attach(model, cfg)
|
|
|
|
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
|
|
# train...
|
|
|
|
ll.save(model, "adapter.pt")
|
|
ll.detach(model)
|
|
ll.load(model, "adapter.pt")
|
|
```
|
|
|
|
## Does it work?
|
|
|
|
```bash
|
|
just check # pytest + smoke + package build + metadata check
|
|
just bnb-smoke # required CUDA bitsandbytes 4bit/8bit smoke
|
|
just qwen-probe # Qwen/Qwen3-0.6B train/save-load probe
|
|
```
|
|
|
|
See [docs/spec/20260426_lora_lite_plan.md](docs/spec/20260426_lora_lite_plan.md) for verification history and exact results.
|
|
|
|
## Variants
|
|
|
|
| Variant | Support | Notes |
|
|
|---|---|---|
|
|
| LoRA | yes | additive low-rank adapter |
|
|
| PiSSA | yes, fp only | mutates `weight` into `W_res`; quantized PiSSA intentionally fails |
|
|
| DeLoRA | yes | normalized additive adapter with learned scalar |
|
|
| IA3 | yes | output gate initialized to ones |
|
|
| DoRA | yes, fp only | reads dense `weight` for column-norm; quantized DoRA fails loudly |
|
|
| HRA | yes | output-side Householder reflection with identity gate; works on bnb |
|
|
| SSVD / OFT / ROAD | no | planned |
|
|
| S-steer / AntiPaSTO | no | should use data-calibrated `group_init`, not plain LoRA tests |
|
|
|
|
## Targeting
|
|
|
|
By default, `lora-lite` targets linear-like modules with `in_features`, `out_features`, and `weight`, excluding `lm_head` and `embed_tokens`.
|
|
|
|
Useful `AdapterConfig` fields (shared across all variants; subclasses add
|
|
variant-specific knobs like `lambda0` on `DeLoRAConfig`):
|
|
|
|
- `target_roles`: subset of `("reader", "writer", "inner")`; `()` means all.
|
|
- `target_names`: regex includes.
|
|
- `exclude_names`: regex excludes.
|
|
- `layers`: layer indices, matching `.layers.<idx>.` in module names.
|
|
|
|
This structural targeting is why LoRA, DeLoRA, and IA3 can run on bnb-style `Linear4bit`/`Linear8bitLt` modules. PiSSA is different because it edits the base weight.
|
|
|
|
## Save format
|
|
|
|
Adapters are just:
|
|
|
|
```python
|
|
torch.save({"cfg": cfg.to_dict(), "state": lora_state_dict}, "adapter.pt")
|
|
```
|
|
|
|
`lora_state_dict` contains full-path keys with `"lora_"` in the name. Missing or unexpected adapter keys fail on load.
|
|
|
|
## Developer docs
|
|
|
|
See [docs/developer_guide.md](docs/developer_guide.md) for the variant API, data-calibrated init, and adapter roadmap.
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@misc{wassname2026loralite,
|
|
title = {LoRA-Lite: A Hackable Adapter Library for Research},
|
|
author = {Michael J. Clark},
|
|
year = {2026},
|
|
url = {https://github.com/wassname/lora-lite/}
|
|
}
|
|
```
|