mirror of
https://github.com/wassname/lora-lite.git
synced 2026-09-09 11:26:14 +08:00
Verify all variants on bnb 4bit/8bit; HRA paper-faithful rewrite
- Test all 6 variants against bnb.Linear8bitLt + Linear4bit in smoke - bnb-friendly (LoRA, IA3, HRA, DeLoRA): identity err <= 2.4e-4 - bnb-incompatible (PiSSA, DoRA): fail-loud TypeError as expected - HRA: rewrite to paper-faithful input-side reflections (h <- (I-2vv^T)h), fixing previous broken output-side formulation - IA3: bypass dtype upcast for bnb (params stay fp16/quantized) - DeLoRA: explicit type check rejecting non-nn.Linear (incl. bnb) - adapter: special-case bnb param assignment via .data - Re-verified Qwen0.6B HRA probe: drop=20.7%, id_err=0, reload=0
This commit is contained in:
@@ -1,4 +1,9 @@
|
||||
"""Vanilla LoRA. Reference variant. y = Wx + (alpha/r) * B @ A @ x."""
|
||||
"""Vanilla LoRA. Hu et al. 2021 https://arxiv.org/abs/2106.09685
|
||||
|
||||
h = W x + (alpha/r) B A x
|
||||
|
||||
Identity at t=0 from B=0. Faithful to the paper.
|
||||
"""
|
||||
from einops import einsum
|
||||
from torch import nn
|
||||
import torch
|
||||
|
||||
Reference in New Issue
Block a user