diff --git a/docs/audit/REVIEW_PROMPT_VARIANTS_V2.md b/docs/audit/REVIEW_PROMPT_VARIANTS_V2.md
new file mode 100644
index 0000000..b231a8b
--- /dev/null
+++ b/docs/audit/REVIEW_PROMPT_VARIANTS_V2.md
@@ -0,0 +1,74 @@
+# Per-variant paper-faithfulness audit V2 (with reference implementations)
+
+Re-audit of `lora-lite` after adding canonical reference implementation URLs to
+each variant docstring. Your job: for each variant, **directly compare** our
+implementation against the reference impl (peft and/or paper-author repo), not
+just against the paper text. This is round 2 — the previous review (you can
+read `docs/audit/variants_review.md`) found:
+
+- HRA gate=0 init kills `lora_U` gradient on step 0
+- DeLoRA same pattern with lambda0=0
+- IA3 targets q/v not paper k/v/ffn-down (deviation documented but untested)
+- PiSSA bf16 init err 0.31 on Qwen
+- Saved adapters don't preserve PiSSA W_res mutation
+
+Your job now is to verify those findings against the **reference code**, and
+look for anything the prior review missed once you have the reference in hand.
+
+## Inputs
+
+- Our code: `src/lora_lite/variants/{lora,pissa,dora,ia3,hra,delora}.py`
+- Adapter plumbing: `src/lora_lite/{adapter.py,target.py,variant.py,config.py}`
+- Papers (text): `docs/papers/*_*.txt`
+- **Reference implementations** (just added):
+  - `docs/refs/peft_lora_layer.py`   — peft LoRA Linear (and PiSSA init paths)
+  - `docs/refs/peft_lora_dora.py`    — peft DoRA helper module
+  - `docs/refs/peft_lora_variants.py` — peft per-variant init dispatch (PiSSA, OLoRA, etc.)
+  - `docs/refs/peft_ia3_layer.py`    — peft IA3 layer
+  - `docs/refs/peft_hra_layer.py`    — peft HRA layer (clean, has apply_GS toggle)
+  - `docs/refs/peft_delora_layer.py` — peft DeLoRA layer (upstreamed)
+  - `docs/refs/orig_pissa_init.py`   — PiSSA paper authors' init script (MuLabPKU)
+  - `docs/refs/orig_hra_layer.py`    — HRA paper authors' OFT-with-HRA layer (DaShenZi721)
+  - `docs/refs/orig_delora.py`       — DeLoRA paper authors' fork-of-peft impl (ExplainableML)
+- Logs: `logs/smoke.log`, `logs/qwen_probe.log`
+- Prior review: `docs/audit/variants_review.md` (do NOT just restate it)
+
+## What to deliver per variant (LoRA, PiSSA, DoRA, IA3, HRA, DeLoRA)
+
+1. **Reference impl ground-truth** — what does the *reference* code actually do
+   for: parameter shapes, initialization, scale factor, forward equation,
+   save/load, target placement? Quote ≤10 lines with file/line cites from
+   `docs/refs/`.
+
+2. **Our code** — quote our impl (≤10 lines, with `src/lora_lite/variants/<v>.py:LN` cites).
+
+3. **Diff** — bullet list of every meaningful difference.
+   Mark each one as: `[OK-doc]` (acceptable, documented), `[OK-undoc]` (acceptable,
+   should add to docstring), `[BUG]` (likely wrong), `[STYLE]` (cosmetic).
+
+4. **Did the prior review get it right?** Quote the relevant prior verdict
+   line and either confirm or correct.
+
+5. **Verdict** — Faithful / Faithful-with-doc-gap / Partial / Buggy.
+   One-line reason.
+
+## Final aggregate
+
+Markdown table:
+
+| variant | prior verdict | new verdict | new bugs found | doc gaps |
+
+And a 5-bullet "what to fix next" list, ordered by severity.
+
+## Hard rules
+
+- Quote evidence from `docs/refs/` files. If you can't find the relevant
+  reference function, say so explicitly — don't guess.
+- Do NOT edit code. Output review only.
+- Be specific about line numbers from the references. "peft does X" is not
+  enough; "peft_lora_layer.py:L1234 does X" is.
+- If you find a NEW bug not flagged in `variants_review.md`, mark it
+  `[NEW-BUG]` and explain the failure mode.
+- If the prior review was wrong (false positive), mark it `[OVERTURN]`.
+
+Write to stdout. I will redirect to `docs/audit/variants_review_v2.md`.
diff --git a/docs/audit/variants_review_v2.md b/docs/audit/variants_review_v2.md
new file mode 100644
index 0000000..d9dfc6e
--- /dev/null
+++ b/docs/audit/variants_review_v2.md
@@ -0,0 +1,1278 @@
+[client] initialize (running)
+
+[client] session/new (running)
+I’ll use the `gsd` workflow here since this is a multi-step audit, and I’ll keep it review-only: read refs, compare implementation, then produce the markdown report to stdout.
+
+[tool] Read SKILL.md (running)
+  input: {"call_id":"call_4LnMf0WHkHoYdG0a5N21Rrgd","process_id":"77034","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /home/wassname/.claude/skills/gsd/SKILL.md"],"cwd":"/me...
+  files: /home/wassname/.claude/skills/gsd/SKILL.md
+
+[tool] Read SKILL.md (completed)
+  kind: read
+  input: {"call_id":"call_4LnMf0WHkHoYdG0a5N21Rrgd","process_id":"77034","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /home/wassname/.claude/skills/gsd/SKILL.md"],"cwd":"/me...
+  files: /home/wassname/.claude/skills/gsd/SKILL.md
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+    ---
+    name: gsd
+    description: "Get Shit Done workflow for non-trivial tasks: spec -> implement -> test -> review -> wrap. Mandatory for anything with more than one step."
+    ---
+    
+    <!--
+    Design rationale and sources — for human readers, not the agent.
+    
+    - GSD spec-first: spec-driven development, scope fencing, verify-before-done
+    - Manus file-as-memory: one file is working memory, checklist is state, append-only log
+    - Boomerang: return concrete artifacts not summaries
+    - DCP: invisible dedup, superseded writes, error purging
+    
+    Core principles:
+    - Spec file is the only persistent artifact (single-file invariant).
+    - Agent shouldn't know about its own resource limits.
+    - Review scales with input size: frontier for plans, cheap for bulk.
+    - Observable outputs: show evidence, not narration.
+    -->
+    
+    # GSD (Get Shit Done)
+    
+    Mandatory for all non-trivial tasks (more than one step). Skip only for single-line fixes.
+    
+    ## Part 1: Core workflow
+    
+    ### 1.1 Spec-first planning
+    ... (192 more lines)
+I’m going to gather the exact line-numbered evidence first: prior verdicts, our variant code, plumbing, and the reference implementations. The main thing to avoid is smearing paper-faithfulness and reference-faithfulness together.
+
+[tool] Read variants_review.md (running)
+  input: {"call_id":"call_ZMecpefRlTCdKyqbYXoIQ4dq","process_id":"78042","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/audit/variants_review.md | sed -n '1,260p'"],"cwd":"/media/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/audit/variants_review.md
+
+[tool] Run for f in src/lora_lite/{adapter.py,target.py,variant.py,config.py}; do printf '\n### %s\n' "$f"; nl -ba "$f" | sed -n '1,280p'; done (running)
+  input: {"call_id":"call_ToqQP8XSZ8r22VF9IYDGRrHF","process_id":"51617","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","for f in src/lora_lite/{adapter.py,target.py,variant.py,config.py}; do ...
+
+[tool] Run for f in src/lora_lite/variants/{lora,pissa,dora,ia3,hra,delora}.py; do printf '\n### %s\n' "$f"; nl -ba "$f" | sed -n '1,260p'; done (running)
+  input: {"call_id":"call_i9nfOVjK5lGEtnZXiX8w3UVI","process_id":"66364","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","for f in src/lora_lite/variants/{lora,pissa,dora,ia3,hra,delora}.py; do...
+
+[tool] Read variants_review.md (completed)
+  kind: read
+  input: {"call_id":"call_ZMecpefRlTCdKyqbYXoIQ4dq","process_id":"78042","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/audit/variants_review.md | sed -n '1,260p'"],"cwd":"/media/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/audit/variants_review.md
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+         1	[client] initialize (running)
+         2	
+         3	[client] session/new (running)
+         4	I’ll use the `gsd` skill because this is a multi-step audit, then I’ll read the variant code, paper excerpts, logs, and refs only as needed. No code edits, just a verdict with quoted evidence.
+         5	
+         6	[tool] Read SKILL.md (running)
+         7	  input: {"call_id":"call_ZJ8jUrIxVXwLtDtWUHoN5Nyw","process_id":"8344","turn_id":"019dc96e-e7cd-79f2-b04d-65a1c4d26637","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /home/wassname/.claude/skills/gsd/SKILL.md"],"cwd":"/med...
+         8	  files: /home/wassname/.claude/skills/gsd/SKILL.md
+         9	
+        10	[tool] Read SKILL.md (completed)
+        11	  kind: read
+        12	  input: {"call_id":"call_ZJ8jUrIxVXwLtDtWUHoN5Nyw","process_id":"8344","turn_id":"019dc96e-e7cd-79f2-b04d-65a1c4d26637","command":["/usr/bin/zsh","-lc","sed -n '1,220p' /home/wassname/.claude/skills/gsd/SKILL.md"],"cwd":"/med...
+        13	  files: /home/wassname/.claude/skills/gsd/SKILL.md
+        14	  output:
+        15	    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        16	    ---
+        17	    name: gsd
+        18	    description: "Get Shit Done workflow for non-trivial tasks: spec -> implement -> test -> review -> wrap. Mandatory for anything with more than one step."
+        19	    ---
+        20	    
+        21	    <!--
+        22	    Design rationale and sources — for human readers, not the agent.
+        23	    
+        24	    - GSD spec-first: spec-driven development, scope fencing, verify-before-done
+        25	    - Manus file-as-memory: one file is working memory, checklist is state, append-only log
+        26	    - Boomerang: return concrete artifacts not summaries
+        27	    - DCP: invisible dedup, superseded writes, error purging
+    ... (233 more lines)
+
+[tool] Run for f in src/lora_lite/{adapter.py,target.py,variant.py,config.py}; do printf '\n### %s\n' "$f"; nl -ba "$f" | sed -n '1,280p'; done (completed)
+  kind: execute
+  input: {"call_id":"call_ToqQP8XSZ8r22VF9IYDGRrHF","process_id":"51617","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","for f in src/lora_lite/{adapter.py,target.py,variant.py,config.py}; do ...
+  output:
+    ### src/lora_lite/adapter.py
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+         1	"""attach / detach / save / load. The whole runtime."""
+         2	from __future__ import annotations
+         3	import torch
+         4	from torch import nn
+         5	from torch.utils.hooks import RemovableHandle
+         6	
+         7	from .config import LoraLiteConfig
+         8	from .variant import REGISTRY
+         9	from .target import find_targets
+        10	
+        11	
+        12	_ATTACHED_ATTR = "_lora_lite_attached"
+        13	
+        14	
+        15	def _hook(layer, args, y):
+        16	    (x,) = args
+        17	    cfg: LoraLiteConfig = layer._lora_cfg
+        18	    x_cast = x.to(cfg.dtype)
+        19	    out = layer._lora_variant.forward(layer, x_cast, y)
+        20	    return out.to(y.dtype)
+        21	
+        22	
+        23	def _pre_hook(layer, args):
+        24	    (x,) = args
+        25	    cfg: LoraLiteConfig = layer._lora_cfg
+        26	    x_cast = x.to(cfg.dtype)
+    ... (250 more lines)
+
+[tool] Run for f in src/lora_lite/variants/{lora,pissa,dora,ia3,hra,delora}.py; do printf '\n### %s\n' "$f"; nl -ba "$f" | sed -n '1,260p'; done (completed)
+  kind: execute
+  input: {"call_id":"call_i9nfOVjK5lGEtnZXiX8w3UVI","process_id":"66364","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","for f in src/lora_lite/variants/{lora,pissa,dora,ia3,hra,delora}.py; do...
+  output:
+    ### src/lora_lite/variants/lora.py
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+         1	"""Vanilla LoRA. Hu et al. 2021  https://arxiv.org/abs/2106.09685
+         2	
+         3	    h = W x + (alpha/r) B A x
+         4	
+         5	Identity at t=0 from B=0. Faithful to the paper.
+         6	
+         7	Reference implementations (for review/cross-check):
+         8	  - peft Linear.update_layer + lora_A/B init, forward:
+         9	    https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py
+        10	    (see docs/refs/peft_lora_layer.py for offline copy)
+        11	"""
+        12	from einops import einsum
+        13	from torch import nn
+        14	import torch
+        15	
+        16	from ..variant import register, ParamSpec
+        17	
+        18	
+        19	@register
+        20	class LoRA:
+        21	    name = "lora"
+        22	
+        23	    @staticmethod
+        24	    def param_specs(d_in, d_out, cfg):
+        25	        return {
+        26	            "lora_A": ParamSpec((cfg.r, d_in), init="kaiming", trainable=True),
+    ... (355 more lines)
+The prior review file appears to include a captured transcript at the top, so I’ll isolate the actual verdict lines with search rather than assuming the whole file is clean markdown. Next I’m pulling focused reference snippets around the implementations.
+
+[tool] Search Verdict|BUG|HRA|DeLoRA|IA3|PiSSA|LoRA|DoRA|prior verdict|new verdict|Faithful|Partial|Buggy in variants_review.md (running)
+  input: {"call_id":"call_e6cm2JLiyHQ9BWA2TVDwczBs","process_id":"84327","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"Verdict|BUG|HRA|DeLoRA|IA3|PiSSA|LoRA|DoRA|prior verdict|new ve...
+
+[tool] Search class Linear|def update_layer|lora_A|lora_B|scaling|reset_lora_parameters|pissa_init|subtract_mutated_init|forward\(|magnitude|dora|use_dora|save|load in peft_lora_layer.py (running)
+  input: {"call_id":"call_M0zQbm2NDBihQ79YxcPRZgiK","process_id":"9967","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|def update_layer|lora_A|lora_B|scaling|reset_lora_p...
+
+[tool] Search class Linear|ia3_l|feedforward|is_feedforward|forward\(|update_layer|reset_ia3_parameters|target_modules|save|load|Parameter|ones in peft_ia3_layer.py (running)
+  input: {"call_id":"call_mXxKBEh7P0d4sdJjx6BqGFXB","process_id":"17225","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|ia3_l|feedforward|is_feedforward|forward\\(|update...
+
+[tool] Search class Linear|delora|lambda|lora_A|lora_B|forward|reset|norm|frozen|init|Parameter|scaling|weight_norm|save|load|dora in peft_delora_layer.py (running)
+  input: {"call_id":"call_xWW1G18NS65tbdAzP1zglU4E","process_id":"4667","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|delora|lambda|lora_A|lora_B|forward|reset|norm|froz...
+
+[tool] Search class Linear|hra|HRA|apply_GS|hra_u|lora|forward|Householder|reflect|lambda|reset|Parameter|orthogonal|init|scaling in peft_hra_layer.py (running)
+  input: {"call_id":"call_LCfRNo5JdxbNAhjTLXS0Rrzx","process_id":"59701","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|hra|HRA|apply_GS|hra_u|lora|forward|Householder|re...
+
+[tool] Search Verdict|BUG|HRA|DeLoRA|IA3|PiSSA|LoRA|DoRA|prior verdict|new verdict|Faithful|Partial|Buggy in variants_review.md (completed)
+  kind: search
+  input: {"call_id":"call_e6cm2JLiyHQ9BWA2TVDwczBs","process_id":"84327","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"Verdict|BUG|HRA|DeLoRA|IA3|PiSSA|LoRA|DoRA|prior verdict|new ve...
+  output:
+    151:    1	"""Vanilla LoRA. Hu et al. 2021  https://arxiv.org/abs/2106.09685
+    155:         5	Identity at t=0 from B=0. Faithful to the paper.
+    165:        15	class LoRA:
+    186:    1	"""PiSSA: top-r SVD of W into A,B; replace W with W_res = W - B@A.
+    199:        14	class PiSSA:
+    213:        28	                "PiSSA mutates layer.weight into W_res, so v1 only supports plain nn.Linear. "
+    229:    1	"""IA3-style output gating. Liu et al. 2022  https://arxiv.org/abs/2205.05638
+    234:         6	    The original IA3 gates only three positions per transformer block:
+    246:        18	    (Llama uses gate * up; gating the up branch is the IA3-spirit choice).
+    255:        27	class IA3:
+    264:    1	"""DoRA: weight-decomposed LoRA. Liu et al. 2024  https://arxiv.org/abs/2402.09353
+    282:        19	class DoRA:
+    315:    1	"""DeLoRA: column-normalised A, B, scaled by lambda * ||W||_F / r.
+    340:        26	class DeLoRA:
+    385:    1	"""HRA: Householder Reflection Adaptation. Yuan et al. 2024  https://arxiv.org/abs/2405.17484
+    403:        19	you want the regularized HRA variant.
+    451:[tool] Search PiSSA|SVD|singular|residual|A =|B =|W_res|Eq|equation|principal|initialize|initial in pissa_*.txt (running)
+    452:  input: {"call_id":"call_E9ZDiQW4LYUvXfsbLERUR8VE","process_id":"40256","turn_id":"019dc96e-e7cd-79f2-b04d-65a1c4d26637","command":["/usr/bin/zsh","-lc","rg -n \"PiSSA|SVD|singular|residual|A =|B =|W_res|Eq|equation|principal...
+    454:[tool] Search IA3|l_k|l_v|l_ff|key|value|feed-forward|activation|element-wise|Eq|equation|rescale|learned vector in ia3_*.txt (running)
+    455:  input: {"call_id":"call_g8eWpsF9Wv2q0sr9uIOTKAXt","process_id":"6527","turn_id":"019dc96e-e7cd-79f2-b04d-65a1c4d26637","command":["/usr/bin/zsh","-lc","rg -n \"IA3|l_k|l_v|l_ff|key|value|feed-forward|activation|element-wise|...
+    457:[tool] Search LoRA|low-rank|A|B|alpha|rank|Eq|equation|ΔW|Delta|zero|Gaussian|Kaiming|initial in lora_*.txt (running)
+    458:  input: {"call_id":"call_bWnP0o9dE2sMNEN6N5a9w6Jp","process_id":"2585","turn_id":...
+
+[tool] Search class Linear|def update_layer|lora_A|lora_B|scaling|reset_lora_parameters|pissa_init|subtract_mutated_init|forward\(|magnitude|dora|use_dora|save|load in peft_lora_layer.py (completed)
+  kind: search
+  input: {"call_id":"call_M0zQbm2NDBihQ79YxcPRZgiK","process_id":"9967","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|def update_layer|lora_A|lora_B|scaling|reset_lora_p...
+  output:
+    docs/refs/peft_lora_variants.py:29:from .dora import DoraConv1dLayer, DoraConv2dLayer, DoraConv3dLayer, DoraEmbeddingLayer, DoraLinearLayer
+    docs/refs/peft_lora_variants.py:38:        for ArrowLoraLinearLayer(). A layer of the base model with LoRA adapter loaded on it will be like:
+    docs/refs/peft_lora_variants.py:41:                (base_layer): Linear4bit or Linear (lora_dropout): ModuleDict( ... ) (lora_A): ModuleDict( ... )
+    docs/refs/peft_lora_variants.py:42:                (lora_B): ModuleDict( ... ) (lora_embedding_A): ParameterDict( ... ) (lora_embedding_B): ParameterDict(
+    docs/refs/peft_lora_variants.py:43:                ... ) (lora_magnitude_vector): ModuleDict( ... ) (lora_arrow): ModuleDict(
+    docs/refs/peft_lora_variants.py:49:            module (Linear): LoRA Layer of the model, containing base_layer, lora_A, lora_B, etc.
+    docs/refs/peft_lora_variants.py:71:    def forward(
+    docs/refs/peft_lora_variants.py:98:        arrow.gen_know_sub(module.lora_A, module.lora_B)
+    docs/refs/peft_lora_variants.py:101:        arrow.build_prototypes(module.lora_A, module.lora_B)
+    docs/refs/peft_lora_variants.py:107:            lora_A=module.lora_A,
+    docs/refs/peft_lora_variants.py:108:            lora_B=module.lora_B,
+    docs/refs/peft_lora_variants.py:110:            scaling=module.scaling,
+    docs/refs/peft_lora_variants.py:136:        if not module.lora_magnitude_vector:
+    docs/refs/peft_lora_variants.py:137:            # first dora layer being added, add lora_magnitude_vector to the list of learnable parameters
+    docs/refs/peft_lora_variants.py:138:            module.adapter_layer_names = module.adapter_layer_names[:] + ("lora_magnitude_vector",)
+    docs/refs/peft_lora_variants.py:140:        dora_layer = DoraLinearLayer(fan_in_fan_out=getattr(module, "fan_in_fan_out", False))
+    docs/refs/peft_lora_variants.py:141:        lora_A = module.lora_A[adapter_name].weight
+    docs/refs/peft_lora_variants.py:142:        lora_B = module.lora_B[adapter_name].weight
+    docs/refs/peft_lora_variants.py:143...
+
+[tool] Search class Linear|ia3_l|feedforward|is_feedforward|forward\(|update_layer|reset_ia3_parameters|target_modules|save|load|Parameter|ones in peft_ia3_layer.py (completed)
+  kind: search
+  input: {"call_id":"call_mXxKBEh7P0d4sdJjx6BqGFXB","process_id":"17225","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|ia3_l|feedforward|is_feedforward|forward\\(|update...
+  output:
+    30:    adapter_layer_names = ("ia3_l",)
+    32:    def __init__(self, base_layer: nn.Module, is_feedforward: bool, **kwargs) -> None:
+    34:        self.ia3_l = nn.ParameterDict({})
+    38:        self.is_feedforward = is_feedforward
+    56:    def update_layer(self, adapter_name: str, config: IA3Config, **kwargs):
+    62:        if self.is_feedforward:
+    66:        self.ia3_l[adapter_name] = nn.Parameter(weight)
+    68:            self.reset_ia3_parameters(adapter_name)
+    72:    def reset_ia3_parameters(self, adapter_name):
+    73:        if adapter_name in self.ia3_l.keys():
+    74:            # initialize learned vector with torch.ones
+    75:            nn.init.constant_(self.ia3_l[adapter_name], 1.0)
+    78:class Linear(nn.Module, IA3Layer):
+    85:        is_feedforward: bool = False,  # Set to True if the layer is treated as a feedforward layer
+    86:        is_target_conv_1d_layer: bool = False,  # whether target module is a conv1d layer. useful while unloading later
+    90:        IA3Layer.__init__(self, base_layer, is_feedforward=is_feedforward)
+    94:        self.update_layer(adapter_name, config=config)
+    115:            if active_adapter in self.ia3_l.keys():
+    117:                ia3_l = transpose(self.ia3_l[active_adapter].data, self.fan_in_fan_out)
+    121:                    orig_weights = torch.mul(orig_weights, ia3_l)
+    129:                    base_layer.weight.data = torch.mul(base_layer.weight.data, ia3_l).to(orig_dtype)
+    131:                if not self.is_feedforward and (base_layer.bias is not None):
+    132:                    scaling = self.ia3_l[active_adapter].reshape(base_layer.bias.shape)
+    149:            if active_adapter in self.ia3_l.keys():
+    152:                ia3_l = transpose(self.ia3_l[active_adapter].data, self.fan_in_fan_out) + 1e-8
+    154:                base_layer.weight.data = torch.div(base_layer.weight.data, ia3_l).to(orig_dtype)
+    156:                if not self.is_feedforward and (base_layer.bias is not None):
+    157:                    scaling = self.ia3_l[active_adapter].reshape(base_layer.bias.s...
+
+[tool] Search class Linear|delora|lambda|lora_A|lora_B|forward|reset|norm|frozen|init|Parameter|scaling|weight_norm|save|load|dora in peft_delora_layer.py (completed)
+  kind: search
+  input: {"call_id":"call_xWW1G18NS65tbdAzP1zglU4E","process_id":"4667","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|delora|lambda|lora_A|lora_B|forward|reset|norm|froz...
+  output:
+    docs/refs/peft_delora_layer.py:32:        "delora_A",
+    docs/refs/peft_delora_layer.py:33:        "delora_B",
+    docs/refs/peft_delora_layer.py:34:        "delora_lambda",
+    docs/refs/peft_delora_layer.py:39:        "delora_dropout",
+    docs/refs/peft_delora_layer.py:40:        "delora_w_norm",
+    docs/refs/peft_delora_layer.py:43:    def __init__(self, base_layer: nn.Module, **kwargs) -> None:
+    docs/refs/peft_delora_layer.py:46:        self.delora_dropout = nn.ModuleDict({})
+    docs/refs/peft_delora_layer.py:47:        self.delora_A = nn.ParameterDict({})
+    docs/refs/peft_delora_layer.py:48:        self.delora_B = nn.ParameterDict({})
+    docs/refs/peft_delora_layer.py:49:        self.delora_lambda = nn.ParameterDict({})
+    docs/refs/peft_delora_layer.py:50:        # Use persistent buffers so they are included in state_dict and saved.
+    docs/refs/peft_delora_layer.py:51:        self.delora_w_norm = BufferDict({}, persistent=True)
+    docs/refs/peft_delora_layer.py:65:        A: torch.Tensor, B: torch.Tensor, delora_lambda: torch.Tensor, r: int, w_norm: torch.Tensor
+    docs/refs/peft_delora_layer.py:67:        """Compute delta = B @ diag(delora_lambda/r / (||A_i||*||B^j||)) @ A, scaled by provided w_norm (per-input channel)"""
+    docs/refs/peft_delora_layer.py:68:        An = torch.clamp(A.norm(dim=1), min=1e-4)
+    docs/refs/peft_delora_layer.py:69:        Bn = torch.clamp(B.norm(dim=0), min=1e-4)
+    docs/refs/peft_delora_layer.py:70:        diag = torch.diag_embed(delora_lambda / r / (An * Bn))
+    docs/refs/peft_delora_layer.py:72:        delta = delta * w_norm.unsqueeze(0)
+    docs/refs/peft_delora_layer.py:76:        if adapter not in self.delora_A or adapter not in self.delora_B:
+    docs/refs/peft_delora_layer.py:80:            self.delora_A[adapter],
+    docs/refs/peft_delora_layer.py:81:            self.delora_B[adapter],
+    docs/refs/peft_delora_layer.py:82:            self.delora_lambda[adapter],
+    docs/refs/peft_delora_layer.py:84:            self.delora_w_norm[adapter],
+    docs/refs/peft_delora_layer.py:92:        del...
+
+[tool] Search class Linear|hra|HRA|apply_GS|hra_u|lora|forward|Householder|reflect|lambda|reset|Parameter|orthogonal|init|scaling in peft_hra_layer.py (completed)
+  kind: search
+  input: {"call_id":"call_LCfRNo5JdxbNAhjTLXS0Rrzx","process_id":"59701","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"class Linear|hra|HRA|apply_GS|hra_u|lora|forward|Householder|re...
+  output:
+    docs/refs/peft_hra_layer.py:25:from .config import HRAConfig
+    docs/refs/peft_hra_layer.py:28:class HRALayer(BaseTunerLayer):
+    docs/refs/peft_hra_layer.py:30:    adapter_layer_names = ("hra_u",)
+    docs/refs/peft_hra_layer.py:32:    other_param_names = ("hra_r", "hra_apply_GS")
+    docs/refs/peft_hra_layer.py:34:    def __init__(self, base_layer: nn.Module, **kwargs) -> None:
+    docs/refs/peft_hra_layer.py:36:        self.hra_r = {}
+    docs/refs/peft_hra_layer.py:37:        self.hra_apply_GS = {}
+    docs/refs/peft_hra_layer.py:38:        self.hra_u = nn.ParameterDict({})
+    docs/refs/peft_hra_layer.py:42:        # flag to enable/disable casting of input to weight dtype during forward call
+    docs/refs/peft_hra_layer.py:58:        config: HRAConfig,
+    docs/refs/peft_hra_layer.py:61:        """Internal function to create hra adapter
+    docs/refs/peft_hra_layer.py:66:            config (`HRAConfig`): The adapter configuration for this layer.
+    docs/refs/peft_hra_layer.py:68:        apply_GS = config.apply_GS
+    docs/refs/peft_hra_layer.py:69:        init_weights = config.init_weights
+    docs/refs/peft_hra_layer.py:75:        self.hra_r[adapter_name] = r
+    docs/refs/peft_hra_layer.py:76:        self.hra_apply_GS[adapter_name] = apply_GS
+    docs/refs/peft_hra_layer.py:78:        # Determine shape of HRA weights
+    docs/refs/peft_hra_layer.py:81:            self.hra_u[adapter_name] = nn.Parameter(torch.empty(self.in_features, r), requires_grad=True)
+    docs/refs/peft_hra_layer.py:83:            self.hra_u[adapter_name] = nn.Parameter(
+    docs/refs/peft_hra_layer.py:88:            raise TypeError(f"HRA is not implemented for base layers of type {type(base_layer).__name__}")
+    docs/refs/peft_hra_layer.py:91:        if init_weights:
+    docs/refs/peft_hra_layer.py:92:            self.reset_hra_parameters(adapter_name)
+    docs/refs/peft_hra_layer.py:94:            self.reset_hra_parameters_random(adapter_name)
+    docs/refs/peft_hra_layer.py:100:    def reset_hra_parameters(self, adapter_name: str):
+    docs/refs/peft_hra_layer.py:101:     ...
+I’ve found one important likely new issue already: upstream PEFT DeLoRA scales by a per-input-channel weight norm, not a single Frobenius scalar. I’m going to verify that against the original author fork before calling it a new bug.
+
+[tool] Read peft_lora_layer.py (running)
+  input: {"call_id":"call_2Y4OTSQuzK2OzQuxYVZAKes5","process_id":"38333","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_layer.py | sed -n '190,245p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_layer.py
+
+[tool] Read peft_lora_layer.py (running)
+  input: {"call_id":"call_opCVMbkydjfHxo72RkbW8Gpz","process_id":"20581","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_layer.py | sed -n '360,395p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_layer.py
+
+[tool] Read orig_pissa_init.py (running)
+  input: {"call_id":"call_25wGw89PzHeRkgbVkFe0nEgN","process_id":"21854","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_pissa_init.py | sed -n '35,65p'"],"cwd":"/media/w...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_pissa_init.py
+
+[tool] Read peft_lora_layer.py (running)
+  input: {"call_id":"call_Cujjy1Wat05YCOIbK5onwiPA","process_id":"16041","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_layer.py | sed -n '941,976p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_layer.py
+
+[tool] Read peft_lora_dora.py (running)
+  input: {"call_id":"call_5vteXTncTSo1hXvTGWuBoAB6","process_id":"13301","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_dora.py | sed -n '86,166p'"],"cwd":"/media/w...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_dora.py
+
+[tool] Read peft_lora_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_2Y4OTSQuzK2OzQuxYVZAKes5","process_id":"38333","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_layer.py | sed -n '190,245p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       190	            lora_dropout_layer = nn.Dropout(p=lora_dropout)
+       191	        else:
+       192	            lora_dropout_layer = nn.Identity()
+       193	
+       194	        self.lora_dropout.update(nn.ModuleDict({adapter_name: lora_dropout_layer}))
+       195	
+       196	        # Actual trainable parameters
+       197	        self.lora_A[adapter_name] = nn.Linear(self.in_features, r, bias=False)
+       198	        self.lora_B[adapter_name] = nn.Linear(r, self.out_features, bias=lora_bias)
+       199	
+       200	        # Tying adapters is only implemented for Linear layers
+       201	        # where the source is the embedding layer.
+       202	        # Currently, this is the most prevelant way of tying layers (weight tying)
+       203	        if tied_adapter:
+       204	            lora_A_params = tied_adapter["lora_A"]
+       205	            lora_B_params = tied_adapter["lora_B"]
+       206	
+       207	            self.lora_A[adapter_name].weight = torch.nn.Parameter(lora_A_params)
+       208	            self.lora_B[adapter_name].weight = torch.nn.Parameter(lora_B_params)
+       209	
+       210	        self.lora_bias[adapter_name] = lora_bias
+       211	
+       212	        if use_rslora:
+       213	            self.scaling[adapter_name] = lora_alpha / math.sqrt(r)
+       214	        else:
+       215	            self.scaling[adapter_name] = lora_alpha / r
+       216	
+    ... (29 more lines)
+
+[tool] Read peft_lora_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_opCVMbkydjfHxo72RkbW8Gpz","process_id":"20581","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_layer.py | sed -n '360,395p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       360	    def pissa_init(self, adapter_name, init_lora_weights):
+       361	        weight = self.get_base_layer().weight
+       362	        dtype = weight.dtype
+       363	        if dtype not in [torch.float32, torch.float16, torch.bfloat16]:
+       364	            raise TypeError(
+       365	                "Please initialize PiSSA under float32, float16, or bfloat16. "
+       366	                "Subsequently, re-quantize the residual model to help minimize quantization errors."
+       367	            )
+       368	        weight = transpose(weight.to(torch.float32), self.fan_in_fan_out)
+       369	        if init_lora_weights == "pissa":
+       370	            # USV^T = W <-> VSU^T = W^T, where W^T = weight.data in R^{out_channel, in_channel},
+       371	            V, S, Uh = torch.linalg.svd(weight.data, full_matrices=False)
+       372	            Vr = V[:, : self.r[adapter_name]]
+       373	            Sr = S[: self.r[adapter_name]]
+       374	            Sr /= self.scaling[adapter_name]
+       375	            Uhr = Uh[: self.r[adapter_name]]
+       376	        elif len(init_lora_weights.split("_niter_")) == 2:
+       377	            Vr, Sr, Ur = svd_lowrank(
+       378	                weight.data, self.r[adapter_name], niter=int(init_lora_weights.split("_niter_")[-1])
+       379	            )
+       380	            Sr /= self.scaling[adapter_name]
+       381	            Uhr = Ur.t()
+       382	        else:
+       383	            raise ValueError(
+       384	                f"init_lora_weights should be 'pissa' or 'pissa_niter_[number of iters]', got {init_lora_weights} instead."
+       385	            )
+       386	
+    ... (9 more lines)
+
+[tool] Read peft_ia3_layer.py (running)
+  input: {"call_id":"call_6x5M2DzKlZugvBcPBhDgk5lo","process_id":"52042","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_ia3_layer.py | sed -n '56,80p'; nl -ba docs/refs/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_ia3_layer.py
+
+[tool] Read orig_pissa_init.py (completed)
+  kind: read
+  input: {"call_id":"call_25wGw89PzHeRkgbVkFe0nEgN","process_id":"21854","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_pissa_init.py | sed -n '35,65p'"],"cwd":"/media/w...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_pissa_init.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        35	    torch_dtype=(
+        36	        torch.float16
+        37	        if script_args.bits == "fp16"
+        38	        else (torch.bfloat16 if script_args.bits == "bf16" else torch.float32)
+        39	    ),
+        40	    device_map="auto",
+        41	)
+        42	tokenizer = AutoTokenizer.from_pretrained(script_args.base_model_path)
+        43	tokenizer.pad_token_id = tokenizer.eos_token_id
+        44	lora_config = LoraConfig(
+        45	    r=script_args.lora_r,
+        46	    lora_alpha=script_args.lora_alpha,
+        47	    init_lora_weights=True if script_args.init_weights=="True" else script_args.init_weights,
+        48	    lora_dropout=script_args.lora_dropout,
+        49	    target_modules=script_args.target_modules,
+        50	)
+        51	peft_model = get_peft_model(model, lora_config)
+        52	
+        53	# Save PiSSA modules:
+        54	peft_model.peft_config["default"].init_lora_weights = True
+        55	peft_model.save_pretrained(os.path.join(script_args.output_dir, "pissa_init"))
+        56	# Save residual model:
+        57	peft_model = peft_model.unload()
+        58	peft_model.save_pretrained(script_args.output_dir)
+        59	# Save the tokenizer:
+        60	tokenizer.save_pretrained(script_args.output_dir)
+
+[tool] Read peft_lora_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_Cujjy1Wat05YCOIbK5onwiPA","process_id":"16041","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_layer.py | sed -n '941,976p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       941	    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+       942	        self._check_forward_args(x, *args, **kwargs)
+       943	        adapter_names = kwargs.pop("adapter_names", None)
+       944	        variant_kwargs = {k: kwargs.pop(k, None) for k in VARIANT_KWARG_KEYS}  # don't pass these to base_layer
+       945	
+       946	        if self.disable_adapters:
+       947	            if self.merged:
+       948	                self.unmerge()
+       949	            result = self.base_layer(x, *args, **kwargs)
+       950	        elif adapter_names is not None:
+       951	            result = self._mixed_batch_forward(x, *args, adapter_names=adapter_names, **variant_kwargs, **kwargs)
+       952	        elif self.merged:
+       953	            result = self.base_layer(x, *args, **kwargs)
+       954	        else:
+       955	            result = self.base_layer(x, *args, **kwargs)
+       956	            torch_result_dtype = result.dtype
+       957	
+       958	            lora_A_keys = self.lora_A.keys()
+       959	            for active_adapter in self.active_adapters:
+       960	                if active_adapter not in lora_A_keys:
+       961	                    continue
+       962	
+       963	                lora_A = self.lora_A[active_adapter]
+       964	                lora_B = self.lora_B[active_adapter]
+       965	                dropout = self.lora_dropout[active_adapter]
+       966	                scaling = self.scaling[active_adapter]
+       967	                x = self._cast_input_dtype(x, lora_A.weight.dtype)
+    ... (9 more lines)
+
+[tool] Read peft_lora_dora.py (completed)
+  kind: read
+  input: {"call_id":"call_5vteXTncTSo1hXvTGWuBoAB6","process_id":"13301","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_lora_dora.py | sed -n '86,166p'"],"cwd":"/media/w...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_lora_dora.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        86	    def get_weight_norm(self, weight, lora_weight, scaling, adapter_name: Optional[str] = None) -> torch.Tensor:
+        87	        # calculate L2 norm of weight matrix, column-wise
+        88	        weight = transpose(weight, self.fan_in_fan_out)
+        89	        weight = weight + scaling * lora_weight
+        90	        weight_norm = torch.linalg.norm(weight, dim=1).to(weight.dtype)
+        91	        return weight_norm
+        92	
+        93	    @cache_decorator("lora-weight")
+        94	    def get_lora_weight(self, lora_A, lora_B, adapter_name: Optional[str] = None):
+        95	        # Don't use `lora_weight = lora_B.weight @ lora_A.weight` because this causes errors with FSDP. Instead,
+        96	        # calculate the same but using forward.
+        97	        x_eye = torch.eye(lora_A.weight.shape[1], device=lora_A.weight.device, dtype=lora_A.weight.dtype)
+        98	        lora_weight = lora_B(lora_A(x_eye)).T
+        99	        return lora_weight
+       100	
+       101	    def update_layer(self, *, base_layer, lora_A, lora_B, scaling, place_on_cpu=False) -> None:
+       102	        # temporarily convert fp16 to fp32, as fp16 can cause trouble on CPU with PyTorch < 2.2
+       103	        dtype_is_fp16 = lora_A.dtype == torch.float16
+       104	        if dtype_is_fp16:
+       105	            lora_A = lora_A.float()
+       106	            lora_B = lora_B.float()
+       107	
+       108	        with gather_params_ctx(base_layer.parameters()):
+       109	            if base_layer.__class__.__name__ == "Linear4bit":
+       110	                # We have to create a copy of the base layer, otherwise, FSDP will throw an error. 8bit does not work
+       111	                # yet because Int8Params cannot be correctly deep-copied (attributes vanish)
+       112	                base_layer = deepcopy(base_layer)
+    ... (54 more lines)
+
+[tool] Read peft_ia3_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_6x5M2DzKlZugvBcPBhDgk5lo","process_id":"52042","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_ia3_layer.py | sed -n '56,80p'; nl -ba docs/refs/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_ia3_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        56	    def update_layer(self, adapter_name: str, config: IA3Config, **kwargs):
+        57	        init_ia3_weights = config.init_ia3_weights
+        58	        inference_mode = config.inference_mode
+        59	
+        60	        # This code works for linear layers, override for other layer types
+        61	        # Actual trainable parameters
+        62	        if self.is_feedforward:
+        63	            weight = torch.randn((1, self.in_features))
+        64	        else:
+        65	            weight = torch.randn((self.out_features, 1))
+        66	        self.ia3_l[adapter_name] = nn.Parameter(weight)
+        67	        if init_ia3_weights:
+        68	            self.reset_ia3_parameters(adapter_name)
+        69	        self._move_adapter_to_device_of_base_layer(adapter_name)
+        70	        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+        71	
+        72	    def reset_ia3_parameters(self, adapter_name):
+        73	        if adapter_name in self.ia3_l.keys():
+        74	            # initialize learned vector with torch.ones
+        75	            nn.init.constant_(self.ia3_l[adapter_name], 1.0)
+        76	
+        77	
+        78	class Linear(nn.Module, IA3Layer):
+        79	    # (IA)^3 implemented in a dense layer
+        80	    def __init__(
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       161	    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+    ... (25 more lines)
+
+[tool] Read peft_hra_layer.py (running)
+  input: {"call_id":"call_tdrRy87nG7HWzt0pyxWOMB5h","process_id":"62390","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_hra_layer.py | sed -n '209,264p'"],"cwd":"/media/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_hra_layer.py
+
+[tool] Read peft_hra_layer.py (running)
+  input: {"call_id":"call_wcSspixdogAwumrsFaLNIN6i","process_id":"94071","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_hra_layer.py | sed -n '78,110p'"],"cwd":"/media/w...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_hra_layer.py
+
+[tool] Read peft_delora_layer.py (running)
+  input: {"call_id":"call_BiuiWIYCjjP1C2MSuBRvrabj","process_id":"14337","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_delora_layer.py | sed -n '112,155p'; nl -ba docs/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_delora_layer.py
+
+[tool] Read orig_hra_layer.py (running)
+  input: {"call_id":"call_K12I8uQGGqEcuqDJ1SNCrsqE","process_id":"8173","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_hra_layer.py | sed -n '52,62p'; nl -ba docs/refs/o...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_hra_layer.py
+
+[tool] Read orig_delora.py (running)
+  input: {"call_id":"call_0C71TzU8xeBpNWvOwibnhBeX","process_id":"92303","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_delora.py | sed -n '302,353p'; nl -ba docs/refs/o...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_delora.py
+
+[tool] Read peft_delora_layer.py (running)
+  input: {"call_id":"call_WUHTervAVvc9qfAXHNlpoc86","process_id":"57442","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_delora_layer.py | sed -n '28,75p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_delora_layer.py
+
+[tool] Read peft_hra_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_wcSspixdogAwumrsFaLNIN6i","process_id":"94071","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_hra_layer.py | sed -n '78,110p'"],"cwd":"/media/w...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_hra_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        78	        # Determine shape of HRA weights
+        79	        base_layer = self.get_base_layer()
+        80	        if isinstance(base_layer, nn.Linear):
+        81	            self.hra_u[adapter_name] = nn.Parameter(torch.empty(self.in_features, r), requires_grad=True)
+        82	        elif isinstance(base_layer, nn.Conv2d):
+        83	            self.hra_u[adapter_name] = nn.Parameter(
+        84	                torch.empty(self.in_features * base_layer.kernel_size[0] * base_layer.kernel_size[0], r),
+        85	                requires_grad=True,
+        86	            )
+        87	        else:
+        88	            raise TypeError(f"HRA is not implemented for base layers of type {type(base_layer).__name__}")
+        89	
+        90	        # Initialize weights
+        91	        if init_weights:
+        92	            self.reset_hra_parameters(adapter_name)
+        93	        else:
+        94	            self.reset_hra_parameters_random(adapter_name)
+        95	
+        96	        # Move new weights to device
+        97	        self._move_adapter_to_device_of_base_layer(adapter_name)
+        98	        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+        99	
+       100	    def reset_hra_parameters(self, adapter_name: str):
+       101	        if self.hra_r[adapter_name] % 2 != 0:
+       102	            warnings.warn("The symmetric initialization can NOT be performed when r is odd!")
+       103	            nn.init.kaiming_uniform_(self.hra_u[adapter_name], a=math.sqrt(5))
+       104	        else:
+    ... (6 more lines)
+
+[tool] Read peft_hra_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_tdrRy87nG7HWzt0pyxWOMB5h","process_id":"62390","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_hra_layer.py | sed -n '209,264p'"],"cwd":"/media/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_hra_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       209	        rank = self.hra_r[adapter_name]
+       210	        apply_GS = self.hra_apply_GS[adapter_name]
+       211	        opt_u = self.hra_u[adapter_name]
+       212	        shape = opt_u.shape
+       213	
+       214	        if apply_GS:
+       215	            weight = [(opt_u[:, 0] / opt_u[:, 0].norm()).view(-1, 1)]
+       216	            for i in range(1, rank):
+       217	                ui = opt_u[:, i].view(-1, 1)
+       218	                for j in range(i):
+       219	                    ui = ui - (weight[j].t() @ ui) * weight[j]
+       220	                weight.append((ui / ui.norm()).view(-1, 1))
+       221	            weight = torch.cat(weight, dim=1)
+       222	            weight = torch.eye(shape[0], device=opt_u.device, dtype=opt_u.dtype) - 2 * weight @ weight.t()
+       223	
+       224	        else:
+       225	            opt_u = opt_u / opt_u.norm(dim=0)
+       226	            weight = torch.eye(shape[0], device=opt_u.device, dtype=opt_u.dtype)
+       227	            if reverse:
+       228	                indices = range(rank - 1, -1, -1)
+       229	            else:
+       230	                indices = range(rank)
+       231	
+       232	            for i in indices:
+       233	                ui = opt_u[:, i].view(-1, 1)
+       234	                weight = weight - 2 * weight @ ui @ ui.t()
+       235	
+    ... (29 more lines)
+
+[tool] Read peft_delora_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_BiuiWIYCjjP1C2MSuBRvrabj","process_id":"14337","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_delora_layer.py | sed -n '112,155p'; nl -ba docs/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_delora_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       112	        self.delora_A[adapter_name] = nn.Parameter(torch.empty(r, self.in_features))
+       113	        self.delora_B[adapter_name] = nn.Parameter(torch.empty(self.out_features, r))
+       114	        self.delora_lambda[adapter_name] = nn.Parameter(torch.empty(1))
+       115	        if module_dropout > 0.0:
+       116	            module_dropout_layer = nn.Dropout(p=module_dropout)
+       117	        else:
+       118	            module_dropout_layer = nn.Identity()
+       119	        self.delora_dropout.update(nn.ModuleDict({adapter_name: module_dropout_layer}))
+       120	
+       121	        # Initialize weights
+       122	        self.reset_delora_parameters(adapter_name, init_weights, delora_lambda)
+       123	
+       124	        # Move new weights to device
+       125	        self._move_adapter_to_device_of_base_layer(adapter_name)
+       126	        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+       127	
+       128	    def reset_delora_parameters(
+       129	        self,
+       130	        adapter_name: str,
+       131	        init_weights: bool = True,
+       132	        delora_lambda: float = 15.0,
+       133	    ) -> None:
+       134	        if adapter_name not in self.delora_A.keys():
+       135	            return
+       136	
+       137	        if init_weights is True:
+       138	            nn.init.kaiming_uniform_(self.delora_A[adapter_name], a=math.sqrt(5))
+    ... (57 more lines)
+
+[tool] Read peft_delora_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_WUHTervAVvc9qfAXHNlpoc86","process_id":"57442","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/peft_delora_layer.py | sed -n '28,75p'"],"cwd":"/media...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/peft_delora_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        28	
+        29	class DeloraLayer(BaseTunerLayer):
+        30	    # All names of layers that may contain (trainable) adapter weights
+        31	    adapter_layer_names = (
+        32	        "delora_A",
+        33	        "delora_B",
+        34	        "delora_lambda",
+        35	    )
+        36	    # All names of other parameters that may contain adapter-related parameters
+        37	    other_param_names = (
+        38	        "r",
+        39	        "delora_dropout",
+        40	        "delora_w_norm",
+        41	    )
+        42	
+        43	    def __init__(self, base_layer: nn.Module, **kwargs) -> None:
+        44	        self.base_layer = base_layer
+        45	        self.r = {}
+        46	        self.delora_dropout = nn.ModuleDict({})
+        47	        self.delora_A = nn.ParameterDict({})
+        48	        self.delora_B = nn.ParameterDict({})
+        49	        self.delora_lambda = nn.ParameterDict({})
+        50	        # Use persistent buffers so they are included in state_dict and saved.
+        51	        self.delora_w_norm = BufferDict({}, persistent=True)
+        52	        # Mark the weight as unmerged
+        53	        self._disable_adapters = False
+        54	        self.merged_adapters = []
+    ... (21 more lines)
+
+[tool] Read orig_delora.py (completed)
+  kind: read
+  input: {"call_id":"call_0C71TzU8xeBpNWvOwibnhBeX","process_id":"92303","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_delora.py | sed -n '302,353p'; nl -ba docs/refs/o...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_delora.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       302	            self.delora_A = nn.Linear(in_features, r, bias=False)
+       303	            self.delora_B = nn.Linear(r, out_features, bias=False)
+       304	
+       305	            self.delora_lambda = nn.Parameter(torch.full((1,), delora_lambda), requires_grad=True)
+       306	
+       307	            # Frozen parameters
+       308	            self.frozen_C = nn.Parameter(torch.empty_like(self.delora_A.weight).copy_(self.delora_A.weight))
+       309	            self.frozen_C.requires_grad = False
+       310	            self.frozen_D = nn.Parameter(torch.empty_like(self.delora_B.weight).copy_(self.delora_B.weight))
+       311	            self.frozen_D.requires_grad = False
+       312	
+       313	            # Freezing the pre-trained weight matrix
+       314	            self.weight.requires_grad = False
+       315	
+       316	        self.reset_parameters()
+       317	        if fan_in_fan_out:
+       318	            self.weight.data = self.weight.data.T
+       319	
+       320	    def reset_parameters(self):
+       321	        nn.Linear.reset_parameters(self)
+       322	        if hasattr(self, "delora_A"):
+       323	            # initialize A the same way as the default for nn.Linear and B to zero
+       324	            nn.init.kaiming_uniform_(self.delora_A.weight, a=math.sqrt(5))
+       325	            nn.init.kaiming_uniform_(self.delora_B.weight, a=math.sqrt(5))
+       326	            nn.init.constant_(self.delora_lambda, self.delora_lambda_value)
+       327	            self.frozen_C.data = self.delora_A.weight.data
+       328	            self.frozen_D.data = self.delora_B.weight.data
+    ... (39 more lines)
+
+[tool] Read orig_hra_layer.py (completed)
+  kind: read
+  input: {"call_id":"call_K12I8uQGGqEcuqDJ1SNCrsqE","process_id":"8173","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_hra_layer.py | sed -n '52,62p'; nl -ba docs/refs/o...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_hra_layer.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        52	        self.oft_r[adapter_name] = nn.Parameter(
+        53	            torch.cat([torch.eye(r, device=weight.device, dtype=weight.dtype),
+        54	                       torch.zeros(shape[0] - r, r, device=weight.device, dtype=weight.dtype)], dim=0))
+        55	
+        56	    def reset_adapter_parameters(self, adapter_name: str):
+        57	        # nn.init.zeros_(self.oft_r[adapter_name])
+        58	        nn.init.kaiming_uniform_(self.oft_r[adapter_name], a=1 / self.eps[adapter_name])
+        59	
+        60	    def reset_adapter_parameters_random(self, adapter_name: str):
+        61	        # nn.init.kaiming_uniform_(self.oft_r[adapter_name], a=math.sqrt(5))
+        62	        nn.init.kaiming_uniform_(self.oft_r[adapter_name], a=1 / self.eps[adapter_name])
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       300	        # scaling factor for each of the smaller block matrix
+       301	        eps = eps * 1 / torch.sqrt(torch.tensor(oft_r.shape[0]))
+       302	        I = (  # noqa: E741
+       303	            torch.zeros((oft_r.size(1), oft_r.size(1)), device=oft_r.device, dtype=oft_r.dtype)
+       304	            .unsqueeze(0)
+       305	            .expand_as(oft_r)
+       306	        )
+       307	        diff = oft_r - I
+       308	        norm_diff = torch.norm(oft_r - I, dim=(1, 2), keepdim=True)
+       309	        mask = (norm_diff <= eps).bool()
+       310	        out = torch.where(mask, oft_r, I + eps * (diff / norm_diff))
+       311	        return out
+       312	
+       313	    def forward(self, x: torch.Tensor, *args, **kwargs) -> torch.Tensor:
+       314	        previous_dtype = x.dtype
+    ... (18 more lines)
+
+[tool] Read orig_delora.py (running)
+  input: {"call_id":"call_ah4F1fipTlU1wMmtswZGndZ5","process_id":"58167","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_delora.py | sed -n '348,358p'"],"cwd":"/media/was...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_delora.py
+
+[tool] Read orig_delora.py (completed)
+  kind: read
+  input: {"call_id":"call_ah4F1fipTlU1wMmtswZGndZ5","process_id":"58167","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba docs/refs/orig_delora.py | sed -n '348,358p'"],"cwd":"/media/was...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/docs/refs/orig_delora.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+       348	        # Get ABCD
+       349	        ABCD = delora_B_weight @ diag12 @ delora_A_weight
+       350	        ABCD = ABCD - self.frozen_D @ diag34 @ self.frozen_C
+       351	
+       352	        # W scaling
+       353	        Wnorm = self.weight.data.norm(dim=0)  # shape: (in_features,)
+       354	        ABCD = torch.mul(ABCD, Wnorm.unsqueeze(0))  # shape: (out_features, in_features)
+       355	
+       356	        return ABCD
+       357	
+       358	    def train(self, mode: bool = True):
+
+[tool] Read qwen_probe.log (running)
+  input: {"call_id":"call_QNV0Ba2iReMmAW5AUbuGc72t","process_id":"70431","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/qwen_probe.log | sed -n '84,110p'"],"cwd":"/media/wassname/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/qwen_probe.log
+
+[tool] Read smoke.log (running)
+  input: {"call_id":"call_G6zvop5SgFkVybpFkSeaqW35","process_id":"82319","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/smoke.log | sed -n '114,176p'"],"cwd":"/media/wassname/SGIr...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/smoke.log
+
+[tool] Search q_proj|v_proj|k_proj|lambda0|variant_kwargs|target_names|variant in scripts (running)
+  input: {"call_id":"call_QzqQbhG7VjJ9UPWbbuCm3as7","process_id":"73468","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"q_proj|v_proj|k_proj|lambda0|variant_kwargs|target_names|varian...
+
+[tool] Read qwen_probe.log (completed)
+  kind: read
+  input: {"call_id":"call_QNV0Ba2iReMmAW5AUbuGc72t","process_id":"70431","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/qwen_probe.log | sed -n '84,110p'"],"cwd":"/media/wassname/...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/qwen_probe.log
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+
+[tool] Read smoke.log (completed)
+  kind: read
+  input: {"call_id":"call_G6zvop5SgFkVybpFkSeaqW35","process_id":"82319","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/smoke.log | sed -n '114,176p'"],"cwd":"/media/wassname/SGIr...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/smoke.log
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+
+[tool] Search q_proj|v_proj|k_proj|lambda0|variant_kwargs|target_names|variant in scripts (completed)
+  kind: search
+  input: {"call_id":"call_QzqQbhG7VjJ9UPWbbuCm3as7","process_id":"73468","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","rg -n \"q_proj|v_proj|k_proj|lambda0|variant_kwargs|target_names|varian...
+  output:
+    scripts/qwen_train_probe.py:17:    "model.layers.0.self_attn.q_proj",
+    scripts/qwen_train_probe.py:18:    "model.layers.0.self_attn.v_proj",
+    scripts/qwen_train_probe.py:22:def cfg_for_variant(variant: str, dtype: torch.dtype, r: int, alpha: float) -> ll.LoraLiteConfig:
+    scripts/qwen_train_probe.py:24:        variant=variant,
+    scripts/qwen_train_probe.py:26:        alpha=r if variant == "pissa" else alpha,
+    scripts/qwen_train_probe.py:29:        target_names=(r"model\.layers\.0\.self_attn\.(q_proj|v_proj)$",),
+    scripts/qwen_train_probe.py:31:        variant_kwargs={"lambda0": 0.1} if variant == "delora" else {},
+    scripts/qwen_train_probe.py:80:def run_variant(args, variant: str, input_ids: torch.Tensor, labels: torch.Tensor, dtype: torch.dtype):
+    scripts/qwen_train_probe.py:83:    cfg = cfg_for_variant(variant, dtype, args.r, args.alpha)
+    scripts/qwen_train_probe.py:138:    out_path = args.out_dir / f"{variant}_adapter.pt"
+    scripts/qwen_train_probe.py:163:        "variant": variant,
+    scripts/qwen_train_probe.py:181:    parser.add_argument("--variants", nargs="+", default=["lora", "pissa", "delora", "ia3", "dora", "hra"])
+    scripts/qwen_train_probe.py:201:    print("SHOULD: exact q_proj/v_proj layer-0 targets, lora-only grads, lossN<loss0, perturb>0, reload<tol. ELSE hook/target/train/save bug.")
+    scripts/qwen_train_probe.py:202:    rows = [run_variant(args, variant, input_ids, labels, dtype) for variant in args.variants]
+    docs/developer_guide.md:3:This is the implementation note for people adding adapter variants. The README is only for prospective users.
+    docs/developer_guide.md:15:A variant is a registered class with a small static interface:
+    docs/developer_guide.md:20:    name = "myvariant"
+    docs/developer_guide.md:42:        layer.lora_* ← variant.param_specs(layer, cfg)
+    docs/developer_guide.md:43:        variant.init(layer, cfg)
+    docs/developer_guide.md:44:        hook(layer, lambda x, y: variant.forward(layer, x, y))
+    docs/developer_guide.md:60:Activation-aware variants imple...
+
+[tool] Read qwen_probe.log (running)
+  input: {"call_id":"call_OhxCJuNFEPRmzaaFxGHbUu4Z","process_id":"42843","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/qwen_probe.log | sed -n '1,40p'"],"cwd":"/media/wassname/SG...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/qwen_probe.log
+
+[tool] Read smoke.log (running)
+  input: {"call_id":"call_4LDa92YA8bBz3kwUeLYLIBG0","process_id":"16740","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/smoke.log | sed -n '1,90p'"],"cwd":"/media/wassname/SGIronW...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/smoke.log
+
+[tool] Read qwen_train_probe.py (running)
+  input: {"call_id":"call_yK1qJHALfCEDcJmKM8bJi3lw","process_id":"2496","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba scripts/qwen_train_probe.py | sed -n '15,35p'"],"cwd":"/media/was...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/scripts/qwen_train_probe.py
+
+[tool] Read qwen_probe.log (completed)
+  kind: read
+  input: {"call_id":"call_OhxCJuNFEPRmzaaFxGHbUu4Z","process_id":"42843","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/qwen_probe.log | sed -n '1,40p'"],"cwd":"/media/wassname/SG...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/qwen_probe.log
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+         1	New task added (id 84).
+         2	SHOULD: exact q_proj/v_proj layer-0 targets, lora-only grads, lossN<loss0, perturb>0, reload<tol. ELSE hook/target/train/save bug.
+         3	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 11183.76it/s]
+         4	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 10483.91it/s]
+         5	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 11231.62it/s]
+         6	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 11419.02it/s]
+         7	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 11501.48it/s]
+         8	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 10138.41it/s]
+         9	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 11177.81it/s]
+        10	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 10532.84it/s]
+        11	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 10290.29it/s]
+        12	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 11146.68it/s]
+        13	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 10274.97it/s]
+        14	
Loading weights:   0%|          | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 311/311 [00:00<00:00, 10701.60it/s]
+        15	variant  	  targets	  trainable	  id_err	  perturb	  loss0	  lossN	  drop%	   grad...
+
+[tool] Read smoke.log (completed)
+  kind: read
+  input: {"call_id":"call_4LDa92YA8bBz3kwUeLYLIBG0","process_id":"16740","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba logs/smoke.log | sed -n '1,90p'"],"cwd":"/media/wassname/SGIronW...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/logs/smoke.log
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+         1	/media/wassname/SGIronWolf/projects5/2026/lora-lite/.venv/lib/python3.13/site-packages/bitsandbytes/autograd/_functions.py:123: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
+         2	  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
+         3	/media/wassname/SGIronWolf/projects5/2026/lora-lite/.venv/lib/python3.13/site-packages/bitsandbytes/backends/cuda/ops.py:213: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious.     Use _check(i >= 0) instead.
+         4	  torch._check_is_size(blocksize)
+         5	
+         6	=== variant=lora dtype=torch.float32 ===
+         7	  attached 28 targets, trainable params=17408
+         8	  t=0 identity: max|y_adapt - y_base| = 0.000e+00  (base scale 4.609e+00)
+         9	  SHOULD: err<4.6e-06. PASS.
+        10	  save/load: max|y_loaded - y_adapt| = 0.000e+00
+        11	  SHOULD: err2<4.6e-06. PASS.
+        12	  loss[0]=1.0882  loss[-1]=1.0220  drop=6.1%
+        13	  SHOULD: drop>5%. PASS.
+        14	
+        15	=== variant=pissa dtype=torch.float32 ===
+        16	  attached 28 targets, trainable params=17408
+        17	  t=0 identity: max|y_adapt - y_base| = 1.550e-06  (base scale 4.609e+00)
+        18	  SHOULD: err<2.3e-03. PASS.
+        19	  save/load: max|y_loaded - y_adapt| = 0.000e+00
+        20	  SHOULD: err2<2.3e-03. PASS.
+        21	  loss[0]=1.0823  loss[-1]=0.9583  drop=11.5%
+        22	  SHOULD: drop>5%. PASS.
+        23	
+        24	=== variant=delora dtype=torch.float32 ===
+        25	  attached 28 targets, trainable params=17436
+        26	  t=0 identity: max|y_adapt - y_base| = 0.000e+00  (base scale 4.609e+00)
+        27	  SHOULD: err<4.6e-06. PASS.
+    ... (54 more lines)
+
+[tool] Read qwen_train_probe.py (completed)
+  kind: read
+  input: {"call_id":"call_yK1qJHALfCEDcJmKM8bJi3lw","process_id":"2496","turn_id":"019dc987-7ec1-7ee2-8133-220e153d9394","command":["/usr/bin/zsh","-lc","nl -ba scripts/qwen_train_probe.py | sed -n '15,35p'"],"cwd":"/media/was...
+  files: /media/wassname/SGIronWolf/projects5/2026/lora-lite/scripts/qwen_train_probe.py
+  output:
+    sed: /tmp/.mount_vscodiOHg5sd/lib/x86_64-linux-gnu/libselinux.so.1: no version information available (required by sed)
+        15	PROMPT = "LoRA-lite probe: Paris is the capital of France. The answer is"
+        16	EXPECTED_TARGETS = {
+        17	    "model.layers.0.self_attn.q_proj",
+        18	    "model.layers.0.self_attn.v_proj",
+        19	}
+        20	
+        21	
+        22	def cfg_for_variant(variant: str, dtype: torch.dtype, r: int, alpha: float) -> ll.LoraLiteConfig:
+        23	    return ll.LoraLiteConfig(
+        24	        variant=variant,
+        25	        r=r,
+        26	        alpha=r if variant == "pissa" else alpha,
+        27	        dtype=dtype,
+        28	        target_roles=(),
+        29	        target_names=(r"model\.layers\.0\.self_attn\.(q_proj|v_proj)$",),
+        30	        layers=(0,),
+        31	        variant_kwargs={"lambda0": 0.1} if variant == "delora" else {},
+        32	    )
+        33	
+        34	
+        35	def adapter_state(model: torch.nn.Module) -> dict[str, torch.Tensor]:
+# Per-Variant Paper-Faithfulness Audit V2
+
+## LoRA
+
+**Reference impl ground-truth**
+
+PEFT creates `A: in -> r`, `B: r -> out`, scales by `alpha/r`, initializes `A` Kaiming and `B` zero, then adds `B(A(dropout(x))) * scaling` to the base output:
+
+```text
+docs/refs/peft_lora_layer.py:197 self.lora_A[adapter_name] = nn.Linear(self.in_features, r, bias=False)
+docs/refs/peft_lora_layer.py:198 self.lora_B[adapter_name] = nn.Linear(r, self.out_features, bias=lora_bias)
+docs/refs/peft_lora_layer.py:215 self.scaling[adapter_name] = lora_alpha / r
+docs/refs/peft_lora_layer.py:269 nn.init.kaiming_uniform_(self.lora_A[adapter_name].weight, a=math.sqrt(5))
+docs/refs/peft_lora_layer.py:274 nn.init.zeros_(self.lora_B[adapter_name].weight)
+docs/refs/peft_lora_layer.py:969 result = result + lora_B(lora_A(dropout(x))) * scaling
+```
+
+Save/load ground-truth: adapter params are named adapter layers in PEFT (`lora_A`, `lora_B`) and are state-dict parameters; no base-weight mutation for vanilla LoRA.
+
+**Our code**
+
+```text
+src/lora_lite/variants/lora.py:26 "lora_A": ParamSpec((cfg.r, d_in), init="kaiming", trainable=True),
+src/lora_lite/variants/lora.py:27 "lora_B": ParamSpec((d_out, cfg.r), init="zeros", trainable=True),
+src/lora_lite/variants/lora.py:38 scale = cfg.alpha / cfg.r
+src/lora_lite/variants/lora.py:39 h = einsum(x, layer.lora_A, "... i, r i -> ... r")
+src/lora_lite/variants/lora.py:40 delta = einsum(h, layer.lora_B, "... r, o r -> ... o")
+src/lora_lite/variants/lora.py:41 return y + scale * delta
+```
+
+**Diff**
+
+- [OK-doc] Parameter shapes, init, scale, and additive forward match PEFT for `dropout=0`.
+- [NEW-BUG] `cfg.dropout` is exposed but ignored: PEFT applies `dropout(x)` before `A` (`peft_lora_layer.py:969`), while our forward always uses raw `x`. Failure mode: setting `LoraLiteConfig(dropout>0)` silently has no effect. This is partly admitted in `src/lora_lite/config.py:11`, but not in the LoRA docstring.
+- [STYLE] PEFT supports bias, multiple adapters, fan-in/fan-out, merge/unmerge; lora-lite intentionally omits those.
+
+**Did the prior review get it right?**
+
+Prior verdict: `docs/audit/variants_review.md:1175` says “Faithful. Kaiming is not literally ‘Gaussian’... and the scaling matches.” Confirm for default `dropout=0`, but prior missed the exposed-dropout no-op.
+
+**Verdict: Faithful-with-doc-gap**
+
+Faithful to PEFT LoRA at default settings; doc/config should say dropout is unsupported or implement it.
+
+## PiSSA
+
+**Reference impl ground-truth**
+
+PEFT PiSSA SVDs the base weight in fp32, divides singular values by scaling, installs `A,B`, and mutates the base weight to the residual:
+
+```text
+docs/refs/peft_lora_layer.py:368 weight = transpose(weight.to(torch.float32), self.fan_in_fan_out)
+docs/refs/peft_lora_layer.py:371 V, S, Uh = torch.linalg.svd(weight.data, full_matrices=False)
+docs/refs/peft_lora_layer.py:374 Sr /= self.scaling[adapter_name]
+docs/refs/peft_lora_layer.py:387 lora_A = torch.diag(torch.sqrt(Sr)) @ Uhr
+docs/refs/peft_lora_layer.py:388 lora_B = Vr @ torch.diag(torch.sqrt(Sr))
+docs/refs/peft_lora_layer.py:391 weight = weight.data - self.scaling[adapter_name] * lora_B @ lora_A
+docs/refs/peft_lora_layer.py:393 self.get_base_layer().weight.data = weight
+```
+
+Original PiSSA save flow explicitly saves both PiSSA modules and the residual model:
+
+```text
+docs/refs/orig_pissa_init.py:53 # Save PiSSA modules:
+docs/refs/orig_pissa_init.py:55 peft_model.save_pretrained(os.path.join(script_args.output_dir, "pissa_init"))
+docs/refs/orig_pissa_init.py:56 # Save residual model:
+docs/refs/orig_pissa_init.py:57 peft_model = peft_model.unload()
+docs/refs/orig_pissa_init.py:58 peft_model.save_pretrained(script_args.output_dir)
+```
+
+**Our code**
+
+```text
+src/lora_lite/variants/pissa.py:47 U, S, Vh = torch.linalg.svd(W, full_matrices=False)
+src/lora_lite/variants/pissa.py:52 B = (Ur * sqrtS).to(cfg.dtype)
+src/lora_lite/variants/pissa.py:53 A = (sqrtS[:, None] * Vhr).to(cfg.dtype)
+src/lora_lite/variants/pissa.py:54 layer.lora_B.data.copy_(B)
+src/lora_lite/variants/pissa.py:55 layer.lora_A.data.copy_(A)
+src/lora_lite/variants/pissa.py:60 scale = cfg.alpha / cfg.r
+src/lora_lite/variants/pissa.py:61 layer.weight.data.copy_((W - scale * BA).to(layer.weight.dtype))
+```
+
+Adapter save/load only stores `lora_` tensors:
+
+```text
+src/lora_lite/adapter.py:98 sd = {k: v.detach().cpu() for k, v in model.state_dict().items() if "lora_" in k}
+src/lora_lite/adapter.py:105 handles = attach(model, cfg)  # creates empty params with right shapes
+```
+
+**Diff**
+
+- [OK-doc] Same residualization idea as PEFT: base `W <- W - scale * BA`, forward adds `scale * BAx`.
+- [OK-undoc] PEFT divides `Sr` by `scaling` before forming `A,B`; our code does not, but subtracts/adds `scale * BA`. Both preserve identity when using the same scale. The trained parameterization differs by a scale factor, so this should be noted.
+- [BUG] PiSSA save/load is not reference-faithful: original PiSSA saves both adapter and mutated residual model (`orig_pissa_init.py:53-58`), while lora-lite saves only `lora_` tensors and recomputes residual on load. Loading onto a base that is not exactly the same silently changes the effective model.
+- [BUG] bf16/Qwen identity error remains real: `logs/qwen_probe.log:17` reports `pissa id_err=0.3125` while still ending `ALL QWEN PROBES PASS` at line 22.
+- [NEW-BUG] Same dropout no-op as LoRA: PEFT forward path still uses `dropout(x)` for LoRA-family layers (`peft_lora_layer.py:969`); our PiSSA ignores `cfg.dropout`.
+
+**Did the prior review get it right?**
+
+Prior verdict: `docs/audit/variants_review.md:1254` says “Potentially material bf16 initialization error... Save/load assumes loading into the same unmodified base model...” Confirmed. Reference code makes the save/load issue stronger because the PiSSA author flow saves the residual model separately.
+
+**Verdict: Partial**
+
+Core PiSSA math is close; persistence and bf16 identity are not reference-faithful.
+
+## DoRA
+
+**Reference impl ground-truth**
+
+PEFT DoRA stores a trainable magnitude vector initialized to `||W + scale*BA||` and in forward detaches the denominator norm:
+
+```text
+docs/refs/peft_lora_dora.py:86 def get_weight_norm(self, weight, lora_weight, scaling, adapter_name: Optional[str] = None)
+docs/refs/peft_lora_dora.py:89 weight = weight + scaling * lora_weight
+docs/refs/peft_lora_dora.py:90 weight_norm = torch.linalg.norm(weight, dim=1).to(weight.dtype)
+docs/refs/peft_lora_dora.py:130 self.weight = nn.Parameter(weight_norm, requires_grad=True)
+docs/refs/peft_lora_dora.py:143 weight_norm = self.get_weight_norm(
+docs/refs/peft_lora_dora.py:152 weight_norm = weight_norm.detach()
+docs/refs/peft_lora_dora.py:165 result_dora = (mag_norm_scale - 1) * base_result + mag_norm_scale * lora_result * scaling
+```
+
+**Our code**
+
+```text
+src/lora_lite/variants/dora.py:36 "lora_A": ParamSpec((cfg.r, d_in), init="kaiming", trainable=True),
+src/lora_lite/variants/dora.py:37 "lora_B": ParamSpec((d_out, cfg.r), init="zeros", trainable=True),
+src/lora_lite/variants/dora.py:39 "lora_m": ParamSpec((d_out,), init="zeros", trainable=True),
+src/lora_lite/variants/dora.py:51 col_norm = W.norm(dim=1).to(layer.lora_m.dtype)
+src/lora_lite/variants/dora.py:60 V = layer.weight + scale * BA
+src/lora_lite/variants/dora.py:61 v_norm = V.norm(dim=1).clamp_min(1e-12)
+src/lora_lite/variants/dora.py:66 return (layer.lora_m / v_norm) * combined
+```
+
+**Diff**
+
+- [OK-doc] Parameterization, magnitude shape, initialization, and forward value match PEFT for plain `nn.Linear` at `dropout=0`.
+- [OK-doc] Gradient differs from PEFT: reference detaches `weight_norm` (`peft_lora_dora.py:152`); our docstring explicitly says it differentiates through `||V||`.
+- [NEW-BUG] Dropout no-op again: PEFT LoRA wrapper passes `dropout(x)` before DoRA variant invocation; our DoRA ignores `cfg.dropout`.
+- [STYLE] PEFT handles dequantization and Linear4bit for DoRA magnitude init; lora-lite raises on non-plain `nn.Linear`, documented.
+
+**Did the prior review get it right?**
+
+Prior verdict: `docs/audit/variants_review.md:1277` says “Faithful for plain `nn.Linear`...” and `docs/audit/variants_review.md:1305` flags the denominator-gradient concern. Confirmed, but direct PEFT reference upgrades the denominator difference from paper-only concern to concrete reference divergence.
+
+**Verdict: Partial**
+
+Forward is faithful; training gradients intentionally diverge from PEFT DoRA.
+
+## IA3
+
+**Reference impl ground-truth**
+
+PEFT IA3 uses different vector shapes and placement depending on whether the target is feedforward. Non-feedforward scales output; feedforward scales input before the base layer:
+
+```text
+docs/refs/peft_ia3_layer.py:62 if self.is_feedforward:
+docs/refs/peft_ia3_layer.py:63     weight = torch.randn((1, self.in_features))
+docs/refs/peft_ia3_layer.py:65     weight = torch.randn((self.out_features, 1))
+docs/refs/peft_ia3_layer.py:75 nn.init.constant_(self.ia3_l[adapter_name], 1.0)
+docs/refs/peft_ia3_layer.py:177 if self.is_feedforward:
+docs/refs/peft_ia3_layer.py:181     interm = (x * ia3_scaling).to(previous_dtype)
+docs/refs/peft_ia3_layer.py:182     result = self.base_layer(interm, *args, **kwargs)
+docs/refs/peft_ia3_layer.py:184 result = self.base_layer(x, *args, **kwargs)
+docs/refs/peft_ia3_layer.py:186 result = (result * ia3_scaling).to(result_dtype)
+```
+
+**Our code**
+
+```text
+src/lora_lite/variants/ia3.py:37 return {"lora_g": ParamSpec((d_out,), init="ones", trainable=True)}
+src/lora_lite/variants/ia3.py:44 def forward(layer: nn.Linear, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
+src/lora_lite/variants/ia3.py:45     return y * layer.lora_g
+```
+
+**Diff**
+
+- [OK-doc] For non-feedforward projections, output gating with a ones vector matches PEFT’s non-feedforward path.
+- [BUG] Feedforward IA3 is not reference-faithful. PEFT uses shape `(1, in_features)` and multiplies the input to the feedforward linear (`peft_ia3_layer.py:177-182`); our IA3 always uses `(d_out,)` and multiplies output. This matters for the paper’s FFN-intermediate gate.
+- [BUG] The IA3 docstring recommends `up_proj` (`src/lora_lite/variants/ia3.py:13`), but reference/paper-faithful FFN gating is better represented as input gating of `down_proj` in Llama/Qwen-style MLPs. Our variant cannot express that with its current forward-only hook.
+- [OK-doc] Qwen probe intentionally targets `q_proj|v_proj`, not IA3 paper placement: `scripts/qwen_train_probe.py:29`.
+- [NEW-BUG] IA3 default targeting is not IA3-like: `LoraLiteConfig.target_roles=("reader","writer")` can attach to many linears, and the variant has no feedforward role handling.
+
+**Did the prior review get it right?**
+
+Prior verdict: `docs/audit/variants_review.md:1325` says target semantics are not paper-faithful and Qwen uses `q_proj/v_proj`; confirmed. The prior review under-specified the feedforward arithmetic: direct PEFT shows it is input-side scaling, not merely “choose the right target.”
+
+**Verdict: Partial**
+
+Projection gates are IA3-like; FFN IA3 is structurally missing.
+
+## HRA
+
+**Reference impl ground-truth**
+
+PEFT HRA stores `hra_u` as `(in_features, r)`, initializes symmetrically for even rank, optionally Gram-Schmidts, builds Householder products, and applies `W @ R` without a zero gate:
+
+```text
+docs/refs/peft_hra_layer.py:81 self.hra_u[adapter_name] = nn.Parameter(torch.empty(self.in_features, r), requires_grad=True)
+docs/refs/peft_hra_layer.py:106 half_u = torch.zeros(shape[0], shape[1] // 2)
+docs/refs/peft_hra_layer.py:108 self.hra_u[adapter_name] = nn.Parameter(torch.repeat_interleave(half_u, 2, dim=1))
+docs/refs/peft_hra_layer.py:214 if apply_GS:
+docs/refs/peft_hra_layer.py:222 weight = torch.eye(shape[0], device=opt_u.device, dtype=opt_u.dtype) - 2 * weight @ weight.t()
+docs/refs/peft_hra_layer.py:225 opt_u = opt_u / opt_u.norm(dim=0)
+docs/refs/peft_hra_layer.py:234 weight = weight - 2 * weight @ ui @ ui.t()
+docs/refs/peft_hra_layer.py:258 new_weight = torch.mm(orig_weight, new_weight)
+```
+
+Original HRA/OFT repo initializes around an identity-like matrix and applies base forward plus transformation logic, not a scalar zero gate:
+
+```text
+docs/refs/orig_hra_layer.py:52 self.oft_r[adapter_name] = nn.Parameter(
+docs/refs/orig_hra_layer.py:53     torch.cat([torch.eye(r, device=weight.device, dtype=weight.dtype),
+docs/refs/orig_hra_layer.py:58 nn.init.kaiming_uniform_(self.oft_r[adapter_name], a=1 / self.eps[adapter_name])
+```
+
+**Our code**
+
+```text
+src/lora_lite/variants/hra.py:51 "lora_U": ParamSpec((cfg.r, d_in), init="kaiming", trainable=True),
+src/lora_lite/variants/hra.py:53 "lora_gate": ParamSpec((), init="zeros", trainable=True),
+src/lora_lite/variants/hra.py:65 for i in range(U.shape[0]):
+src/lora_lite/variants/hra.py:67 sq = (u * u).sum().clamp_min(1e-12)
+src/lora_lite/variants/hra.py:68 coeff = einsum(Rx, u, "... i, i -> ...") * (2.0 / sq)
+src/lora_lite/variants/hra.py:69 Rx = Rx - coeff.unsqueeze(-1) * u
+src/lora_lite/variants/hra.py:70 return x + layer.lora_gate * (Rx - x)
+```
+
+**Diff**
+
+- [OK-undoc] Shape is transposed relative to PEFT: our `(r,d_in)` is equivalent to PEFT `(d_in,r)` for the loop implementation.
+- [BUG] Zero scalar gate is not in PEFT or original HRA. It preserves identity, but causes step-0 zero gradient for `lora_U`. This was already documented in the new docstring and is confirmed by the reference.
+- [BUG] PEFT’s identity-preserving init for even `r` repeats columns in pairs (`peft_hra_layer.py:106-108`), so products cancel without killing gradients. Our implementation should use the symmetric pair init instead of a zero gate.
+- [BUG] PEFT has `apply_GS` orthogonalization (`peft_hra_layer.py:214-222`); our code has no equivalent. Without it, the “strict orthogonal HRA” reference mode is unavailable.
+- [STYLE] PEFT materializes `new_weight = W @ R`; our pre-hook computes `R x` then base computes `W(Rx)`. Forward-equivalent for linear layers.
+
+**Did the prior review get it right?**
+
+Prior verdict: `docs/audit/variants_review.md:1407` says “zero-gate formulation is materially different from paper HRA and creates an initial dead-`U` step.” Confirmed. Direct PEFT reference also reveals the better fix: symmetric repeated-column init.
+
+**Verdict: Buggy**
+
+The core Householder transform is present, but initialization is reference-incompatible and creates avoidable dead gradients.
+
+## DeLoRA
+
+**Reference impl ground-truth**
+
+PEFT DeLoRA stores `A`, `B`, trainable `lambda`, and a persistent per-input-channel `w_norm`; init uses Kaiming `A`, zero `B` by default, and forward multiplies `x` by `w_norm`, normalizes `A/B`, scales by `lambda/r`, then applies `B`:
+
+```text
+docs/refs/peft_delora_layer.py:50 # Use persistent buffers so they are included in state_dict and saved.
+docs/refs/peft_delora_layer.py:51 self.delora_w_norm = BufferDict({}, persistent=True)
+docs/refs/peft_delora_layer.py:112 self.delora_A[adapter_name] = nn.Parameter(torch.empty(r, self.in_features))
+docs/refs/peft_delora_layer.py:113 self.delora_B[adapter_name] = nn.Parameter(torch.empty(self.out_features, r))
+docs/refs/peft_delora_layer.py:138 nn.init.kaiming_uniform_(self.delora_A[adapter_name], a=math.sqrt(5))
+docs/refs/peft_delora_layer.py:139 nn.init.zeros_(self.delora_B[adapter_name])
+docs/refs/peft_delora_layer.py:144 self.delora_lambda[adapter_name].data.fill_(float(delora_lambda))
+docs/refs/peft_delora_layer.py:150 w_norm = torch.norm(w.data, dim=0).detach()
+docs/refs/peft_delora_layer.py:250 h = nn.functional.linear(x_d * self.delora_w_norm[adapter], self.delora_A[adapter])
+docs/refs/peft_delora_layer.py:255 scaling = (self.delora_lambda[adapter] / self.r[adapter]) / (An * Bn)
+```
+
+Original author fork differs from upstream PEFT: it initializes both `A,B` Kaiming and subtracts a frozen copy so identity holds even with nonzero `lambda`:
+
+```text
+docs/refs/orig_delora.py:324 nn.init.kaiming_uniform_(self.delora_A.weight, a=math.sqrt(5))
+docs/refs/orig_delora.py:325 nn.init.kaiming_uniform_(self.delora_B.weight, a=math.sqrt(5))
+docs/refs/orig_delora.py:327 self.frozen_C.data = self.delora_A.weight.data
+docs/refs/orig_delora.py:328 self.frozen_D.data = self.delora_B.weight.data
+docs/refs/orig_delora.py:349 ABCD = delora_B_weight @ diag12 @ delora_A_weight
+docs/refs/orig_delora.py:350 ABCD = ABCD - self.frozen_D @ diag34 @ self.frozen_C
+docs/refs/orig_delora.py:353 Wnorm = self.weight.data.norm(dim=0)
+docs/refs/orig_delora.py:354 ABCD = torch.mul(ABCD, Wnorm.unsqueeze(0))
+```
+
+**Our code**
+
+```text
+src/lora_lite/variants/delora.py:48 "lora_A": ParamSpec((cfg.r, d_in), init="kaiming", trainable=True),
+src/lora_lite/variants/delora.py:49 "lora_B": ParamSpec((d_out, cfg.r), init="kaiming", trainable=True),
+src/lora_lite/variants/delora.py:51 (), init=lambda t: t.fill_(lam0), trainable=True
+src/lora_lite/variants/delora.py:63 layer.lora_wnorm.data.fill_(W.norm().item())
+src/lora_lite/variants/delora.py:70 A = F.normalize(layer.lora_A, dim=1)
+src/lora_lite/variants/delora.py:71 B = F.normalize(layer.lora_B, dim=0)
+src/lora_lite/variants/delora.py:72 scale = layer.lora_lambda * layer.lora_wnorm / cfg.r
+src/lora_lite/variants/delora.py:75 return y + scale * delta
+```
+
+**Diff**
+
+- [BUG] Against upstream PEFT, `B` init differs: PEFT default uses `B=0` (`peft_delora_layer.py:139`), our code uses Kaiming `B`.
+- [BUG] Against original author repo, our code omits the frozen-copy subtraction (`orig_delora.py:327-350`). This confirms the prior “lambda0=0 dead A/B vs lambda0>0 non-identity” finding.
+- [NEW-BUG] Our `lora_wnorm` is a scalar Frobenius norm (`W.norm().item()`), but both PEFT and author repo use per-input-channel norm `norm(dim=0)` (`peft_delora_layer.py:150`, `orig_delora.py:353-354`). Failure mode: the DeLoRA update has the wrong feature-wise scaling and cannot match reference behavior except in degenerate equal-column-norm cases.
+- [NEW-BUG] `lora_wnorm` is registered as a non-trainable parameter because `ParamSpec.make` always returns `nn.Parameter`; PEFT uses a persistent buffer (`peft_delora_layer.py:50-51`). Optimizers will usually ignore it due to `requires_grad=False`, but state semantics are not reference-clean.
+- [OK-doc] `lambda0=0` identity and dead first-step `A/B` gradients are documented in our docstring.
+- [NEW-BUG] Dropout no-op: PEFT DeLoRA has `delora_dropout` and applies it at `peft_delora_layer.py:246`; our code ignores `cfg.dropout`.
+
+**Did the prior review get it right?**
+
+Prior verdict: `docs/audit/variants_review.md:1471` says “no Eq.9 frozen-copy init; zero-lambda dead A/B first step; nonzero-lambda breaks identity.” Confirmed. Direct reference comparison adds the per-input-channel norm bug.
+
+**Verdict: Buggy**
+
+It is DeLoRA-shaped, but not faithful to either upstream PEFT or the original author implementation.
+
+## Aggregate
+
+| variant | prior verdict | new verdict | new bugs found | doc gaps |
+|---|---:|---:|---|---|
+| lora | Faithful | Faithful-with-doc-gap | `cfg.dropout` silently ignored | LoRA docstring should mention dropout unsupported |
+| pissa | Partial | Partial | dropout ignored | parameter scaling differs from PEFT; save/load caveat should cite residual-base requirement |
+| dora | Faithful | Partial | dropout ignored | PEFT denominator detach is documented, but verdict should say reference-gradient divergence |
+| ia3 | Partial | Partial | FFN gate requires input-side scaling; doc recommends wrong `up_proj` stand-in | document non-feedforward-only faithfulness |
+| hra | Partial | Buggy | none beyond prior, but PEFT reveals symmetric init fix | document lack of `apply_GS` |
+| delora | Partial | Buggy | wrong scalar `W` norm; dropout ignored; `w_norm` should be buffer-like | distinguish PEFT upstream vs original-author initialization |
+
+## What To Fix Next
+
+1. Fix DeLoRA scaling: replace scalar `W.norm()` with per-input-channel `W.norm(dim=0)`, and decide whether to follow upstream PEFT `B=0` or author frozen-copy init.
+2. Replace HRA zero gate with PEFT-style symmetric repeated-column initialization so identity holds without dead `U` gradients.
+3. Fix IA3 FFN support: add input-side gating for feedforward targets and update docs from `up_proj` to the correct `down_proj`/FFN-intermediate semantics.
+4. Make PiSSA persistence explicit: either save/load the residual base mutation or refuse/load-warn unless the exact same base is guaranteed.
+5. Either implement adapter dropout across LoRA-family variants or remove/rename `cfg.dropout` so users do not get a silent no-op.
+
+[done] end_turn
diff --git a/docs/refs/orig_delora.py b/docs/refs/orig_delora.py
new file mode 100644
index 0000000..b727d5e
--- /dev/null
+++ b/docs/refs/orig_delora.py
@@ -0,0 +1,446 @@
+import importlib
+import math
+import re
+import warnings
+from dataclasses import asdict, dataclass, field
+from enum import Enum
+from typing import List, Optional, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers.pytorch_utils import Conv1D
+
+from ..utils import PeftConfig, PeftType, transpose
+
+
+def is_bnb_available():
+    return importlib.util.find_spec("bitsandbytes") is not None
+
+
+if is_bnb_available():
+    import bitsandbytes as bnb
+
+
+@dataclass
+class DeloraConfig(PeftConfig):
+    """
+    This is the configuration class to store the configuration of a [`~peft.Delora`].
+
+    Args:
+        r (`int`): Delora attention dimension
+        target_modules (`Union[List[str],str]`): The names of the modules to apply Delora to.
+        delora_lambda (`float`): The lambda parameter for Delora scaling.
+        delora_dropout (`float`): The dropout probability for Delora layers.
+        merge_weights (`bool`):
+            Whether to merge the weights of the Delora layers with the base transformer model in `eval` mode.
+        fan_in_fan_out (`bool`): Set this to True if the layer to replace stores weight like (fan_in, fan_out)
+        enable_delora ( `List[bool]`): Used with `delora.MergedLinear`.
+        bias (`str`): Bias type for Delora. Can be 'none', 'all' or 'delora_only'
+        modules_to_save (`List[str]`):List of modules apart from Delora layers to be set as trainable
+            and saved in the final checkpoint.
+    """
+
+    r: int = field(default=8, metadata={"help": "Delora attention dimension"})
+    target_modules: Optional[Union[List[str], str]] = field(
+        default=None,
+        metadata={
+            "help": "List of module names or regex expression of the module names to replace with Delora."
+            "For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' "
+        },
+    )
+    delora_lambda: int = field(default=None, metadata={"help": "Delora lambda"})
+    delora_dropout: float = field(default=None, metadata={"help": "Delora dropout"})
+    Wdecompose_target_modules: Optional[Union[List[str], str]] = field(
+        default=None,
+        metadata={
+            "help": "List of module names or regex expression of the module names to only tune the magnitude part"
+            "For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' "
+        },
+    )
+    merge_weights: bool = field(
+        default=False, metadata={"help": "Merge weights of the original model and the Delora model"}
+    )
+    fan_in_fan_out: bool = field(
+        default=False,
+        metadata={"help": "Set this to True if the layer to replace stores weight like (fan_in, fan_out)"},
+    )
+    enable_delora: Optional[List[bool]] = field(default=None, metadata={"help": "Used with `delora.MergedLinear`."})
+    bias: str = field(default="none", metadata={"help": "Bias type for Delora. Can be 'none', 'all' or 'delora_only'"})
+    modules_to_save: Optional[List[str]] = field(
+        default=None,
+        metadata={
+            "help": "List of modules apart from Delora layers to be set as trainable and saved in the final checkpoint. "
+            "For example, in Sequence Classification or Token Classification tasks, "
+            "the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved."
+        },
+    )
+
+    def __post_init__(self):
+        self.peft_type = PeftType.DELORA
+
+
+class DeloraModel(torch.nn.Module):
+    """
+    Creates Decoupled Low Rank Adapter (Delora) model from a pretrained transformers model.
+
+    Args:
+        model ([`transformers.PreTrainedModel`]): The model to be adapted.
+        config ([`DeloraConfig`]): The configuration of the Delora model.
+
+    Returns:
+        `torch.nn.Module`: The Delora model.
+
+    Example::
+
+        >>> from transformers import AutoModelForSeq2SeqLM, DeloraConfig >>> from peft import DeloraModel, DeloraConfig >>>
+        config = DeloraConfig(
+            peft_type="DELORA", task_type="SEQ_2_SEQ_LM", r=8, delora_lambda=32, target_modules=["q", "v"],
+            delora_dropout=0.01, )
+        >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> delora_model = DeloraModel(config, model)
+
+    **Attributes**:
+        - **model** ([`transformers.PreTrainedModel`]) -- The model to be adapted.
+        - **peft_config** ([`DeloraConfig`]): The configuration of the Delora model.
+    """
+
+    def __init__(self, config, model):
+        super().__init__()
+        self.peft_config = config
+        print(self.peft_config)
+        self.model = model
+        self._find_and_replace()
+        mark_only_delora_as_trainable(self.model, self.peft_config.bias)
+        self.forward = self.model.forward
+
+    def _find_and_replace(self):
+        loaded_in_8bit = getattr(self.model, "is_loaded_in_8bit", False)
+        if loaded_in_8bit and not is_bnb_available():
+            raise ImportError(
+                "To use Delora with 8-bit quantization, please install the `bitsandbytes` package. "
+                "You can install it with `pip install bitsandbytes`."
+            )
+        is_target_modules_in_base_model = False
+        is_hf_device_map_available = hasattr(self.model, "hf_device_map")
+        kwargs = {
+            "r": self.peft_config.r,
+            "delora_lambda": self.peft_config.delora_lambda,
+            "delora_dropout": self.peft_config.delora_dropout,
+            "fan_in_fan_out": self.peft_config.fan_in_fan_out,
+            "merge_weights": (self.peft_config.merge_weights or self.peft_config.inference_mode)
+            and not is_hf_device_map_available,
+        }
+        key_list = [key for key, _ in self.model.named_modules()]
+        for key in key_list:
+            if isinstance(self.peft_config.target_modules, str):
+                target_module_found = re.fullmatch(self.peft_config.target_modules, key)
+            else:
+                target_module_found = any(key.endswith(target_key) for target_key in self.peft_config.target_modules)
+
+            if target_module_found:
+                if not is_target_modules_in_base_model:
+                    is_target_modules_in_base_model = True
+                parent, target, target_name = self._get_submodules(key)
+                bias = target.bias is not None
+                if loaded_in_8bit and isinstance(target, bnb.nn.Linear8bitLt):
+                    kwargs.update(
+                        {
+                            "has_fp16_weights": target.state.has_fp16_weights,
+                            "memory_efficient_backward": target.state.memory_efficient_backward,
+                            "threshold": target.state.threshold,
+                            "index": target.index,
+                        }
+                    )
+                    if self.peft_config.enable_delora is None:
+                        print("8 bit delora")
+                        new_module = Linear8bitLt(target.in_features, target.out_features, bias=bias, **kwargs)
+                    else:
+                        kwargs.update({"enable_delora": self.peft_config.enable_delora})
+                        new_module = MergedLinear8bitLt(target.in_features, target.out_features, bias=bias, **kwargs)
+                elif isinstance(target, torch.nn.Linear) and self.peft_config.enable_delora is None:
+                    new_module = Linear(target.in_features, target.out_features, bias=bias, **kwargs)
+                elif self.peft_config.enable_delora is not None:
+                    kwargs.update({"enable_delora": self.peft_config.enable_delora})
+                    if isinstance(target, Conv1D):
+                        in_features, out_features = (
+                            target.weight.ds_shape if hasattr(target.weight, "ds_shape") else target.weight.shape
+                        )
+                    else:
+                        in_features, out_features = target.in_features, target.out_features
+                        if kwargs["fan_in_fan_out"]:
+                            warnings.warn(
+                                "fan_in_fan_out is set to True but the target module is not a Conv1D. "
+                                "Setting fan_in_fan_out to False."
+                            )
+                            kwargs["fan_in_fan_out"] = self.peft_config.fan_in_fan_out = False
+                    new_module = MergedLinear(in_features, out_features, bias=bias, **kwargs)
+                self._replace_module(parent, target_name, new_module, target)
+        if not is_target_modules_in_base_model:
+            raise ValueError(
+                f"Target modules {self.peft_config.target_modules} not found in the base model. "
+                f"Please check the target modules and try again."
+            )
+
+    def _get_submodules(self, key):
+        parent = self.model.get_submodule(".".join(key.split(".")[:-1]))
+        target_name = key.split(".")[-1]
+        target = self.model.get_submodule(key)
+        return parent, target, target_name
+
+    def _replace_module(self, parent_module, child_name, new_module, old_module):
+        setattr(parent_module, child_name, new_module)
+        new_module.weight = old_module.weight
+        if old_module.bias is not None:
+            new_module.bias = old_module.bias
+        if getattr(old_module, "state", None) is not None:
+            new_module.state = old_module.state
+            new_module.to(old_module.weight.device)
+
+        # dispatch to correct device
+        for name, module in new_module.named_modules():
+            if "delora_" in name:
+                module.to(old_module.weight.device)
+
+    def __getattr__(self, name: str):
+        """Forward missing attributes to the wrapped module."""
+        try:
+            return super().__getattr__(name)  # defer to nn.Module's logic
+        except AttributeError:
+            return getattr(self.model, name)
+
+    @property
+    def modules_to_save(self):
+        return None
+
+    def get_peft_config_as_dict(self, inference: bool = False):
+        config = {k: v.value if isinstance(v, Enum) else v for k, v in asdict(self.peft_config).items()}
+        if inference:
+            config["inference_mode"] = True
+        return config
+
+    def _set_adapter_layers(self, enabled=True):
+        for module in self.model.modules():
+            if isinstance(module, DeloraLayer):
+                module.disable_adapters = False if enabled else True
+
+    def enable_adapter_layers(self):
+        self._set_adapter_layers(enabled=True)
+
+    def disable_adapter_layers(self):
+        self._set_adapter_layers(enabled=False)
+
+
+# Below code is based on https://github.com/microsoft/LoRA/blob/main/loralib/layers.py
+# and modified to work with PyTorch FSDP
+
+
+#  ------------------------------------------------------------------------------------------
+#  Copyright (c) Microsoft Corporation. All rights reserved.
+#  Licensed under the MIT License (MIT). See LICENSE in the repo root for license information.
+#  ------------------------------------------------------------------------------------------
+
+
+# had to adapt it for `delora_only` to work
+def mark_only_delora_as_trainable(model: nn.Module, bias: str = "none") -> None:
+    for n, p in model.named_parameters():
+        if "delora_" not in n:
+            p.requires_grad = False
+    if bias == "none":
+        return
+    elif bias == "all":
+        for n, p in model.named_parameters():
+            if "bias" in n:
+                p.requires_grad = True
+    elif bias == "delora_only":
+        for m in model.modules():
+            if isinstance(m, DeloraLayer) and hasattr(m, "bias") and m.bias is not None:
+                m.bias.requires_grad = True
+    else:
+        raise NotImplementedError
+
+
+class DeloraLayer:
+    def __init__(
+        self,
+        r: int,
+        delora_lambda_value: int,
+        delora_dropout: float,
+        merge_weights: bool,
+    ):
+        self.r = r
+        self.delora_lambda_value = delora_lambda_value
+        # Optional dropout
+        if delora_dropout > 0.0:
+            self.delora_dropout = nn.Dropout(p=delora_dropout)
+        else:
+            self.delora_dropout = lambda x: x
+        # Mark the weight as unmerged
+        self.merged = False
+        self.merge_weights = merge_weights
+        self.disable_adapters = False
+
+
+class Linear(nn.Linear, DeloraLayer):
+    # Delora implemented in a dense layer
+    def __init__(
+        self,
+        in_features: int,
+        out_features: int,
+        r: int = 0,
+        delora_lambda: float = 1.,
+        delora_dropout: float = 0.0,
+        fan_in_fan_out: bool = False,  # Set this to True if the layer to replace stores weight like (fan_in, fan_out)
+        merge_weights: bool = False,
+        **kwargs,
+    ):
+        nn.Linear.__init__(self, in_features, out_features, **kwargs)
+        DeloraLayer.__init__(self, r=r, delora_lambda_value=delora_lambda, delora_dropout=delora_dropout, merge_weights=merge_weights)
+
+        self.fan_in_fan_out = fan_in_fan_out
+
+        if r > 0:
+            self.delora_A = nn.Linear(in_features, r, bias=False)
+            self.delora_B = nn.Linear(r, out_features, bias=False)
+
+            self.delora_lambda = nn.Parameter(torch.full((1,), delora_lambda), requires_grad=True)
+
+            # Frozen parameters
+            self.frozen_C = nn.Parameter(torch.empty_like(self.delora_A.weight).copy_(self.delora_A.weight))
+            self.frozen_C.requires_grad = False
+            self.frozen_D = nn.Parameter(torch.empty_like(self.delora_B.weight).copy_(self.delora_B.weight))
+            self.frozen_D.requires_grad = False
+
+            # Freezing the pre-trained weight matrix
+            self.weight.requires_grad = False
+
+        self.reset_parameters()
+        if fan_in_fan_out:
+            self.weight.data = self.weight.data.T
+
+    def reset_parameters(self):
+        nn.Linear.reset_parameters(self)
+        if hasattr(self, "delora_A"):
+            # initialize A the same way as the default for nn.Linear and B to zero
+            nn.init.kaiming_uniform_(self.delora_A.weight, a=math.sqrt(5))
+            nn.init.kaiming_uniform_(self.delora_B.weight, a=math.sqrt(5))
+            nn.init.constant_(self.delora_lambda, self.delora_lambda_value)
+            self.frozen_C.data = self.delora_A.weight.data
+            self.frozen_D.data = self.delora_B.weight.data
+
+    def get_ABCD(self):
+        # Get weights
+        delora_A_weight = self.delora_A.weight  # shape: (r, in_features)
+        delora_B_weight = self.delora_B.weight  # shape: (out_features, r)
+
+        # Get norms
+        delora_A_norm = delora_A_weight.norm(dim=1)  # shape: (r,)
+        delora_B_norm = delora_B_weight.norm(dim=0)  # shape: (r,)
+        frozen_C_norm = self.frozen_C.norm(dim=1)  # shape: (r,)
+        frozen_D_norm = self.frozen_D.norm(dim=0)  # shape: (r,)
+
+        # AB normalization
+        diag12 = torch.div(self.delora_lambda / self.r, torch.mul(delora_A_norm, delora_B_norm))
+        diag12 = torch.diag_embed(diag12)
+
+        diag34 = torch.div(self.delora_lambda / self.r, torch.mul(frozen_C_norm, frozen_D_norm))
+        diag34 = torch.diag_embed(diag34)
+
+        # Get ABCD
+        ABCD = delora_B_weight @ diag12 @ delora_A_weight
+        ABCD = ABCD - self.frozen_D @ diag34 @ self.frozen_C
+
+        # W scaling
+        Wnorm = self.weight.data.norm(dim=0)  # shape: (in_features,)
+        ABCD = torch.mul(ABCD, Wnorm.unsqueeze(0))  # shape: (out_features, in_features)
+
+        return ABCD
+
+    def train(self, mode: bool = True):
+        nn.Linear.train(self, mode)
+        self.delora_A.train(mode)
+        self.delora_B.train(mode)
+        self.delora_lambda.requires_grad = mode
+
+        if not mode and self.merge_weights and not self.merged:
+            # Merge the weights and mark it
+            if self.r > 0:
+                self.weight.data += self.get_ABCD().to(self.weight.device, dtype=self.weight.dtype)
+            self.merged = True
+        elif self.merge_weights and self.merged:
+            # Make sure that the weights are not merged
+            if self.r > 0:
+                self.weight.data -= self.get_ABCD()
+            self.merged = False
+
+    def eval(self):
+        nn.Linear.eval(self)
+        self.delora_A.eval()
+        self.delora_B.eval()
+
+    def forward(self, x: torch.Tensor):
+        previous_dtype = self.weight.dtype
+        if self.disable_adapters:
+            if self.r > 0 and self.merged:
+                self.weight.data -= self.get_ABCD()
+                self.merged = False
+
+            result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
+        elif self.r > 0 and not self.merged:
+            result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
+            if self.r > 0:
+                result += F.linear(self.delora_dropout(x), self.get_ABCD(), bias=None)
+        else:
+            result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
+
+        if result.dtype != previous_dtype:
+            result = result.to(previous_dtype)
+
+        return result
+
+
+class MergedLinear(nn.Linear, DeloraLayer):
+    # Delora implemented in a dense layer
+    def __init__(
+        self,
+        in_features: int,
+        out_features: int,
+        r: int = 0,
+        delora_lambda: int = 1,
+        delora_dropout: float = 0.0,
+        enable_delora: List[bool] = [False],
+        fan_in_fan_out: bool = False,
+        merge_weights: bool = True,
+        **kwargs,
+    ):
+        raise NotImplementedError
+
+
+if is_bnb_available():
+
+    class Linear8bitLt(bnb.nn.Linear8bitLt, DeloraLayer):
+        # Delora implemented in a dense layer
+        def __init__(
+            self,
+            in_features,
+            out_features,
+            r: int = 0,
+            delora_lambda: int = 1,
+            delora_dropout: float = 0.0,
+            Wdecompose: bool = False,
+            **kwargs,
+        ):
+            raise NotImplementedError
+
+    class MergedLinear8bitLt(bnb.nn.Linear8bitLt, DeloraLayer):
+        # Delora implemented in a dense layer
+        def __init__(
+            self,
+            in_features: int,
+            out_features: int,
+            r: int = 0,
+            delora_lambda: int = 1,
+            delora_dropout: float = 0.0,
+            enable_delora: List[bool] = [False],
+            **kwargs,
+        ):
+            raise NotImplementedError
diff --git a/docs/refs/orig_hra_layer.py b/docs/refs/orig_hra_layer.py
new file mode 100644
index 0000000..2f617bb
--- /dev/null
+++ b/docs/refs/orig_hra_layer.py
@@ -0,0 +1,420 @@
+# Copyright 2023-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import math
+import warnings
+from typing import Any, List, Optional, Set, Tuple
+
+import torch
+import torch.nn as nn
+
+from peft.tuners.lycoris_utils import LycorisLayer, check_adapters_to_merge
+
+
+class OFTLayer(nn.Module, LycorisLayer):
+    # All names of layers that may contain adapter weights
+    adapter_layer_names = ("oft_r",)
+
+    # other_param_names is defined on parent class
+
+    def __init__(self, base_layer: nn.Module):
+        super().__init__()
+        LycorisLayer.__init__(self, base_layer)
+
+        # OFT info
+        self.oft_r = nn.ParameterDict({})
+        self.coft = {}
+        self.eps = {}
+        self.block_share = {}
+
+    @property
+    def _available_adapters(self) -> Set[str]:
+        return {*self.oft_r}
+
+    def create_adapter_parameters(self, adapter_name: str, r: int, shape: Tuple[int, ...], block_share: bool):
+        # if block_share:
+        #     self.oft_r[adapter_name] = nn.Parameter(torch.empty(1, math.ceil(shape[0] / r), math.ceil(shape[0] / r)))
+        # else:
+        #     self.oft_r[adapter_name] = nn.Parameter(torch.empty(r, math.ceil(shape[0] / r), math.ceil(shape[0] / r)))
+        weight = getattr(self.get_base_layer(), "weight", None)
+        # self.oft_r[adapter_name] = nn.Parameter(torch.cat([weight.new_ones(r, r), weight.new_zeros(shape[0]-r, r)], dim=0))
+        self.oft_r[adapter_name] = nn.Parameter(
+            torch.cat([torch.eye(r, device=weight.device, dtype=weight.dtype),
+                       torch.zeros(shape[0] - r, r, device=weight.device, dtype=weight.dtype)], dim=0))
+
+    def reset_adapter_parameters(self, adapter_name: str):
+        # nn.init.zeros_(self.oft_r[adapter_name])
+        nn.init.kaiming_uniform_(self.oft_r[adapter_name], a=1 / self.eps[adapter_name])
+
+    def reset_adapter_parameters_random(self, adapter_name: str):
+        # nn.init.kaiming_uniform_(self.oft_r[adapter_name], a=math.sqrt(5))
+        nn.init.kaiming_uniform_(self.oft_r[adapter_name], a=1 / self.eps[adapter_name])
+
+    def update_layer(
+            self,
+            adapter_name: str,
+            r: int,
+            module_dropout: float,
+            init_weights: bool,
+            coft: bool = False,
+            eps: float = 6e-5,
+            block_share: bool = False,
+            **kwargs,
+    ) -> None:
+        """Internal function to create oft adapter
+
+        Args:
+            adapter_name (`str`): Name for the adapter to add.
+            r (`int`): Rank for the added adapter.
+            module_dropout (`float`): The dropout probability for disabling adapter during training.
+            init_weights (`bool`): Whether to initialize weights.
+            coft (`bool`): Whether to use the constrained variant of OFT or not.
+            eps (`float`):
+                The control strength of COFT. The freedom of rotation. Only has an effect if `coft` is set to True.
+            block_share (`bool`): Whether to share the OFT parameters between blocks or not.
+        """
+        if r <= 0:
+            raise ValueError(f"`r` should be a positive integer value but the value passed is {r}")
+
+        self.r[adapter_name] = r
+        self.module_dropout[adapter_name] = module_dropout
+        self.coft[adapter_name] = coft
+        self.block_share[adapter_name] = block_share
+
+        # Determine shape of OFT weights
+        base_layer = self.get_base_layer()
+        if isinstance(base_layer, nn.Linear):
+            shape = tuple(base_layer.weight.shape)
+        elif isinstance(base_layer, nn.Conv2d):
+            shape = (
+                base_layer.out_channels,
+                base_layer.in_channels * base_layer.kernel_size[0] * base_layer.kernel_size[1],
+            )
+        else:
+            raise TypeError(f"OFT is not implemented for base layers of type {type(base_layer).__name__}")
+
+        # self.eps[adapter_name] = eps * math.ceil(shape[0] / r) * math.ceil(shape[0] / r)
+        self.eps[adapter_name] = eps
+
+        # Create weights with provided shape
+        self.create_adapter_parameters(adapter_name, r, shape, block_share)
+
+        # Initialize weights
+        # if init_weights:
+        #     self.reset_adapter_parameters(adapter_name)
+        # else:
+        #     self.reset_adapter_parameters_random(adapter_name)
+
+        # Move new weights to device
+        weight = getattr(self.get_base_layer(), "weight", None)
+        if weight is not None:
+            # the layer is already completely initialized, this is an update
+            if weight.dtype.is_floating_point or weight.dtype.is_complex:
+                self.to(weight.device, dtype=weight.dtype)
+            else:
+                self.to(weight.device)
+        self.set_adapter(self.active_adapters)
+
+    def unscale_layer(self, scale=None) -> None:
+        # scale is not used
+        pass
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[List[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`List[str]`, *optional*):
+                The list of adapter names that should be merged. If `None`, all active adapters will be merged.
+                Defaults to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self._available_adapters:
+                base_layer = self.get_base_layer()
+
+                orig_weights = base_layer.weight.data
+                if isinstance(base_layer, nn.Linear):
+                    orig_weights = torch.transpose(orig_weights, 0, 1)
+                elif isinstance(base_layer, nn.Conv2d):
+                    orig_weights = orig_weights.view(
+                        [
+                            base_layer.out_channels,
+                            base_layer.in_channels * base_layer.kernel_size[0] * base_layer.kernel_size[1],
+                        ]
+                    )
+                    orig_weights = torch.transpose(orig_weights, 0, 1)
+                delta_weight = self.get_delta_weight(active_adapter)
+                if orig_weights.shape[1] != delta_weight.shape[1]:
+                    # when in channels is not divisible by r
+                    delta_weight = delta_weight[: orig_weights.shape[1], : orig_weights.shape[1]]
+                # delta_weight=delta_weight.to(orig_weights.device, dtype=orig_weights.dtype)
+                new_weights = torch.mm(orig_weights, delta_weight)
+                if isinstance(base_layer, nn.Linear):
+                    new_weights = torch.transpose(new_weights, 0, 1)
+                elif isinstance(base_layer, nn.Conv2d):
+                    new_weights = torch.transpose(new_weights, 0, 1)
+                    new_weights = new_weights.view(
+                        [
+                            base_layer.out_channels,
+                            base_layer.in_channels,
+                            base_layer.kernel_size[0],
+                            base_layer.kernel_size[1],
+                        ]
+                    )
+
+                if safe_merge and not torch.isfinite(new_weights).all():
+                    raise ValueError(
+                        f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                    )
+
+                base_layer.weight.data = new_weights
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self._available_adapters:
+                base_layer = self.get_base_layer()
+                new_weights = base_layer.weight.data
+                if isinstance(base_layer, nn.Linear):
+                    new_weights = torch.transpose(new_weights, 0, 1)
+                elif isinstance(base_layer, nn.Conv2d):
+                    new_weights = new_weights.view(
+                        [
+                            base_layer.out_channels,
+                            base_layer.in_channels * base_layer.kernel_size[0] * base_layer.kernel_size[1],
+                        ]
+                    )
+                    new_weights = torch.transpose(new_weights, 0, 1)
+                delta_weight = self.get_delta_weight(active_adapter)
+                if new_weights.shape[1] != delta_weight.shape[1]:
+                    # when in channels is not divisible by r
+                    delta_weight = delta_weight[: new_weights.shape[1], : new_weights.shape[1]]
+                delta_inv = torch.inverse(delta_weight)
+                orig_weights = torch.mm(new_weights, delta_inv)
+
+                if isinstance(base_layer, nn.Linear):
+                    orig_weights = torch.transpose(orig_weights, 0, 1)
+                elif isinstance(base_layer, nn.Conv2d):
+                    orig_weights = torch.transpose(orig_weights, 0, 1)
+                    orig_weights = orig_weights.reshape(
+                        [
+                            base_layer.out_channels,
+                            base_layer.in_channels,
+                            base_layer.kernel_size[0],
+                            base_layer.kernel_size[1],
+                        ]
+                    )
+                base_layer.weight.data = orig_weights
+
+    def get_delta_weight(self, adapter_name: str) -> torch.Tensor:
+        # rank = self.r[adapter_name]
+        # coft = self.coft[adapter_name]
+        # eps = self.eps[adapter_name]
+        # opt_r = self.oft_r[adapter_name]
+
+        # if coft:
+        #     with torch.no_grad():
+        #         opt_r.copy_(self._project_batch(opt_r, eps=eps))
+        #
+        # orth_rotate = self._cayley_batch(opt_r)
+        # weight = self._block_diagonal(orth_rotate, rank)
+        rank = self.r[adapter_name]
+        hrft_v = self.oft_r[adapter_name]
+        in_features = self.oft_r[adapter_name].size(0)
+        device = self.oft_r[adapter_name].device
+        dtype = self.oft_r[adapter_name].dtype
+
+        # unit_v_list = [hrft_v[:, i].view(-1,1) / (torch.sqrt(torch.sum(hrft_v[:,i] ** 2) + self.eps[adapter_name])) for i in range(8)]
+
+        # weight = torch.eye(in_features, device=device, dtype=dtype)
+        # for unit_v in unit_v_list:
+        #     weight = torch.mm(weight, torch.eye(in_features, device=device, dtype=dtype) - 2 * unit_v @ unit_v.t())
+
+        U_list = []
+        U_list.append((hrft_v[:, 0] / hrft_v[:, 0].norm()).view(-1, 1))
+        for i in range(1, rank):
+            Ui = hrft_v[:, i].view(-1, 1)
+            for j in range(i):
+                Ui = Ui - (U_list[j].t() @ Ui) * U_list[j]
+            U_list.append((Ui / Ui.norm()).view(-1, 1))
+        U_list = torch.cat(U_list, dim=1)
+        weight = torch.eye(in_features, device=device, dtype=dtype) - 2 * U_list @ U_list.t()
+        # weight = torch.eye(in_features, device=device) - 2 * U_list @ U_list.t()
+
+        return weight
+
+    # Copied from https://github.com/Zeju1997/oft/blob/84cebb965df69781e3d9c3c875f5980b421eaf24/oft-control/oft.py#L144
+    def _cayley_batch(self, data: torch.Tensor) -> torch.Tensor:
+        b, r, c = data.shape
+        # Ensure the input matrix is skew-symmetric
+        skew = 0.5 * (data - data.transpose(1, 2))
+        I = torch.eye(r, device=data.device).unsqueeze(0).expand(b, r, c)  # noqa: E741
+
+        # Perform the Cayley parametrization
+        Q = torch.bmm(I - skew, torch.inverse(I + skew))
+
+        return Q
+
+    # Copied from https://github.com/Zeju1997/oft/blob/84cebb965df69781e3d9c3c875f5980b421eaf24/oft-control/oft.py#L155
+    def _block_diagonal(self, oft_r: torch.Tensor, rank: int) -> torch.Tensor:
+        if oft_r.shape[0] == 1:
+            # block share
+            blocks = [oft_r[0, ...] for i in range(rank)]
+        else:
+            blocks = [oft_r[i, ...] for i in range(rank)]
+
+        # Use torch.block_diag to create the block diagonal matrix
+        A = torch.block_diag(*blocks)
+
+        return A
+
+    # Copied from https://github.com/Zeju1997/oft/blob/84cebb965df69781e3d9c3c875f5980b421eaf24/oft-control/oft.py#L52
+    def _project_batch(self, oft_r, eps=1e-5):
+        # scaling factor for each of the smaller block matrix
+        eps = eps * 1 / torch.sqrt(torch.tensor(oft_r.shape[0]))
+        I = (  # noqa: E741
+            torch.zeros((oft_r.size(1), oft_r.size(1)), device=oft_r.device, dtype=oft_r.dtype)
+            .unsqueeze(0)
+            .expand_as(oft_r)
+        )
+        diff = oft_r - I
+        norm_diff = torch.norm(oft_r - I, dim=(1, 2), keepdim=True)
+        mask = (norm_diff <= eps).bool()
+        out = torch.where(mask, oft_r, I + eps * (diff / norm_diff))
+        return out
+
+    def forward(self, x: torch.Tensor, *args, **kwargs) -> torch.Tensor:
+        previous_dtype = x.dtype
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            result = self.base_layer(x, *args, **kwargs)
+            if len(result.shape) == 4:
+                result = result.permute(0, 2, 3, 1)
+
+            base_layer = self.get_base_layer()
+            base_bias = base_layer.bias
+            if base_bias is not None:
+                # Bias should be added after OFT forward
+                result = result - base_bias.data
+
+            # Execute all the adapters
+            for active_adapter in self.active_adapters:
+                if active_adapter not in self._available_adapters:
+                    continue
+
+                module_dropout = self.module_dropout[active_adapter]
+
+                # Modify current execution weights
+                if (not self.training) or (self.training and torch.rand(1) > module_dropout):
+                    result = self._get_delta_activations(active_adapter, result, *args, **kwargs)
+
+            if base_bias is not None:
+                result = result + base_bias.data
+            if len(result.shape) == 4:
+                result = result.permute(0, 3, 1, 2)
+
+        result = result.to(previous_dtype)
+        return result
+
+
+class Linear(OFTLayer):
+    """OFT implemented in Linear layer"""
+
+    def __init__(
+            self,
+            base_layer: nn.Module,
+            adapter_name: str = "default",
+            r: int = 0,
+            module_dropout: float = 0.0,
+            init_weights: bool = True,
+            **kwargs,
+    ):
+        super().__init__(base_layer)
+
+        # Create adapter and set it active
+        self._active_adapter = adapter_name
+        self.update_layer(adapter_name, r, module_dropout, init_weights, **kwargs)
+
+    def _get_delta_activations(
+            self, adapter_name: str, input: torch.Tensor, *args: Any, **kwargs: Any
+    ) -> torch.Tensor:
+        delta_weight = self.get_delta_weight(adapter_name)
+
+        base_layer = self.get_base_layer()
+        base_weight = base_layer.weight.data
+        delta_weight = delta_weight[: base_weight.shape[0], : base_weight.shape[0]]
+
+        # don't add bias here, because the bias will be added after OFT forward
+        return torch.matmul(input, delta_weight)
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "oft." + rep
+
+
+class Conv2d(OFTLayer):
+    """OFT implemented in Conv2d layer"""
+
+    def __init__(
+            self,
+            base_layer: nn.Module,
+            adapter_name: str = "default",
+            r: int = 0,
+            module_dropout: float = 0.0,
+            init_weights: bool = True,
+            **kwargs,
+    ):
+        super().__init__(base_layer)
+
+        # Create adapter and set it active
+        self._active_adapter = adapter_name
+        self.update_layer(adapter_name, r, module_dropout, init_weights, **kwargs)
+
+    def _get_delta_activations(
+            self, adapter_name: str, input: torch.Tensor, *args: Any, **kwargs: Any
+    ) -> torch.Tensor:
+        delta_weight = self.get_delta_weight(adapter_name)
+
+        base_layer = self.get_base_layer()
+        base_weight = base_layer.weight.data
+        delta_weight = delta_weight[: base_weight.shape[0], : base_weight.shape[0]]
+
+        # don't add bias here, because the bias will be added after OFT forward
+        return torch.matmul(input, delta_weight)
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "oft." + rep
diff --git a/docs/refs/orig_pissa_init.py b/docs/refs/orig_pissa_init.py
new file mode 100644
index 0000000..9b1797d
--- /dev/null
+++ b/docs/refs/orig_pissa_init.py
@@ -0,0 +1,60 @@
+# Copyright 2023-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import torch
+import os
+from peft import LoraConfig, get_peft_model
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import argparse
+
+parser = argparse.ArgumentParser(description="Separate the principal singular value and singular vectors from base model")
+parser.add_argument("--base_model_path", type=str, required=True, help="The name or path of the base model.")
+parser.add_argument("--output_dir", type=str, required=True)
+parser.add_argument("--bits", type=str, default="bf16", choices=["bf16", "fp16", "fp32"])
+parser.add_argument("--init_weights", type=str, default="pissa", help="(`['pissa', 'pissa_niter_[number of iters]']`)")
+parser.add_argument("--lora_r", type=int, default=128)
+parser.add_argument("--lora_alpha", type=int, default=128)
+parser.add_argument("--lora_dropout", type=float, default=0)
+parser.add_argument('--target_modules', nargs='+', help='', required=True)
+script_args = parser.parse_args()
+print(script_args)
+
+model = AutoModelForCausalLM.from_pretrained(
+    script_args.base_model_path,
+    torch_dtype=(
+        torch.float16
+        if script_args.bits == "fp16"
+        else (torch.bfloat16 if script_args.bits == "bf16" else torch.float32)
+    ),
+    device_map="auto",
+)
+tokenizer = AutoTokenizer.from_pretrained(script_args.base_model_path)
+tokenizer.pad_token_id = tokenizer.eos_token_id
+lora_config = LoraConfig(
+    r=script_args.lora_r,
+    lora_alpha=script_args.lora_alpha,
+    init_lora_weights=True if script_args.init_weights=="True" else script_args.init_weights,
+    lora_dropout=script_args.lora_dropout,
+    target_modules=script_args.target_modules,
+)
+peft_model = get_peft_model(model, lora_config)
+
+# Save PiSSA modules:
+peft_model.peft_config["default"].init_lora_weights = True
+peft_model.save_pretrained(os.path.join(script_args.output_dir, "pissa_init"))
+# Save residual model:
+peft_model = peft_model.unload()
+peft_model.save_pretrained(script_args.output_dir)
+# Save the tokenizer:
+tokenizer.save_pretrained(script_args.output_dir)
\ No newline at end of file
diff --git a/docs/refs/peft_delora_layer.py b/docs/refs/peft_delora_layer.py
new file mode 100644
index 0000000..59ce8e1
--- /dev/null
+++ b/docs/refs/peft_delora_layer.py
@@ -0,0 +1,274 @@
+# Copyright 2025-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import annotations
+
+import math
+import warnings
+from typing import Any, Optional
+
+import torch
+import torch.nn as nn
+
+from peft.tuners._buffer_dict import BufferDict
+from peft.tuners.tuners_utils import BaseTunerLayer, check_adapters_to_merge
+
+from .config import DeloraConfig
+
+
+class DeloraLayer(BaseTunerLayer):
+    # All names of layers that may contain (trainable) adapter weights
+    adapter_layer_names = (
+        "delora_A",
+        "delora_B",
+        "delora_lambda",
+    )
+    # All names of other parameters that may contain adapter-related parameters
+    other_param_names = (
+        "r",
+        "delora_dropout",
+        "delora_w_norm",
+    )
+
+    def __init__(self, base_layer: nn.Module, **kwargs) -> None:
+        self.base_layer = base_layer
+        self.r = {}
+        self.delora_dropout = nn.ModuleDict({})
+        self.delora_A = nn.ParameterDict({})
+        self.delora_B = nn.ParameterDict({})
+        self.delora_lambda = nn.ParameterDict({})
+        # Use persistent buffers so they are included in state_dict and saved.
+        self.delora_w_norm = BufferDict({}, persistent=True)
+        # Mark the weight as unmerged
+        self._disable_adapters = False
+        self.merged_adapters = []
+        self.kwargs = kwargs
+
+        base_layer_mod = self.get_base_layer()
+        if isinstance(base_layer_mod, nn.Linear):
+            self.in_features, self.out_features = base_layer_mod.in_features, base_layer_mod.out_features
+        else:
+            raise ValueError(f"Unsupported layer type {type(base_layer_mod)}")
+
+    @staticmethod
+    def _compute_delta(
+        A: torch.Tensor, B: torch.Tensor, delora_lambda: torch.Tensor, r: int, w_norm: torch.Tensor
+    ) -> torch.Tensor:
+        """Compute delta = B @ diag(delora_lambda/r / (||A_i||*||B^j||)) @ A, scaled by provided w_norm (per-input channel)"""
+        An = torch.clamp(A.norm(dim=1), min=1e-4)
+        Bn = torch.clamp(B.norm(dim=0), min=1e-4)
+        diag = torch.diag_embed(delora_lambda / r / (An * Bn))
+        delta = B @ diag @ A
+        delta = delta * w_norm.unsqueeze(0)
+        return delta
+
+    def get_delta_weight(self, adapter: str) -> torch.Tensor:
+        if adapter not in self.delora_A or adapter not in self.delora_B:
+            raise ValueError(f"Adapter {adapter} not found.")
+
+        delta = self._compute_delta(
+            self.delora_A[adapter],
+            self.delora_B[adapter],
+            self.delora_lambda[adapter],
+            self.r[adapter],
+            self.delora_w_norm[adapter],
+        )
+        return delta
+
+    def update_layer(
+        self,
+        adapter_name: str,
+        r: int,
+        delora_lambda: float,
+        config: DeloraConfig,
+        **kwargs: Any,
+    ) -> None:
+        """Internal function to create delora adapter
+
+        Args:
+            adapter_name (`str`): Name for the adapter to add.
+            r (`int`): Rank for the added adapter.
+            delora_lambda (`float`): Boundary for the adapter's norm.
+            config (`DeloraConfig`): The adapter configuration for this layer.
+        """
+        module_dropout = config.module_dropout
+        init_weights = config.init_weights
+        inference_mode = config.inference_mode
+
+        if r <= 0:
+            raise ValueError(f"`r` should be a positive integer value but the value passed is {r}")
+
+        self.r[adapter_name] = r
+        self.delora_A[adapter_name] = nn.Parameter(torch.empty(r, self.in_features))
+        self.delora_B[adapter_name] = nn.Parameter(torch.empty(self.out_features, r))
+        self.delora_lambda[adapter_name] = nn.Parameter(torch.empty(1))
+        if module_dropout > 0.0:
+            module_dropout_layer = nn.Dropout(p=module_dropout)
+        else:
+            module_dropout_layer = nn.Identity()
+        self.delora_dropout.update(nn.ModuleDict({adapter_name: module_dropout_layer}))
+
+        # Initialize weights
+        self.reset_delora_parameters(adapter_name, init_weights, delora_lambda)
+
+        # Move new weights to device
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+    def reset_delora_parameters(
+        self,
+        adapter_name: str,
+        init_weights: bool = True,
+        delora_lambda: float = 15.0,
+    ) -> None:
+        if adapter_name not in self.delora_A.keys():
+            return
+
+        if init_weights is True:
+            nn.init.kaiming_uniform_(self.delora_A[adapter_name], a=math.sqrt(5))
+            nn.init.zeros_(self.delora_B[adapter_name])
+        else:
+            nn.init.kaiming_uniform_(self.delora_A[adapter_name], a=math.sqrt(5))
+            nn.init.kaiming_uniform_(self.delora_B[adapter_name], a=math.sqrt(5))
+
+        self.delora_lambda[adapter_name].data.fill_(float(delora_lambda))
+
+        # capture a fixed norm for this adapter to use for future delta computations
+        with torch.no_grad():
+            w = self.get_base_layer().weight
+            if w.device.type != "meta":
+                w_norm = torch.norm(w.data, dim=0).detach()
+            else:
+                # For meta tensors, we can't compute the norm, so use a default value
+                w_norm = torch.ones(w.shape[1], device=w.device)
+            self.delora_w_norm[adapter_name] = w_norm
+
+
+class DeloraLinear(nn.Module, DeloraLayer):
+    # DeLoRA implemented in a dense layer
+    def __init__(
+        self,
+        base_layer,
+        adapter_name: str,
+        config: DeloraConfig,
+        r: int,
+        delora_lambda: float,
+        **kwargs,
+    ) -> None:
+        super().__init__()
+        DeloraLayer.__init__(self, base_layer, **kwargs)
+        self._active_adapter = adapter_name
+        self.update_layer(adapter_name, r, delora_lambda, config=config)
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`list[str]`, *optional*):
+                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
+                to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.delora_A.keys():
+                base_layer = self.get_base_layer()
+                delta_weight = (
+                    self.get_delta_weight(active_adapter)
+                    .detach()
+                    .to(dtype=base_layer.weight.dtype, device=base_layer.weight.device)
+                )
+                with torch.no_grad():
+                    if safe_merge:
+                        orig_weights = base_layer.weight.data.clone()
+                        orig_weights = orig_weights + delta_weight
+
+                        if not torch.isfinite(orig_weights).all():
+                            raise ValueError(
+                                f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                            )
+
+                        base_layer.weight.data = orig_weights
+                    else:
+                        base_layer.weight.data.add_(delta_weight)
+
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        Unmerge all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self.delora_A.keys():
+                self.get_base_layer().weight.data -= self.get_delta_weight(active_adapter)
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        previous_dtype = x.dtype
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            if not self.active_adapters:
+                return self.base_layer(x, *args, **kwargs).to(previous_dtype)
+
+            base_out = self.base_layer(x, *args, **kwargs)
+            add_out = torch.zeros_like(base_out)
+
+            for adapter in self.active_adapters:
+                if adapter not in self.delora_A:
+                    continue
+
+                x_d = self.delora_dropout[adapter](x)
+
+                # Decomposed delta calculation
+                # 1. (x * w_norm) @ A.T
+                h = nn.functional.linear(x_d * self.delora_w_norm[adapter], self.delora_A[adapter])
+
+                # 2. h @ diag
+                An = torch.clamp(self.delora_A[adapter].norm(dim=1), min=1e-4)
+                Bn = torch.clamp(self.delora_B[adapter].norm(dim=0), min=1e-4)
+                scaling = (self.delora_lambda[adapter] / self.r[adapter]) / (An * Bn)
+
+                h = h * scaling
+
+                # 3. h @ B.T
+                h = nn.functional.linear(h, self.delora_B[adapter])
+
+                add_out += h
+
+            result = base_out + add_out.to(base_out.dtype)
+
+        result = result.to(previous_dtype)
+        return result
+
+    def supports_lora_conversion(self, adapter_name: str = "default") -> bool:
+        return True
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "delora." + rep
diff --git a/docs/refs/peft_hra_layer.py b/docs/refs/peft_hra_layer.py
new file mode 100644
index 0000000..e24e28a
--- /dev/null
+++ b/docs/refs/peft_hra_layer.py
@@ -0,0 +1,462 @@
+# Copyright 2024-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import math
+import warnings
+from typing import Any, Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from peft.tuners.tuners_utils import BaseTunerLayer, check_adapters_to_merge
+
+from .config import HRAConfig
+
+
+class HRALayer(BaseTunerLayer):
+    # All names of layers that may contain (trainable) adapter weights
+    adapter_layer_names = ("hra_u",)
+    # All names of other parameters that may contain adapter-related parameters
+    other_param_names = ("hra_r", "hra_apply_GS")
+
+    def __init__(self, base_layer: nn.Module, **kwargs) -> None:
+        self.base_layer = base_layer
+        self.hra_r = {}
+        self.hra_apply_GS = {}
+        self.hra_u = nn.ParameterDict({})
+        # Mark the weight as unmerged
+        self._disable_adapters = False
+        self.merged_adapters = []
+        # flag to enable/disable casting of input to weight dtype during forward call
+        self.cast_input_dtype_enabled = True
+        self.kwargs = kwargs
+
+        base_layer = self.get_base_layer()
+        if isinstance(base_layer, nn.Linear):
+            self.in_features, self.out_features = base_layer.in_features, base_layer.out_features
+        elif isinstance(base_layer, nn.Conv2d):
+            self.in_features, self.out_features = base_layer.in_channels, base_layer.out_channels
+        else:
+            raise ValueError(f"Unsupported layer type {type(base_layer)}")
+
+    def update_layer(
+        self,
+        adapter_name: str,
+        r: int,
+        config: HRAConfig,
+        **kwargs,
+    ) -> None:
+        """Internal function to create hra adapter
+
+        Args:
+            adapter_name (`str`): Name for the adapter to add.
+            r (`int`): Rank for the added adapter.
+            config (`HRAConfig`): The adapter configuration for this layer.
+        """
+        apply_GS = config.apply_GS
+        init_weights = config.init_weights
+        inference_mode = config.inference_mode
+
+        if r <= 0:
+            raise ValueError(f"`r` should be a positive integer value but the value passed is {r}")
+
+        self.hra_r[adapter_name] = r
+        self.hra_apply_GS[adapter_name] = apply_GS
+
+        # Determine shape of HRA weights
+        base_layer = self.get_base_layer()
+        if isinstance(base_layer, nn.Linear):
+            self.hra_u[adapter_name] = nn.Parameter(torch.empty(self.in_features, r), requires_grad=True)
+        elif isinstance(base_layer, nn.Conv2d):
+            self.hra_u[adapter_name] = nn.Parameter(
+                torch.empty(self.in_features * base_layer.kernel_size[0] * base_layer.kernel_size[0], r),
+                requires_grad=True,
+            )
+        else:
+            raise TypeError(f"HRA is not implemented for base layers of type {type(base_layer).__name__}")
+
+        # Initialize weights
+        if init_weights:
+            self.reset_hra_parameters(adapter_name)
+        else:
+            self.reset_hra_parameters_random(adapter_name)
+
+        # Move new weights to device
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+    def reset_hra_parameters(self, adapter_name: str):
+        if self.hra_r[adapter_name] % 2 != 0:
+            warnings.warn("The symmetric initialization can NOT be performed when r is odd!")
+            nn.init.kaiming_uniform_(self.hra_u[adapter_name], a=math.sqrt(5))
+        else:
+            shape = self.hra_u[adapter_name].shape
+            half_u = torch.zeros(shape[0], shape[1] // 2)
+            nn.init.kaiming_uniform_(half_u, a=math.sqrt(5))
+            self.hra_u[adapter_name] = nn.Parameter(torch.repeat_interleave(half_u, 2, dim=1))
+
+    def reset_hra_parameters_random(self, adapter_name: str):
+        nn.init.kaiming_uniform_(self.hra_u[adapter_name], a=math.sqrt(5))
+
+    def scale_layer(self, scale: float) -> None:
+        if scale == 1:
+            return
+
+        for active_adapter in self.active_adapters:
+            if active_adapter not in self.hra_u.keys():
+                continue
+
+            warnings.warn("Scaling operation for HRA not supported! Automatically set scale to 1.")
+
+    def unscale_layer(self, scale=None) -> None:
+        for active_adapter in self.active_adapters:
+            if active_adapter not in self.hra_u.keys():
+                continue
+
+            warnings.warn("Unscaling operation for HRA not supported! Keeping scale at 1.")
+
+
+class HRALinear(nn.Module, HRALayer):
+    """
+    HRA implemented in a dense layer.
+    """
+
+    def __init__(
+        self,
+        base_layer,
+        adapter_name: str,
+        config: HRAConfig,
+        r: int = 0,
+        **kwargs,
+    ) -> None:
+        super().__init__()
+        HRALayer.__init__(self, base_layer, **kwargs)
+        self._active_adapter = adapter_name
+        self.update_layer(adapter_name, r, config=config, **kwargs)
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`List[str]`, *optional*):
+                The list of adapter names that should be merged. If `None`, all active adapters will be merged.
+                Defaults to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.hra_u.keys():
+                base_layer = self.get_base_layer()
+                orig_dtype = base_layer.weight.dtype
+                if safe_merge:
+                    # Note that safe_merge will be slower than the normal merge
+                    # because of the copy operation.
+                    orig_weight = base_layer.weight.data.clone()
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    orig_weight = torch.mm(orig_weight.to(delta_weight.dtype), delta_weight)
+
+                    if not torch.isfinite(orig_weight).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    base_layer.weight.data = orig_weight.to(orig_dtype)
+                else:
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    new_weight = torch.mm(base_layer.weight.data.to(delta_weight.dtype), delta_weight)
+                    base_layer.weight.data = new_weight.to(orig_dtype)
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            base_layer = self.get_base_layer()
+            orig_dtype = base_layer.weight.dtype
+            if active_adapter in self.hra_u.keys():
+                orig_weight = base_layer.weight.data.clone()
+                delta_weight = self.get_delta_weight(active_adapter, reverse=True)
+                new_weight = torch.mm(orig_weight.to(delta_weight.dtype), delta_weight)
+                base_layer.weight.data = new_weight.to(orig_dtype)
+
+    def get_delta_weight(self, adapter_name: str, reverse: bool = False) -> torch.Tensor:
+        rank = self.hra_r[adapter_name]
+        apply_GS = self.hra_apply_GS[adapter_name]
+        opt_u = self.hra_u[adapter_name]
+        shape = opt_u.shape
+
+        if apply_GS:
+            weight = [(opt_u[:, 0] / opt_u[:, 0].norm()).view(-1, 1)]
+            for i in range(1, rank):
+                ui = opt_u[:, i].view(-1, 1)
+                for j in range(i):
+                    ui = ui - (weight[j].t() @ ui) * weight[j]
+                weight.append((ui / ui.norm()).view(-1, 1))
+            weight = torch.cat(weight, dim=1)
+            weight = torch.eye(shape[0], device=opt_u.device, dtype=opt_u.dtype) - 2 * weight @ weight.t()
+
+        else:
+            opt_u = opt_u / opt_u.norm(dim=0)
+            weight = torch.eye(shape[0], device=opt_u.device, dtype=opt_u.dtype)
+            if reverse:
+                indices = range(rank - 1, -1, -1)
+            else:
+                indices = range(rank)
+
+            for i in indices:
+                ui = opt_u[:, i].view(-1, 1)
+                weight = weight - 2 * weight @ ui @ ui.t()
+
+        return weight
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        previous_dtype = x.dtype
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            new_weight = torch.eye(self.in_features, device=x.device)
+
+            for active_adapter in self.active_adapters:
+                if active_adapter not in self.hra_u.keys():
+                    continue
+                delta_weight = self.get_delta_weight(active_adapter)
+                new_weight = torch.mm(new_weight.to(delta_weight.dtype), delta_weight)
+
+            orig_weight = self.get_base_layer().weight.data
+            orig_weight = self._cast_input_dtype(orig_weight, new_weight.dtype)
+            new_weight = torch.mm(orig_weight, new_weight)
+            bias = self._cast_input_dtype(self.base_layer.bias, new_weight.dtype)
+
+            if self.cast_input_dtype_enabled:
+                x = self._cast_input_dtype(x, new_weight.dtype)
+            else:
+                x = x.to(self.get_base_layer().weight.data.dtype)
+            result = F.linear(input=x, weight=new_weight, bias=bias)
+
+        result = result.to(previous_dtype)
+        return result
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "hra." + rep
+
+
+class HRAConv2d(nn.Module, HRALayer):
+    """HRA implemented in Conv2d layer"""
+
+    def __init__(
+        self,
+        base_layer,
+        adapter_name: str,
+        config: HRAConfig,
+        r: int = 0,
+        **kwargs,
+    ):
+        super().__init__()
+        HRALayer.__init__(self, base_layer)
+        self._active_adapter = adapter_name
+        self.update_layer(adapter_name, r, config=config, **kwargs)
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`List[str]`, *optional*):
+                The list of adapter names that should be merged. If `None`, all active adapters will be merged.
+                Defaults to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.hra_u.keys():
+                base_layer = self.get_base_layer()
+                orig_dtype = base_layer.weight.dtype
+                if safe_merge:
+                    # Note that safe_merge will be slower than the normal merge
+                    # because of the copy operation.
+                    orig_weight = base_layer.weight.data.clone()
+                    orig_weight = orig_weight.view(
+                        self.out_features,
+                        self.in_features * base_layer.kernel_size[0] * self.base_layer.kernel_size[0],
+                    )
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    orig_weight = torch.mm(orig_weight.to(delta_weight.dtype), delta_weight)
+                    orig_weight = orig_weight.view(
+                        self.out_features,
+                        self.in_features,
+                        base_layer.kernel_size[0],
+                        base_layer.kernel_size[0],
+                    )
+
+                    if not torch.isfinite(orig_weight).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    base_layer.weight.data = orig_weight.to(orig_dtype)
+                else:
+                    orig_weight = base_layer.weight.data
+                    orig_weight = orig_weight.view(
+                        self.out_features,
+                        self.in_features * self.base_layer.kernel_size[0] * self.base_layer.kernel_size[0],
+                    )
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    orig_weight = torch.mm(orig_weight.to(delta_weight.dtype), delta_weight)
+                    orig_weight = orig_weight.view(
+                        self.out_features,
+                        self.in_features,
+                        base_layer.kernel_size[0],
+                        base_layer.kernel_size[0],
+                    )
+
+                    base_layer.weight.data = orig_weight.to(orig_dtype)
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            base_layer = self.get_base_layer()
+            orig_dtype = base_layer.weight.dtype
+            if active_adapter in self.hra_u.keys():
+                orig_weight = base_layer.weight.data.clone()
+                orig_weight = orig_weight.view(
+                    self.out_features,
+                    self.in_features * base_layer.kernel_size[0] * base_layer.kernel_size[0],
+                )
+                delta_weight = self.get_delta_weight(active_adapter, reverse=True)
+                orig_weight = torch.mm(orig_weight.to(delta_weight.dtype), delta_weight)
+                orig_weight = orig_weight.view(
+                    self.out_features, self.in_features, base_layer.kernel_size[0], base_layer.kernel_size[0]
+                )
+
+                base_layer.weight.data = orig_weight.to(orig_dtype)
+
+    def get_delta_weight(self, adapter_name: str, reverse: bool = False) -> torch.Tensor:
+        rank = self.hra_r[adapter_name]
+        apply_GS = self.hra_apply_GS[adapter_name]
+        opt_u = self.hra_u[adapter_name]
+        shape = opt_u.shape
+
+        if apply_GS:
+            weight = [(opt_u[:, 0] / opt_u[:, 0].norm()).view(-1, 1)]
+            for i in range(1, rank):
+                ui = opt_u[:, i].view(-1, 1)
+                for j in range(i):
+                    ui = ui - (weight[j].t() @ ui) * weight[j]
+                weight.append((ui / ui.norm()).view(-1, 1))
+            weight = torch.cat(weight, dim=1)
+            weight = torch.eye(shape[0], device=opt_u.device, dtype=opt_u.dtype) - 2 * weight @ weight.t()
+
+        else:
+            opt_u = opt_u / opt_u.norm(dim=0)
+            weight = torch.eye(shape[0], device=opt_u.device, dtype=opt_u.dtype)
+            if reverse:
+                indices = range(rank - 1, -1, -1)
+            else:
+                indices = range(rank)
+
+            for i in indices:
+                ui = opt_u[:, i].view(-1, 1)
+                weight = weight - 2 * weight @ ui @ ui.t()
+
+        return weight
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        previous_dtype = x.dtype
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            new_weight = torch.eye(
+                self.in_features * self.base_layer.kernel_size[0] * self.base_layer.kernel_size[0],
+                device=x.device,
+            )
+            for active_adapter in self.active_adapters:
+                if active_adapter not in self.hra_u.keys():
+                    continue
+                delta_weight = self.get_delta_weight(active_adapter)
+                new_weight = torch.mm(new_weight.to(delta_weight.dtype), delta_weight)
+
+            orig_weight = self.base_layer.weight.data
+            orig_weight = orig_weight.view(
+                self.out_features,
+                self.in_features * self.base_layer.kernel_size[0] * self.base_layer.kernel_size[0],
+            )
+            orig_weight = self._cast_input_dtype(orig_weight, new_weight.dtype)
+            bias = self._cast_input_dtype(self.base_layer.bias, new_weight.dtype)
+
+            new_weight = torch.mm(orig_weight, new_weight)
+            new_weight = new_weight.view(
+                self.out_features,
+                self.in_features,
+                self.base_layer.kernel_size[0],
+                self.base_layer.kernel_size[0],
+            )
+
+            if self.cast_input_dtype_enabled:
+                x = self._cast_input_dtype(x, new_weight.dtype)
+            else:
+                x = x.to(self.get_base_layer().weight.data.dtype)
+            result = F.conv2d(
+                input=x,
+                weight=new_weight,
+                bias=bias,
+                padding=self.base_layer.padding[0],
+                stride=self.base_layer.stride[0],
+            )
+
+        result = result.to(previous_dtype)
+        return result
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "hra." + rep
diff --git a/docs/refs/peft_ia3_layer.py b/docs/refs/peft_ia3_layer.py
new file mode 100644
index 0000000..f2aff56
--- /dev/null
+++ b/docs/refs/peft_ia3_layer.py
@@ -0,0 +1,336 @@
+# Copyright 2023-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import warnings
+from typing import Any, Optional
+
+import torch
+import torch.nn as nn
+from transformers.pytorch_utils import Conv1D
+
+from peft.tuners.tuners_utils import BaseTunerLayer, check_adapters_to_merge
+from peft.utils import transpose
+
+from .config import IA3Config
+
+
+class IA3Layer(BaseTunerLayer):
+    # All names of layers that may contain adapter weights
+    adapter_layer_names = ("ia3_l",)
+
+    def __init__(self, base_layer: nn.Module, is_feedforward: bool, **kwargs) -> None:
+        self.base_layer = base_layer
+        self.ia3_l = nn.ParameterDict({})
+        # Mark the weight as unmerged
+        self._disable_adapters = False
+        self.merged_adapters = []
+        self.is_feedforward = is_feedforward
+
+        base_layer = self.get_base_layer()
+        if isinstance(base_layer, nn.Linear):
+            in_features, out_features = base_layer.in_features, base_layer.out_features
+        elif isinstance(base_layer, (nn.Conv2d, nn.Conv3d)):
+            in_features, out_features = base_layer.in_channels, base_layer.out_channels
+        elif isinstance(base_layer, nn.Embedding):
+            in_features, out_features = base_layer.num_embeddings, base_layer.embedding_dim
+        elif isinstance(base_layer, Conv1D):
+            in_features, out_features = (
+                base_layer.weight.ds_shape if hasattr(base_layer.weight, "ds_shape") else base_layer.weight.shape
+            )
+        else:
+            raise ValueError(f"Unsupported layer type {type(base_layer)}")
+        self.in_features = in_features
+        self.out_features = out_features
+
+    def update_layer(self, adapter_name: str, config: IA3Config, **kwargs):
+        init_ia3_weights = config.init_ia3_weights
+        inference_mode = config.inference_mode
+
+        # This code works for linear layers, override for other layer types
+        # Actual trainable parameters
+        if self.is_feedforward:
+            weight = torch.randn((1, self.in_features))
+        else:
+            weight = torch.randn((self.out_features, 1))
+        self.ia3_l[adapter_name] = nn.Parameter(weight)
+        if init_ia3_weights:
+            self.reset_ia3_parameters(adapter_name)
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+    def reset_ia3_parameters(self, adapter_name):
+        if adapter_name in self.ia3_l.keys():
+            # initialize learned vector with torch.ones
+            nn.init.constant_(self.ia3_l[adapter_name], 1.0)
+
+
+class Linear(nn.Module, IA3Layer):
+    # (IA)^3 implemented in a dense layer
+    def __init__(
+        self,
+        base_layer: nn.Module,
+        adapter_name: str,
+        config: IA3Config,
+        is_feedforward: bool = False,  # Set to True if the layer is treated as a feedforward layer
+        is_target_conv_1d_layer: bool = False,  # whether target module is a conv1d layer. useful while unloading later
+        **kwargs,
+    ) -> None:
+        super().__init__()
+        IA3Layer.__init__(self, base_layer, is_feedforward=is_feedforward)
+        self.fan_in_fan_out = config.fan_in_fan_out
+        self.is_target_conv_1d_layer = is_target_conv_1d_layer
+        self._active_adapter = adapter_name
+        self.update_layer(adapter_name, config=config)
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`List[str]`, *optional*):
+                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
+                to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.ia3_l.keys():
+                base_layer = self.get_base_layer()
+                ia3_l = transpose(self.ia3_l[active_adapter].data, self.fan_in_fan_out)
+                orig_dtype = base_layer.weight.data.dtype
+                if safe_merge:
+                    orig_weights = base_layer.weight.data
+                    orig_weights = torch.mul(orig_weights, ia3_l)
+
+                    if not torch.isfinite(orig_weights).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+                    base_layer.weight.data = orig_weights.to(orig_dtype)
+                else:
+                    base_layer.weight.data = torch.mul(base_layer.weight.data, ia3_l).to(orig_dtype)
+
+                if not self.is_feedforward and (base_layer.bias is not None):
+                    scaling = self.ia3_l[active_adapter].reshape(base_layer.bias.shape)
+                    orig_dtype = base_layer.bias.data.dtype
+                    base_layer.bias.data = torch.mul(base_layer.bias.data, scaling.data).to(orig_dtype)
+
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+
+        warnings.warn("Unmerge result can be inaccurate for (IA)^3.")
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self.ia3_l.keys():
+                base_layer = self.get_base_layer()
+                # Add tolerace to avoid division by zero
+                ia3_l = transpose(self.ia3_l[active_adapter].data, self.fan_in_fan_out) + 1e-8
+                orig_dtype = base_layer.weight.data.dtype
+                base_layer.weight.data = torch.div(base_layer.weight.data, ia3_l).to(orig_dtype)
+
+                if not self.is_feedforward and (base_layer.bias is not None):
+                    scaling = self.ia3_l[active_adapter].reshape(base_layer.bias.shape)
+                    orig_dtype = base_layer.bias.data.dtype
+                    base_layer.bias.data = torch.div(base_layer.bias.data, scaling.data + 1e-8).to(orig_dtype)
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        dtype = previous_dtype = x.dtype
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            ia3_scaling = 1
+            for active_adapter in self.active_adapters:
+                if active_adapter not in self.ia3_l.keys():
+                    continue
+                dtype = self.ia3_l[active_adapter].dtype
+                ia3_scaling *= self.ia3_l[active_adapter].flatten()
+
+            if self.is_feedforward:
+                x = x.to(dtype)
+                # TODO: weight.dtype can be != self.ia3_l[self.active_adapters].dtype
+                # e.g. bf16 vs fp32. Is that okay?
+                interm = (x * ia3_scaling).to(previous_dtype)
+                result = self.base_layer(interm, *args, **kwargs)
+            else:
+                result = self.base_layer(x, *args, **kwargs)
+                result_dtype = result.dtype
+                result = (result * ia3_scaling).to(result_dtype)
+
+        return result
+
+
+class _ConvNd(nn.Module, IA3Layer):
+    def __init__(
+        self,
+        base_layer: nn.Module,
+        adapter_name: str,
+        config: IA3Config,
+        is_feedforward: bool = False,  # Set to True if the layer is treated as a feedforward layer
+        **kwargs,
+    ) -> None:
+        super().__init__()
+        IA3Layer.__init__(self, base_layer, is_feedforward=is_feedforward)
+        self.fan_in_fan_out = config.fan_in_fan_out
+        self._active_adapter = adapter_name
+        self._kernel_dim = base_layer.weight.dim()
+
+        self.update_layer(adapter_name, config=config)
+
+    def update_layer(self, adapter_name: str, config: IA3Config, **kwargs):
+        init_ia3_weights = config.init_ia3_weights
+        inference_mode = config.inference_mode
+
+        # Actual trainable parameters
+        num_features = self.in_features if self.is_feedforward else self.out_features
+        weights_size = (1, num_features) + (1,) * (self._kernel_dim - 2)
+        weight = torch.randn(weights_size)
+        self.ia3_l[adapter_name] = nn.Parameter(weight)
+        if init_ia3_weights:
+            self.reset_ia3_parameters(adapter_name)
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`List[str]`, *optional*):
+                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
+                to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.ia3_l.keys():
+                base_layer = self.get_base_layer()
+                orig_dtype = base_layer.weight.data.dtype
+                ia3_scaling = self.ia3_l[active_adapter].data
+                if not self.is_feedforward:
+                    ia3_scaling = ia3_scaling.transpose(0, 1)
+
+                if safe_merge:
+                    output_weight = torch.mul(base_layer.weight.data, ia3_scaling).clone()
+
+                    if not torch.isfinite(output_weight).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    base_layer.weight.data = output_weight.to(orig_dtype)
+                else:
+                    base_layer.weight.data = torch.mul(base_layer.weight.data, ia3_scaling).to(orig_dtype)
+
+                if not self.is_feedforward and (base_layer.bias is not None):
+                    scaling = self.ia3_l[active_adapter].reshape(base_layer.bias.shape)
+                    base_layer.bias.data = torch.mul(base_layer.bias.data, scaling.data).to(orig_dtype)
+
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+
+        warnings.warn("Unmerge result can be inaccurate for (IA)^3.")
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self.ia3_l.keys():
+                base_layer = self.get_base_layer()
+                orig_dtype = base_layer.weight.data.dtype
+                # divide by (IA)^3 vector. Add tolerace to avoid division by zero
+                ia3_scaling = self.ia3_l[active_adapter].data
+                if not self.is_feedforward:
+                    ia3_scaling = ia3_scaling.transpose(0, 1)
+                base_layer.weight.data = torch.div(base_layer.weight.data, ia3_scaling + 1e-8).to(orig_dtype)
+
+                if not self.is_feedforward and (base_layer.bias is not None):
+                    scaling = self.ia3_l[active_adapter].reshape(base_layer.bias.shape)
+                    orig_dtype = base_layer.bias.data.dtype
+                    base_layer.bias.data = torch.mul(base_layer.bias.data, scaling.data).to(orig_dtype)
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        dtype = previous_dtype = x.dtype
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            ia3_scaling = 1
+            for active_adapter in self.active_adapters:
+                if active_adapter not in self.ia3_l.keys():
+                    continue
+                dtype = self.ia3_l[active_adapter].dtype
+                ia3_scaling *= self.ia3_l[active_adapter]
+
+            if self.is_feedforward:
+                x = x.to(dtype)
+                # TODO: weight.dtype can be != self.ia3_l[self.active_adapters].dtype
+                # e.g. bf16 vs fp32. Is that okay?
+                interm = (x * ia3_scaling).to(self.get_base_layer().weight.dtype)
+                result = self.base_layer(interm, *args, **kwargs)
+            else:
+                result = self.base_layer(x, *args, **kwargs)
+                result = result.to(dtype) * ia3_scaling
+
+        result = result.to(previous_dtype)
+        return result
+
+
+class Conv2d(_ConvNd):
+    # IA3 implemented in a 2D convolutional layer
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        if not self._kernel_dim == 4:
+            raise ValueError(f"Conv2d layer kernel must have 4 dimensions, not {self._kernel_dim}")
+
+
+class Conv3d(_ConvNd):
+    # IA3 implemented in a 3D convolutional layer
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        if not self._kernel_dim == 5:
+            raise ValueError(f"Conv2d layer kernel must have 5 dimensions, not {self._kernel_dim}")
diff --git a/docs/refs/peft_lora_dora.py b/docs/refs/peft_lora_dora.py
new file mode 100644
index 0000000..f7a397f
--- /dev/null
+++ b/docs/refs/peft_lora_dora.py
@@ -0,0 +1,287 @@
+# Copyright 2024-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from copy import deepcopy
+from functools import wraps
+from typing import Any, Optional
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from peft.utils.integrations import dequantize_module_weight, gather_params_ctx
+from peft.utils.other import transpose
+
+
+ENABLE_DORA_CACHING = False
+"""Whether to enable DoRA caching, which makes it faster at inference but requires more memory"""
+
+
+def cache_decorator(cache_key: str):
+    """Caching decorator for DoRA
+
+    Caching is only enabled if ENABLE_DORA_CACHING is set to True (default: False), when in eval mode, and when the
+    adapter_name is passed (e.g. not during layer initialization).
+
+    """
+
+    def cache_value(func):
+        @wraps(func)
+        def wrapper(self, *args, **kwargs):
+            # if adapter_name is not passed, no caching
+            adapter_name = kwargs.get("adapter_name")
+            if (not ENABLE_DORA_CACHING) or self.training or (adapter_name is None):
+                self._cache_clear()
+                return func(self, *args, **kwargs)
+
+            cache_key_adapter = f"{cache_key}-{adapter_name}"
+            output = self._cache_get(cache_key_adapter, None)
+            if output is not None:
+                return output
+
+            output = func(self, *args, **kwargs)
+            self._cache_store(cache_key_adapter, output)
+            return output
+
+        return wrapper
+
+    return cache_value
+
+
+class DoraLinearLayer(nn.Module):
+    def __init__(self, fan_in_fan_out):
+        super().__init__()
+        self.fan_in_fan_out = fan_in_fan_out
+        self._dora_cache: dict[str, Any] = {}  # small ad hoc cache; values are not part of the state_dict
+
+    def _cache_store(self, key: str, value: Any) -> None:
+        # cache intermediate values, e.g. weight norm of DoRA
+        self._dora_cache[key] = value
+
+    def _cache_get(self, key: str, default: Optional[Any]) -> Optional[Any]:
+        # retrieve from ad hoc cache
+        return self._dora_cache.get(key, default)
+
+    def _cache_clear(self) -> None:
+        self._dora_cache.clear()
+
+    def train(self, mode: bool = True):
+        if mode:
+            self._cache_clear()
+        super().train(mode=mode)
+        return self
+
+    @cache_decorator("weight-norm")
+    def get_weight_norm(self, weight, lora_weight, scaling, adapter_name: Optional[str] = None) -> torch.Tensor:
+        # calculate L2 norm of weight matrix, column-wise
+        weight = transpose(weight, self.fan_in_fan_out)
+        weight = weight + scaling * lora_weight
+        weight_norm = torch.linalg.norm(weight, dim=1).to(weight.dtype)
+        return weight_norm
+
+    @cache_decorator("lora-weight")
+    def get_lora_weight(self, lora_A, lora_B, adapter_name: Optional[str] = None):
+        # Don't use `lora_weight = lora_B.weight @ lora_A.weight` because this causes errors with FSDP. Instead,
+        # calculate the same but using forward.
+        x_eye = torch.eye(lora_A.weight.shape[1], device=lora_A.weight.device, dtype=lora_A.weight.dtype)
+        lora_weight = lora_B(lora_A(x_eye)).T
+        return lora_weight
+
+    def update_layer(self, *, base_layer, lora_A, lora_B, scaling, place_on_cpu=False) -> None:
+        # temporarily convert fp16 to fp32, as fp16 can cause trouble on CPU with PyTorch < 2.2
+        dtype_is_fp16 = lora_A.dtype == torch.float16
+        if dtype_is_fp16:
+            lora_A = lora_A.float()
+            lora_B = lora_B.float()
+
+        with gather_params_ctx(base_layer.parameters()):
+            if base_layer.__class__.__name__ == "Linear4bit":
+                # We have to create a copy of the base layer, otherwise, FSDP will throw an error. 8bit does not work
+                # yet because Int8Params cannot be correctly deep-copied (attributes vanish)
+                base_layer = deepcopy(base_layer)
+
+            weight = dequantize_module_weight(base_layer)
+            if weight.data.ndim >= 3:  # For handling LoRAs applied to Conv layers.
+                r = lora_A.shape[0]
+                lora_weight = torch.mm(lora_B.view([-1, r]), lora_A.view([r, -1]))
+                lora_weight = lora_weight.reshape(weight.shape)
+            else:
+                lora_weight = lora_B @ lora_A
+
+            if dtype_is_fp16:
+                lora_weight = lora_weight.half()
+            weight_norm = self.get_weight_norm(
+                weight=weight.to(lora_A.device), lora_weight=lora_weight, scaling=scaling
+            )
+
+        if place_on_cpu:
+            weight_norm = weight_norm.to("cpu")
+        self.weight = nn.Parameter(weight_norm, requires_grad=True)
+
+    def forward(self, x, *, lora_A, lora_B, scaling, base_layer, base_result=None, adapter_name="default"):
+        """
+        For DoRA, calculate the extra output from LoRA with DoRA applied. This should be added on top of the base layer
+        output.
+        """
+        lora_weight = self.get_lora_weight(lora_A=lora_A, lora_B=lora_B, adapter_name=adapter_name)
+        lora_weight = lora_weight.to(x.dtype)
+
+        magnitude = self.weight
+        weight = dequantize_module_weight(base_layer)
+        weight = weight.to(x.dtype)
+        weight_norm = self.get_weight_norm(
+            weight=weight, lora_weight=lora_weight.detach(), scaling=scaling, adapter_name=adapter_name
+        )
+        # see section 4.3 of DoRA (https://huggingface.co/papers/2402.09353)
+        # "[...] we suggest treating ||V +∆V ||_c in
+        # Eq. (5) as a constant, thereby detaching it from the gradient
+        # graph. This means that while ||V + ∆V ||_c dynamically
+        # reflects the updates of ∆V , it won’t receive any gradient
+        # during backpropagation"
+        weight_norm = weight_norm.detach()
+        mag_norm_scale = (magnitude / weight_norm).view(1, -1)
+
+        lora_result = lora_B(lora_A(x))
+
+        bias = None
+        if base_result is not None:
+            bias = base_layer.bias
+            if bias is not None:
+                base_result = base_result - bias
+        else:
+            base_result = F.linear(x, transpose(weight, self.fan_in_fan_out))
+
+        result_dora = (mag_norm_scale - 1) * base_result + mag_norm_scale * lora_result * scaling
+        return result_dora
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "lora.dora." + rep
+
+
+class DoraEmbeddingLayer(DoraLinearLayer):
+    @cache_decorator("lora-weight")
+    def get_lora_weight(self, lora_A, lora_B, adapter_name: Optional[str] = None):
+        return (lora_A @ lora_B).T
+
+    def forward(self, x, *, lora_A, lora_B, scaling, base_layer, embed_fn, adapter_name="default"):
+        """
+        For DoRA, calculate the extra output from LoRA with DoRA applied. This should be added on top of the base layer
+        output.
+        """
+        lora_weight = self.get_lora_weight(lora_A=lora_A, lora_B=lora_B, adapter_name=adapter_name)
+        magnitude = self.weight
+        weight = base_layer.weight
+        weight_norm = self.get_weight_norm(
+            weight=weight, lora_weight=lora_weight.detach(), scaling=scaling, adapter_name=adapter_name
+        )
+        # see section 4.3 of DoRA (https://huggingface.co/papers/2402.09353)
+        # "[...] we suggest treating ||V +∆V ||_c in
+        # Eq. (5) as a constant, thereby detaching it from the gradient
+        # graph. This means that while ||V + ∆V ||_c dynamically
+        # reflects the updates of ∆V , it won’t receive any gradient
+        # during backpropagation"
+        weight_norm = weight_norm.detach()
+        mag_norm_scale = magnitude / weight_norm
+        result_dora = mag_norm_scale * (embed_fn(x, lora_A) @ lora_B) * scaling
+        return mag_norm_scale, result_dora
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "lora.dora." + rep
+
+
+class _DoraConvNdLayer(DoraLinearLayer):
+    @cache_decorator("weight-norm")
+    def get_weight_norm(self, weight, lora_weight, scaling, adapter_name: Optional[str] = None) -> torch.Tensor:
+        # calculate L2 norm of weight matrix, column-wise
+        weight = weight + scaling * lora_weight
+        # the following is needed to have compatibility with the 4/5D weight tensors of Conv2D/3D
+        dim = tuple(range(1, weight.dim()))
+        weight_norm = weight.norm(p=2, dim=dim, keepdim=True).transpose(1, 0)
+        return weight_norm
+
+    @cache_decorator("lora-weight")
+    def get_lora_weight(self, lora_A, lora_B, adapter_name: Optional[str] = None) -> torch.Tensor:
+        # Don't use `lora_weight = lora_B.weight @ lora_A.weight` because this causes errors with FSDP. Instead,
+        # calculate the same but using forward.
+        r = lora_A.weight.shape[0]
+        lora_weight = torch.mm(lora_B.weight.view([-1, r]), lora_A.weight.view([r, -1]))
+        return lora_weight
+
+    def forward(
+        self, x, *, lora_A, lora_B, scaling, base_layer, base_result=None, adapter_name: str = "default"
+    ) -> torch.Tensor:
+        """
+        For DoRA, calculate the extra output from LoRA with DoRA applied. This should be added on top of the base layer
+        output.
+        """
+        weight = base_layer.weight
+        lora_weight = self.get_lora_weight(lora_A=lora_A, lora_B=lora_B, adapter_name=adapter_name).reshape(
+            weight.shape
+        )
+        magnitude = self.weight
+        weight_norm = self.get_weight_norm(
+            weight=weight, lora_weight=lora_weight.detach(), scaling=scaling, adapter_name=adapter_name
+        )
+        # see section 4.3 of DoRA (https://huggingface.co/papers/2402.09353)
+        # "[...] we suggest treating ||V +∆V ||_c in
+        # Eq. (5) as a constant, thereby detaching it from the gradient
+        # graph. This means that while ||V + ∆V ||_c dynamically
+        # reflects the updates of ∆V , it won’t receive any gradient
+        # during backpropagation"
+        weight_norm = weight_norm.detach()
+        mag_norm_scale = magnitude / weight_norm
+
+        if base_result is None:
+            base_result = self.conv_fn(
+                x,
+                weight,
+                bias=None,
+                stride=base_layer.stride,
+                padding=base_layer.padding,
+                dilation=base_layer.dilation,
+                groups=base_layer.groups,
+            )
+        else:
+            bias = base_layer.bias
+            if bias is not None:
+                # reshape bias to (1, -1, 1, ...)
+                bias_shape = (1, -1) + (1,) * (base_result.dim() - 2)
+                base_result = base_result - bias.view(*bias_shape)
+
+        result_dora = (mag_norm_scale - 1) * base_result + mag_norm_scale * lora_B(lora_A(x)) * scaling
+        return result_dora
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "lora.dora." + rep
+
+
+class DoraConv1dLayer(_DoraConvNdLayer):
+    def __init__(self, fan_in_fan_out):
+        super().__init__(fan_in_fan_out)
+        self.conv_fn = F.conv1d
+
+
+class DoraConv2dLayer(_DoraConvNdLayer):
+    def __init__(self, fan_in_fan_out):
+        super().__init__(fan_in_fan_out)
+        self.conv_fn = F.conv2d
+
+
+class DoraConv3dLayer(_DoraConvNdLayer):
+    def __init__(self, fan_in_fan_out):
+        super().__init__(fan_in_fan_out)
+        self.conv_fn = F.conv3d
diff --git a/docs/refs/peft_lora_layer.py b/docs/refs/peft_lora_layer.py
new file mode 100644
index 0000000..eba67e7
--- /dev/null
+++ b/docs/refs/peft_lora_layer.py
@@ -0,0 +1,2510 @@
+# Copyright 2023-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import annotations
+
+import copy
+import math
+import warnings
+from collections.abc import Callable
+from contextlib import contextmanager
+from typing import Any, Optional, Union
+
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+import torch.nn.functional as F
+from torch import svd_lowrank
+from transformers.pytorch_utils import Conv1D
+
+from peft.import_utils import is_transformers_ge_v5_4_0
+from peft.tuners._buffer_dict import BufferDict
+from peft.tuners.tuners_utils import BaseTunerLayer, _get_in_out_features, check_adapters_to_merge
+from peft.utils import ALLOWED_COMPUTE_DTYPES, UPCAST_DTYPES
+from peft.utils.integrations import (
+    dequantize_module_weight,
+    gather_params_ctx,
+    get_bnb_param_type,
+    skip_init_on_device,
+)
+from peft.utils.loftq_utils import loftq_init
+from peft.utils.other import transpose
+from peft.utils.warning import PeftWarning
+
+from .config import LoraConfig
+
+
+VARIANT_KWARG_KEYS = ["alora_offsets"]
+
+
+class LoraVariant:
+    """
+    Base class for LoRA variants, e.g. DoRA.
+
+    This class should be subclassed and the methods below should be implemented accordingly. The methods should be
+    implemented as static methods, this makes it easier to combine variants.
+
+    Note for developers: These methods are prone to change and should thus considered to be "private". Use at your own
+    discretion.
+    """
+
+    @staticmethod
+    def init(module: LoraLayer, adapter_name: str) -> None:
+        """Initialization code for the LoRA variant, it's called within `update_layer`"""
+        raise NotImplementedError
+
+    @staticmethod
+    def merge_safe(module: LoraLayer, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        """Safe merging of the weights from `merge(..., safe_merge=True)`, should return a new tensor"""
+        raise NotImplementedError
+
+    @staticmethod
+    def merge_unsafe(module: LoraLayer, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        """Unsafe merging of the weights from `merge(..., safe_merge=False)`, should modify the weight in-place"""
+
+    @staticmethod
+    def unmerge(module: LoraLayer, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        """Remove the adapter weights from the original weights, then return them"""
+
+    @staticmethod
+    def forward(
+        module: LoraLayer,
+        active_adapter: str,
+        x: torch.Tensor,
+        result: torch.Tensor,
+        **kwargs,
+    ) -> torch.Tensor:
+        """
+        The forward pass of the LoRA variant, should return the overall result (not just the diff)
+
+        Args:
+            module (LoraLayer): The module on which the forward pass is called
+            active_adapter (str): The name of the active adapter
+            x (torch.Tensor): The input to the forward call
+            result (torch.Tensor): The result from the base model
+            **kwargs: Additional arguments passed from [`LoraLayer.forward`].
+        """
+        raise NotImplementedError
+
+
+class LoraLayer(BaseTunerLayer):
+    # All names of layers that may contain (trainable) adapter weights
+    adapter_layer_names: tuple[str, ...] = ("lora_A", "lora_B", "lora_embedding_A", "lora_embedding_B")
+    # All names of other parameters that may contain adapter-related parameters
+    other_param_names: tuple[str, ...] = ("r", "lora_alpha", "scaling", "lora_dropout")
+
+    def __init__(self, base_layer: nn.Module, ephemeral_gpu_offload: bool = False, **kwargs) -> None:
+        self.base_layer = base_layer
+        self.r = {}
+        self.lora_alpha = {}
+        self.scaling = {}
+        self.lora_dropout = nn.ModuleDict({})
+        self.lora_A = nn.ModuleDict({})
+        self.lora_B = nn.ModuleDict({})
+        # For Embedding layer
+        self.lora_embedding_A = nn.ParameterDict({})
+        self.lora_embedding_B = nn.ParameterDict({})
+        # Mark the weight as unmerged
+        self._disable_adapters = False
+        self.merged_adapters = []
+        self.use_dora: dict[str, bool] = {}  # not actively used anymore after #2443, keep it for BC
+        self.use_rslora: dict[str, bool] = {}
+        self.lora_bias: dict[str, bool] = {}
+        self.lora_magnitude_vector = torch.nn.ModuleDict()  # for DoRA
+        self._caches: dict[str, Any] = {}  # small ad hoc cache; values are not part of the state_dict
+        self.ephemeral_gpu_offload: bool = ephemeral_gpu_offload
+        # flag to enable/disable casting of input to weight dtype during forward call
+        self.cast_input_dtype_enabled: bool = True
+        self.lora_variant: dict[str, LoraVariant] = {}
+        self.kwargs = kwargs
+
+        base_layer = self.get_base_layer()
+        in_features, out_features = self._get_in_out_features(base_layer)
+        self.in_features = in_features
+        self.out_features = out_features
+
+    def _get_in_out_features(self, module: nn.Module) -> tuple[int, int] | tuple[None, None]:
+        return _get_in_out_features(module)
+
+    def resolve_lora_variant(self, *, config: LoraConfig, **kwargs) -> Optional[LoraVariant]:
+        """Return a matching LoRA variant for this layer type.
+
+        Given the init arguments of this layer, return the correct LoRA variant, if any. E.g., if `use_dora=True`, this
+        method should return the DoRA variant for the given layer. If `use_alora=True`, same for aLoRA.
+
+        If there is no fitting variant, return None.
+
+        Note: If this layer type does not support the LoRA variant at all, please raise an error during __init__ as is
+        convention, and not here.
+
+        """
+        return None
+
+    def update_layer(
+        self,
+        adapter_name: str,
+        r: int,
+        lora_alpha: int,
+        config: LoraConfig,
+        **kwargs,
+    ) -> None:
+        # collect the kwargs
+        lora_dropout = config.lora_dropout
+        init_lora_weights = config.init_lora_weights
+        use_rslora = config.use_rslora
+        lora_bias = config.lora_bias
+        inference_mode = config.inference_mode
+
+        target_name = kwargs.get("target_name", "")  # preserve target_name before overwriting kwargs
+        kwargs["target_name"] = target_name  # restore target_name
+        tied_adapter = kwargs.get("tied_adapter", None)
+
+        # This code works for linear layers, override for other layer types
+        if r <= 0:
+            raise ValueError(f"`r` should be a positive integer value but the value passed is {r}")
+
+        if lora_bias and (getattr(self.get_base_layer(), "bias", None) is None):
+            warnings.warn(
+                f"`lora_bias=True` was passed but the targeted layer of type {type(self.get_base_layer()).__name__} "
+                "has no bias. This means that merging LoRA weights won't be possible.",
+                PeftWarning,
+            )
+
+        lora_variant = self.resolve_lora_variant(config=config)
+        if lora_variant is not None:
+            self.lora_variant[adapter_name] = lora_variant
+
+        self.r[adapter_name] = r
+        self.lora_alpha[adapter_name] = lora_alpha
+        if lora_dropout > 0.0:
+            lora_dropout_layer = nn.Dropout(p=lora_dropout)
+        else:
+            lora_dropout_layer = nn.Identity()
+
+        self.lora_dropout.update(nn.ModuleDict({adapter_name: lora_dropout_layer}))
+
+        # Actual trainable parameters
+        self.lora_A[adapter_name] = nn.Linear(self.in_features, r, bias=False)
+        self.lora_B[adapter_name] = nn.Linear(r, self.out_features, bias=lora_bias)
+
+        # Tying adapters is only implemented for Linear layers
+        # where the source is the embedding layer.
+        # Currently, this is the most prevelant way of tying layers (weight tying)
+        if tied_adapter:
+            lora_A_params = tied_adapter["lora_A"]
+            lora_B_params = tied_adapter["lora_B"]
+
+            self.lora_A[adapter_name].weight = torch.nn.Parameter(lora_A_params)
+            self.lora_B[adapter_name].weight = torch.nn.Parameter(lora_B_params)
+
+        self.lora_bias[adapter_name] = lora_bias
+
+        if use_rslora:
+            self.scaling[adapter_name] = lora_alpha / math.sqrt(r)
+        else:
+            self.scaling[adapter_name] = lora_alpha / r
+
+        self.use_rslora[adapter_name] = use_rslora
+
+        self.use_dora[adapter_name] = config.use_dora
+
+        # for inits that require access to the base weight, use gather_param_ctx so that the weight is gathered when using DeepSpeed
+        if isinstance(init_lora_weights, str) and init_lora_weights.startswith("pissa"):
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.pissa_init(adapter_name, init_lora_weights)
+        elif isinstance(init_lora_weights, str) and init_lora_weights.startswith("corda"):
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.corda_init(adapter_name, init_lora_weights)
+        elif isinstance(init_lora_weights, str) and init_lora_weights.lower() == "olora":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.olora_init(adapter_name)
+        elif init_lora_weights == "loftq":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.loftq_init(adapter_name, config)
+        elif init_lora_weights == "eva":
+            nn.init.zeros_(self.lora_B[adapter_name].weight)
+        elif init_lora_weights == "orthogonal":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.orthogonal_init(adapter_name)
+        elif init_lora_weights == "lora_ga":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.lora_ga_init(adapter_name, config.lora_ga_config)
+        elif init_lora_weights:
+            self.reset_lora_parameters(adapter_name, init_lora_weights)
+        # call this before init of the lora variants
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+
+        if adapter_name in self.lora_variant:
+            self.lora_variant[adapter_name].init(self, adapter_name=adapter_name, config=config, **kwargs)
+
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+        # Check for adapters that were added or removed from the arrow_model.
+        # The arrow model may be modified after creation by adding new experts
+        # (pre-trained or trainable) or by removing existing ones. Whenever such
+        # a change occurs, on_adapter_change() is called to update the set of
+        # active task-specific experts and, if needed, to handle recomputing prototypes
+        # and doing general knowledge subtraction (GKS) again.
+        if hasattr(self, "lora_arrow"):
+            for adapter in self.lora_variant:
+                if adapter in self.lora_arrow:
+                    self.lora_arrow[adapter].on_adapter_change(self.lora_A, self.lora_B)
+
+    def reset_lora_parameters(self, adapter_name, init_lora_weights):
+        if init_lora_weights is not False:
+            if adapter_name in self.lora_A.keys():
+                if init_lora_weights is True:
+                    # initialize A the same way as the default for nn.Linear and B to zero
+                    # https://github.com/microsoft/LoRA/blob/a0a92e0f26c067cf94747bdbf1ce73793fa44d19/loralib/layers.py#L124
+                    nn.init.kaiming_uniform_(self.lora_A[adapter_name].weight, a=math.sqrt(5))
+                elif init_lora_weights.lower() == "gaussian":
+                    nn.init.normal_(self.lora_A[adapter_name].weight, std=1 / self.r[adapter_name])
+                else:
+                    raise ValueError(f"Unknown initialization {init_lora_weights=}")
+                nn.init.zeros_(self.lora_B[adapter_name].weight)
+                if self.lora_bias[adapter_name]:
+                    nn.init.zeros_(self.lora_B[adapter_name].bias)
+            if adapter_name in self.lora_embedding_A.keys():
+                # Initialize A to zeros and B the same way as the default for nn.Embedding, see:
+                # https://github.com/microsoft/LoRA/blob/4c0333854cb905966f8cc4e9a74068c1e507c7b7/loralib/layers.py#L59-L60
+                nn.init.zeros_(self.lora_embedding_A[adapter_name])
+                nn.init.normal_(self.lora_embedding_B[adapter_name])
+                if self.lora_bias[adapter_name]:
+                    # embeddings are not supported at the moment, but still adding this for consistency
+                    nn.init.zeros_(self.lora_embedding_B[adapter_name].bias)
+
+        # Always synchronize non-sharded LoRA weights across TP ranks, regardless of
+        # init_lora_weights, since each rank initializes weights independently.
+        # We could skip some broadcast, for instance when the lora weights are initialized to zero,
+        # but this is a minor optimization and would add extra complexity to the code.
+        if dist.is_available() and dist.is_initialized():
+            base_layer = self.get_base_layer()
+            tp_plan = getattr(base_layer, "_hf_tp_plan", None)
+            device_mesh = getattr(base_layer, "_hf_device_mesh", None)
+            if device_mesh is not None:
+                if tp_plan not in ["colwise", "rowwise", "embedding_rowwise"]:
+                    warnings.warn(
+                        f'tp_plan "{tp_plan}" found on the base layer is not supported for LoRA weight '
+                        'synchronization. Expected one of "colwise", "rowwise", "embedding_rowwise". '
+                        "LoRA weights may not be synchronized across ranks."
+                    )
+                pg = device_mesh.get_group()
+                src = dist.get_global_rank(pg, 0)
+                if tp_plan == "colwise" and adapter_name in self.lora_A:
+                    # The adapter weights need to be on device for broadcasting
+                    self._move_adapter_to_device_of_base_layer(adapter_name)
+                    dist.broadcast(self.lora_A[adapter_name].weight.data, src=src, group=pg)
+                elif tp_plan == "rowwise" and adapter_name in self.lora_B:
+                    # The adapter weights need to be on device for broadcasting
+                    self._move_adapter_to_device_of_base_layer(adapter_name)
+                    dist.broadcast(self.lora_B[adapter_name].weight.data, src=src, group=pg)
+                elif tp_plan == "embedding_rowwise" and adapter_name in self.lora_embedding_B:
+                    self._move_adapter_to_device_of_base_layer(adapter_name)
+                    dist.broadcast(self.lora_embedding_B[adapter_name].data, src=src, group=pg)
+
+    def olora_init(self, adapter_name):
+        base_layer = self.get_base_layer()
+        orig_weight = base_layer.weight
+        bnb_param_type = get_bnb_param_type(orig_weight)
+        dtype = orig_weight.dtype
+
+        if bnb_param_type:
+            # check without importing bitsandbytes and robust to bnb_4bit_quant_storage=float*
+            weight_tensor = dequantize_module_weight(base_layer)
+        elif dtype in [torch.float32, torch.float16, torch.bfloat16]:
+            weight_tensor = orig_weight
+        else:
+            raise TypeError(f"Unsupported data type for the base layer. Got {dtype}.")
+
+        scale_factor = self.scaling[adapter_name]
+        r = self.r[adapter_name]
+        weight_tensor = weight_tensor.to(torch.float32)
+        Q, R = torch.linalg.qr(weight_tensor.data)
+
+        Qr, Rr = Q[:, :r], R[:r]
+
+        self.lora_A[adapter_name].weight.data = Rr.contiguous()
+        self.lora_B[adapter_name].weight.data = Qr.contiguous()
+
+        weight_tensor.data -= scale_factor * self.lora_B[adapter_name].weight @ self.lora_A[adapter_name].weight
+        if bnb_param_type == "4bit":
+            weight_tensor = orig_weight.__class__(
+                weight_tensor,
+                quant_type=orig_weight.quant_type,
+                quant_storage=orig_weight.quant_storage,
+                compress_statistics=orig_weight.compress_statistics,
+                module=orig_weight.module,
+            ).to(orig_weight.device)
+            base_layer.weight = weight_tensor
+        elif bnb_param_type == "8bit":
+            weight_tensor = orig_weight.__class__(
+                weight_tensor,
+                requires_grad=orig_weight.requires_grad,
+                has_fp16_weights=orig_weight.has_fp16_weights,
+            ).to(orig_weight.device)
+            base_layer.weight = weight_tensor
+        else:
+            weight_tensor = weight_tensor.to(dtype)
+            base_layer.weight.data = weight_tensor
+
+    def pissa_init(self, adapter_name, init_lora_weights):
+        weight = self.get_base_layer().weight
+        dtype = weight.dtype
+        if dtype not in [torch.float32, torch.float16, torch.bfloat16]:
+            raise TypeError(
+                "Please initialize PiSSA under float32, float16, or bfloat16. "
+                "Subsequently, re-quantize the residual model to help minimize quantization errors."
+            )
+        weight = transpose(weight.to(torch.float32), self.fan_in_fan_out)
+        if init_lora_weights == "pissa":
+            # USV^T = W <-> VSU^T = W^T, where W^T = weight.data in R^{out_channel, in_channel},
+            V, S, Uh = torch.linalg.svd(weight.data, full_matrices=False)
+            Vr = V[:, : self.r[adapter_name]]
+            Sr = S[: self.r[adapter_name]]
+            Sr /= self.scaling[adapter_name]
+            Uhr = Uh[: self.r[adapter_name]]
+        elif len(init_lora_weights.split("_niter_")) == 2:
+            Vr, Sr, Ur = svd_lowrank(
+                weight.data, self.r[adapter_name], niter=int(init_lora_weights.split("_niter_")[-1])
+            )
+            Sr /= self.scaling[adapter_name]
+            Uhr = Ur.t()
+        else:
+            raise ValueError(
+                f"init_lora_weights should be 'pissa' or 'pissa_niter_[number of iters]', got {init_lora_weights} instead."
+            )
+
+        lora_A = torch.diag(torch.sqrt(Sr)) @ Uhr
+        lora_B = Vr @ torch.diag(torch.sqrt(Sr))
+        self.lora_A[adapter_name].weight.data = lora_A
+        self.lora_B[adapter_name].weight.data = lora_B
+        weight = weight.data - self.scaling[adapter_name] * lora_B @ lora_A
+        weight = transpose(weight.to(dtype), self.fan_in_fan_out)
+        self.get_base_layer().weight.data = weight
+
+    def corda_init(self, adapter_name, init_lora_weights):
+        linear = self.get_base_layer()
+        weight = linear.weight
+        dtype = weight.dtype
+        if dtype not in [torch.float32, torch.float16, torch.bfloat16]:
+            raise TypeError(
+                "Please initialize CorDA under float32, float16, or bfloat16. "
+                "Subsequently, re-quantize the residual model to help minimize quantization errors."
+            )
+        weight = weight.to(torch.float32)
+        # For Conv1D, weight is stored as (in_features, out_features), transposed compared to Linear
+        if isinstance(linear, Conv1D):
+            out_dim = weight.data.size(1)
+            in_dim = weight.data.size(0)
+        else:
+            out_dim = weight.data.size(0)
+            in_dim = weight.data.size(1)
+
+        # Calculate WC from covariance matrix
+        if not hasattr(linear, "eigens"):
+            raise ValueError(
+                "`eigens` attribute not found for layer, please run `preprocess_corda` first. "
+                "More information can be found at examples/corda_finetuning/README.md."
+            )
+        eigens = linear.eigens
+        U = eigens.U_WC
+        S = eigens.S_WC
+        V = eigens.V_WC
+        r = self.r[adapter_name]
+
+        # nan or inf check
+        if torch.isnan(S).any() or torch.isinf(S).any():
+            raise ValueError(
+                "Invalid value found in matrix S. Please file an issue at https://github.com/huggingface/peft/issues."
+            )
+        if torch.isnan(U).any() or torch.isinf(U).any():
+            raise ValueError(
+                "Invalid value found in matrix U. Please file an issue at https://github.com/huggingface/peft/issues."
+            )
+        if torch.isnan(V).any() or torch.isinf(V).any():
+            raise ValueError(
+                "Invalid value found in matrix V. Please file an issue at https://github.com/huggingface/peft/issues."
+            )
+
+        # Sanity check
+        if U.size(0) != out_dim or U.size(1) != r:
+            raise ValueError(
+                f"Matrix U size mismatch: {U.size()} vs. ({out_dim}, {r}). Please make sure the `lora_config` and "
+                "`model` argument of `preprocess_corda` is consistent with `get_peft_model`. If you're using cache "
+                "in `preprocess_corda`, please make sure the cache is built with the same model and LoRA rank."
+            )
+        if S.size(0) != r:
+            raise ValueError(
+                f"Matrix S size mismatch: {S.size()} vs. ({r},). Please make sure the `lora_config` and `model` argument "
+                "of `preprocess_corda` is consistent with `get_peft_model`. If you're using cache in `preprocess_corda`, "
+                "please make sure the cache is built with the same model and LoRA rank."
+            )
+        if V.size(0) != in_dim or V.size(1) != r:
+            raise ValueError(
+                f"Matrix V size mismatch: {V.size()} vs. ({in_dim}, {r}). Please make sure the `lora_config` and "
+                "`model` argument of `preprocess_corda` is consistent with `get_peft_model`. If you're using cache "
+                "in `preprocess_corda`, please make sure the cache is built with the same model and LoRA rank."
+            )
+
+        # Apply alpha
+        S /= self.scaling[adapter_name]
+
+        # Init lora_A and lora_B weights
+        lora_A = V.t().mul(S.sqrt().view(-1, 1)).contiguous()
+        lora_B = U.mul(S.sqrt()).contiguous()
+        self.lora_A[adapter_name].weight.data = lora_A
+        self.lora_B[adapter_name].weight.data = lora_B
+
+        # For Conv1D, lora_B @ lora_A gives (out_dim, in_dim) but weight is (in_dim, out_dim)
+        # So we need to transpose before subtraction
+        delta = self.scaling[adapter_name] * lora_B @ lora_A
+        delta = transpose(delta, fan_in_fan_out=self.fan_in_fan_out)
+        weight = weight.data - delta
+        weight = weight.to(dtype)
+        self.get_base_layer().weight.data = weight
+
+        # Remove redundant fields
+        del linear.eigens
+
+    def loftq_init(self, adapter_name, config: LoraConfig):
+        weight = self.get_base_layer().weight
+        kwargs = {
+            "num_bits": config.loftq_config["loftq_bits"],
+            "reduced_rank": self.r[adapter_name],
+            "num_iter": config.loftq_config["loftq_iter"],
+        }
+
+        qweight, lora_A, lora_B = loftq_init(weight, **kwargs)
+        if adapter_name in self.lora_A.keys():
+            # initialize A the same way as the default for nn.Linear and B to zero
+            self.lora_A[adapter_name].weight.data = lora_A
+            self.lora_B[adapter_name].weight.data = lora_B
+        if adapter_name in self.lora_embedding_A.keys():
+            # initialize a the same way as the default for nn.linear and b to zero
+            self.lora_embedding_A[adapter_name].weight.data = lora_A
+            self.lora_embedding_B[adapter_name].weight.data = lora_B
+        self.get_base_layer().weight.data = qweight
+
+    @torch.no_grad()
+    def orthogonal_init(self, adapter_name):
+        # https://datta0.github.io/posts/rethink-lora-init/#orthogonal-initialisation
+        rank = self.r[adapter_name]
+        if rank % 2 != 0:
+            raise ValueError(f"Orthogonal initialization requires the LoRA rank to be even, got {rank} instead.")
+
+        X = torch.randn(rank, rank)
+        Q, _ = torch.linalg.qr(X)
+        q_odd = Q[0::2, :]  # Odd rows
+        q_even = Q[1::2, :]  # Even rows
+        dtype = self.get_base_layer().weight.dtype
+        lora_A = torch.randn(self.in_features, rank // 2).mm(q_odd).T / 10.0
+        lora_B = torch.randn(rank // 2, self.out_features).T.mm(q_even) / 10.0
+        self.lora_A[adapter_name].weight = nn.Parameter(lora_A.contiguous().to(dtype))
+        self.lora_B[adapter_name].weight = nn.Parameter(lora_B.contiguous().to(dtype))
+
+    def lora_ga_init(self, adapter_name, lora_ga_config):
+        """
+        Initialize LoRA weights using gradient approximation.
+
+        Uses SVD on the gradient matrix to initialize adapters in a way that aligns with the direction of full
+        fine-tuning.
+
+        Expects that `preprocess_loraga` has been called before this, which attaches the `loraga_grad` attribute to the
+        base layer.
+
+        If gradients are not available (e.g., when loading from a saved adapter), falls back to gaussian
+        initialization. The weights will be overwritten by the state_dict anyway.
+        """
+        base_layer = self.get_base_layer()
+
+        # Check for gradient attached by preprocess_loraga
+        if not hasattr(base_layer, "_peft_loraga_grad"):
+            # When loading from saved adapter, gradients won't be available
+            # Fall back to gaussian initialization (weights will be overwritten by state_dict)
+            self.reset_lora_parameters(adapter_name, init_lora_weights=True)
+            return
+
+        grad = base_layer._peft_loraga_grad
+
+        # Check for lora_ga_config
+        if lora_ga_config is None:
+            raise ValueError(
+                "lora_ga_config must be provided when init_lora_weights='lora_ga'. "
+                "Please pass lora_ga_config=LoraGAConfig(...) to LoraConfig."
+            )
+        direction = lora_ga_config.direction
+        scale = lora_ga_config.scale
+        stable_gamma = lora_ga_config.stable_gamma
+        dtype = self.get_base_layer().weight.dtype
+
+        grad = grad.to(torch.float32)
+        weight = self.get_base_layer().weight
+
+        grad = transpose(grad, self.fan_in_fan_out)
+
+        r = self.r[adapter_name]
+
+        # torch.svd_lowrank returns (U, S, V) where grad ≈ U @ diag(S) @ V.T
+        # So V is shape (in_features, k) and we need V.T which is (k, in_features) for lora_A
+        U, S, V = torch.svd_lowrank(grad, q=min(4 * r, min(grad.shape)), niter=4)
+
+        # V is (in_features, k), we need Vh = V.T which is (k, in_features)
+        Vh = V.t()
+
+        U = U[:, : 2 * r]
+        S = S[: 2 * r]
+        Vh = Vh[: 2 * r, :]
+
+        if direction == "ArBr":
+            # Alternating: A takes rows at odd indices [1,3,5,7], B takes columns at even indices [0,2,4,6]
+            lora_A_weight = Vh[1 : 2 * r : 2, :]  # Shape: (r, in_features)
+            lora_B_weight = U[:, 0 : 2 * r : 2]  # Shape: (out_features, r)
+            S_B = S[0 : 2 * r : 2]
+            lora_B_weight = lora_B_weight @ torch.diag(S_B)
+
+        elif direction == "A2rBr":
+            # A takes second half rows [r:2r], B takes first half columns [:r]
+            lora_A_weight = Vh[r : 2 * r, :]  # Shape: (r, in_features)
+            lora_B_weight = U[:, :r]  # Shape: (out_features, r)
+            S_B = S[:r]
+            lora_B_weight = lora_B_weight @ torch.diag(S_B)
+
+        elif direction == "ArB2r":
+            # A takes first half rows [:r], B takes second half columns [r:2r]
+            lora_A_weight = Vh[:r, :]  # Shape: (r, in_features)
+            lora_B_weight = U[:, r : 2 * r]  # Shape: (out_features, r)
+            S_B = S[r : 2 * r]
+            lora_B_weight = lora_B_weight @ torch.diag(S_B)
+
+        elif direction == "random":
+            indices = torch.randperm(2 * r)[:r]
+            lora_A_weight = Vh[indices, :]  # Shape: (r, in_features)
+            S_B = S[indices]
+            lora_B_weight = U[:, indices] @ torch.diag(S_B)  # Shape: (out_features, r)
+
+        scaling_factor = self.scaling[adapter_name]
+        out_features = weight.shape[0]
+
+        if scale == "stable":
+            scale_factor = (out_features**0.25) / (stable_gamma**0.5)
+            lora_B_weight = lora_B_weight * scale_factor
+
+        elif scale == "weight_svd":
+            weight_data = transpose(weight.data.to(torch.float32), self.fan_in_fan_out)
+            _, weight_S, _ = torch.svd_lowrank(weight_data, q=r, niter=4)
+            if S_B[0] > 0:
+                scale_factor = weight_S[0] / S_B[0]
+                lora_B_weight = lora_B_weight * scale_factor
+
+        elif scale == "gd_scale":
+            lora_A_weight = lora_A_weight / scaling_factor
+            lora_B_weight = lora_B_weight / scaling_factor
+
+        # Convert to target dtype first to ensure weight offset matches adapter precision
+        lora_A_weight = lora_A_weight.to(dtype)
+        lora_B_weight = lora_B_weight.to(dtype)
+
+        # Assign LoRA weights
+        # lora_A should be (r, in_features), lora_B should be (out_features, r)
+        self.lora_A[adapter_name].weight.data = lora_A_weight.contiguous()
+        self.lora_B[adapter_name].weight.data = lora_B_weight.contiguous()
+
+        # Modify base weights: W_new = W_old - scaling * (B @ A)
+        # Important: compute offset in fp32 using dtype-converted weights to match forward pass precision
+        weight_data = transpose(weight.data.to(torch.float32), self.fan_in_fan_out)
+        weight_offset = scaling_factor * (lora_B_weight.float() @ lora_A_weight.float())
+        weight_data = weight_data - weight_offset
+        weight_data = transpose(weight_data.to(dtype), self.fan_in_fan_out)
+        self.get_base_layer().weight.data = weight_data
+
+        # Remove redundant fields
+        del base_layer._peft_loraga_grad
+
+    def _cache_store(self, key: str, value: Any) -> None:
+        # cache intermediate values, e.g. weight norm of DoRA
+        self._caches[key] = value
+
+    def _cache_pop(self, key: str) -> Any:
+        # retrieve and remove from ad hoc cache
+        value = self._caches.pop(key)
+        return value
+
+    def set_scale(self, adapter: str, scale: float | int) -> None:
+        """Set the scale of the given adapter to the initial scale multiplied by the provided factor
+
+        The initial scale is determined by the configured `r` (rank) and `lora_alpha`.
+        """
+        if adapter not in self.scaling:
+            # Ignore the case where the adapter is not in the layer
+            return
+        if self.use_rslora.get(adapter, False):
+            self.scaling[adapter] = scale * self.lora_alpha[adapter] / math.sqrt(self.r[adapter])
+        else:
+            self.scaling[adapter] = scale * self.lora_alpha[adapter] / self.r[adapter]
+
+    def scale_layer(self, scale: float | int) -> None:
+        """Multiply the current scale of all active adapters by the provided factor"""
+        if scale == 1:
+            return
+
+        for active_adapter in self.active_adapters:
+            if active_adapter not in self.lora_A.keys():
+                continue
+
+            self.scaling[active_adapter] *= scale
+
+    def unscale_layer(self, scale: Optional[float | int] = None) -> None:
+        """Divide the current scale of all active adapters by the provided factor. If `scale=None` is passed, reset to
+        initial scale
+
+        The initial scale is determined by the configured `r` (rank) and `lora_alpha`.
+
+        """
+        for active_adapter in self.active_adapters:
+            if active_adapter not in self.lora_A.keys():
+                continue
+
+            if scale is None:
+                if self.use_rslora.get(active_adapter, False):
+                    self.scaling[active_adapter] = self.lora_alpha[active_adapter] / math.sqrt(self.r[active_adapter])
+                else:
+                    self.scaling[active_adapter] = self.lora_alpha[active_adapter] / self.r[active_adapter]
+            else:
+                self.scaling[active_adapter] = self.scaling[active_adapter] / scale
+
+    def _check_forward_args(self, x, *args, **kwargs):
+        """Check if the arguments are compatible with the configs and state of the model"""
+        adapter_names = kwargs.get("adapter_names", None)
+        if adapter_names is None:
+            return
+
+        if len(x) != len(adapter_names):
+            msg = (
+                "Length of `adapter_names` should be the same as the number of inputs, but got "
+                f"{len(adapter_names)} and {len(x)} respectively."
+            )
+            raise ValueError(msg)
+
+        if self.merged:
+            # It is unclear what would be the right thing to do if users pass adapter_names and there are merged
+            # adapters. Therefore, it is better to raise an error in this case.
+            msg = "Cannot pass `adapter_names` when there are merged adapters, please call `unmerge_adapter` first."
+            raise ValueError(msg)
+
+        # DoRA is not supported (yet), check that it's not being used. Don't check "__base__", as this is the
+        # placeholder for the base model.
+        unique_adapters = {name for name in adapter_names if name != "__base__"}
+        for adapter_name in unique_adapters:
+            if self.use_dora.get(adapter_name, False):
+                msg = "Cannot pass `adapter_names` when DoRA is enabled."
+                raise ValueError(msg)
+
+    def _mixed_batch_forward(
+        self, x: torch.Tensor, *args: Any, adapter_names: list[str], **kwargs: Any
+    ) -> torch.Tensor:
+        # This is a special method that handles the case when users pass the argument `adapter_names`. This is an
+        # extra argument that allows mixing different adapters in the same batch at inference time.
+        variant_kwargs = {k: kwargs.pop(k, None) for k in VARIANT_KWARG_KEYS}  # don't pass these to base_layer
+        result = self.base_layer(x, *args, **kwargs)
+        torch_result_dtype = result.dtype
+
+        unique_adapters = set(adapter_names)
+        sub_batch_indices_list = []
+        for adapter in unique_adapters:
+            sub_batch_indices_list.append([index for index, item in enumerate(adapter_names) if item == adapter])
+        alora_offsets = variant_kwargs.get("alora_offsets", None)
+        for i, active_adapter in enumerate(unique_adapters):
+            if active_adapter == "__base__":
+                continue
+            if active_adapter not in self.lora_A.keys():
+                continue
+
+            lora_A = self.lora_A[active_adapter]
+            lora_B = self.lora_B[active_adapter]
+            dropout = self.lora_dropout[active_adapter]
+            scaling = self.scaling[active_adapter]
+
+            # getting the sub-batch, passing it to LoRA layers and updating the corresponding indices of the linear
+            # layer output
+            sub_batch = x[sub_batch_indices_list[i]].to(lora_A.weight.dtype)
+            if active_adapter not in self.lora_variant:  # vanilla LoRA
+                lora_output = lora_B(lora_A(dropout(sub_batch))) * scaling
+                result[sub_batch_indices_list[i]] += lora_output.to(torch_result_dtype)
+            else:
+                if alora_offsets is not None:
+                    variant_kwargs["alora_offsets"] = [alora_offsets[j] for j in sub_batch_indices_list[i]]
+                lora_output = self.lora_variant[active_adapter].forward(
+                    self,
+                    active_adapter=active_adapter,
+                    x=sub_batch,
+                    result=result[sub_batch_indices_list[i]],
+                    **variant_kwargs,
+                    **kwargs,
+                )
+                result[sub_batch_indices_list[i]] = lora_output.to(torch_result_dtype)
+
+        return result
+
+
+# Below code is based on https://github.com/microsoft/LoRA/blob/main/loralib/layers.py
+# and modified to work with PyTorch FSDP
+
+
+#  ------------------------------------------------------------------------------------------
+#  Copyright (c) Microsoft Corporation. All rights reserved.
+#  Licensed under the MIT License (MIT). See LICENSE in the repo root for license information.
+#  ------------------------------------------------------------------------------------------
+
+
+class Linear(nn.Module, LoraLayer):
+    # Lora implemented in a dense layer
+    def __init__(
+        self,
+        base_layer,
+        adapter_name: str,
+        config: LoraConfig,
+        r: int = 0,
+        lora_alpha: int = 1,
+        is_target_conv_1d_layer: bool = False,
+        **kwargs,
+    ) -> None:
+        super().__init__()
+        LoraLayer.__init__(self, base_layer, **kwargs)
+        self.fan_in_fan_out = config.fan_in_fan_out
+
+        self._active_adapter = adapter_name
+        self.update_layer(
+            adapter_name,
+            r,
+            lora_alpha=lora_alpha,
+            config=config,
+            **kwargs,
+        )
+        self.is_target_conv_1d_layer = is_target_conv_1d_layer
+
+    def resolve_lora_variant(self, config: LoraConfig, **kwargs) -> Optional[LoraVariant]:
+        if config.arrow_config is not None:
+            from .variants import ArrowLinearVariant
+
+            return ArrowLinearVariant()
+
+        if config.use_bdlora is not None:
+            from .variants import BdLoraLinearVariant
+
+            return BdLoraLinearVariant()
+
+        use_alora = config.alora_invocation_tokens is not None
+        if not config.use_dora and not use_alora:
+            return None
+
+        from .variants import ALoraLinearVariant, DoraLinearVariant
+
+        if use_alora:
+            return ALoraLinearVariant()
+        else:
+            return DoraLinearVariant()
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`list[str]`, *optional*):
+                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
+                to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.lora_A.keys():
+                base_layer = self.get_base_layer()
+                if safe_merge:
+                    # Note that safe_merge will be slower than the normal merge
+                    # because of the copy operation.
+                    orig_weight = base_layer.weight.data.clone()
+                    orig_dtype = orig_weight.dtype
+                    if active_adapter not in self.lora_variant:  # vanilla LoRA
+                        delta_weight = self.get_delta_weight(active_adapter)
+                        orig_weight += delta_weight.to(orig_dtype)
+                    else:
+                        orig_weight = self.lora_variant[active_adapter].merge_safe(self, active_adapter, orig_weight)
+
+                    if not torch.isfinite(orig_weight).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    base_layer.weight.data = orig_weight
+
+                    if self.lora_bias[active_adapter]:
+                        if getattr(base_layer, "bias", None) is None:
+                            raise RuntimeError(
+                                "Impossible to merge LoRA with `lora_bias=True` because the base layer has no bias."
+                            )
+                        new_bias = base_layer.bias + self.lora_B[active_adapter].bias * self.scaling[active_adapter]
+                        if not torch.isfinite(new_bias).all():
+                            raise ValueError(
+                                f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                            )
+                        base_layer.bias.data = new_bias.to(orig_dtype)
+
+                else:
+                    if active_adapter not in self.lora_variant:  # vanilla LoRA
+                        delta_weight = self.get_delta_weight(active_adapter)
+                        base_layer.weight.data += delta_weight
+                    else:
+                        self.lora_variant[active_adapter].merge_unsafe(self, active_adapter, base_layer.weight)
+
+                    if self.lora_bias[active_adapter]:
+                        if getattr(base_layer, "bias", None) is None:
+                            raise RuntimeError(
+                                "Impossible to merge LoRA with `lora_bias=True` because the base layer has no bias."
+                            )
+                        base_layer.bias.data += self.lora_B[active_adapter].bias * self.scaling[active_adapter]
+
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self.lora_A.keys():
+                weight = self.get_base_layer().weight
+                if active_adapter not in self.lora_variant:  # vanilla LoRA
+                    orig_dtype = weight.dtype
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    weight.data -= delta_weight.to(orig_dtype)
+                else:
+                    unmerged = self.lora_variant[active_adapter].unmerge(self, active_adapter, weight)
+                    weight.data = unmerged
+
+                if self.lora_bias[active_adapter]:
+                    self.get_base_layer().bias.data -= self.lora_B[active_adapter].bias * self.scaling[active_adapter]
+
+    def get_delta_weight(self, adapter) -> torch.Tensor:
+        """
+        Compute the delta weight for the given adapter.
+
+        Args:
+            adapter (str):
+                The name of the adapter for which the delta weight should be computed.
+        """
+        device = self.lora_B[adapter].weight.device
+        dtype = self.lora_B[adapter].weight.dtype
+
+        # In case users wants to merge the adapter weights that are in
+        # (b)float16 while being on CPU, we need to cast the weights to float32, perform the merge and then cast back to
+        # (b)float16 because some CPUs have slow bf16/fp16 matmuls.
+        cast_to_fp32 = device.type == "cpu" and (dtype == torch.float16 or dtype == torch.bfloat16)
+
+        weight_A = self.lora_A[adapter].weight
+        weight_B = self.lora_B[adapter].weight
+
+        if cast_to_fp32:
+            weight_A = weight_A.float()
+            weight_B = weight_B.float()
+
+        output_tensor = transpose(weight_B @ weight_A, self.fan_in_fan_out) * self.scaling[adapter]
+
+        if cast_to_fp32:
+            output_tensor = output_tensor.to(dtype=dtype)
+
+            # cast back the weights
+            self.lora_A[adapter].weight.data = weight_A.to(dtype)
+            self.lora_B[adapter].weight.data = weight_B.to(dtype)
+
+        return output_tensor
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        self._check_forward_args(x, *args, **kwargs)
+        adapter_names = kwargs.pop("adapter_names", None)
+        variant_kwargs = {k: kwargs.pop(k, None) for k in VARIANT_KWARG_KEYS}  # don't pass these to base_layer
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif adapter_names is not None:
+            result = self._mixed_batch_forward(x, *args, adapter_names=adapter_names, **variant_kwargs, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            result = self.base_layer(x, *args, **kwargs)
+            torch_result_dtype = result.dtype
+
+            lora_A_keys = self.lora_A.keys()
+            for active_adapter in self.active_adapters:
+                if active_adapter not in lora_A_keys:
+                    continue
+
+                lora_A = self.lora_A[active_adapter]
+                lora_B = self.lora_B[active_adapter]
+                dropout = self.lora_dropout[active_adapter]
+                scaling = self.scaling[active_adapter]
+                x = self._cast_input_dtype(x, lora_A.weight.dtype)
+                if active_adapter not in self.lora_variant:  # vanilla LoRA
+                    result = result + lora_B(lora_A(dropout(x))) * scaling
+                else:
+                    result = self.lora_variant[active_adapter].forward(
+                        self,
+                        active_adapter=active_adapter,
+                        x=x,
+                        result=result,
+                        **variant_kwargs,
+                        **kwargs,
+                    )
+
+            result = result.to(torch_result_dtype)
+
+        return result
+
+    def supports_lora_conversion(self, adapter_name: str = "default") -> bool:
+        return True
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "lora." + rep
+
+
+class _LoraEmbeddingAHolder(nn.Module):
+    """
+    A "fake" module to hold the lora_embedding_A weights for the TP hooks.
+    """
+
+    def __init__(self, lora_embedding_A_weight):
+        super().__init__()
+        self.weight = lora_embedding_A_weight.T  # lora_embedding_A shape is (r, vocab_size)
+
+
+class Embedding(nn.Module, LoraLayer):
+    # LoRA implemented in a Embedding layer
+    def __init__(
+        self,
+        base_layer: nn.Module,
+        adapter_name: str,
+        config: LoraConfig,
+        r: int = 0,
+        lora_alpha: int = 1,
+        init_lora_weights: Union[bool, str] = True,
+        **kwargs,
+    ) -> None:
+        if config.lora_bias:
+            # lora_bias=True is not supported (yet) for embedding layers, as they use nn.Parameter
+            raise ValueError(f"lora_bias={config.lora_bias} is not supported for {self.__class__.__name__}.")
+
+        super().__init__()
+        LoraLayer.__init__(self, base_layer)
+        self.fan_in_fan_out = config.fan_in_fan_out
+
+        tp_plan = getattr(base_layer, "_hf_tp_plan", None)
+
+        self.device_mesh = getattr(base_layer, "_hf_device_mesh", None)
+        self.tp_layer = None
+
+        self.input_fns = {}
+        self.output_fns = {}
+
+        if tp_plan is not None:
+            if not is_transformers_ge_v5_4_0:
+                raise RuntimeError("Tensor Parallel with LoRA is only supported for transformers v5.4.0 and above. ")
+
+            if tp_plan != "embedding_rowwise":
+                raise ValueError(
+                    f'Unsupported tensor parallel plan {tp_plan} for embedding layers. Only "embedding_rowwise" is '
+                    "supported."
+                )
+
+            from transformers.integrations.tensor_parallel import ALL_PARALLEL_STYLES
+
+            self.tp_layer = copy.deepcopy(ALL_PARALLEL_STYLES[tp_plan])
+
+        self._active_adapter = adapter_name
+        self.update_layer(
+            adapter_name,
+            r,
+            lora_alpha=lora_alpha,
+            config=config,
+        )
+
+    def resolve_lora_variant(self, *, config: LoraConfig, **kwargs) -> Optional[LoraVariant]:
+        if not config.use_dora:
+            return None
+
+        from .variants import DoraEmbeddingVariant
+
+        return DoraEmbeddingVariant()
+
+    def update_layer(
+        self,
+        adapter_name: str,
+        r: int,
+        lora_alpha: int,
+        config: LoraConfig,
+        **kwargs,
+    ) -> None:
+        lora_dropout = config.lora_dropout
+        init_lora_weights = config.init_lora_weights
+        use_rslora = config.use_rslora
+        lora_bias = config.lora_bias
+        inference_mode = config.inference_mode
+
+        if r <= 0:
+            raise ValueError(f"`r` should be a positive integer value but the value passed is {r}")
+
+        lora_variant = self.resolve_lora_variant(config=config)
+        if lora_variant is not None:
+            self.lora_variant[adapter_name] = lora_variant
+
+        self.r[adapter_name] = r
+        self.lora_alpha[adapter_name] = lora_alpha
+        if lora_dropout > 0.0:
+            lora_dropout_layer = nn.Dropout(p=lora_dropout)
+        else:
+            lora_dropout_layer = nn.Identity()
+
+        self.lora_dropout[adapter_name] = lora_dropout_layer
+        # Actual trainable parameters
+        weight_A = torch.randn((r, self.in_features))
+        weight_B = torch.randn((self.out_features, r))
+        self.lora_embedding_A[adapter_name] = nn.Parameter(weight_A)
+        self.lora_embedding_B[adapter_name] = nn.Parameter(weight_B)
+        self.lora_bias[adapter_name] = lora_bias
+
+        if use_rslora:
+            self.scaling[adapter_name] = lora_alpha / math.sqrt(r)
+        else:
+            self.scaling[adapter_name] = lora_alpha / r
+
+        self.use_rslora[adapter_name] = use_rslora
+
+        self.use_dora[adapter_name] = config.use_dora
+
+        if init_lora_weights == "loftq":
+            self.loftq_init(adapter_name)
+        elif init_lora_weights == "lora_ga":
+            # Embedding layers don't support LoRA-GA, fall back to standard initialization
+            self.reset_lora_parameters(adapter_name, True)
+        elif init_lora_weights:
+            self.reset_lora_parameters(adapter_name, init_lora_weights)
+
+        # call this before init of the lora variants
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+
+        if adapter_name in self.lora_variant:
+            self.lora_variant[adapter_name].init(self, adapter_name=adapter_name, config=config, **kwargs)
+
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+        # If there is tensor parallelism, we register the hooks for `self._embed`.
+        if self.tp_layer is not None:
+            mod = _LoraEmbeddingAHolder(self.lora_embedding_A[adapter_name])
+
+            def input_fn(inputs):
+                return self.tp_layer._prepare_input_fn(mod, inputs, self.device_mesh)
+
+            def output_fn(outputs):
+                return self.tp_layer._prepare_output_fn(mod, outputs, self.device_mesh)
+
+            self.input_fns[adapter_name] = input_fn
+            self.output_fns[adapter_name] = output_fn
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`list[str]`, *optional*):
+                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
+                to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.lora_embedding_A.keys():
+                base_layer = self.get_base_layer()
+                orig_dtype = base_layer.weight.dtype
+                if safe_merge:
+                    # Note that safe_merge will be slower than the normal merge
+                    # because of the copy operation.
+                    orig_weight = base_layer.weight.data.clone()
+                    if active_adapter not in self.lora_variant:  # vanilla LoRA
+                        orig_weight += self.get_delta_weight(active_adapter).to(orig_dtype)
+                    else:
+                        orig_weight = self.lora_variant[active_adapter].merge_safe(self, active_adapter, orig_weight)
+
+                    if not torch.isfinite(orig_weight).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    base_layer.weight.data = orig_weight
+                else:
+                    if active_adapter not in self.lora_variant:  # vanilla LoRA
+                        base_layer.weight.data += self.get_delta_weight(active_adapter).to(orig_dtype)
+                    else:
+                        self.lora_variant[active_adapter].merge_unsafe(self, active_adapter, base_layer.weight)
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            orig_dtype = self.get_base_layer().weight.dtype
+            if active_adapter in self.lora_embedding_A.keys():
+                weight = self.get_base_layer().weight
+                if active_adapter not in self.lora_variant:  # vanilla LoRA
+                    weight.data -= self.get_delta_weight(active_adapter).to(orig_dtype)
+                else:
+                    unmerged = self.lora_variant[active_adapter].unmerge(self, active_adapter, weight)
+                    weight.data = unmerged
+
+    def get_delta_weight(self, adapter) -> torch.Tensor:
+        """
+        Compute the delta weight for the given adapter.
+
+        Args:
+            adapter (str):
+                The name of the adapter for which the delta weight should be computed.
+        """
+        device = self.lora_embedding_B[adapter].device
+        dtype = self.lora_embedding_A[adapter].dtype
+
+        # In case users wants to merge the adapter weights that are in
+        # (b)float16 while being on CPU, we need to cast the weights to float32, perform the merge and then cast back to
+        # (b)float16 because some CPUs have slow bf16/fp16 matmuls.
+        cast_to_fp32 = device.type == "cpu" and (dtype == torch.float16 or dtype == torch.bfloat16)
+
+        weight_A = self.lora_embedding_A[adapter]
+        weight_B = self.lora_embedding_B[adapter]
+
+        if cast_to_fp32:
+            weight_A = weight_A.float()
+            weight_B = weight_B.float()
+
+        output_tensor = transpose(weight_B @ weight_A, True) * self.scaling[adapter]
+
+        if cast_to_fp32:
+            output_tensor = output_tensor.to(dtype=dtype)
+
+            # cast back the weights
+            self.lora_embedding_A[adapter] = weight_A.to(dtype)
+            self.lora_embedding_B[adapter] = weight_B.to(dtype)
+
+        return output_tensor
+
+    def _mixed_batch_forward(
+        self, x: torch.Tensor, *args: Any, adapter_names: list[str], **kwargs: Any
+    ) -> torch.Tensor:
+        # This is a special method that handles the case when users pass the argument `adapter_names`. This is an
+        # extra argument that allows mixing different adapters in the same batch at inference time.
+        result = self.base_layer(x, *args, **kwargs)
+
+        # Some embedding layers (e.g., Gemma3TextScaledWordEmbedding) apply scaling in their forward method.
+        # Since base_layer(x) already includes this scaling, we need to apply it to LoRA contributions too.
+        embed_scale = self._get_embed_scale()
+
+        unique_adapters = set(adapter_names)
+        sub_batch_indices_list = []
+        for adapter in unique_adapters:
+            sub_batch_indices_list.append([index for index, item in enumerate(adapter_names) if item == adapter])
+
+        for i, active_adapter in enumerate(unique_adapters):
+            if active_adapter == "__base__":
+                continue
+            if active_adapter not in self.lora_embedding_A.keys():
+                continue
+
+            embedding_A = self.lora_embedding_A[active_adapter].T
+            embedding_B = self.lora_embedding_B[active_adapter].T
+            scaling = self.scaling[active_adapter]
+
+            # getting the sub-batch, passing it to LoRA layers and updating the corresponding indices of the linear
+            # layer output
+            sub_batch = x[sub_batch_indices_list[i]]
+            after_A = self._embed(sub_batch, embedding_A)
+            adapter_output = (after_A @ embedding_B) * scaling
+
+            # Apply embed_scale to match the base layer's scaling
+            if embed_scale is not None:
+                adapter_output = adapter_output * embed_scale.to(adapter_output.dtype)
+
+            result[sub_batch_indices_list[i]] += adapter_output
+
+        return result
+
+    def _embed(
+        self,
+        input: torch.Tensor,
+        weight: torch.Tensor,
+        input_fn: Callable[[tuple[torch.Tensor]], torch.Tensor] | None = None,
+        output_fn: Callable[[torch.Tensor], torch.Tensor] | None = None,
+    ) -> torch.Tensor:
+        base_layer = self.get_base_layer()
+        if input_fn is not None:
+            input = input_fn((input,))
+        output = F.embedding(
+            input,
+            weight,
+            padding_idx=base_layer.padding_idx,
+            max_norm=base_layer.max_norm,
+            norm_type=base_layer.norm_type,
+            scale_grad_by_freq=base_layer.scale_grad_by_freq,
+            sparse=base_layer.sparse,
+        )
+        if output_fn is not None:
+            output = output_fn(output)
+        return output
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        # TODO: no dtype conversion here, unlike in Linear, is that correct?
+        self._check_forward_args(x, *args, **kwargs)
+        adapter_names = kwargs.pop("adapter_names", None)
+        variant_kwargs = {k: kwargs.pop(k, None) for k in VARIANT_KWARG_KEYS}  # don't pass these to base_layer
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif adapter_names is not None:
+            result = self._mixed_batch_forward(x, *args, adapter_names=adapter_names, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            result = self.base_layer(x, *args, **kwargs)
+            torch_result_dtype = result.dtype
+
+            # Some embedding layers (e.g., Gemma3TextScaledWordEmbedding) apply scaling in their forward method.
+            # Since base_layer(x) already includes this scaling, we need to apply it to LoRA contributions too.
+            embed_scale = self._get_embed_scale()
+
+            for active_adapter in self.active_adapters:
+                if active_adapter not in self.lora_embedding_A:
+                    continue
+
+                if active_adapter not in self.lora_variant:  # vanilla LoRA
+                    embedding_A = self.lora_embedding_A[active_adapter].T
+                    embedding_B = self.lora_embedding_B[active_adapter].T
+                    scaling = self.scaling[active_adapter]
+                    # input and ouput function hooks for TP support.
+                    input_fn = self.input_fns.get(active_adapter, None)
+                    output_fn = self.output_fns.get(active_adapter, None)
+                    after_A = self._embed(x, embedding_A, input_fn=input_fn, output_fn=output_fn)
+                    adapter_output = (after_A @ embedding_B) * scaling
+
+                    # Apply embed_scale to match the base layer's scaling
+                    if embed_scale is not None:
+                        adapter_output = adapter_output * embed_scale.to(adapter_output.dtype)
+
+                    result = result + adapter_output
+                else:
+                    result = self.lora_variant[active_adapter].forward(
+                        self,
+                        active_adapter=active_adapter,
+                        x=x,
+                        result=result,
+                        **variant_kwargs,
+                        **kwargs,
+                    )
+            result = result.to(torch_result_dtype)
+
+        return result
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "lora." + rep
+
+
+class _ConvNd(nn.Module, LoraLayer):
+    # Lora implemented in a conv(2,3)d layer
+    def __init__(
+        self,
+        base_layer: nn.Module,
+        adapter_name: str,
+        config: LoraConfig,
+        r: int = 0,
+        lora_alpha: int = 1,
+        **kwargs,
+    ) -> None:
+        super().__init__()
+        LoraLayer.__init__(self, base_layer)
+        if kwargs.get("use_alora", False):
+            raise ValueError("aLoRA does not support adapting conv layers.")
+        if base_layer.groups > 1:
+            warnings.warn("LoRA adapter added to ConvNd layer with groups > 1. Merging is not supported.")
+
+        if r % base_layer.groups != 0:
+            raise ValueError(
+                f"Targeting a {base_layer.__class__.__name__} with groups={base_layer.groups} and rank {r}. "
+                "Currently, support is limited to conv layers where the rank is divisible by groups. "
+                "Either choose a different rank or do not target this specific layer."
+            )
+
+        self._active_adapter = adapter_name
+        self._kernel_dim = base_layer.weight.dim()
+
+        self.update_layer(
+            adapter_name,
+            r,
+            lora_alpha=lora_alpha,
+            config=config,
+        )
+
+    def update_layer(
+        self,
+        adapter_name: str,
+        r: int,
+        lora_alpha: int,
+        config: LoraConfig,
+        **kwargs,
+    ) -> None:
+        lora_dropout = config.lora_dropout
+        init_lora_weights = config.init_lora_weights
+        use_rslora = config.use_rslora
+        lora_bias = config.lora_bias
+        inference_mode = config.inference_mode
+
+        if r <= 0:
+            raise ValueError(f"`r` should be a positive integer value but the value passed is {r}")
+
+        if lora_bias and (getattr(self.get_base_layer(), "bias", None) is None):
+            warnings.warn(
+                f"`lora_bias=True` was passed but the targeted layer of type {type(self.get_base_layer()).__name__} "
+                "has no bias. This means that merging LoRA weights won't be possible.",
+                PeftWarning,
+            )
+
+        lora_variant = self.resolve_lora_variant(config=config)
+        if lora_variant is not None:
+            self.lora_variant[adapter_name] = lora_variant
+
+        self.r[adapter_name] = r
+        self.lora_alpha[adapter_name] = lora_alpha
+        if lora_dropout > 0.0:
+            lora_dropout_layer = nn.Dropout(p=lora_dropout)
+        else:
+            lora_dropout_layer = nn.Identity()
+
+        self.lora_dropout[adapter_name] = lora_dropout_layer
+        # Actual trainable parameters
+        base_layer = self.get_base_layer()
+        kernel_size = base_layer.kernel_size
+        stride = base_layer.stride
+        padding = base_layer.padding
+        conv_layer = type(base_layer)
+        out_kernel = out_stride = (1,) * (self._kernel_dim - 2)
+        self.lora_A[adapter_name] = conv_layer(self.in_features, r, kernel_size, stride, padding, bias=False)
+        self.lora_B[adapter_name] = conv_layer(
+            r, self.out_features, out_kernel, out_stride, groups=base_layer.groups, bias=lora_bias
+        )
+        self.lora_bias[adapter_name] = lora_bias
+
+        if use_rslora:
+            self.scaling[adapter_name] = lora_alpha / math.sqrt(r)
+        else:
+            self.scaling[adapter_name] = lora_alpha / r
+
+        self.use_rslora[adapter_name] = use_rslora
+
+        self.use_dora[adapter_name] = config.use_dora
+
+        if init_lora_weights == "loftq":
+            self.loftq_init(adapter_name)
+        elif init_lora_weights == "lora_ga":
+            # Conv layers don't support LoRA-GA, fall back to standard initialization
+            self.reset_lora_parameters(adapter_name, True)
+        elif init_lora_weights:
+            self.reset_lora_parameters(adapter_name, init_lora_weights)
+
+        # call this before init of the lora variants
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+
+        if adapter_name in self.lora_variant:
+            self.lora_variant[adapter_name].init(self, adapter_name=adapter_name, config=config, **kwargs)
+
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+    def _get_dora_factor_view(self):
+        return (-1,) + (1,) * (self._kernel_dim - 1)
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights inside the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`list[str]`, *optional*):
+                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
+                to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.lora_A.keys():
+                base_layer = self.get_base_layer()
+                orig_dtype = base_layer.weight.dtype
+
+                if base_layer.groups > 1:
+                    # https://github.com/huggingface/peft/pull/2403
+                    raise NotImplementedError("Merging is not supported for _ConvNd layers with groups > 1!")
+
+                if safe_merge:
+                    # Note that safe_merge will be slower than the normal merge
+                    # because of the copy operation.
+                    orig_weight = base_layer.weight.data.clone()
+                    if active_adapter not in self.lora_variant:  # vanilla LoRA
+                        delta_weight = self.get_delta_weight(active_adapter)
+                        orig_weight += delta_weight.to(orig_dtype)
+                    else:
+                        orig_weight = self.lora_variant[active_adapter].merge_safe(self, active_adapter, orig_weight)
+
+                    if not torch.isfinite(orig_weight).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    base_layer.weight.data = orig_weight
+
+                    if self.lora_bias[active_adapter]:
+                        if getattr(base_layer, "bias", None) is None:
+                            raise RuntimeError(
+                                "Impossible to merge LoRA with `lora_bias=True` because the base layer has no bias."
+                            )
+                        new_bias = base_layer.bias + self.lora_B[active_adapter].bias * self.scaling[active_adapter]
+                        if not torch.isfinite(new_bias).all():
+                            raise ValueError(
+                                f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                            )
+                        base_layer.bias.data = new_bias.to(orig_dtype)
+
+                else:
+                    if active_adapter not in self.lora_variant:  # vanilla LoRA
+                        delta_weight = self.get_delta_weight(active_adapter)
+                        base_layer.weight.data += delta_weight.to(orig_dtype)
+                    else:
+                        self.lora_variant[active_adapter].merge_unsafe(self, active_adapter, base_layer.weight)
+
+                    if self.lora_bias[active_adapter]:
+                        if getattr(base_layer, "bias", None) is None:
+                            raise RuntimeError(
+                                "Impossible to merge LoRA with `lora_bias=True` because the base layer has no bias."
+                            )
+                        base_layer.bias.data += self.lora_B[active_adapter].bias * self.scaling[active_adapter]
+
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self.lora_A.keys():
+                weight = self.get_base_layer().weight
+                if active_adapter not in self.lora_variant:  # vanilla LoRA
+                    orig_dtype = weight.dtype
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    weight.data -= delta_weight.to(orig_dtype)
+                else:
+                    unmerged = self.lora_variant[active_adapter].unmerge(self, active_adapter, weight)
+                    weight.data = unmerged
+
+                if self.lora_bias[active_adapter]:
+                    self.get_base_layer().bias.data -= self.lora_B[active_adapter].bias * self.scaling[active_adapter]
+
+    def get_delta_weight(self, adapter) -> torch.Tensor:
+        """
+        Compute the delta weight for the given adapter.
+
+        Args:
+            adapter (str):
+                The name of the adapter for which the delta weight should be computed.
+        """
+        device = self.lora_B[adapter].weight.device
+        dtype = self.lora_A[adapter].weight.dtype
+
+        # In case users wants to merge the adapter weights that are in
+        # (b)float16 while being on CPU, we need to cast the weights to float32, perform the merge and then cast back to
+        # (b)float16 because some CPUs have slow bf16/fp16 matmuls.
+        cast_to_fp32 = device.type == "cpu" and (dtype == torch.float16 or dtype == torch.bfloat16)
+
+        weight_A = self.lora_A[adapter].weight
+        weight_B = self.lora_B[adapter].weight
+
+        if cast_to_fp32:
+            weight_A = weight_A.float()
+            weight_B = weight_B.float()
+
+        # https://github.com/bmaltais/kohya_ss/blob/feb6728762a8f463d15ba936d189d4c3abfaa1ab/networks/lora.py#L117
+        if self.get_base_layer().weight.size()[2:4] == (1, 1):
+            # conv2d 1x1
+            output_tensor = (weight_B.squeeze(3).squeeze(2) @ weight_A.squeeze(3).squeeze(2)).unsqueeze(2).unsqueeze(
+                3
+            ) * self.scaling[adapter]
+        else:
+            output_tensor = self.conv_fn(weight_A.transpose(0, 1), weight_B)
+
+            if self.get_base_layer().groups > 1:
+                output_tensor = output_tensor * self.scaling[adapter]
+            else:
+                output_tensor = output_tensor.transpose(0, 1) * self.scaling[adapter]
+
+        if cast_to_fp32:
+            output_tensor = output_tensor.to(dtype=dtype)
+
+            # cast back the weights
+            self.lora_A[adapter].weight.data = weight_A.to(dtype)
+            self.lora_B[adapter].weight.data = weight_B.to(dtype)
+
+        return output_tensor
+
+    def forward(self, x: torch.Tensor, *args, **kwargs) -> torch.Tensor:
+        self._check_forward_args(x, *args, **kwargs)
+        adapter_names = kwargs.pop("adapter_names", None)
+        variant_kwargs = {k: kwargs.pop(k, None) for k in VARIANT_KWARG_KEYS}  # don't pass these to base_layer
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif adapter_names is not None:
+            result = self._mixed_batch_forward(x, *args, adapter_names=adapter_names, **kwargs)
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+
+        else:
+            result = self.base_layer(x, *args, **kwargs)
+            torch_result_dtype = result.dtype
+
+            for active_adapter in self.active_adapters:
+                if active_adapter not in self.lora_A.keys():
+                    continue
+                lora_A = self.lora_A[active_adapter]
+                lora_B = self.lora_B[active_adapter]
+                dropout = self.lora_dropout[active_adapter]
+                scaling = self.scaling[active_adapter]
+                x = self._cast_input_dtype(x, lora_A.weight.dtype)
+
+                if active_adapter not in self.lora_variant:  # vanilla LoRA
+                    result = result + lora_B(lora_A(dropout(x))) * scaling
+                else:
+                    result = self.lora_variant[active_adapter].forward(
+                        self,
+                        active_adapter=active_adapter,
+                        x=x,
+                        result=result,
+                        **variant_kwargs,
+                        **kwargs,
+                    )
+
+            result = result.to(torch_result_dtype)
+        return result
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "lora." + rep
+
+
+class Conv2d(_ConvNd):
+    # Lora implemented in a conv2d layer
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        if not self._kernel_dim == 4:
+            raise ValueError(f"Conv2d layer kernel must have 4 dimensions, not {self._kernel_dim}")
+        self.conv_fn = F.conv2d
+
+    def resolve_lora_variant(self, *, config: LoraConfig, **kwargs) -> Optional[LoraVariant]:
+        if not config.use_dora:
+            return None
+
+        from .variants import DoraConv2dVariant
+
+        return DoraConv2dVariant()
+
+
+class Conv1d(_ConvNd):
+    # Lora implemented in a conv1d layer
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        if not self._kernel_dim == 3:
+            raise ValueError(f"Conv1d layer kernel must have 3 dimensions, not {self._kernel_dim}")
+        self.conv_fn = F.conv1d
+
+    def resolve_lora_variant(self, *, config: LoraConfig, **kwargs) -> Optional[LoraVariant]:
+        if not config.use_dora:
+            return None
+
+        from .variants import DoraConv1dVariant
+
+        return DoraConv1dVariant()
+
+
+class Conv3d(_ConvNd):
+    # Lora implemented in a conv3d layer
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        if not self._kernel_dim == 5:
+            raise ValueError(f"Conv3d layer kernel must have 5 dimensions, not {self._kernel_dim}")
+        self.conv_fn = F.conv3d
+
+    def resolve_lora_variant(self, *, config: LoraConfig, **kwargs) -> Optional[LoraVariant]:
+        if not config.use_dora:
+            return None
+
+        from .variants import DoraConv3dVariant
+
+        return DoraConv3dVariant()
+
+
+class MultiheadAttention(nn.Module, LoraLayer):
+    """LoRA implemented in a multihead attention layer
+
+    This is currently only implemented for the case of `_qkv_same_embed_dim = True`, i.e. query, key, and value having
+    the same dimension.
+
+    Note: LoRA is applied to both the in_proj (query/key/value) and out_proj. There is currently no way to specify only
+    one of them. Don't try to apply LoRA to the out_proj of MultiheadAttention by targeting that layer specifically,
+    since the forward method of that layer is not being used, hence the LoRA adapter would be ignored.
+
+    This is a little bit hacky because of the way that MultiheadAttention is implemented in PyTorch: There are no
+    `nn.Linear` layers which we can hook onto or, in case of output projection, `.forward` is not used. This
+    implementation works around these problems by merging the weights before the forward call and unmerging them after
+    the forward call.
+    """
+
+    def __init__(
+        self,
+        base_layer,
+        adapter_name: str,
+        config: LoraConfig,
+        r: int = 0,
+        lora_alpha: int = 1,
+        **kwargs,
+    ) -> None:
+        # TODO work with separate weights
+        if not getattr(base_layer, "_qkv_same_embed_dim", True):
+            # default for this value appears to be True:
+            # https://github.com/pytorch/pytorch/blob/701ba5203fe68d55d655bd4d6c008be94cf34ea5/torch/nn/modules/activation.py#L1128-L1130
+            raise ValueError(
+                f"Only same embed for query/key/value is supported as of now for {self.__class__.__name__}."
+            )
+        if config.use_dora:
+            # TODO: probably not so hard to implement
+            raise ValueError(f"{self.__class__.__name__} does not support DoRA (yet), please set use_dora to False")
+        if kwargs.get("use_alora", False):
+            raise ValueError(f"{self.__class__.__name__} does not support aLoRA (yet), please set use_alora to False")
+        super().__init__()
+        LoraLayer.__init__(self, base_layer, **kwargs)
+
+        # Note: LoRA is applied to both in_proj and out_proj. There is currently no way to only specify one of them.
+        if isinstance(base_layer.out_proj, nn.Linear):
+            self.base_layer.out_proj = Linear(
+                base_layer.out_proj,
+                adapter_name,
+                r=r,
+                lora_alpha=lora_alpha,
+                config=config,
+                **kwargs,
+            )
+        else:
+            raise ValueError(f"out_proj must be an instance of nn.Linear for {self.__class__.__name__}.")
+
+        self._active_adapter = adapter_name
+        self.update_layer(adapter_name, r, lora_alpha=lora_alpha, config=config)
+
+    @property
+    def embed_dim(self) -> int:
+        return self.get_base_layer().embed_dim
+
+    @property
+    def kdim(self) -> Optional[int]:
+        return self.get_base_layer().kdim
+
+    @property
+    def vdim(self) -> Optional[int]:
+        return self.get_base_layer().vdim
+
+    @property
+    def _qkv_same_embed_dim(self) -> bool:
+        return self.get_base_layer()._qkv_same_embed_dim
+
+    @property
+    def num_heads(self) -> int:
+        return self.get_base_layer().num_heads
+
+    @property
+    def dropout(self) -> float:
+        return self.get_base_layer().dropout
+
+    @property
+    def batch_first(self) -> bool:
+        return self.get_base_layer().batch_first
+
+    @property
+    def head_dim(self) -> int:
+        return self.get_base_layer().head_dim
+
+    @property
+    def in_proj_weight(self) -> nn.Parameter:
+        return self.get_base_layer().in_proj_weight
+
+    @property
+    def in_proj_bias(self) -> nn.Parameter:
+        return self.get_base_layer().in_proj_bias
+
+    @property
+    def out_proj(self) -> nn.Module:
+        return self.get_base_layer().out_proj.get_base_layer()
+
+    @property
+    def bias_k(self) -> Optional[nn.Parameter]:
+        return self.get_base_layer().bias_k
+
+    @property
+    def bias_v(self) -> Optional[nn.Parameter]:
+        return self.get_base_layer().bias_v
+
+    def merge_masks(self, *args, **kwargs) -> tuple[Optional[torch.Tensor], Optional[int]]:
+        return self.get_base_layer().merge_masks(*args, **kwargs)
+
+    @property
+    def add_zero_attn(self) -> bool:
+        return self.get_base_layer().add_zero_attn
+
+    def update_layer(self, *args, **kwargs) -> None:
+        super().update_layer(*args, **kwargs)
+        # Note: LoRA is applied to both in_proj and out_proj. There is currently no way to only specify one of them.
+        self.base_layer.out_proj.update_layer(*args, **kwargs)
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        """
+        Merge the active adapter weights into the base weights
+
+        Args:
+            safe_merge (`bool`, *optional*):
+                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
+                before merging the weights. This is useful if you want to check if the merge operation will produce
+                NaNs. Defaults to `False`.
+            adapter_names (`List[str]`, *optional*):
+                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
+                to `None`.
+        """
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        # Implementation follows this:
+        # https://github.com/Baijiong-Lin/LoRA-Torch/blob/4bfed6820b64fcf47064c30f30606a190a4f0d2e/loratorch/layers.py#L73-L79
+        # Notably, instead of mutating the weight, we delete the original weight and replace it by the merged weight
+        # TODO: work with separate weights
+        for active_adapter in adapter_names:
+            if active_adapter in self.lora_A.keys():
+                base_layer = self.get_base_layer()
+                orig_dtype = base_layer.out_proj.weight.dtype
+                if safe_merge:
+                    # TODO: work with separate weights
+                    # merging in_proj (nn.Parameter)
+                    orig_weight_in = base_layer.in_proj_weight.data.detach().clone()
+                    orig_weight_in += self.get_delta_weight(active_adapter).to(orig_dtype)
+                    if not torch.isfinite(orig_weight_in).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    # merging out_proj (subclass of nn.Linear)
+                    orig_weight_out = base_layer.out_proj.weight.data.detach().clone()
+                    orig_weight_out += base_layer.out_proj.get_delta_weight(active_adapter).to(orig_dtype)
+                    if not torch.isfinite(orig_weight_out).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    # unregister parameter implicitly and overwrite using merged weights; gradients are computed after
+                    # forward and, thus, after unmerging (see forward()), therefore this is safe to do.
+                    del base_layer.in_proj_weight
+                    base_layer.in_proj_weight = orig_weight_in
+
+                    del base_layer.out_proj.get_base_layer().weight
+                    base_layer.out_proj.get_base_layer().weight = orig_weight_out
+                    base_layer.out_proj.merge(adapter_names=[active_adapter])
+                else:
+                    # merging in_proj (nn.Parameter)
+                    # TODO: work with separate weights
+                    delta_weight = self.get_delta_weight(active_adapter).to(orig_dtype)
+                    weight_merged = base_layer.in_proj_weight.data.detach() + delta_weight
+
+                    # unregister parameter implicitly and overwrite using merged weights; gradients are computed after
+                    # forward and, thus, after unmerging (see forward()), therefore this is safe to do.
+                    del base_layer.in_proj_weight
+                    base_layer.in_proj_weight = weight_merged
+
+                    # merging out_proj (subclass of nn.Linear)
+                    delta_weight = base_layer.out_proj.get_delta_weight(active_adapter).to(orig_dtype)
+                    weight_merged = base_layer.out_proj.weight.data.detach() + delta_weight
+                    del base_layer.out_proj.get_base_layer().weight
+                    base_layer.out_proj.get_base_layer().weight = weight_merged
+                    base_layer.out_proj.merge(adapter_names=[active_adapter])
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        """
+        This method unmerges all merged adapter layers from the base weights.
+        """
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+
+        # TODO work with separate weights
+        base_layer = self.get_base_layer()
+        orig_dtype = base_layer.out_proj.base_layer.weight.dtype
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self.lora_A.keys():
+                # Ensure that requires_grad=False for the base weights after unmerging. This may not matter since
+                # requires_grad was False when the optimizer was initialized, but still let's try to be correct here.
+
+                # in_proj
+                delta_weight = self.get_delta_weight(active_adapter).to(orig_dtype)
+                old_weight = base_layer.in_proj_weight.data - delta_weight
+                del base_layer.in_proj_weight
+                base_layer.register_parameter("in_proj_weight", nn.Parameter(old_weight, requires_grad=False))
+
+                # out_proj
+                delta_weight = base_layer.out_proj.get_delta_weight(active_adapter).to(orig_dtype)
+                old_weight = base_layer.out_proj.base_layer.weight.data - delta_weight
+                del base_layer.out_proj.base_layer.weight
+                base_layer.out_proj.base_layer.register_parameter(
+                    "weight", nn.Parameter(old_weight, requires_grad=False)
+                )
+
+        self.get_base_layer().out_proj.unmerge()
+
+    def unload_and_optionally_merge_module(
+        self, merge: bool, safe_merge: bool, adapter_names: Optional[list[str]]
+    ) -> nn.MultiheadAttention:
+        """
+        Merging and unloading of the MultiheadAttention module
+
+        This requires an extra step for MultiheadAttention, which is why there is this special method instead of
+        relying on the normal merge_and_unload code path.
+        """
+        if merge:
+            self.merge(safe_merge=safe_merge, adapter_names=adapter_names)
+        base_layer = self.get_base_layer()
+
+        # extra steps: re-register weights, take care of out_proj layer
+        # in_proj
+        weight = base_layer.in_proj_weight
+        del base_layer.in_proj_weight
+        base_layer.register_parameter("in_proj_weight", nn.Parameter(weight.data, requires_grad=weight.requires_grad))
+
+        # out_proj
+        out_proj_layer = base_layer.out_proj.get_base_layer()
+        weight = out_proj_layer.weight
+        del out_proj_layer.weight
+        out_proj_layer.register_parameter("weight", nn.Parameter(weight.data, requires_grad=weight.requires_grad))
+
+        base_layer.out_proj = out_proj_layer
+        return base_layer
+
+    def get_delta_weight(self, adapter) -> torch.Tensor:
+        """
+        Compute the delta weight for the given adapter.
+
+        Args:
+            adapter (str):
+                The name of the adapter for which the delta weight should be computed.
+        """
+        device = self.lora_B[adapter].weight.device
+        dtype = self.lora_B[adapter].weight.dtype
+
+        # In case users wants to merge the adapter weights that are in
+        # float16 while being on CPU, we need to cast the weights to float32, perform the merge and then cast back to
+        # float16 because the `@` and matmul operation in general is not supported in torch + cpu + fp16.
+        cast_to_fp32 = device.type == "cpu" and dtype == torch.float16
+
+        weight_A = self.lora_A[adapter].weight
+        weight_B = self.lora_B[adapter].weight
+
+        if cast_to_fp32:
+            weight_A = weight_A.float()
+            weight_B = weight_B.float()
+
+        output_tensor = (weight_B @ weight_A) * self.scaling[adapter]
+
+        if cast_to_fp32:
+            output_tensor = output_tensor.to(dtype=dtype)
+
+            # cast back the weights
+            self.lora_A[adapter].weight.data = weight_A.to(dtype)
+            self.lora_B[adapter].weight.data = weight_B.to(dtype)
+
+        return output_tensor
+
+    def _check_forward_args(self, x, *args, **kwargs):
+        if "adapter_names" in kwargs:
+            raise TypeError(f"lora.{self.__class__.__name__} does not support mixed adapter batches.")
+        super()._check_forward_args(x, *args, **kwargs)
+
+    def forward(self, query: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        previous_dtype = query.dtype
+        self._check_forward_args(query, *args, **kwargs)
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(query, *args, **kwargs)
+        elif self.merged:
+            result = self.base_layer(query, *args, **kwargs)
+        else:
+            out_proj = self.get_base_layer().out_proj
+            if out_proj.active_adapters != self.active_adapters:
+                # We have a case that in_proj and out_proj have diverging merged adapters. We cannot
+                # really deal with this correctly, thus it's better to raise than possibly create a hard to debug mess
+                cls_name = self.get_base_layer().__class__.__name__
+                raise ValueError(
+                    f"The out_proj layer of {cls_name} has merged layers but {cls_name} itself doesn't; please ensure "
+                    "that either both or none have merged layers"
+                )
+
+            # Merge all adapters that are active for this module, i.e. the LoRA weights for in_proj and out_proj.
+            # in_proj uses nn.Parameters, therefore, there is no forward method to be used and we have to explicitly
+            # merge for the LoRA weights to have an effect:
+            # https://github.com/pytorch/pytorch/blob/6ebb26d572d5fcdc6ac0d1297bdf8d1eb5d20722/torch/nn/modules/activation.py#L1020
+            # For out_proj, we have an nn.Linear (or rather: NonDynamicallyQuantizableLinear), but its forward method
+            # is not used:
+            # https://github.com/pytorch/pytorch/blob/6ebb26d572d5fcdc6ac0d1297bdf8d1eb5d20722/torch/nn/modules/activation.py#L1267-L1271
+            # Therefore, its LoRA weights also need to be merged to have an effect.
+            active_adapters = [a for a in self.active_adapters if a in self.lora_A]
+            try:
+                self.merge(adapter_names=active_adapters)
+                result = self.base_layer(query, *args, **kwargs)
+            finally:
+                # it's safe to call unmerge(), which unmerges all adapters, because we checked that not self.merged,
+                # i.e. there is was no merged layer before
+                self.unmerge()
+
+        result = (result[0].to(previous_dtype), result[1].to(previous_dtype) if result[1] is not None else result[1])
+        return result
+
+    # The decorator is needed in case low_cpu_mem_usage=True is used, as we don't want the base layer weights to be
+    # moved to meta device. This requires the use of PEFT's implementation of init_empty_weight instead of using the one
+    # from accelerate.
+    @skip_init_on_device
+    def _restore_weights(self):
+        # Restore the weights as registered parameters on the base layer.
+        # This is necessary because the way that weights are merged/unmerged (which is necessary for forward to work
+        # correctly), the Module "forgets" these attributes. Therefore, we need to call register_parameter explicitly.
+        # We cannot call register_parameter for merging/unmerging because that cuts them off from the autograd graph.
+        # Note that this is hacky, since we need to ensure that _restore_weights is called by each method that needs it.
+
+        # in_proj
+        # TODO work with separate weights
+        base_layer = self.get_base_layer()
+        weight = base_layer.in_proj_weight
+        del base_layer.in_proj_weight
+        base_layer.register_parameter("in_proj_weight", nn.Parameter(weight.data, requires_grad=weight.requires_grad))
+
+        # out_proj
+        base_layer = base_layer.out_proj.get_base_layer()
+        weight = base_layer.weight
+        del base_layer.weight
+        base_layer.register_parameter("weight", nn.Parameter(weight.data, requires_grad=weight.requires_grad))
+
+    def state_dict(self, *args, **kwargs):
+        self._restore_weights()
+        return super().state_dict(*args, **kwargs)
+
+    def named_modules(self, *args, **kwargs):
+        # Note: no need to also implement modules(), as modules() calls named_modules() under the hood
+        self._restore_weights()
+        return super().named_modules(*args, **kwargs)
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        return "lora." + rep
+
+
+class _LoraParameterProxy(nn.Module):
+    """This proxies an `nn.Parameter` that is targeted with LoRA.
+    Intended to be used in conjunction with `nn.utils.parametrize`, see `ParamWrapper`.
+    """
+
+    def __init__(self, delta_weight):
+        super().__init__()
+        self.delta_weight = delta_weight
+
+    @staticmethod
+    def _low_prec_add(x, y):
+        # addition in fp8 is not directly supported, need to use a higher precision
+        orig_dtype = x.dtype
+        upcast_dtype = y.dtype
+        if upcast_dtype not in ALLOWED_COMPUTE_DTYPES:
+            raise RuntimeError(
+                f"There is an attempt to upcast the targeted parameter to {upcast_dtype} "
+                f"but the only supported are: {ALLOWED_COMPUTE_DTYPES}."
+            )
+
+        # this operation can be quite costly
+        x = x.to(upcast_dtype)
+        z = x + y
+        # clamp to valid range before casting down, as this is *not* performed automatically and can thus result in NANs
+        info = torch.finfo(orig_dtype)
+        z = z.clamp(min=info.min, max=info.max)
+        return z.to(orig_dtype)
+
+    def forward(self, W):
+        if any(getattr(torch, dtype_name, None) == W.dtype for dtype_name in UPCAST_DTYPES):
+            return self._low_prec_add(W, self.delta_weight)
+        return W + self.delta_weight
+
+
+# copied from:
+# https://github.com/pytorch/pytorch/blob/5e386eec9426f174eea130c0c012d9f65ebe65fb/torch/nn/utils/parametrize.py#L75-L79
+def _register_parameter_or_buffer(module, name, X):
+    if isinstance(X, nn.Parameter):
+        module.register_parameter(name, X)
+    else:
+        module.register_buffer(name, X)
+
+
+class ParamWrapper(nn.Module, LoraLayer):
+    """A LoRA wrapper for `nn.Parameter`. This layer is dispatched if users target a parameter directly with
+    `lora_config.target_parameters`
+        Note:
+        - When accessing the wrapped nn.Parameter directly, e.g. via `module.weight`, the LoRA weights are *not*
+          applied.
+        - It is currently not implemented to target multiple parameters on the same module. To achieve this, it is
+          currently required to create a separate LoRA adapter (with another adapter name) and activate both at the
+          same time.
+    """
+
+    def __init__(
+        self,
+        base_layer,
+        adapter_name: str,
+        parameter_name: str,
+        config: LoraConfig,
+        r: int = 0,
+        lora_alpha: int = 1,
+        is_target_conv_1d_layer: bool = False,
+        **kwargs,
+    ) -> None:
+        self.parameter_name = parameter_name
+        super().__init__()
+        LoraLayer.__init__(self, base_layer, **kwargs)
+
+        if config.lora_dropout:
+            # It's not possible to factor out x from lora_B(lora_A(dropout(x))), so dropout can't be correctly
+            # implemented
+            raise ValueError(f"lora.{self.__class__.__name__} does not work with lora_dropout != 0.")
+        if config.fan_in_fan_out:
+            raise ValueError(f"lora.{self.__class__.__name__} does not work with fan_in_fan_out.")
+        if config.lora_bias:
+            raise ValueError(f"lora.{self.__class__.__name__} does not work with lora_bias=True.")
+        if config.use_dora:
+            raise ValueError(f"lora.{self.__class__.__name__} does not work with use_dora=True.")
+        if is_target_conv_1d_layer:
+            raise ValueError(f"lora.{self.__class__.__name__} does not work with is_target_conv_1d_layer=True.")
+
+        self.fan_in_fan_out = config.fan_in_fan_out
+        self._did_swap_in_out_features = False  # ensure we swap only once
+        self._active_adapter = adapter_name
+        self.update_layer(
+            adapter_name,
+            r,
+            lora_alpha=lora_alpha,
+            config=config,
+        )
+
+    def _get_in_out_features(self, module: nn.Module) -> tuple[int, int] | tuple[None, None]:
+        # For ParamWrapper, we don't derive the in_features and out_features based on the base layer type, but directly
+        # from the targeted parameter.
+        param = self.get_param()
+        if param.ndim == 3:
+            num_experts, in_features, out_features = param.shape
+        else:
+            num_experts, in_features, out_features = 1, param.shape[1], param.shape[0]
+        if param.ndim not in (2, 3):
+            raise ValueError(
+                f"lora.{self.__class__.__name__} was initialized with {param.ndim} dimensional Parameter, but only 2d "
+                "and 3d are supported."
+            )
+        # we have to store the num_experts attribute here, as the parent class only stores in_features and out_features.
+        self.num_experts = num_experts
+        return in_features, out_features
+
+    def update_layer(
+        self,
+        adapter_name: str,
+        r: int,
+        lora_alpha: int,
+        config: LoraConfig,
+        **kwargs,
+    ) -> None:
+        # same method as in lora.Linear but taking into account that there can be multiple experts (3d parameter)
+        lora_dropout = config.lora_dropout
+        init_lora_weights = config.init_lora_weights
+        use_rslora = config.use_rslora
+        lora_bias = config.lora_bias
+        inference_mode = config.inference_mode
+        # This code works for linear layers, override for other layer types
+        if r <= 0:
+            raise ValueError(f"`r` should be a positive integer value but the value passed is {r}")
+
+        lora_variant = self.resolve_lora_variant(config=config)
+        if lora_variant is not None:
+            raise ValueError(f"lora.{self.__class__.__name__} does not work with LoRA variants like DoRA.")
+
+        # for some MoE layers, the order is (experts, out_features, in_features)
+        is_transposed = getattr(self.get_base_layer(), "is_transposed", False)
+        swap_in_out_features = (self.get_param().ndim == 3) and not is_transposed
+        if swap_in_out_features and not self._did_swap_in_out_features:
+            self.in_features, self.out_features = self.out_features, self.in_features
+            self._did_swap_in_out_features = True
+
+        self.r[adapter_name] = r
+        self.lora_alpha[adapter_name] = lora_alpha
+        if lora_dropout > 0.0:
+            # It's not possible to factor out x from lora_B(lora_A(dropout(x))), so dropout can't be correctly
+            # implemented
+            raise ValueError(f"lora.{self.__class__.__name__} does not work with lora_dropout != 0.")
+        else:
+            lora_dropout_layer = nn.Identity()
+
+        self.lora_dropout.update(nn.ModuleDict({adapter_name: lora_dropout_layer}))
+        # Actual trainable parameters
+        # Difference to normal update_layer: consider experts. LoRA layers still use nn.Linear for consistency with
+        # lora.Linear.
+        self.lora_A[adapter_name] = nn.Linear(self.in_features, r * self.num_experts, bias=False)
+        self.lora_B[adapter_name] = nn.Linear(r * self.num_experts, self.out_features, bias=lora_bias)
+        self.lora_bias[adapter_name] = lora_bias
+
+        if use_rslora:
+            self.scaling[adapter_name] = lora_alpha / math.sqrt(r)
+        else:
+            self.scaling[adapter_name] = lora_alpha / r
+
+        self.use_rslora[adapter_name] = use_rslora
+
+        self.use_dora[adapter_name] = config.use_dora
+
+        # for inits that require access to the base weight, use gather_param_ctx so that the weight is gathered when using DeepSpeed
+        if isinstance(init_lora_weights, str) and init_lora_weights.startswith("pissa"):
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.pissa_init(adapter_name, init_lora_weights)
+        elif isinstance(init_lora_weights, str) and init_lora_weights.startswith("corda"):
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.corda_init(adapter_name, init_lora_weights)
+        elif isinstance(init_lora_weights, str) and init_lora_weights.lower() == "olora":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.olora_init(adapter_name)
+        elif init_lora_weights == "loftq":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.loftq_init(adapter_name)
+        elif init_lora_weights == "eva":
+            nn.init.zeros_(self.lora_B[adapter_name].weight)
+        elif init_lora_weights == "orthogonal":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.orthogonal_init(adapter_name)
+        elif init_lora_weights == "lora_ga":
+            with gather_params_ctx(self.get_base_layer().weight):
+                self.lora_ga_init(adapter_name, config.lora_ga_config)
+        elif init_lora_weights:
+            self.reset_lora_parameters(adapter_name, init_lora_weights)
+        # call this before init of the lora variants
+        self._move_adapter_to_device_of_base_layer(adapter_name)
+
+        if adapter_name in self.lora_variant:
+            self.lora_variant[adapter_name].init(self, config=config, **kwargs)
+
+        self.set_adapter(self.active_adapters, inference_mode=inference_mode)
+
+    def _move_adapter_to_device_of_base_layer(self, adapter_name: str, device: Optional[torch.device] = None) -> None:
+        """
+        Move the adapter of the given name to the device of the base layer. Needs special handling for nn.Parameter
+        """
+        device = self.get_param().device
+        meta = torch.device("meta")
+        param = self.get_param()
+
+        for adapter_layer_name in self.adapter_layer_names + self.other_param_names:
+            adapter_layer = getattr(self, adapter_layer_name, None)
+            if not isinstance(adapter_layer, (nn.ModuleDict, nn.ParameterDict, BufferDict)):
+                continue
+            if adapter_name not in adapter_layer:
+                continue
+            if any(p.device == meta for p in adapter_layer.parameters()):
+                continue
+
+            if param.dtype.is_floating_point or param.dtype.is_complex:
+                adapter_layer[adapter_name] = adapter_layer[adapter_name].to(device, dtype=param.dtype)
+            else:
+                adapter_layer[adapter_name] = adapter_layer[adapter_name].to(device)
+
+    def get_param(self):
+        param = getattr(self.get_base_layer(), self.parameter_name)
+        return param
+
+    def get_delta_weight(self, adapter_name, *args, **kwargs):
+        if self.num_experts == 1:
+            # could actually be a normal layer or experts stacked block-diagonally, acting like a single layer
+            delta_weight = Linear.get_delta_weight(self, adapter_name, *args, **kwargs)
+        else:
+            weight_A = self.lora_A[adapter_name].weight
+            weight_B = self.lora_B[adapter_name].weight
+            # shape: experts x rank x in_features
+            weight_A = weight_A.reshape(self.num_experts, -1, weight_A.shape[-1])
+            # shape: out_features x rank x experts
+            weight_B = weight_B.reshape(weight_B.shape[0], -1, self.num_experts)
+            # fan_in_fan_out must be False, so no transpose call here
+            if not self._did_swap_in_out_features:
+                delta_weight = torch.einsum("o r e, e r i -> e i o", weight_B, weight_A) * self.scaling[adapter_name]
+            else:
+                # for some MoE layers, the order is (experts, out_features, in_features)
+                delta_weight = torch.einsum("o r e, e r i -> e o i", weight_B, weight_A) * self.scaling[adapter_name]
+
+        param = self.get_param()
+        if param.dtype in ALLOWED_COMPUTE_DTYPES:
+            delta_weight = delta_weight.to(param.device, param.dtype)
+        else:
+            # don't cast dW to weight dtype if it is in torch.float8_e4m3fn etc. as these low precision dtypes because
+            # we want to perform the W+dW addition in high precision before downcasting, see _low_prec_add.
+            delta_weight = delta_weight.to(param.device)
+        return delta_weight
+
+    @contextmanager
+    def _activate_lora(self, active_adapters: list[str]):
+        if not active_adapters or not any(adapter in self.lora_A for adapter in active_adapters):
+            # no active adapters for this layer
+            yield
+            return
+
+        delta_weight = None
+        for active_adapter in active_adapters:
+            if active_adapter not in self.lora_A:
+                continue
+            if delta_weight is None:
+                delta_weight = self.get_delta_weight(active_adapter)
+            else:
+                delta_weight = delta_weight + self.get_delta_weight(active_adapter)
+
+        base_layer = self.get_base_layer()
+        requires_grad_before = self.get_param().requires_grad
+        nn.utils.parametrize.register_parametrization(
+            base_layer, self.parameter_name, _LoraParameterProxy(delta_weight)
+        )
+        # set requires_grad, as it defaults to False
+        base_layer.parametrizations[self.parameter_name].original.requires_grad_(requires_grad_before)
+        try:
+            with nn.utils.parametrize.cached():
+                yield
+        finally:
+            self._remove_parametrizations()
+
+    def _remove_parametrizations(self):
+        # Remove the parametrization of this specific parameter
+        base_layer = self.get_base_layer()
+        parameter_name = self.parameter_name
+        if parameter_name not in base_layer.parametrizations:
+            raise ValueError(
+                "Something went wrong, please report this issue on PEFT: https://github.com/huggingface/peft/issues"
+            )
+
+        param_list = base_layer.parametrizations[parameter_name]
+        if len(param_list) == 1:
+            # last parametrization, we can safely remove it completely
+            nn.utils.parametrize.remove_parametrizations(base_layer, parameter_name, leave_parametrized=False)
+            return
+
+        # If there are multiple parametrizations for the same parameter_name, we only want to remove the LoRA proxy.
+        # Unfortunately, PyTorch does not support this directly, so we need to take care of it manually. To achieve
+        # this, we check the ParameterList from the back until we find the _LoraParameterProxy instance and then remove
+        # it.
+        reversed_indices = reversed(range(len(param_list)))
+        for i in reversed_indices:
+            module = param_list[i]
+            if isinstance(module, _LoraParameterProxy):
+                del param_list[i]
+                break
+        else:  # no break encountered
+            # this should not happen, but raising an error is probably not necessary
+            warnings.warn(
+                f"Could not find any LoRA parametrization on {self}, please open an issue on "
+                "https://github.com/huggingface/peft/issues and report this warning."
+            )
+
+    def merge(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> None:
+        # same as lora.Linear.merge but not hard-coding base_layer.weight and without special cases like variants removed
+        adapter_names = check_adapters_to_merge(self, adapter_names)
+        if not adapter_names:
+            # no adapter to merge
+            return
+
+        for active_adapter in adapter_names:
+            if active_adapter in self.lora_A.keys():
+                base_layer = self.get_base_layer()
+                param = getattr(base_layer, self.parameter_name)
+                if safe_merge:
+                    # Note that safe_merge will be slower than the normal merge
+                    # because of the copy operation.
+                    orig_weight = param.data.clone()
+                    orig_dtype = orig_weight.dtype
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    orig_weight += delta_weight.to(orig_dtype)
+
+                    if not torch.isfinite(orig_weight).all():
+                        raise ValueError(
+                            f"NaNs detected in the merged weights. The adapter {active_adapter} seems to be broken"
+                        )
+
+                    param.data = orig_weight
+
+                else:
+                    delta_weight = self.get_delta_weight(active_adapter)
+                    param.data += delta_weight
+
+                self.merged_adapters.append(active_adapter)
+
+    def unmerge(self) -> None:
+        # same as lora.Linear.unmerge but not hard-coding base_layer.weight and without special cases like variants removed
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+        while len(self.merged_adapters) > 0:
+            active_adapter = self.merged_adapters.pop()
+            if active_adapter in self.lora_A.keys():
+                param = getattr(self.get_base_layer(), self.parameter_name)
+                orig_dtype = param.dtype
+                delta_weight = self.get_delta_weight(active_adapter)
+                param.data -= delta_weight.to(orig_dtype)
+
+    def _check_forward_args(self, x, *args, **kwargs):
+        """Check if the arguments are compatible with the configs and state of the model"""
+        if kwargs.get("adapter_names", None):
+            raise ValueError(f"lora.{self.__class__.__name__} does not support mixed adapter batches yet.")
+        super()._check_forward_args(x, *args, **kwargs)
+
+    def unload_and_optionally_merge_module(self, merge: bool, safe_merge: bool, adapter_names: Optional[list[str]]):
+        base_layer = self.base_layer
+        # ParamWrappers can be nested, so merge and retrieve base layer recursively
+        if merge:
+            self.merge(safe_merge=safe_merge, adapter_names=adapter_names)
+            while isinstance(base_layer, ParamWrapper):
+                base_layer.merge(safe_merge=safe_merge, adapter_names=adapter_names)
+                base_layer = base_layer.base_layer
+        else:
+            base_layer = self.get_base_layer()
+        return base_layer
+
+    def forward(self, x: torch.Tensor, *args: Any, **kwargs: Any) -> torch.Tensor:
+        self._check_forward_args(x, *args, **kwargs)
+        adapter_names = kwargs.pop("adapter_names", None)
+
+        if self.disable_adapters:
+            if self.merged:
+                self.unmerge()
+            result = self.base_layer(x, *args, **kwargs)
+        elif adapter_names is not None:
+            raise ValueError(f"lora.{self.__class__.__name__} does not support mixed batch inference")
+        elif self.merged:
+            result = self.base_layer(x, *args, **kwargs)
+        else:
+            with self._activate_lora(self.active_adapters):
+                result = self.base_layer(x, *args, **kwargs)
+        return result
+
+    def __repr__(self) -> str:
+        rep = super().__repr__()
+        idx = rep.find("(") + 1
+        # insert the name of the parameter to allow the repr to be disambiguous when multiple parameters on the same
+        # module are being targeted
+        rep = f"{rep[:idx]}\n  parameter_name='{self.parameter_name}',{rep[idx:]}"
+        return "lora." + rep
+
+
+def dispatch_default(
+    target: torch.nn.Module,
+    adapter_name: str,
+    config: LoraConfig,
+    parameter_name: Optional[str] = None,
+    **kwargs,
+) -> Optional[torch.nn.Module]:
+    new_module = None
+
+    if isinstance(target, BaseTunerLayer):
+        target_base_layer = target.get_base_layer()
+    else:
+        target_base_layer = target
+
+    if parameter_name is not None:
+        new_module = ParamWrapper(target, adapter_name, parameter_name=parameter_name, config=config, **kwargs)
+    elif isinstance(target_base_layer, torch.nn.Embedding):
+        new_module = Embedding(target, adapter_name, config=config, **kwargs)
+    elif isinstance(target_base_layer, torch.nn.Conv2d):
+        new_module = Conv2d(target, adapter_name, config=config, **kwargs)
+    elif isinstance(target_base_layer, torch.nn.Conv3d):
+        new_module = Conv3d(target, adapter_name, config=config, **kwargs)
+    elif isinstance(target_base_layer, nn.Conv1d):
+        new_module = Conv1d(target, adapter_name, config=config, **kwargs)
+    elif isinstance(target_base_layer, torch.nn.MultiheadAttention):
+        new_module = MultiheadAttention(target, adapter_name, config=config, **kwargs)
+    elif isinstance(target_base_layer, torch.nn.Linear):
+        if config.fan_in_fan_out:
+            warnings.warn(
+                "fan_in_fan_out is set to True but the target module is `torch.nn.Linear`. "
+                "Setting fan_in_fan_out to False."
+            )
+            config.fan_in_fan_out = False
+        new_module = Linear(target, adapter_name, config=config, **kwargs)
+    elif isinstance(target_base_layer, Conv1D):
+        if not config.fan_in_fan_out:
+            warnings.warn(
+                "fan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True."
+            )
+            config.fan_in_fan_out = True
+        new_module = Linear(target, adapter_name, is_target_conv_1d_layer=True, config=config, **kwargs)
+
+    return new_module
diff --git a/docs/refs/peft_lora_variants.py b/docs/refs/peft_lora_variants.py
new file mode 100644
index 0000000..8d990b7
--- /dev/null
+++ b/docs/refs/peft_lora_variants.py
@@ -0,0 +1,923 @@
+# Copyright 2023-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import annotations
+
+import collections
+import warnings
+from typing import Any, Optional
+
+import torch
+from accelerate.utils.imports import is_xpu_available
+from torch import nn
+
+from peft.tuners.lora.config import BdLoraConfig
+from peft.utils.other import transpose
+
+from .arrow import ArrowLoraLinearLayer
+from .config import LoraConfig, PeftConfig
+from .dora import DoraConv1dLayer, DoraConv2dLayer, DoraConv3dLayer, DoraEmbeddingLayer, DoraLinearLayer
+from .layer import Conv1d, Conv2d, Conv3d, Embedding, Linear, LoraVariant, _ConvNd
+
+
+class ArrowLinearVariant(LoraVariant):
+    @staticmethod
+    def init(module: Linear, adapter_name: str, config: LoraConfig, **kwargs):
+        """
+        Initialise the ArrowLoraLinearLayer() inside lora_arrow. lora_arrow is nn.ModuleDict(), serving as a container
+        for ArrowLoraLinearLayer(). A layer of the base model with LoRA adapter loaded on it will be like:
+        ----------------------------------------------------
+             (qkv_proj): lora.Linear4bit or lora.Linear(
+                (base_layer): Linear4bit or Linear (lora_dropout): ModuleDict( ... ) (lora_A): ModuleDict( ... )
+                (lora_B): ModuleDict( ... ) (lora_embedding_A): ParameterDict( ... ) (lora_embedding_B): ParameterDict(
+                ... ) (lora_magnitude_vector): ModuleDict( ... ) (lora_arrow): ModuleDict(
+                    (arrow_router): ArrowLoraLinearLayer() )
+            )
+        ----------------------------------------------------
+
+        Args:
+            module (Linear): LoRA Layer of the model, containing base_layer, lora_A, lora_B, etc.
+            adapter_name (str): name of the adapter that will be put in lora_arrow.
+            The adapter_name is "arrow_router" by default, set in create_arrow_model() in ./arrow.py
+        """
+        # Checking for arrow necessary config
+        arrow_config = config.arrow_config
+        if arrow_config is None:
+            raise ValueError("ArrowLinearVariant.init() did not receive an arrow_config")
+
+        # 1-a) build the ArrowLoRALayer
+        arrow_layer = ArrowLoraLinearLayer(
+            in_features=module.in_features,
+            arrow_config=arrow_config,
+        ).to(module.weight.device)
+
+        # 1-b) register a container if it doesn’t exist yet
+        if not hasattr(module, "lora_arrow"):
+            module.lora_arrow = nn.ModuleDict()
+
+        module.lora_arrow[adapter_name] = arrow_layer
+
+    @staticmethod
+    def forward(
+        module: Linear,
+        *,
+        active_adapter: str,
+        x: torch.Tensor,
+        result: torch.Tensor,
+        **kwargs,
+    ) -> torch.Tensor:
+        """
+        Parameters mirror those in PEFT’s `LoraVariant.forward`. Called every time the host Linear does a fwd pass.
+
+        build_prototypes() and gen_know_sub() should run only once before routing. Both are implemented in
+        ArrowLoraLinearLayer (see ./arrow.py). They are lazily invoked in the forward pass below. Attributes of
+        ArrowLoraLinearLayer() class ensure they execute only a single time.
+
+        Args:
+            module (Linear): LoRA Layer of the model
+            active_adapter (str): name of the arrow route, which should be active to perform arrow.
+            x (torch.Tensor): input to the layer
+            result (torch.Tensor): output of the base layer.
+
+        Return value:
+            output of the base model + delta weight computed by arrow layer.
+        """
+        arrow = module.lora_arrow[active_adapter]  # ArrowLoraLinearLayer
+        # Apply GenKnowSub the 1st time if applcable. By calling arrow/on_adapter_change(),
+        # gen_know_sub() is redone for newly added adapters after arrow.create_arrow_model().
+        arrow.gen_know_sub(module.lora_A, module.lora_B)
+        # lazily build prototypes the 1st time after GenKnowSub. By calling arrow/on_adapter_change(),
+        # build_prototypes() is redone for newly added adapters after arrow.create_arrow_model().
+        arrow.build_prototypes(module.lora_A, module.lora_B)
+
+        # A forward path of ArrowLoraLinearLayer is called so routing performs.
+        # Accept and ignore extra variant kwargs (e.g., 'alora_offsets') for compatibility
+        delta = arrow(
+            x,
+            lora_A=module.lora_A,
+            lora_B=module.lora_B,
+            dropout=module.lora_dropout[active_adapter],
+            scaling=module.scaling,
+        )
+        return result + delta
+
+    """
+    Since Arrow is a Mixture-of-Experts (MoE) approach, merging adapters is not meaningful or even possible: for each
+    token, the top-k LoRA experts are dynamically selected and routed. Because of this per-token routing, there is no
+    single set of weights that can represent a merged adapter.
+    """
+
+    @staticmethod
+    def merge_safe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        raise RuntimeError("Cannot merge an active Arrow router adapter. Remove it first.")
+
+    @staticmethod
+    def merge_unsafe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        raise RuntimeError("Cannot merge an active Arrow router adapter. Remove it first.")
+
+    @staticmethod
+    def unmerge(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        raise RuntimeError("Cannot unmerge an active Arrow router adapter. Remove it first.")
+
+
+class DoraLinearVariant(LoraVariant):
+    @staticmethod
+    def init(module: Linear, adapter_name: str, **kwargs: Any) -> None:
+        if not module.lora_magnitude_vector:
+            # first dora layer being added, add lora_magnitude_vector to the list of learnable parameters
+            module.adapter_layer_names = module.adapter_layer_names[:] + ("lora_magnitude_vector",)
+
+        dora_layer = DoraLinearLayer(fan_in_fan_out=getattr(module, "fan_in_fan_out", False))
+        lora_A = module.lora_A[adapter_name].weight
+        lora_B = module.lora_B[adapter_name].weight
+        place_on_cpu = module.ephemeral_gpu_offload and (lora_A.device.type == "cpu" or lora_B.device.type == "cpu")
+        if module.ephemeral_gpu_offload:
+            if lora_A.device.type in ["cuda", "xpu"]:
+                lora_B = lora_B.to(lora_A.device)
+            else:
+                if lora_B.device.type not in ["cuda", "xpu"]:
+                    if is_xpu_available():
+                        lora_B = lora_B.to("xpu")
+                    else:
+                        lora_B = lora_B.to("cuda")
+                lora_A = lora_A.to(lora_B.device)
+        scaling = module.scaling[adapter_name]
+        dora_layer.update_layer(
+            base_layer=module.get_base_layer(),
+            lora_A=lora_A,
+            lora_B=lora_B,
+            scaling=scaling,
+            place_on_cpu=place_on_cpu,
+        )
+        module.lora_magnitude_vector[adapter_name] = dora_layer
+
+    @staticmethod
+    def merge_safe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+
+        # since delta_weight already includes scaling, set it to 1 here
+        weight_norm = (
+            module.lora_magnitude_vector[active_adapter]
+            .get_weight_norm(orig_weight, transpose(delta_weight, module.fan_in_fan_out), scaling=1)
+            .detach()
+        )
+        # We need to cache weight_norm because it has to be based on the original weights. We
+        # cannot calculate it on the fly based on the merged weights when unmerging because its a
+        # different value
+        module._cache_store(f"{active_adapter}-weight_norm", weight_norm)
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        dora_factor = transpose(dora_factor.view(-1, 1), module.fan_in_fan_out)
+        new_weight = dora_factor * (orig_weight + delta_weight)
+        new_weight = new_weight.to(orig_dtype)
+        return new_weight
+
+    @staticmethod
+    def merge_unsafe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+        weight_norm = (
+            module.lora_magnitude_vector[active_adapter]
+            .get_weight_norm(orig_weight, transpose(delta_weight, module.fan_in_fan_out), scaling=1)
+            .detach()
+        )
+        # We need to cache weight_norm because it has to be based on the original weights. We
+        # cannot calculate it on the fly based on the merged weights when unmerging because its a
+        # different value
+        module._cache_store(f"{active_adapter}-weight_norm", weight_norm)
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        dora_factor = transpose(dora_factor.view(-1, 1), module.fan_in_fan_out)
+        new_weight = dora_factor * (orig_weight.data + delta_weight)
+        new_weight = new_weight.to(orig_dtype)
+        orig_weight.data = new_weight
+
+    @staticmethod
+    def unmerge(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+        weight_norm = module._cache_pop(f"{active_adapter}-weight_norm")
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        new_weight = orig_weight.data / dora_factor.view(-1, 1) - delta_weight
+        new_weight = new_weight.to(orig_dtype)
+        return new_weight
+
+    @staticmethod
+    def forward(
+        module: Linear,
+        active_adapter: str,
+        x: torch.Tensor,
+        result: torch.Tensor,
+        **kwargs,
+    ) -> torch.Tensor:
+        lora_A = module.lora_A[active_adapter]
+        lora_B = module.lora_B[active_adapter]
+        dropout = module.lora_dropout[active_adapter]
+        scaling = module.scaling[active_adapter]
+
+        if isinstance(dropout, nn.Identity) or not module.training:
+            base_result = result
+        else:
+            x = dropout(x)
+            base_result = None
+
+        result = result + module.lora_magnitude_vector[active_adapter](
+            x,
+            lora_A=lora_A,
+            lora_B=lora_B,
+            scaling=scaling,
+            base_layer=module.get_base_layer(),
+            base_result=base_result,
+            adapter_name=active_adapter,
+        )
+        return result
+
+
+class DoraEmbeddingVariant(DoraLinearVariant):
+    @staticmethod
+    def init(module: Embedding, adapter_name: str, **kwargs: Any) -> None:
+        if module.lora_magnitude_vector is None:
+            # first dora layer being added, add lora_magnitude_vector to the list of learnable parameters
+            module.adapter_layer_names = module.adapter_layer_names[:] + ("lora_magnitude_vector",)
+
+        dora_layer = DoraEmbeddingLayer(fan_in_fan_out=True)
+        lora_embedding_A = module.lora_embedding_A[adapter_name]
+        lora_embedding_B = module.lora_embedding_B[adapter_name]
+        scaling = module.scaling[adapter_name]
+        dora_layer.update_layer(
+            base_layer=module.get_base_layer(), lora_A=lora_embedding_A, lora_B=lora_embedding_B, scaling=scaling
+        )
+        module.lora_magnitude_vector[adapter_name] = dora_layer
+
+    @staticmethod
+    def merge_safe(module: Embedding, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+
+        # since delta_weight already includes scaling, set it to 1 here
+        weight_norm = (
+            module.lora_magnitude_vector[active_adapter]
+            .get_weight_norm(orig_weight, delta_weight.T, scaling=1)
+            .detach()
+        )
+        # We need to cache weight_norm because it has to be based on the original weights. We
+        # cannot calculate it on the fly based on the merged weights when unmerging because its a
+        # different value
+        module._cache_store(f"{active_adapter}-weight_norm", weight_norm)
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        dora_factor = dora_factor.view(1, -1)
+        new_weight = dora_factor * (orig_weight + delta_weight)
+        new_weight = new_weight.to(orig_dtype)
+        return new_weight
+
+    @staticmethod
+    def merge_unsafe(module: Embedding, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+        weight_norm = (
+            module.lora_magnitude_vector[active_adapter]
+            .get_weight_norm(orig_weight, delta_weight.T, scaling=1)
+            .detach()
+        )
+        # We need to cache weight_norm because it has to be based on the original weights. We
+        # cannot calculate it on the fly based on the merged weights when unmerging because its a
+        # different value
+        module._cache_store(f"{active_adapter}-weight_norm", weight_norm)
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        dora_factor = dora_factor.view(1, -1)
+        new_weight = dora_factor * (orig_weight.data + delta_weight)
+        new_weight = new_weight.to(orig_dtype)
+        orig_weight.data = new_weight
+
+    @staticmethod
+    def unmerge(module: Embedding, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+        weight_norm = module._cache_pop(f"{active_adapter}-weight_norm")
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        new_weight = orig_weight.data / dora_factor.view(1, -1) - delta_weight
+        new_weight = new_weight.to(orig_dtype)
+        return new_weight
+
+    @staticmethod
+    def forward(
+        module: Embedding,
+        active_adapter: str,
+        x: torch.Tensor,
+        result: torch.Tensor,
+        **kwargs,
+    ) -> torch.Tensor:
+        embedding_A = module.lora_embedding_A[active_adapter].T
+        embedding_B = module.lora_embedding_B[active_adapter].T
+        scaling = module.scaling[active_adapter]
+
+        mag_norm_scale, dora_result = module.lora_magnitude_vector[active_adapter](
+            x,
+            lora_A=embedding_A,
+            lora_B=embedding_B,
+            scaling=scaling,
+            base_layer=module.get_base_layer(),
+            embed_fn=module._embed,
+            adapter_name=active_adapter,
+        )
+
+        # Some embedding layers (e.g., Gemma3TextScaledWordEmbedding) apply scaling in their forward method.
+        # Since base_layer(x) already includes this scaling, we need to apply it to DoRA contributions too.
+        # Note: embed_scale is applied AFTER weight norm calculation to preserve DoRA's weight geometry semantics.
+        embed_scale = module._get_embed_scale()
+        if embed_scale is not None:
+            dora_result = dora_result * embed_scale.to(dora_result.dtype)
+
+        result = mag_norm_scale * result + dora_result
+        return result
+
+
+class _DoraConvNdVariant(LoraVariant):
+    @staticmethod
+    def init_convd_variant(module: _ConvNd, adapter_name: str, dora_layer: nn.Module) -> None:
+        if module.lora_magnitude_vector is None:
+            # first dora layer being added, add lora_magnitude_vector to the list of learnable parameters
+            module.adapter_layer_names = module.adapter_layer_names[:] + ("lora_magnitude_vector",)
+
+        lora_A = module.lora_A[adapter_name].weight
+        lora_B = module.lora_B[adapter_name].weight
+        scaling = module.scaling[adapter_name]
+        dora_layer.update_layer(base_layer=module.get_base_layer(), lora_A=lora_A, lora_B=lora_B, scaling=scaling)
+        module.lora_magnitude_vector[adapter_name] = dora_layer
+
+    @staticmethod
+    def merge_safe(module: _ConvNd, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+
+        # since delta_weight already includes scaling, set it to 1 here
+        weight_norm = (
+            module.lora_magnitude_vector[active_adapter].get_weight_norm(orig_weight, delta_weight, scaling=1).detach()
+        )
+        # We need to cache weight_norm because it has to be based on the original weights. We
+        # cannot calculate it on the fly based on the merged weights when unmerging because its a
+        # different value
+        module._cache_store(f"{active_adapter}-weight_norm", weight_norm)
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        new_weight = dora_factor.view(*module._get_dora_factor_view()) * (orig_weight + delta_weight)
+        new_weight = new_weight.to(orig_dtype)
+        return new_weight
+
+    @staticmethod
+    def merge_unsafe(module: _ConvNd, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+        # since delta_weight already includes scaling, set it to 1 here
+        weight_norm = (
+            module.lora_magnitude_vector[active_adapter].get_weight_norm(orig_weight, delta_weight, scaling=1).detach()
+        )
+        # We need to cache weight_norm because it has to be based on the original weights. We
+        # cannot calculate it on the fly based on the merged weights when unmerging because its a
+        # different value
+        module._cache_store(f"{active_adapter}-weight_norm", weight_norm)
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        new_weight = dora_factor.view(*module._get_dora_factor_view()) * (orig_weight.data + delta_weight)
+        new_weight = new_weight.to(orig_dtype)
+        orig_weight.data = new_weight
+
+    @staticmethod
+    def unmerge(module: _ConvNd, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        orig_dtype = orig_weight.dtype
+        delta_weight = module.get_delta_weight(active_adapter)
+        weight_norm = module._cache_pop(f"{active_adapter}-weight_norm")
+        dora_factor = module.lora_magnitude_vector[active_adapter].weight / weight_norm
+        new_weight = orig_weight.data / dora_factor.view(*module._get_dora_factor_view()) - delta_weight
+        new_weight = new_weight.to(orig_dtype)
+        return new_weight
+
+    @staticmethod
+    def forward(
+        module: _ConvNd,
+        active_adapter: str,
+        x: torch.Tensor,
+        result: torch.Tensor,
+        **kwargs,
+    ) -> torch.Tensor:
+        lora_A = module.lora_A[active_adapter]
+        lora_B = module.lora_B[active_adapter]
+        dropout = module.lora_dropout[active_adapter]
+        scaling = module.scaling[active_adapter]
+
+        if isinstance(dropout, nn.Identity) or not module.training:
+            base_result = result
+        else:
+            x = dropout(x)
+            base_result = None
+
+        result = result + module.lora_magnitude_vector[active_adapter](
+            x,
+            lora_A=lora_A,
+            lora_B=lora_B,
+            scaling=scaling,
+            base_layer=module.get_base_layer(),
+            base_result=base_result,
+            adapter_name=active_adapter,
+        )
+        return result
+
+
+class DoraConv1dVariant(_DoraConvNdVariant):
+    @staticmethod
+    def init(module: Conv1d, adapter_name: str, config: LoraConfig, **kwargs: Any) -> None:
+        dora_layer = DoraConv1dLayer(fan_in_fan_out=False)
+        _DoraConvNdVariant.init_convd_variant(module, adapter_name, dora_layer=dora_layer)
+
+
+class DoraConv2dVariant(_DoraConvNdVariant):
+    @staticmethod
+    def init(module: Conv2d, adapter_name: str, config: LoraConfig, **kwargs: Any) -> None:
+        dora_layer = DoraConv2dLayer(fan_in_fan_out=False)
+        _DoraConvNdVariant.init_convd_variant(module, adapter_name, dora_layer=dora_layer)
+
+
+class DoraConv3dVariant(_DoraConvNdVariant):
+    @staticmethod
+    def init(module: Conv3d, adapter_name: str, config: LoraConfig, **kwargs: Any) -> None:
+        dora_layer = DoraConv3dLayer(fan_in_fan_out=False)
+        _DoraConvNdVariant.init_convd_variant(module, adapter_name, dora_layer=dora_layer)
+
+
+class QALoraLinearVariant(LoraVariant):
+    @staticmethod
+    def init(module: Linear, adapter_name: str, config: LoraConfig, **kwargs: Any) -> None:
+        """
+        Initializes QALoRA specific parameters for a given adapter.
+
+        Args:
+            module (Linear): The linear module to be adapted.
+            adapter_name (str): The name of the adapter.
+            config (LoraConfig): The config of the LoRA adapter.
+            **kwargs: Additional keyword arguments.
+        """
+        qalora_group_size = config.qalora_group_size
+        if module.in_features is not None and module.in_features % qalora_group_size != 0:
+            raise ValueError(
+                f"`use_qalora=True` requires `module.in_features` ({module.in_features}) to be"
+                f"divisible by 'qalora_group_size' ({qalora_group_size})"
+            )
+
+        if "qalora_group_size" not in module.other_param_names:
+            module.other_param_names = module.other_param_names + ("qalora_group_size",)
+
+        if not hasattr(module, "qalora_group_size"):
+            module.qalora_group_size = {}
+        module.qalora_group_size[adapter_name] = qalora_group_size
+
+        old_lora_A_layer = module.lora_A[adapter_name]
+        r = old_lora_A_layer.out_features
+        device = old_lora_A_layer.weight.device
+        dtype = old_lora_A_layer.weight.dtype
+
+        new_lora_A_layer = nn.Linear(
+            old_lora_A_layer.in_features // module.qalora_group_size[adapter_name],
+            r,
+            bias=False,
+            device=device,
+            dtype=dtype,
+        )
+        module.lora_A[adapter_name] = new_lora_A_layer
+
+    @staticmethod
+    def get_delta_weight(module: Linear, active_adapter: str) -> torch.Tensor:
+        raise NotImplementedError("QALoRA for GPTQ layers does not support 'get_delta_weight'.")
+
+    @staticmethod
+    def merge_safe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        raise NotImplementedError("QALoRA for GPTQ layers does not support 'safe_merge'.")
+
+    @staticmethod
+    def merge_unsafe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        raise NotImplementedError("QALoRA for GPTQ layers does not support 'merge_unsafe'.")
+
+    @staticmethod
+    def unmerge(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        raise NotImplementedError("QALoRA for GPTQ layers does not support 'unmerge'.")
+
+    @staticmethod
+    def forward(
+        module: Linear,
+        active_adapter: str,
+        x: torch.Tensor,
+        result: torch.Tensor,
+        **kwargs,
+    ) -> torch.Tensor:
+        lora_A_weight = module.lora_A[active_adapter].weight
+        lora_B_weight = module.lora_B[active_adapter].weight
+        dropout = module.lora_dropout[active_adapter]
+        scaling = module.scaling[active_adapter]
+        group_size = module.qalora_group_size[active_adapter]
+
+        x_dropped = dropout(x) if module.training and not isinstance(dropout, nn.Identity) else x
+        orig_shape = x_dropped.shape
+
+        # Reshape to 2D
+        if len(orig_shape) > 2:
+            x_flat = x_dropped.view(-1, module.in_features)
+        else:
+            x_flat = x_dropped
+
+        batch_size, in_features = x_flat.shape
+        pooled_features = in_features // group_size
+
+        x_pooled = x_flat.view(batch_size, pooled_features, group_size).mean(dim=2)
+
+        x_pooled_scaled = x_pooled * pooled_features
+
+        # LoRA computation
+        delta = x_pooled_scaled @ lora_A_weight.t() @ lora_B_weight.t() * scaling
+
+        # Reshape back
+        if len(orig_shape) > 2:
+            delta = delta.view(orig_shape[:-1] + (delta.size(-1),))
+
+        return result + delta
+
+
+class ALoraLinearVariant(LoraVariant):
+    @staticmethod
+    def init(module: Linear, adapter_name: str, config: LoraConfig, **kwargs: Any) -> None:
+        pass
+
+    @staticmethod
+    def merge_safe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        raise NotImplementedError("aLoRA does not support safe merging.")
+
+    @staticmethod
+    def merge_unsafe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        raise NotImplementedError("aLoRA does not support merging.")
+
+    @staticmethod
+    def unmerge(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        raise NotImplementedError("aLoRA does not support unmerging.")
+
+    @staticmethod
+    def forward(
+        module: Linear,
+        active_adapter: str,
+        x: torch.Tensor,
+        result: torch.Tensor,
+        **kwargs,
+    ) -> torch.Tensor:
+        alora_offsets = kwargs.get("alora_offsets", None)
+        lora_A = module.lora_A[active_adapter]
+        lora_B = module.lora_B[active_adapter]
+        dropout = module.lora_dropout[active_adapter]
+        scaling = module.scaling[active_adapter]
+        x = x.to(lora_A.weight.dtype)
+        result_shape = result.shape
+        B = result_shape[0]  # batch
+        if len(result_shape) == 3:
+            T = result_shape[1]  # tokens
+        else:
+            T = 1
+        D = result_shape[-1]  # dimensions
+        Dx = x.shape[-1]
+        device = result.device
+        if alora_offsets is None:  # use base model only, but ensure 0 gradient
+            mask = torch.zeros((B, T), dtype=torch.bool)
+        else:
+            # If alora_offsets[i] is None, this means that the invocation sequence was not found in the
+            # input. As a result, the weights should not be activated anywhere (equivalent to base model).
+            # Convert None -> 0 and clip to T
+            offsets = torch.tensor(
+                [0 if o is None else min(int(o), T) for o in alora_offsets],
+                device=device,
+                dtype=torch.long,
+            )
+            # Mask True on the last `offsets[i]` positions for each row i
+            pos = torch.arange(T, device=device).unsqueeze(0)  # [1, T]
+            mask = pos >= (T - offsets).unsqueeze(1)
+
+        # Flatten for vectorization
+        x_flat = x.view(-1, Dx)
+        res_flat = result.view(-1, D)
+        mask_flat = mask.view(-1)
+
+        # Compute adapter on the selected tokens only
+        res_flat[mask_flat] += lora_B(lora_A(dropout(x_flat[mask_flat]))) * scaling
+        return result
+
+
+def calculate_alora_offsets(
+    peft_config: PeftConfig, active_adapter: str, input_ids: torch.Tensor, adapter_names: Optional[list[str]] = None
+) -> list[int]:
+    """
+    This is a helper function for Activated LoRA (aLoRA) that searches each input token sequence for the last
+    occurrence of the appropriate "alora_invocation_tokens" invocation sequence. The calculated alora_offset is the
+    location of the *start* of the invocation tokens, counting backward from the end (will therefore always be >=
+    len(alora_invocation_tokens). If adapter_names is passed, then each input uses the appropriate invocation sequence
+    for the specified adapter for that row. Logic is provided to handle mixed collections of adapters for which not all
+    are aLoRAs (e.g. some base model, some LoRA).
+    """
+    if input_ids is None:
+        return []
+
+    batch_size = input_ids.shape[0]
+    alora_offsets = [None] * batch_size
+
+    cached_invocation_tensors = {}
+    adapters_to_process_indices = collections.defaultdict(list)
+
+    for i in range(batch_size):
+        current_adapter_name = adapter_names[i] if adapter_names and i < len(adapter_names) else active_adapter
+
+        if current_adapter_name == "__base__":
+            alora_offsets[i] = None
+            continue
+
+        if current_adapter_name not in peft_config:
+            warnings.warn(f"Adapter '{current_adapter_name}' not found in peft_config. Using base model for row {i}.")
+            alora_offsets[i] = None
+            continue
+
+        current_peft_config = peft_config[current_adapter_name]
+
+        invocation_tokens = getattr(current_peft_config, "alora_invocation_tokens", None)
+        if invocation_tokens is None:
+            alora_offsets[i] = None  # Not an aLoRA adapter or wrong type
+            continue
+
+        if current_adapter_name not in cached_invocation_tensors:
+            cached_invocation_tensors[current_adapter_name] = torch.tensor(
+                invocation_tokens, dtype=torch.long, device=input_ids.device
+            )
+
+        adapters_to_process_indices[current_adapter_name].append(i)
+
+    for adapter_name_to_process, indices in adapters_to_process_indices.items():
+        current_invocation_ids_tensor = cached_invocation_tensors[adapter_name_to_process]
+        invocation_len = len(current_invocation_ids_tensor)
+
+        for i in indices:
+            sequence = input_ids[i]
+            seq_len = len(sequence)
+            best_match_start_idx = -1
+
+            possible_starts = (sequence == current_invocation_ids_tensor[0]).nonzero(as_tuple=True)[0]
+
+            for start_idx_tensor in possible_starts:
+                idx = start_idx_tensor.item()
+                if idx + invocation_len <= seq_len:
+                    if torch.equal(sequence[idx : idx + invocation_len], current_invocation_ids_tensor):
+                        if idx > best_match_start_idx:
+                            best_match_start_idx = idx
+
+            if best_match_start_idx != -1:
+                offset_val = seq_len - best_match_start_idx
+                alora_offsets[i] = offset_val if offset_val > 0 else None
+            else:  # Invocation sequence not found in input
+                alora_offsets[i] = None
+    return alora_offsets
+
+
+def is_alora_relevant_in_batch(model: nn.Module, adapter_names: Optional[list[str]] = None):
+    """
+    Helper function to determine if the current batch has any aLoRA adapters.
+    """
+    is_alora_relevant = False
+    if getattr(model.active_peft_config, "alora_invocation_tokens", None):
+        is_alora_relevant = True
+    elif adapter_names:
+        for name in adapter_names:
+            if name == "__base__":
+                continue
+            config_ = model.peft_config.get(name)
+            if config_ and getattr(config_, "alora_invocation_tokens", None):
+                is_alora_relevant = True
+                break
+
+    return is_alora_relevant
+
+
+def get_alora_offsets_for_forward(
+    model: nn.Module, input_ids: torch.Tensor = None, inputs_embeds: torch.Tensor = None, **kwargs
+):
+    """
+    Wrapper around calculate_alora_offsets, for the .forward of the model. It only calculates alora_offsets if the
+    batch contains aLoRA adapters.
+    """
+    adapter_names_for_offset_calc = kwargs.get("adapter_names", None)
+    if not is_alora_relevant_in_batch(model, adapter_names_for_offset_calc):
+        # Nothing to compute
+        return kwargs
+    alora_offsets = kwargs.get("alora_offsets")
+    if alora_offsets is None:
+        if input_ids is None and inputs_embeds is not None:
+            warnings.warn(
+                "Cannot calculate aLoRA offsets when only inputs_embeds are provided. Disabling aLoRA for this forward pass."
+            )
+            kwargs["alora_offsets"] = None
+        elif input_ids is not None:
+            kwargs["alora_offsets"] = calculate_alora_offsets(
+                model.peft_config,
+                model.active_adapter,
+                input_ids,
+                adapter_names=adapter_names_for_offset_calc,
+            )
+        else:
+            kwargs["alora_offsets"] = None
+    return kwargs
+
+
+def get_alora_offsets_for_generate(model: nn.module, *args, **kwargs):
+    """
+    Wrapper around calculate_alora_offsets, for the .generate of the model. It only calculates alora_offsets if the
+    batch contains aLoRA adapters.
+    """
+    adapter_names_for_offset_calc = kwargs.get("adapter_names")
+    if not is_alora_relevant_in_batch(model, adapter_names_for_offset_calc):
+        # Nothing to compute
+        return kwargs
+    alora_offsets_from_kwargs = kwargs.get("alora_offsets")
+    if alora_offsets_from_kwargs is None:
+        current_input_ids = kwargs.get("input_ids")
+        if current_input_ids is None:  # args[0] is usually input_ids
+            if args and isinstance(args[0], torch.Tensor):
+                current_input_ids = args[0]
+            else:
+                current_input_ids = None
+
+        if current_input_ids is not None:
+            if current_input_ids.ndim == 1:
+                current_input_ids = current_input_ids.unsqueeze(0)
+            calculated_offsets = calculate_alora_offsets(
+                model.peft_config,
+                model.active_adapter,
+                current_input_ids,
+                adapter_names=adapter_names_for_offset_calc,
+            )
+            kwargs["alora_offsets"] = calculated_offsets
+
+        else:
+            warnings.warn(
+                "Cannot calculate aLoRA offsets during generate as input_ids are not available. Disabling aLoRA."
+            )
+
+            kwargs["alora_offsets"] = None
+    return kwargs
+
+
+class BlockDiagonalLinear(nn.Module):
+    def __init__(
+        self,
+        in_features: int,
+        out_features: int,
+        nblocks: int,
+        init_zero: bool = False,
+        dtype: torch.dtype = torch.float32,
+        device: torch.device = torch.device("cpu"),
+    ):
+        super().__init__()
+        self.in_features = in_features
+        self.out_features = out_features
+        self.nblocks = nblocks
+        if self.in_features % nblocks != 0 or self.out_features % nblocks != 0:
+            raise ValueError(
+                f"self.in_features={self.in_features} or self.out_features={self.out_features} not divisible by {self.nblocks}"
+            )
+        # Create weight with specified dtype and device
+        self.weight = nn.Parameter(torch.empty(out_features, in_features // nblocks, dtype=dtype, device=device))
+
+        if init_zero:
+            torch.nn.init.zeros_(self.weight)
+        else:
+            torch.nn.init.kaiming_uniform_(self.weight)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        first_dims = x.shape[:-1]
+        if x.dim() != 2:
+            x = x.reshape(-1, x.shape[-1])
+        B = x.shape[0]
+        nb = self.nblocks
+        m = x.shape[-1] // nb
+        n = self.out_features // nb
+        x = x.reshape(B, nb, m)
+        w = self.weight.view(nb, n, m)
+        out = torch.einsum("bim,inm->bin", x, w)
+        return out.reshape(*first_dims, -1)
+
+    def weight_as_blockdiagonal_matrix(self):
+        """Returns weight in a format similar to a vanilla LoRA adapter. For this, we stack the blocks on the diagonal,
+        leaving the off-diagonals padded with zero."""
+        return torch.block_diag(*torch.chunk(self.weight, self.nblocks, dim=0))
+
+
+class BdLoraLinearVariant(LoraVariant):
+    @staticmethod
+    def init(module: Linear, adapter_name: str, config: LoraConfig, **kwargs) -> None:
+        use_bdlora = config.use_bdlora
+        target_name = kwargs.get("target_name", "")
+
+        # Handle case where use_bdlora is a dict (from saved config) instead of BdLoraConfig object
+        if isinstance(use_bdlora, dict):
+            use_bdlora = BdLoraConfig(**use_bdlora)
+
+        lora_a_blockdiagonal_pattern = use_bdlora.target_modules_bd_a or []
+        lora_b_blockdiagonal_pattern = use_bdlora.target_modules_bd_b or []
+        nblocks = use_bdlora.nblocks
+
+        has_lora_a_blockdiagonal = any(pattern in target_name for pattern in lora_a_blockdiagonal_pattern)
+        has_lora_b_blockdiagonal = any(pattern in target_name for pattern in lora_b_blockdiagonal_pattern)
+
+        if has_lora_a_blockdiagonal and has_lora_b_blockdiagonal:
+            raise ValueError(f"Target {target_name} matches both A and B block-diagonal patterns")
+        if use_bdlora.match_strict and not (has_lora_a_blockdiagonal or has_lora_b_blockdiagonal):
+            raise ValueError(
+                f"Target {target_name} matches neither A nor B block-diagonal patterns."
+                "If this is intentional, set match_strict=False in BdLoraConfig during initialization. "
+            )
+
+        if has_lora_a_blockdiagonal:
+            r = module.lora_A[adapter_name].out_features
+            base_layer = module.get_base_layer()
+            layer = BlockDiagonalLinear(
+                base_layer.in_features,
+                r,
+                nblocks=nblocks,
+                init_zero=False,
+                dtype=base_layer.weight.dtype,
+                device=base_layer.weight.device,
+            )
+            module.lora_A[adapter_name] = layer
+        elif has_lora_b_blockdiagonal:
+            r = module.lora_B[adapter_name].in_features
+            base_layer = module.get_base_layer()
+            layer = BlockDiagonalLinear(
+                r,
+                base_layer.out_features,
+                nblocks=nblocks,
+                init_zero=True,
+                dtype=base_layer.weight.dtype,
+                device=base_layer.weight.device,
+            )
+            module.lora_B[adapter_name] = layer
+
+    @staticmethod
+    def forward(module: Linear, active_adapter: str, x: torch.Tensor, result: torch.Tensor, **kwargs) -> torch.Tensor:
+        lora_A = module.lora_A[active_adapter]
+        lora_B = module.lora_B[active_adapter]
+        dropout = module.lora_dropout[active_adapter]
+        scaling = module.scaling[active_adapter]
+        x = dropout(x)
+        # Cast input dtype to match lora_A weight dtype
+        x = module._cast_input_dtype(x, lora_A.weight.dtype)
+        result += lora_B(lora_A(x)) * scaling
+        return result
+
+    @staticmethod
+    def _get_weight_from_module_maybe_blockdiagonal(module: nn.Module) -> torch.Tensor:
+        if isinstance(module, BlockDiagonalLinear):
+            return module.weight_as_blockdiagonal_matrix()
+        else:
+            return module.weight  # type: ignore
+
+    @staticmethod
+    def _get_bdlora_delta_weight(module: Linear, adapter: str) -> torch.Tensor:
+        """Similar to get_delta_weight for a linear module, but we have to eventually reshape the blocks
+        of the weights."""
+        device = module.lora_B[adapter].weight.device
+        # Use base layer dtype to ensure compatibility with merge/unmerge operations
+        base_layer = module.get_base_layer()
+        dtype = base_layer.weight.dtype
+        cast_to_fp32 = device.type == "cpu" and (dtype == torch.float16 or dtype == torch.bfloat16)
+
+        weight_A = BdLoraLinearVariant._get_weight_from_module_maybe_blockdiagonal(module.lora_A[adapter])
+        weight_B = BdLoraLinearVariant._get_weight_from_module_maybe_blockdiagonal(module.lora_B[adapter])
+
+        if cast_to_fp32:
+            weight_A = weight_A.float()
+            weight_B = weight_B.float()
+
+        output_tensor = transpose(weight_B @ weight_A, module.fan_in_fan_out) * module.scaling[adapter]
+
+        if cast_to_fp32:
+            output_tensor = output_tensor.to(dtype=dtype)
+
+        # Ensure output tensor matches base layer dtype
+        return output_tensor.to(dtype=dtype)
+
+    @staticmethod
+    def merge_safe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        return orig_weight + BdLoraLinearVariant._get_bdlora_delta_weight(module, active_adapter)
+
+    @staticmethod
+    def merge_unsafe(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> None:
+        orig_weight.data += BdLoraLinearVariant._get_bdlora_delta_weight(module, active_adapter)
+
+    @staticmethod
+    def unmerge(module: Linear, active_adapter: str, orig_weight: torch.Tensor) -> torch.Tensor:
+        return orig_weight - BdLoraLinearVariant._get_bdlora_delta_weight(module, active_adapter)
diff --git a/src/lora_lite/variants/delora.py b/src/lora_lite/variants/delora.py
index 094e9a3..5d9bf88 100644
--- a/src/lora_lite/variants/delora.py
+++ b/src/lora_lite/variants/delora.py
@@ -1,6 +1,6 @@
 """DeLoRA: column-normalised A, B, scaled by lambda * ||W||_F / r.
 
-Bini et al. 2025  https://arxiv.org/abs/2503.18225
+Bini et al. 2025 (ICLR'25)  https://arxiv.org/abs/2503.18225
 
 Paper Eq. 8:    W' = W + (lambda * ||W||_F / r) B Xi A
 where Xi_{i,i} = 1 / (||b_i|| ||a_i||) makes each rank-1 component unit-norm.
@@ -12,7 +12,22 @@ Identity at t=0: paper uses kaiming init for both A and B with `lambda` initiali
 to 0 (or small) so the effective delta starts near zero. We honour that:
 default lambda0 == 0 gives bit-identity; user can override via variant_kwargs.
 
+KNOWN GRADIENT ISSUE (flagged by external review 2026-04-26):
+  With lambda0=0 the *forward* is identity but `A,B` get zero gradient on step 0
+  (delta = lambda * ... -> d_output/d_A is proportional to lambda). Only
+  `lora_lambda` moves first step. With lambda0>0, A,B train but identity is broken.
+  Paper's true initialization (frozen-copy trick, see Eq. 9) achieves both;
+  we do NOT implement that here.
+
 The frozen ||W||_F factor is captured once at init() into a buffer `lora_wnorm`.
+
+Reference implementations (for review/cross-check):
+  - DeLoRA paper authors (ExplainableML/DeLoRA) -- their fork of peft:
+    https://github.com/ExplainableML/DeLoRA/blob/main/peft/src/peft/tuners/delora.py
+    (offline: docs/refs/orig_delora.py)
+  - peft DeLoRA (upstreamed):
+    https://github.com/huggingface/peft/blob/main/src/peft/tuners/delora/layer.py
+    (offline: docs/refs/peft_delora_layer.py)
 """
 import torch
 import torch.nn.functional as F
diff --git a/src/lora_lite/variants/dora.py b/src/lora_lite/variants/dora.py
index 395c879..34b966f 100644
--- a/src/lora_lite/variants/dora.py
+++ b/src/lora_lite/variants/dora.py
@@ -6,6 +6,17 @@ At t=0:  B=0 -> V=W -> y_new = (m_init / ||W||_c) (Wx + 0) = Wx   when m_init =
 
 Limitation: requires materializing the dense weight to compute ||V||_c. v1 supports
 plain nn.Linear only; bnb 4/8-bit layers raise loudly.
+
+DEVIATION (numerical):
+  - We differentiate through ||V||_c every forward. The paper's sec. 4.3 suggests
+    a 'cost-saving' variant that detaches ||V|| in backward (gradient only flows
+    through V); we do NOT do that. Real impact: slower step, slightly different
+    gradient direction. Faithful to the eq.5 forward, not the optimized one.
+
+Reference implementations (for review/cross-check):
+  - peft DoRA (separate file under lora/):
+    https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/dora.py
+    (offline: docs/refs/peft_lora_dora.py)
 """
 import torch
 import torch.nn.functional as F
diff --git a/src/lora_lite/variants/hra.py b/src/lora_lite/variants/hra.py
index e2cfeb6..8161d8b 100644
--- a/src/lora_lite/variants/hra.py
+++ b/src/lora_lite/variants/hra.py
@@ -13,10 +13,25 @@ Identity at t=0: `lora_gate` is initialized to 0 and gates each Householder
 vector, so the effective u_i starts at 0 -> H_i = I -> R = I -> y' = y.
 At training time the gate scales the active reflection direction.
 
+KNOWN GRADIENT ISSUE (flagged by external review 2026-04-26):
+  Forward is `x + gate * (Rx - x)`. With gate=0 at init, d_output/d_U is
+  proportional to gate, so on step 0 ONLY `lora_gate` receives gradient;
+  `lora_U` is dead. Once gate moves off zero, U starts learning. This deviates
+  from the paper, which has no such gate -- paper uses orthogonal init of U so
+  R != I from step 0. We trade paper-faithful init for identity-at-init.
+
 OMITTED: paper also adds an orthogonality regularizer
     lambda * sum_i (u_i^T u_j)^2          (Eq. 6 / Sec. 3.3)
 which is a loss term, not a forward-pass change. Add it in your training loop if
 you want the regularized HRA variant.
+
+Reference implementations (for review/cross-check):
+  - HRA paper authors (DaShenZi721/HRA), llama variant of OFT layer with HRA:
+    https://github.com/DaShenZi721/HRA/blob/master/llama/peft/oft/layer_GS_HRA.py
+    (offline: docs/refs/orig_hra_layer.py)
+  - peft HRA layer (cleaner, includes apply_GS toggle for orthogonalization):
+    https://github.com/huggingface/peft/blob/main/src/peft/tuners/hra/layer.py
+    (offline: docs/refs/peft_hra_layer.py)
 """
 import torch
 from einops import einsum
diff --git a/src/lora_lite/variants/ia3.py b/src/lora_lite/variants/ia3.py
index a60fcdb..f9782e3 100644
--- a/src/lora_lite/variants/ia3.py
+++ b/src/lora_lite/variants/ia3.py
@@ -16,6 +16,11 @@ DEVIATION FROM PAPER:
 
     `up_proj` is the closest stand-in for "FFN intermediate" in gated-MLP blocks
     (Llama uses gate * up; gating the up branch is the IA3-spirit choice).
+
+Reference implementations (for review/cross-check):
+  - peft IA3 layer (uses ia3_l elementwise scaling, fan_in_fan_out aware):
+    https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/layer.py
+    (offline: docs/refs/peft_ia3_layer.py)
 """
 import torch
 from torch import nn
diff --git a/src/lora_lite/variants/lora.py b/src/lora_lite/variants/lora.py
index 2b55eb9..681a4cc 100644
--- a/src/lora_lite/variants/lora.py
+++ b/src/lora_lite/variants/lora.py
@@ -3,6 +3,11 @@
     h = W x + (alpha/r) B A x
 
 Identity at t=0 from B=0. Faithful to the paper.
+
+Reference implementations (for review/cross-check):
+  - peft Linear.update_layer + lora_A/B init, forward:
+    https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py
+    (see docs/refs/peft_lora_layer.py for offline copy)
 """
 from einops import einsum
 from torch import nn
diff --git a/src/lora_lite/variants/pissa.py b/src/lora_lite/variants/pissa.py
index 0c32a26..f57788a 100644
--- a/src/lora_lite/variants/pissa.py
+++ b/src/lora_lite/variants/pissa.py
@@ -2,6 +2,21 @@
 
 Meng et al. 2024  https://arxiv.org/abs/2404.02948
 W_eff(t=0) = W_res + B@A = W (numerically; bf16 round-trip not bit-exact).
+
+DEVIATION FROM PAPER (documented):
+  - Paper sets adapter scale = 1 (no alpha/r factor); we keep LoRA's alpha/r
+    pipeline so callers must pass alpha=r to get paper-faithful identity.
+  - Saved adapter does NOT include W_res; load() recomputes PiSSA init on the
+    *same-seed base* before overwriting A/B. Reload is exact only on identical
+    base weights.
+
+Reference implementations (for review/cross-check):
+  - PiSSA original (NeurIPS'24 spotlight) init script (SVD on dequant W):
+    https://github.com/MuLabPKU/PiSSA/blob/main/utils/init_pissa.py
+    (offline: docs/refs/orig_pissa_init.py)
+  - peft PiSSA flavor (init_lora_weights='pissa') in:
+    https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py
+    (offline: docs/refs/peft_lora_layer.py, see pissa_init / loftq_init paths)
 """
 import torch
 from einops import einsum