Adapters as Representational Hypotheses

What does each PEFT method believe about transformer internals?

Why care?

We want to understand how transformers work. There are many approaches -- probing, ablation, SAEs -- but most of them observe rather than intervene. Probing finds representations that predict behavior, but high probe accuracy does not mean the model uses that representation (Belinkov, 2022). CCS discovers latent knowledge but cannot intervene on it (Burns et al., 2022). Intervention shortcuts both problems: if modifying a representation reliably changes behavior, we have causal evidence of what we control (I argued this in AntiPaSTO).

There is an underappreciated source of exactly this kind of causal evidence: the PEFT adapter literature.

Each adapter constrains how you can update pretrained weights. When one adapter architecture outperforms another under controlled conditions -- same model, same data, same parameter budget -- the winning method's structural assumptions are supported as a better description of the weight manifold. This is a natural experiment running across hundreds of papers, and almost nobody reads it as science about representations.

GDM's interpretability team recently pivoted toward "pragmatic interpretability" -- directly solving problems on the critical path to AGI going well, grounded in proxy tasks with empirical feedback (Nanda et al., 2025). Adapter benchmarks are precisely this: empirical feedback on which structural assumptions about transformer internals hold up under intervention.

If the adapter generalizes out-of-distribution, that tells us the geometric structure it exploits is causally relevant to behavior, not merely correlated. From my AntiPaSTO paper:

Each adapter architecture encodes a claim about how to intervene in transformer internals. LoRA hypothesizes weight changes are low-rank. OFT hypothesizes orthogonal transformations preserve semantic structure. VeRA hypothesizes shared random projections plus learned scaling suffice. DeLoRA hypothesizes direction and magnitude should decouple. PiSSA hypothesizes principal components matter most. Our choice -- Cayley rotations of SVD singular vectors -- hypothesizes that the model's own learned basis defines the natural intervention manifold. Adapters that generalize out-of-distribution tell us which geometric structures are causally relevant to behavior, not merely correlated with it.

This is a pragmatic, interventionist program: we learn about internals by seeing which interventions work. An adapter that transfers where others fail reveals something real about the geometry of the representation. Below, we catalog each major PEFT method as a hypothesis, extract pseudocode for the intervention, and weigh the evidence.

Evidence scoring

We grade evidence on independent dimensions. Each method gets points for the dimensions it satisfies:

Dim	Pts	Meaning
PE	1	Parameter-efficient: competitive with full FT at <1% params
BL	1	Beats LoRA on raw performance at comparable budget
BF	1	Matches or beats full fine-tuning
DE	1	Data-efficient: faster convergence or works with less data
OOD	2	Generalizes out-of-distribution
WA	1	Widely adopted: used as baseline by other papers

Total = sum of applicable dimensions (max 7). Higher = stronger evidence that the method's structural hypothesis is correct.

---

1. LoRA -- Low-Rank Adaptation

Paper: Hu et al. 2021 (ICLR 2022) Code: peft/tuners/lora/layer.py Saved: docs/lora_low_rank_adaptation.md

Hypothesis: Weight changes needed for task adaptation are low-rank. The residual between pretrained and fine-tuned weights lives in a small subspace, so we can parameterize \Delta W = BA with B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times d}, r \ll d.

# ── LoRA intervention ──────────────
def lora_forward(x, W, A, B, α, r):
    # W frozen, A and B learned
    scaling = α / r
    ΔW = B @ A                        # ΔW ∈ ℝ^{d_out × d_in}, rank r
    return (W + scaling * ΔW) @ x     # equivalently: W(x) + scaling * B(A(x))

Evidence: Parameter-efficient (matches full FT with 0.01% params on GPT-3). Universal baseline adopted by the entire field. Authors demonstrate comparable performance to full fine-tuning on GPT-3 175B across multiple NLU benchmarks. Subsequent work (Biderman et al. 2024) finds LoRA underperforms full FT on harder tasks and larger scale -- the low-rank assumption holds for surface-level adaptation but breaks where deep weight restructuring is needed.

Grade: PE+WA=2 (parameter-efficient, universal baseline, but ceiling on hard tasks)

---

2. OFT -- Orthogonal Fine-Tuning

Paper: Qiu et al. 2023 Code: peft/tuners/oft/layer.py Saved: docs/oft_orthogonal_finetuning.md See also: BOFT (Liu et al. 2023), OFTv2 (2025)

Hypothesis: Orthogonal transformations preserve the semantic structure of pretrained weights. The pairwise angles between neuron weight vectors (the "hyperspherical energy") encode learned knowledge; any useful adaptation should preserve these angles. W_{\text{new}} = R \cdot W where R \in O(d).

# ── OFT intervention ──────────────
def oft_forward(x, W, Q):
    # Q: learned skew-symmetric params (upper triangle of block matrices)
    Q_skew = skew_symmetric(Q, block_size)     # Q_skew ∈ ℝ^{b×k×k}, antisymmetric
    R = cayley(Q_skew)                         # R = (I + Q_skew)(I - Q_skew)^{-1} ∈ O(k)
    R_full = block_diag(R)                     # R_full ∈ O(d), block-diagonal
    # Paper: w̃ᵢ = R · wᵢ for each row, so W' = W @ R^T
    return (W @ R_full.T) @ x                  # rotate weight rows orthogonally

Evidence: Authors demonstrate OFT preserves "hyperspherical energy" (pairwise neuron angles) during adaptation, which LoRA does not. Strong results on controllable image generation (ControlNet) and subject-driven generation (DreamBooth), where semantic preservation matters. BOFT extends this with butterfly-factorized orthogonal matrices for better parameter efficiency. OFTv2 reduces computational cost from O(d^3) to O(d^2) via input-centric reformulation and outperforms QLoRA.

However: the orthogonality constraint is rigid. It prevents magnitude changes entirely, limiting adaptation on tasks that require rescaling neuron importance. The hypothesis is strongest where you want to rotate representations without distorting them.

Grade: PE=1 (parameter-efficient, strong on vision/generation, limited on NLU where magnitude changes matter)

---

3. VeRA -- Vector-based Random Matrix Adaptation

Paper: Kopiczko et al. 2023 (ICLR 2024) Code: peft/tuners/vera/layer.py Saved: docs/vera_vector_random_matrix_adaptation.md

Hypothesis: Random projections are sufficient structure; all a layer needs to learn is how much of each projected direction to use. A single pair of frozen random matrices (A, B) shared across all layers, combined with per-layer learned scaling vectors (\lambda_d, \lambda_b), can match LoRA. The implication: the specific learned subspace matters far less than you'd think -- only the per-layer scaling matters.

# ── VeRA intervention ─────────────
def vera_forward(x, W, A, B, λ_d, λ_b):
    # A ∈ ℝ^{r×d_in}, B ∈ ℝ^{d_out×r}: frozen random, shared across ALL layers
    # λ_d ∈ ℝ^r, λ_b ∈ ℝ^r: learned per-layer scaling vectors
    ΔW = (λ_b[:, None] * B) @ (λ_d[:, None] * A)   # ΔW ∈ ℝ^{d_out × d_in}
    return (W + ΔW) @ x
    # forward: result + λ_b * linear(λ_d * linear(dropout(x), A), B)

Evidence: 10x fewer trainable parameters than LoRA while maintaining competitive performance across diverse NLU benchmarks. The fact that random projections work at all is surprising and informative: it suggests that the JL lemma-style argument applies -- random subspaces approximately preserve the structure needed for adaptation, and per-layer gating is the real bottleneck.

Grade: PE=1 (extreme parameter efficiency, competitive with LoRA, random-projection ceiling on complex tasks)

---

4. DoRA -- Weight-Decomposed Low-Rank Adaptation

Paper: Liu et al. 2024 (ICML 2024) Code: peft/tuners/lora/dora.py Saved: docs/dora_weight_decomposed_lora.md

Hypothesis: Full fine-tuning decomposes weight updates into magnitude and direction components that evolve differently. LoRA conflates these two. Decomposing W = m \cdot \frac{V}{\|V\|_c} and updating them separately (magnitude as a learned scalar, direction via LoRA) better approximates full FT dynamics.

# ── DoRA intervention ─────────────
def dora_forward(x, W, A, B, m, α):
    # m ∈ ℝ^{d_out}: learned magnitude per output neuron
    # A, B: LoRA matrices for directional update
    ΔW = B @ A                                    # directional update, rank r
    V = W + α * ΔW                                # updated weight (direction)
    V̂_norm = norm(V, dim=1).detach()              # column norms, detached
    scale = m / V̂_norm                            # magnitude / direction_norm
    return scale * (W @ x) + scale * α * (B @ A @ x)

Evidence: Authors analyze full FT weight updates and find they exhibit distinct magnitude vs. direction patterns that LoRA misses. DoRA consistently outperforms LoRA on LLaMA (commonsense reasoning), LLaVA (visual instruction tuning), and VL-BART (image/video-text), across multiple scales. No additional inference overhead (magnitudes merge). Has become a widely-adopted LoRA variant and default in many pipelines.

Grade: PE+BL+WA=3 (beats LoRA across multiple domains, now a standard strong baseline)

Implications: The magnitude/direction decomposition reveals something about how full FT works internally. Weight updates are not just "adding stuff" -- they redistribute energy across neurons (magnitude) independently of rotating their selectivity (direction). This connects to the neuroscience intuition that gain modulation and selectivity tuning are separate mechanisms.

---

5. DeLoRA -- Decoupled Low-Rank Adaptation

Paper: Bini, Girrbach, Akata 2025 (ICLR 2025) Code: peft/tuners/delora/layer.py Saved: docs/delora_decoupled_low_rank_adaptation.md

Hypothesis: The direction of a weight update (which features to mix) and its strength (how far to deviate from pretrained weights) should be explicitly decoupled. LoRA conflates them via learning rate; ETHER fixes them. DeLoRA normalizes each rank-1 component of BA by its norms and introduces a learnable scalar \lambda controlling the distance bound. This yields robustness (bounded deviation) without sacrificing expressivity (arbitrary rank).

# ── DeLoRA intervention ───────────
def delora_forward(x, W, A, B, λ, r, w_norm):
    # A ∈ ℝ^{r×d_in}, B ∈ ℝ^{d_out×r}: learned (like LoRA)
    # λ ∈ ℝ^r: learned per-component scaling (strength)
    # w_norm ∈ ℝ^{d_in}: frozen ||W||_col from init
    Â_norm = clamp(norm(A, dim=1), min=1e-4)        # ∈ ℝ^r
    B̂_norm = clamp(norm(B, dim=0), min=1e-4)        # ∈ ℝ^r
    scaling = (λ / r) / (Â_norm * B̂_norm)            # normalize each rank-1 component
    ΔW = B @ diag(scaling) @ A                       # direction normalized, strength via λ
    return W(x) + (x * w_norm) @ A.T @ diag(scaling) @ B.T

The key insight: \Delta W = B \cdot \text{diag}\left(\frac{\lambda}{r \cdot \|a_i\| \cdot \|b^j\|}\right) \cdot A. Each rank-1 outer product b_i a_i^\top is normalized to unit norm, then scaled by \lambda_i / r. The angular component (which direction in weight space to move) trains freely; the radial component (how far) is controlled by \lambda.

Evidence: DeLoRA matches or surpasses LoRA, DoRA, and ETHER on subject-driven generation (DreamBooth), NLU (GLUE), and instruction tuning (LLaMA), while showing much better robustness to learning rate and training duration. The bounded deviation prevents catastrophic overwriting that plagues LoRA at high LR. Same authors as ETHER (Bini, Girrbach, Akata), extending their work on direction/strength decoupling.

Grade: PE+BL=2 (beats LoRA on robustness and competitive on performance; ICLR 2025)

Implications: The strength/direction decoupling means gradient updates drive angular learning only -- the optimizer doesn't waste capacity fighting magnitude dynamics. Predictions: methods that explicitly decouple direction from strength will systematically show better OOD transfer, because the direction captures what to change while the strength captures how much, and only the former should be task-invariant.

---

6. PiSSA -- Principal Singular Values and Singular Vectors Adaptation

Paper: Meng, Wang, Zhang 2024 (NeurIPS 2024) Code: github.com/GraphPKU/PiSSA Saved: docs/pissa_principal_singular_values_adaptation.md

Hypothesis: The principal components of each weight matrix are what matter for adaptation. LoRA initializes adapters with random noise + zeros, so it starts far from the important subspace and converges slowly. PiSSA initializes A and B from the top-r SVD of W, then freezes the residual W_{\text{res}}. Same architecture as LoRA, but trains the most important directions first.

# ── PiSSA initialization + intervention ──
def pissa_init(W, r):
    U, S, Vt = svd(W)                              # W ∈ ℝ^{m×n}
    A = U[:, :r] @ diag(sqrt(S[:r]))                # A ∈ ℝ^{m×r}, principal left
    B = diag(sqrt(S[:r])) @ Vt[:r, :]               # B ∈ ℝ^{r×n}, principal right
    W_res = U[:, r:] @ diag(S[r:]) @ Vt[r:, :]     # residual, frozen
    return A, B, W_res

def pissa_forward(x, W_res, A, B):
    return (W_res + A @ B) @ x                      # same as LoRA at inference

The decomposition: W = \underbrace{U_{:r} S_{:r} V_{:r}^\top}_{\text{adapter (learned)}} + \underbrace{U_{r:} S_{r:} V_{r:}^\top}_{\text{residual (frozen)}}. LoRA updates noise; PiSSA updates the signal.

Evidence: PiSSA consistently outperforms LoRA across 11 models (184M--70B) on 5 NLG and 8 NLU tasks under identical setups. Gemma-7B on GSM8K: PiSSA 77.7% vs LoRA 74.5%. QPiSSA (quantized) on LLaMA-3-70B GSM8K: 86.05% vs QLoRA 81.73%. Faster convergence because the optimizer starts in the high-signal subspace. The initialization cost is negligible (fast SVD, a few seconds).

Grade: PE+BL+BF+DE=4 (beats LoRA, approaches/beats full FT, faster convergence, NeurIPS 2024)

Implications: PiSSA tells us something crucial about which weight-space directions matter: the top singular directions encode the most task-relevant structure. This is the "principal components carry the signal" hypothesis. It also suggests that LoRA's random init wastes early training steps re-discovering what SVD gives you for free. Connects to the broader question: is model adaptation about modifying the dominant signal or the residual noise? PiSSA says: the signal, always the signal.

---

7. SVFT -- Singular Vector Fine-Tuning

Paper: Lingam et al. 2024 Code: github.com/VijayLingam95/SVFT Saved: docs/svft_svd_coefficient_finetuning.md

Hypothesis: The structure of \Delta W should depend on the specific weight matrix W. SVFT fixes both left and right singular vectors (from $W$'s own SVD) and learns only a sparse set of coefficients for their outer products. The weight matrix's own geometry defines the intervention basis; we just rescale which combinations of its existing directions to amplify or suppress.

# ── SVFT intervention ─────────────
def svft_init(W, k):
    U, S, Vt = svd(W)                          # W ∈ ℝ^{m×n}
    # select k (i,j) pairs from {0..m-1} x {0..n-1}
    indices = select_sparse_pairs(k)            # e.g. band-diagonal, random
    c = zeros(k)                                # learned coefficients
    return U, Vt, indices, c                    # U, Vt frozen

def svft_forward(x, W, U, Vt, indices, c):
    ΔW = sum(c[t] * outer(U[:, i], Vt[j, :]) for t, (i,j) in enumerate(indices))
    return (W + ΔW) @ x                        # sparse combo of singular vector outer products

The key: \Delta W = \sum_{t} c_t \cdot u_{i_t} v_{j_t}^\top, where u_i, v_j come from $W$'s SVD. Only the c_t scalars are learned. Different sparsity patterns (band-diagonal, random, etc.) give different expressivity/efficiency tradeoffs.

Evidence: SVFT recovers up to 96% of full fine-tuning performance with only 0.006--0.25% of parameters, outperforming LoRA/DoRA/BOFT which only recover 85% with 0.03--0.8% of params. Strong results on language (GLUE, commonsense reasoning) and vision benchmarks. The weight-dependent structure is the key differentiator.

Grade: PE+BL=2 (beats LoRA/DoRA on performance/parameter tradeoff, weight-aware structure)

Implications: SVFT is the purest test of "does the model's own SVD basis define the right intervention space?" The answer appears to be yes: learning just coefficients over the model's own singular vectors is far more efficient than learning new arbitrary directions. This provides direct evidence that these singular vectors aren't arbitrary artifacts but encode meaningful computational directions. If combined with PiSSA's "top components matter most," we get a clear picture: the SVD basis is the natural coordinate system, and the singular values are the knobs.

---

8. SSVD -- Structured SVD-Guided Fine-Tuning

Paper: Wang, Watanabe, Van hamme 2025 Saved: docs/ssvd_structured_svd_finetuning.md

Hypothesis: Input-space (right singular vectors V) and output-space (left singular vectors U) serve fundamentally different roles. Adaptation should rotate the input feature space to align with domain-shifted inputs while preserving the output semantic mappings. The right singular vectors define "what the layer listens to"; the left define "what it says". In domain shift, what you listen to changes, but what you say should stay.

# ── SSVD intervention ─────────────
def ssvd_init(W, k):
    U, Σ, Vt = svd(W)                           # W ∈ ℝ^{m×n}
    K = zeros(k, k)                              # learned skew-symmetric matrix
    ΔΣ = zeros(k)                                # learned singular value shifts
    return U, Σ, Vt, K, ΔΣ                      # U, Σ, Vt frozen; K, ΔΣ learned

def ssvd_forward(x, U, Σ, Vt, K, ΔΣ, k):
    G_k = cayley(K)                              # G_k = (I-K)(I+K)^{-1} ∈ O(k)
    Σ̂ = Σ.clone()
    Σ̂[:k] += ΔΣ                                  # shift top-k singular values
    V̂t = Vt.clone()
    V̂t[:k] = G_k @ Vt[:k]                        # rotate top-k right singular vectors
    return U @ diag(Σ̂) @ V̂t @ x                  # W' = U (Σ+ΔΣ) G Vt x

W' = U (\Sigma + \Delta\Sigma) \, G_k \, V^\top

Only k(k-1)/2 + k parameters (skew-symmetric entries + singular value shifts). Uses Cayley-Neumann approximation for efficiency.

Evidence: SSVD achieves comparable performance to LoRA, DoRA, PiSSA, VeRA, and SVFT on domain-shifted ASR (child speech, dialectal variation) across 0.1B--2B models, with significantly fewer trainable parameters. On OWSM-1B: SSVD matches LoRA WER with 10M fewer params. At larger scales, a convergence hierarchy emerges: SSVD > PiSSA > DoRA > LoRA. The gap grows with model scale, suggesting the asymmetric hypothesis becomes more valid as models get larger.

Grade: PE+BL+DE=3 (matches/beats LoRA with fewer params on domain-shifted ASR, faster convergence at scale)

Implications: SSVD's asymmetric treatment of U vs V is novel and deeply informative. It says: the model's "output vocabulary" (left singular vectors = what abstract features get produced) is already correct and should be preserved. Only the "input receptive fields" (right singular vectors = how raw features map into the abstract space) need updating for domain shift. This is exactly the right inductive bias for acoustic adaptation (accents change the input distribution, not the semantic targets). Predictions: this asymmetry should also work for visual domain adaptation (camera changes, lighting) but fail for tasks that require redefining the output space (new task types, new label semantics).

---

9. IA3 -- Infused Adapter by Inhibiting and Amplifying Inner Activations

Paper: Liu et al. 2022 Code: peft/tuners/ia3/layer.py Saved: docs/ia3_few_shot_peft.md

Hypothesis: Task adaptation is mostly about rescaling what the model already computes, not restructuring it. A learned vector that element-wise scales activations at key, value, and FFN layers suffices. The pretrained model already extracts the right features; you just need to amplify the relevant ones and suppress the irrelevant ones. This is the "gain control" hypothesis -- adaptation as a gating/attention mechanism over existing channels.

# ── IA3 intervention ──────────────
def ia3_forward(x, W, λ, is_feedforward):
    # λ ∈ ℝ^d: learned scaling vector, init to 1.0 (identity)
    if is_feedforward:
        return W @ (x * λ)             # scale input channels: amplify/suppress features
    else:
        return (W @ x) * λ             # scale output channels: amplify/suppress neurons

Merge into weights: W_{\text{merged}} = W \odot \lambda (element-wise scaling of rows or columns). Extremely few trainable parameters -- just one $d$-dimensional vector per adapted layer.

Evidence: Authors claim (IA)3 with T0-3B outperforms ICL with GPT-3 175B on Super-NaturalInstructions while being orders of magnitude cheaper. Competitive with LoRA on RAFT leaderboard (rank 2 vs 3) with far fewer params. Strong on T5-family models. However, scaling-only methods have a clear expressivity ceiling: they cannot introduce new feature interactions, only reweight existing ones.

Grade: PE=1 (parameter-efficient, strong on T5-family, expressivity-limited compared to LoRA/DoRA)

Implications: IA3's success tells us that a surprisingly large fraction of "task adaptation" is just reweighting. The pretrained model already computes many useful features; the bottleneck is which ones to attend to for a given task, not computing new features from scratch. This connects to the neuroscience concept of "gain modulation" -- neurons don't change their tuning curves, just their amplitude. The limitation is equally informative: IA3 struggles on tasks requiring novel feature combinations, confirming that some adaptations genuinely require new weight-space directions, not just rescaling.

---

10. ROAD -- Rotary Adaptation

Paper: Petrushkov 2024 Code: peft/tuners/road/layer.py Saved: docs/road_rotary_adaptation.md

Hypothesis: Adaptation is a rotation of activation pairs, with independently controllable angle (which direction to rotate) and magnitude (how much to scale). The output space splits into 2D subspaces, and within each, a learned rotation + scaling suffices. This explicitly decouples "what to change" (angle \theta) from "how much" (magnitude \alpha), making the adaptation strength a continuous knob.

# ── ROAD intervention ─────────────
def road_forward(x, W, θ, α, group_size):
    # θ ∈ ℝ^{d/2}: learned rotation angles per pair
    # α ∈ ℝ^{d/2}: learned magnitudes per pair, init 1.0
    result = W @ x                                  # base linear output ∈ ℝ^d
    x1, x2 = split_groups(result, group_size)       # split into paired halves
    y1 = α * cos(θ) * x1 - α * sin(θ) * x2         # 2D rotation + scale
    y2 = α * sin(θ) * x1 + α * cos(θ) * x2         # per pair
    return interleave(y1, y2)

R_i = \alpha_i \begin{pmatrix} \cos\theta_i & -\sin\theta_i \\ \sin\theta_i & \cos\theta_i \end{pmatrix}

Applied element-wise (no matrix multiply needed at inference). Merges into weights via W_{\text{new}} = R \cdot W.

Evidence: ROAD is the only PEFT method besides LoRA that supports mixed adapter batches (different adapters for different samples in the same batch). Authors claim competitive with LoRA on standard benchmarks. The explicit angle/magnitude decoupling makes it ideal for contrastive steering: scale only \alpha for bidirectional control while preserving learned rotation directions \theta.

Grade: PE=1 (parameter-efficient, clean decoupling, competitive with LoRA, limited published benchmarks)

Implications: ROAD's decoupling of angle from magnitude is the cleanest formulation of the "direction vs strength" principle that also appears in DeLoRA and DoRA. The 2D rotation structure connects to RoPE (rotary position embeddings) -- both use paired rotations in subspaces, suggesting this is a natural symmetry of transformer representations. For steering applications, ROAD's explicit \alpha parameter is the most interpretable knob: \alpha = 1 is identity, \alpha > 1 amplifies, \alpha < 1 attenuates, \alpha = -1 reverses.

---

11. AntiPaSTO -- Antiparallel Steering via SVD Rotations

Disclosure: this is my own work. I give it the highest grade here, so read the evidence with appropriate skepticism.

Paper: Clark 2025 Code: github.com/wassname/AntiPaSTO Saved: docs/antipasto_antiparallel_steering.md

Hypothesis: The model's own SVD basis defines the natural intervention manifold. Steering is best done by rotating singular vectors via Cayley transform on a learned skew-symmetric matrix, parameterized by a single coefficient \alpha \in [-1, +1]. The Cayley transform guarantees exact orthogonality and exact reversibility: R(-\alpha) = R(\alpha)^{-1}. Separating rotation (learned direction) from magnitude (\alpha) yields antiparallel steering -- the same adapter produces opposite behavioral shifts at \alpha = \pm 1.

The core claim synthesizes SSVD + PiSSA + DeLoRA: use the model's own top-r SVD basis (PiSSA), rotate right singular vectors via Cayley (SSVD), decouple direction from strength (DeLoRA), and add learnable singular value shifts.

# ── AntiPaSTO intervention ────────
def antipasto_init(W, r):
    U, S, Vt = svd(W)                              # W ∈ ℝ^{m×n}
    U_r, S_r, V_r = U[:, :r], S[:r], Vt[:r].T      # top-r components
    W_res = W - U_r @ diag(S_r) @ V_r.T             # residual (frozen)
    A_v = zeros(r, r)                                # skew-symmetric rotation params for V
    ΔS = zeros(r)                                    # learnable singular value shifts
    return U_r, S_r, V_r, W_res, A_v, ΔS            # U,S,V,W_res frozen; A_v,ΔS learned

def antipasto_forward(x, U, S, V, W_res, A_v, ΔS, α):
    # α ∈ [-1, +1]: steering coefficient (continuous knob)
    X = α * A_v / 2                                  # scale skew-symmetric params
    R_v = solve(I - X, I + X)                        # Cayley: (I - αA/2)^{-1}(I + αA/2) ∈ O(r)
    V_rot = V @ R_v                                  # rotate input-space basis
    S_scaled = S + α * ΔS                            # shift singular values
    # Efficient: x @ V_rot @ diag(S_scaled) @ U^T + x @ W_res^T
    h = (x @ V_rot) * S_scaled @ U.T                 # adapted path
    return h + x @ W_res.T                           # + residual

W'(\alpha) = U \, \text{diag}(S + \alpha \Delta S) \, R_v(\alpha) \, V^\top + W_{\text{res}}

where R_v(\alpha) = (I - \alpha A/2)^{-1}(I + \alpha A/2) is the Cayley transform of skew-symmetric A. Only r(r-1)/2 + r learned parameters per layer.

Evidence: AntiPaSTO beats prompting baselines by 6.9x on DailyDilemmas honesty evaluation using Gemma-3-1B. Maintains bidirectional control (\alpha = \pm 1) where prompting triggers refusal. Trains with only 800 contrastive word pairs (no preference labels). Transfers out-of-distribution from template sentences to real ethical dilemmas. The OOD transfer suggests the SVD rotation basis captures something causally relevant about the model's honesty computations.

Grade: PE+DE+OOD=4 (OOD transfer from templates to real dilemmas, trains on 800 pairs, bidirectional control)

Caveat: Primary evidence is on models up to 4B parameters. The paper notes larger models "need further exploration" and results show high seed variance. The OOD transfer claim is strong but narrow (one trait, one evaluation benchmark).

Implications: Synthesizes PiSSA's SVD initialization, SSVD's input-space rotation, DeLoRA's direction/strength decoupling, and OFT's Cayley parameterization. The OOD transfer from templates to real dilemmas suggests the SVD manifold is causally relevant, not just efficient. The antiparallel property (+\alpha and -\alpha produce opposite effects) is a natural consequence of rotational symmetry. Caveat: single-author, one trait, high seed variance; would be more confident with independent replication.

---

12. AdaLoRA -- Adaptive Budget Allocation for LoRA

Paper: Zhang et al. 2023 (ICLR 2023) Code: peft/tuners/adalora Saved: docs/adalora_adaptive_budget.md

Hypothesis: Not all layers need the same rank. The optimal rank distribution across layers is adaptive and should be learned during training. Some weight matrices need high-rank updates (they are task-critical); others need almost none. SVD-based importance scoring can dynamically prune less important singular values, reallocating budget where it matters.

# ── AdaLoRA intervention ──────────
def adalora_forward(x, W, P, Λ, Q):
    # P ∈ ℝ^{d_out×r}, Q ∈ ℝ^{r×d_in}: left/right singular vectors (learned)
    # Λ ∈ ℝ^r: singular values (learned, prunable via importance mask)
    ΔW = P @ diag(Λ) @ Q                          # SVD-parameterized update
    return (W + ΔW) @ x

def prune_step(P, Λ, Q, budget):
    importance = compute_importance(P, Λ, Q)       # sensitivity-based scoring
    mask = top_k(importance, budget)               # keep top-budget components
    Λ_pruned = Λ * mask                            # zero out unimportant
    return Λ_pruned

Evidence: Authors claim AdaLoRA achieves comparable or better performance than LoRA with 30-50% fewer total parameters on DeBERTaV3-base across NLU tasks. The adaptive rank allocation concentrates budget on query/value projections and early/late layers. Orthogonal regularization on P, Q prevents degenerate solutions. However, the pruning adds training complexity and the final rank pattern is model/task-specific, limiting transferability of the insight.

Grade: PE=1 (parameter-efficient, smarter budget allocation, added complexity for modest gains)

---

13. BOFT -- Butterfly Orthogonal Fine-Tuning

Paper: Liu et al. 2023 (ICLR 2024) Code: peft/tuners/boft Saved: docs/boft_butterfly_orthogonal.md

Hypothesis: Orthogonal transformations (OFT's key insight) are right, but the full block-diagonal parameterization is wasteful. Butterfly factorizations -- the same structure behind the FFT -- can represent arbitrary orthogonal transformations with O(d \log d) parameters instead of O(d^2), while maintaining the information-theoretic expressiveness needed for adaptation.

# ── BOFT intervention ─────────────
def boft_forward(x, W, butterfly_blocks, n_layers):
    R = eye(d)
    for l in range(n_layers):                       # log(d) butterfly layers
        B_l = block_diag(butterfly_blocks[l])       # sparse butterfly factor
        R = R @ B_l                                 # compose: R ∈ O(d)
    return (R @ W) @ x                             # orthogonal rotation of W

Each butterfly layer has d/2 independent 2\times2 rotation blocks arranged in a permuted pattern. Composing \log_2(d) layers can represent any orthogonal matrix.

Evidence: BOFT matches or exceeds OFT performance on DreamBooth and ControlNet with 2-4x fewer parameters. Authors demonstrate it preserves hyperspherical energy like OFT. The butterfly structure provides a principled trade-off between expressiveness and parameter count. Strong on vision/generation tasks where semantic preservation matters. ICLR 2024 acceptance validates the contribution.

Grade: PE=1 (strict improvement over OFT in parameter efficiency, same hypothesis, ICLR 2024)

---

14. GOFT -- Givens Orthogonal Fine-Tuning

Paper: Ma et al. 2024 (ICML 2024) Code: github.com/ArthurLeoM/peft-givens Saved: docs/goft_givens_orthogonal.md

Hypothesis: Any orthogonal transformation in SO(d) can be decomposed into O(d) Givens rotations (planar rotations in 2D subplanes), reducing parameter complexity from O(d^2) to O(d). This is the most parameter-efficient parameterization of orthogonal adaptation. Beyond strict orthogonality, soft orthogonality regularization allows controlled norm and angular adjustment.

# ── GOFT intervention ─────────────
def goft_forward(x, W, θ_list, pairs):
    # θ_list ∈ ℝ^{d}: rotation angles for d Givens rotations
    # pairs: which (i,j) dimensions each rotation acts on
    R = eye(d)
    for θ, (i, j) in zip(θ_list, pairs):
        G = givens_rotation(d, i, j, θ)            # identity except 2x2 block at (i,j)
        R = R @ G                                   # compose all rotations
    # With soft orthogonality, also learn norm adjustments
    return (R @ W) @ x

Evidence: Authors claim GOFT outperforms OFT and BOFT on LLaMA-2-7B SFT (MT-Bench, AlpacaEval), DreamBooth, and offline RL tasks while using significantly fewer parameters. The parallel rotation strategy achieves O(\log d) sparse matrix multiplication. ICML 2024 acceptance. The Givens decomposition is mathematically elegant and provably equivalent to full orthogonal transformations.

Grade: PE=1 (most parameter-efficient orthogonal method, strong results, ICML 2024)

---

15. HRA -- Householder Reflection Adaptation

Paper: Yuan et al. 2024 Code: peft/tuners/hra Saved: docs/hra_householder_reflection.md

Hypothesis: Orthogonal adaptations are equivalent to specific low-rank adaptations when parameterized via Householder reflections. A chain of r Householder reflections H_1 H_2 \cdots H_r (each defined by a single vector v_i) constructs an orthogonal matrix with exactly r \times d learnable parameters -- bridging the low-rank and orthogonal adaptation paradigms.

# ── HRA intervention ──────────────
def hra_forward(x, W, V):
    # V ∈ ℝ^{r×d}: r Householder reflection vectors
    R = eye(d)
    for i in range(r):
        v = V[i]                                    # reflection normal ∈ ℝ^d
        H_i = eye(d) - 2 * outer(v, v) / dot(v, v) # Householder reflector
        R = R @ H_i                                  # compose: R ∈ O(d)
    return (R @ W) @ x

Each reflection flips the space across a hyperplane. Composing r of them gives a rank-r "distance" from identity while staying exactly orthogonal.

Evidence: Authors demonstrate HRA achieves competitive or better results than LoRA and OFT on LLaMA fine-tuning and image generation. The theoretical equivalence between Householder chains and adaptive low-rank updates is the main contribution: same expressiveness as rank-r LoRA with guaranteed orthogonality. Regularization on reflection plane orthogonality improves stability.

Grade: PE=1 (bridges orthogonal and low-rank paradigms, competitive performance)

Implications: HRA reveals that the "low-rank vs orthogonal" dichotomy is a false one. A chain of r Householder reflections is both orthogonal and equivalent to a rank-r perturbation. This means LoRA's success (low rank works) and OFT's success (orthogonality works) are compatible: the effective adaptation might be low-rank and approximately orthogonal simultaneously. If true, the right constraint isn't "low rank" or "orthogonal" alone, but "low-rank orthogonal" -- small rotations that stay on the Stiefel manifold.

---

16. RandLoRA -- Random Matrix LoRA

Paper: Albert et al. 2025 (ICLR 2025) Code: peft/tuners/randlora Saved: docs/randlora_random_matrix.md

Hypothesis: LoRA's rank bottleneck (\text{rank}(\Delta W) \leq r) limits expressiveness. By summing n = d/r scaled random rank-r bases, the update \Delta W = \sum_j B_j \Lambda_j A \Gamma_j achieves full rank while learning only diagonal scaling matrices. Each frozen random basis B_j, A spans a different subspace; the learnable scalings \Lambda_j, \Gamma_j select how much of each to use.

# ── RandLoRA intervention ─────────
def randlora_forward(x, W, A, B_list, Λ_list, Γ_list):
    # A ∈ ℝ^{r×d_in}: shared frozen random matrix
    # B_list: n frozen random matrices, each B_j ∈ ℝ^{d_out×r}
    # Λ_list: n learned diagonal scalings, each Λ_j ∈ ℝ^{r×r}
    # Γ_list: n learned diagonal scalings, each Γ_j ∈ ℝ^{d×d}
    ΔW = sum(B_j @ Λ_j @ A @ Γ_j for B_j, Λ_j, Γ_j  # sum of n rank-r terms = full rank
             in zip(B_list, Λ_list, Γ_list))
    return (W + ΔW) @ x

Evidence: RandLoRA outperforms LoRA as parameter budget expands, while remaining parameter-efficient. DinoV2, CLIP, and LLaMA-3-8B experiments show LoRA hits a rank ceiling (increasing rank has diminishing returns) while RandLoRA continues to improve. Loss landscape analysis shows RandLoRA's local minima are closer to full fine-tuning's. ICLR 2025.

Grade: PE=1 (full-rank with learned scalings only, ICLR 2025, strong on vision-language)

---

17. FourierFT -- Fourier Fine-Tuning

Paper: Gao et al. 2024 (ICML 2024) Code: peft/tuners/fourierft Saved: docs/fourierft_spectral.md

Hypothesis: Weight updates \Delta W are spectrally sparse -- they can be represented by a small number of Fourier coefficients. Instead of parameterizing \Delta W in the spatial domain (like LoRA), learn a sparse set of spectral coefficients and reconstruct via inverse DFT. This exploits the observation that useful weight changes tend to be smooth/structured rather than random.

# ── FourierFT intervention ────────
def fourierft_forward(x, W, coeffs, freq_indices, shape):
    # coeffs ∈ ℂ^k: learned spectral coefficients (k << m*n)
    # freq_indices: which frequency components to learn
    spectrum = zeros(shape, dtype=complex)
    spectrum[freq_indices] = coeffs                  # sparse spectrum
    ΔW = real(ifft2(spectrum))                       # inverse 2D DFT
    return (W + ΔW) @ x

Evidence: Authors claim FourierFT achieves higher compression than LoRA by exploiting frequency-domain sparsity. Competitive with LoRA on GLUE and commonsense reasoning using fewer parameters. ICML 2024 acceptance. The spectral sparsity hypothesis is interesting but the evidence for why weight changes should be low-frequency is largely empirical.

Grade: PE=1 (novel parameterization, ICML 2024, competitive compression)

---

18. C3A -- Circular Convolution Adaptation

Paper: Phoveran et al. 2024 (ACL 2025) Code: peft/tuners/c3a Saved: docs/c3a_circular_convolution.md

Hypothesis: Weight updates have circulant structure -- the matrix \Delta W is approximately a circulant matrix (each row is a cyclic shift of the previous). Circulant matrices are diagonalized by the DFT, so efficient computation via FFT is possible. Unlike LoRA which is rank-limited, circulant matrices can have full rank with only d parameters (one generating vector).

# ── C3A intervention ──────────────
def c3a_forward(x, W, c):
    # c ∈ ℝ^d: generating vector for circulant matrix
    ΔW = circulant(c)                               # ΔW[i,j] = c[(j-i) mod d]
    # Efficient via FFT: ΔW @ x = ifft(fft(c) * fft(x))
    return (W + ΔW) @ x

Evidence: Authors claim C3A achieves higher effective rank than LoRA with similar parameter count and compute. Competitive on GLUE, commonsense reasoning, and instruction tuning. ACL 2025 acceptance. The FFT-based computation is genuinely efficient. However, the assumption of circulant structure in weight updates is strong and may not hold universally.

Grade: PE=1 (full-rank with fewer params, ACL 2025, circulant assumption is strong)

---

19. LoHa -- Low-Rank Hadamard Product

Paper: Hyeon-Woo et al. 2021 (FedPara; adapted in LyCORIS) Code: peft/tuners/loha Saved: docs/loha_hadamard_product.md

Hypothesis: Weight updates have multiplicative structure that a single low-rank factorization misses. By combining two low-rank decompositions via Hadamard (element-wise) product, more complex interaction patterns can be captured. (A_1 B_1) \odot (A_2 B_2) can represent higher-rank updates than either factor alone.

# ── LoHa intervention ─────────────
def loha_forward(x, W, A1, B1, A2, B2):
    # Each pair (Ai, Bi): rank-r decomposition
    ΔW = (A1 @ B1) * (A2 @ B2)                     # Hadamard product, potentially full-rank
    return (W + ΔW) @ x

Evidence: Part of the LyCORIS toolkit. Authors claim LoHa achieves richer expressiveness than LoRA for the same parameter count, particularly for image generation (Stable Diffusion fine-tuning) where complex spatial interactions matter. The Hadamard product inherently captures pairwise feature interactions that additive low-rank matrices cannot.

Grade: PE=1 (richer than LoRA for vision, part of LyCORIS ecosystem)

---

20. LoKr -- Low-Rank Kronecker Product

Paper: Yeh et al. 2023 (LyCORIS) Code: peft/tuners/lokr Saved: docs/lokr_lycor.md

Hypothesis: Weight updates have tensor product structure. The Kronecker factorization \Delta W = A \otimes B decomposes a large matrix into the tensor product of two smaller ones, exploiting multi-scale or block-structured patterns in adaptation. Especially efficient for high-dimensional or convolutional weight matrices.

# ── LoKr intervention ─────────────
def lokr_forward(x, W, A, B):
    # A ∈ ℝ^{m1×m2}, B ∈ ℝ^{n1×n2}, where m1*n1 = d_out, m2*n2 = d_in
    ΔW = kron(A, B)                                 # ΔW ∈ ℝ^{d_out × d_in}
    return (W + ΔW) @ x

Evidence: Part of LyCORIS. Kronecker structure is especially effective for convolutional layers where the weight tensor naturally factorizes across spatial and channel dimensions. Compact parameterization for large weight matrices. Less commonly used for LLMs where the spatial structure assumption doesn't hold as well.

Grade: PE=1 (efficient for conv layers, niche use case for transformers)

---

21. MiSS -- Matrix Shard Sharing

Paper: JL-er 2024 Code: peft/tuners/miss Saved: docs/miss_matrix_shard_sharing.md

Hypothesis: Weight updates share structural motifs across layers. Instead of learning independent low-rank matrices per layer, share "shards" (small matrix blocks) across layers through a weight-magnitude-based scoring system. Layers with similar function should reuse similar update patterns, and the scoring identifies which layers are similar.

# ── MiSS intervention ─────────────
def miss_forward(x, W, shared_shards, scores):
    # shared_shards: global bank of small matrix blocks
    # scores: per-layer importance weights selecting which shards to use
    ΔW = assemble(shared_shards, scores)            # weighted combination of shards
    return (W + ΔW) @ x

Evidence: Successor to Bone (deprecated). PEFT benchmark comparison shows "excellent results" in both performance and memory efficiency. Adaptive rank allocation via shard scoring. Reduced memory compared to full per-layer LoRA matrices. However, the shard sharing mechanism adds implementation complexity.

Grade: PE=1 (memory-efficient, good benchmark results per PEFT team)

---

22. VBLoRA -- Vector Bank LoRA

Paper: Li et al. 2024 (NeurIPS 2024) Code: peft/tuners/vblora Saved: docs/vblora_vector_bank.md

Hypothesis: Adapter weight matrices are sparse combinations of shared atomic vectors. Instead of learning full low-rank matrices, maintain a shared "vector bank" and select/combine top-k vectors per layer. This is a codebook/dictionary learning approach: the adaptation vocabulary is shared globally, and each layer's adapter is a sparse code over it.

# ── VBLoRA intervention ───────────
def vblora_forward(x, W, bank, indices, coeffs):
    # bank ∈ ℝ^{V×d}: shared vector bank (V vectors)
    # indices ∈ ℤ^k: top-k selected vectors per layer
    # coeffs ∈ ℝ^k: combination weights
    selected = bank[indices]                         # k most relevant vectors
    ΔW = sum(coeffs[i] * outer(selected[i]) for i in range(k))  # sparse reconstruction
    return (W + ΔW) @ x

Evidence: Authors claim VBLoRA uses 0.4% of LoRA's parameters while maintaining comparable performance. NeurIPS 2024 acceptance. The extreme compression is remarkable and suggests that adapter weight diversity across layers is much lower than assumed -- most of the information is in which vectors to select and how much of each, not in the vectors themselves.

Grade: PE=1 (extreme compression, NeurIPS 2024, intriguing theoretical implications)

---

23. SHiRA -- Sparse High-Rank Adapters

Paper: KKB et al. 2024 (NeurIPS 2024 Workshop) Code: peft/tuners/shira Saved: docs/shira_sparse_high_rank.md

Hypothesis: The right parameterization isn't low-rank or full-rank, but sparse high-rank. Directly fine-tune 1-2% of the base model's weights, selected by importance scoring. The updated weights can have full rank (no rank bottleneck), but the sparsity pattern constrains which parameters change. The hypothesis: a small fraction of weights are "task-critical knobs" that, when tuned, achieve most of adaptation's benefit.

# ── SHiRA intervention ────────────
def shira_forward(x, W, mask, ΔW_sparse):
    # mask ∈ {0,1}^{d_out × d_in}: 1-2% of entries are 1
    # ΔW_sparse: learned updates at mask positions only
    W_adapted = W + mask * ΔW_sparse                # sparse but full-rank update
    return W_adapted @ x

Evidence: Authors claim SHiRA outperforms LoRA especially on concept loss when using multiple adapters (critical for diffusion model fine-tuning). Sparse adapters are cheaper to switch between than LoRA. NeurIPS 2024 Workshop. The importance-scoring approach connects to structured pruning literature.

Grade: PE=1 (sparse high-rank, good multi-adapter properties, workshop paper)

---

24. LN Tuning -- LayerNorm Tuning

Paper: undated Code: peft/tuners/ln_tuning

Hypothesis: Normalization layers (LayerNorm/RMSNorm) are the distribution controllers of the network. Tuning only their affine parameters (\gamma, \beta) adapts how each layer normalizes its inputs, which is sufficient for many tasks because distribution shift is the primary thing that changes between pretraining and fine-tuning.

# ── LN Tuning intervention ────────
def ln_tuning_forward(x, W_frozen, γ, β):
    # Only γ ∈ ℝ^d and β ∈ ℝ^d are trainable (LayerNorm params)
    x_norm = (x - mean(x)) / std(x) * γ + β       # adapted normalization
    return W_frozen @ x_norm                        # rest of network frozen

Evidence: Authors claim LN Tuning with ~0.5% trainable parameters can match LoRA performance on some NLU tasks. The extreme simplicity is informative: if tuning only normalization suffices, then much of "task adaptation" is really "distribution matching." Less effective on tasks requiring new feature representations rather than feature rescaling.

Grade: PE=1 (extremely few params, competitive on some tasks, limited expressiveness)

---

25. Prompt & Prefix Tuning -- Learned Virtual Tokens

Papers: Prompt Tuning (Lester et al. 2021), Prefix Tuning (Li & Liang 2021), P-Tuning v2 (Liu et al. 2022), Adaption Prompt / LLaMA-Adapter (Zhang et al. 2023), Multitask Prompt Tuning (Asai et al. 2023), CPT (Tsachiblau 2024)

Hypothesis: The model's prompt/context is the primary interface for task specification. Learning "virtual tokens" (continuous embeddings prepended to the input) provides enough signal for downstream tasks without modifying any model weights. The hypothesis: the model's computation is already capable of the target task; it just needs the right "instruction" in embedding space. This is the "models are instruction-following programs" view.

# ── Prompt Tuning intervention ────
def prompt_tuning_forward(x, model, P):
    # P ∈ ℝ^{k×d}: k learned prompt vectors (virtual tokens)
    x_prompted = cat([P, x], dim=seq)               # prepend prompts
    return model(x_prompted)                        # model is fully frozen

Variants: Prefix Tuning adds prompts to key/value projections at every layer. P-Tuning v2 applies deep prompts to all layers. LLaMA-Adapter uses zero-initialized gating. CPT uses adversarial-inspired optimization for context-aware prompts.

Evidence: Prompt Tuning scales with model size: at T5-XXL (11B), it matches full fine-tuning with 0.01% parameters. However, it struggles on smaller models and hard sequence labeling tasks. Prefix Tuning achieves comparable results with ~0.1% parameters on generation tasks. The prompt paradigm is fundamentally different from weight adaptation: it modifies the input rather than the computation. When it works, it suggests the model already has the capability; when it fails, it reveals genuine capability gaps.

Grade: PE=1 (scales with model size, conceptually different from weight methods)

---

26. Poly / X-LoRA -- Mixture of Adapters

Papers: Polytropon (Ponti et al. 2022), X-LoRA (Buehler 2024)

Hypothesis: Task adaptation isn't monolithic -- it's compositional. A shared library of "skill modules" (small adapters) can be recombined via learned routing to handle diverse tasks. The routing coefficients select which skills to activate for each input, forming a mixture-of-experts over adapter space.

# ── X-LoRA intervention ───────────
def xlora_forward(x, W, adapters, gating_net):
    # adapters: list of LoRA experts {(A_i, B_i)}
    # gating_net: maps hidden states to mixing weights
    gate = softmax(gating_net(x))                    # ∈ ℝ^{n_experts}
    ΔW = sum(gate[i] * (B_i @ A_i) for i in range(n_experts))
    return (W + ΔW) @ x

Evidence: X-LoRA achieves better composite performance than individual LoRAs by dynamically routing through appropriate expert adapters. Polytropon demonstrates cross-task transfer via shared skill libraries. The compositionality assumption is powerful but adds routing overhead and complexity. More suited to multi-task deployment than single-task fine-tuning.

Grade: PE=1 (compositional multi-task, routing overhead)

---

27. ETHER -- Efficient fine-THEning by oRthogonal transformation

Paper: Bini, Girrbach, Akata 2024 Code: Not in PEFT (standalone) Saved: docs/ether_orthogonal_steering.md See also: BiPDO (2024), repeng/representation engineering

Hypothesis: Fixed-strength orthogonal transformations are sufficient for behavioral steering. ETHER learns a single orthogonal rotation matrix applied to weight matrices, with the constraint that the transformation distance from identity is bounded. Unlike OFT which allows flexible-strength orthogonal updates, ETHER deliberately constrains the deviation, trading expressiveness for robustness and reversibility.

# ── ETHER intervention ────────────
def ether_forward(x, W, R):
    # R ∈ O(d): learned orthogonal matrix, close to identity
    return (R @ W) @ x                              # fixed-strength rotation

Evidence: Bini, Girrbach, Akata (same authors as DeLoRA). ETHER demonstrates that fixed-strength orthogonal transformations can achieve competitive task adaptation while preventing catastrophic forgetting. The bounded deviation is both a feature (robustness) and a limitation (ceiling on complex tasks). ETHER's constraints motivated DeLoRA's more flexible direction/strength decoupling.

Grade: PE=1 (robust fixed-strength rotations, foundational for DeLoRA)

Implications: ETHER represents the "minimal intervention" extreme of the orthogonal hypothesis. By showing that bounded rotations work for many tasks, it establishes a baseline: how much deviation from pretrained weights is actually needed? The answer appears to be "less than you think for behavioral steering, more than you think for complex task adaptation." This informed DeLoRA's key insight: decouple strength from direction, and let each evolve independently.

---

28. OFTv2 -- Input-Centric Orthogonal Fine-Tuning

Paper: 2025 (EMNLP 2025) Code: peft/tuners/oft (improved implementation) Saved: docs/oftv2_input_centric.md

Hypothesis: OFT's computational bottleneck (cubic complexity from weight-centric matrix-matrix multiplication) is an implementation artifact, not a fundamental limitation. By reformulating to input-centric matrix-vector multiplication and using Cayley-Neumann series for approximate matrix inversion, OFT can be made practical at scale (quadratic complexity, 10x faster, 3x less memory).

# ── OFTv2 intervention ────────────
def oftv2_forward(x, W, Q):
    # Instead of computing R = cayley(Q) then R @ W,
    # directly compute R @ (W @ x) via matrix-vector ops
    z = W @ x                                        # standard linear output
    Q_skew = skew_symmetric(Q)
    # Cayley-Neumann: (I - Q)^{-1} ≈ I + Q + Q^2 + ... (truncated)
    Rx = z + Q_skew @ z + Q_skew @ (Q_skew @ z)    # Neumann approximation
    return Rx

Evidence: Authors claim 10x faster training and 3x lower GPU memory than OFT without performance loss. Supports quantized foundation models and outperforms QLoRA in training stability. EMNLP 2025 acceptance. The key insight is purely computational: the same mathematical operation (orthogonal rotation) can be implemented much more efficiently in input-centric form.

Grade: PE=1 (same hypothesis as OFT, much more practical)

---

29. Bone -- Block-Affine Adaptation (Deprecated)

Paper: JL-er 2024 Code: Deprecated in PEFT, replaced by MiSS

Hypothesis: Weight updates have block-affine structure. Each block of the weight matrix undergoes an independent affine transformation (rotation + shift), combining HRA-style Householder reflections with per-block bias terms.

Evidence: Superseded by MiSS (entry 21), which generalizes the shard-sharing idea more cleanly. Listed for completeness.

Grade: (deprecated, see MiSS)

---

30. Trainable Tokens -- Vocabulary Extension

Code: peft/tuners/trainable_tokens

Hypothesis: Not a weight adaptation method. Extends the vocabulary embedding matrix with new learnable token embeddings (e.g., for reasoning/thinking tokens). Combinable with LoRA. Listed for completeness but outside the scope of the weight-adaptation hypothesis framework.

---