add flatness-seeking adapter family

TASK.md

@@ -2,9 +2,15 @@ TASK write a new file, from the old part.
 
 ## Status: DONE
 
+### Extension: flatness- and curvature-aware adapters
+- [x] Freeze primary texts for BAR, FMLoRA/EFMLoRA, Bi-LoRA, LoRA-MGPO, and DISAM under `docs/`
+- [x] Expand the catalog from 34 to 38 methods with pseudocode and calibrated evidence summaries
+- [x] Separate flatness seeking, explicit low-loss curvature, and function/context curvature in Theme 8
+- [x] UAT: [catalog](adapters_as_hypotheses.md) has 38 sequential entries and 36 active pseudocode blocks; [evidence map](adapters_vargdown.argdown) has seven new verbatim quote blocks within their linked line ranges; fresh-eyes review confirmed the corrected EFMLoRA math and curvature taxonomy
+
 ### Task 1: adapters_as_hypotheses.md
 - [x] Preamble with pragmatic interpretability framing
-- [x] 34 catalog entries; 32 active methods have pseudocode, while deprecated Bone and out-of-scope Trainable Tokens are marked as boundary entries
+- [x] 38 catalog entries; 36 active methods have pseudocode, while deprecated Bone and out-of-scope Trainable Tokens are marked as boundary entries
 - [x] All papers saved to docs/ (full size, no truncation)
 - [x] Sub-agent review completed, fixes applied
 

adapters_as_hypotheses.md

@@ -10,7 +10,7 @@ We fine tune transformers effeciently with low rank adapters - adding a new tran
 
 This is an underused source of *suggestive* evidence. Most interpretability *observes* (probing, SAEs); adapters *intervene*. If a structural constraint helps, the structure it encodes is load-bearing. The evidence is confounded by optimization dynamics (a method can win because its parameterization is optimizer-friendly, not because its structural hypothesis is correct), but the patterns are consistent enough to be worth taking seriously.
 
-I went through 34 PEFT methods in [HuggingFace PEFT](https://github.com/huggingface/peft) and the broader literature. For each one I extracted pseudocode for the intervention, stated the hypothesis it encodes, and weighed the evidence. Three claims emerged:
+I went through 38 PEFT methods in [HuggingFace PEFT](https://github.com/huggingface/peft) and the broader literature. For each one I extracted pseudocode for the intervention, stated the hypothesis it encodes, and weighed the evidence. Three claims emerged:
 
 1. **SVD basis outperforms random-initialized and standard bases.** Methods that initialize or constrain updates in the model's own singular-vector basis (PiSSA, SVFT, SSVD, CLOVER, PSOFT) consistently outperform random-basis alternatives at comparable budgets. SVD is linear and transformers are not, and almost no papers compare against other structured bases (ICA, Fisher eigenvectors, gradient covariance), so "SVD > random" is solid but "SVD is the right basis" is stronger than the data warrants.
 2. **Direction and strength decouple.** Methods that separate *which way* to move in weight space from *how far* (DoRA, DeLoRA, ROAD, AntiPaSTO) show better stability and sometimes better OOD transfer. An honest alternative: this could be an optimization benefit (giving Adam better-conditioned knobs) rather than a structural insight.
@@ -940,6 +940,109 @@ def flat_lora_loss(x, y, W, A, B, σ):
 
 ---
 
+## 35. BAR -- Balancedness-Aware Regularization
+
+**Paper:** [Li, Zhang, He 2024](https://arxiv.org/abs/2410.14802) (NeurIPS 2024)
+**Code:** [github.com/BingcongLi/BAR](https://github.com/BingcongLi/BAR)
+**Saved:** [docs/bar_balancedness_aware_regularization.md](docs/bar_balancedness_aware_regularization.md)
+
+**Hypothesis:** Much of SAM's benefit for factorized adapters comes from balancing the norms of LoRA's two factors, rather than directly minimizing Hessian curvature. BAR makes that implicit effect explicit. Its nBAR variant expands one factor and contracts the other according to their gradient norms, then applies the ordinary optimizer update.
+
+```py
+# Claude: nBAR training step
+def nbar_step(loss, W, A, B, α, η, optimizer):
+    g_A, g_B = ∇(loss(W + B @ A), (A, B))
+    s = +1 if norm(g_A) >= norm(g_B) else -1
+    A ← (1 + s * α * η) * A
+    B ← (1 - s * α * η) * B
+    A, B ← optimizer.step((A, B), (g_A, g_B))
+    return A, B
+```
+
+**Evidence:** The authors report few-shot OPT-1.3B averages of 78.5 for oBAR and 79.2 for nBAR, versus 77.6 for LoRA and 78.4 for LoRA-SAM. BAR runs at about 1.03--1.05x LoRA in those experiments, while LoRA-SAM takes 3.28--4.43x. RoBERTa and GPT-2 experiments also favor BAR in most reported comparisons. These are few-shot and ordinary held-out evaluations, not controlled OOD tests; balancedness is a proposed explanation for SAM's benefit rather than a curvature estimate.
+
+**Grade:** PE+BL+DE=3.5 (beats LoRA in few-shot tests and retains near-LoRA training cost)
+
+---
+
+## 36. FMLoRA / EFMLoRA -- Flat Minima LoRA
+
+**Paper:** [Deng et al. 2025](https://arxiv.org/abs/2508.00522) (AAAI 2026)
+**Saved:** [docs/fmlora_flat_minima_lora.md](docs/fmlora_flat_minima_lora.md)
+
+**Hypothesis:** A full-weight SAM perturbation can be reconstructed from LoRA-factor gradients and represented by perturbing only one LoRA factor. FMLoRA performs the two-step SAM update; EFMLoRA reuses an exponential moving average of previous perturbations to recover one-forward, one-backward training.
+
+```py
+# Claude: EFMLoRA training step
+def efmlora_step(loss, W, A, B, Ê_B, ρ, β, scale, optimizer):
+    g_A, g_B = ∇(loss(W + scale * (B + Ê_B) @ A), (A, B))
+    Ĝ_W = 0.5 / scale * (g_B @ pinv(A.T) + pinv((B + Ê_B).T) @ g_A)
+    E_W = ρ * Ĝ_W / norm(Ĝ_W)               # Claude: full-weight SAM direction
+    E_B = E_W @ pinv(A) / scale               # Claude: transfer into one LoRA factor
+    A, B ← optimizer.step((A, B), (g_A, g_B))
+    Ê_B ← (1 - β) * Ê_B + β * E_B
+    return A, B, Ê_B
+```
+
+**Evidence:** The authors report RoBERTa few-shot averages of 83.1 for FMLoRA and 82.3 for EFMLoRA, versus 80.0 for LoRA and 81.3 for LoRA-SAM. On full GLUE fine-tuning, EFMLoRA averages 89.4 versus 88.4 for LoRA and 88.9 for full fine-tuning. The paper also reports gains on GPT-2, CLIP few-shot classification, and Qwen-VL-Chat. Its "distribution shift" language mostly refers to few-shot transfer or ordinary downstream test sets; it does not use a leave-one-domain-out OOD protocol.
+
+**Grade:** PE+BL+BF+DE=5 (beats LoRA, slightly beats full FT on reported averages, and is strongest in few-shot settings)
+
+---
+
+## 37. Bi-LoRA -- Bi-directional Low-Rank Adaptation
+
+**Paper:** [Liu et al. 2025](https://arxiv.org/abs/2508.19564) (ICLR 2026)
+**Code:** [github.com/CrazyElements/Bi-LoRA](https://github.com/CrazyElements/Bi-LoRA)
+**Saved:** [docs/bi_lora_sharpness_aware.md](docs/bi_lora_sharpness_aware.md)
+
+**Hypothesis:** Task adaptation and sharpness exploration should use separate low-rank modules. A primary LoRA branch descends the task loss while an auxiliary branch ascends it inside a norm ball. Because the adversarial branch evolves independently, its perturbations need not collapse into the primary LoRA subspace. The auxiliary branch is discarded after training.
+
+```py
+# Claude: Bi-LoRA training step
+def bilora_step(loss, W, A_1, B_1, A_2, B_2, η_1, η_2, ρ):
+    W̃ = W + B_1 @ A_1 + B_2 @ A_2
+    G_W = ∇(loss(W̃), W̃)
+    B_1, A_1 ← B_1 - η_1 * G_W @ A_1.T, A_1 - η_1 * B_1.T @ G_W
+    B_2, A_2 ← B_2 + η_2 * G_W @ A_2.T, A_2 + η_2 * B_2.T @ G_W
+    A_2, B_2 ← project_product_norm(A_2, B_2, ρ)
+    return A_1, B_1, A_2, B_2
+
+def bilora_merge(W, A_1, B_1):
+    return W + B_1 @ A_1                       # Claude: discard adversarial branch
+```
+
+**Evidence:** The authors fine-tune on MetaMathQA, Code-Feedback, WizardLM, and Alpaca, then evaluate on separate benchmarks including GSM8K, HumanEval, MT-Bench, MMLU, DROP, and BBH. Against LoRA, reported Llama-2 gains are +2.11 on GSM8K, +2.45 on HumanEval, and +0.34 on MT-Bench. Bi-LoRA mostly improves on Flat-LoRA in the same table, while costing one gradient step per iteration. These cross-dataset evaluations are more informative than same-dataset validation, but they are not a controlled domain-generalization study.
+
+**Grade:** PE+BL+BF=3.5 (beats LoRA broadly and beats full FT on some reported tasks)
+
+---
+
+## 38. LoRA-MGPO -- Momentum-Guided Perturbation Optimization
+
+**Paper:** [Chang et al. 2025](https://aclanthology.org/2025.findings-emnlp.34/) (Findings EMNLP 2025)
+**Code:** [github.com/llm172/LoRA-MGPO](https://github.com/llm172/LoRA-MGPO)
+**Saved:** [docs/lora_mgpo_momentum_perturbation.md](docs/lora_mgpo_momentum_perturbation.md)
+
+**Hypothesis:** Optimizer momentum supplies a cheap, stable approximation to SAM's adversarial direction. Perturb LoRA parameters along the previous first-moment vector, normalize the radius using an EMA of gradient norms, and compute only one gradient at the perturbed point.
+
+```py
+# Claude: LoRA-MGPO training step
+def mgpo_step(loss, W, θ, m, ḡ, ρ, β, optimizer):
+    ε_θ = ρ * m / (norm(m) * ḡ)                # Claude: θ = (A, B)
+    Ã, B̃ = unpack(θ + ε_θ)
+    g = ∇(loss(W + B̃ @ Ã), θ + ε_θ)
+    θ, m ← optimizer.step(θ, m, g)
+    ḡ ← β * ḡ + (1 - β) * norm(g)
+    return θ, m, ḡ
+```
+
+**Evidence:** The authors report a T5 GLUE average of 88.81 versus 82.08 for LoRA and 87.91 for full fine-tuning. On Llama-2, MGPO is the strongest reported PEFT method on MT-Bench, GSM8K, and HumanEval at several ranks, though it remains below full fine-tuning on GSM8K and HumanEval. The large GLUE margin is concentrated in CoLA and MRPC and comes from the authors' own setup. The experiments support optimization stability and conventional generalization, not an explicit curvature measurement or controlled OOD transfer.
+
+**Grade:** PE+BL+BF=3.5 (beats LoRA and slightly beats full FT on the reported GLUE average)
+
+---
+
 ## Scorecard
 
 Sorted by evidence strength (max 8). See [scoring legend](#evidence-scoring) above.
@@ -947,11 +1050,15 @@ Sorted by evidence strength (max 8). See [scoring legend](#evidence-scoring) abo
 |    # | Method        | Score | Breakdown   | Theme            |
 | ---: | ------------- | ----: | ----------- | ---------------- |
 |    6 | PiSSA         |   5.0 | PE+BL+BF+DE | SVD basis        |
+|   36 | FMLoRA       |   5.0 | PE+BL+BF+DE | flatness         |
 |    4 | DoRA          |   4.5 | PE+BL+BF+WA | dir/strength     |
 |   11 | AntiPaSTO*    |   4.5 | PE+DE+OOD   | SVD+rotation     |
 |   34 | Flat-LoRA     |   4.0 | PE+BL+OOD   | flatness         |
 |   13 | BOFT          |   4.0 | PE+BF+DE    | orthogonal       |
 |    5 | DeLoRA        |   3.5 | PE+BL+DE    | dir/strength     |
+|   35 | BAR           |   3.5 | PE+BL+DE    | flatness         |
+|   37 | Bi-LoRA       |   3.5 | PE+BL+BF    | flatness         |
+|   38 | LoRA-MGPO     |   3.5 | PE+BL+BF    | flatness         |
 |    8 | SSVD          |   3.5 | PE+BL+DE    | SVD basis        |
 |   31 | CLOVER        |   3.5 | PE+BL+BF    | SVD+architecture |
 |   32 | PSOFT         |   3.5 | PE+BL+DE    | SVD+orthogonal   |
@@ -985,7 +1092,7 @@ Sorted by evidence strength (max 8). See [scoring legend](#evidence-scoring) abo
 
 ## Themes: What the Evidence Tells Us
 
-Looking across all 34 methods, the successful adapters share a recipe: choose coordinates that align with pretrained structure, constrain updates to preserve that structure, and control update strength explicitly.
+Looking across all 38 methods, the successful adapters share a recipe: choose coordinates that align with pretrained structure, constrain updates to preserve that structure, and control update strength explicitly.
 
 The pattern is strong enough to organize the literature by theme rather than by year.
 
@@ -1000,13 +1107,13 @@ The *direction-versus-strength* split follows naturally. DoRA, DeLoRA, ROAD, and
 
 The *rank* debate is secondary once basis is accounted for. Full-rank updates help on harder tasks (RandLoRA, C3A), but a good low-rank subspace beats a poorly chosen full-rank update (PiSSA, SVFT). "Which subspace" matters more than "how many free directions".
 
-*Curvature* is probably best treated as space-specific rather than as one adapter principle. Flat-LoRA's authors report that smoothing task loss around the merged weights improves LoRA, including under corruption and instruction-following shifts. CrispEdit instead constrains curvature of a capability loss, while TRAM regularizes predictive-distribution changes in function space. Counterexamples with flat non-generalizing minima make weight-space flatness weak evidence about semantic depth. Context curvature of a steering effect remains a plausible but untested diagnostic. The quote-anchored argument and counterevidence are in [the Vargdown evidence map](adapters_vargdown.argdown).
+*Flatness and curvature* form one related family, but the differentiated variable matters. Flat-LoRA, FMLoRA, Bi-LoRA, and MGPO seek finite-neighborhood flatness through random, adversarial, or momentum-guided perturbations. BAR instead isolates factor balancedness as a proposed implicit effect of SAM. This flatness-seeking family does not diagonalize a Hessian. The explicit low-loss-curvature hypothesis constrains updates to low-eigenvalue directions of a named loss; CrispEdit is the closest adjacent method. Function and context curvature differentiate predictions or steering effects with respect to inputs, activations, or context paths; TRAM supports this axis, while context curvature of a steering effect remains a plausible but untested diagnostic. Counterexamples with flat non-generalizing minima make weight-space flatness weak evidence about semantic depth. The quote-anchored argument and counterevidence are in [the Vargdown evidence map](adapters_vargdown.argdown).
 
 Finally, methods that respect *functional architecture* are promising but early. CLOVER's joint Q-K and V-O treatment outperforms per-matrix updates in reported setups, and ReFT shows targeted activation interventions can be far more parameter-efficient than weight updates. Both suggest that treating transformer layers as computation graphs -- not bags of independent matrices -- is a productive direction.
 
 ### What I now believe (and didn't before)
 
-Before writing this catalog, I thought of adapters mainly as engineering trade-offs: LoRA is cheap, full FT is better, pick your budget. After reading 34 adapter papers carefully, I updated on three things:
+Before writing this catalog, I thought of adapters mainly as engineering trade-offs: LoRA is cheap, full FT is better, pick your budget. After reading 38 adapter papers carefully, I updated on three things:
 
 1. **The SVD basis outperforms random-initialized and standard bases.** The consistent advantage of SVD-initialized methods (PiSSA > LoRA, SVFT recovering 96% of full FT with 0.006% params, CLOVER's joint SVD beating per-matrix LoRA) is hard to explain as coincidence. The model's singular vectors appear to encode meaningful computational directions that the optimizer discovers faster when given them as a starting point. But SVD is linear and transformers are not; the advantage could be a warm-start effect; and almost no papers compare against other structured bases. "SVD > random" is solid, "SVD is the right basis" remains open. Strength of evidence: moderate (multiple independent groups, multiple modalities, but all within-paper comparisons).
 

adapters_vargdown.argdown

@@ -1,6 +1,6 @@
 ===
 title: Adapters as Representational Hypotheses -- Which Geometric Priors About Transformer Internals Hold Under Intervention?
-author: Compiled from 34 PEFT methods plus adjacent evidence (2021--2026)
+author: Compiled from 38 PEFT methods plus adjacent evidence (2021--2026)
 model:
     mode: strict
 ===
@@ -426,25 +426,29 @@ model:
   +> [Natural Manifold]
 
 
-// ══════════════════════════════════════════════════════════════════════
-// THEME 8: CURVATURE IS SPACE-SPECIFIC
-// Adapter: Flat-LoRA. Adjacent evidence: CrispEdit, SGD subspaces,
-// sharpness counterexamples, TRAM, and steering generalization.
-// ══════════════════════════════════════════════════════════════════════
+// Claude: ══════════════════════════════════════════════════════════════
+// Claude: THEME 8: FLATNESS AND CURVATURE ARE SPACE-SPECIFIC
+// Claude: Flatness-seeking family: Flat-LoRA, FMLoRA, Bi-LoRA, MGPO;
+// Claude: BAR is the SAM-balancedness surrogate.
+// Claude: Adjacent evidence: CrispEdit, DISAM, SGD subspaces, sharpness
+// Claude: counterexamples, TRAM, and steering generalization.
+// Claude: ══════════════════════════════════════════════════════════════
 
 # Curvature
 
-[Curvature Is Space-Specific]: Curvature only becomes an intervention
-  hypothesis after naming the scalar landscape and the space or path in
-  which it is measured. Weight-, capability-, and function-space curvature
-  make different predictions about robustness and generalization.
-  + <Full Weight Flatness>
+[Curvature Is Space-Specific]: Flatness and curvature are related, but the
+  differentiated variable must be named. Perturbation-based flatness seeking,
+  explicit Hessian or Gauss-Newton eigenspace constraints, and function or
+  context curvature make different predictions about generalization.
+  + <Flatness Seeking Adapters>
+  + <Explicit Low Loss Curvature>
   + <Curvature Direction Ambiguity>
+  + <Flatness Under Domain Shift>
   + <Flatness OOD Limits>
   + <Function Space Curvature>
 
 
-<Full Weight Flatness>
+<Flatness Seeking Adapters>
 
 (1) [Flat-LoRA Smooths Merged Weights]: Flat-LoRA trains low-rank factors
     under random perturbations of the merged weight matrix so the solution
@@ -453,7 +457,49 @@ model:
     [evidence](docs/flat_lora_full_parameter_flatness.md#L1-L35)
     > Despite recent progress in improving LoRA’s performance, the relationship between the LoRA optimization space and the full parameter space is often overlooked. **A solution that appears flat in the loss landscape of the LoRA space may still exhibit sharp directions in the full parameter space, potentially compromising generalization. We introduce Flat-LoRA, which aims to identify a low-rank adaptation situated in a flat region of the full parameter space.** Instead of adopting the well-established sharpness-aware minimization approach, which incurs significant computation and memory overheads, we employ a Bayesian expectation loss objective to preserve training efficiency. Further, we design a refined random perturbation generation strategy for improved performance and carefully manage memory overhead using random seeds.
     {reason: "ICML 2025; authors' abstract for their own method; the main perturbation scheme covers adapted linear matrices, while all-layer perturbation is reported separately in the appendix; no independent replication found", credence: 0.78}
-(2) [CrispEdit Protects Capability Loss]: CrispEdit projects model-editing
+(2) [BAR Makes SAM Balancedness Explicit]: BAR replaces SAM's extra
+    adversarial step with a factor-norm regularizer derived from the claimed
+    implicit balancedness dynamics of SAM. #observation
+    [Li, Zhang, He 2024](https://arxiv.org/abs/2410.14802)
+    [evidence](docs/bar_balancedness_aware_regularization.md#L15-L29)
+    > Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization of SAM for scale-invariant problems involving two groups of variables. **Instead of focusing on commonly used sharpness, this work introduces a concept termed balancedness, defined as the difference between the squared norm of two variables.** This allows us to depict richer global behaviors of SAM. In particular, our theoretical and empirical findings reveal that i) SAM promotes balancedness; and ii) the regularization on balancedness is data-responsive – outliers have stronger impact. The latter coincides with empirical observations that SAM outperforms SGD in the presence of outliers. Leveraging the implicit regularization, we develop a resource-efficient SAM variant, balancedness-aware regularization (BAR), tailored for scale-invariant problems such as finetuning language models with LoRA.
+    {reason: "NeurIPS 2024; authors' abstract and theoretical framing; BAR is reused as a baseline by Flat-LoRA, FMLoRA, and Bi-LoRA, but its tests are few-shot or ordinary held-out rather than controlled OOD", credence: 0.82}
+(3) [FMLoRA Transfers Full-Space Perturbations]: FMLoRA reconstructs a
+    full-weight SAM direction from LoRA gradients and transfers it into one
+    factor; EFMLoRA reuses an EMA perturbation for near-LoRA cost. #observation
+    [Deng et al. 2025](https://arxiv.org/abs/2508.00522)
+    [evidence](docs/fmlora_flat_minima_lora.md#L8-L28)
+    > Little research explores the correlation between the expressive ability and generalization ability of the low-rank adaptation (LoRA). Sharpness-Aware Minimization (SAM) improves model generalization for both Convolutional Neural Networks (CNNs) and Transformers by encouraging convergence to locally flat minima. However, the connection between sharpness and generalization has not been fully explored for LoRA due to the lack of tools to either empirically seek flat minima or develop theoretical methods. **In this work, we propose Flat Minima LoRA (FMLoRA) and its efficient version i.e., EFMLoRA, to seek flat minima for LoRA. Concretely, we theoretically demonstrate that perturbations in the full parameter space can be transferred to the low-rank subspace.** This approach eliminates the potential interference introduced by perturbations across multiple matrices in the low-rank subspace. Our extensive experiments on large language models and vision-language models demonstrate that EFMLoRA achieves optimize efficiency comparable to that of LoRA while simultaneously attaining comparable or even better performance. For example, on the GLUE dataset with RoBERTa-large, EFMLoRA outperforms LoRA and full fine-tuning by 1.0% and 0.5% on average, respectively. On vision-language models e.g., Qwen-VL-Chat, there are performance improvements of 1.5% and 1.0% on the SQA and VizWiz datasets, respectively. These empirical results also verify that the generalization of LoRA is closely related to sharpness, which is omitted by previous methods.
+    {reason: "AAAI 2026; authors' abstract and own benchmark comparisons; reported gains cover several architectures, but the claimed distribution shift is mostly few-shot or ordinary downstream evaluation", credence: 0.72}
+(4) [Bi-LoRA Separates Descent and Ascent]: Bi-LoRA trains a primary LoRA
+    branch for task descent and an auxiliary low-rank branch for adversarial
+    ascent, then discards the auxiliary branch. #observation
+    [Liu et al. 2025](https://arxiv.org/abs/2508.19564)
+    [evidence](docs/bi_lora_sharpness_aware.md#L16-L20)
+    > Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large pre-trained models. Yet LoRA can face generalization challenges. One promising way to improve the generalization is Sharpness-Aware Minimization (SAM), which has proven effective for small-scale training scenarios. **In this paper, we propose Bi-directional Lo w-R ank A daptation (Bi-LoRA), which introduces an auxiliary adversarial LoRA module. This design explicitly decouples sharpness optimization, handled by the auxiliary module, from task adaptation, performed by the primary module.** Such a separation yields two key benefits. First, it transforms SAM’s sequential computation of adversarial perturbation and gradient descent into a parallel form, which roughly halves the time and conquers the main obstacle of applying SAM in LoRA. Second, it provides perturbations from the auxiliary module that do not collapse into the restricted optimization subspace of the primary module, enabling broader sharpness exploration and flatter minima. Bi-LoRA simultaneously achieves both efficiency and effectiveness within a single framework, as validated by extensive experiments across diverse architectures and tasks.
+    {reason: "ICLR 2026; authors' method section; same-paper comparisons mostly favor Bi-LoRA over LoRA, LoRA-SAM, and Flat-LoRA on cross-dataset LLM tests, without a controlled domain-generalization protocol", credence: 0.76}
+(5) [MGPO Reuses Optimizer Momentum]: LoRA-MGPO perturbs trainable LoRA
+    factors along the optimizer's previous first moment and normalizes the
+    radius with an EMA of gradient norms. #observation
+    [Chang et al. 2025](https://aclanthology.org/2025.findings-emnlp.34/)
+    [evidence](docs/lora_mgpo_momentum_perturbation.md#L28-L34)
+    > Parameter-efficient fine-tuning (PEFT), partic-ularly Low-Rank Adaptation (LoRA), adapts large language models (LLMs) by training only a small fraction of parameters. However, as the rank of the low-rank matrices used for adap-tation increases, LoRA often exhibits an un-stable "double descent" phenomenon, charac-terized by transient divergence in the training loss, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima. **To address this, we introduce LoRA-MGPO , a framework that incorporates Momentum-Guided Pertur-bation Optimization (MGPO). MGPO stabi-lizes training dynamics by mitigating the dou-ble descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer’s state, thus avoiding dual gra-dient computations.** Additionally, an adaptive normalization scheme scales the magnitude of perturbations based on an exponential mov-ing average (EMA) of gradient norms, further enhancing stability. While EMA controls the magnitude of the perturbations, MGPO guides their direction, ensuring a more stable opti-mization trajectory.
+    {reason: "Findings EMNLP 2025; authors' method section and released code; improvements are conventional NLU and cross-dataset NLG results, while the link to flat minima is indirect", credence: 0.64}
+----
+(6) [Flatness Seeking Is Indirect Curvature Control]: Flat-LoRA, FMLoRA,
+    Bi-LoRA, and MGPO seek low loss in a finite weight neighborhood; BAR
+    instead isolates factor balancedness as a proposed implicit effect of
+    SAM. Near a stationary point the four neighborhood objectives are
+    sensitive to the largest Hessian eigenvalues, but none of the five
+    computes a Hessian eigenspace or restricts updates to low-eigenvalue
+    directions.
+    {reason: "local Taylor expansion links neighborhood sharpness to curvature; BAR implements a derived surrogate, and the other four implement perturbation objectives rather than spectral projection", inference: 0.90}
+  +> [Curvature Is Space-Specific]
+
+
+<Explicit Low Loss Curvature>
+
+(1) [CrispEdit Protects Capability Loss]: CrispEdit projects model-editing
     updates into the low-curvature subspace of a separate capability loss,
     estimated with Gauss-Newton curvature and K-FAC. #observation
     [Ikram et al. 2026](https://arxiv.org/abs/2602.15823)
@@ -461,11 +507,11 @@ model:
     > We present CrispEdit, a scalable and principled second-order editing algorithm that treats capability preservation as an explicit constraint, unifying and generalizing several existing editing approaches. **CrispEdit formulates editing as constrained optimization and enforces the constraint by projecting edit updates onto the low-curvature subspace of the capability-loss landscape.** At the crux of CrispEdit is expressing capability constraint via Bregman divergence, whose quadratic form yields the Gauss–Newton Hessian exactly and even when the base model is not trained to convergence. We make this second-order procedure efficient at the LLM scale using Kronecker-factored approximate curvature (K-FAC) and a novel matrix-free projector that exploits Kronecker structure to avoid constructing massive projection matrices.
     {reason: "May 2026 preprint; authors' abstract for a model-editing method rather than an adapter; capability is evaluated on a designated reference set", credence: 0.66}
 ----
-(3) [Low Curvature Protects the Chosen Loss]: Flat-LoRA seeks task-loss
-    insensitivity around merged weights; CrispEdit seeks capability-loss
-    insensitivity along edit directions. Neither makes low curvature a
+(2) [Low Curvature Protects the Chosen Loss]: CrispEdit's spectral projector
+    protects a designated capability loss. This is the closest prior art to
+    an explicit low-loss-curvature adapter, but it does not make curvature a
     detector of the target behavior's semantic depth.
-    {reason: "the methods constrain different scalar landscapes; the distinction follows from their objectives rather than their reported benchmark gains", inference: 0.82}
+    {reason: "CrispEdit names the protected scalar loss and reference set; projecting an update away from its stiff directions says what is preserved, not why the edited behavior generalizes", inference: 0.84}
   +> [Curvature Is Space-Specific]
 
 
@@ -494,6 +540,33 @@ model:
   +> [Curvature Is Space-Specific]
 
 
+<Flatness Under Domain Shift>
+
+(1) [Plain SAM Can Fail Under Domain Shift]: DISAM's authors report that
+    ordinary SAM can underperform ERM when source domains converge at
+    different rates. #observation
+    [Zhang et al. 2024](https://arxiv.org/abs/2405.18861)
+    [evidence](docs/disam_domain_shift_sharpness.md#L46-L58)
+    > Nonetheless, these methods cannot solve generalizability scenarios that involve training data of multiple domains with domain shifts like Domain Generalization (DG) (Ben-David et al., 2010; Li et al., 2017). **In this study, we observed that sometimes SAM even has a detrimental impact in situations where there exist domain shifts across multiple domains as shown in Figure 1(1(a)).** While a few studies have incorporated SAM-based methods in domain generalization tasks (Wang et al., 2023b; Foret et al., 2021), they cannot ensure consistent improvements in generalizability during domain shifts due to their reliance on the i.i.d assumption. Upon a thorough analysis of the behavior of SAM under domain shifts, we discovered that the degradation of the training process caused by SAM from the disparity in convergence degree among different domains as shown in Figure 1(1(a)). Given the inconsistency in the degree and direction of convergence among different domains during training (Arjovsky et al., 2019; Krueger et al., 2021), the straightforward application of SAM for perturbations may not only disrupt convergence but also generate perturbation directions that are not adequately coherent to the geometric characteristics of the entire loss landscape.
+    {reason: "ICLR 2024; authors' diagnosis from DomainBed experiments; this directly limits the inference from ordinary SAM or flat-adapter gains to OOD transfer", credence: 0.83}
+(2) [DISAM Uses a Genuine OOD Protocol]: DISAM calibrates the SAM
+    perturbation using source-domain loss variance and evaluates on domains
+    excluded from training. #observation
+    [Zhang et al. 2024](https://arxiv.org/abs/2405.18861)
+    [evidence](docs/disam_domain_shift_sharpness.md#L1611-L1627)
+    > We evaluate DISAM on five datasets PACS (Li et al., 2017), VLCS (Fang et al., 2013) OfficeHome (Venkateswara et al., 2017), TerraIncognita (Beery et al., 2018) (abbreviated as TerraInc), and DomainNet (Peng et al., 2019), following the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021). For fair comparison, we adhere to the training and evaluation protocol outlined in DomainBed. Evaluation. **The standard leave-one-domain-out strategy is used in evaluation. Specially, the unseen domain is used to evaluate the out-of-domain generalization, and the validation sets of source domains are used to measure the in-domain generalization, while the others are used for training.** Final accuracy is averaged across all settings, and the performance is the averaging over three trials with distinct random seeds. Detailed statistics for each case of all datasets are provided in Appendix C.
+    [evidence](docs/disam_domain_shift_sharpness.md#L2186-L2192)
+    > We propose incorporating our domain-inspired adaptive adjustment into three SAM-based methods: SAM (Foret et al., 2021), GSAM (Zhuang et al., 2022), and SAGM (Wang et al., 2023b) on five datasets of DomainBed with ResNet50 backbone. Table 1 shows that our Domain-Inspired SAM can mitigate issues arising from SAM’s training under domain shifts, by comparing averaged in-domain and out-of-domain performance of leading SAM methods, with and without DISAM. In-domain results show domain-inspired perturbations enhance convergence, especially on the TerraInc dataset with substantial domain gaps. **In Out-of-domain results, DISAM consistently improves generalization, with average improvements of 1.9% for SAM, 1.7% for GSAM, and 1.9% for SAGM.** Notably, SAM performs well when the performance gap between in-domain and out-of-domain is small but worse than ERM on datasets like TerraInc with large gaps, which proves our analysis of SAM’s shortcomings under domain shifts. This shows SAM’s inconsistent convergence for large domain shifts, which DISAM addresses by incorporating domain-inspired adaptive adjustments based on domain-level convergence degree. Incorporating CORAL constraints, a recognized effective traditional DG method on DomainBed improves SAGM with DISAM and sets new state-of-the-art results on all settings.
+    {reason: "ICLR 2024; standard leave-one-domain-out DomainBed protocol over five datasets, three trials, plus CLIP prompt-tuning experiments; author-reported results but materially stronger OOD evidence than downstream test accuracy", credence: 0.86}
+----
+(3) [OOD Flatness Needs Shift Information]: DISAM is not a LoRA adapter,
+    but it is strong evidence against treating generic weight-space flatness
+    as domain-general by default. Its gains require domain labels and a
+    domain-loss variance term when constructing the perturbation.
+    {reason: "plain SAM sometimes loses to ERM while domain-calibrated SAM improves leave-one-domain-out accuracy; the added domain information, not flatness alone, distinguishes the methods", inference: 0.82}
+  +> [Curvature Is Space-Specific]
+
+
 <Flatness OOD Limits>
 
 (1) [Flatness Need Not Generalize]: Flat non-generalizing minimizers exist,

docs/fmlora_flat_minima_lora.md

@@ -0,0 +1,598 @@
+Title: Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond
+
+URL Source: https://arxiv.org/html/2508.00522
+
+Published Time: Tue, 16 Dec 2025 02:15:18 GMT
+
+Markdown Content:
+Jiaxin Deng 1\equalcontrib, Qingcheng Zhu 2\equalcontrib, Junbiao Pang 1, Linlin Yang 3, Zhongqian Fu 4, Baochang Zhang 5
+
+###### Abstract
+
+Little research explores the correlation between the expressive ability and generalization ability of the low-rank adaptation (LoRA). Sharpness-Aware Minimization (SAM) improves model generalization for both Convolutional Neural Networks (CNNs) and Transformers by encouraging convergence to locally flat minima. However, the connection between sharpness and generalization has not been fully explored for LoRA due to the lack of tools to either empirically seek flat minima or develop theoretical methods. In this work, we propose Flat Minima LoRA (FMLoRA) and its efficient version i.e., EFMLoRA, to seek flat minima for LoRA. Concretely, we theoretically demonstrate that perturbations in the full parameter space can be transferred to the low-rank subspace. This approach eliminates the potential interference introduced by perturbations across multiple matrices in the low-rank subspace. Our extensive experiments on large language models and vision-language models demonstrate that EFMLoRA achieves optimize efficiency comparable to that of LoRA while simultaneously attaining comparable or even better performance. For example, on the GLUE dataset with RoBERTa-large, EFMLoRA outperforms LoRA and full fine-tuning by 1.0% and 0.5% on average, respectively. On vision-language models e.g., Qwen-VL-Chat, there are performance improvements of 1.5% and 1.0% on the SQA and VizWiz datasets, respectively. These empirical results also verify that the generalization of LoRA is closely related to sharpness, which is omitted by previous methods.
+
+## Introduction
+
+Parameter-Efficient Fine-Tuning (PEFT) methods only update a small subset of parameters, e.g., adapters (hu2022lora) or prompt weights (li2021prefix) for Large language models (LLMs) with substantially lower memory and computational costs. Specifically, Low-Rank Adaptation (LoRA) (hu2022lora) stands out for achieving performance comparable to full fine-tuning (FT) while being considerably more efficient.
+
+![Image 1: Refer to caption](https://arxiv.org/html/2508.00522v3/x1.png)
+
+Figure 1: Comparison of Methods: LoRA, FMLoRA, and EFMLoRA.
+
+Many works have been proposed to enhance the performance of LoRA by introducing more dedicated budgets for rank allocation (zhang2023adaptive), decomposing optimization for direction and magnitude updates (liu2024dora), or designing better initialization strategies for LoRA parameters (meng2024pissa), etc. These studies demonstrate the significant potential to improve LoRA performance. However, most existing approaches fail to effectively address bias inheritance, where LLMs may propagate and amplify their inherent biases, significantly impacting model performance and robustness on downstream tasks (li2025understanding). Therefore, a natural question is: how to model and understand the generalization of LoRA for various LLMs and beyond, e.g., vision-language models?
+
+It is widely believed that a flatter loss landscape can lead to better generalization performance(hochreiter1994simplifying)(hochreiter1997flat). For instance, Foret et al. proposed Sharpness-Aware Minimization (SAM)(foret-2020-SAM-ICLR), which seeks parameter regions where the training loss remains uniformly flat. SAM and its variants have demonstrated State-Of-The-Art (SOTA) performances across various applications, such as classification(kwon-2021-asam-ICML), transfer learning(zhuang-2022-GSAM-ICLR), domain generalization(dong2024implicit) and federated learning(FedGAMMA).
+
+To the best of our knowledge, compared to theoretical analysis, e.g.,(neyshabur2017exploring), empirically connecting sharpness and generalization ability of LoRA is a practical approach, e.g.,(andriushchenko2023modern). For the second line of research, a naive approach is to combine SAM with LoRA. However, if perturbations in SAM are applied simultaneously to two low-rank subspaces of LoRA, they may change the maximum loss within the neighborhood of LoRA’s full parameter space(dinh2017sharp); besides, SAM incurs a computational cost twice that of Stochastic Gradient Descent (SGD)(deng2024effective). The key question in the second line of research is how to efficiently find flat minima in LoRA, aiming to better understand the connection between sharpness and generalization.
+
+In this paper, we propose a novel PEFT method, FMLoRA, that promotes convergence toward flatter minima. Specifically, we theoretically uncover that perturbations in the full parameter space can be equivalently re-parameterized as perturbations within the low-rank space. In addition, we propose EFMLoRA to accelerate FMLoRA by an Exponential Moving Average (EMA) strategy. We validate that EFMLoRA improves generalization performance on downstream tasks while maintaining computational efficiency comparable to that of LoRA. Fig.[1](https://arxiv.org/html/2508.00522v3#Sx1.F1 "Figure 1 ‣ Introduction ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") compares three methods: LoRA, FMLoRA, and EFMLoRA. We conducted comprehensive experiments on diverse tasks (fine-tuning, few-shot learning) and various model types (RoBERTa (liu2019roberta), GPT-2 (radford2019language), CLIP (zanella2024low), Qwen-VL-Chat (Bai2023QwenVLAV)) and scales. We find that EFMLoRA achieves model accuracy very close to, or even surpass both full fine-tuning and LoRA across many tasks. Our main contribution can be summarized as follows:
+
+*   •We propose FMLoRA, a novel PEFT training method that integrates SAM into the LoRA framework. Furthermore, EFMLoRA provides an efficient tool for empirically understanding the connection between sharpness and generalization in LLMs and beyond. We empirically show that reducing sharpness is highly correlated with improved generalization in PEFT tasks, which has been rarely explored in PEFT studies before.
+*   •We conduct comprehensive experiments on LLMs (e.g., RoBERTa, GPT-2) and vision-language models (e.g., CLIP, Qwen-VL-Chat) across various tasks including fine-tuning and few-shot learning. Results show that EFMLoRA achieves optimize efficiency comparable to that of LoRA while simultaneously attaining comparable or even better performance.
+
+## Related Works
+
+### Low-rank Adaption
+
+Hu et al. proposed LoRA (hu2022lora) as a PEFT method that introduced low-rank adapters into each layer of a pre-trained model. Recent advancements in LoRA can be broadly categorized into two directions: 1) advanced architectures and 2) optimization methods. In the first research line, for example, LoraHub (huang2023lorahub) trained multiple adapters and strategically combined them based on the domain during inference. LoRA-FA (zhang2023lora) chose to freeze the projection-down weight of \mathbf{A} and update the projection-up weight of \mathbf{B} in each LoRA layer. DoRA (liu2024dora) improved LoRA by incorporating a learnable magnitude vector to re-scale the normalized product of low-rank matrices. HydraLoRA (tian2024hydralora) extended the LoRA framework with an asymmetric architecture that shared a common \mathbf{A} matrix for efficiency while dynamically assigning samples to multiple \mathbf{B} matrices via a MoE mechanism. In the second line, for example, LoRA+ (hayou2024lora+) applied different learning rates to the two low-rank matrices. Additionally, Galore (zhao2024galore) employed SVD to compress the gradients and its first and second momentum of full training into a low-rank space, thereby reducing the memory footprint during pre-training and fine-tuning. Recently, Li et al. (li2024flat) proposed combining SAM with LoRA for better generalization, but they used random perturbation. Our method belongs to the second research line. Different from (li2024flat), our method transfers the perturbation from the full parameter space to a single low-rank parameter space without changing the maximum perturbed loss, avoiding misalignment with SAM’s training behavior.
+
+### Sharpness and Generalization Ability
+
+Research on the relationship between sharpness and generalization could be traced back to (hochreiter1997flat). Following the observation by (keskar-2016-large_batch-ICLR) that larger batch sizes tended to increase sharpness and generalization error. (jastrzkebski2017three) extended this by finding a correlation between the sharpness and the ratio of learning rate to batch size. (dinh-2017-sharp_minima-ICML) showed that one can easily construct networks with good generalization but with arbitrary large sharpness by reparameterization. (jiang-2019-fantastic-ICLR) performed a large-scale empirical study on various generalization measures and showed that sharpness-based measures have the highest correlation with generalization. Theoretical understandings on the generalization error using sharpness-related measures were provided in (neyshabur2017exploring), (wanggeneralization). Collectively, these studies justified the goal of seeking flatter minima to improve generalization. However, to the best of our knowledge, the correlation between sharpness and generalization for LoRA has barely been discussed due to the lack of theoretical understanding or efficient tools for empirical analysis. Our method provides an efficient tool for empirical analysis in this domain.
+
+### Recap of SAM
+
+Foret et al.(foret-2020-SAM-ICLR) proposed the SAM to enhance model generalization as follows:
+
+\displaystyle\mathop{\min}\limits_{\mathbf{w}}[(\mathop{\max}\limits_{||\bm{\varepsilon}||\leq\rho}L(\mathbf{w}+\bm{\varepsilon})-L(\mathbf{w}))+L(\mathbf{w})+\lambda||\mathbf{w}||_{2}^{2}],(1)
+
+where \mathbf{w} represents the weights of the network, \bm{\varepsilon} represents the perturbation of weights \mathbf{w} in a Euclidean ball with the radius \rho(\rho>0), L(\cdot) is the loss function, and \lambda||\mathbf{w}||_{2}^{2} is a standard L2 regularization term.
+
+SAM utilizes Taylor expansion to search for the maximum perturbed loss (\mathop{\max}\limits_{||\bm{\varepsilon}||\leq\rho}L(\mathbf{w}+\bm{\varepsilon})) in local parameter space as follows:
+
+\displaystyle\mathop{\arg\max}\limits_{||\bm{\varepsilon}||\leq\rho}\;L(\mathbf{w}+\bm{\varepsilon})\approx\mathop{\arg\max}\limits_{||\bm{\varepsilon}||\leq\rho}\;{\bm{\varepsilon}^{\top}}{\nabla_{\mathbf{w}}}L(\mathbf{w}).(2)
+
+By solving Eq.([2](https://arxiv.org/html/2508.00522v3#Sx2.E2 "Equation 2 ‣ Recap of SAM ‣ Related Works ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), SAM obtains the perturbation as follows:
+
+\displaystyle\hat{\bm{\varepsilon}}=\rho{\nabla_{\mathbf{w}}}L(\mathbf{w})/||{\nabla_{\mathbf{w}}}L(\mathbf{w})||.(3)
+
+Substituting the perturbation \hat{\bm{\varepsilon}} back into Eq.([1](https://arxiv.org/html/2508.00522v3#Sx2.E1 "Equation 1 ‣ Recap of SAM ‣ Related Works ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), we then have:
+
+\displaystyle{\nabla_{\mathbf{w}}}\mathop{\max}\limits_{||\bm{\varepsilon}||\leq\rho}L(\mathbf{w}+\bm{\varepsilon})\approx{\nabla_{\mathbf{w}}}L({\mathbf{w}}+\hat{\bm{\varepsilon}}({\mathbf{w}}))(4)
+\displaystyle={\nabla_{\mathbf{w}}}L({\mathbf{w}}){|_{{\mathbf{w}}+\hat{\bm{\varepsilon}}({\mathbf{w}})}}+\frac{{d\hat{\bm{\varepsilon}}({\mathbf{w}})}}{{d{\mathbf{w}}}}{\nabla_{\mathbf{w}}}L({\mathbf{w}}){|_{{\mathbf{w}}+\hat{\bm{\varepsilon}}({\mathbf{w}})}}.
+
+By dropping the second-order terms in Eq.([4](https://arxiv.org/html/2508.00522v3#Sx2.E4 "Equation 4 ‣ Recap of SAM ‣ Related Works ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), SAM calculates the gradient at \mathbf{w}+\bm{\hat{\varepsilon}} as follows:
+
+\displaystyle{\nabla_{\mathbf{w}}}\mathop{\max}\limits_{||\bm{\varepsilon}||\leq\rho}L(\mathbf{w}+\bm{\varepsilon})\approx{\nabla_{\mathbf{w}}}L(\mathbf{w}){|_{\mathbf{w}+\bm{\hat{\varepsilon}}}}.(5)
+
+Finally, SAM uses the gradients from Eq.([5](https://arxiv.org/html/2508.00522v3#Sx2.E5 "Equation 5 ‣ Recap of SAM ‣ Related Works ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")) for optimization.
+
+### SAM Variants
+
+Recently, SAM variants could be broadly categorized into three groups: 1) studies on the perturbation radius \rho in SAM, 2) studies that speed up the optimization process of SAM, and 3) redefinitions of sharpness in SAM. For the first direction, Kwon et al.(kwon-2021-asam-ICML) proposed Adaptive SAM (ASAM), which adapted the perturbation radius in a scale-aware manner, allowing SAM to be effectively applied to scale-invariant neural networks. For the second group, Kim et al.(kim2023exploring) introduced a multi-step ascent approach to improve SAM. Li et al.(li2024friendly) introduced Friendly SAM (F-SAM), which improved generalization by removing the detrimental influence of the full gradient component and instead utilizing batch-specific gradients to guide optimization more effectively. For the third group, Zhuang et al.(zhuang-2022-GSAM-ICLR) pointed out that SAM did not always favor flat minima. Consequently, they proposed GSAM, which minimized the surrogate gap and the perturbed loss to better encourage flatness. Zhang et al. introduced the first-order flatness(zhang-2023-gradient-CVPR), which assessed the maximal gradient norm within a perturbation radius. Consequently, they proposed GAM which explicitly seeks minima characterized by uniformly small curvature.
+
+## Method
+
+### SAM on LoRA
+
+LoRA achieves parameter efficiency by modeling the low-rank decomposed weight(li2022low). Specifically, the weight change for each layer \mathbf{W}_{0}\in\mathbb{R}^{n\times m} is represented as \Delta\mathbf{W}=s\mathbf{B}\mathbf{A}, where s is a scaling factor, \mathbf{B}\in\mathbb{R}^{n\times r}, \mathbf{A}\in\mathbb{R}^{r\times m}, with rank r\ll\min(n,m). Given an input \mathbf{x}, the forward is as follows:
+
+\displaystyle\mathbf{y}=\mathbf{W}_{0}\mathbf{x}+\Delta\mathbf{W}\mathbf{x}=(\mathbf{W}_{0}+s\mathbf{B}\mathbf{A})\mathbf{x},(6)
+
+where matrix \mathbf{A} is typically initialized by the Kaiming’s method(he2015delving), \mathbf{B} is set to zeros. \mathbf{W}_{0} remains unchanged during fine-tuning, while \mathbf{B} and \mathbf{A} are trained. During inference, \Delta\mathbf{W} is merged into \mathbf{W_{0}}.
+
+If SAM is naively combined with LoRA, the optimization loss can be rewritten as follows:
+
+\displaystyle\min_{\mathbf{A},\mathbf{B}}~~\mathop{\max}\limits_{\scriptstyle||{{\bf{E}}^{\bf{A}}}|{|_{F}}\leq\rho,\hfill\atop\scriptstyle||{{\bf{E}}^{\bf{B}}}|{|_{F}}\leq\rho\hfill}L({\mathbf{W_{0}}}+{s}(\mathbf{B}+{\mathbf{E}^{\mathbf{B}}})(\mathbf{A}+{\mathbf{E}^{\mathbf{A}}})),(7)
+
+where \mathbf{E}^{\mathbf{B}}\in\mathbb{R}^{n\times r} and \mathbf{E}^{\mathbf{A}}\in\mathbb{R}^{r\times m} represent the perturbations applied to the parameters \mathbf{B} and \mathbf{A}, respectively, and \rho is the radius of perturbations. There are two key challenges:
+
+*   •Two separate perturbations in two low-rank subspaces interfere with each other, leading to an inconsistency between the maximum loss obtained when perturbing in the low-rank subspaces and the maximum loss obtained when perturbing in the full parameter space.
+*   •SAM requires computing gradients twice per iteration, resulting in approximately twice the computational cost compared to LoRA.
+
+### FMLoRA
+
+To deal with the first challenge, we propose to re-parameterize the perturbation from the full parameter space to a single low-rank parameter space. Concretely, the loss in the full parameter space can be formulated as follows:
+
+\displaystyle\min_{\mathbf{A},\mathbf{B}}~~\max_{\|\mathbf{E}^{\mathbf{W}}\|_{F}\leq\rho}~~L(\mathbf{W_{0}}+s\mathbf{B}\mathbf{A}+\mathbf{E}^{\mathbf{W}}).(8)
+
+To solve the minimax problem in Eq.([8](https://arxiv.org/html/2508.00522v3#Sx3.E8 "Equation 8 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), it is necessary to first find optimal \hat{\mathbf{E}}^{\mathbf{W}}\in\mathbb{R}^{n\times m}. Analogous to SAM, we approximate the optimal perturbation \hat{\mathbf{E}}^{\mathbf{W}} to maximize L(\mathbf{W}+\mathbf{E}^{\mathbf{W}}) where \mathbf{W}=\mathbf{W_{0}}+s\mathbf{B}\mathbf{A} as follows:
+
+\displaystyle\hat{\bm{\varepsilon}}^{\mathbf{w}}=\rho\text{sign}(\mathbf{g}^{\mathbf{w}})\frac{\mathbf{g}^{\mathbf{w}}}{||\mathbf{g}^{\mathbf{w}}||},(9)
+
+where \mathbf{g}^{\mathbf{w}}=\text{Vector}(\nabla L_{\mathbf{W}}(\mathbf{W})) and \hat{\bm{\varepsilon}}^{\mathbf{w}}=\text{Vector}(\hat{\mathbf{E}}^{\mathbf{W}}), in which the \text{Vector}(\cdot) function represents a vectorized operation. However, the solution for \hat{\mathbf{E}}^{\mathbf{W}} explicitly depends on the gradient of the matrix \mathbf{W}. That is, the form of solution in Eq.([9](https://arxiv.org/html/2508.00522v3#Sx3.E9 "Equation 9 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")) is undesirable since \nabla L_{\mathbf{W}}(\mathbf{W}) is unknown during LoRA optimization.
+
+In this paper, we propose to approximate the unknown gradient \nabla L_{\mathbf{W}}(\mathbf{W}) using standard LoRA gradients, which can be computed in two ways:
+
+\displaystyle(1)\quad\nabla L_{\mathbf{W}}(\mathbf{W})=\frac{1}{s}\nabla L_{\mathbf{B}}(\mathbf{W_{0}}+s\mathbf{BA})(\mathbf{A}^{\top})^{+},(10)
+\displaystyle(2)\quad\nabla L_{\mathbf{W}}(\mathbf{W})=\frac{1}{s}(\mathbf{B}^{\top})^{+}\nabla L_{\mathbf{A}}(\mathbf{W_{0}}+s\mathbf{BA}),(11)
+
+where (\mathbf{A}^{\top})^{+} and (\mathbf{B}^{\top})^{+} represent the pseudo-inverse of \mathbf{A}^{\top} and \mathbf{B}^{\top}, respectively. The accuracy of the pseudo-inverse depends on the condition number of matrix. A smaller condition number leads to a more accurate pseudo-inverse. Matrices with lower condition numbers are better suited for stable representation. In LoRA, we found that the condition number is typically low, around 3.
+
+To obtain a more accurate estimate of the gradient of the full weights, we combine the above two approaches to compute \nabla L_{\mathbf{W}}(\mathbf{W}) as follows:
+
+\displaystyle\overline{\nabla{L}_{\mathbf{W}}(\mathbf{W})}\displaystyle=0.5*(\frac{1}{s}\nabla L_{\mathbf{B}}(\mathbf{W_{0}}+s\mathbf{BA})(\mathbf{A}^{\top})^{+}
+\displaystyle+\frac{1}{s}(\mathbf{B}^{\top})^{+}\nabla L_{\mathbf{A}}(\mathbf{W_{0}}+s\mathbf{BA})).(12)
+
+Let {\bar{\mathbf{g}}^{\mathbf{W}}}=\text{Vector}(\overline{\nabla{L}_{\mathbf{W}}(\mathbf{W})}). Then the perturbation in Eq.([9](https://arxiv.org/html/2508.00522v3#Sx3.E9 "Equation 9 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")) could be rewritten as follows:
+
+\displaystyle\bar{\mathbf{E}}^{\mathbf{W}}=\text{Matrix}(\rho\text{sign}({\bar{\mathbf{g}}^{\mathbf{W}}})\frac{{\bar{\mathbf{g}}^{\mathbf{W}}}}{||{\bar{\mathbf{g}}^{\mathbf{W}}}||}),(13)
+
+where \text{Matrix}(\cdot) denotes the operation that converts a vector into a matrix. We transfer the perturbation from the full parameter space to a single low-rank parameter space without changing the maximum loss in the local region of the parameters. We apply no perturbation to matrix \mathbf{A}, i.e., {\mathbf{E}^{\mathbf{A}}}=\mathbf{0}, and ensure that the loss under perturbations in the low-rank subspace in Eq.([7](https://arxiv.org/html/2508.00522v3#Sx3.E7 "Equation 7 ‣ SAM on LoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")) matches the inner maximum loss in Eq.([8](https://arxiv.org/html/2508.00522v3#Sx3.E8 "Equation 8 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), as follows:
+
+\displaystyle L({\mathbf{W_{0}}}\displaystyle+{s}(\mathbf{B}+{\mathbf{E}^{\mathbf{B}}})\mathbf{A})(14)
+\displaystyle=\max_{\|\mathbf{E}^{\mathbf{W}}\|_{F}\leq\rho}L(\mathbf{W_{0}}+s\mathbf{B}\mathbf{A}+\mathbf{E}^{\mathbf{W}}).
+
+Substituting \bar{\mathbf{E}}^{\mathbf{W}} into Eq.([14](https://arxiv.org/html/2508.00522v3#Sx3.E14 "Equation 14 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), we obtain:
+
+\displaystyle{\mathbf{E}}^{\mathbf{B}}\approx\frac{1}{s}\bar{\mathbf{E}}^{\mathbf{W}}\mathbf{A}^{+},(15)
+
+where \mathbf{A}^{+} is the pseudo-inverse of \mathbf{A}. An alternative approach is to transfer the perturbation to matrix \mathbf{A}. Following the observations from HydraLoRA(tian2024hydralora), matrix \mathbf{A} shows high parameter similarity across heads, likely due to initialization, making it capture domain-common features, while matrix \mathbf{B} remains distinct and domain-specific. Since different tasks require different perturbations, we adopt the approach of transferring the perturbation to the matrix \mathbf{B}, as expressed in Eq.([14](https://arxiv.org/html/2508.00522v3#Sx3.E14 "Equation 14 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")). The detailed derivation of Eq.([10](https://arxiv.org/html/2508.00522v3#Sx3.E10 "Equation 10 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")) and the pseudo-algorithm for FMLoRA are provided in the supplementary file.
+
+#### Balancedness of FMLoRA.
+
+Balancedness is well-appreciated in domains such as matrix factorization/sensing (ge2017no)(du2018algorithmic). It is also observed that balanced neural networks are easier to optimize relative to unbalanced ones (neyshabur2015path). Recently, Balancedness B_{t}:=\frac{1}{2}(||\mathbf{x}_{t}||^{2}-||\mathbf{y}_{t}||^{2}) (where \mathbf{x}_{t} and \mathbf{y}_{t} are variables) turns out to be an intriguing alternative to sharpness on the scale-invariant problem (li2024implicit).
+
+To investigate the balancedness of our proposed method, we express the update process of FMLoRA analogously to Eq.(4) in (li2024implicit) as follows:
+
+\displaystyle{\tilde{\mathbf{x}}_{t}}={{\mathbf{x}}_{t}}+\rho\frac{1}{s}\frac{{{\mathbf{G}_{t}}}}{{\left\|{{\mathbf{G}_{t}}}\right\|}}{\mathbf{y}_{t}}^{+}\displaystyle,\quad{\tilde{\mathbf{y}}_{t}}={{\mathbf{y}}_{t}},(16)
+\displaystyle{\mathbf{g}_{{\tilde{\mathbf{x}}_{t}}}}={{\tilde{\mathbf{G}}}_{t}}\tilde{\mathbf{y}}_{t}\displaystyle,\quad{\mathbf{g}_{{\tilde{\mathbf{y}}_{t}}}}={{\tilde{\mathbf{G}}}_{t}}^{\top}\tilde{\mathbf{x}}_{t},
+\displaystyle{{\mathbf{x}}_{t+1}}={{\mathbf{x}}_{t}}-\eta{\mathbf{g}_{{\tilde{\mathbf{x}}_{t}}}}\displaystyle,\quad{{\mathbf{y}}_{t+1}}={{\mathbf{y}}_{t}}-\eta{\mathbf{g}_{{\tilde{\mathbf{y}}_{t}}}},
+
+where {\mathbf{x}}_{t}=\text{Vector}(\mathbf{B}_{t}), {\mathbf{y}}_{t}=\text{Vector}(\mathbf{A}_{t}), {\mathbf{G}_{t}}=\nabla L({\mathbf{x}}_{t}{\mathbf{y}}_{t}^{\top}) is the gradient of the full parameter space at the original parameter point, {\tilde{\mathbf{G}}_{t}}=\nabla L(\tilde{\mathbf{x}}_{t}\tilde{\mathbf{y}}_{t}^{\top}) is the gradient of the full parameter space at the perturbed parameter point, and \mathbf{y}_{t}^{+} is the pseudo inverse of \mathbf{y}_{t}.
+
+###### Theorem 1.
+
+Let B_{t}:=\frac{1}{2}(||\mathbf{x}_{t}||^{2}-||\mathbf{y}_{t}||^{2}). For the learning rate \eta\Rightarrow 0, the limiting flow of FMLoRA guarantees that:
+
+\displaystyle\left|{\frac{1}{2}\frac{{d({{\left\|{{\mathbf{x}_{t}}}\right\|}^{2}}-{{\left\|{{\mathbf{y}_{t}}}\right\|}^{2}})}}{{dt}}}\right|\leq\left|{\rho\frac{1}{s}\frac{1}{{\left\|{\mathbf{y}_{t}}\right\|}}\left\|{{\mathbf{g}_{{{{\rm{\tilde{\mathbf{x}}}}}_{t}}}}}\right\|}\right|.(17)
+
+Theorem 1 indicates that the balancedness of FMLoRA is influenced by the perturbation range \rho, the norm of the gradient at the perturbed point, the \ell_{2}-norm of \mathbf{y}_{t}, and the scale constraint of LoRA. To ensure that the balancedness of FMLoRA gradually decreases during training, we reduce \rho progressively. In addition, the norm of the gradient with respect to \mathbf{y}_{t} at the perturbed point also decreases due to the weight decay. The \ell_{2}-norm of \mathbf{y}_{t} is bounded within a certain range, these factors collectively contribute to the reduction in the balancedness of FMLoRA.
+
+![Image 2: Refer to caption](https://arxiv.org/html/2508.00522v3/x2.png)
+
+Figure 2: Parameter update process for EFMLoRA.
+
+### Efficient FMLoRA
+
+The optimization processes of FMLoRA also require two gradient computations per iteration. To enhance optimization efficiency, we propose Efficient FMLoRA (EFMLoRA), which estimates the subsequent perturbation {\mathbf{E}}^{\mathbf{B}} in Eq.([15](https://arxiv.org/html/2508.00522v3#Sx3.E15 "Equation 15 ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")) by maintaining an Exponential Moving Average (EMA) of previous perturbations as follows:
+
+\displaystyle{{\hat{\mathbf{E}}}^{\mathbf{B}}_{t}}=(1-\beta){{\hat{\mathbf{E}}}^{\mathbf{B}}_{t-1}}+\beta{\mathbf{E}^{\mathbf{B}}_{t}},(18)
+
+where \beta\in(0,1) is the momentum coefficient that determines the update rate of the exponential moving average. {\mathbf{E}^{\mathbf{B}}_{t}} is the perturbation on matrix \mathbf{B}_{t} at t-th iteration, {{\hat{\mathbf{E}}}^{\mathbf{B}}_{t}} is the EMA perturbation at t-th iteration. Fig.[2](https://arxiv.org/html/2508.00522v3#Sx3.F2 "Figure 2 ‣ Balancedness of FMLoRA. ‣ FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") illustrates the parameter update process of EFMLoRA: (1) Calculate the gradient at the perturbed point (\mathbf{W}_{0}, \mathbf{B}_{t-1}+\hat{\mathbf{E}}^{\mathbf{B}}_{t-1}, \mathbf{A}_{t-1}). (2) Calculate the perturbation {\mathbf{E}}^{\mathbf{B}}_{t}=\frac{1}{s}\bar{\mathbf{E}}^{\mathbf{W}}\mathbf{A}^{+}_{t-1}. (3) Return to the original parameter point (\mathbf{W}_{0},\mathbf{B}_{t-1},\mathbf{A}_{t-1}). (4) Update the parameters to (\mathbf{W}_{0},\mathbf{B}_{t},\mathbf{A}_{t}). (5) Calculate the EMA perturbation by Eq.([18](https://arxiv.org/html/2508.00522v3#Sx3.E18 "Equation 18 ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")) and update the parameters to the next perturbed point (\mathbf{W}_{0},\mathbf{B}_{t}+\hat{\mathbf{E}}^{\mathbf{B}}_{t},\mathbf{A}_{t}). During this optimization process, each optimization step requires only a single forward and backward. The algorithmic pseudocode is provided in the supplementary file.
+
+Table 1: Experiments on few-shot RoBERTa (355M). Results marked with * are taken from (li2024implicit).
+
+To theoretically analyze the error of EFMLoRA, some necessary assumptions are listed below, all of which are common and standard when analyzing SAM optimization(du-2022-ESAM-ICLR)(zhuang-2022-GSAM-ICLR).
+
+###### Assumption 1.
+
+(Smooth) L(\mathbf{w}) is \tau-Lipschitz smooth in \mathbf{w}, i.e., \left\|{\nabla L(\mathbf{w})-\nabla L(\mathbf{v})}\right\|\leq\tau\left\|{\mathbf{w}-\mathbf{v}}\right\|.
+
+###### Assumption 2.
+
+(Bounded gradients). By the assumption that an upper bound exists on the gradient of each mini-batch. There exists G>0 for each mini-batch such that \mathbb{E}\left[{\left\|{\nabla L(\mathbf{w})}\right\|}\right]\leq G.
+
+###### Assumption 3.
+
+(Bounded variance of stochastic gradients). Given the training set \mathbf{D} and a mini-batch \mathbf{B}\in\mathbf{D}. There exists \sigma\geq 0, the variance of stochastic gradient L_{\mathbf{B}}(\mathbf{w}) is bounded by \mathbb{E}\left[{{{\left\|{\nabla{L_{\mathbf{B}}}(\mathbf{w})-\nabla{L_{\mathbf{D}}}(\mathbf{w})}\right\|}^{2}}}\right]\leq\sigma^{2}.
+
+###### Assumption 4.
+
+(Convex) We assume that the loss function f:\mathbb{R}^{n}\rightarrow\mathbb{R} is convex and twice differentiable over an open domain. That is, for all x,y\in\text{dom}(f), it satisfies: f(y)\geq f(x)+\nabla f(x)^{\top}(y-x).
+
+This convexity assumption is reasonable in the fine-tuning stage, as the model is typically close to a local minimum and the loss landscape is approximately convex in a local neighborhood (jang2024lora).
+
+###### Theorem 2.
+
+[EMA perturbation approximate perturbation of SAM due to the convex of the loss landscape] Assume that during fine-tuning, the solution is already close to a local minimum and the local loss function is convex. Let the model weights at i-th iteration be \mathbf{w}_{t}. Under Assumptions [1](https://arxiv.org/html/2508.00522v3#Thmassumption1 "Assumption 1. ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"), [2](https://arxiv.org/html/2508.00522v3#Thmassumption2 "Assumption 2. ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"), and [3](https://arxiv.org/html/2508.00522v3#Thmassumption3 "Assumption 3. ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"), let {\rho_{t}}=\frac{{{\rho_{0}}}}{{\sqrt{t}}}, the error between the sharpness calculated using the EMA perturbation (S^{\text{EMA}}) and that calculated using the original SAM perturbation (S^{\text{SAM}}) is bounded as follows:
+
+\displaystyle|\underbrace{\left[{L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t}})-L({\mathbf{w}_{t}})}\right]}_{S^{\text{EMA}}}-\underbrace{\left[{L({\mathbf{w}_{t}}+{\bm{\tilde{\varepsilon}}_{t}})-L({\mathbf{w}_{t}})}\right]}_{S^{\text{SAM}}}|(19)
+\displaystyle\leq\left({\left({1+{{(1-\beta)}^{t-1}}}\right)\tau{\rho_{0}}+G+{\sigma^{2}}}\right)
+\displaystyle\quad\quad\cdot\left({\left({1+{{(1-\beta)}^{t-1}}}\right){\rho_{0}}+\frac{{{\rho_{0}}}}{{\sqrt{t}}}}\right).
+
+Theorem [2](https://arxiv.org/html/2508.00522v3#Thmtheorem2 "Theorem 2. ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") demonstrates that as t increases, the difference between S^{\text{EMA}} and S^{\text{SAM}} gradually decreases. The perturbation estimated by the EMA can effectively approximate the original SAM perturbation.
+
+Table 2: Experiments on finetuning RoBERTa (355M). Results marked with \dagger are taken from (hu2022lora), and those with * are taken from (li2024implicit).
+
+Table 3: GPT-2 medium (M) and large (L) with different adaptation methods on the E2E NLG Challenge. Results marked with \dagger are taken from (hu2022lora).
+
+### Memory and Time Complexity
+
+LoRA reduces the number of trainable parameters by decomposing weight updates as \Delta\mathbf{W}\approx\mathbf{B}\mathbf{A}, where \mathbf{B}\in\mathbb{R}^{n\times r} and \mathbf{A}\in\mathbb{R}^{r\times m} with r\ll\min(n,m). Both FMLoRA and EFMLoRA retain this parameter efficiency:
+
+\displaystyle\text{P}_{\text{LoRA}}\displaystyle=\text{P}_{\text{FMLoRA}}=\text{P}_{\text{EFMLoRA}}(20)
+\displaystyle=O(nr+rm)\ll O(nm).
+
+However, FMLoRA and EFMLoRA introduce additional memory overhead. Specifically, FMLoRA temporarily stores the original values of \mathbf{B} and \mathbf{A}, as well as the gradients of \mathbf{A}. The memory usage of FMLoRA is calibrated as follows:
+
+\displaystyle\text{M}_{\text{FMLoRA}}=\text{M}_{\text{LoRA}}+O(5\times(nr+rm)),(21)
+
+where \text{M}_{\text{LoRA}} indicates the memory required by LoRA. The memory of EFMLoRA needs to maintain the EMA perturbation on \mathbf{B} as follows:
+
+\displaystyle\text{M}_{\text{EFMLoRA}}=\text{M}_{\text{LoRA}}+O(2\times(nr+rm)).(22)
+
+Notably, modern optimizers like AdamW already require O(2\times(nr+rm)) memory for momentum and second-moment statistics when applied to LoRA.
+
+For time complexity, suppose that the time complexity of optimizing the model with LoRA is O(T), which mainly includes the time for forward and backward. Theoretically, the time complexity of FMLoRA is approximately as follows:
+
+\displaystyle\text{T}_{\text{FMLoRA}}\approx O(2T)=2\times\text{T}_{\text{LoRA}}.(23)
+
+In contrast, the time complexity of EFMLoRA can be approximated as follows:
+
+\displaystyle\text{T}_{\text{EFMLoRA}}\approx O(T)=\text{T}_{\text{LoRA}}.(24)
+
+We implement QR decomposition by Householder transformations, with time complexity of O(r^{2}n) for an r\times n matrix, e.g., r is rank, n is the input dimension in LORA.
+
+## Experiments and Discussions
+
+The best and second-best results are highlighted in bold and underline, respectively. Additional experimental details are provided in the supplementary file.
+
+### Experiments on Large Language Models
+
+Few-shot with RoBERTa-large. We first consider few-shot learning with EFMLoRA. Following the setup of (li2024implicit), we adopt RoBERTa-large—a 355M-parameter language model—as the backbone. The results in Table [1](https://arxiv.org/html/2508.00522v3#Sx3.T1 "Table 1 ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") show that FMLoRA outperforms all other methods with the highest average score (83.1), particularly excelling on SST-2, SNLI, and MNLI. EFMLoRA follows closely with an average score of 82.3. It consistently surpasses baseline LoRA (+2.3), LoRA-SAM (+1.0), and both BAR variants. These results highlight its superior generalization ability under distribution shift and limited supervision. We conjecture that the performance gap between SAM and EFMLoRA comes from EFMLoRA eliminating the mutual interference between perturbations in the two low-rank subspaces.
+
+Fine-tuning with RoBERTa-large. We apply EFMLoRA to finetune RoBERTa-large. Our implementation follows (hu2022lora), using the same hyperparameters as those in its GitHub repository. The results can be found in Table [2](https://arxiv.org/html/2508.00522v3#Sx3.T2 "Table 2 ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"). we observe that EFMLoRA achieves the highest scores on all datasets, and achieves the highest accuracy on average over these datasets. Specifically, on average over these datasets, EFMLoRA surpasses standard LoRA with a margin of 1.0. Additionally, EFMLoRA even achieve better performance than full fine-tuning on some datasets. This superior performance may be attributed to overfitting in full fine-tuning, where optimizing all model parameters can lead to overfitting on the training data, thus reducing the model’s generalization to the test set. This effect is particularly pronounced on small datasets, such as MRPC, which contains only 3.7k training data.
+
+Fine-tuning with GPT-2. Having shown that FMLoRA is effective for NLU tasks, we now explore whether EFMLoRA can improve LoRA in NLG models like GPT-2 Medium and Large (radford2019language). To enable a direct comparison, we adopt the experimental setup of (li2021prefix) with minimal deviation. Table[3](https://arxiv.org/html/2508.00522v3#Sx3.T3 "Table 3 ‣ Efficient FMLoRA ‣ Method ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") demonstrates the effectiveness of EFMLoRA on the E2E NLG Challenge (novikova2017e2e) with GPT-2 Medium and Large models. Compared with existing PEFT methods such as Adapter and LoRA, EFMLoRA consistently achieves superior performance across all metrics. Notably, it achieves this improvement without increasing the number of trainable parameters, maintaining the same efficiency as standard LoRA.
+
+### Experiments on Vision Language Models
+
+Few-shot with CLIP. Recent advances in few-shot adaptation of Vision-Language Models (VLMs) have significantly enhanced their generalization. CLIP-LoRA (zanella2024low) explores the application of LoRA in this few-shot VLM setting. In our work, we also apply FMLoRA and EFMLoRA to VLMs to evaluate their effectiveness. For a fair comparison, our experimental setup follows that of CLIP-LoRA. We consider five datasets for fine-grained classification of satellite imagery (EuroSAT (helber2019eurosat), Ox-fordPets (parkhi2012cats), Flower102 (nilsback2008automated), Caltech101 (fei2004learning), DTD (cimpoi2014describing)). These datasets offer a thorough benchmarking framework for evaluating few-shot visual classification tasks. Table[4](https://arxiv.org/html/2508.00522v3#Sx4.T4 "Table 4 ‣ Experiments on Vision Language Models ‣ Experiments and Discussions ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") demonstrates that FMLoRA and EFMLoRA outperformed Adapter and LoRA in most settings. In the low-data regimes (1-shot and 4-shot), EFMLoRA shows clear advantages. These results highlight the effectiveness of EFMLoRA in improving generalization in few-shot adaptation of vision-language models.
+
+Table 4: Detailed results for five datasets with CLIP-Adapter, CLIP-LoRA and EFMLoRA.
+
+Fine-tuning with Qwen-VL-Chat. Qwen-VL-Chat (Bai2023QwenVLAV) is a multimodal conversational large language model capable of understanding both images and text. We apply EFMLoRA to fine-tune Qwen-VL-Chat, following the same experimental setup as in (zhou2024empirical). Table [5](https://arxiv.org/html/2508.00522v3#Sx4.T5 "Table 5 ‣ Experiments on Vision Language Models ‣ Experiments and Discussions ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") presents the results on the ScienceQA (lu2022learn) and VizWiz (gurari2018vizwiz) datasets. The results in Table [5](https://arxiv.org/html/2508.00522v3#Sx4.T5 "Table 5 ‣ Experiments on Vision Language Models ‣ Experiments and Discussions ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") demonstrate that the perturbation size \rho significantly influences the performance of EFMLoRA when fine-tuning Qwen-VL-Chat. By tuning \rho, EFMLoRA adapts to different tasks, enabling improved generalization—achieving higher accuracy than LoRA. Specifically, a larger \rho (e.g., \rho=0.2) yields the best accuracy on ScienceQA, while a smaller \rho (e.g., \rho=0.05) performs better on VizWiz. This suggests that different tasks benefit from different levels of perturbation. Therefore, selecting an appropriate \rho based on the task characteristics is crucial for achieving optimal fine-tuning performance on multimodal large language models.
+
+Table 5: EFMLoRA Fine-Tuning Results on Qwen-VL-Chat with different \rho.
+
+Table 6: Runtime (Hour) and memory (GB) of LoRA, FMLoRA and EFMLoRA on fine-tuning GPT-2 Medium/Large.
+
+### Runtime and Memory Consumption
+
+The results in Table[6](https://arxiv.org/html/2508.00522v3#Sx4.T6 "Table 6 ‣ Experiments on Vision Language Models ‣ Experiments and Discussions ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") confirm the theoretical time complexity analysis. As expected, FMLoRA has approximately double the runtime of LoRA (2.1× on both GPT-2 Medium and Large), consistent with its theoretical complexity of O(2T) due to two forward and backward passes for sharpness optimization. In contrast, EFMLoRA operates with near-LoRA efficiency, requiring only 1.1× and 1.2× more time on GPT-2 Medium and Large, respectively. This supports the theoretical claim that EFMLoRA maintains a time complexity close to O(T) while benefiting from sharpness-aware optimization. In addition, EFMLoRA maintains a memory usage almost identical to that of LoRA, with only negligible increases (less than 0.4 GB across both model scales). These results demonstrate that EFMLoRA achieves near-LoRA efficiency in both memory and runtime.
+
+### Conclusion
+
+In this work, we propose FMLoRA, a novel PEFT method that integrates sharpness-aware optimization into the LoRA framework to promote convergence toward flatter minima. We theoretically demonstrate that perturbations in the full parameter space can be equivalently represented within the low-rank subspace. To improve computational efficiency, we introduce EFMLoRA, which leverages an exponential moving average to approximate perturbations, significantly reducing runtime overhead while maintaining effectiveness. Extensive experiments across various large language and vision-language models demonstrate that EFMLoRA achieves comparable or even superior generalization performance to full fine-tuning and LoRA. Our results emphasize the importance of reducing sharpness to improve generalization in PEFT methods, offering valuable insights and practical tools for future research on the link between sharpness and generalization in LLMs and beyond.
+
+## A. Proofs
+
+### A.1 Proof of Eq.(10) and Eq.(11)
+
+###### Proof.
+
+we propose to approximate the unknown gradient \nabla L_{\mathbf{W}}(\mathbf{W}) using standard LoRA gradients, which can be computed in two ways:
+
+\displaystyle(1)\nabla L_{\mathbf{B}}\displaystyle(\mathbf{W_{0}}+s\mathbf{BA})=s\nabla L_{\mathbf{W}}(\mathbf{W})\mathbf{A}^{\top}
+\displaystyle\Rightarrow\quad\nabla L_{\mathbf{W}}(\mathbf{W})=\frac{1}{s}\nabla L_{\mathbf{B}}(\mathbf{W_{0}}+s\mathbf{BA})(\mathbf{A}^{\top})^{+},(25)
+\displaystyle(2)\nabla L_{\mathbf{A}}\displaystyle(\mathbf{W_{0}}+s\mathbf{BA})=s\mathbf{B}^{\top}\nabla L_{\mathbf{W}}(\mathbf{W})
+\displaystyle\Rightarrow\quad\nabla L_{\mathbf{W}}(\mathbf{W})=\frac{1}{s}(\mathbf{B}^{\top})^{+}\nabla L_{\mathbf{A}}(\mathbf{W_{0}}+s\mathbf{BA}),(26)
+
+∎
+
+### A.2 Proof of Theorem 1
+
+###### Proof.
+
+The update process of the FMLoRA is as follows:
+
+\displaystyle{\tilde{\mathbf{x}}_{t}}={{\mathbf{x}}_{t}}+\rho\frac{1}{s}\frac{{{\mathbf{G}_{t}}}}{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}{\mathbf{y}_{t}}^{+}\displaystyle,\quad{\tilde{\mathbf{y}}_{t}}={{\mathbf{y}}_{t}}(27)
+\displaystyle{\mathbf{g}_{{\tilde{\mathbf{x}}_{t}}}}={{\tilde{\mathbf{G}}}_{t}}\tilde{\mathbf{y}}_{t}\displaystyle,\quad{\mathbf{g}_{{\tilde{\mathbf{y}}_{t}}}}={{\tilde{\mathbf{G}}}_{t}}^{\top}\tilde{\mathbf{x}}_{t}
+\displaystyle{{\mathbf{x}}_{t+1}}={{\mathbf{x}}_{t}}-\eta{\mathbf{g}_{{\tilde{\mathbf{x}}_{t}}}}\displaystyle,\quad{{\mathbf{y}}_{t+1}}={{\mathbf{y}}_{t}}-\eta{\mathbf{g}_{{\tilde{\mathbf{y}}_{t}}}}
+
+where {\mathbf{x}}_{t}=\text{Vector}(\mathbf{B}_{t}) is the vectorized form of matrix \mathbf{B}_{t}, {\mathbf{y}}_{t} is the vectorized form of matrix \mathbf{A}_{t}, {\mathbf{G}_{t}}=\nabla L({\mathbf{x}}_{t}{\mathbf{y}}_{t}^{\top}) is the gradient of the full parameter space at the original point during gradient descent, {\tilde{\mathbf{G}}_{t}}=\nabla L(\tilde{\mathbf{x}}_{t}\tilde{\mathbf{y}}_{t}^{\top}) is the gradient of the full parameter space at the perturbed point, and \mathbf{y}^{+} is the pseudo inverse of \mathbf{y}. Let balancedness B_{t}:=\frac{1}{2}(||\mathbf{x}_{t}||^{2}-||\mathbf{y}_{t}||^{2}). Then, we have that:
+
+\displaystyle\frac{1}{2}\frac{{d({{\left\|{{\mathbf{x}_{t}}}\right\|}^{2}}-{{\left\|{{\mathbf{y}_{t}}}\right\|}^{2}})}}{{dt}}(28)
+\displaystyle=\frac{1}{2}\frac{{d({{\left\|{{\mathbf{x}_{t}}}\right\|}^{2}})}}{{dt}}-\frac{1}{2}\frac{{d({{\left\|{{\mathbf{y}_{t}}}\right\|}^{2}})}}{{dt}}
+\displaystyle={\mathbf{x}_{t}}^{\top}\frac{{d{\mathbf{x}_{t}}}}{{dt}}-{{\mathbf{y}}_{t}}^{\top}\frac{{d{\mathbf{y}_{t}}}}{{dt}}
+\displaystyle=-{\mathbf{x}_{t}}^{\top}({\mathbf{\tilde{G}}_{t}}{\mathbf{y}_{t}})+({\mathbf{y}_{t}}^{\top}({\mathbf{\tilde{G}}_{t}}^{\top}({{\mathbf{x}}_{t}}+\rho\frac{1}{s}\frac{{{\mathbf{G}_{{t}}}}}{{{{\left\|{\mathbf{G}_{t}}\right\|}_{F}}}}\mathbf{y}_{t}^{+})))
+\displaystyle=-{\mathbf{x}_{t}}^{\top}({\mathbf{\tilde{G}}_{t}}{\mathbf{y}_{t}})+({\mathbf{y}_{t}}^{\top}({\mathbf{\tilde{G}}_{t}}^{\top}{\mathbf{{x}}_{t}}+\rho\frac{1}{s}{\mathbf{\tilde{G}}_{t}}^{\top}\frac{{{\mathbf{G}_{{t}}}}}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\mathbf{y}_{t}^{+}))
+\displaystyle=-{\mathbf{x}_{t}}^{\top}{\mathbf{\tilde{G}}_{t}}{\mathbf{y}_{t}}+({\mathbf{x}_{t}}^{\top}{\mathbf{\tilde{G}}_{t}}{\mathbf{y}_{t}}){{}^{\top}}+\rho\frac{1}{s}{\mathbf{y}_{t}}^{\top}{\mathbf{\tilde{G}}_{t}}^{\top}\frac{{{\mathbf{G}_{{t}}}}}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\mathbf{y}_{t}^{+}
+\displaystyle=\rho\frac{1}{s}{\mathbf{y}_{t}}^{\top}{\mathbf{\tilde{G}}_{t}}^{\top}\frac{{{\mathbf{G}_{{t}}}}}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\mathbf{y}_{t}^{+}
+\displaystyle=\rho\frac{1}{s}\frac{1}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\left[{{\mathbf{y}_{t}}^{\top}{\mathbf{\tilde{G}}_{t}}^{\top}{\mathbf{G}_{{t}}}\mathbf{y}_{t}^{+}}\right]
+
+Because \frac{1}{{{s}}}\mathbf{g_{x}}={\mathbf{G}_{{t}}}{\mathbf{y}_{t}} and {\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}={\mathbf{\tilde{G}}_{t}}{\mathbf{\tilde{y}}_{t}}, we have:
+
+\displaystyle\frac{1}{2}\frac{{d({{\left\|{{\mathbf{x}_{t}}}\right\|}^{2}}-{{\left\|{{\mathbf{y}_{t}}}\right\|}^{2}})}}{{dt}}(29)
+\displaystyle=\rho\frac{1}{s}\frac{1}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\left[{{\mathbf{y}_{t}}^{\top}{\mathbf{\tilde{G}}_{t}}^{\top}{\mathbf{G}_{{t}}}\mathbf{y}_{t}^{+}}\right]
+\displaystyle=\rho\frac{1}{{{s^{2}}}}\frac{1}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\left[{{{({\mathbf{\tilde{G}}_{t}}{\mathbf{y}_{t}})}^{\top}}\mathbf{{g}_{x}}{{(\mathbf{y}_{t}^{\top})}^{+}}\mathbf{y}_{t}^{+}}\right]
+\displaystyle=\rho\frac{1}{{{s^{2}}}}\frac{1}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\left[{{{({\mathbf{\tilde{G}}_{t}}{{}\mathbf{y}_{t}})}^{\top}}\mathbf{g_{x}}{{(\mathbf{y}_{t}^{+})}^{\top}}\mathbf{y}_{t}^{+}}\right]
+\displaystyle=\rho\frac{1}{{{s^{2}}}}\frac{1}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\left[{{{({\mathbf{\tilde{G}}_{t}}{\mathbf{y}_{t}})}^{\top}}\mathbf{g_{x}}{{\left\|{\mathbf{y}_{t}^{+}}\right\|}^{2}}}\right]
+\displaystyle=\rho\frac{1}{{{s^{2}}}}\frac{1}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\left[{{{({\mathbf{g}_{{{\mathbf{{\tilde{x}}}}_{t}}}}({\mathbf{\tilde{y}}_{t}}^{+})^{\top}{\mathbf{y}_{t}})}^{\top}}\mathbf{g_{x}}{{\left\|{\mathbf{y}_{t}^{+}}\right\|}^{2}}}\right]
+\displaystyle=\rho\frac{1}{{{s^{2}}}}\frac{1}{{{{\left\|{{\mathbf{G}_{t}}}\right\|}_{F}}}}\left[{{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}^{\top}\mathbf{g_{x}}{{\left\|{\mathbf{y}_{t}^{+}}\right\|}^{2}}}\right]
+
+Taking the absolute value of balancedness B_{t} gives:
+
+\displaystyle\left|{\frac{1}{2}\frac{{d({{\left\|{{\mathbf{x}_{t}}}\right\|}^{2}}-{{\left\|{{\mathbf{y}_{t}}}\right\|}^{2}})}}{{dt}}}\right|(30)
+\displaystyle=\left|{\rho\frac{1}{s}\frac{{{{\left\|{\mathbf{y}_{t}^{+}}\right\|}^{2}}}}{{{{\left\|{{\mathbf{{g}}_{\mathbf{x}}}{{(\mathbf{y}_{t}^{+})}^{\top}}}\right\|}_{F}}}}({{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}^{\top}\mathbf{g_{x}}})}\right|
+\displaystyle=\left|{\rho\frac{1}{s}\frac{{{{\left\|{\mathbf{y}_{t}^{+}}\right\|}^{2}}}}{{\left\|{\mathbf{{g}_{x}}}\right\|\left\|{{{(\mathbf{y}_{t}^{+})}^{\top}}}\right\|}}({{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}^{\top}\mathbf{g_{x}}})}\right|
+\displaystyle=\left|{\rho\frac{1}{s}\frac{{\left\|{\mathbf{y}_{t}^{+}}\right\|}}{{\left\|{\mathbf{{g}_{x}}}\right\|}}({{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}^{\top}\mathbf{g_{x}}})}\right|
+\displaystyle\leq\left|{\rho\frac{1}{s}\frac{{\left\|{\mathbf{y}_{t}^{+}}\right\|}}{{\left\|{\mathbf{g_{x}}}\right\|}}\left\|{{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}}\right\|\left\|{\mathbf{{{g}}_{x}}}\right\|}\right|
+\displaystyle=\left|{\rho\frac{1}{s}\left\|{\mathbf{y}_{t}^{+}}\right\|\left\|{{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}}\right\|}\right|
+\displaystyle=\left|{\rho\frac{1}{s}\left\|{\frac{{\mathbf{y}_{t}^{\top}}}{{{{\left\|{\mathbf{y}_{t}}\right\|}^{2}}}}}\right\|\left\|{{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}}\right\|}\right|
+\displaystyle=\left|{\rho\frac{1}{s}\frac{1}{{\left\|{\mathbf{y}_{t}}\right\|}}\left\|{{\mathbf{g}_{{{{\mathbf{\tilde{x}}}}_{t}}}}}\right\|}\right|
+
+The proof is thus completed. ∎
+
+###### Lemma 1.
+
+Let A_{t+1}=\alpha A_{t}+\beta with some \alpha\in(0,1), then we have
+
+A_{t+1}\leq\alpha^{t+1}A_{0}+\frac{\beta}{1-\alpha}.
+
+###### Proof.
+
+The proof can be completed by simply unrolling A_{t+1} and using the fact 1+\alpha+\alpha^{2}+\dots+\alpha^{t}\leq\frac{1}{1-\alpha}. ∎
+
+### A.3 Proof of Theorem 2
+
+###### Proof.
+
+Assume that \bm{\varepsilon}_{t} is the perturbation at time step t, and \bm{\hat{\varepsilon}}_{t-1} is the EMA perturbation from the previous step. Let \nabla L(\mathbf{w}_{t}+\bm{\hat{\varepsilon}}_{t-1}) denote the gradient used for updating at time t. The standard SAM perturbation at step t is defined as \bm{\tilde{\varepsilon}}_{t}=\rho_{t}\frac{\nabla L(\mathbf{w}_{t})}{\|\nabla L(\mathbf{w}_{t})\|}, and the EMA perturbation at step t is computed as \bm{\hat{\varepsilon}}_{t}=(1-\beta)\bm{\hat{\varepsilon}}_{t-1}+\beta\bm{\varepsilon}_{t}. Based on Assumption 4, we have that:
+
+\displaystyle\left[{L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})-L({\mathbf{w}_{t}})}\right]-\left[{L({\mathbf{w}_{t}}+{\bm{\tilde{\varepsilon}}_{t}})-L({\mathbf{w}_{t}})}\right](31)
+\displaystyle=L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})-L({\mathbf{w}_{t}}+{\bm{\tilde{\varepsilon}}_{t}})
+\displaystyle\leq-\nabla L{({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})^{\top}}({\mathbf{w}_{t}}+{\bm{\tilde{\varepsilon}}_{t}}-{\mathbf{w}_{t}}-{\bm{\hat{\varepsilon}}_{t-1}})
+\displaystyle=\nabla L{({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})^{\top}}({\bm{\hat{\varepsilon}}_{t-1}}-{\bm{\tilde{\varepsilon}}_{t}})
+\displaystyle\leq\left|{\nabla L{{({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})}^{\top}}({\bm{\hat{\varepsilon}}_{t-1}}-{\bm{\tilde{\varepsilon}}_{t}})}\right|
+\displaystyle\leq\left\|{\nabla L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})}\right\|\left\|{{\bm{\hat{\varepsilon}}_{t-1}}-{\bm{\tilde{\varepsilon}}_{t}}}\right\|(32)
+
+For the first term \left\|{\nabla L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})}\right\| in Eq.([32](https://arxiv.org/html/2508.00522v3#Sx5.E32 "Equation 32 ‣ A.3 Proof of Theorem 2 ‣ A. Proofs ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), Based on Assumption 1, Assumption 2 and Lemma 1, we have:
+
+\displaystyle\left\|{\nabla L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})}\right\|(33)
+\displaystyle=\left\|{\nabla L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})-\nabla L({\mathbf{w}_{t}})+\nabla L({\mathbf{w}_{t}})}\right\|
+\displaystyle\leq\left\|{\nabla L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})-\nabla L({\mathbf{w}_{t}})}\right\|+\left\|{\nabla L({\mathbf{w}_{t}})}\right\|
+\displaystyle\leq\tau\left\|{{\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}}-{\mathbf{w}_{t}}}\right\|+\left\|{\nabla L({\mathbf{w}_{t}})}\right\|
+\displaystyle=\tau\left\|{{\bm{\hat{\varepsilon}}_{t-1}}}\right\|+\left\|{\nabla L({\mathbf{w}_{t}})-\nabla{L_{\rm{D}}}({\mathbf{w}_{t}})+\nabla{L_{\rm{D}}}({\mathbf{w}_{t}})}\right\|
+\displaystyle=\tau\left\|{{\bm{\hat{\varepsilon}}_{t-1}}}\right\|+\left\|{\nabla{L_{\rm{D}}}({\mathbf{w}_{t}})}\right\|+{\sigma^{2}}
+\displaystyle=\tau\left\|{(1-\beta){\bm{\hat{\varepsilon}}_{t-2}}+\beta\bm{\varepsilon}_{t-1}}\right\|+G+{\sigma^{2}}
+\displaystyle\leq\tau((1-\beta)\left\|{{\bm{\hat{\varepsilon}}_{t-2}}}\right\|+\beta{\rho_{0}})+G+{\sigma^{2}}
+\displaystyle\leq\tau{(1-\beta)^{t-1}}\left\|{{\bm{\hat{\varepsilon}}_{0}}}\right\|+\tau{\rho_{0}}+G+{\sigma^{2}}
+
+For the second term \left\|{{\bm{\hat{\varepsilon}}_{t-1}}-{\bm{\tilde{\varepsilon}}_{t}}}\right\| in Eq.([32](https://arxiv.org/html/2508.00522v3#Sx5.E32 "Equation 32 ‣ A.3 Proof of Theorem 2 ‣ A. Proofs ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond")), we have:
+
+\displaystyle\left\|{{\bm{\hat{\varepsilon}}_{t-1}}-{\bm{\tilde{\varepsilon}}_{t}}}\right\|(34)
+\displaystyle\leq\left\|{{\bm{\hat{\varepsilon}}_{t-1}}}\right\|+\left\|{{\bm{\tilde{\varepsilon}}_{t}}}\right\|
+\displaystyle=\left\|{{\bm{\hat{\varepsilon}}_{t-1}}}\right\|+{\rho_{\rm{t}}}
+\displaystyle\leq{(1-\beta)^{t-1}}\left\|{{\bm{\hat{\varepsilon}}_{0}}}\right\|+{\rho_{0}}+{\rho_{t}}
+
+Let {\bm{\hat{\varepsilon}}_{0}}={\bm{\tilde{\varepsilon}}_{0}}=\rho_{0}\frac{\nabla L(\mathbf{w}_{0})}{\|\nabla L(\mathbf{w}_{0})\|}, {\rho_{t}}=\frac{{{\rho_{0}}}}{{\sqrt{t}}}, we have:
+
+\displaystyle\left[{L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t-1}})-L({\mathbf{w}_{t}})}\right]-\left[{L({\mathbf{w}_{t}}+{\bm{\tilde{\varepsilon}}_{t}})-L({\mathbf{w}_{t}})}\right](35)
+\displaystyle\leq\left(\tau{(1-\beta)^{t-1}}\left\|{{\bm{\hat{\varepsilon}}_{0}}}\right\|+\tau{\rho_{0}}+G+{\sigma^{2}}\right)
+\displaystyle\quad\cdot\left({(1-\beta)^{t-1}}\left\|{{\bm{\hat{\varepsilon}}_{0}}}\right\|+{\rho_{0}}+{\rho_{t}}\right)
+\displaystyle=\left({\left({1+{{(1-\beta)}^{t-1}}}\right)\tau{\rho_{0}}+G+{\sigma^{2}}}\right)
+\displaystyle\quad\quad\cdot\left({\left({1+{{(1-\beta)}^{t-1}}}\right){\rho_{0}}+\frac{{{\rho_{0}}}}{{\sqrt{t}}}}\right)
+
+The proof is thus completed. ∎
+
+## B. Experimental Details
+
+### B.1 Details on datasets
+
+Our evaluations are carried out on commonly-used datasets in the literature.
+
+Datasets for few-shot learning of RoBERTa-large. We consider classification datasets: SST-2 (socher2013recursive), SST-5 (socher2013recursive), TREC (voorhees2000building), MNLI (williams2018broad), SNLI (bowman2015large), and RTE (dagan2005pascal). We follow Malladi et al. (malladi2023kernel) in limiting the test set to 1, 000 examples for fast iteration. For training and validation, we set k = 512, which mean that we have 512 examples per class for both training and validation.
+
+Table 7: The hyperparameters used for RoBERTa large with LoRA on the GLUE benchmark.
+
+Table 8: Hyperparameters used for few-shot learning with RoBERTa-large.
+
+Table 9: Hyperparameters used for GPT2.
+
+GLUE benchmark. GLUE is designed to provide a general-purpose evaluation of language understanding (wangglue). Those adopted in our work include MNLI (inference, (williams2018broad)), SST-2 (sentiment analysis, (socher2013recursive)), MRPC (paraphrase detection, (dolan2005automatically)), CoLA (linguistic acceptability (warstadt2019neural)), QNLI (inference (rajpurkar2018know)), QQP 1 1 1 https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs (question-answering), RTE 2 2 2 https://paperswithcode.com/dataset/rte (inference), and STS-B (textual similarity (cer2017semeval)). These datasets are released under different permissive licenses.
+
+E2E NLG Challenge. The E2E NLG Challenge dataset (novikova2017e2e) is a standard benchmark for end-to-end data-to-text natural language generation. It consists of around 42,000 training instances, along with 4,600 each for validation and testing, all within the restaurant domain. Inputs are structured as sequences of slot-value pairs and paired with one or more reference texts. The dataset is released under the Creative Commons BY-NC-SA 4.0 license.
+
+Datasets for few-shot learning of CLIP. We consider five datasets for fine-grained classification of satellite imagery (EuroSAT (helber2019eurosat)), pet breeds (Ox-fordPets (parkhi2012cats)), flowers (Flower102 (nilsback2008automated)), general objects (Caltech101 (fei2004learning)), textures (DTD (cimpoi2014describing)). These datasets offer a thorough benchmarking framework for evaluating few-shot visual classification tasks.
+
+Datasets for fine-tuning with Qwen-VL-Chat. We use two representative datasets: ScienceQA (lu2022learn) and VizWiz (gurari2018vizwiz). ScienceQA is a multimodal multiple-choice QA dataset covering elementary science, with questions accompanied by text and images. VizWiz is a real-world visual QA dataset collected from blind users, featuring diverse and often low-quality images, posing challenges for robust multimodal understanding.
+
+### B.2 Details on models
+
+We summarize the adopted language models in our evaluation. All model checkpoints are obtained from HuggingFace.
+
+RoBERTa-large. This is a 355 M parameter model. The model checkpoint 3 3 3 https://huggingface.co/FacebookAI/roberta-large is released under the MIT license.
+
+GPT2-medium. This is a 345 M parameter model. Its checkpoint 4 4 4 https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-medium-pytorch˙model.bin is under MIT License.
+
+GPT2-large. This is a 774 M parameter model. Its checkpoint 5 5 5 https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-large-pytorch˙model.bin is under MIT License.
+
+CLIP. This is a model that learns to connect images and text by mapping them into a shared semantic space using contrastive learning.
+
+Qwen-VL-Chat. Qwen-VL-Chat (Bai2023QwenVLAV) is a multimodal conversational large language model capable of understanding both images and text.
+
+Table 10: Hyperparameters used for few-shot learning with CLIP.
+
+Algorithm 1 Pseudocode of the FMLoRA
+
+Require: The training dataset, the learning rate \eta, the batch size b, parameters \rho and \beta.
+
+1:for
+
+t=1,2,\cdot\cdot\cdot
+do
+
+2: Randomly sample a mini-batch;
+
+3: Evaluate the gradient at the current point;
+
+4: Apply Equation (12) to compute the gradient in the full parameter space
+
+{\bar{\mathbf{g}}^{\mathbf{W}}}
+;
+
+5: Use Equation (13) to calculate the perturbation
+
+\bar{\mathbf{E}}^{\mathbf{W}}
+;
+
+6: Compute the perturbation
+
+\bar{\mathbf{E}}^{\mathbf{B}}=\frac{1}{s}\bar{\mathbf{E}}^{\mathbf{W}}\mathbf{A}^{+}
+on matrix
+
+\mathbf{B}
+according to Equation (14);
+
+7: Evaluate the gradient at the perturbed point (
+
+\mathbf{W}_{0}
+,
+
+\mathbf{B}+\bar{\mathbf{E}}^{\mathbf{B}}
+,
+
+\mathbf{A}
+);
+
+8: Return to the original (unperturbed) parameter point (
+
+\mathbf{W}_{0}
+,
+
+\mathbf{B}
+,
+
+\mathbf{A}
+);
+
+9: Update the weights using the gradient obtained in Step 6;
+
+10:end for
+
+### B.3 Details on hyperparameters
+
+Few-shot Learning with RoBERTa. We adopt the k-shot learning setup from (malladi2023fine), focusing on classification tasks with k=512 training samples per class and 1000 samples for testing. Prompt-based finetuning is used, following the same prompt templates as in (malladi2023fine, Table 13). We use AdamW as the optimizer and tune hyperparameters based on Table [8](https://arxiv.org/html/2508.00522v3#Sx6.T8 "Table 8 ‣ B.1 Details on datasets ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"). All results are averaged over three random seeds.
+
+Fine-tuning with RoBERTa-large. AdamW is adopted as the base optimizer, and hyperparameters are in Table [7](https://arxiv.org/html/2508.00522v3#Sx6.T7 "Table 7 ‣ B.1 Details on datasets ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"). However, we employ single GPU rather than multiple ones and use gradient accumulation rather than parallelism due to memory constraint. We consider the GLUE benchmark and report the mismatched accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for other datasets. Larger values indicate better results for all datasets. Experiments are conducted over three random trials for all datasets.
+
+GPT2 medium/large on E2E NLG Challenge. We use the batch size, learning rate, and beam search beam size described in (hu2022lora). AdamW is adopted as base optimizer. The hyperparameters can be found in Table [9](https://arxiv.org/html/2508.00522v3#Sx6.T9 "Table 9 ‣ B.1 Details on datasets ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"). The result for each run is taken from the last epoch.
+
+Few-shot Learning with CLIP. We follow the setting of previous work (zanella2024low). The hyperparameters are tuned from those in Table [10](https://arxiv.org/html/2508.00522v3#Sx6.T10 "Table 10 ‣ B.2 Details on models ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"). We only apply low-rank matrices on the query, key and value matrices with r=2. We regularize the input of the LoRA module by a dropout layer with p=0.25. The number of iterations is set equal to 500 times N/K (the number of labeled samples per class).
+
+Fine-tuning with Qwen-VL-Chat. We conduct experiments follow the setting of previous work (zhou2024empirical). The hyperparameters can be found in Table [11](https://arxiv.org/html/2508.00522v3#Sx6.T11 "Table 11 ‣ B.3 Details on hyperparameters ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond").
+
+Table 11: Hyperparameters used for fine-tuning with Qwen-VL-Chat.
+
+![Image 3: Refer to caption](https://arxiv.org/html/2508.00522v3/x3.png)
+
+Figure 3: Approximation ability of EMA perturbations across datasets
+
+![Image 4: Refer to caption](https://arxiv.org/html/2508.00522v3/x4.png)
+
+Figure 4: Evolution of balancedness across layers during training with Adam and FMLoRA.
+
+## C. Algorithm
+
+The two algorithms presented in [1](https://arxiv.org/html/2508.00522v3#alg1 "Algorithm 1 ‣ B.2 Details on models ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") and [2](https://arxiv.org/html/2508.00522v3#alg2 "Algorithm 2 ‣ C. Algorithm ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") describe the training procedures of the proposed FMLoRA and its accelerated variant EFMLoRA.
+
+Algorithm 2 Pseudocode of the EFMLoRA
+
+Require: The training dataset, the learning rate \eta, the batch size b, parameters \rho and \beta.
+
+1:for
+
+t=1,2,\cdot\cdot\cdot
+do
+
+2: Randomly sample a mini-batch;
+
+3:if
+
+t=1
+then
+
+4: Evaluate the gradient at the current point;
+
+5: EMA perturbation
+
+\hat{\mathbf{E}}^{\mathbf{B}}_{1}=\bar{\mathbf{E}}^{{\mathbf{B}}}_{1}
+;
+
+6: Update the weights using the gradient obtained in Step 4;
+
+7: Update the parameters to the next perturbation point
+
+(\mathbf{W}_{0}
+,
+
+\mathbf{B}_{1}+\hat{\mathbf{E}}^{\mathbf{B}}_{1}
+,
+
+\mathbf{A}_{1})
+.
+
+8:else
+
+9: Calculate the gradient at the perturbation point
+
+(\mathbf{W}_{0}
+,
+
+\mathbf{B}_{t-1}+\hat{\mathbf{E}}^{\mathbf{B}}_{t-1}
+,
+
+\mathbf{A}_{t-1})
+.
+
+10: Compute the perturbation
+
+\bar{\mathbf{E}}^{\mathbf{B}}_{t}=\frac{1}{s}\bar{\mathbf{E}}^{\mathbf{W}}\mathbf{A}^{+}_{t-1}
+on matrix
+
+\mathbf{B}
+according to Equation (14);
+
+11: Return to the original parameter point
+
+(\mathbf{W}_{0}
+,
+
+\mathbf{B}_{t-1}
+,
+
+\mathbf{A}_{t-1})
+.
+
+12: Calculate the EMA perturbation
+
+{{\hat{\mathbf{E}}}^{\mathbf{B}}_{t}}=(1-\beta){{\hat{\mathbf{E}}}^{\mathbf{B}}_{t-1}}+\beta{\bar{\mathbf{E}}^{\mathbf{B}}_{t}}
+.
+
+13: Update the weights to
+
+(\mathbf{W}_{0}
+,
+
+\mathbf{B}_{t}
+,
+
+\mathbf{A}_{t})
+using the gradient obtained in Step 9;
+
+14: Update the parameters to the next perturbation point
+
+(\mathbf{W}_{0}
+,
+
+\mathbf{B}_{t}+\hat{\mathbf{E}}^{\mathbf{B}}_{t}
+,
+
+\mathbf{A}_{t})
+.
+
+15:end if
+
+16:end for
+
+## D. More experiments
+
+### D.1 The approximate ability of EMA perturbation
+
+We consider few shot learning with LoRA on RoBERTa-large. Fig.[3](https://arxiv.org/html/2508.00522v3#Sx6.F3 "Figure 3 ‣ B.3 Details on hyperparameters ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond") illustrates the evolution of the difference in sharpness, \left[{L({\mathbf{w}_{t}}+{\bm{\hat{\varepsilon}}_{t}})-L({\mathbf{w}_{t}})}\right]-\left[{L({\mathbf{w}_{t}}+{\bm{\tilde{\varepsilon}}_{t}})-L({\mathbf{w}_{t}})}\right], as described in Theorem 2, during training on six datasets (SNLI, SST-2, SST-5, MNLI, RTE, and TREC). S^{\text{EMA}} denotes the sharpness computed using EMA perturbations, while S^{\text{SAM}} refers to the original SAM sharpness. As training progresses, the absolute difference consistently decreases across all datasets, demonstrating that the EMA perturbation becomes increasingly effective at approximating the SAM perturbations. This validates the use of EMA perturbations as a computationally efficient surrogate for SAM perturbations. This result empirically supports Theorem 2.
+
+### D.2 The change in balancedness during FMLoRA training
+
+We consider few shot learning with LoRA on RoBERTa-large. For dataset MNLI, 1st, 12th and 24th query layers’ 2|B_{t,l}| are plotted, where t denotes the iteration and l denotes the layer index. The layers are chosen to represent early, middle, and final stages of RoBERTa. Balancedness of FMLoRA and Adam on different layers are plotted in Fig.[4](https://arxiv.org/html/2508.00522v3#Sx6.F4 "Figure 4 ‣ B.3 Details on hyperparameters ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"). Balancedness may increase or decrease across different layers. As shown in Fig.[4](https://arxiv.org/html/2508.00522v3#Sx6.F4 "Figure 4 ‣ B.3 Details on hyperparameters ‣ B. Experimental Details ‣ Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond"), the balancedness of FMLoRA in the first query layer of RoBERTa-large gradually decreases during training, while in the 12th layer, it first decreases and then increases. In contrast, the balancedness in the 24th layer continuously increases. An increase typically occurs when parameter magnitudes in both low-rank subspaces grow simultaneously. This behavior can be influenced by factors such as the learning rate, optimization algorithm, weight decay, and other regularization strategies. Despite these occasional increases, FMLoRA generally maintains lower balancedness than Adam in most layers, suggesting its capacity to induce implicit regularization during training.

docs/lora_mgpo_momentum_perturbation.md

@@ -0,0 +1,465 @@
+Title: 2025.findings-emnlp.34.pdf
+
+URL Source: https://aclanthology.org/2025.findings-emnlp.34.pdf
+
+Published Time: Fri, 31 Oct 2025 19:44:11 GMT
+
+Number of Pages: 12
+
+Markdown Content:
+> Findings of the Association for Computational Linguistics: EMNLP 2025 , pages 648–659 November 4-9, 2025 ©2025 Association for Computational Linguistics
+
+# LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization
+
+Yupeng Chang 1 Chenlu Guo 1 Yi Chang 1,2,3 Yuan Wu 1*
+
+> 1
+
+School of Artificial Intelligence, Jilin University
+
+> 2
+
+Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China
+
+> 3
+
+International Center of Future Science, Jilin University {changyp23, guocl23}@mails.jlu.edu.cn, {yichang, yuanwu}@jlu.edu.cn
+
+Abstract
+
+Parameter-efficient fine-tuning (PEFT), partic-ularly Low-Rank Adaptation (LoRA), adapts large language models (LLMs) by training only a small fraction of parameters. However, as the rank of the low-rank matrices used for adap-tation increases, LoRA often exhibits an un-stable "double descent" phenomenon, charac-terized by transient divergence in the training loss, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima. To address this, we introduce LoRA-MGPO , a framework that incorporates Momentum-Guided Pertur-bation Optimization (MGPO). MGPO stabi-lizes training dynamics by mitigating the dou-ble descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer’s state, thus avoiding dual gra-dient computations. Additionally, an adaptive normalization scheme scales the magnitude of perturbations based on an exponential mov-ing average (EMA) of gradient norms, further enhancing stability. While EMA controls the magnitude of the perturbations, MGPO guides their direction, ensuring a more stable opti-mization trajectory. Experiments on a suite of natural language understanding and genera-tion benchmarks show that LoRA-MGPO con-sistently achieves superior performance over LoRA and other PEFT methods. The analysis indicates that LoRA-MGPO leads to smoother loss curves, faster convergence, and improved generalization by stabilizing the training pro-cess and mitigating the attraction to sharp min-ima. The code is publicly available at https: //github.com/llm172/LoRA-MGPO .
+
+1 Introduction
+
+Large language models (LLMs) have driven signif-icant advancements in natural language process-ing, establishing new performance benchmarks
+
+> *Corresponding authors
+
+on tasks ranging from text generation to seman-tic understanding (Chang et al., 2024b; Wei et al., 2022). However, the conventional method of full-parameter fine-tuning (Full FT) requires updating billions of parameters, incurring prohibitive mem-ory and computational costs. To overcome this limitation, parameter-efficient fine-tuning (PEFT) methods have emerged as an effective alternative, enabling efficient adaptation by optimizing only a small subset of model parameters (Lester et al., 2021; Fu et al., 2023). Among these methods, Low-Rank Adaptation (LoRA) (Hu et al., 2021) is distinguished by its computational efficiency and architectural simplic-ity. LoRA approximates the weight update ma-trix ∆W as a low-rank decomposition, where the original pre-trained weights W0 remain frozen. The trainable matrices B and A, with rank r ≪
+
+min( m, n ), drastically reduce the number of train-able parameters, improving efficiency without al-tering the model architecture. Despite its efficiency, LoRA’s training dynam-ics can be unstable. As shown in Figure 1, fine-tuning LLaMA-2-7B (Touvron et al., 2023) on MetaMathQA (Yu et al., 2024) often exhibits a "double descent" trajectory with initial conver-gence, transient divergence, and eventual stabi-lization. This phenomenon worsens with higher ranks and is not unique to LoRA; Full FT can ex-hibit even more severe double descent, highlighting the general challenge of stabilizing fine-tuning in high-capacity models (Nakkiran et al., 2019). Such non-monotonic behavior delays convergence and impairs generalization due to unstable gradients and the attraction to sharp local minima (Li et al., 2024a). Addressing these stability issues is crucial. Sharpness-Aware Minimization (SAM) (Foret et al., 2020) improves generalization by seeking flatter minima. However, its application is hin-dered by the dual gradient computation require-
+
+648 0 1000 2000 3000 4000 5000 6000
+
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> Training Loss
+> LoRA-rank32
+> 01000 2000 3000 4000 5000 6000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> LoRA-rank64
+> 01000 2000 3000 4000 5000 6000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> LoRA-rank128
+> 01000 2000 3000 4000 5000 6000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> 1.2
+> 1.4
+> 1.6
+> Full FT
+
+Figure 1: Training loss curves of Full FT and LoRA (Hu et al., 2021) methods with LLaMA-2-7B (Touvron et al., 2023) on the MetaMathQA dataset (Yu et al., 2024). For LoRA, rank ( r) and alpha ( α) are set to the same values (r = α ∈ { 32 , 64 , 128 }), with a fixed learning rate of 5e − 4.
+
+ment, which doubles the training cost (Becker et al., 2024; Li et al., 2024b). More efficient variants like momentum-guided SAM reuse optimizer states to avoid this overhead but may not guarantee stable convergence. To further enhance stability, comple-mentary techniques such as applying an exponen-tial moving average (EMA) to smooth optimization dynamics have been shown to suppress parameter oscillations and improve convergence in certain scenarios (Wang et al., 2021). Building on these insights, we propose LoRA-MGPO , a novel framework that integrates Momentum-Guided Perturbation Optimization (MGPO) into LoRA to mitigate the detrimental effects of double descent. Our contributions are twofold: 1. Mitigating Double Descent: MGPO stabi-lizes training by addressing double descent, typically observed at higher ranks in LoRA. By reusing momentum vectors, it guides weight perturbations towards flatter minima, preventing transient divergences in loss. 2. Adaptive Perturbation Normalization:
+
+MGPO introduces an adaptive scheme that scales perturbation magnitude based on an exponential moving average (EMA) of gradient norms, decoupling perturbation intensity from optimization dynamics and further enhancing stability. We evaluate LoRA-MGPO on a suite of natural lan-guage understanding (NLU) and generation (NLG) benchmarks. Our results show that it consistently achieves superior performance over standard LoRA and other state-of-the-art PEFT methods. Crucially, we demonstrate that LoRA-MGPO effectively mit-igates the double descent phenomenon, leading to more stable training dynamics, smoother loss curves, and faster convergence, all of which con-tribute to better generalization and the avoidance of sharp minima.
+
+2 Method
+
+In this section, we first provide a concise overview of the Low-Rank Adaptation (LoRA) framework. We then introduce LoRA-MGPO , an extension of LoRA that integrates Momentum-Guided Perturba-tion Optimization (MGPO) to enhance its stability and efficiency. We describe how MGPO reuses op-timizer momentum for guided perturbations of the trainable parameters and incorporates an adaptive normalization scheme to stabilize training.
+
+2.1 Review of LoRA
+
+While full fine-tuning directly updates the entire pre-trained weight matrix W0 ∈ Rm×n, its pro-hibitive computational cost makes it impractical for large-scale models. Low-Rank Adaptation (LoRA) (Hu et al., 2021) offers a parameter-efficient al-ternative. LoRA freezes W0 and injects a train-able low-rank decomposition, ∆W = BA , where
+
+B ∈ Rm×r and A ∈ Rr×n are trainable matrices with rank r ≪ min( m, n ). The weight update is incorporated into the forward pass as:
+
+Y = X(W0 + αr BA ), (1) where X is the input, α is a scaling hyperparameter, and r is the rank of the decomposition. Typically,
+
+A is initialized with a Kaiming normal distribution, and B with zeros. While effective, LoRA can suf-fer from training instability, particularly the double descent phenomenon, when r increases without appropriate optimization strategies to maintain sta-bility (Li et al., 2024a). 649 2.2 LoRA with Momentum-Guided Perturbation Optimization
+
+To address the training instabilities in LoRA, we propose LoRA-MGPO , which integrates Momentum-Guided Perturbation Optimization (MGPO). Inspired by Sharpness-Aware Minimiza-tion (SAM), MGPO is redesigned for computa-tional efficiency and parameter efficiency. It di-rectly perturbs the trainable LoRA parameters by reusing the optimizer’s first-moment estimate, guid-ing the perturbations toward stable directions. Ad-ditionally, MGPO incorporates adaptive normaliza-tion to dynamically scale the perturbation, enhanc-ing training stability.
+
+2.2.1 Motivation: SAM for LoRA and Its Limitations
+
+The goal of SAM (Foret et al., 2020) is to find parameters in flat loss regions to improve gener-alization. A direct application to LoRA would involve perturbing the full weight matrix, solv-ing min A,B max ∥ϵ∥F ≤ρ L (W0 + BA + ϵ). This approach is ill-suited for PEFT due to two critical flaws: (1) its dual gradient computation require-ment doubles the training cost, and (2) creating and storing the full-space perturbation ϵ counteracts the memory savings of LoRA. MGPO is explicitly designed to resolve these inefficiencies.
+
+2.2.2 Momentum-Guided Perturbation of LoRA Parameters
+
+MGPO achieves the stability benefits of SAM by perturbing the trainable parameters θ = ( A, B )
+
+directly, using information readily available in the optimizer’s state. At each training step t, instead of computing a new gradient for the perturbation direction, it reuses the optimizer’s first-moment vector (momentum) from the previous step, mt−1.The optimization objective is:
+
+min
+
+> θ
+
+L(θt + ϵθt ), (2) where the perturbation ϵθt applied to the LoRA parameters θt = ( At, B t) is constructed using the state from step t − 1:
+
+ϵθt = ρ · mt−1
+
+∥mt−1∥2
+
+· 1¯g(t−1) . (3) Here, ρ is the perturbation radius. Using the histori-cal momentum vector is a deliberate design choice, as it represents a smoothed average of past gradi-ents, filtering out the noise from any single mini-batch and providing a more stable direction for assessing landscape sharpness. This vector is main-tained by the optimizer itself. After computing the gradient on the perturbed parameters, the momen-tum for the current step is updated as:
+
+mt = μmt−1 + ∇˜θt L. (4) The decay factor μ (e.g., ‘beta1‘ in AdamW) is reused from the optimizer’s standard settings. The scalar ¯g(t−1) is a global normalization factor, de-tailed next. This formulation entirely avoids the second gradient computation and any operations in the full weight space.
+
+Two-Stage Update Mechanism MGPO is imple-mented efficiently within each training step t. First, using the state from step t − 1, we compute the per-turbation ϵθt and apply it to the current parameters
+
+θt to get a perturbed version, ˜θt:
+
+˜θt = θt + ϵθt . (5) Second, the loss and its gradient are computed with respect to these perturbed parameters: ∇˜θt L. This single gradient is then used by the optimizer to up-date both the original parameters from θt to θt+1
+
+and the momentum from mt−1 to mt. For infer-ence, the final, unperturbed parameters θT are used.
+
+2.2.3 Adaptive Perturbation Normalization
+
+To ensure robustness across training stages, we in-troduce an Adaptive Perturbation Normalization (APN) scheme. The normalization factor ¯g(t) used in Equation 3 is a scalar computed via an exponen-tial moving average (EMA) of the global L2-norm of the LoRA parameter gradients. Following the principle of using the actually computed gradient, the update rule is:
+
+¯g(t) = β¯g(t−1) + (1 − β)∥∇ ˜θt L∥ 2, (6) where β is the EMA decay rate. This mechanism makes the perturbation scale-invariant relative to the gradient dynamics. For instance, during early training with large gradients, the normalization fac-tor increases, reducing the effective perturbation size to prevent destabilization. Conversely, in later stages, it ensures the perturbation remains suffi-ciently large to be effective. This adaptive scaling enhances training stability.
+
+3 Experiments
+
+3.1 Experimental Setup Baselines To provide a comprehensive evalua-tion, we compare LoRA-MGPO against a carefully 650 Table 1: Performance of T5-Base on five GLUE tasks, comparing LoRA-MGPO with full fine-tuning and other LoRA variants (rank r = 8 ). Scores are reported for the primary metric of each task, averaged over 3 runs, with standard deviations shown in subscripts. Bold indicates the best score, while underlining denotes the second best.
+
+Method MNLI SST2 CoLA QNLI MRPC Avg
+
+Train Size 393k 67k 8.5k 105k 3.7k
+
+Full FT 86.33 ±0.00 94.75 ±0.21 80.70 ±0.24 93.19 ±0.22 84.56 ±0.73 87.91 LoRA 85.30 ±0.04 94.04 ±0.11 69.35 ±0.05 92.96 ±0.09 68.38 ±0.01 82.08
+
+LoRA Variants with Modified Structure
+
+DoRA 85.67 ±0.09 94.04 ±0.53 72.04 ±0.94 93.04 ±0.06 68.08 ±0.51 82.57 AdaLoRA 85.45 ±0.11 93.69 ±0.20 69.16 ±0.24 91.66 ±0.05 68.14 ±0.28 81.62
+
+LoRA Variants with Original Structure
+
+PiSSA 85.75 ±0.07 94.07 ±0.06 74.27 ±0.39 93.15 ±0.14 76.31 ±0.51 84.71 rsLoRA 85.73 ±0.10 94.19 ±0.23 72.32 ±1.12 93.12 ±0.09 52.86 ±2.27 79.64 LoRA+ 85.81 ±0.09 93.85 ±0.24 77.53 ±0.20 93.14 ±0.03 74.43 ±1.39 84.95 LoRA-GA 85.70 ±0.09 94.11 ±0.18 80.57 ±0.20 93.18 ±0.06 85.29 ±0.24 87.77 LoRA-MGPO 86.58 ±0.11 94.72 ±0.46 82.32 ±0.18 93.79 ±0.46 86.62 ±0.68 88.81
+
+selected set of baselines. These include Full Fine-Tuning (Full FT), serving as a strong performance benchmark, and vanilla LoRA (Hu et al., 2021), our primary point of comparison. We further in-clude two categories of state-of-the-art LoRA vari-ants. The first category, variants with architec-tural modifications , comprises methods that alter the LoRA structure itself, such as DoRA (Liu et al., 2024), which introduces learnable magnitude vec-tors, and AdaLoRA (Zhang et al., 2023), which dynamically allocates rank budgets. The second category, variants improving the training process or initialization , includes rsLoRA (Kalajdzievski, 2023), which stabilizes update magnitudes; LoRA+ (Hayou et al., 2024), which employs different learn-ing rates for the LoRA matrices; and PiSSA (Meng et al., 2024), which refines initialization using SVD. Finally, we compare against methods focused on
+
+gradient alignment , such as LoRA-GA (Wang et al., 2024a) and LoRA-Pro (Wang et al., 2024b), which aim to align LoRA’s gradient updates more closely with those of full fine-tuning.
+
+Datasets Our experiments span a range of tasks in natural language understanding and generation. For NLU, we evaluate on five tasks from the widely-used General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018): MNLI, SST-2, CoLA, QNLI, and MRPC. These tasks cover natural language inference, sentiment analy-sis, grammatical acceptability, and paraphrase iden-tification. For NLG, we fine-tune the LLaMA-2-7B (Tou-vron et al., 2023) model on a 52k randomly sam-pled subset of the WizardLM dataset (Xu et al., 2024). We evaluate the model on the MT-Bench dataset (Zheng et al., 2024a), which consists of 80 multi-turn questions designed to assess conversa-tional abilities across various aspects. The quality of the responses is evaluated by GPT-4, and we report the first-turn score as the primary evaluation metric. For mathematical reasoning, we use a 100k ran-dom sample from MetaMathQA (Yu et al., 2024), with evaluation on the GSM8K test set (Cobbe et al., 2021). For code generation, fine-tuning is performed on a 100k randomly sampled subset of the CodeFeedback dataset (Zheng et al., 2024b), with evaluation on HumanEval (Chen et al., 2021).
+
+Implementation Details For fair comparison, our experimental setup closely follows that of LoRA-GA (Wang et al., 2024a). Across all experi-ments, we use the AdamW optimizer (Loshchilov and Hutter, 2019) with weight decay set to 0 and a cosine learning rate schedule with a warm-up ratio of 0.03. LoRA adapters are applied to all linear layers within the transformer blocks, with the rank
+
+r set to 8 and scaling factor α to 16 by default. For our two task families, the settings are as follows. For Natural Language Understanding (NLU) on GLUE, we fine-tune T5-base (Raffel et al., 2020) with a learning rate of 1 × 10 −4, a sequence length of 128, and a batch size of 32. The MGPO hy-651 Table 2: Fine-tuning results of LLaMA-2-7B on MT-Bench, GSM8K, and HumanEval. Performance is evaluated using primary task metrics: MT-Bench score, GSM8K accuracy, and HumanEval Pass@1. PEFT methods are tested with rank r = 8 , and additional tests at ranks 32 and 128 are included to evaluate performance scaling. Results are averaged over three random seeds, with standard deviations provided. Bold and underlining denote the best and second-best scores, respectively.
+
+Method MT-Bench GSM8K HumanEval Avg
+
+Full FT 5.30 ±0.11 59.36 ±0.85 35.31 ±2.13 33.32
+
+LoRA 5.61 ±0.10 42.08 ±0.04 14.76 ±0.17 20.82 DoRA 5.97 ±0.02 53.07 ±0.75 19.75 ±0.41 26.26 AdaLoRA 5.57 ±0.05 50.72 ±1.39 17.80 ±0.44 24.70 PiSSA 5.30 ±0.02 44.54 ±0.27 16.02 ±0.78 21.95 rsLoRA 5.25 ±0.03 45.62 ±0.10 16.01 ±0.79 22.29 LoRA+ 5.71 ±0.08 52.11 ±0.62 18.17 ±0.52 25.33 LoRA-GA 5.95 ±0.16 53.60 ±0.30 19.81 ±1.46 26.45 LoRA-GA (rank=32) 5.79 ±0.09 55.12 ±0.30 20.18 ±0.19 27.03 LoRA-GA (rank=128) 6.13 ±0.07 55.07 ±0.18 23.05 ±0.37 28.08 LoRA-MGPO 6.27 ±0.12 54.56 ±0.44 21.02 ±0.39 27.28 LoRA-MGPO (rank=32) 6.21 ±0.15 55.74 ±0.21 21.34 ±0.47 27.76 LoRA-MGPO (rank=128) 6.48 ±0.23 56.96 ±0.35 24.87 ±0.54 29.44 perparameters are ρ = 0 .05 , μ = 0 .9 (AdamW’s ‘beta1‘), and β = 0 .9. For Natural Language Gener-ation (NLG), we fine-tune LLaMA-2-7B (Touvron et al., 2023) with a learning rate of 2 × 10 −5 and a sequence length of 1024. We use a per-device batch size of 4 with 8 gradient accumulation steps for an effective batch size of 32. The MGPO hyperparam-eters are ρ = 0 .01 , μ = 0 .8 (AdamW’s ‘beta1‘), and β = 0 .8. All experiments were conducted on NVIDIA H20 96GB GPUs, repeated three times with different random seeds, and we report the av-erage and standard deviation of the results. Further details on optimizer settings, specific LoRA target modules, and the software environment are pro-vided in the Appendix.
+
+3.2 Main Results Performance on Natural Language Understand-ing (NLU) We first evaluated LoRA-MGPO on a standard suite of NLU tasks from the GLUE bench-mark (Wang et al., 2018), using the T5-base model. As detailed in Table 1, our method demonstrates strong and consistent performance. The improve-ments are particularly notable on challenging, low-resource benchmarks such as CoLA and MRPC, where LoRA-MGPO surpasses not only all other PEFT methods but also full fine-tuning. Success on these tasks often hinges on capturing subtle lin-guistic nuances. The stability afforded by LoRA-MGPO likely prevents the fine-tuning process from corrupting the rich knowledge encoded in the base model; by preventing erratic weight updates, our method may better preserve the pre-trained model’s nuanced understanding of syntax and semantics. Quantitatively, LoRA-MGPO achieves the highest scores among all PEFT methods on five out of five tasks, obtains the best average score, and outper-forms the next-best PEFT method, LoRA-GA, by a margin of 1.04 points.
+
+Performance on Natural Language Genera-tion (NLG) We further assessed our method on three challenging NLG tasks using the LLaMA-2-7B model, with results summarized in Ta-ble 2. LoRA-MGPO consistently secures top performance among all PEFT baselines. On the conversational MT-Bench, its top score suggests that stable training helps maintain the model’s coherence and instruction-following capabilities. For structured reasoning tasks like mathemati-cal problem-solving (GSM8K) and code genera-tion (HumanEval), where logical consistency is paramount, LoRA-MGPO again emerges as the strongest PEFT method. A stable optimization tra-jectory may reduce the risk of the model deviating from a correct reasoning path during fine-tuning, as each update step is more measured, preventing 652 0 500 1000 1500 2000 2500 3000
+
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> 1.2
+> 1.4
+> 1.6
+> Training Loss
+> rank 16
+> Full FT
+> LoRA-MGPO
+> LoRA
+> 0500 1000 1500 2000 2500 3000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> 1.2
+> 1.4
+> 1.6
+> rank 32
+> Full FT
+> LoRA-MGPO
+> LoRA
+> 0500 1000 1500 2000 2500 3000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> 1.2
+> 1.4
+> 1.6
+> rank 64
+> Full FT
+> LoRA-MGPO
+> LoRA
+> 0500 1000 1500 2000 2500 3000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> 1.2
+> 1.4
+> 1.6
+> rank 128
+> Full FT
+> LoRA-MGPO
+> LoRA
+
+Figure 2: Training loss dynamics across different rank configurations: A comparative analysis of LoRA, LoRA-MGPO, and full fine-tuning on LLaMA-2-7B with MetaMathQA. Rank ( r) and alpha ( α) follow r = α ∈{16 , 32 , 64 , 128 } with a fixed learning rate of 5e − 4.0 500 1000 1500 2000 2500 3000
+
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> Training Loss
+> lr = 2e-4
+> Full FT
+> LoRA-MGPO
+> LoRA
+> 0500 1000 1500 2000 2500 3000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> 1.2
+> lr = 3e-4
+> Full FT
+> LoRA-MGPO
+> LoRA
+> 0500 1000 1500 2000 2500 3000
+> Steps
+> 0.0
+> 0.2
+> 0.4
+> 0.6
+> 0.8
+> 1.0
+> 1.2
+> lr = 4e-4
+> Full FT
+> LoRA-MGPO
+> LoRA
+> 0500 1000 1500 2000 2500 3000
+> Steps
+> 0.00
+> 0.25
+> 0.50
+> 0.75
+> 1.00
+> 1.25
+> 1.50
+> 1.75
+> 2.00
+> lr = 6e-4
+> Full FT
+> LoRA-MGPO
+> LoRA
+
+Figure 3: Learning rate sensitivity analysis: A comparison of training loss for LoRA, LoRA-MGPO, and full fine-tuning on LLaMA-2-7B with MetaMathQA. The analysis spans learning rates {2e − 4, 3e − 4, 4e − 4, 6e − 4},with rank ( r) and alpha ( α) fixed at 128. Full FT
+
+> LoRA (Baseline)
+> LoRA + MGPO
+> LoRA-MGPO (Full)
+> 0
+> 10
+> 20
+> 30
+> 40
+> 50
+> 60
+
+(a) Performance on NLG Tasks
+
+> MT-Bench
+> GSM8K
+> HumanEval
+> 5.30 5.61 5.69 6.27
+> 59.36
+> 42.08
+> 54.12 54.56
+> 35.31
+> 14.76
+> 20.43 21.02
+> Full FT
+> LoRA (Baseline)
+> LoRA + MGPO
+> LoRA-MGPO (Full)
+> 82
+> 84
+> 86
+> 88
+> 90
+
+(b) Average Performance on GLUE
+
+> 87.91
+> 82.08
+> 86.76
+> 88.81
+> GLUE Average
+
+Figure 4: Ablation study of LoRA-MGPO on NLG and NLU tasks. (a) LLaMA-2-7B performance across three NLG tasks. (b) T5-Base performance on the GLUE benchmark. "LoRA (Baseline)" refers to standard LoRA, "LoRA + MGPO" refers to an ablation with only momentum-guided perturbation, and "LoRA-MGPO (Full)" includes both momentum-guided perturbation and adaptive normalization.
+
+catastrophic error accumulation common in multi-step generation. While full fine-tuning still holds an edge on the reasoning tasks, our method narrows the gap and outperforms it on MT-Bench. Notably, as the LoRA rank increases from 8 to 128, the performance of LoRA-MGPO scales gracefully, validating its ability to effectively leverage a higher parameter budget while maintaining the training stability that standard LoRA often lacks at higher ranks.
+
+3.3 Analysis and Ablation Studies Effectiveness in Mitigating Double Descent To empirically validate LoRA-MGPO’s core claim of mitigating double descent, we conducted a con-trolled analysis of its training dynamics, focusing on the impacts of rank and learning rate. The re-sults, presented in Figure 2 and Figure 3, offer compelling visual evidence of our method’s sta-bility. Figure 2 illustrates that as the LoRA rank
+
+r increases, the double descent phenomenon in standard LoRA becomes progressively more se-653 Table 3: Comparison of computational efficiency and performance across LoRA, LoRA-MGPO, and Full FT methods, trained for one epoch on the WizardLM dataset using LLaMA-2-7B.
+
+Method #Params Memory Cost Training Time MT-Bench GSM8K HumanEval Full FT 6738M >96 GB - 5.30 ±0.11 59 .36 ±0.85 35 .31 ±2.13
+
+LoRA 320M 81.73 GB 5h 48min 5.61 ±0.10 42 .08 ±0.04 14 .76 ±0.17
+
+LoRA-MGPO 320M 90.56 GB 6h 52min 6.27 ±0.12 54 .56 ±0.44 21 .02 ±0.39
+
+Table 4: Ablation study of LoRA-MGPO vs. random noise perturbation on three NLG benchmarks. Exper-iments use LLaMA-3.1-8B-Base (Dubey et al., 2024) with rank r = 8 . Scores are averaged over three random seeds, with standard deviations in subscripts. Bold indi-cates the best method.
+
+> Method MTBench GSM8k HumanEval
+> Full FT 5.88 ±0.23 73.69 ±0.28 51.63 ±1.27
+> LoRA 6.15 ±0.02 67.78 ±1.25 43.09 ±0.35
+> LoRA + Random Noise 6.43 ±0.26 68.05 ±1.12 42.92 ±0.41
+> LoRA-MGPO 7.51 ±0.07 70.23 ±1.08 45.13 ±0.63
+
+vere, exhibiting a sharp rebound at r = 128 . In stark contrast, LoRA-MGPO’s loss curve remains smooth and monotonically decreasing across all ranks. Similarly, Figure 3 shows that while higher learning rates induce significant oscillations in stan-dard LoRA, LoRA-MGPO maintains a stable con-vergence path. These findings provide strong empir-ical evidence that our method effectively stabilizes fine-tuning and potentially broadens the effective learning rate window.
+
+Ablation Study To rigorously dissect the indi-vidual and combined contributions of our method’s two key components—Momentum-Guided Pertur-bation (MGPO) and Adaptive Perturbation Normal-ization (APN)—we conducted a detailed ablation study, with results shown in Figure 4. The findings clearly validate our design choices. The first abla-tion step, labeled ‘LoRA + MGPO‘ , applies only the MGPO component and yields a substantial per-formance lift over the vanilla ‘LoRA (Baseline)‘ .On the NLU task suite, for instance, this single component boosts the average score from 82.08 to 86.76, demonstrating that the core strategy of using momentum to guide perturbations towards flatter loss regions is fundamentally effective. However, the full potential is unlocked when introducing APN. Our complete model, labeled
+
+‘LoRA-MGPO (Full)‘ , combines both compo-nents and achieves the final NLU score of 88.81. The significant improvement from 86.76 to 88.81 underscores the critical role of adaptive normal-ization. It suggests that while MGPO provides a stable perturbation direction , its effectiveness is maximized only when the perturbation magnitude
+
+is dynamically scaled in response to the gradient landscape. The consistent superiority of the full model across all NLU and NLG tasks confirms that these two components are not merely additive but work in synergy, fulfilling the design goals of our framework.
+
+Comparison with Random Noise Perturbation
+
+To further validate that our performance gains stem from a principled optimization strategy rather than simple regularization, we compared LoRA-MGPO to LoRA augmented with undirected, isotropic ran-dom noise. The results in Table 4 are revealing: adding random noise provides only inconsistent and marginal benefits, and can even be detrimen-tal in some cases (e.g., HumanEval). In contrast, LoRA-MGPO yields consistent and significant im-provements across all tasks. This disparity highlights a fundamental differ-ence in mechanism. Random noise acts as a general regularizer by pushing parameters out of their im-mediate trajectory, which can occasionally help escape sharp minima by chance. However, the di-rection is arbitrary and uncorrelated with the loss landscape’s structure. Our momentum-guided per-turbation, conversely, is informed . It leverages the recent history of the optimization path—a strong indicator of relevant high-curvature directions—to perform a targeted exploration. This principled approach makes the search for flat minima non-stochastic and significantly more effective and reli-able than undirected noise injection.
+
+Computational Cost Analysis Finally, we an-alyzed the practical overhead of our method (Ta-ble 3). As expected, LoRA-MGPO operates with the same minimal number of trainable parameters as standard LoRA, making it vastly more memory-efficient than Full FT. In terms of training time, LoRA-MGPO introduces a modest and acceptable 654 overhead compared to vanilla LoRA (6h 52m vs. 5h 48m in our NLG setup). Given the significant performance improvements it delivers, this analy-sis confirms that LoRA-MGPO presents a highly favorable trade-off between computational cost and model performance, underscoring its practical via-bility.
+
+4 Related Work
+
+Parameter-Efficient Fine-Tuning (PEFT) The prohibitive computational and storage costs of full-parameter fine-tuning (Howard and Ruder, 2018; Devlin, 2018) have spurred the development of PEFT techniques for adapting large language mod-els (Houlsby et al., 2019; Ding et al., 2023). By selectively updating a small subset of parameters, PEFT methods can achieve performance competi-tive with full fine-tuning while being significantly more efficient (Han et al., 2024). Among the diverse PEFT strategies, Low-Rank Adaptation (LoRA) (Hu et al., 2021) has gained prominence for its simplicity and effectiveness. Recent works have enhanced LoRA along several directions. One line of work introduces architectural modifications ; for instance, DoRA (Liu et al., 2024) integrates learn-able magnitude vectors, while AdaLoRA (Zhang et al., 2023) dynamically allocates rank budgets. Another direction focuses on improving the train-ing process and initialization , such as adjusting scaling factors in rsLoRA (Kalajdzievski, 2023), us-ing separate learning rates in LoRA+ (Hayou et al., 2024), or refining initialization with PiSSA (Meng et al., 2024) and NLoRA (Guo et al., 2025). A third direction aims to improve the quality of the param-eter updates, for instance by alleviating training biases with BA-LoRA (Chang et al., 2024a) or by more closely aligning LoRA’s gradients with those of full fine-tuning, as seen in LoRA-GA (Wang et al., 2024a) and LoRA-Pro (Wang et al., 2024b). Additional work has further explored LoRA’s ap-plication in multi-task learning, such as (Liu et al., 2025b,a). Distinct from these approaches, our work focuses directly on the underlying optimization dy-namics. Rather than altering LoRA’s architecture or mimicking full fine-tuning gradients, we intro-duce a novel training framework to stabilize the optimization process itself.
+
+Optimization Stability in PEFT The training stability of PEFT methods, particularly LoRA, is a critical concern. Empirical studies have revealed that as LoRA’s rank increases, performance can de-grade after an initial improvement, a behavior anal-ogous to the double descent phenomenon (Belkin et al., 2019; Nakkiran et al., 2019). This insta-bility highlights the challenge of navigating high-dimensional and non-convex loss landscapes dur-ing fine-tuning. To promote smoother optimization and find flatter minima, Sharpness-Aware Mini-mization (SAM) (Foret et al., 2020) has been influ-ential. However, its requirement for dual gradient computations imposes a significant computational burden (Becker et al., 2024; Li et al., 2024b). More recent work has explored more efficient directional perturbation strategies. Momentum-guided meth-ods, for example, reuse optimizer momentum to avoid the extra gradient step, reducing computa-tional cost without sacrificing the directional guid-ance (Becker et al., 2024). Other techniques, such as applying an exponential moving average (EMA) to model weights, also contribute to stability by smoothing the trajectory of parameter updates (Wang et al., 2021). While these components— efficient perturbation and smoothing—are individu-ally effective, they are typically studied in isolation. This leaves a clear gap for a unified framework that synergistically combines these strategies to enhance both the efficiency and stability of PEFT. Our work, LoRA-MGPO, is designed to fill this gap.
+
+5 Conclusion
+
+In this work, we addressed the double descent phe-nomenon in Low-Rank Adaptation (LoRA), an in-stability that can affect the fine-tuning of large lan-guage models. We proposed LoRA-MGPO , an optimization framework that integrates Momentum-Guided Perturbation Optimization (MGPO). This method aims to find flatter minima by reusing optimizer momentum to guide weight perturba-tions, combined with an adaptive normalization scheme to improve robustness. Our experimental results across a range of natural language under-standing (NLU) and natural language generation (NLG) tasks show that LoRA-MGPO provides im-proved performance over standard LoRA and other common PEFT baselines. This improvement is re-flected in more stable convergence trajectories and reduced training instability. LoRA-MGPO offers a practical approach to overcoming some of the op-timization challenges in LoRA while maintaining its parameter efficiency. Future research may ex-plore extending this framework to other parameter-655 efficient methods or adapting it for different do-mains, such as vision and speech.
+
+Limitations
+
+First, LoRA-MGPO’s use of momentum vectors for perturbation directions assumes relatively stable optimizer dynamics, which might limit its effective-ness during early training stages or in the presence of highly non-stationary gradient conditions. Sec-ond, while the adaptive perturbation normalization via EMA-smoothed gradients improves robustness, its performance may be sensitive to sudden changes in gradient magnitude distributions, potentially re-quiring adjustments to the smoothing hyperparam-eters depending on the specific task.
+
+Ethics Statement
+
+Our research focuses on LoRA-MGPO, a general-purpose optimization algorithm designed to im-prove the stability of parameter-efficient fine-tuning (PEFT). The experiments use publicly avail-able, pre-trained models (LLaMA-2-7B, T5-base) and standard academic benchmarks. We acknowl-edge that these foundational models may inherit and potentially amplify societal biases present in their training data. The primary goal of this work is to provide a more reliable and resource-efficient tool for adapting and studying such models within the research community. By enhancing PEFT tech-niques, our work contributes to broader efforts aimed at reducing the computational costs involved in large-scale model adaptation.
+
+Acknowledgments
+
+This work is supported by the National Key Research and Development Program of China (No.2023YFF0905400), the National Natural Sci-ence Foundation of China (No.U2341229) and the Reform Commission Foundation of Jilin Province (No.2024C003).
+
+References
+
+Marlon Becker, Frederick Altrock, and Benjamin Risse. 2024. Momentum-sam: Sharpness aware minimiza-tion without computational overhead. arXiv preprint arXiv:2401.12033 .Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. 2019. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849–15854. Yupeng Chang, Yi Chang, and Yuan Wu. 2024a. Ba-lora: Bias-alleviating low-rank adaptation to mitigate catastrophic inheritance in large language models.
+
+arXiv preprint arXiv:2408.04556 .Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024b. A sur-vey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology ,15(3):1–45. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Ka-plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 .Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 .Jacob Devlin. 2018. Bert: Pre-training of deep bidi-rectional transformers for language understanding.
+
+arXiv preprint arXiv:1810.04805 .Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelli-gence , 5(3):220–235. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 .Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-aware min-imization for efficiently improving generalization.
+
+arXiv preprint arXiv:2010.01412 .Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 12799–12807. Chenlu Guo, Yuan Wu, and Yi Chang. 2025. Nlora: Nystr \" om-initiated low-rank adaptation for large language models. arXiv preprint arXiv:2502.14482 .Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey.
+
+arXiv preprint arXiv:2403.14608 .Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. Lora+: Efficient low rank adaptation of large models.
+
+Preprint , arXiv:2402.12354. 656 Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In In-ternational conference on machine learning , pages 2790–2799. PMLR. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification.
+
+arXiv preprint arXiv:1801.06146 .Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap-tation of large language models. arXiv preprint arXiv:2106.09685 .Damjan Kalajdzievski. 2023. A rank stabilization scaling factor for fine-tuning with lora. Preprint ,arXiv:2312.03732. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 .Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, and Xiaolin Huang. 2024a. Flat-lora: Low-rank adaption over a flat loss landscape. arXiv preprint arXiv:2409.14396 .Tao Li, Qinghua Tao, Weihao Yan, Zehao Lei, Yingwen Wu, Kun Fang, Mingzhen He, and Xiaolin Huang. 2024b. Revisiting random weight perturbation for efficiently improving generalization. arXiv preprint arXiv:2404.00357 .Jinda Liu, Yi Chang, and Yuan Wu. 2025a. R-lora: Random initialization of multi-head lora for multi-task learning. arXiv preprint arXiv:2502.15455 .Jinda Liu, Bo Cheng, Yi Chang, and Yuan Wu. 2025b. Align, don’t divide: Revisiting the lora architecture in multi-task learning. arXiv preprint arXiv:2508.05078 .Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. Preprint ,arXiv:2402.09353. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR .Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. Pissa: Principal singular values and singular vec-tors adaptation of large language models. Preprint ,arXiv:2404.02948. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2019. Deep double descent: Where bigger models and more data hurt. Preprint , arXiv:1912.02292. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the lim-its of transfer learning with a unified text-to-text transformer. Journal of machine learning research ,21(140):1–67. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda-tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations .Shaowen Wang, Linxi Yu, and Jian Li. 2024a. Lora-ga: Low-rank adaptation with gradient approximation.
+
+Preprint , arXiv:2407.05000. Yizhou Wang, Yue Kang, Can Qin, Huan Wang, Yi Xu, Yulun Zhang, and Yun Fu. 2021. Rethinking adam: A twofold exponential moving average approach. arXiv preprint arXiv:2106.11514 .Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. 2024b. Lora-pro: Are low-rank adapters properly optimized? Preprint , arXiv:2407.18242. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models.
+
+arXiv preprint arXiv:2206.07682 .Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In ICLR .Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR .Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adalora: Adap-tive budget allocation for parameter-efficient fine-tuning. Preprint , arXiv:2303.10512. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024a. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS .Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 657 2024b. OpenCodeInterpreter: Integrating code gen-eration with execution and refinement. In Findings of ACL .
+
+# Appendix
+
+Contents
+
+A Models and Datasets 11
+
+A.1 Details of Models . . . . . . . . . 11 A.2 Details of Datasets . . . . . . . . 11
+
+B Baselines and Implementation 11
+
+B.1 Baseline Methods . . . . . . . . . 11 B.2 Implementation Details . . . . . . 12 B.3 Hyperparameter Settings for Base-lines . . . . . . . . . . . . . . . . 12
+
+A Models and Datasets
+
+A.1 Details of Models
+
+In this work, we primarily utilize two pre-trained language models: LLaMA-2-7B and T5-base. • LLaMA-2-7B : A 7-billion parameter, decoder-only transformer model from the LLaMA-2 series, primarily used for genera-tion tasks. More details are available at its Hugging Face repository *.• T5-base : A 220-million parameter encoder-decoder transformer model, widely used for a variety of natural language understanding tasks. More details are available at its Hug-ging Face repository †.Our experiments were conducted using the im-plementations of these models provided by the Hug-ging Face Transformers library.
+
+A.2 Details of Datasets
+
+Table 5 summarizes the GLUE benchmark datasets (Wang et al., 2018). For our Natural Language Gen-eration (NLG) experiments, we used the following evaluation metrics: Accuracy for GSM8K; Pass@1 for HumanEval; and a score based on GPT-4 evalu-ation for MT-Bench.
+
+B Baselines and Implementation
+
+B.1 Baseline Methods
+
+Our study includes several baseline methods for a comprehensive comparison. Full Fine-Tuning
+
+serves as a strong performance benchmark. Vanilla
+
+> *https://huggingface.co/meta-llama/LLaMA-2-7B
+> †https://huggingface.co/t5-base
+
+658 Table 5: GLUE Benchmark Datasets and Evaluation Metrics
+
+> Dataset Task Type Classes Train Examples Metric Description
+> CoLA Acceptability 28.5k Matthews Corr. Grammatical acceptability SST-2 Sentiment 267k Accuracy Sentiment analysis MRPC Paraphrase 23.7k Accuracy/F1 Paraphrase detection MNLI NLI 3393k Accuracy Multi-genre NLI QNLI NLI/QA 2108k Accuracy QA/NLI converted from SQuAD
+
+LoRA (Hu et al., 2021) is our primary point of com-parison from the PEFT literature. We also compare against LoRA variants that introduce structural modifications (DoRA (Liu et al., 2024), AdaLoRA (Zhang et al., 2023)) and those that refine the training process or initialization (rsLoRA (Kala-jdzievski, 2023), LoRA+ (Hayou et al., 2024), PiSSA (Meng et al., 2024)). Finally, we include methods focused on gradient alignment (LoRA-GA (Wang et al., 2024a), LoRA-Pro (Wang et al., 2024b)).
+
+B.2 Implementation Details LoRA Configuration. As stated in the main text, LoRA adapters were applied to all linear layers within the transformer blocks for both LLaMA-2-7B and T5-base models.
+
+Initialization of MGPO. The implementation of our method requires an initial state for the momen-tum vector and the adaptive normalization factor. Following standard optimizer practice, the momen-tum ‘ m‘ is initialized to zeros. The adaptive nor-malization factor ‘ ¯g‘ is initialized using the L2-norm of the gradient computed in the first training step.
+
+Hyperparameters. Our method introduces two primary hyperparameters: the perturbation radius ‘ρ‘ and the EMA decay rate ‘ β‘. ‘ ρ‘ controls the magnitude of the weight perturbation, influencing the search for flatter minima. ‘ β‘ controls the tem-poral smoothing window for the adaptive normal-ization. The values used in our main experiments were effective across the evaluated tasks, as evi-denced by the strong performance reported in Sec-tion 3.
+
+B.3 Hyperparameter Settings for Baselines
+
+To ensure a fair and robust comparison, we adhered to the hyperparameter settings recommended in the original papers or official codebases of our baseline methods wherever possible. General settings, such as the learning rate schedule and batch size, were kept consistent across all methods as described in Section 3. Key method-specific hyperparameters are detailed below. • DoRA (Liu et al., 2024): We utilized the offi-cial implementation provided by the authors, maintaining its default configuration for the magnitude and directional components. • AdaLoRA (Zhang et al., 2023): We followed the setup from the original paper, with the rank budget dynamically allocated starting from a higher initial rank and pruned during training. • LoRA+ (Hayou et al., 2024): Following the authors’ recommendation, the learning rate for the LoRA matrix A was set to our default value ( 1 × 10 −4 for NLU, 2 × 10 −5 for NLG), while the learning rate for matrix B was set 16 times higher. • LoRA-GA and LoRA-Pro (Wang et al., 2024a,b): For these methods focused on gra-dient alignment, we used the hyperparameter settings as specified in their respective papers and official implementations to ensure a faith-ful comparison. For all other baselines, we used their standard, publicly available implementations without modifi-cation to their core components. 659