wassname/adapters_as_hypotheses
1
0
Fork 0
mirror of https://github.com/wassname/adapters_as_hypotheses.git synced 2026-08-11 11:14:21 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
eee889676ab511dcdf3efea9588b0c164f444cd5
adapters_as_hypotheses/adapters_vargdown.argdown
T

677 lines
70 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
===
title: Adapters as Representational Hypotheses -- Which Geometric Priors About Transformer Internals Hold Under Intervention?
author: Compiled from 38 PEFT methods plus adjacent evidence (2021--2026)
model:
mode: strict
===
// ══════════════════════════════════════════════════════════════════════
// MAIN THESIS
// Each adapter constrains how you update pretrained weights. When one
// architecture outperforms another under controlled conditions, the
// winner's structural assumptions are supported as a better description
// of the weight manifold. This is a natural experiment running across
// hundreds of papers, and almost nobody reads it as science about
// representations.
//
// Pseudocode for each adapter lives in adapters_as_hypotheses.md (README)
// Evidence files are in docs/ (frozen copies of papers as markdown)
// ══════════════════════════════════════════════════════════════════════
[Natural Manifold]: The model's own geometric structure -- its SVD basis
combined with orthogonal constraints and direction/magnitude decoupling --
defines the natural intervention manifold for transformer weight adaptation.
Adapters exploiting this structure generalize better, converge faster, and
reveal causally relevant properties of internals.
+ <SVD Basis Cluster>
+ <Orthogonal Preservation>
+ <Direction Magnitude Decoupling>
+ <Functional Architecture>
+ <Shared Structure Evidence>
+ <Prompt Tuning Orthogonal Paradigm>
- <Gain Control Sufficiency>
- <Rank Is Secondary>
- <SVD Is Linear Decomposition>
// ══════════════════════════════════════════════════════════════════════
// THEME 1: SVD AS THE NATURAL COORDINATE SYSTEM
// Methods: PiSSA, SVFT, SSVD, CLOVER, AntiPaSTO, AdaLoRA, PSOFT
// ══════════════════════════════════════════════════════════════════════
# SVD Basis
<SVD Basis Cluster>
(1) [SVD Init Wins]: PiSSA's SVD initialization consistently outperforms
LoRA's random initialization across 11 models and 13 tasks, showing
the principal components carry the adaptation signal. #observation
[Meng et al. 2024](https://arxiv.org/abs/2404.02948)
[evidence](docs/pissa_principal_singular_values_adaptation.md#L1-L20)
> PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices A and B with the principal components of the original matrix W, and put the remaining components into a residual matrix W res which is frozen during fine-tuning. Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance. Comparative experiments of PiSSA and LoRA across 11 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that **PiSSA consistently outperforms LoRA under identical experimental setups.** On the GSM8K benchmark, Gemma-7B fine-tuned with PiSSA achieves an accuracy of 77.7%, surpassing LoRA's 74.53% by 3.25%.
{reason: "NeurIPS 2024, 11 models up to 70B, controlled comparisons with identical architecture (same as LoRA)", credence: 0.88}
(2) [SVD Coefficients Suffice]: SVFT recovers 96% of full FT performance
by learning only sparse coefficients over the model's own singular
vectors -- no new directions needed. #observation
[Lingam et al. 2024](https://arxiv.org/abs/2405.19597)
[evidence](docs/svft_svd_coefficient_finetuning.md#L1-L30)
> We propose SVFT, a simple approach that fundamentally differs from existing methods: the structure imposed on delta-W depends on the specific weight matrix W. Specifically, SVFT updates W as a sparse combination of outer products of its singular vectors, training only the coefficients (scales) of these sparse combinations. (...) Extensive experiments on language and vision benchmarks show that **SVFT recovers up to 96% of full fine-tuning performance while training only 0.006 to 0.25% of parameters, outperforming existing methods that only recover up to 85% performance** using 0.03 to 0.8% of the trainable parameter budget.
{reason: "preprint under review; strong controlled comparison across language and vision; 96% of full FT with 0.006% params is striking but unpublished", credence: 0.74}
(3) [Asymmetric SVD Works]: SSVD shows that selectively rotating input-space
(right) singular vectors while preserving output-space (left) vectors
matches LoRA with far fewer params, and a convergence hierarchy emerges
at scale: SSVD > PiSSA > DoRA > LoRA. #observation
[Wang, Watanabe, Van hamme 2025](https://arxiv.org/abs/2509.02830)
[evidence](docs/ssvd_structured_svd_finetuning.md#L1-L40)
> This work presents the first comprehensive integration and benchmarking of these PEFT methods within ESPnet. We further introduce structured SVD-guided (SSVD) fine-tuning, which selectively rotates input-associated right singular vectors while keeping output-associated vectors fixed to preserve semantic mappings. This design enables robust domain adaptation with minimal trainable parameters and improved efficiency. (...) **SSVD achieves comparable performance with significantly fewer trainable parameters and higher efficiency than LoRA and SoTA LoRA variants**, approaching fully fine-tuned model performance.
{reason: "first comprehensive PEFT benchmark for speech across 0.1B-2B; convergence hierarchy consistent across scales", credence: 0.80}
----
(4) [SVD Is Natural Basis]: The weight matrix's own SVD provides the right
coordinate system for adaptation -- 6 of the top 7 scoring methods use it.
{reason: "PiSSA beats random init, SVFT shows coefficients alone suffice, SSVD hierarchy holds across scales; convergent evidence from independent groups", inference: 0.85}
+> [Natural Manifold]
<Joint SVD Evidence>
(1) [Cross-Layer Coupling]: CLOVER's joint SVD across Q-K and V-O attention
matrix pairs beats per-matrix LoRA by 7.6% on LLaMA-2-7B, validated
across 4 model families on 8 commonsense tasks. #observation
[Meng, Tang, Jiang, Zhang 2024](https://arxiv.org/abs/2411.17426)
[evidence](docs/clover_joint_svd.md#L258-L275)
> CLOVER consistently outperforms all other methods across all models and tasks. Specifically, on LLaMA 7B, CLOVER outperforms LoRA, DoRA, and PiSSA by 9%, 5.3%, and 1%, respectively. On LLaMA 13B, CLOVER outperforms these methods by 5.7%, 4.7%, and 0.8%. On LLaMA-2-7B, CLOVER surpasses LoRA, DoRA, HiRA, and PiSSA by 7.6%, 5.5%, 3.8%, and 0.7%. Even on LLaMA-3-8B, with fewer trainable parameters, CLOVER outperforms by 7.1%, 2.7%, 1.2%, and 0.6%.
{reason: "Under review (not yet published at venue); controlled comparison using DoRA's own hyperparams; 4 LLaMA variants x 8 tasks; beats PiSSA by 0.7% which is modest but consistent", credence: 0.72}
----
(2) [Functional SVD Matters]: Attention matrices cooperate functionally (Q-K
define pattern, V-O define extraction), and joint decomposition captures
this coupling.
{reason: "mechanistic intuition is sound; Q and K jointly specify attention weights so their updates should be coordinated. CLOVER paper also shows 82% of features project outside PiSSA's subspace, but CLOVER covers all directions. Single paper but multi-model validation.", inference: 0.72}
+> [Natural Manifold]
<PSOFT Synthesis>
(1) [OFT in SVD Coords]: PSOFT combines PiSSA's SVD initialization with
OFT's Cayley rotation, operating in the principal subspace. Beats LoRA
on 35 NLP and CV tasks with ~80% memory reduction vs OFT. #observation
[Wu et al. 2026](https://arxiv.org/abs/2505.11235)
[evidence](docs/psoft_principal_subspace_oft.md#L1-L50)
> PSOFT improves parameter and memory efficiency without compromising performance. Although GOFT and PSOFT have the same parameter counts, **PSOFT reduces memory usage by about 80% and avoids OOM issues.** It further achieves up to an 18x improvement in parameter efficiency over BOFT, OFTv2, and LoRA variants, attaining the best average performance across all baselines with the lowest memory footprint.
{reason: "ICLR 2026; 35 NLP+CV tasks across 4 models (DeBERTa, ViT, LLaMA-3.2-3B, LLaMA-3.1-8B); code available; bridges PiSSA and OFT with theoretical geometry guarantee", credence: 0.82}
----
(2) [SVD Plus Orthogonal Synergy]: The best performing adapters combine
SVD basis selection with orthogonal constraints, suggesting both priors
are complementary.
{reason: "PSOFT + AntiPaSTO + SSVD all combine SVD with Cayley rotations and all score well; convergent design choices", inference: 0.75}
+> [Natural Manifold]
<AntiPaSTO OOD Transfer>
(1) [OOD Via SVD Rotation]: AntiPaSTO achieves out-of-distribution transfer
from template persona pairs to real moral dilemmas (6.9x prompting baseline)
using Cayley rotations of SVD singular vectors. #observation
[Clark 2025](https://arxiv.org/abs/2601.07473)
[evidence](docs/antipasto_antiparallel_steering.md#L1-L40)
> Trained on 800 such pairs, our method transfers to 1,360 unseen moral dilemmas where honesty conflicts with other values, achieving **6.9x the Steering F1 of prompting** on Gemma-3-1B. We demonstrate two key advantages over prompting: OOD transfer (train on simple persona pairs, test on complex moral reasoning) and suppression bypass (steer when prompting triggers refusal).
{reason: "single-author, one trait (honesty), models up to 4B, high seed variance (std ~5-7); but OOD transfer is the strongest possible test of causal relevance", credence: 0.55}
(2) [SVD Causal Not Correlational]: If SVD rotation transfers OOD, the
geometric structure it exploits is causally relevant to behavior,
not merely correlated.
{reason: "OOD transfer is strong evidence of causality (rules out surface pattern matching), but narrow: 1 trait, 1 eval benchmark, 1 author. Independent replication would raise this considerably.", inference: 0.65}
+> [Natural Manifold]
<LoRA Baseline Limitation>
(1) [LoRA Ceiling]: Biderman et al. 2024 find LoRA underperforms full FT
on harder tasks and larger scale, suggesting the low-rank assumption
holds for surface adaptation but breaks where deep restructuring
is needed. #observation
[Biderman et al. 2024](https://arxiv.org/abs/2405.09673)
[evidence](docs/biderman_lora_limitations.md#L1-L50)
> Our results show that, in the standard low-rank settings, **LoRA substantially underperforms full finetuning.** Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. (...) Finally, we show that full finetuning learns perturbations with a rank that is **10-100x greater than typical LoRA configurations**, possibly explaining some of the reported gaps.
{reason: "TMLR 2024 (published, peer-reviewed); Llama-2-7B on code and math. In this local corpus, the paper is referenced in multiple docs (ROAD, DeLoRA, MiSS, CLOVER), mostly as supporting related-work evidence rather than direct replication.", credence: 0.80}
----
(2) [Random Init Wastes Steps]: LoRA's random initialization works (universal
baseline) but converges slower than SVD-informed init, suggesting random
subspaces are not special -- just adequate.
{reason: "PiSSA shows 3.25% gain on GSM8K just from better init; VeRA shows random projections work at all; together: random is adequate, SVD is better", inference: 0.70}
+> [Natural Manifold]
// ══════════════════════════════════════════════════════════════════════
// THEME 2: ORTHOGONAL TRANSFORMATIONS PRESERVE STRUCTURE
// Methods: OFT, BOFT, GOFT, HRA, ETHER, OFTv2, ROAD
// ══════════════════════════════════════════════════════════════════════
# Orthogonality
<Orthogonal Preservation>
(1) [Hyperspherical Energy Preserved]: OFT provably preserves pairwise
neuron angles (hyperspherical energy) during adaptation, yielding
stable finetuning and strong data efficiency (converges with 5%
of training data on controllable generation). #observation
[Qiu et al. 2023](https://arxiv.org/abs/2306.07280)
[evidence](docs/oft_orthogonal_finetuning.md#L1-L40)
> We introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, **OFT can provably preserve hyper-spherical energy which characterizes the pairwise neuron relationship on the unit hypersphere.** We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. (...) OFT is more sample-efficient in training and converges well with only 5% of the original dataset, while both ControlNet and LoRA cannot converge until 50% of the data is present.
{reason: "NeurIPS 2023; provable preservation is rare among PEFT methods; data efficiency claim is strong but limited to vision generation", credence: 0.82}
(2) [Butterfly Orthogonal Scales]: BOFT extends OFT with butterfly factorization,
reducing parameters from O(d^2) to O(d log d) while matching or exceeding
OFT performance. Outperforms full FT on vision tasks. #observation
[Liu et al. 2023](https://arxiv.org/abs/2311.06243)
[evidence](docs/boft_butterfly_orthogonal.md#L1-L30)
> We propose an efficient orthogonal parameterization using butterfly structures. (...) By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. (...) **BOFT outperforms current state-of-the-art methods by a considerable margin**, validating its superior parameter-efficiency and generalization ability.
{reason: "ICLR 2024; subsumes OFT theoretically; broad empirical validation across vision and language", credence: 0.80}
----
(3) [Orthogonality Prevents Catastrophic Forgetting]: Orthogonal constraints
bound deviation from pretrained weights, preventing the catastrophic
overwriting that plagues unconstrained methods at high learning rates.
{reason: "OFT converges on 5% data, ETHER is LR-robust, BOFT beats full FT on vision; consistent signal that bounded deviation helps. But strict orthogonality prevents magnitude changes, limiting it on tasks needing rescaling.", inference: 0.78}
+> [Natural Manifold]
<Givens and Householder>
(1) [Minimal Orthogonal Params]: GOFT decomposes orthogonal transformations
into O(d) Givens rotations (planar 2D rotations), achieving the most
parameter-efficient orthogonal parameterization. #observation
[Ma et al. 2024](https://arxiv.org/abs/2404.04316)
[evidence](docs/goft_givens_orthogonal.md#L1-L44)
> We first use O(d) Givens rotations to accomplish arbitrary orthogonal transformation in SO(d) with provable equivalence, reducing parameter complexity from O(d^2) to O(d). Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.
{reason: "ICML 2024; provable O(d) reduction from O(d^2); but advantage over BOFT is modest on benchmarks", credence: 0.70}
(2) [Orthogonal Equals Low-Rank]: HRA proves that a chain of r Householder
reflections is both orthogonal AND equivalent to a rank-r perturbation,
bridging the low-rank and orthogonal paradigms. #observation
[Yuan et al. 2024](https://arxiv.org/abs/2405.17484)
[evidence](docs/hra_householder_reflection.md#L1-L30)
> We bridge the gap between these two techniques, proposing a simple but effective adaptation method based on Householder reflections. Given a pre-trained model, our method fine-tunes its layers by multiplying each frozen weight matrix with an orthogonal matrix constructed by a chain of learnable Householder reflections (HRs). This HR-based orthogonal fine-tuning is equivalent to an adaptive low-rank adaptation. Moreover, we show that the orthogonality of the reflection planes corresponding to the HRs impacts the model capacity and regularity.
{reason: "theoretical bridge resolving 'low-rank vs orthogonal' dichotomy; merged into HuggingFace PEFT; empirical results competitive but not dominant", credence: 0.72}
----
(3) [Low-Rank Orthogonal Sweet Spot]: The effective adaptation is simultaneously
low-rank and approximately orthogonal -- small rotations on the Stiefel manifold.
{reason: "HRA's theoretical bridge is convincing; GOFT's practical efficiency confirms orthogonal methods can be made lean. But 'low-rank orthogonal' as the true operating point needs more direct testing.", inference: 0.65}
+> [Natural Manifold]
<ETHER Bounded Deviation>
(1) [Fixed-Strength Rotations Work]: ETHER demonstrates that bounded orthogonal
transformations (close to identity) achieve competitive adaptation while
preventing catastrophic forgetting, with high learning-rate robustness. #observation
[Bini, Girrbach, Akata 2024](https://arxiv.org/abs/2405.20271)
[evidence](docs/ether_orthogonal_steering.md#L1-L30)
> We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. By design, ETHER transformations require a minimal number of parameters, are less likely to deteriorate model performance, and exhibit robustness to hyperparameter and learning rate choices. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters (~10-100 times lower than LoRA or OFT) across multiple image synthesis and natural language tasks without exhaustive hyperparameter tuning.
{reason: "foundational for DeLoRA which won ICLR 2025; the bounded-deviation principle is well-validated; 10-100x fewer params than LoRA/OFT", credence: 0.72}
----
(2) [Minimal Intervention Often Suffices]: For behavioral steering, bounded
rotations suffice; for complex task adaptation, they hit a ceiling.
{reason: "ETHER works for many tasks but DeLoRA's extension (unbounding magnitude) does better, confirming the ceiling", inference: 0.70}
+> [Orthogonal Preservation]
// ══════════════════════════════════════════════════════════════════════
// THEME 3: DIRECTION/MAGNITUDE DECOUPLING
// Methods: DoRA, DeLoRA, ROAD, AntiPaSTO, SSVD
// ══════════════════════════════════════════════════════════════════════
# Decoupling
<Direction Magnitude Decoupling>
(1) [FT Decouples Naturally]: DoRA's weight decomposition analysis reveals
that full fine-tuning exhibits distinct magnitude vs. direction update
patterns that LoRA conflates. QDoRA slightly outperforms full FT. #observation
[Liu et al. 2024](https://arxiv.org/abs/2402.09353)
[evidence](docs/dora_weight_decomposed_lora.md#L1-L40)
> Drawing on Weight Normalization, which achieves faster convergence via improving the conditioning of the gradient with weight reparameterization, we introduce a novel weight decomposition analysis that initially reparameterizes model weights into magnitude and directional components, subsequently examining the changes in magnitude and direction introduced by LoRA and FT. **Our analysis reveals that LoRA and FT exhibit markedly distinct patterns of updates**, leading us to surmise that these variations mirror the learning capability of each method. (...) DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding.
{reason: "ICML 2024 (NVIDIA); thorough analysis + consistent improvements across 3 model families; widely adopted as LoRA replacement", credence: 0.85}
(2) [Bounded Deviation Robust]: DeLoRA normalizes each rank-1 component and
introduces scalar lambda controlling distance bound, yielding robustness
to LR and training duration where LoRA fails. #observation
[Bini, Girrbach, Akata 2025](https://arxiv.org/abs/2503.18225)
[evidence](docs/delora_decoupled_low_rank_adaptation.md#L1-L30)
> We propose Decoupled Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and scales learnable low-rank matrices. **By bounding the distance of the transformation, DeLoRA effectively decouples the angular learning from the adaptation strength, enhancing robustness without compromising performance.** Through evaluations on subject-driven image generation, natural language understanding, and instruction tuning, we show that DeLoRA matches or surpasses performance of competing PEFT methods, while exhibiting stronger robustness.
{reason: "ICLR 2025; same group as ETHER so a coherent research line; robustness claim well-supported by ablations", credence: 0.82}
----
(3) [Decoupling Is Sound Engineering]: Separating what-to-change (direction)
from how-much (magnitude) consistently improves training stability and
often improves final performance.
{reason: "DoRA beats LoRA everywhere, DeLoRA adds robustness, ROAD provides cleanest per-pair formulation; 3 independent groups converge on this. Whether it's fundamental or just good optimization is unclear.", inference: 0.80}
+> [Natural Manifold]
<ROAD Explicit Decoupling>
(1) [Angle Plus Magnitude Per Pair]: ROAD parameterizes adaptation as learned
rotation angle theta plus magnitude alpha per activation pair, the cleanest
direction/strength separation. #observation
[Liao, Monz 2024](https://arxiv.org/abs/2409.00119)
[evidence](docs/road_rotary_adaptation.md#L1-L30)
> We introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with < 0.1% trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.
{reason: "clean formulation connecting to RoPE; supports mixed-adapter batching which is unique; limited vision/generation benchmarks", credence: 0.62}
----
(2) [Continuous Steering Knob]: The explicit magnitude parameter provides
a continuous interpolation between identity and full adaptation, ideal
for controllable steering.
{reason: "mathematically clean; alpha as continuous knob is the most interpretable version of direction/strength decoupling", inference: 0.65}
+> [Direction Magnitude Decoupling]
// ══════════════════════════════════════════════════════════════════════
// THEME 4: GAIN CONTROL -- ADAPTATION AS RESCALING
// Methods: IA3, VeRA, LN Tuning
// Evidence AGAINST the full thesis: if mere rescaling works, the
// geometric structure may not be needed.
// ══════════════════════════════════════════════════════════════════════
# Gain Control
<Gain Control Sufficiency>
(1) [Scaling Vectors Competitive]: IA3 achieves competitive adaptation using
only d-dimensional scaling vectors at key, value, and FFN positions --
no new weight directions needed. T0-3B with IA3 outperforms GPT-3 175B
with ICL. #observation
[Liu et al. 2022](https://arxiv.org/abs/2205.05638)
[evidence](docs/ia3_few_shot_peft.md#L1-L30)
> We introduce (IA)3 that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. (...) We validate the effectiveness of T-Few on completely unseen tasks by applying it to the RAFT benchmark, **attaining super-human performance for the first time and outperforming the state-of-the-art by 6% absolute.**
{reason: "strong result on RAFT; super-human is notable; but T5-family specific and scaling-only has known expressivity ceiling", credence: 0.72}
(2) [Random Projections Plus Scaling]: VeRA matches LoRA with 10x fewer
trainable parameters using only per-layer scaling of shared frozen
random matrices. #observation
[Kopiczko et al. 2023](https://arxiv.org/abs/2310.11454)
[evidence](docs/vera_vector_random_matrix_adaptation.md#L1-L30)
> We present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. **It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead.**
{reason: "ICLR 2024; surprising result that random projections suffice with per-layer gating; JL-lemma argument is plausible", credence: 0.75}
----
(3) [Rescaling Gets 60-80%]: A surprisingly large fraction of task adaptation
is reweighting existing features, suggesting the model already computes
the right features -- the bottleneck is channel selection.
{reason: "IA3 and VeRA perform well on standard benchmarks but both hit expressivity ceilings on hard tasks. This means rescaling handles easy adaptation but not deep restructuring.", inference: 0.45}
-> [Natural Manifold]
// The contra is weak (0.45) because gain control works on easy tasks
// but fails precisely where geometric structure matters most. The
// ceiling IS the evidence for the manifold hypothesis.
// ══════════════════════════════════════════════════════════════════════
// THEME 5: RANK IS SECONDARY TO BASIS CHOICE
// Methods: LoRA, RandLoRA, C3A, FourierFT, LoHa, LoKr
// ══════════════════════════════════════════════════════════════════════
# Rank Debate
<Rank Is Secondary>
(1) [Full Rank Helps Hard Tasks]: RandLoRA achieves full-rank updates by
summing scaled random rank-r bases, and its loss landscape is closer
to full FT's than LoRA's. #observation
[Albert et al. 2025](https://arxiv.org/abs/2502.00987)
[evidence](docs/randlora_random_matrix.md#L1-L30)
> This paper aims to answer this question by introducing RandLoRA, a parameter-efficient method that performs full-rank updates using a learned linear combinations of low-rank, non-trainable random matrices. (...) Our findings reveal that full-rank updates are beneficial across vision and language tasks individually, and even more so for vision-language tasks, where RandLoRA significantly reduces -- and sometimes eliminates -- the performance gap between standard fine-tuning and LoRA, demonstrating its efficacy.
{reason: "ICLR 2025; eliminates FT-LoRA gap on vision-language; loss landscape analysis compelling; improvement task-dependent", credence: 0.72}
(2) [Spectral Sparsity]: FourierFT achieves comparable results via sparse
Fourier coefficients, suggesting weight updates are spectrally smooth. #observation
[Gao et al. 2024](https://arxiv.org/abs/2405.03003)
[evidence](docs/fourierft_spectral.md#L1-L30)
> We introduce FourierFT, which treats DeltaW as a matrix in the spatial domain and learns only a small fraction of its spectral coefficients. With the trained spectral coefficients, we implement the inverse discrete Fourier transform to recover DeltaW. Empirically, our FourierFT method shows comparable or better performance with fewer parameters than LoRA on various tasks, including natural language understanding, natural language generation, instruction tuning, and image classification. For example, when performing instruction tuning on the LLaMA2-7B model, FourierFT surpasses LoRA with only 0.064M trainable parameters, compared to LoRA's 33.5M.
{reason: "ICML 2024; 500x fewer params than LoRA on LLaMA2 instruction tuning; spectral sparsity is structurally different from SVD", credence: 0.68}
----
(3) [Basis Matters More Than Rank]: PiSSA's well-chosen rank-r subspace
outperforms RandLoRA's full-rank random subspace, suggesting which
directions you update matters more than how many.
{reason: "PiSSA at rank r beats LoRA at same rank; RandLoRA at full rank sometimes beats PiSSA but not always. The basis selection (SVD vs random) appears to be the primary factor.", inference: 0.40}
-> [Natural Manifold]
// Weak contra (0.40): the rank debate slightly undermines the thesis
// because if full rank sometimes wins, the SVD manifold might not
// capture everything. But basis choice dominates rank choice.
<Alternative Parameterizations>
(1) [Circulant Structure]: C3A achieves full-rank updates via circulant
matrices (FFT-computable), competitive on GLUE and instruction tuning. #observation
[Chen et al. 2025](https://arxiv.org/abs/2407.19342)
[evidence](docs/c3a_circular_convolution.md#L1-L30)
> We introduce Circular Convolution Adaptation (C3A), which not only achieves high-rank adaptation with enhanced performance but also excels in both computational power and memory utilization. **Extensive experiments demonstrate that C3A consistently outperforms LoRA and its variants across various fine-tuning tasks.** Our code is available at Hugging Face PEFT.
{reason: "ACL 2025; merged into HuggingFace PEFT; circulant assumption is a strong inductive bias; competitive but not dominant on all benchmarks", credence: 0.62}
(2) [Hadamard Products]: LoHa's element-wise product of two low-rank
decompositions captures pairwise feature interactions that additive
low-rank misses. #observation
[Nam, Moon, Oh 2022](https://arxiv.org/abs/2108.06098)
[evidence](docs/loha_hadamard_product.md#L1-L30)
> We propose a communication-efficient parameterization, FedPara, which re-parameterizes weight parameters of layers using low-rank weights followed by the Hadamard product. **Compared to the conventional low-rank parameterization, our FedPara method is not restricted to low-rank constraints, and thereby it has a far larger capacity.** This property enables to achieve comparable performance while requiring 3 to 10 times lower communication costs than the model with the original layers.
{reason: "ICLR 2022; effective for vision/diffusion via LyCORIS ecosystem; Hadamard product overcomes low-rank bottleneck; niche for LLMs", credence: 0.55}
(3) [Kronecker Products]: LoKr's tensor product structure enables compact
adaptation across convolutional and linear layers, with systematic
evaluation across fine-tuning methodologies. #observation
[Yeh et al. 2024](https://arxiv.org/abs/2309.14859)
[evidence](docs/lokr_lycor.md#L1-L30)
> This paper introduces LyCORIS (Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion), an open-source library that offers a wide selection of fine-tuning methodologies for Stable Diffusion. Furthermore, we present **a thorough framework for the systematic assessment of varied fine-tuning techniques.** This framework employs a diverse suite of metrics and delves into multiple facets of fine-tuning.
{reason: "systematic LyCORIS benchmark; Kronecker/spatial structure assumption less natural for attention weights; mainly validated on diffusion models", credence: 0.50}
----
(4) [Multiple Structures Work]: Multiple non-SVD parameterizations (circulant,
Hadamard, Kronecker, Fourier) achieve competitive results, suggesting
SVD is not the only viable basis.
{reason: "variety of working approaches, but none consistently beats SVD methods on controlled benchmarks; SVD has strongest cross-domain evidence", inference: 0.50}
-> [Natural Manifold]
// ══════════════════════════════════════════════════════════════════════
// THEME 6: FUNCTIONAL ARCHITECTURE
// Methods: CLOVER, ReFT, X-LoRA, Prompt Tuning
// ══════════════════════════════════════════════════════════════════════
# Functional Architecture
<Functional Architecture>
(1) [Activation Intervention]: ReFT achieves 15-65x more parameter efficiency
than LoRA by intervening on hidden states at specific (layer, position)
sites rather than on weights at all. #observation
[Wu, Arora et al. 2024](https://arxiv.org/abs/2404.03592)
[evidence](docs/reft_representation_finetuning.md#L1-L40)
> ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT). (...) **LoReFT uses 15x-65x times fewer parameters while achieving state-of-the-art performance on commonsense reasoning, instruction-following, and natural language understanding** against the strongest PEFTs.
{reason: "Stanford NLP; widely cited; strong on instruction following and math. Fundamentally different paradigm (activation vs weight) so comparison not apples-to-apples", credence: 0.72}
(2) [Compositional Routing]: X-LoRA dynamically routes through expert LoRA
adapters, achieving better composite task performance than individual
adapters. #assumption
{reason: "compositional multi-task; routing adds overhead; more suited to deployment than single-task science", credence: 0.50}
----
(3) [Beyond Individual Matrices]: Respecting the model's functional architecture
(which matrices cooperate, where activations matter) provides additional
efficiency gains beyond weight-matrix-level priors.
{reason: "CLOVER (joint SVD) beats per-matrix methods; ReFT (activation intervention) is 15-65x more efficient; both exploit structure LoRA ignores. But evidence is early -- 1-2 papers each.", inference: 0.60}
+> [Natural Manifold]
// ══════════════════════════════════════════════════════════════════════
// THEME 7: SHARED STRUCTURE AND COMPRESSION
// Methods: MiSS, VBLoRA, SHiRA, AdaLoRA
// ══════════════════════════════════════════════════════════════════════
# Compression
<Shared Structure Evidence>
(1) [Adaptive Rank Allocation]: AdaLoRA's SVD-parameterized pruning shows
that different layers need different ranks -- query/value projections
and early/late layers are most important. #observation
[Zhang et al. 2023](https://arxiv.org/abs/2303.10512)
[evidence](docs/adalora_adaptive_budget.md#L1-L30)
> We propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition. Such a novel approach allows us to effectively prune the singular values of unimportant updates. (...) **Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings.**
{reason: "ICLR 2023; validates SVD parameterization and adaptive rank; modest gains over LoRA add complexity", credence: 0.70}
(2) [Vector Bank Compression]: VBLoRA uses 0.4% of LoRA's parameters via
shared atomic vectors, suggesting adapter diversity across layers is
much lower than assumed. #observation
[Li, Han, Ji 2024](https://arxiv.org/abs/2405.15179)
[evidence](docs/vblora_vector_bank.md#L1-L30)
> Our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable top-k admixture module. (...) When fine-tuning the Llama2-13B model, **VB-LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results.** This method has been merged into the Hugging Face PEFT package.
{reason: "NeurIPS 2024; merged into HuggingFace PEFT; extreme compression is remarkable; implies shared structure across layers", credence: 0.72}
(3) [Sparse High-Rank]: SHiRA directly fine-tunes 1-2% of base weights
selected by importance, achieving full rank without low-rank
bottleneck. #observation
[Bhardwaj et al. 2024](https://arxiv.org/abs/2406.13175)
[evidence](docs/shira_sparse_high_rank.md#L1-L40)
> We propose SHiRA, a new high rank adapter paradigm to demonstrate that **changing as few as 1-2% parameters of the original network is sufficient for adaptation.** (...) We conduct extensive experiments on LLMs (LLaMA-7B, LLaMAv2-7B) and LVMs (Stable Diffusion, SDXL) where we demonstrate that SHiRA significantly outperforms LoRA on both single- and multi-adapter tasks. On LLMs, we show that SHiRA achieves up to 2.7% better accuracy than LoRA on commonsense reasoning.
{reason: "NeurIPS 2024; Qualcomm; multi-adapter fusion is unique angle; extensive LLM+LVM experiments; challenges low-rank assumption", credence: 0.62}
----
(4) [Compression Signals Shared Structure]: The extreme compressibility of
adapter weights (VBLoRA 0.4%, SVFT 0.006%) suggests weight updates
share structure that priors can exploit.
{reason: "multiple methods achieve extreme compression suggesting high redundancy in naive parameterizations; supports the thesis that the right basis captures this shared structure", inference: 0.60}
+> [Natural Manifold]
// Claude: ══════════════════════════════════════════════════════════════
// Claude: THEME 8: FLATNESS AND CURVATURE ARE SPACE-SPECIFIC
// Claude: Flatness-seeking family: Flat-LoRA, FMLoRA, Bi-LoRA, MGPO;
// Claude: BAR is the SAM-balancedness surrogate.
// Claude: Adjacent evidence: CrispEdit, DISAM, SGD subspaces, sharpness
// Claude: counterexamples, TRAM, and steering generalization.
// Claude: ══════════════════════════════════════════════════════════════
# Curvature
[Curvature Is Space-Specific]: Flatness and curvature are related, but the
differentiated variable must be named. Perturbation-based flatness seeking,
explicit Hessian or Gauss-Newton eigenspace constraints, and function or
context curvature make different predictions about generalization.
+ <Flatness Seeking Adapters>
+ <Explicit Low Loss Curvature>
+ <Curvature Direction Ambiguity>
+ <Flatness Under Domain Shift>
+ <Flatness OOD Limits>
+ <Function Space Curvature>
<Flatness Seeking Adapters>
(1) [Flat-LoRA Smooths Merged Weights]: Flat-LoRA trains low-rank factors
under random perturbations of the merged weight matrix so the solution
is flat in the full task-loss landscape. #observation
[Li et al. 2024](https://arxiv.org/abs/2409.14396)
[evidence](docs/flat_lora_full_parameter_flatness.md#L1-L35)
> Despite recent progress in improving LoRA’s performance, the relationship between the LoRA optimization space and the full parameter space is often overlooked. **A solution that appears flat in the loss landscape of the LoRA space may still exhibit sharp directions in the full parameter space, potentially compromising generalization. We introduce Flat-LoRA, which aims to identify a low-rank adaptation situated in a flat region of the full parameter space.** Instead of adopting the well-established sharpness-aware minimization approach, which incurs significant computation and memory overheads, we employ a Bayesian expectation loss objective to preserve training efficiency. Further, we design a refined random perturbation generation strategy for improved performance and carefully manage memory overhead using random seeds.
{reason: "ICML 2025; authors' abstract for their own method; the main perturbation scheme covers adapted linear matrices, while all-layer perturbation is reported separately in the appendix; no independent replication found", credence: 0.78}
(2) [BAR Makes SAM Balancedness Explicit]: BAR replaces SAM's extra
adversarial step with a factor-norm regularizer derived from the claimed
implicit balancedness dynamics of SAM. #observation
[Li, Zhang, He 2024](https://arxiv.org/abs/2410.14802)
[evidence](docs/bar_balancedness_aware_regularization.md#L15-L29)
> Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization of SAM for scale-invariant problems involving two groups of variables. **Instead of focusing on commonly used sharpness, this work introduces a concept termed balancedness, defined as the difference between the squared norm of two variables.** This allows us to depict richer global behaviors of SAM. In particular, our theoretical and empirical findings reveal that i) SAM promotes balancedness; and ii) the regularization on balancedness is data-responsive – outliers have stronger impact. The latter coincides with empirical observations that SAM outperforms SGD in the presence of outliers. Leveraging the implicit regularization, we develop a resource-efficient SAM variant, balancedness-aware regularization (BAR), tailored for scale-invariant problems such as finetuning language models with LoRA.
{reason: "NeurIPS 2024; authors' abstract and theoretical framing; BAR is reused as a baseline by Flat-LoRA, FMLoRA, and Bi-LoRA, but its tests are few-shot or ordinary held-out rather than controlled OOD", credence: 0.82}
(3) [FMLoRA Transfers Full-Space Perturbations]: FMLoRA reconstructs a
full-weight SAM direction from LoRA gradients and transfers it into one
factor; EFMLoRA reuses an EMA perturbation for near-LoRA cost. #observation
[Deng et al. 2025](https://arxiv.org/abs/2508.00522)
[evidence](docs/fmlora_flat_minima_lora.md#L8-L28)
> Little research explores the correlation between the expressive ability and generalization ability of the low-rank adaptation (LoRA). Sharpness-Aware Minimization (SAM) improves model generalization for both Convolutional Neural Networks (CNNs) and Transformers by encouraging convergence to locally flat minima. However, the connection between sharpness and generalization has not been fully explored for LoRA due to the lack of tools to either empirically seek flat minima or develop theoretical methods. **In this work, we propose Flat Minima LoRA (FMLoRA) and its efficient version i.e., EFMLoRA, to seek flat minima for LoRA. Concretely, we theoretically demonstrate that perturbations in the full parameter space can be transferred to the low-rank subspace.** This approach eliminates the potential interference introduced by perturbations across multiple matrices in the low-rank subspace. Our extensive experiments on large language models and vision-language models demonstrate that EFMLoRA achieves optimize efficiency comparable to that of LoRA while simultaneously attaining comparable or even better performance. For example, on the GLUE dataset with RoBERTa-large, EFMLoRA outperforms LoRA and full fine-tuning by 1.0% and 0.5% on average, respectively. On vision-language models e.g., Qwen-VL-Chat, there are performance improvements of 1.5% and 1.0% on the SQA and VizWiz datasets, respectively. These empirical results also verify that the generalization of LoRA is closely related to sharpness, which is omitted by previous methods.
{reason: "AAAI 2026; authors' abstract and own benchmark comparisons; reported gains cover several architectures, but the claimed distribution shift is mostly few-shot or ordinary downstream evaluation", credence: 0.72}
(4) [Bi-LoRA Separates Descent and Ascent]: Bi-LoRA trains a primary LoRA
branch for task descent and an auxiliary low-rank branch for adversarial
ascent, then discards the auxiliary branch. #observation
[Liu et al. 2025](https://arxiv.org/abs/2508.19564)
[evidence](docs/bi_lora_sharpness_aware.md#L16-L20)
> Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large pre-trained models. Yet LoRA can face generalization challenges. One promising way to improve the generalization is Sharpness-Aware Minimization (SAM), which has proven effective for small-scale training scenarios. **In this paper, we propose Bi-directional Lo w-R ank A daptation (Bi-LoRA), which introduces an auxiliary adversarial LoRA module. This design explicitly decouples sharpness optimization, handled by the auxiliary module, from task adaptation, performed by the primary module.** Such a separation yields two key benefits. First, it transforms SAM’s sequential computation of adversarial perturbation and gradient descent into a parallel form, which roughly halves the time and conquers the main obstacle of applying SAM in LoRA. Second, it provides perturbations from the auxiliary module that do not collapse into the restricted optimization subspace of the primary module, enabling broader sharpness exploration and flatter minima. Bi-LoRA simultaneously achieves both efficiency and effectiveness within a single framework, as validated by extensive experiments across diverse architectures and tasks.
{reason: "ICLR 2026; authors' method section; same-paper comparisons mostly favor Bi-LoRA over LoRA, LoRA-SAM, and Flat-LoRA on cross-dataset LLM tests, without a controlled domain-generalization protocol", credence: 0.76}
(5) [MGPO Reuses Optimizer Momentum]: LoRA-MGPO perturbs trainable LoRA
factors along the optimizer's previous first moment and normalizes the
radius with an EMA of gradient norms. #observation
[Chang et al. 2025](https://aclanthology.org/2025.findings-emnlp.34/)
[evidence](docs/lora_mgpo_momentum_perturbation.md#L28-L34)
> Parameter-efficient fine-tuning (PEFT), partic-ularly Low-Rank Adaptation (LoRA), adapts large language models (LLMs) by training only a small fraction of parameters. However, as the rank of the low-rank matrices used for adap-tation increases, LoRA often exhibits an un-stable "double descent" phenomenon, charac-terized by transient divergence in the training loss, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima. **To address this, we introduce LoRA-MGPO , a framework that incorporates Momentum-Guided Pertur-bation Optimization (MGPO). MGPO stabi-lizes training dynamics by mitigating the dou-ble descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer’s state, thus avoiding dual gra-dient computations.** Additionally, an adaptive normalization scheme scales the magnitude of perturbations based on an exponential mov-ing average (EMA) of gradient norms, further enhancing stability. While EMA controls the magnitude of the perturbations, MGPO guides their direction, ensuring a more stable opti-mization trajectory.
{reason: "Findings EMNLP 2025; authors' method section and released code; improvements are conventional NLU and cross-dataset NLG results, while the link to flat minima is indirect", credence: 0.64}
----
(6) [Flatness Seeking Is Indirect Curvature Control]: Flat-LoRA, FMLoRA,
Bi-LoRA, and MGPO seek low loss in a finite weight neighborhood; BAR
instead isolates factor balancedness as a proposed implicit effect of
SAM. Near a stationary point the four neighborhood objectives are
sensitive to the largest Hessian eigenvalues, but none of the five
computes a Hessian eigenspace or restricts updates to low-eigenvalue
directions.
{reason: "local Taylor expansion links neighborhood sharpness to curvature; BAR implements a derived surrogate, and the other four implement perturbation objectives rather than spectral projection", inference: 0.90}
+> [Curvature Is Space-Specific]
<Explicit Low Loss Curvature>
(1) [CrispEdit Protects Capability Loss]: CrispEdit projects model-editing
updates into the low-curvature subspace of a separate capability loss,
estimated with Gauss-Newton curvature and K-FAC. #observation
[Ikram et al. 2026](https://arxiv.org/abs/2602.15823)
[evidence](docs/crispedit_low_curvature_editing.md#L39-L46)
> We present CrispEdit, a scalable and principled second-order editing algorithm that treats capability preservation as an explicit constraint, unifying and generalizing several existing editing approaches. **CrispEdit formulates editing as constrained optimization and enforces the constraint by projecting edit updates onto the low-curvature subspace of the capability-loss landscape.** At the crux of CrispEdit is expressing capability constraint via Bregman divergence, whose quadratic form yields the Gauss–Newton Hessian exactly and even when the base model is not trained to convergence. We make this second-order procedure efficient at the LLM scale using Kronecker-factored approximate curvature (K-FAC) and a novel matrix-free projector that exploits Kronecker structure to avoid constructing massive projection matrices.
{reason: "May 2026 preprint; authors' abstract for a model-editing method rather than an adapter; capability is evaluated on a designated reference set", credence: 0.66}
----
(2) [Low Curvature Protects the Chosen Loss]: CrispEdit's spectral projector
protects a designated capability loss. This is the closest prior art to
an explicit low-loss-curvature adapter, but it does not make curvature a
detector of the target behavior's semantic depth.
{reason: "CrispEdit names the protected scalar loss and reference set; projecting an update away from its stiff directions says what is preserved, not why the edited behavior generalizes", inference: 0.84}
+> [Curvature Is Space-Specific]
<Curvature Direction Ambiguity>
(1) [Low-Curvature Complement Can Train]: Projecting SGD updates out of the
dominant training-loss Hessian subspace can preserve training progress.
#observation
[Song, Ahn, Yun 2024](https://arxiv.org/abs/2405.16002)
[evidence](docs/sgd_low_curvature_subspace.md#L31-L40)
> Given this alignment, this paper explores whether neural networks can be trained within the dominant subspace, which, if feasible, could lead to more efficient training methods. **Our primary observation is that when the SGD update is projected onto the dominant subspace, the training loss does not decrease further. This suggests that the observed alignment between the gradient and the dominant subspace is spurious. Surprisingly, projecting out the dominant subspace proves to be just as effective as the original update, despite removing the majority of the original update component.** We observe similar behavior across practical setups, including the large learning rate regime (also known as Edge of Stability), Sharpness-Aware Minimization, momentum, and adaptive optimizers.
{reason: "ICLR 2025 empirical result on training-loss Hessians; main experiments use small supervised vision and text-classification settings", credence: 0.74}
(2) [High-Curvature Dimensions Can Drive Improvement]: ES probes on LLM
reward landscapes instead attribute improvement to a small stiff
subspace. #observation
[Liang et al. 2026](https://arxiv.org/abs/2602.00170)
[evidence](docs/blessing_dimensionality_variance_curvature.md#L20-L30)
> We also observe a second seemingly separate phenomenon: under fixed hyperparameters, the stochastic fine-tuning reward often rises, peaks, and then degrades in both ES and GRPO. **We argue that both effects reflect a shared geometric property of fine-tuning landscapes: they are low-dimensional in curvature. A small set of high-curvature dimensions dominates improvement, producing (i) heterogeneous time scales that yield rise–then–decay under fixed stochasticity, as captured by a minimal quadratic stochastic-ascent model, and (ii) degenerate improving updates, where many random perturbations share similar components along these directions.** Using ES as a geometric probe on fine-tuning reward landscapes of GSM8K, ARC-C, and WinoGrande across Qwen2.5-Instruct models (0.5B–7B), we show that reward-improving perturbations remain empirically accessible with small populations across scales.
{reason: "February 2026 preprint; authors' interpretation of ES reward probes on Qwen2.5-Instruct 0.5B-7B, not a direct high-versus-low projection intervention", credence: 0.55}
----
(3) [No Universal Useful-Curvature Side]: These results are not direct
replications because they use different objectives, scales, and
optimizers. Together they make a universal high- or low-curvature
prescription improbable.
{reason: "the apparent conflict is largely regime-dependent; the safe conclusion is that the scalar objective and optimizer must be specified", inference: 0.72}
+> [Curvature Is Space-Specific]
<Flatness Under Domain Shift>
(1) [Plain SAM Can Fail Under Domain Shift]: DISAM's authors report that
ordinary SAM can underperform ERM when source domains converge at
different rates. #observation
[Zhang et al. 2024](https://arxiv.org/abs/2405.18861)
[evidence](docs/disam_domain_shift_sharpness.md#L46-L58)
> Nonetheless, these methods cannot solve generalizability scenarios that involve training data of multiple domains with domain shifts like Domain Generalization (DG) (Ben-David et al., 2010; Li et al., 2017). **In this study, we observed that sometimes SAM even has a detrimental impact in situations where there exist domain shifts across multiple domains as shown in Figure 1(1(a)).** While a few studies have incorporated SAM-based methods in domain generalization tasks (Wang et al., 2023b; Foret et al., 2021), they cannot ensure consistent improvements in generalizability during domain shifts due to their reliance on the i.i.d assumption. Upon a thorough analysis of the behavior of SAM under domain shifts, we discovered that the degradation of the training process caused by SAM from the disparity in convergence degree among different domains as shown in Figure 1(1(a)). Given the inconsistency in the degree and direction of convergence among different domains during training (Arjovsky et al., 2019; Krueger et al., 2021), the straightforward application of SAM for perturbations may not only disrupt convergence but also generate perturbation directions that are not adequately coherent to the geometric characteristics of the entire loss landscape.
{reason: "ICLR 2024; authors' diagnosis from DomainBed experiments; this directly limits the inference from ordinary SAM or flat-adapter gains to OOD transfer", credence: 0.83}
(2) [DISAM Uses a Genuine OOD Protocol]: DISAM calibrates the SAM
perturbation using source-domain loss variance and evaluates on domains
excluded from training. #observation
[Zhang et al. 2024](https://arxiv.org/abs/2405.18861)
[evidence](docs/disam_domain_shift_sharpness.md#L1611-L1627)
> We evaluate DISAM on five datasets PACS (Li et al., 2017), VLCS (Fang et al., 2013) OfficeHome (Venkateswara et al., 2017), TerraIncognita (Beery et al., 2018) (abbreviated as TerraInc), and DomainNet (Peng et al., 2019), following the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021). For fair comparison, we adhere to the training and evaluation protocol outlined in DomainBed. Evaluation. **The standard leave-one-domain-out strategy is used in evaluation. Specially, the unseen domain is used to evaluate the out-of-domain generalization, and the validation sets of source domains are used to measure the in-domain generalization, while the others are used for training.** Final accuracy is averaged across all settings, and the performance is the averaging over three trials with distinct random seeds. Detailed statistics for each case of all datasets are provided in Appendix C.
[evidence](docs/disam_domain_shift_sharpness.md#L2186-L2192)
> We propose incorporating our domain-inspired adaptive adjustment into three SAM-based methods: SAM (Foret et al., 2021), GSAM (Zhuang et al., 2022), and SAGM (Wang et al., 2023b) on five datasets of DomainBed with ResNet50 backbone. Table 1 shows that our Domain-Inspired SAM can mitigate issues arising from SAM’s training under domain shifts, by comparing averaged in-domain and out-of-domain performance of leading SAM methods, with and without DISAM. In-domain results show domain-inspired perturbations enhance convergence, especially on the TerraInc dataset with substantial domain gaps. **In Out-of-domain results, DISAM consistently improves generalization, with average improvements of 1.9% for SAM, 1.7% for GSAM, and 1.9% for SAGM.** Notably, SAM performs well when the performance gap between in-domain and out-of-domain is small but worse than ERM on datasets like TerraInc with large gaps, which proves our analysis of SAM’s shortcomings under domain shifts. This shows SAM’s inconsistent convergence for large domain shifts, which DISAM addresses by incorporating domain-inspired adaptive adjustments based on domain-level convergence degree. Incorporating CORAL constraints, a recognized effective traditional DG method on DomainBed improves SAGM with DISAM and sets new state-of-the-art results on all settings.
{reason: "ICLR 2024; standard leave-one-domain-out DomainBed protocol over five datasets, three trials, plus CLIP prompt-tuning experiments; author-reported results but materially stronger OOD evidence than downstream test accuracy", credence: 0.86}
----
(3) [OOD Flatness Needs Shift Information]: DISAM is not a LoRA adapter,
but it is strong evidence against treating generic weight-space flatness
as domain-general by default. Its gains require domain labels and a
domain-loss variance term when constructing the perturbation.
{reason: "plain SAM sometimes loses to ERM while domain-calibrated SAM improves leave-one-domain-out accuracy; the added domain information, not flatness alone, distinguishes the methods", inference: 0.82}
+> [Curvature Is Space-Specific]
<Flatness OOD Limits>
(1) [Flatness Need Not Generalize]: Flat non-generalizing minimizers exist,
and sharpness-minimizing algorithms can generalize for reasons beyond
minimizing sharpness. #observation
[Wen, Li, Ma 2023](https://arxiv.org/abs/2307.11007)
[evidence](docs/sharpness_generalization_counterexample.md#L16-L30)
> This work critically examines this explanation. Through theoretical and empirical investigation, we identify the following three scenarios for two-layer ReLU networks: (1) flatness provably implies generalization; **(2) there exist non-generalizing flattest models and sharpness minimization algorithms fail to generalize, and (3) perhaps most surprisingly, there exist non-generalizing flattest models, but sharpness minimization algorithms still generalize. Our results suggest that the relationship between sharpness and generalization subtly depends on the data distributions and the model architectures and sharpness minimization algorithms do not only minimize sharpness to achieve better generalization.** This calls for the search for other explanations for the generalization of over-parameterized neural networks.
{reason: "theorem-backed counterexamples plus experiments on stylized two-layer networks; strong against a universal implication, weak about modern LLM adapters specifically", credence: 0.84}
(2) [Simple Features Can Be Shortcuts]: Simplicity bias can select one easy
feature while ignoring predictive complex features, making apparently
benign shifts destructive. #observation
[Shah et al. 2020](https://arxiv.org/abs/2006.07710)
[evidence](docs/simplicity_bias_pitfalls.md#L40-L48)
> Through theoretical analysis and targeted experiments on these datasets, we make four observations: **(i) SB of SGD and variants can be extreme: neural networks can exclusively rely on the simplest feature and remain invariant to all predictive complex features. (ii) The extreme aspect of SB could explain why seemingly benign distribution shifts and small adversarial perturbations significantly degrade model performance.** (iii) Contrary to conventional wisdom, SB can also hurt generalization on the same data distribution, as SB persists even when the simplest feature has less predictive power than the more complex features. (iv) Common approaches to improve generalization and robustness—ensembles and adversarial training—can fail in mitigating SB and its pitfalls. Given the role of SB in training neural networks, we hope that the proposed datasets and methods serve as an effective testbed to evaluate novel algorithmic approaches aimed at avoiding the pitfalls of SB.
{reason: "NeurIPS 2020 theoretical analysis and controlled synthetic/image experiments; the paper does not connect feature simplicity to loss curvature", credence: 0.78}
----
(3) [Weight Flatness Is Weak Semantic Evidence]: Weight-space flatness can
improve a training procedure without showing that the learned feature is
deep, causal, or robust under the shift of interest.
{reason: "Wen et al. break flatness-implies-generalization in stylized settings; Shah et al. establish shortcut risk but supply no curvature link", inference: 0.78}
+> [Curvature Is Space-Specific]
<Function Space Curvature>
(1) [TRAM Regularizes Function Curvature]: TRAM combines SAM parameter
sharpness with trust-region divergence in function space. #observation
[Sherborne et al. 2023](https://arxiv.org/abs/2310.03646)
[evidence](docs/tram_function_space_curvature.md#L58-L65)
> Sharpness-aware minimization (SAM) reports improving domain generalization by reducing the loss surface curvature in the parameter space. However, **generalization during fine-tuning is often more dependent on the transferability of representations in the function space. Trust-region methods (TR) target this goal by regularizing representation curvature** to reduce catastrophic forgetting of pre-trained task-agnostic information while adopting task-specific skills. We consider unifying these strategies for low curvature in both parameter space and function space to improve out-of-domain (OOD) generalization. We propose Trust Region Aware Minimization (TRAM), a SAM algorithm fine-tuning for low parameter sharpness and smooth, informative representations preserving pre-trained structure. TRAM uses a trust region bound to inform the SAM adversarial neighborhood, introducing an awareness of function curvature within optimization for flatter minima.
{reason: "authors' abstract and method framing; reports OOD experiments in vision and language, but this is function divergence rather than a context-path second derivative", credence: 0.68}
(2) [TRAM Has Multiple Trust Regions]: TRAM estimates divergence against a
previous step, the pretrained model, a noised input, or a Fisher metric.
#observation
[Sherborne et al. 2023](https://arxiv.org/abs/2310.03646)
[evidence](docs/tram_function_space_curvature.md#L1159-L1172)
> We propose four variants of TRAM based on different trust region estimations. **TRAM-$\theta_{t-1}$ uses divergence against the previous step; TRAM-$\theta_0$ is a simplifying heuristic of this divergence against the pre-trained model only; and TRAM-$x$ uses noised input divergence, $d_x$.** TRAM-Fisher extends FSAM by measuring the Fisher Information metric around the trust region.
{reason: "Table 2 caption from the authors' paper; all four are trust-region estimates, not steering-effect curvature", credence: 0.80}
(3) [Static Steering Assumes a Constant Direction]: SVF motivates
context-dependent steering by replacing one global vector with a local
gradient field. #observation
[Li, Li, Huang 2026](https://arxiv.org/abs/2602.01654)
[evidence](docs/steering_vector_fields_context_aware.md#L5-L12)
> Reliability also degrades in long-form generation and multi-attribute steering. We take a geometric view of these failures. **A static SV applies the same update vector everywhere in representation space, implicitly assuming that the concept-improving direction is constant across contexts. When the locally effective direction varies with the current activation, a single global vector can become misaligned, which yields weak or reversed effects. Guided by this perspective, we propose Steering Vector Fields (SVF), which learns a differentiable concept scoring function whose local gradient defines the steering direction at each activation, making interventions explicitly context-dependent.** This formulation supports coordinated multi-layer interventions in a shared, aligned concept space, and enables efficient long-form and multi-attribute control within a unified framework.
{reason: "February 2026 preprint; authors' motivation and method description with reported OOD results on hallucination and truthfulness tasks; first-order context dependence rather than curvature measurement", credence: 0.58}
(4) [Steering Generalization Is Measurable]: One-shot optimized steering
vectors provide a direct cross-input generalization target. #observation
[Dunefsky, Cohan 2025](https://arxiv.org/abs/2502.18862)
[evidence](docs/one_shot_steering_generalization.md#L15-L22)
> Steering vectors (SVs) have emerged as a promising approach for interpreting and controlling LLMs, but current methods typically require large contrastive datasets that are often impractical to construct and may capture spurious correlations. **We propose directly optimizing SVs through gradient descent on a single training example, and systematically investigate how these SVs generalize.** We consider several SV optimization techniques and find that the resulting SVs effectively mediate safety-relevant behaviors in multiple models.
{reason: "authors' abstract; measures cross-input transfer but contains no curvature measure", credence: 0.68}
----
(5) [Context-Curvature Diagnostic Is Open]: For a fixed steering direction
v and ordered context path x(t), estimate the second derivative of its
behavioral effect with
kappa_v = abs(e_v(x(t+epsilon)) - 2e_v(x(t)) + e_v(x(t-epsilon))) / epsilon^2.
Here e_v(x) is the steered-minus-unsteered behavioral effect, and x(t)
must vary context while preserving the behavior being tested. Unordered
paraphrases support sensitivity or variance, not a second derivative.
Low kappa_v might predict OOD transfer after controlling for in-distribution
effect size, but a correlation would indicate robustness rather than prove
semantic depth. #assumption
{reason: "TRAM supplies the function-space prior, SVF motivates context dependence, and Dunefsky-Cohan supply the transfer target; none of these sources tests this second-order diagnostic", inference: 0.55}
+> [Curvature Is Space-Specific]
// ══════════════════════════════════════════════════════════════════════
// ADDITIONAL METHODS (Prompt Tuning, LN Tuning, Bone, Trainable Tokens)
// These are boundary cases that don't strongly support or oppose the thesis
// ══════════════════════════════════════════════════════════════════════
# Boundary Cases
<Prompt Tuning Orthogonal Paradigm>
(1) [Input Not Weights]: Prompt/prefix tuning modifies the input embedding
space rather than weight matrices. At T5-XXL scale, it matches full FT
with 0.01% params. #assumption
{reason: "scales with model size; conceptually different from weight adaptation; when it works, it suggests the model already has the capability", credence: 0.65}
----
(2) [Capability Already Present]: When prompt tuning works, it reveals that
the target task is within the model's existing computational capacity --
no weight restructuring needed.
{reason: "informative boundary: prompt tuning success = no intervention needed; prompt tuning failure = genuine capability gap requiring weight changes", inference: 0.55}
+> [Natural Manifold]
// ══════════════════════════════════════════════════════════════════════
// CONTRA: LINEARITY ASSUMPTION
// ══════════════════════════════════════════════════════════════════════
<SVD Is Linear Decomposition>
(1) [Nonlinear Structure Missed]: SVD is a linear decomposition of a nonlinear
system. It captures dominant linear subspaces but may miss nonlinear
interactions that matter for complex tasks. #assumption
{reason: "fundamental theoretical limitation; no PEFT method addresses this; but empirically SVD methods still win", credence: 0.75}
(2) [Checkpoint Specific]: SVD basis is model-specific and not transferable
across checkpoints without recomputation. #assumption
{reason: "practical limitation; PiSSA notes fast SVD takes only seconds, so cost is low", credence: 0.60}
----
(3) [Linearity Might Not Matter]: For the adaptations tested so far, linear
subspaces suffice. The nonlinear structure may only matter for
capabilities not yet probed.
{reason: "strong practical success despite theoretical limitation; but we haven't tested on sufficiently hard capability changes", inference: 0.45}
-> [Natural Manifold]
Reference in New Issue View Git Blame Copy Permalink