Title: 2406.13175v2.pdf URL Source: https://arxiv.org/pdf/2406.13175 Published Time: Tue, 28 Jan 2025 02:10:35 GMT Number of Pages: 30 Markdown Content: # Sparse High Rank Adapters Kartikeya Bhardwaj ∗ § Nilesh Prasad Pandey ∗† Sweta Priyadarshi † Viswanath Ganapathy † Shreya Kadambi Rafael Esteves Shubhankar Borse Paul Whatmough § Risheek Garrepalli Mart Van Baalen Harris Teague § Markus Nagel § Qualcomm AI Research ‡§{kbhardwa,pwhatmou,hteague,markusn}@qti.qualcomm.com ## Abstract Low Rank Adaptation (LoRA) has gained massive attention in the recent generative AI research. One of the main advantages of LoRA is its ability to be fused with pretrained models, adding no overhead during inference. However, from a mobile deployment standpoint, we can either avoid inference overhead in the fused mode but lose the ability to switch adapters rapidly, or suffer significant (up to 30% higher) inference latency while enabling rapid switching in the unfused mode. LoRA also exhibits concept-loss when multiple adapters are used concurrently. In this paper, we propose Sparse High Rank Adapters (SHiRA), a new paradigm which incurs no inference overhead, enables rapid switching, and significantly reduces concept-loss. Specifically, SHiRA can be trained by directly tuning only 1-2% of the base model weights while leaving others unchanged. This results in a highly sparse adapter which can be switched directly in the fused mode. We further provide theoretical and empirical insights on how high sparsity in SHiRA can aid multi-adapter fusion by reducing concept loss. Our extensive experiments on LVMs and LLMs demonstrate that finetuning only a small fraction of the parameters in the base model significantly outperforms LoRA while enabling both rapid switching and multi-adapter fusion. Finally, we provide a latency- and memory-efficient SHiRA implementation based on Parameter-Efficient Finetuning (PEFT) Library which trains at nearly the same speed as LoRA while consuming up to 16% lower peak GPU memory, thus making SHiRA easy to adopt for practical use cases. To demonstrate rapid switching benefits during inference, we show that loading SHiRA on a base model can be 5×-16 × faster than LoRA fusion on a CPU. ¶ ## 1 Introduction Low Rank Adaptation (LoRA) [ 13 ] is an established technique to tune the behavior of large generative models such as Large Language Models (LLMs) [ 30 , 29 ] and Stable Diffusion [ 24 , 22 ]. As the name suggests, LoRA requires very few parameters since it trains low rank projection weights that consume very low memory during the finetuning process while producing excellent results. Moreover, these low rank weights can be fused analytically into the base model, thereby incurring no additional overhead during inference. Despite its success, there are still several limitations of low rank adaptation methods. First, if LoRA parameters are fused into the corresponding pretrained base model weights, they modify the entire weight tensor. Therefore, deploying LoRA on large models such as LLaMA-1/2 (7B+ parameters) or Stable Diffusion (1.5B+ parameters) on mobile devices would require changing a large number of weights during inference. Consequently, for mobile scenarios, if an application requires rapid adapter switching , existing low rank methods would incur a significant memory and latency cost. This is a major deployment challenge because, unlike large GPUs, local memory of small AI accelerators is limited and cannot store all weights at the same time. These challenges can be partially addressed by > ∗ Equal contribution. †Work done while employed at Qualcomm AI Research. ‡Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. ¶Code: https://github.com/Qualcomm-AI-research/SHiRA .38th Conference on Neural Information Processing Systems (NeurIPS 2024). > arXiv:2406.13175v2 [cs.LG] 27 Jan 2025 LoRA > Car dog in space Elephant House, mountain Thunder Bird knight > BLUEFIRE > SHiRA-SNIP > PAINTINGS MULTI-ADAPTER Figure 1: Sparse Hi gh Rank A dapters (SHiRA): Changing about 1-2% weights of the pretrained generative model is often sufficient to achieve high performance. Due to its extreme sparsity, SHiRA enables rapid switching and also reduced concept loss during multi-adapter fusion. In contrast, LoRA modifies majority of parameters when fused, thus prohibiting rapid switching on mobile devices, and also experiences concept loss during multi-adapter fusion. For LoRA, elephant for single “paintings” adapter case has artifacts (extra/broken tusks); bird and knight for multi-adapter case lose “paintings” concept and keep only the “blue fire” effects. SHiRA does not experience these issues. running LoRA in unfused mode; however, unfused inference can incur as high as 30 % additional latency compared to the base model [ 1 ] (see section 2.1 for details). This increased inference time in unfused mode and time for adapter switching significantly hampers user experience; hence, this is an important problem which has been a focus of recent research by various industries [ 9]. Second, LoRA has a well-known limitation called concept loss when using multiple concurrent adapters, e.g., combining multiple style transfer adapters, etc. Specifically, it has been well documented [ 34 , 26 , 8 ]that a simple additive merging of multiple LoRA adapters leads to concept loss of one or more adapters. Finally, recent literature also contributes important theoretical and empirical knowledge towards the value of high rank adapters . For instance, Kalajdzievski [ 16 ] shows that the high rank adapters can greatly outperform low rank adapters when used with correct scaling factors. This calls for further investigation into whether other high rank adapters would significantly outperform LoRA. In view of the above, we address the following key problems in this paper: ( i) How can we perform rapid switching for fused adapters? (ii ) Is there a simpler solution for multi-adapter fusion to reduce concept loss? ( iii ) Can we build high rank adapters that have high expressive power without significantly increasing the training or inference costs? To this end, we propose Sparse Hi gh Rank A dapters (SHiRA), a single solution to all three problems above. SHiRA is a highly sparse but a high rank adapter which relies on training only a very small subset of parameters from the original pretrained network. One of the crucial insights we demonstrate is that even finetuning merely 1-2% parameters of the pretrained generative model is sufficient to achieve high performance on many adapter tasks (see Fig. 1). However, unlike LoRA layers that modify all parameters in the weight tensors in the fused mode, SHiRA still keeps a very low percentage of parameters that need to be switched, thus enabling rapid switching at inference time. Moreover, since the pretrained weights are huge, SHiRA being a very sparse adapter greatly aids multi-adapter fusion by significantly reducing concept loss. Finally, we theoretically and emprically analyze the high rank vs. sparsity properties of SHiRA and why that helps with adapter performance. Overall, we make the following key contributions :• We propose SHiRA, a new high rank adapter paradigm to demonstrate that changing as few as 1-2% parameters of the original network is sufficient for adaptation. Our crucial insight is that even the most basic masking criteria (to identify the top 1-2% parameters) enable SHiRA to significantly outperform LoRA on diverse vision and language tasks. • SHiRA enables on-device rapid adapter switching and provides a natural multi-adapter fusion technique due to high sparsity, thus, significantly reducing concept loss . We also theoretically analyze SHiRA through the lens of high rank adaptation vs. sparsity. • We conduct extensive experiments on LLMs (LLaMA-7B, LLaMAv2-7B) and LVMs (Stable Diffusion, SDXL) where we demonstrate that SHiRA significantly outperforms LoRA on both single- and multi-adapter tasks. On LLMs, we show that SHiRA achieves up to 2.7% better accuracy than LoRA on commonsense reasoning. SHiRA also complements advanced variants of LoRA such as DoRA [20] and can be easily applied on top of them. 2• Finally, on the training side, we provide a PEFT-based latency- and memory-efficient implementation for SHiRA which trains nearly as fast as standard LoRA while consuming 16% lower peak GPU memory. Beyond PEFT, we provide a simple way to turn any trainer into SHiRA finetuning. For inference, we demonstrate that SHiRA weights can be loaded on a CPU up to 5×-16 × faster than equivalent LoRA fusing, thereby enabling rapid switching. The rest of this paper is organized as follows: section 2 presents the background and related work. We propose SHiRA in section 3 while describing its theoretical properties in section 4. We then conduct extensive experiments for SHiRA in section 5. Finally, we discuss the key findings in section 6 and conclude the paper in section 7. ## 2 Background and Related Work 2.1 Background: Edge Deployment Challenges for LoRA There are three existing deployment options for LoRA: ( i) fuse the adapter offline and then deploy on-device: this changes a large fraction of the weight tensors compared to base model which prohibits rapid switching since it will increase DRAM traffic considerably; ( ii ) keep the adapter unfused and run the inference in unfused mode: this can help with rapid switching but would incur significant addi-tional (up to 30% higher) latency as shown in [ 1] since we would have LoRA branches in the forward pass during inference; ( iii ) use the Huggingface/Diffusers pipeline [ 1] (built for server-grade GPUs) for mobile inference. This pipeline consists of load →fuse →inference →unfuse →unload to switch adapters. Here, unfused LoRA-A and LoRA-B weights (see Fig. 2(a)) are first loaded into the memory and then fused into the base model by computing Wnew = W + AB ; this new weight is used for inference. To switch the adapter, we can unfuse the adapter as W = Wnew − AB and then unload existing LoRA weights to load the new ones. We provide further evidence in Appendix A to demonstrate that such a pipeline is not feasible for edge devices. This is primarily because edge devices are memory-limited and not all weights of large generative models can be stored in the local memory at the same time. Hence, loading and fusing needs to happen layerwise on a mobile device that obviously results in massive inference latency costs. 2.2 Related Work LoRA, its variants, and sparse adapters. Many LoRA variants exist in literature: DoRA [ 20 ], LoRA+ [ 11 ], VeRA [ 17 ], LoRA-FA [ 35 ], RS-LoRA [ 16 ], among many others. The crucial difference between this literature and our work is that we develop a high rank adapter without increasing training and inference costs. Also, for such methods, the final fused adapter still updates all elements in the pretrained weight tensor, thus prohibiting rapid switching. Moreover, for completeness, we will also show that SHiRA is orthogonal to and can be applied on top of some of the latest, more advanced LoRA variants such as DoRA [20] while preserving the benefits of rapid switching. A few other LoRA variants have also explored a combination of sparsity and low rank adaptation. Ex-amples include RoSA [ 21 ], SoRA [ 6], Sparse-Adapters [ 12 ], etc. Among these, Sparse-Adapters [ 12 ]explores the use of popular pruning techniques (e.g., SNIP [ 19 ]) to prune out adapters to improve their efficiency. SoRA [ 6 ] proposes an adaptive rank version of LoRA by gating elements of down and up projection layers and pruning out the zero entries at inference. Finally, RoSA [ 21 ] combines a sparse adapter with a low rank one to achieve some high rank benefits. However, since they combine their method with LoRA, the fused adapter weight still overwrites the entire pretrained weight tensor. Partial Finetuning. Our work is most closely related to partial finetuning techniques that were mostly proposed in the pre-LoRA era [ 36 , 28 , 3, 33 , 10 ]. These methods use a mix of fixed sparse masks [ 28 ] or learned masks [ 36 , 10 ] to finetune a pretrained network. Note that, these techniques have been mostly explored for relatively small language models, and not for recent LLMs and diffusion models. Since the LoRA models exploded in popularity, it has been unclear if other sparse finetuning techniques would achieve comparable results to LoRA on generic adapter tasks, particularly in the vision domain. One significant limitation of partial finetuning, as opposed to LoRA-based methods, is its high GPU memory consumption , making it impractical to be used for large generative models. Consequently, the reduced memory consumption for finetuning was a key factor to LoRA’s success and its widespread adoption. To this end, we provide a memory- and latency-efficient PEFT-based implementation for SHiRA which trains as efficiently as LoRA, thus requiring significantly lower memory consumption compared to prior partial finetuning techniques. Further, we explore the effectiveness of sparse finetuning on both large language and vision models and provide a detailed analysis on rapid switching and multi-adapter fusion of the high rank adapters. 3Backward Pass > Forward Pass > Trainable weights > Frozen weights > Weights > 1 > 1 > 111 > 11 > 1 > Masked Gradients Mask Original Gradients > Linear Layer > Input Features Output Features > Non -Zero Gradients Zero Gradients > Pretrained > Weights, W > b. Sparse Hi gh Rank Adaptation > (SHiRA ) > a. Low Rank Adaptation > (LoRA ) > LoRA -A > LoRA -B > + > Rank r > h > x > Fused weight at inference would modify > all elements of pretrained weight W > ⊙ =Figure 2: (a) LoRA when fused into the pretrained model modifies all weights and prevents rapid adapter switching. (b) SHiRA does not require additional weights during training but finetunes very few pretrained weights. Our approach relies on a sparse mask for gradient-masking during training. We show that finetuning as low as 1-2% parameters is sufficient to achieve high accuracy. A notable concurrent work is SpIEL [ 4] which scales partial finetuning to modern LLMs and also has a PEFT implementation that results in comparable speed and memory as LoRA. The main differences between SpIEL and SHiRA are as follows: ( i) SpIEL works with dynamic masks while SHiRA uses a static mask. ( ii ) Dynamic mask in SpIEL requires users to install custom sparse linear layer kernels for the GPUs. In contrast, SHiRA does not require installing any custom kernels and directly works with native Pytorch. Hence, SHiRA’s biggest advantage is its ease of training/inference deployment. (iii ) We also analyze multi-adapter fusion properties, e.g., impact of sparsity on orthogonality between adapters, which were not discussed in SpIEL. ( iv ) Finally, SHiRA demonstrates its effectiveness on both vision and language tasks, whereas SpIEL only discusses the language tasks. Multi-Adapter Fusion. Existing Multi-adapter fusion methods focus on preventing concept loss [ 8 ,34 , 26 ]. However, these methods usually either just use the base LoRA as it is (and then perform some non-trivial postprocessing on them) [ 34 , 26 ], or some create some minor variants [ 8 ]. In contrast, we introduce a new adapter for the concept loss problem where multiple concepts naturally do not interfere with each other. In that respect, our work is orthogonal to the prior multi-adapter fusion work since our adapter can be further postprocessed using such techniques. ## 3 Proposed Approach 3.1 Sparse High Rank Adapters (SHiRA) SHiRA exploits highly sparse trainable parameters in the pretrained model. In its simplest form, our adapter can be trained by masking gradients such that only a fraction of original weights get updated. Specifically, we do not add any new weights to the forward pass like LoRA (see Fig. 2(a)) but rather make a small percentage of existing weights trainable (see Fig. 2(b) top). To this end, we first create an extremely sparse ( ∼98 -99% zeros) mask M ∈ Rn×m = {0, 1}n×m, where n, m are dimensions of the pretrained weight matrix. M is then used to mask the gradients during backpropagation using a Hadamard product (see Fig. 2(b) bottom). Thus, very few parameters get updated during training and our adapter consists of just those sparse weights. Concrete gradient masking-based and another latency-/memory-efficient PEFT implementations for SHiRA are discussed in section 3.3. We consider the following masks M (only 1-2% trainable parameters, see also Appendix B): 1. SHiRA-Struct: In this structured mask, certain rows or columns of the weight as well as its diagonal are set to be trainable. All other rows/columns are not trainable. The diagonal makes the mask high rank whereas the structured trainable rows/columns – set to 1 to enable gradient flow to corresponding parameters – lead to a rank 1 adapter. Thus, SHiRA-Struct is a combination of a high rank but very sparse adapter and a rank 1 adapter. 2. SHiRA-Rand: This mask is obtained by randomly setting 1-2% parameters as trainable. 3. SHiRA-WM: Here we pick top-K parameters to train based on their weight magnitudes (WM), the absolute value of the weight for each layer. 4SHiRA Adapter 1 SHiRA Adapter 2 Fused Multi -Adapter > + α2 =+ α1 > Sparse > Weights Indices > + > Storing [Sparse Weights + Indices] consumes > much less memory than pretrained weights > Weights trained > for SHiRA > b. Multi -adapter fusion a. Rapid adapter switching > Base Model > Weights that changed > during adaptation > Frozen weights > Non -Zero Weights for > SHiRA Adapter 2 Zero Weights > Non -Zero Weights for > SHiRA Adapter 1 > Base Model Weights Figure 3: (a) Rapid adapter switching: The sparse finetuned weights can be stored as weights and their indices. At inference time, these weights can be loaded on the base model weights. Since only 1-2% weights need to be overwritten, the adapter can be efficiently switched with different weights at inference, eliminating the need for a separate fusion stage. (b) Multi-adapter fusion: Concept-loss can be reduced if multiple adapters do not significantly interfere with each other. 4. SHiRA-Grad: This is a gradient-based mask. We first collect gradients on a small calibra-tion set and then pick top 1-2% weights that receive the highest gradient magnitudes. 5. SHiRA-SNIP: The SNIP metric from the pruning literature [ 19 ] combines weight magnitude and gradient strategies, i.e., SNIP equals magnitude of the gradient times the weight. 3.2 Rapid Adapter Switching, Multi-Adapter Fusion, and High Rank Since very few base weights change during the SHiRA training, we can simply extract them out and store them as sparse weights and their indices (see Fig. 3(a)). Hence, SHiRA is comparable to LoRA in model size but overwrites only a fraction of the pretrained weights at inference time. In contrast, LoRA fuses into base weights as Wnew = W + AB and changes the entire weight. Note that, we do not actually need to fuse SHiRA but rather just need to overwrite the modified value at the correct index in the pretrained weight tensor. This enables rapid switching on resource-constrained devices. To verify that SHiRA indeed provides rapid switching benefits compared to LoRA, we provide an optimized implementation based on scatter_op to overwrite base model weights instead of fusing them like LoRA. We demonstrate that on a CPU, weight loading for SHiRA adapters can be up to 5×-16 × faster than equivalent LoRA fusing for inference (see Appendix C and Fig 7). Next, we discuss multi-adapter fusion in SHiRA. Given two adapters A1 and A2 with sparse masks M1 and M2, we ask the following questions: ( i) What is the impact of sparsity on relative interference between adapters in the multi-adapter setting? ( ii ) Is it possible to create masks that result in nearly orthogonal SHiRA weights so they do not significantly interfere with each other at inference time? Getting adapters that do not interfere with each other is essential to avoid concept-loss. To this end, we define specific metrics in section 4.2 to analyze orthogonality properties between adapter weights for various SHiRA strategies. We theoretically show that at least one of the SHiRA methods, i.e., SHiRA-Struct can in fact create near-orthogonal adapters. We further experimentally demonstrate in section 5.2.2 that SHiRA-Struct indeed outperforms other methods for multi-adapter fusion. Finally, since we do not have any low rank weights in the forward pass, our proposed adapters can be high rank albeit highly sparse. We theoretically analyze the rank vs. sparsity properties in section 4. 3.3 Memory- and Latency-Efficient SHiRA Training We have created two implementations for SHiRA: ( i) a backward hook-based gradient masking to turn any trainer into SHiRA finetuning (see Appendix D), and ( ii ) a PEFT-based implementation. As discussed in Appendix E, the PEFT-based SHiRA implementation consumes 16 .63 % lower peak GPU memory and trains almost at a similar speed as LoRA . On the contrary, DoRA exhibits a 40 .99% and 28 .9% increase in memory and training time respectively compared to LoRA. ## 4 Theoretical Insights for SHiRA 4.1 Rank vs. Sparsity Below we discuss parameter and learning complexity, parallels between LoRA and SHiRA, as well as its optimization properties from the lens of rank and sparsity. Lemma 4.1. The parameter complexity and learning complexity of SHiRA is equal to the number of non-zero elements in the adapter. Appendix F.1 provides the proof. This lemma suggests that despite high rank property of SHiRA, it would not require significantly larger datasets to converge. 5Lemma 4.2. If we specify a sparsity factor, the LoRA is r rank approximation of SHiRA with approximation error bounded by σ2 > r+1 , the (r + 1) th singular value of the SHiRA adapter. The above lemma is proved in section F.2. As a consequence of this lemma, any r rank LoRA adapter of size (m, n ) can be seen as an approximation of a SHiRA adapter with mr + rn non-zero elements. Lemma 4.3. Scaling factor for SHiRA is independent of the rank of the adapter and can be set to 1. Please see the proof in Appendix F.3. Lemma 4.3 states that we do not need scaling factors to stabilize the training and, therefore, we do not need additional hyperparameters like α or independent learning rates for separate A and B matrices like in LoRA[ 13 ] or LoRA+ [ 11 ]. Of note, the scaling factor α can still be used at inference time to vary the intensity of the adapter. 4.2 Adapter Weight Orthogonality in Multi-Adapter Fusion In this section, we provide theoretical and empirical insights by studying properties of SHiRA and LoRA adapter designs for multi-adapter fusion. Lemma 4.4. Consider two adapters, ∆W1 and ∆W2. If one of the adapters, ∆W1 or ∆W2 lies in the null space of the other, then the adapters will not interfere multiplicatively. Proof is given in Appendix F.4. The above lemma implies that two adapters can be efficiently fused without interference if they are orthogonal. In order to analyze the orthogonality between any two adapter weights, we define the following metrics: Definition 1. Adapter Weight Orthogonality Magnitude (AWOM) is defined as the l2 norm of the product AT > 1 A2 for two sparse adapter weights A1, A2 ∈ Rn×m. AWOM enables us to understand how far the product AT > 1 A2 is from a zero matrix O ∈ Rm×m (Oi,j = {0}∀ i, j ). Definition 2. Adapter Weight Orthogonality Ratio (AWOR) is defined as the sparsity ratio of the product AT > 1 A2. Specifically, AWOR = h 1 −  ||A T > 1A2|| 0 > m2 i , where m2 is #elements in AT > 1 A2.Together, AWOM and AWOR can provide us an idea of relative orthogonality between adapter weights A1 and A2. Next, we analyze how at least one of the SHiRA strategies (i.e., SHiRA-Struct) can result in near-orthogonal adapters. Recall that, SHiRA-Struct adapters train certain rows/columns and the diagonal elements while keeping all other parameters frozen. Hence, the final trained adapter (after subtracting the pretrained weight) contains a structured pattern of rows/columns and diagonal elements, everything else being zero. Now, without loss of generality, consider two SHiRA-Struct adapters for a layer with square m × m weights: A1 = I + S1 and A2 = I + S2, where S1 and S2 are row-wise patterns of trained weights for two different tasks, and I is an identity matrix. Also, S1 and S2 are non-overlapping, e.g., both have same number of non-zero rows but are offset from each other such that they do not have any common trained rows. Then, the following result holds: Lemma 4.5. Non-overlapping SHiRA-Struct adapters are nearly orthogonal: AWOR for non-overlapping SHiRA-Struct adapters is at most the sum of sparsity of individual adapters. Since all SHiRA masks are highly sparse, AT > 1 A2 has a lot of zeros, thus making the adapters nearly orthogonal. Proof is provided in Appendix F.5. We demonstrate the orthogonality properties of various adapters and report the simulation results in Fig. 4. For our experiment, we compute AWOM and AWOR for a variety of adapter designs - Figure 4: Comparison of average AWOM (left) and AWOR (right) for 50 randomly initialized adapters. We compare different adapters, namely - Dense, Sparse LoRA, SHiRA-WM and SHiRA-Struct. dense, sparse-LoRA [ 12 ] (sparse LoRA A and B weights), SHiRA-WM and SHiRA-Struct based adapters. As shown in Fig. 4, both dense and sparse LoRA have low AWOR for adapters with larger dimen-sions, e.g., 4096 × 4096 which is typical in LLMs. This signifies that these adapter weights are non-orthogonal. On the con-trary, SHiRA-WM achieves much higher AWOR than the LoRA variants. More inter-estingly, SHiRA-Struct is nearly orthogo-nal. Note that, due to high sparsity, AWOM also tends to be much lower for SHiRA adapters than the dense counterparts. Com-bined with the fact that AWOR of SHiRA 6adapters is 63-96% higher sparsity than LoRA, this may suggest that AT > 1 A2 would be closer to zero for SHiRA adapters, thus potentially bringing them closer to orthogonality and less interference. Finally, although we have shown interesting properties for SHiRA-Struct, it is still a rank 1 + diagonal adapter. Hence, we need to tradeoff single adapter performance (which strongly depends on adapter’s expressive power) against the multi-adapter fusion capabilities. For instance, next we will see that while SHiRA-Struct is good for vision, SHiRA-SNIP performs well across both LVMs and LLMs. Remark 1. The orthogonality property shown here can lead to disentangled representation for adapter outputs before they merge into the base model. However, this property does not hold for other SHiRA masks that do not have a regular sparsity pattern like SHiRA-Struct even if other SHiRA strategies are still more orthogonal than LoRA weights (e.g., see SHiRA-WM AWOR in Fig. 4(right)). Interestingly, for unstructured sparse masks like SHiRA-WM, SHiRA-Grad, SHiRA-SNIP, etc., both overlapping and non-overlapping adapters have similar orthogonality properties. We discuss this in more detail in section 5.3.2. Finally, this analysis only focuses on orthogonality of adapter weights and not on orthogonality of subspaces. We leave the subspace analysis of SHiRA for future work. ## 5 Experiments 5.1 Training Setup and Datasets For the vision tasks, we use the RealisticVision-v3 model checkpoint for Stable Diffusion-v1.5, and finetune it using different adapters on two style transfer datasets collected using public domain images. The first dataset is called Bluefire which provides a “blue fire” effect to images. The second dataset is a painting dataset which gives a “paintings” effect (see Appendix section G for more details). For both these datasets, we conduct single- and multi-adapter experiments. To quantify the image quality, we use the Human Preference Score-V2 (HPSv2) [32]. On the language domain, we experiment with LLaMA 7B [ 29 ], LLaMA2-7B [ 30 ] and evaluate it on various commonsense reasoning benchmarks such as HellaSwag, PIQA, SIQA, BoolQ, Arc-easy, Arc-challenge, OpenBookQA and Winogrande. Similar to our vision investigations, we conduct single- and multi-adapter experiments on LLMs as well. Specifically, for language finetuning, we follow the setup adopted by [ 14 , 20 ] for training and evaluating LoRA [ 13 ], DoRA [ 20 ], and SHiRA based finetuned models on downstream tasks. Finally, we also explore generalizability of SHiRA to other popular LoRA models and applications such as SDXL [ 22 ] and DreamBooth [ 25 ]. Detailed training setups are provided in the Appendix H. 5.2 Vision Results 5.2.1 Impact of Various SHiRA Masks We first evaluate the image quality for SHiRA and LoRA on Paintings and Blue-fire datasets for both single and multi-adapter usecases. Fig. 1 demonstrates com-parison between SHiRA-SNIP and LoRA. As evident, by merely changing 2% pre-trained weights, SHiRA generates high quality images for both finetuning tasks. > Style Method %Params HPSv2 score( ↑) > α= 1 α= 0 .5 > Paintings LoRA 3.84 24 .7±1.831 .3±1.5 > SHiRA-Struct 1.99 31 .2±1.733 .0±1.8 > SHiRA-Grad 2.05 30 .3±1.832 .3±1.8 > SHiRA-SNIP 2.05 29 .8±1.831 .6±1.8 > Bluefire LoRA 3.84 32 .6±1.933 .6±1.6 > SHiRA-Struct 1.99 34 .2±1.634 .1±1.5 > SHiRA-Grad 2.05 34 .2±1.533 .7±1.7 > SHiRA-SNIP 2.05 33 .7±1.733 .7±1.6 Table 1: HPSv2 score of various adapters on Paintings and Bluefire. SHiRA-Struct outperforms all other methods. Next, we compare various types of SHiRA masks in Fig. 5. Clearly, all SHiRA schemes produce impressive images for different prompts and sig-nificantly outperform LoRA. We fur-ther quantify the image quality using HPSv2 for each of the masks. The results are presented in Table 1. As evident, all variants of SHiRA con-sistently achieve superior or similar HPSv2 scores than LoRA, especially for larger α (see details on scaling factor α in Appendix I). More results are provided in Appendices J and K: see Table 10 and Fig. 10, 11, 12. 5.2.2 SHiRA Adapters aid Multi-Adapter Fusion As explained in section 4.2, high sparsity of SHiRA reduces their AWOM and increases the AWOR metrics by increasing the number of zeros in AT > 1 A2 product even for unstructured schemes such as SHiRA-WM, SHiRA-Grad, and SHiRA-SNIP. We hypothesized that this may lead to improved multi-adapter fusion performance. This was also pointed out by [ 26 , 8, 31 ]: naively merging multiple LoRA adapters leads to poor performance and concept loss. 7LoRA > thunder bird Cat Ship, sunset, sea House, Prairie fox night flower > SHiRA-Struct SHiRA-Grad > BLUEFIRE > SHiRA-SNIP > PAINTINGS MULTI-ADAPTER Figure 5: Comparison between different SHiRA masking methods for single- and multi-adapter image generation. For multi-adapter fusion, SHiRA-Struct outperforms all other adapters by generating exceptional images with high frequency details and good concept fusion (e.g., see fox and flower). We now validate the effectiveness of various SHiRA schemes on multi-adapter fusion. The right two columns in Fig. 1 and Fig. 5 show our results. SHiRA is clearly better at capturing both concepts than LoRA. For example, both bird and knight images in Fig. 1 generated with LoRA lose most of the paintings concept. Similarly, for the fox image in Fig. 5, LoRA does not show significant bluefire concept. In contrast, SHiRA-Struct and SHiRA-SNIP consistently perform well on many different prompts and produce exceptional images for multi-adapter fusion. Please refer to Appendix K.1 (Fig. 10, 11, 12, and 13) for additional results. For certain classes that were not included in the training set for both adapters (e.g., see Koala in Fig. 10, 12, and 13 in Appendix), we observe that LoRA produces significant artifacts whereas SHiRA generates high quality images. 5.3 Language Results 5.3.1 Single Adapter SHiRA Finetuning Similar to vision results, we demonstrate the effectiveness of SHiRA on language tasks. For our experiments, each adapter (i.e., weight-magnitude, gradient-magnitude, and SNIP based SHiRA) is trained on the combined 170K sample commonsense reasoning dataset released by [ 14 , 20 ]. Similar to [ 20 ], we train our SHiRA adapters for 3 epochs and compare it against the LoRA baselines. As shown in Table 2, various SHiRA adapters outperform LoRA by 1.9-2.7% on an average on LLaMA-7B. Importantly, SHiRA only modifies 1% base parameter weights as compared to 66.72% (4.5B weights ) changed by LoRA in the fused mode, thus enabling rapid switching on edge devices. Interestingly, we found that SHiRA-Struct does not perform well on language tasks likely because it is a rank 1 + diagonal adapter and may not have sufficient expressive power. Moreover, when compared to newer techniques like DoRA [ 20 ], our proposed work takes an orthogo-nal approach by finetuning very few parameters of the pretrained weights. This strategy allows for an efficient integration of our adapter with methods like DoRA to improve the expressiveness of the adapters. As we show in Table 2, our proposed adapter benefits from DoRA based finetuning and achieves almost comparable performance (within 0.3%) to DoRA on an average, with an added benefit of changing only 1% parameters at inference time. In contrast, DoRA would lead to 66.72% (4.5B weights ≈ 9GB memory in FP16 format) parameter change in the fused mode. Therefore, SHiRA is orthogonal to other existing low rank methods and can be efficiently integrated with them. 8Model %Params %C BoolQ( ↑) PIQA( ↑) Arc-e( ↑) Arc-c( ↑) WG( ↑) OBQA( ↑) HS( ↑) SIQA( ↑) Avg.( ↑) > LoRA 0.83 66.72 68.9 80.7 77.8 61.3 78.8 74.8 78.1 77.4 74.7 (+0%) SHiRA-Grad 1.0 1.0 68.4 80.9 80.2 64.7 80.4 78.2 80.3 79.4 76.6 (+1.9%) SHiRA-WM 1.0 1.0 69.6 81.6 81.5 66.5 79.8 79.4 79.6 77.8 77.0 (+2.3%) > SHiRA-SNIP 1.0 1.0 68.3 80.6 81.5 67.9 80.0 79.6 82.1 79.1 77.4 (+2.7%) DoRA 0.84 66.72 68.5 82.9 81.4 65.8 80.8 81.0 84.8 79.6 78.1 (+0%) SHiRA-WM-DoRA 6.25 ∗1.0 70.9 81.9 81.7 64.9 80.8 79.2 84.5 78.6 77.8 (-0.3%) Table 2: Evaluation of LLaMA-7B on Commonsense Reasoning. WG and HS denote WinoGrande and HellaSwag, respectively. %C represents parameters changed in the fused mode. ( ↑): the higher the better. Green denotes improvement. ∗Trained by masking a high-rank DoRA with a WM mask of top 1% weights, thus changing only 1% of the model during both training and inference. > Model %Params %C BoolQ( ↑)PIQA( ↑)Arc-e( ↑)Arc-c( ↑)WG( ↑)OBQA( ↑)HS( ↑)SIQA( ↑)Avg.( ↑) > LoRA 0.83 66.72 69.90 79.9 79.8 64.7 82.6 81.0 83.6 79.5 77.61 (+0%) DoRA 0.84 66.72 71.8 83.7 83.7 68.2 82.6 82.4 89.1 76.0 79.68 (+2.07%) > SHiRA-SNIP 1.0 1.0 70.42 81.71 83.25 68.6 80.51 81.0 89.78 79.01 79.28 (+1.67%) Table 3: Results for LLaMA2-7B on Commonsense Reasoning. Finally, we experiment with LLaMA2-7B [ 30 ] and demonstrate that SHiRA-SNIP – which achieved the best results on LLaMA-7B – yields significant accuracy gains compared to LoRA and nearly the same accuracy as DoRA (within 0.4%, see Table 3). 5.3.2 Multi-Adapter Fusion on LLMs We now extend our LLM experiments to the multi-adapter fusion setting. To this end, we create a new setup where we independently train multiple adapters on training sets of individual commonsense reasoning benchmarks, i.e., one adapter each for BoolQ, PIQA, and Arc-Easy. In contrast, each adapter in section 5.3.1 was trained on a combined dataset containing 170K samples from all eight commonsense benchmarks as proposed in [ 14 , 20 ]. In the present section, the goal is to evaluate how much accuracy drop various adapters experience when we perform multi-adapter fusion. Due to its simplicity towards constructing a mask, we will use SHiRA-WM in the rest of this paper. Further, we explore two settings - overlapping and non-overlapping SHiRA-WM adapters. The overlapping mask consists of top 1% parameters being trained for all tasks. On the other hand, the non-overlapping setting trains the top 1% weights for the first task, next top 1% for the second task, and so on. We compare the performance of both LoRA and SHiRA across the multi-adapter fusion of these three tasks. As shown in Table 4, both overlapping and non-overlapping multi-SHiRA outperform multi-LoRA on all three commonsense benchmarks. This is inline with our theoretical analysis in section 4.2 where we suggest that even unstructured sparse SHiRA adapters such as SHiRA-WM would have more orthogonal behavior than LoRA due to high sparsity (see higher AWOR of SHiRA-WM in Fig. 4(right)). In comparison, independently trained LoRA adapters would have no such property and suffer greatly during multi-adapter fusion. As a result, we see that both SHiRA models outperform LoRA by more than 6.5% accuracy on average. Further analysis of the properties of these trained adapters is discussed in Appendix K.3 (see Table 13 and Fig. 9). Of note, this experiment also demonstrates the value of creating a good mask for single adapter performance: Non-overlapping masks achieve lower single adapter accuracy than the corresponding overlapping masks since they train less important parameters. Hence, creating an optimal mask for SHiRA should be of significant interest to future research. 5.4 Content/Style Personalization: Generalizing SHiRA to SDXL and DreamBooth Finally, we extend SHiRA to focus on DreamBooth [ 25 ] using a much bigger vision model called SDXL [ 22 ]. We follow a similar setup as adopted by [ 2]. Specifically, one content (vase) and two style (wooden sculpture and canvas) datasets with five images each were collected from the DreamBooth dataset [ 25 ] and public domains, respectively. These datasets were used to train various content and style adapters. For our experiments, we use SDXL [ 23 ] as our base model and train both LoRA and SHiRA adapters with comparable trainable parameters on individual single-concept datasets. During training, prompts containing special identifier tokens like " " or "