Abstract
In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the manifold formed by the recently proposed Group-and-Shuffle ($\mathcal{GS}$) orthogonal matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a $\text{spectra restoration}$ transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two $\mathcal{GS}$ orthogonal matrices is capable of uniting concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the $\href{this https URL}{link}$.
Abstract (translated)
在模型训练这一快速发展的领域中，人们始终对参数高效微调以及利用少量训练数据使模型适应特定任务的各种技术抱有实践兴趣。然而，一个悬而未决的问题是：如何将针对不同任务微调得到的多个适配器合并为一个，使其能在两项任务上都取得合理结果？具体而言，生成模型中主题适配器与风格适配器的合并问题仍未解决。本文旨在证明，在正交微调（OFT）场景下，可利用结构化正交参数化及其几何特性，获得免训练的适配器合并公式。特别地，我们推导了近期提出的分组与洗牌（$\mathcal{GS}$）正交矩阵构成的流形结构，并得到了两点间测地线近似的高效公式。此外，我们提出一种**谱恢复变换**，通过恢复合并适配器的谱特性实现更高质量的融合。我们在主题驱动生成任务中的实验表明，所提出的$\mathcal{GS}$正交矩阵合并方法能够融合不同适配器的概念与风格特征。据我们所知，这是首个针对乘法正交适配器的免训练合并方法。代码可通过$\href{this https URL}{链接}$获取。
URL
https://arxiv.org/abs/2604.05183
