Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy
Summary: arXiv:2603.26299v1 Announce Type: cross
Abstract
Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems, yet challenging because LoRA update directions span different subspaces and contribute unevenly. When merged naively, such mismatches can weaken the directions most critical to certain task losses while overemphasizing relatively less important ones, ultimately reducing the model’s ability to represent all tasks faithfully.
Introduction
The rapid advancement in artificial intelligence (AI) has led to the development of various methods for enhancing model performance across tasks. One such technique is Low-Rank Adaptation (LoRA), which allows for efficient model tuning by introducing low-rank matrices to update the weights of neural networks. However, the challenge arises when attempting to merge multiple LoRA modules, as their update directions may not align well, leading to compromised performance.
The Challenge of LoRA Merging
Merging LoRA modules presents two primary challenges:
- Subspace Coverage: The directions represented by different LoRA modules may cover diverse areas of the representation space. If not properly managed, this can lead to inadequate coverage of the necessary directions for various tasks.
- Directional Anisotropy: This refers to the imbalance of influence among the different directions. Some directions may become overly emphasized while others are neglected, which can result in suboptimal task performance.
Proposed Solution: TARA-Merging
To address these challenges, we propose TARA-Merging (Task-Rank Anisotropy Alignment). This innovative approach focuses on aligning merging weights using a preference-weighted cross-entropy pseudo-loss. The key elements of TARA-Merging include:
- Preservation of Task-Relevant LoRA Subspaces: By ensuring that the merging process retains the essential directions relevant to specific tasks, TARA-Merging maintains the integrity of the original LoRA updates.
- Mitigation of Anisotropy: The direction-wise reweighting strategy effectively reduces the influence imbalance, allowing for a more equitable representation of all tasks.
Empirical Validation
The effectiveness of TARA-Merging was evaluated across eight vision benchmarks and six natural language inference (NLI) benchmarks. The results demonstrated that TARA-Merging consistently outperformed both vanilla merging approaches and other LoRA-aware baselines. Key findings include:
- Enhanced robustness across diverse tasks.
- Improved generalization capabilities, indicating better adaptation to unseen data.
- A clear demonstration of the importance of addressing both subspace coverage and anisotropy in the merging process.
Conclusion
The research presented in this paper highlights the complexities involved in merging LoRA modules and introduces a robust solution in the form of TARA-Merging. By focusing on both subspace coverage and anisotropy, this approach paves the way for more effective AI systems that can perform well across a variety of tasks. Future work will explore further refinements to the merging process and its applications to additional domains.