Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions
Summary: arXiv:2604.11841v1 Announce Type: cross
Abstract
Low-rank adaptation (LoRA) is a widely used strategy for efficient fine-tuning of large language models (LLMs), but its strictly linear structure fundamentally limits expressive capacity. The bilinear formulation of weight updates captures only first-order dependencies between low-rank factors, restricting the modeling of nonlinear and higher-order parameter interactions. In this paper, we propose Polynomial Expansion Rank Adaptation (PERA), a novel method that introduces structured polynomial expansion directly into the low-rank factor space.
Introduction
Fine-tuning large language models has become a critical area of research, and low-rank adaptation (LoRA) is a popular method that enhances this process. However, the limitations of LoRA’s first-order interaction model pose challenges for capturing complex relationships within data. Our proposed method, Polynomial Expansion Rank Adaptation (PERA), addresses these limitations by integrating polynomial expansions into the adaptation framework.
Methodology
PERA introduces structured polynomial expansions to the low-rank factor space. By expanding each low-rank factor, we are able to synthesize high-order interaction terms before composition. This transformation allows the adaptation space to evolve into a polynomial manifold, which is capable of modeling richer nonlinear couplings without increasing rank or inference costs. The methodology can be summarized as follows:
- Structured polynomial expansion is applied to low-rank factors.
- High-order interactions are synthesized prior to factor composition.
- The adaptation space is transformed into a polynomial manifold.
- No increase in rank or inference cost is required.
Theoretical Analysis
We provide a comprehensive theoretical analysis that demonstrates PERA’s enhanced expressive capacity compared to existing linear adaptation approaches. Our analysis reveals that the integration of high-order interactions significantly improves feature utilization, allowing for a more nuanced understanding of the data.
Empirical Results
To validate our theoretical claims, we conducted extensive experiments across diverse benchmarks. The results consistently show that PERA outperforms state-of-the-art methods. Key findings include:
- Incorporating high-order nonlinear components, particularly square terms, is crucial for enhancing expressive capacity.
- PERA maintains strong and robust performance across various rank settings.
- The method demonstrates superior adaptability and efficiency in fine-tuning large language models.
Conclusion
Polynomial Expansion Rank Adaptation (PERA) represents a significant advancement in the field of low-rank fine-tuning for large language models. By introducing polynomial expansions into the low-rank factor space, PERA enhances expressive capacity and feature utilization while maintaining efficiency. Our empirical results confirm the effectiveness of PERA, making it a promising method for future research and applications in machine learning.
For those interested in exploring PERA further, our code is available at GitHub.