REAM: Merging Improves Pruning of Experts in LLMs
Summary: arXiv:2604.04356v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
Introduction
The rise of large language models has transformed the field of natural language processing (NLP), enabling a variety of applications from chatbots to automated content generation. However, deploying these massive models presents significant challenges, particularly in terms of memory usage. Weight pruning and quantization have emerged as common solutions to mitigate these issues, but they often come at the cost of performance degradation. This article discusses a new approach—Router-weighted Expert Activation Merging (REAM)—that improves upon existing methods by preserving the model’s effectiveness while reducing its memory footprint.
Background
Mixture-of-Experts (MoE) architectures leverage multiple expert models, selectively activating only a subset during inference. While effective, this approach can lead to high memory consumption, especially in large models. Traditional pruning methods, such as REAP, focus on removing experts entirely, which can risk losing critical information and performance capabilities.
Introducing REAM
REAM presents an innovative solution to the challenges faced by MoE models. Rather than deleting experts, REAM merges their weights, allowing the model to retain more of its original performance characteristics. This method not only reduces memory usage but also enhances the model’s ability to generalize across various tasks.
Methodology
To evaluate the effectiveness of REAM, we conducted extensive testing across multiple MoE LLMs. Our focus was on two primary benchmarks:
- Multiple-choice (MC) question answering: Assessing the model’s ability to select the correct answer from a list of options.
- Generative (GEN) tasks: Evaluating the model’s capability to produce coherent and contextually relevant text.
Results and Discussion
The results demonstrate a notable trade-off between MC and GEN performance influenced by the calibration data mix. By systematically varying the data types—general, math, and coding—we explored the Pareto frontier of performance trade-offs. Our findings indicate that REAM frequently outperforms existing baselines and matches the performance of original uncompressed models in many scenarios.
Conclusion
In conclusion, the Router-weighted Expert Activation Merging (REAM) method represents a significant advancement in optimizing Mixture-of-Experts large language models. By merging rather than discarding experts, REAM successfully balances memory efficiency and performance, setting a new standard for future developments in model optimization.