The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
Summary: arXiv:2604.09780v1 Announce Type: new
Abstract
Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their “expert specialization” remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself.
Key Findings
This research provides several important insights into the nature of expert specialization in MoEs:
- Expert usage similarity is fundamentally tied to the hidden state similarities present in the models.
- Specialization emerges from the representation space rather than being an inherent feature of the routing architecture.
- Evidence was gathered across five pre-trained models, confirming these findings at both token and sequence levels.
Mechanisms Behind Specialization
Further analysis revealed that load-balancing loss plays a crucial role in maintaining routing diversity. Specifically, it suppresses shared hidden state directions to prevent specialization collapse, particularly when data diversity is limited, such as in small batch scenarios. This finding presents a theoretical basis for understanding the conditions under which expert specialization can thrive or deteriorate.
Challenges in Interpretation
Despite the clear mechanistic explanations provided, the patterns of specialization observed in pre-trained MoEs often elude human interpretation. Some notable challenges include:
- Expert overlap between different models responding to the same queries is no greater than that observed between entirely different questions, approximately 60%.
- Prompt-level routing does not consistently predict routing at the rollout level, indicating a disconnect between initial input and expert activation.
- Deeper layers of models frequently exhibit near-identical activation of experts across semantically unrelated inputs, particularly within reasoning models.
Conclusion
In conclusion, while the operational efficiency of MoEs is well established, the understanding of expert specialization remains a complex and challenging problem. The findings of this study suggest that comprehending expert specialization is as intricate as deciphering the geometry of hidden states in large language models, a conundrum that has persisted in the research community. As the field advances, further exploration into these dynamics will be essential for harnessing the full potential of MoEs in artificial intelligence.