Route Experts by Sequence, not by Token
Summary: arXiv:2511.06494v2 Announce Type: replace-cross
Abstract
Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity. Prior adaptive routing methods introduce additional modules and hyperparameters, often requiring costly retraining from scratch.
We propose Sequence-level TopK (SeqTopK), a minimal modification that shifts the expert budget from the token level to the sequence level. By selecting the top T · K experts across all T tokens, SeqTopK enables end-to-end learned dynamic allocation — assigning more experts to difficult tokens and fewer to easy ones — while preserving the same overall budget.
SeqTopK requires only a few lines of code, adds less than 1% overhead, and remains fully compatible with pretrained MoE models. Experiments across math, coding, law, and writing show consistent improvements over TopK and prior parameter-free adaptive methods, with gains that become substantially larger under higher sparsity (up to 16.9%).
These results highlight SeqTopK as a simple, efficient, and scalable routing strategy, particularly well-suited for the extreme sparsity regimes of next-generation LLMs. Code is available at https://github.com/Y-Research-SBU/SeqTopK.
Key Features of SeqTopK
- Dynamic Expert Allocation: Unlike traditional methods, SeqTopK allows for a flexible distribution of expert resources based on the complexity of the tasks at hand.
- Minimal Overhead: The implementation requires minimal changes to existing models, making it easy to integrate without significant resource investment.
- Compatibility: SeqTopK works seamlessly with pretrained MoE models, allowing users to leverage existing architectures without needing extensive retraining.
- Scalability: The method is designed to handle extreme sparsity, making it ideal for future iterations of large language models that will require efficient resource management.
Impact on Large Language Models
The introduction of SeqTopK represents a significant advancement in the field of machine learning, particularly in the development of large language models. By optimizing the allocation of experts based on the sequence rather than individual tokens, researchers can improve model performance across a variety of domains.
The ability to adaptively assign resources allows models to respond more effectively to complex queries, ultimately enhancing their utility in real-world applications such as legal document analysis, coding assistance, and creative writing.
Conclusion
As the demand for more sophisticated and capable AI models grows, innovations like SeqTopK will play a crucial role in shaping the future landscape of artificial intelligence. By providing a robust and efficient routing mechanism for experts, SeqTopK opens new avenues for research and application in the rapidly evolving field of machine learning.