One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging
Summary: Weight-space model merging combines independently fine-tuned models without accessing original training data, offering a practical alternative to joint training. While merging succeeds in multitask settings, its behavior in multilingual contexts remains poorly understood.
Abstract
Recent advancements in machine learning have led to significant progress in the field of natural language processing, particularly in machine translation. One of the prominent techniques is weight-space model merging, which allows for the combination of independently fine-tuned models without the need for access to the original training data. This approach presents a viable alternative to the more resource-intensive process of joint training.
Exploration of Multilingual Model Merging
While weight-space merging has shown success in multitask scenarios, its effectiveness in multilingual contexts remains largely unexplored. In our study, we systematically investigate the behavior of weight-space merging specifically for multilingual machine translation. This involves fully fine-tuning a language model on large-scale bilingual corpora and evaluating several standard merging strategies.
Key Findings
Our experiments reveal several critical insights:
- Merging tends to degrade performance, particularly when the target languages differ significantly.
- To understand the underlying reasons for this performance decline, we analyze internal representations through span-conditioned neuron selectivity and layer-wise centered kernel alignment.
- We find that language-specific neurons tend to cluster in embedding layers and upper transformer blocks, while intermediate layers show a higher degree of shared representation across languages.
Redistribution of Language-Specific Neurons
One of the most significant observations from our research is that fine-tuning appears to redistribute rather than sharpen language selectivity. Specifically:
- Neurons associated with supervised and closely related languages become less exclusive in their activation patterns.
- Conversely, neurons that correspond to unsupervised languages exhibit a tendency to become more isolated.
This redistribution leads to increased representational divergence in the higher layers of the model, which are crucial for the generation of translations. Such modifications in neuron behavior suggest that multilingual fine-tuning could be reshaping the model’s geometric representation in a manner that diminishes its compatibility with traditional weight-space merging assumptions.
Conclusion
Our research provides a comprehensive explanation for the challenges faced during weight-space merging in multilingual translation scenarios. The findings highlight the complexities involved in merging models trained on diverse languages and underscore the need for further exploration in this domain. As the field of multilingual machine translation continues to evolve, understanding the underlying mechanics of model merging will be essential for developing more effective and robust translation systems.