OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
In the rapidly evolving field of artificial intelligence, continual pre-training has emerged as a vital technique for adapting large language models (LLMs) to various target languages and domains. However, one of the significant challenges faced by researchers and practitioners is the optimal mixture ratio of training data, which remains a sensitive hyperparameter. Tuning this parameter can be both time-consuming and costly, often requiring weeks of computational resources. A suboptimal choice can lead to inefficient use of these resources, causing delays in model development and deployment.
Introduction to OptiMer
To address these challenges, a recent paper titled “OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training,” proposes a novel approach to continual pre-training. The authors introduce OptiMer, a method that decouples the selection of mixture ratios from the training phase. Instead of fixing the mixture ratios before training begins, OptiMer allows for flexibility and optimization post-hoc.
Methodology
The methodology behind OptiMer involves several key steps:
- Training Multiple Models: The approach begins by training one continual pre-training (CPT) model for each dataset. This allows for a tailored adaptation to the specific characteristics of each dataset.
- Extracting Distribution Vectors: After training, the distribution vector for each model is extracted. This vector represents the parameter shift induced by the corresponding dataset, encapsulating the unique information learned from it.
- Post-Hoc Optimization: Using Bayesian optimization, the optimal composition weights for these distribution vectors are searched and determined post-hoc. This step is crucial as it allows for an efficient exploration of the weight space without the need for retraining.
Experimental Results
Experiments conducted on the Gemma 3 27B model across multiple languages, including Japanese and Chinese, as well as various domains such as Mathematics and Code, demonstrate the effectiveness of OptiMer. The results indicate that:
- OptiMer consistently outperforms traditional data mixture and model averaging baselines, achieving a remarkable 15-35 times lower search cost.
- The optimized weights can be interpreted as mixture ratios, and retraining with these ratios significantly enhances the performance of data mixture continual pre-training.
- The same vector pool can be re-optimized for different objectives without retraining, allowing for the creation of target-tailored models on demand.
Conclusion
This work establishes a paradigm shift in how data mixture ratio selection is approached in continual pre-training. Traditionally viewed as a pre-training decision, it can now be reformulated as a post-hoc optimization over distribution vectors. This flexibility not only streamlines the pre-training process but also provides a more efficient means of adapting LLMs to diverse tasks and datasets. As AI research continues to progress, methods like OptiMer could pave the way for more efficient and effective model development, ultimately advancing the capabilities of artificial intelligence systems.