To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
Summary: arXiv:2604.00715v1 Announce Type: cross
Retrieval-augmented generation (RAG) is a transformative approach that enhances the performance of language models (LMs) by supplying relevant context during testing, particularly in knowledge-intensive scenarios. Despite its promising capabilities, the relationship between the parametric knowledge acquired during pretraining and the non-parametric knowledge accessed through retrieval mechanisms remains inadequately explored. This gap is particularly evident when considering fixed data budgets.
In a recent study, researchers systematically investigate the trade-off between the size of the pretraining corpus and the size of the retrieval store over a broad spectrum of model and data scales. The team trained OLMo-2-based LMs with parameter counts varying from 30 million to 3 billion, utilizing up to 100 billion tokens from DCLM data. They adjusted both the scale of pretraining data (ranging from 1 to 150 times the number of parameters) and the size of the retrieval store (from 1 to 20 times), assessing performance across a diverse array of benchmarks that included reasoning tasks, scientific question answering (QA), and open-domain QA.
Key Findings
The study yielded several significant insights:
- Performance Improvement: Retrieval consistently enhances performance compared to parametric-only baselines, regardless of model scale.
- Three-Dimensional Scaling Framework: The researchers introduced a novel framework that illustrates performance as a function of model size, pretraining tokens, and retrieval corpus size.
- Optimal Data Allocation: This scaling manifold allows for the estimation of optimal allocations of a fixed data budget between pretraining and retrieval, highlighting the importance of strategic data management in model design.
- Marginal Utility of Retrieval: The benefit derived from retrieval varies significantly based on the model scale, the type of task, and the extent of pretraining saturation.
Practical Guidance for Language Modeling
The findings of this research provide a quantitative basis for understanding how retrieval can effectively complement pretraining. They offer practical recommendations for allocating data resources, which is crucial for the design of scalable language modeling systems. By identifying when to leverage retrieval in conjunction with pretraining, developers can optimize the performance of their language models and ensure that they are equipped to handle a variety of tasks efficiently.
Conclusion
As the landscape of AI language modeling continues to evolve, understanding the dynamics between pretraining and retrieval becomes increasingly important. The insights derived from this study pave the way for future research and development, ultimately leading to more efficient and capable language models that can better serve knowledge-intensive applications. The balance between memorization and retrieval is not just a theoretical consideration; it is a practical challenge that can significantly impact the effectiveness of AI systems in real-world scenarios.