Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability
In the rapidly evolving field of artificial intelligence, particularly in the realm of large language models (LLMs), a significant debate has emerged regarding the efficacy of supervised finetuning (SFT) and reinforcement learning (RL). The prevailing narrative suggests that while SFT tends to memorize information, RL is more adept at generalizing knowledge. A recent paper, titled Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability, challenges this notion by examining the complexities surrounding reasoning SFT, particularly when utilizing long chain-of-thought (CoT) supervision.
The authors of the study argue that the cross-domain generalization observed in reasoning SFT is not simply absent; rather, it is dependent on several factors including optimization dynamics, the quality and structure of training data, and the inherent capability of the base model. Their findings reveal that many of the failures previously reported in generalization can be attributed to under-optimization artifacts.
Key Findings
-
Optimization Dynamics:
The study highlights a dip-and-recovery pattern wherein cross-domain performance may initially degrade before showing improvement with extended training. This suggests that short training checkpoints can lead to an underestimation of a model’s generalization capabilities. -
Data Quality and Structure:
The quality of the training data plays a crucial role; low-quality solutions adversely affect generalization. Conversely, verified long-CoT traces contribute to consistent gains across different domains. -
Model Capability:
The research indicates that more robust models are better equipped to internalize transferable procedural patterns, such as backtracking, even from simplified tasks like toy arithmetic games. In contrast, less capable models tend to mimic superficial verbosity without grasping the underlying procedural logic. -
Asymmetric Generalization:
Interestingly, the findings illustrate that while reasoning capabilities improve, safety may degrade. This reframes the critical question from whether reasoning SFT can generalize to the conditions under which it generalizes and the associated costs of such generalization.
Conclusion
The implications of this research are profound for the future of AI development. As the landscape of language models continues to evolve, understanding the conditional aspects of generalization in reasoning SFT can lead to more effective training methodologies. By recognizing the interplay between optimization, data quality, and model capability, researchers and developers can better harness the full potential of LLMs, paving the way for advancements in reasoning and generalization that are both reliable and safe. The study encourages a shift in focus towards these underlying factors, challenging the simplistic dichotomy of memorization versus generalization in AI.