Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
Summary: arXiv:2604.08880v1 Announce Type: cross
Abstract: Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capacity gap: distillation may fail when the teacher-student capability mismatch is large. We revisit the capacity gap from a practical perspective by re-examining commonly used experimental settings. Notably, we find that CoT distillation often degrades performance compared to the student’s pre-distillation baseline, an issue obscured when only post-distillation comparisons are reported. We therefore propose a more realistic evaluation protocol and find that the impact of capacity gap effects does not consistently dominate across tasks and settings, especially when candidate teachers differ substantially in performance. Our results offer practical guidance for selecting teacher-student pairs in CoT distillation.
Introduction
The field of artificial intelligence has evolved significantly with the introduction of various distillation techniques. Among these, Chain-of-Thought (CoT) distillation has gained prominence for its ability to transfer complex reasoning skills from larger, more capable models (teachers) to smaller models (students). However, a notable challenge remains: the capacity gap between teachers and students. This gap can hinder the effectiveness of the distillation process, leading to suboptimal performance in student models.
Understanding the Capacity Gap
The capacity gap refers to the mismatch in capabilities between the teacher and student models. When the teacher has significantly more capacity or knowledge than the student, the distillation process may fail to yield the desired outcomes. This phenomenon has been documented in previous studies, yet it has not been thoroughly explored from a practical standpoint.
Revisiting Experimental Settings
In this study, we revisit the experimental settings commonly used to evaluate CoT distillation. Our analysis reveals that relying solely on post-distillation performance metrics can be misleading. Many studies have reported improvements in student performance after distillation without considering their pre-distillation baselines. This oversight can obscure instances where distillation actually degrades performance.
Proposed Evaluation Protocol
To address these issues, we propose a more realistic evaluation protocol. This protocol emphasizes the significance of comparing student performance before and after distillation. By doing so, we can better understand the true impact of the capacity gap on the distillation process.
Key Findings
- CoT distillation may result in performance degradation compared to the student’s pre-distillation baseline.
- The impact of capacity gap effects varies across different tasks and settings.
- When selecting teacher-student pairs for CoT distillation, it is crucial to consider the performance disparity between candidates.
Conclusion
This study sheds light on the complexities of Chain-of-Thought distillation and the challenges posed by capacity gaps. Our findings advocate for a more nuanced approach to evaluating distillation processes, emphasizing the importance of baseline comparisons. As the field continues to advance, understanding these dynamics will be essential for optimizing the transfer of reasoning capabilities in AI models.
Practical Guidance
For practitioners in the field, our results provide valuable insights for selecting teacher-student pairs in CoT distillation. By carefully considering the capabilities of both models, one can enhance the likelihood of successful distillation outcomes, ultimately leading to more efficient and effective AI systems.