Reinforcing Structured Chain-of-Thought for Video Understanding
Summary: arXiv:2603.25942v1 Announce Type: cross
Multi-modal Large Language Models (MLLMs) have emerged as powerful tools for video understanding, demonstrating significant potential in parsing complex visual data. However, several challenges persist that hinder their effectiveness in this domain. A primary concern is the phenomenon known as “thinking drift,” where the reasoning process becomes disconnected from the task at hand. Additionally, MLLMs often exhibit weak temporal comprehension, which is critical for understanding the sequential nature of video content. These issues remain even when advanced Reinforcement Learning (RL) techniques, such as Group Relative Policy Optimization (GRPO), are employed.
Another limitation of current RL methodologies is their reliance on Supervised Fine-Tuning (SFT). This process is not only resource-intensive due to the requirement for extensive Chain-of-Thought (CoT) annotation but also involves multiple training stages. Such fixed reasoning paths restrict the MLLMs’ generalization capabilities and can introduce systemic biases.
Introducing Summary-Driven Reinforcement Learning (SDRL)
To address these pressing issues, we propose a novel framework known as Summary-Driven Reinforcement Learning (SDRL). This single-stage RL approach eliminates the necessity for SFT by employing a Structured CoT format that follows the sequence: Summarize -> Think -> Answer. By rethinking the integration of reasoning processes, SDRL aims to overcome the limitations imposed by traditional methods.
Key Innovations in SDRL
SDRL introduces two innovative self-supervised mechanisms that are integrated into the GRPO objective:
- Consistency of Vision Knowledge (CVK): This mechanism enforces factual grounding by minimizing the Kullback-Leibler (KL) divergence among the generated summaries. It ensures that the generated content remains consistent with the visual information presented in the video.
- Dynamic Variety of Reasoning (DVR): This feature encourages exploration by dynamically adjusting the diversity of reasoning based on the accuracy of the group’s outputs. By promoting a variety of thinking pathways, SDRL allows the model to explore different reasoning strategies, enhancing overall performance.
Balancing Alignment and Exploration
The innovative integration of CVK and DVR effectively strikes a balance between alignment and exploration within the reasoning process. This dual supervision not only focuses on the correctness of the final answer but also emphasizes the quality and diversity of the reasoning pathway taken to reach that answer. Such a holistic approach is essential for enhancing the interpretability and reliability of MLLMs in video understanding tasks.
Results and Impact
Our method has demonstrated state-of-the-art performance across seven public VideoQA datasets, establishing SDRL as a promising advancement in the field of video understanding. By addressing the challenges of thinking drift, weak temporal comprehension, and the limitations of existing RL techniques, SDRL paves the way for more robust and adaptable MLLMs capable of nuanced video analysis.
As the demand for sophisticated video understanding tools continues to grow, innovations like SDRL will be crucial in enhancing the capabilities of AI systems, ensuring they can effectively interpret and reason about complex visual information.