A Progressive Training Strategy for Vision-Language Models to Counteract Spatio-Temporal Hallucinations in Embodied Reasoning
Summary: arXiv:2604.10506v1 Announce Type: new
Vision-Language Models (VLMs) have significantly advanced the field of artificial intelligence, particularly in static image understanding. However, these models face substantial challenges when it comes to spatiotemporal reasoning. One of the most pressing issues is known as “multi-image reasoning hallucination.” This phenomenon results in a drastic decline in performance when transitioning from forward temporal queries to reverse ones, highlighting a reliance on superficial shortcuts rather than genuine causal comprehension.
Challenges in Spatiotemporal Reasoning
The limitations of current VLMs in dynamic reasoning can be attributed to several factors:
- Dependence on Superficial Shortcuts: Many VLMs utilize patterns in data rather than understanding the underlying causal relationships.
- Performance Gaps: The significant disparity in performance between forward and reverse temporal queries indicates a lack of true comprehension of temporal dynamics.
- Limited Dataset Availability: Existing datasets often do not cover the intricacies of spatiotemporal reasoning, making it difficult for models to learn effectively.
Introducing a New Chain-of-Thought (CoT) Dataset
To address these challenges, researchers have developed a novel Chain-of-Thought (CoT) dataset designed to break down complex reasoning tasks into manageable spatiotemporal steps and clear judgments. This dataset serves as a foundation for training VLMs to enhance their reasoning capabilities.
A Progressive Training Framework
The proposed training strategy involves a two-step process:
- Supervised Pre-training: The initial phase involves training the model on the CoT dataset, which instills logical structures necessary for understanding intricate reasoning.
- Fine-tuning with Weakly-Labeled Data: Following the pre-training, models are fine-tuned using scalable weakly-labeled datasets. This step aims to broaden the generalization capabilities of the models, allowing them to adapt to diverse scenarios.
Experimental Results
Recent experiments have demonstrated the effectiveness of this progressive training approach. Notably, the method has:
- Improved backbone accuracy significantly, validating the training framework’s efficacy.
- Reduced the forward-backward performance gap from over 70% to just 6.53%, indicating a marked enhancement in the models’ capacity for dynamic reasoning.
Conclusion
This research confirms that a well-structured progressive training strategy can effectively mitigate the inherent temporal biases present in current Vision-Language Models. By fostering a deeper understanding of causal relationships and spatiotemporal dynamics, this approach paves the way for more robust and reliable AI systems capable of sophisticated reasoning.