Reasoning Fails Where Step Flow Breaks
Summary: arXiv:2604.06695v1 Announce Type: new
Abstract: Large reasoning models (LRMs) that generate long chains of thought now perform well on multi-step math, science, and coding tasks. However, their behavior is still unstable and hard to interpret, and existing analysis tools struggle with such long, structured reasoning traces. We introduce Step-Saliency, which pools attention-gradient scores into step-to-step maps along the question-thinking-summary trajectory. Across several models, Step-Saliency reveals two recurring information-flow failures: Shallow Lock-in, where shallow layers over-focus on the current step and barely use earlier context, and Deep Decay, where deep layers gradually lose saliency on the thinking segment and the summary increasingly attends to itself and the last few steps. Motivated by these patterns, we propose StepFlow, a saliency-inspired test-time intervention that adjusts shallow saliency patterns measured by Step-Saliency via Odds-Equal Bridge and adds a small step-level residual in deep layers via Step Momentum Injection. StepFlow improves accuracy on math, science, and coding tasks across multiple LRMs without retraining, indicating that repairing information flow can recover part of their missing reasoning performance.
Introduction
In recent years, large reasoning models (LRMs) have demonstrated the capability to handle complex tasks that require multi-step reasoning, such as solving mathematical problems, conducting scientific inquiries, and executing programming challenges. Despite their impressive performance, these models often exhibit inconsistencies in their reasoning processes, making it difficult for researchers to understand their decision-making mechanisms.
Challenges in Current Reasoning Models
Current analysis tools for LRMs fall short when it comes to interpreting long and structured reasoning traces. As a result, there is a pressing need to develop better methods for analyzing how these models process information over multiple steps. Understanding the flow of information in LRMs is crucial for enhancing their reliability and interpretability.
Introducing Step-Saliency
To address the limitations of existing analysis methods, we introduce Step-Saliency. This innovative approach pools attention-gradient scores to create step-to-step maps along the trajectory of question, thinking, and summary. Our findings reveal two significant patterns of information-flow failures:
- Shallow Lock-in: Shallow layers tend to focus excessively on the current step, neglecting valuable context from earlier steps.
- Deep Decay: Deep layers experience a gradual loss of saliency regarding the thinking segment, resulting in the summary increasingly attending to itself and only the most recent steps.
StepFlow: A Proposed Intervention
In light of the identified information-flow issues, we propose StepFlow, a saliency-inspired intervention designed to enhance the reasoning capabilities of LRMs during test time. StepFlow employs two key mechanisms:
- Odds-Equal Bridge: This mechanism adjusts shallow saliency patterns based on the insights gained from Step-Saliency analysis.
- Step Momentum Injection: This technique introduces a small, step-level residual in deep layers to counteract the effects of Deep Decay.
Results and Implications
Our experiments demonstrate that StepFlow significantly improves accuracy on various tasks across multiple LRMs, including math, science, and coding challenges. Remarkably, this enhancement occurs without the need for retraining the models. These findings suggest that by repairing information flow, we can recover a portion of the reasoning performance that is typically compromised in large reasoning models.
Conclusion
The introduction of Step-Saliency and the subsequent development of StepFlow mark a significant advancement in the analysis and improvement of large reasoning models. As the field continues to evolve, it is imperative to focus on enhancing the interpretability and reliability of these systems to foster greater trust and usability in real-world applications.