On the Step Length Confounding in LLM Reasoning Data Selection
Summary: arXiv:2604.06834v1 Announce Type: cross
Abstract
Large reasoning models have recently demonstrated strong performance on complex tasks that require long chain-of-thought reasoning, through supervised fine-tuning on large-scale and high-quality datasets. To construct such datasets, existing pipelines generate long reasoning data from more capable Large Language Models (LLMs) and apply manually heuristic or naturalness-based selection methods to filter high-quality samples. Despite the proven effectiveness of naturalness-based data selection, which ranks data by the average log probability assigned by LLMs, our analysis shows that, when applied to LLM reasoning datasets, it systematically prefers samples with longer reasoning steps (i.e., more tokens per step) rather than higher-quality ones, a phenomenon we term step length confounding.
Introduction
The advent of Large Language Models has revolutionized the field of natural language processing, particularly in tasks requiring complex reasoning. However, the methodology used in selecting training data for these models has come under scrutiny. The effectiveness of data selection methods, particularly those based on the naturalness of samples, can lead to unintended biases in the datasets used for training.
Step Length Confounding
Our analysis identifies a significant issue in the data selection process known as step length confounding. This phenomenon occurs when the selection criteria favor longer reasoning steps over higher-quality reasoning. This bias arises primarily due to the influence of low-probability first tokens in reasoning sequences. Consequently, longer reasoning steps dilute the impact of these low probabilities, leading to an inflated average log probability score.
Proposed Solutions
To address the step length confounding issue, we propose two innovative methods aimed at refining the data selection process:
- ASLEC-DROP: This method involves dropping the first-token probabilities when calculating the average log probability, ensuring that the selection process is not skewed by the initial tokens.
- ASLEC-CASL: This method applies a causal debiasing regression technique to remove the confounding effects of the first tokens, allowing for a more accurate representation of sample quality.
Experimental Validation
We conducted experiments across four different LLMs and five evaluation benchmarks to assess the effectiveness of our proposed methods. The results indicate a significant improvement in the quality of selected samples when employing ASLEC-DROP and ASLEC-CASL. This validation highlights the importance of addressing biases in data selection to enhance the performance of reasoning models.
Conclusion
In conclusion, while naturalness-based data selection methods have been widely adopted for training reasoning models, our findings reveal the critical issue of step length confounding. By implementing the proposed ASLEC-DROP and ASLEC-CASL methods, we can mitigate this bias and improve the overall quality of reasoning data selection. Future work should focus on further refining these methods and exploring their applicability across diverse datasets and reasoning tasks.