The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
Summary: arXiv:2603.29025v1 Announce Type: cross
Abstract
Large language models (LLMs) have shown remarkable capabilities in generating human-like text; however, they often encounter challenges when a prominent surface cue conflicts with an unstated feasibility constraint. This article explores these failures through a structured framework consisting of diagnosis, measurement, bridging, and treatment. By conducting a causal-behavioral analysis of the well-known “car wash problem” across six different models, we reveal that approximately context-independent sigmoid heuristics dominate decision-making processes. Specifically, the distance cue tends to exert influence that is 8.7 to 38 times greater than the actual goal itself. Additionally, token-level attribution illustrates patterns that align more closely with keyword associations rather than the expected compositional inference.
Key Findings
- Heuristic Override Benchmark (HOB): We developed the HOB, which consists of 500 instances divided into four heuristic families and five constraint families. The benchmark includes minimal pairs and explicitness gradients, showcasing its generality across 14 different models.
- Model Performance: Under strict evaluation conditions (10 out of 10 correct), no model managed to exceed a performance threshold of 75%. Moreover, presence constraints proved to be the most challenging, with an accuracy rate of merely 44%.
- Impact of Hints: Introducing minimal hints, such as emphasizing the key object within the task, resulted in an average recovery of +15 percentage points (pp). This suggests that the root cause of the models’ failures lies in their inability to infer constraints rather than a lack of knowledge.
- Conservative Bias: In a surprising twist, 12 out of the 14 models displayed worse performance when the feasibility constraint was removed, with declines of up to -39 pp, indicating a conservative bias in reasoning.
Further Insights
Our parametric probes confirm that the observed sigmoid pattern extends beyond simple distance cues; it generalizes across cost, efficiency, and semantic-similarity heuristics. Furthermore, goal-decomposition prompting, which compels models to enumerate preconditions before arriving at an answer, was found to improve performance by an additional 6 to 9 pp.
Conclusion
Together, these findings characterize heuristic override as a systematic vulnerability in reasoning processes of large language models. This research not only highlights the limitations of current models but also provides a new benchmark for evaluating progress toward overcoming these challenges. As LLMs continue to evolve, understanding and addressing the implications of heuristic overrides will be essential for developing more reliable and effective artificial intelligence systems.