Robust Reasoning Benchmark
Summary: arXiv:2604.08571v1
Announce Type: cross
Abstract: While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their underlying reasoning processes remain highly overfit to standard textual formatting. We propose a perturbation pipeline consisting of 14 techniques to evaluate robustness of LLM reasoning. We apply this pipeline to the AIME 2024 dataset and evaluate 8 state-of-the-art models on the resulting benchmark.
Key Findings
Our research reveals significant insights into the robustness of reasoning in LLMs:
- Frontier models exhibit resilience against perturbations.
- Open weights reasoning models suffer catastrophic collapses, with average accuracy drops reaching up to 55% across various perturbations and even 100% on some.
- The structural fragility of these models is exposed through rigorous testing.
Methodology
To further disentangle mechanical parsing failures from downstream reasoning failures, we implemented a unique approach:
- Models were forced to solve multiple unperturbed mathematical problems sequentially within a single context window.
- This isolation was crucial in examining the models’ working memory capacity.
Results
The results of our evaluations indicate alarming trends among open weight models:
- Models ranging from 7B to 120B parameters, including Claude Opus 4.6, exhibited notable accuracy decay on subsequent problems.
- This degradation suggests that intermediate reasoning steps can permanently pollute standard dense attention mechanisms.
Implications
Our findings highlight a critical area for future research in LLM development:
- To achieve reliable reasoning, future architectures must integrate explicit contextual resets within a model’s own Chain-of-Thought.
- This leads to fundamental open questions regarding the optimal granularity of atomic reasoning tasks and how they can be effectively structured.
Conclusion
The Robust Reasoning Benchmark provides essential insights into the limitations of current LLM architectures. As the field evolves, understanding and addressing these weaknesses will be vital for developing more resilient and reliable AI systems. The ongoing research in this domain may pave the way for breakthroughs that enhance the reasoning capabilities of AI, ultimately leading to more sophisticated and dependable models.