Discover and Prove: An Open-source Agentic Framework for Hard Mode Automated Theorem Proving in Lean 4
In a significant advancement in the field of automated theorem proving (ATP), researchers have introduced an innovative framework called Discover And Prove (DAP). This framework aims to enhance the capability of large language models (LLMs) in solving complex mathematical problems under what they refer to as “Hard Mode.” The study, documented in arXiv:2604.15839v1, challenges existing benchmarks that have traditionally favored simpler problem formats.
Introduction to Hard Mode vs. Easy Mode
Most current ATP benchmarks utilize a design approach referred to as “Easy Mode,” where the final answer is embedded within the formal statement. This method simplifies the tasks for automated systems, leading to potentially inflated assessments of their capabilities. In contrast, “Hard Mode” presents a more rigorous challenge, requiring systems to independently discover answers before constructing formal proofs.
Key Contributions of the Research
The research makes two significant contributions to the field:
- Release of MiniF2F-Hard and FIMO-Hard: These are expert-reannotated Hard Mode variants of two widely-used ATP benchmarks, enabling more realistic assessments of automated systems.
- Introduction of Discover And Prove (DAP): This agentic framework employs LLMs for natural-language reasoning and incorporates explicit self-reflection, allowing for the discovery of solutions and the rewriting of Hard Mode statements into Easy Mode formats suitable for existing ATP provers.
Achievements of DAP
DAP has set a new standard in the realm of automated theorem proving. Notably, it has achieved remarkable results on two key benchmarks:
- On CombiBench, DAP increased the number of solved problems from 7 (the previous state-of-the-art, Pass@16) to 10.
- On PutnamBench, DAP became the first system to formally prove 36 theorems in Hard Mode.
Insights into LLM Performance
One of the most striking insights revealed by this research is the performance gap between state-of-the-art LLMs and formal provers. While LLMs achieved over 80% answer accuracy on the same problems where traditional provers managed under 10%, this disparity highlights the unique utility of Hard Mode benchmarks. These benchmarks are particularly effective in measuring the true capabilities of automated systems.
Future Directions
The introduction of DAP and the Hard Mode benchmarks signifies a paradigm shift in the evaluation of automated theorem proving systems. As researchers continue to refine these frameworks and methodologies, the potential for LLMs and ATP systems to tackle increasingly complex mathematical challenges becomes more promising. The implications of this research extend beyond theoretical mathematics, potentially impacting fields such as computer science, artificial intelligence, and beyond.
Conclusion
In conclusion, the Discover And Prove framework represents a significant leap forward in automated theorem proving, pushing the boundaries of what is possible for LLMs in solving complex mathematical problems. As the research community continues to explore these new methodologies, the future of ATP looks increasingly bright.