FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
Summary: arXiv:2604.03893v1 Announce Type: new
Abstract
Breakthroughs in frontier theory often depend on the combination of concrete diagrammatic notations with rigorous logic. While multimodal large language models (MLLMs) show promise in general scientific tasks, current benchmarks often focus on local information extraction rather than the global structural logic inherent in formal scientific notations. In this work, we introduce FeynmanBench, the first benchmark centered on Feynman diagram tasks.
Introduction
FeynmanBench is designed to evaluate AI’s capacity for multistep diagrammatic reasoning, which involves:
- Satisfying conservation laws and symmetry constraints
- Identifying graph topology
- Converting between diagrammatic and algebraic representations
- Constructing scattering amplitudes under specific conventions and gauges
Methodology
To support large-scale and reproducible evaluation, we developed an automated pipeline that produces diverse Feynman diagrams. This pipeline is accompanied by verifiable topological annotations and amplitude results, ensuring a comprehensive dataset.
Database Overview
Our database spans various interactions within the Standard Model of particle physics, including:
- Electromagnetic interactions
- Weak interactions
- Strong interactions
It encompasses over 100 distinct types of Feynman diagrams and includes more than 2000 tasks, providing a robust foundation for benchmarking.
Experiments and Findings
Experiments conducted on state-of-the-art MLLMs have revealed several systematic failure modes, including:
- Unstable enforcement of physical constraints
- Violations of global topological conditions
These findings underscore the necessity for physics-grounded benchmarks that rigorously test visual reasoning capabilities over scientific notation.
Conclusion
FeynmanBench provides a logically rigorous test of whether AI can effectively engage in scientific discovery, particularly within the realm of theoretical physics. By addressing the limitations of current benchmarks, we aim to enhance the performance and reliability of MLLMs in tackling complex scientific problems.
Future Work
As AI continues to evolve, further research will be essential to refine benchmarks like FeynmanBench. Future iterations may include:
- Expanded datasets covering additional areas of physics
- Enhanced algorithms that better capture the intricacies of diagrammatic reasoning
- Collaboration with physicists to ensure the relevance and accuracy of benchmarks
We believe that FeynmanBench will play a pivotal role in advancing the capabilities of MLLMs in scientific reasoning, paving the way for future innovations in AI-assisted research.