Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?
Summary: arXiv:2604.03016v1 Announce Type: new
Abstract: Multimodal Large Language Models (MLLMs) are evolving from passive observers into active agents, solving problems through Visual Expansion (invoking visual tools) and Knowledge Expansion (open-web search). However, existing evaluations fall short: they lack flexible tool integration, test visual and search tools separately, and evaluate primarily by final answers. Consequently, they cannot verify if tools were actually invoked, applied correctly, or used efficiently. To address this, we introduce Agentic-MME, a process-verified benchmark for Multimodal Agentic Capabilities.
The Agentic-MME benchmark contains 418 real-world tasks across 6 domains and 3 difficulty levels to evaluate capability synergy. It features over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task. Each task includes a unified evaluation framework supporting sandboxed code and APIs, alongside a human reference trajectory annotated with stepwise checkpoints along dual-axis: S-axis and V-axis.
Why Agentic-MME Matters
As MLLMs continue to progress, understanding their capabilities becomes increasingly significant. Traditional evaluations typically concentrate on the final outcomes, neglecting the processes and steps undertaken to reach those conclusions. Agentic-MME aims to fill this gap by emphasizing the importance of process verification.
Key Features of Agentic-MME
- Real-World Task Variety: The benchmark encompasses a wide range of tasks, ensuring comprehensive evaluation across multiple domains.
- Stepwise Checkpoints: With over 2,000 checkpoints, researchers can gain insights into the decision-making processes of MLLMs.
- Unified Evaluation Framework: The framework facilitates the use of sandboxed tools and APIs, allowing for flexible integration and real-time testing.
- Dual-Axis Evaluation: The S-axis and V-axis provide a nuanced understanding of agentic capabilities and the effectiveness of tool usage.
Challenges in Multimodal Agentic Problem Solving
Despite the advancements, challenges remain. Experimental results indicate that even the best-performing model, Gemini3-pro, achieves only 56.3% overall accuracy, with a significant drop to 23.0% on Level-3 tasks. This underscores the inherent difficulty in real-world multimodal agentic problem solving.
Conclusion
Agentic-MME represents a significant step towards enhancing the evaluation of multimodal intelligence. By focusing on process verification and providing a robust framework, it seeks to ensure that MLLMs evolve into true problem-solving agents. The future of artificial intelligence lies not only in their ability to produce correct answers but also in the efficiency and correctness of the processes they employ to arrive at those answers.