Towards Knowledgeable Deep Research: Framework and Benchmark
Summary: arXiv:2604.07720v2 Announce Type: replace
Abstract: Deep Research (DR) requires LLM agents to autonomously perform multi-step information seeking, processing, and reasoning to generate comprehensive reports. In contrast to existing studies that mainly focus on unstructured web content, a more challenging DR task should additionally utilize structured knowledge to provide a solid data foundation, facilitate quantitative computation, and lead to in-depth analyses.
In this paper, we refer to this novel task as Knowledgeable Deep Research (KDR), which requires DR agents to generate reports with both structured and unstructured knowledge. Furthermore, we propose the Hybrid Knowledge Analysis framework (HKA), a multi-agent architecture that reasons over both kinds of knowledge and integrates the texts, figures, and tables into coherent multimodal reports.
Key Features of the Hybrid Knowledge Analysis Framework
The key design is the Structured Knowledge Analyzer, which utilizes both coding and vision-language models to produce figures, tables, and corresponding insights. This innovative approach enables DR agents to enhance their analytical capabilities significantly.
KDR-Bench: A Comprehensive Evaluation Tool
To support systematic evaluation, we construct KDR-Bench, which covers the following:
- 9 domains
- 41 expert-level questions
- A large number of structured knowledge resources (e.g., 1,252 tables)
This comprehensive benchmark is designed to evaluate the performance of DR agents effectively.
Evaluation Metrics
We further annotate the main conclusions and key points for each question and propose three categories of evaluation metrics including:
- General-purpose metrics
- Knowledge-centric metrics
- Vision-enhanced metrics
These metrics allow for a nuanced assessment of DR agents’ capabilities in handling both structured and unstructured knowledge.
Experimental Results and Findings
Experimental results demonstrate that HKA consistently outperforms most existing DR agents on general-purpose and knowledge-centric metrics. Notably, it even surpasses the Gemini DR agent on vision-enhanced metrics, which highlights its effectiveness in deep, structure-aware knowledge analysis.
Conclusion
In conclusion, we hope this work can serve as a new foundation for structured knowledge analysis in DR agents and facilitate future multimodal DR studies. The implications of our findings are significant, as they advance the capabilities of AI in conducting comprehensive and informed research autonomously.