Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation
Summary: arXiv:2604.00265v1 Announce Type: cross
Introduction
In the rapidly evolving field of artificial intelligence, the need for reliable benchmarks is paramount. The recent introduction of the Question-Asking Navigation (QAsk-Nav) benchmark addresses this need by providing a reproducible framework for Collaborative Instance Object Navigation (CoIN). This innovative benchmark allows for a clear and distinct evaluation of both embodied navigation and collaborative question-asking capabilities, marking a significant advancement in the intersection of AI navigation and human interaction.
What is Collaborative Instance Object Navigation (CoIN)?
CoIN involves an embodied agent tasked with reaching a specific target that is defined through free-form natural language. This process occurs under conditions of partial observability, relying solely on egocentric visual observations and interactive dialogue with a human. The dialogue component serves to resolve ambiguities that may arise from visually similar object instances, enhancing both the efficiency and effectiveness of the navigation task.
Limitations of Existing Benchmarks
Prior benchmarks in the CoIN domain have primarily concentrated on assessing navigation success. However, they have not facilitated a systematic evaluation of the collaborative interaction aspect, which is crucial for understanding how well an AI can work alongside human users. QAsk-Nav addresses this critical gap by introducing several key features:
- Lightweight Question-Asking Protocol: QAsk-Nav includes an independent scoring system for the question-asking component, allowing for a comprehensive evaluation of collaborative interaction.
- Enhanced Navigation Protocol: The benchmark provides a diverse array of realistic and high-quality target descriptions, enriching the context and challenges faced by the AI agents.
- Open-Source Dataset: QAsk-Nav boasts an extensive dataset comprising 28,000 quality-checked reasoning and question-asking traces, offering valuable resources for training and analyzing the interactive capabilities of CoIN models.
Introducing Light-CoNav
Using the QAsk-Nav benchmark, researchers have developed Light-CoNav, a lightweight unified model for collaborative navigation. This model is not only significantly smaller—three times less in size—but also remarkably faster, achieving speeds up to 70 times faster than existing modular methods. More impressively, Light-CoNav has demonstrated superior performance in generalizing to unseen objects and environments, outperforming current state-of-the-art CoIN approaches.
Conclusion
The QAsk-Nav benchmark represents a significant step forward in the field of AI, particularly in enhancing the collaborative capabilities of embodied agents. By focusing on both navigation and interaction, this benchmark opens new avenues for research and development in AI, ultimately leading to more effective and human-like interactions. For further details, visit the project page at https://benchmarking-interaction.github.io/.