Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition
Summary: arXiv:2604.09121v1 Announce Type: cross
Abstract
Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale training data. However, two important aspects remain underexplored. First, Word Error Rate (WER), the dominant evaluation metric for decades, treats all words equally and often fails to reflect the semantic correctness of an utterance at the sentence level. Second, interactive correction—an essential component of human communication—has rarely been systematically studied in ASR research.
Integrating Perspectives for Enhanced ASR
In this paper, we integrate these two perspectives under an agentic framework for interactive ASR. We propose leveraging LLM-as-a-Judge as a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy. This marks a significant shift in how ASR systems evaluate performance, moving towards a more nuanced understanding of language meaning and context.
Designing Human-like Interaction
Furthermore, we design an LLM-driven agent framework to simulate human-like multi-turn interaction. This framework enables iterative refinement of recognition outputs through semantic feedback, allowing ASR systems to learn and adapt from their interactions. The ability to engage in dialogue and self-correct is crucial for creating more effective and responsive speech recognition systems.
Experimental Validation
Extensive experiments are conducted on standard benchmarks, including:
- GigaSpeech (English)
- WenetSpeech (Chinese)
- ASRU 2019 Code-Switching Test Set
Both objective and subjective evaluations demonstrate the effectiveness of the proposed framework in improving semantic fidelity and interactive correction capability. Our results indicate that incorporating semantic awareness in ASR not only enhances recognition accuracy but also leads to more coherent and meaningful interactions.
Future Directions
We are committed to advancing research in this field and will release the code to facilitate future exploration in interactive and agentic ASR. By providing access to our framework, we hope to inspire further studies that will build on our findings and contribute to the development of more sophisticated speech recognition technologies.
Conclusion
The integration of semantic coherence evaluation and interactive correction into ASR systems represents a crucial step towards achieving human-like interaction in automated speech systems. As technology continues to evolve, the focus on semantic understanding and user engagement will be paramount in shaping the future of speech recognition.