INTERACT: An AI-Driven Extended Reality Framework for Accessible Communication Featuring Real-Time Sign Language Interpretation and Emotion Recognition
Video conferencing has become a cornerstone of professional collaboration in today’s digital landscape. However, existing platforms often fall short in providing adequate support for deaf, hard-of-hearing, and multilingual users. According to the World Health Organisation, over 430 million individuals globally require rehabilitation for disabling hearing loss, a number projected to rise above 700 million by 2050. Traditional accessibility measures have been hampered by high costs, limited availability, and various logistical barriers.
In response to these challenges, Extended Reality (XR) technologies present innovative avenues for fostering immersive and inclusive communication. The recent paper titled “INTERACT” introduces an AI-driven XR platform designed specifically to enhance accessibility in communication. The platform integrates real-time speech-to-text conversion, International Sign Language (ISL) rendering through 3D avatars, multilingual translation, and emotion recognition within an immersive virtual environment.
Key Features of INTERACT
- Real-Time Speech-to-Text Conversion: Utilizing advanced speech recognition technology, INTERACT converts spoken language into written text on the fly, making it accessible for users with hearing impairments.
- 3D Avatars for ISL Rendering: The platform employs lifelike 3D avatars to represent sign language interpreters, facilitating better communication for deaf users.
- Multilingual Translation: INTERACT incorporates multilingual translation capabilities, ensuring effective communication across diverse language speakers.
- Emotion Recognition: By employing AI algorithms, the platform can identify and interpret user emotions, enhancing the contextual relevance of interactions.
- Immersive Virtual Environment: Built on the CORTEX2 framework and deployed on Meta Quest 3 headsets, users can engage in a more interactive and immersive communication experience.
Technical Implementation
INTERACT leverages a suite of cutting-edge technologies to deliver its features. It incorporates the following:
- Whisper: An advanced speech recognition system that ensures accurate transcription of spoken language.
- NLLB: A multilingual translation model that facilitates communication between speakers of different languages.
- RoBERTa: This AI model is used for emotion classification, enabling the platform to detect and interpret user emotions effectively.
- Google MediaPipe: Employed for gesture extraction, this technology aids in the rendering of sign language through the avatars.
Pilot Evaluations and User Feedback
Pilot evaluations of INTERACT were conducted in two phases. The first phase involved technical experts from academia and industry, followed by trials with members of the deaf community. The results were overwhelmingly positive, with:
- 92% user satisfaction reported among participants.
- Transcription accuracy exceeding 85%.
- Emotion detection precision at 90%.
- A mean overall experience rating of 4.6 out of 5.0.
- 90% of participants expressing willingness to engage in further testing.
The outcomes of these evaluations suggest a strong potential for INTERACT to advance accessibility across various sectors, including educational, cultural, and professional settings. An extended version of this study, featuring comprehensive pilot data and implementation details, has been published as an Open Research Europe article by Tantaroudas et al. (2026a).