MI-QUALITY: Machine Interpreting Quality Assessment and Linguistic Evaluation for AI Speech Translation

Machine interpreting (MI) refers to AI-driven systems designed to perform real-time translation of spoken language, leveraging technologies such as automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). These technologies, driven by advancements in AI, are rapidly transforming multilingual communication across industries. Despite its growing integration in a variety of contexts—including business, diplomacy, and healthcare—MI remains challenged by issues of error propagation, linguistic variability, and issues in capturing cultural nuance, contextual meanings, and idiomatic expressions. While MI tools demonstrate potential, significant gaps remain in systematically evaluating their quality, particularly in comparison with professional human interpreters. This research seeks to answer the main question: How does the quality of speech-to-speech automated translation ("machine interpreting") compare to human interpreting across linguistic pairs, and what are the implications for AI integration in multilingual communication? In so doing, this project aims to bridge this gap in knowledge on human and machine interpreting quality by conducting a multidimensional evaluation of MI and human interpreting outputs across five language pairs—Arabic-English, Farsi-English, Greek-English, Spanish-English, and Italian-English—employing a framework that assesses dimensions including linguistic accuracy, contextual integrity, fluency, and technical performance. Speech materials, sourced or created by the research team, will be interpreted by both selected MI systems (Google Interpreter’s Mode and either KUDO AI or Interprefy Aivia) and human professionals, and the outputs will be analyzed using Advanced Natural Language Processing software (OpenAI GPT API) and expert human evaluations. This project contributes to understanding MI's capabilities while addressing its limitations, and the practical implications of integrating automated language technologies into multilingual communication. By systematically evaluating MI and human interpreting, the study aims to enhance quality standards and inform the responsible use of AI in its usage in multilingual settings. Aligned with Qatar Foundation’s and HBKU’s research priorities on AI in Society, and feeding into Qatar National Digital Agenda 2030, the study supports Qatar's strategic goals of becoming a leader in technological innovation and cross-cultural communication. Practical outcomes include the development of a novel evaluation framework, insights into MI capabilities, and guidelines for quality assessment, benefiting industries and communities reliant on multilingual communication. Academic outputs will include high-impact publications, a workshop, and knowledge contributions to translation studies, human-machine interaction, and computational linguistics. By bridging AI advancements with human linguistic expertise, this project advances knowledge at the intersection of technology and society, enhancing Qatar’s role in shaping the future of AI-driven communication.

Hamad Bin Khalifa University (HBKU)