Interactive Multimodal AI for High-precision Medical Diagnosis

The proposed project aims to advance medical artificial intelligence (AI) by developing an interactive multimodal AI system capable of performing high-precision diagnostic tasks guided by natural language instructions. This research addresses a critical challenge in AI-assisted medicine: the need for systems that not only process multimodal data (e.g., vision and text) but also interact with clinicians to perform complex visual reasoning tasks, such as identifying, segmenting, and counting medical structures from clinical images. Modern diagnostic workflows increasingly depend on image-intensive procedures, ranging from histopathological slides and radiology scans to microscopic cell images. However, current AI systems are typically static, task specific, non-interactive and heavily dependent on large amounts of manually labeled datasets. Moreover, they often lack transparency and adaptability to specific diagnostic questions posed by physicians. To this end, this project addresses these limitations by creating a general-purpose interactive AI model that interprets language-based instructions and performs corresponding visual analysis tasks in a clinical setting.

Hamad Bin Khalifa University (HBKU)