Development and Evaluation of an Agentic LLM-based Rag-enabled Framework for Evidence-based Patient Education

Abstract

Patient Education (PE) is crucial for empowering individuals in managing their health, yet significant challenges persist in delivering effective and accessible Patient Education Materials (PEMs), especially for diverse linguistic populations such as Arabic speakers. Large Language Models (LLMs) offer a promising avenue to address these challenges. This thesis details the development and evaluation of the Amal Framework, an LLM-based agentic retrieval-augmented generation (ARAG) system designed to produce evidence-based PEMs, with a specific application for the Arabic language. The name Amal in Arabic translates to "hope" in English, it was chosen to reflect the project’s aspiration to inspire optimism and provide reliable support for individuals navigating their health journeys. The research first establishes a foundational understanding of LLMs, Retrieval Augmented Generation (RAG), prompt engineering, and agentic workflows. It then presents a comprehensive scoping review of the current use of LLMs in patient education, identifying key trends and research gaps, notably the underutilization of RAG and opensource models for non-English PEMs, and the lack of standardized evaluation. The core of this work lies in the design and implementation of the Amal Framework. This involved creating a robust data pipeline, which included sourcing and processing a substantial corpus of Arabic medical texts from reputable sources. A multi-component generation pipeline was developed, integrating carefully selected open-source LLMs with the ARAG system and a novel Validation Agent (VA) designed to ensure the safety, accuracy, and appropriateness of the generated PEMs. A rigorous two-part evaluation methodology was devised. The first part assessed the quality of the generated Arabic PEMs across multiple experimental setups, using automated LLM-based scoring and subsequent expert validation on metrics including accuracy, readability, comprehensiveness, appropriateness, and safety. The second part specifically evaluated the performance of the VA in identifying and blocking harmful or unsuitable medical advice using a specialized benchmark dataset. The findings from these evaluations demonstrate the efficacy of the Amal Framework in generating high-quality, evidence-based Arabic PEMs and the effectiveness of the integrated VA in enhancing safety. This research contributes a novel, systematically evaluated framework for leveraging advanced AI to improve patient education, particularly for underserved linguistic communities, and offers a blueprint for the responsible and effective deployment of LLMs in healthcare. The thesis concludes by discussing the implications of these findings, the limitations of the current work, and directions for future research in this rapidly evolving field.

Date of Award	2025
Original language	American English
Awarding Institution	HBKU College of Science and Engineering