Reliability of Large Language Models for Transportation Applications

As the use of LLMs is becoming central to organizational functioning, it is necessary that the output of LLMs be robust, reliable and resilient. One way to enhance reliability is to quantify the uncertainty of their output. There are two forms of uncertainties (i) related to the inherent ambiguity of the prompt and (ii) lack of knowledge in LLMs. In this project we will quantify both forms of uncertainty and propose a reliability score associated with each prompt output. The reliability score will be then integrated into an open-source LLM (including Fanar) and used for applications of interest to the Ministry of Transportation.

Hamad Bin Khalifa University (HBKU)