Beyond Accuracy: Assessing the Trustworthiness of Deep Learning Models for Coronary CT Angiography

The use of high-performing deep learning models in clinical settings raises concerns about trust, especially because these models often lack interpretability. In this study, we fine-tuned four different architectures: DenseNet121, InceptionV3, InceptionResNetV2, and ViT-B/16. for the detection of atherosclerosis on coronary CT angiography (CCTA) and jointly evaluated their predictive performance and explainability. Using k-fold cross-validation and held-out test data, DenseNet121 and ViT-B/16 achieved higher accuracy, precision, and recall than the Inception models; ViT-B/16 reached a test accuracy of 96.17%, followed by DenseNet121 with 95.80%, and both significantly outperformed the Inception architectures in statistical comparisons. We applied multiple XAI techniques, including LIME, SHAP, and Integrated Gradients, to characterize how each model arrived at its predictions. DenseNet121 provided localized, vessel-specific saliency focused on clinically relevant coronary segments, whereas ViT-B/16 displayed more holistic, patch-level attention that captured broader vascular context while maintaining strong predictive performance. This combined performance-interpretability analysis advances trustworthy AI for coronary artery disease detection on CCTA by linking automated predictions to clinically meaningful image patterns and supporting future development of explainable decision-support tools in cardiology.