Evaluating Explainability Techniques for Machine Learning in Healthcare - A Human-Centered Approach through Expert Interviews

Sammanfattning

Background Machine learning (ML) has significant potential to improve healthcare by enhancing diagnostic accuracy, supporting earlier diagnoses and optimizing treatment strategies. However, the lack of transparency in many ML models can hinder their adoption in clinical practice. Explainable Artificial Intelligence (XAI) has been proposed as a solution to this challenge, aiming to make model outputs more interpretable and trustworthy. Yet, little is known about how different stakeholders perceive and interact with XAI in healthcare contexts. Aim This study aimed to explore how different explainability techniques – specifically Local SHAP, Global SHAP, and Attention Mechanism – influence stakeholders’ trust, usability, and decision-making in healthcare, with a focus on two clinical domains: sepsis-related mortality and psychology. Method A qualitative, cross-sectional design was employed. Semi-structured interviews were conducted with 20 participants across three stakeholder groups: clinical experts and researchers, data scientists, and members of the general public. Participants evaluated visualizations of XAI techniques based on real ML prediction tasks. Thematic analysis was used to identify key patterns in the data. Results Five themes were identified. Participants emphasized that trust in XAI systems depends not only on explainability but also on factors such as perceived reliability, contextual relevance, and human oversight. Local SHAP was seen as the most intuitive format, while Global SHAP and Attention Mechanism were often viewed as too abstract. Across all groups, AI/XAI was valued as a support tool, not a replacement for professional judgment. Adoption was seen as contin- gent on tailored explanations, workflow integration, training, and interdisciplinary collaboration. Conclusion Explainability alone is insufficient to ensure trust, usability, or effective decision-making. Advancing XAI in healthcare requires human-centered design, adaptive explanation formats, and institutional readiness. This study offers one of the first qualitative, multi-stakeholder evaluations of XAI across clinical domains, providing practical guidance for developing systems that are both technically robust and meaningful to end-users.