Clinicians’ Explainability Requirements for AI Decision-Support Systems in Emergency Care

Sammanfattning

Introduction: As artificial intelligence decision-support systems are increasingly emerging, questions arise regarding what clinicians require from such systems. Explainability, the degree to which a system’s reasoning and suggestions are transparent and interpretable, has been posed as a central factor in clinical adoption. However, little is known about what explainability requirements clinicians themselves express in emergency care settings. Research Question: The primary research question of this thesis is: What explainability requirements do clinicians have when using AI decision-support systems in emergency care? Method: In order to gather data about requirements, the study adopted a qualitative approach, with semi-structured interviews conducted with ten participants across varying roles and experience levels in emergency and pre-hospital care. Data were analysed using a thematic analysis. Results: Four themes were identified: strengthening trust through system features, usability, clinical relevance, and human-AI role alignment. Participants required explanations that allow independent verification of AI reasoning, communicate system limitations and fragility of suggestions, and are delivered concisely within time-critical environments. Explanations were further required to be contextually relevant and adaptable to clinical role and experience. Across all themes, clinicians positioned AI as a supportive tool second to their own and others’ judgment, with professional responsibility remaining with the clinician despite any explainability measures. Discussion: The findings suggest that current established explainability methods such as SHAP and LIME do not meet the requirements that clinicians themselves express, particularly regarding situational inapplicability, fragility of suggestions, iterative interaction and references to peer-reviewed sources. Safe and effective integration of AI decision-support systems in emergency care may therefore require moving beyond model-centric explainability towards more user-centred, context-dependent design. The study is limited by its small sample size and primarily Swedish healthcare context, which restricts generalisability. Further research should validate these findings across larger and more diverse populations and evaluate identified requirements in real clinical environments.