Data-Driven Evaluation of Clinical Workflow Efficiency in AI-Enabled Digital Health Platforms : AI Efficiency in Digital Health Platforms

Sammanfattning

Documentation in healthcare imposes a substantial administrative burden on clinicians. With research indicating that doctors spend at least two hours daily on electronic health records (EHRs) outside of patient care. Clinvvo is an artificial intelligence (AI)-based clinical documentation platform that was used throughout Swedish primary care, psychiatry and gynecology. It uses Whisper-based Automatic Speech Recognition (ASR) and Embedding-Driven Natural Language Generation (NLG) to automate the transcribing and structuring of notes. Although Clinvvo processed over 500,000 consultations annually since it was operationally implemented in Sweden, no systemic performance evaluation has occurred on the system to date, therefore preventing the opportunity to use the results to optimize the system through an evidence-based approach. This research addresses the measurement gap in the evaluation of clinical AI systems by developing a "Measurement First" framework for designing a data collection infrastructure for deployed clinical AI systems within the bounds of the General Data Protection Regulation (GDPR). The four objectives guiding this research project are: (1) a complete data inventory of available sources of data (identifying 19 potential sources of data) to support the development of the required metrics; (2) the formal definition of 12 metrics across three dimensions of ASR, NLG, and Workflow; (3) an implementation assessment demonstrating that 97% of timestamps for consultations were captured across 384 consultations from 19 clinicians; and (4) the phased implementation roadmap of a balanced approach between preserving clinician privacy while achieving complete measurement of the data collected. Clinvvo proved effective in practice, successfully capturing 91% of transcripts and 68% of clinician feedback without disrupting existing workflows. Primary care clinicians completed templates at a rate of 80% (an average of 4.25 out of five stars) compared to psychiatrists who had a completion rate of 62% (an average of 3.95 out of five stars) and identified the need for specialty specific redesign. This research provides a method for evaluating AI-assisted documentation platforms that is generally applicable, and also identifies how to provide actionable guidance for complying with the AI Act through the design of measurement infrastructure before the AI system is deployed, not after the system has been implemented.