Data-Driven Automated Reporting Solution for External Collaborations - LLM-driven KPI Definition

Sammanfattning

This thesis presents a proof-of-concept, developed with AstraZeneca (AZ), that exploresautomating progress reporting for external collaborations by testing whethera large language model (LLM)-driven system can extract objectives from contractsand translate them into tailor-made key performance indicators (KPIs). Objectiveextraction is quite reliable, reaching several highs of accuracy around the 85%-mark,but converting objectives into KPIs that stakeholders judge as relevant, clear, actionable,and measurable, is substantially less solid. Fewer than half of the KPIsmet each quality criterion on average, and 39% met none. Survey responses notedthat KPIs were often unclear, overly generic, or poorly timed, and skewed towardsimple counts (e.g., “number of models”) that miss quality and impact.From interviews conducted at AZ, a set of general KPIs, that were deemed meaningfulto measure in a collaboration project, could be demonstrated. The final evaluationsuggests that these KPIs (e.g., external engagement and budget coherence)outperform collaboration-specific KPIs generated directly from objectives. This underscoresthe difficulty of creating bespoke target measures in diverse contexts.Despite these issues, the approach offers practical value. In principle, the pipelineshould be better suited for agreements with explicit milestones (e.g., business orcommercialisation contracts), where more clearly defined expected outcomes supportbetter-formed KPIs. However, this cannot be conclusively established by theimplementation in this thesis, due to limited data.Ultimately, translating qualitative objectives into quantitative, decision-grade KPIsremains inherently difficult. Contemporary LLMs are capable across many aspects ofautomation, but evidently less reliable for high-judgement and context-specific KPIdesign that balances relevance, clarity, actionability, and measurability, at least byfollowing the approach outlined in this thesis. Therefore, the most defensible neartermusefulness is in metadata extraction and recommendation, while still requiringa human-in-the-loop as a safeguard. In turn, this can improve customer relationshipmanagement (CRM) metadata completeness and enable collaboration healthinsights and automated reporting.