Extracting Arm-Level Statistics from Randomized Controlled Trials with LLMs : A Comparison of Zero-Shot and Few-Shot Prompting

Sammanfattning

Extracting numerical data from RCT reports is time-consuming and error-prone. While LLMs show potential on this task, previous work concluded that models are not reliable enough for research synthesis without human oversight. We tasked three multimodal models (GPT-5.2, Gemini 3 Pro, and Claude Haiku 4.5) with extracting arm-level statistics (including data from figures) from full-text PDFs, specifically addressing whether providing full-text PDFs as few-shot examples improved performance compared to a zero-shot baseline. All three models performed comparably (F1 range: 82.9%–85.5%, Exact Match range: 51.9%–58.3%). Although GPT-5.2 achieved the highest numerical point estimate for zero-shot performance (F1: 85.5%), the 95% bootstrap confidence intervals spanned zero, showing no significant differences among the models. The inclusion of full-text few-shot examples yielded no performance benefits; in fact, F1-scores decreased slightly across all models (Claude -0.1, Gemini -0.6, GPT -1.1). The 95% paired cluster bootstrap intervals effectively ruled out the possibility of substantial improvement, a result likely attributable to the distracting nature of large context windows. Crucially, a qualitative discrepancy analysis revealed that a large portion of discrepancies stemmed from external factors (e.g., errors in the gold standard, ambiguous reporting) rather than intrinsic model errors. Consequently, we consider our reported metrics to be conservative estimates of true performance and conclude that models are likely more fit for the task of assisting meta-analytic extraction than previously assumed.