Automated Prompt Optimization for LLM-based Test Update Localization

Sammanfattning

Maintaining consistency between production code and its associated test cases isa critical, yet costly, task in software development. As systems evolve, identifyingwhich tests require updates becomes increasingly complex. Recently, Large Language Models (LLMs) have shown great potential for automatically localizing testcases. Yet the performance of existing LLM-based automated approaches is stronglyaffected by how prompts are formulated. While prior work has shown that promptengineering can significantly influence LLM behavior, many existing approaches relyon static, manually crafted prompts that do not adapt to varying contexts.Our thesis proposes an automated prompt optimization approach, named Adaptive Reasoning-guided Gradient-based Optimization (ARGO), for improving LLMbased test update localization in a multi-agent pipeline. Although several automatedprompt optimization paradigms exist, our study focuses specifically on text-gradientbased optimization as a way to automatically revise prompts based on feedback fromlocalization errors. As part of this investigation, it also examines whether incorporating configurable prompt strategies, such as structural prompt modifications, canfurther guide the optimization process. The approach is integrated into an existingtest localization framework and evaluated using a ground-truth dataset of real-worldcode changes and corresponding test updates.Our evaluation compares automatically optimized prompts against manually craftedbaseline prompts across multiple real-world repositories, while accounting for thenon-deterministic nature of LLM outputs. The analysis considers not only the directeffect of prompt optimization on the target module, but also how changes in thatmodule influence downstream collaborating modules in the multi-agent pipeline.The results demonstrate that adaptive and structurally-aware prompt optimizationcan improve the performance and practical applicability of LLM-based test updatelocalization, contributing to more reliable automation in software maintenance workflows.