Intermediate Representations for LLM-Based Test Generation: A Comparative Study in Automotive Software Verification

Sammanfattning

Large Language Models (LLMs) can generate test scripts from natural-languagerequirements, but direct generation often misses boundary behaviors, fault handling, and requirement-specific signal details. We investigate whether intermediaterepresentations (IRs)—artifacts inserted between requirements and executable testscripts—can improve the completeness, traceability, and practical utility of LLMbased test generation in an industrial automotive context.We design a unified four-path framework: three generation paths (A–C) that varyonly the IR type while holding constant the LLM family, discovery infrastructure,and output format—no IR (direct generation), a structured JSON reasoning document, and an executable Python state-machine specification (xSpec)—and one augmentation path (D) that uses the executable IR in a separate enhancement mode toidentify gaps in existing human-authored tests rather than generating test scriptsfrom scratch. The framework is evaluated on 31 functional requirements from VolvoGroup’s body/cabin electronics domain, with three runs per configuration to accountfor LLM nondeterminism.Results show that introducing IR improves static test-script quality—as measured bybehavioral completeness and requirement traceability—over direct generation. Allquality assessments are based on static analysis of test-script content. ComparingIR types, the executable IR outperforms the structured IR on traceability, but thetwo are statistically indistinguishable on completeness, signal correctness, and valuecorrectness—indicating that structuring the LLM’s reasoning provides most of themeasurable benefit, with executability adding a targeted advantage. The augmentation path preserves 94.7% of existing human-authored behaviors while identifyingcoverage gaps with 87.5% precision and 63.2% recall, and a median coverage improvement of +0.302. Most identified gaps (62.5%) are fault-condition behaviors,the category where existing test suites are weakest.These findings support a staged adoption strategy: structured IRs for immediatequality gains, executable IRs where traceability matters most, and IR-guided augmentation to extend existing trusted tests rather than replace them.