A Requirements-Aware Prompting Framework for LLM-Based Code Generation - Design and Empirical Evaluation

Sammanfattning

Large language models (LLMs) are increasingly used to generate software code fromnatural-language descriptions, yet the quality of the input requirement is rarelytreated as an independent variable in such systems. In practice, software requirements are often ambiguous, incomplete, inconsistent, or difficult to verify, creatinguncertainty for LLM-based code generation. This thesis presents a requirementsaware prompting framework grounded in the ISO/IEC/IEEE 29148 requirementsquality standard and empirically evaluates how requirement quality and promptstructure influence LLM behaviour during code generation.The proposed framework operationalises four requirement-quality attributes: unambiguity, completeness, verifiability, and consistency by incorporating them intostructured prompts through quality-aware constraints and behavioural routing policies. Six complementary empirical studies were conducted across three large language models (GPT-4o-mini, GPT-4.1, and Claude Sonnet 4.6) using both benchmark and real-world software engineering datasets. Benchmark evaluation includedHumanEval, a collection of 164 Python programming tasks with hidden unit tests,and MBPP Sanitized, a benchmark of 427 short programming problems, out ofwhich 413 were evaluated under synthetically degraded requirement-quality conditions intended as a methodological stress test. Real-world evaluation used thePURE corpus, comprising 2,503 natural-language software requirements extractedfrom publicly available Software Requirements Specification (SRS) documents, with1,000 requirements selected for behavioural analysis, alongside a corroborative SRSdataset of 47 industrial-style requirements derived from the FarmBot system.Results indicate that prompt structure has limited influence on functional correctness of the LLM generated code. Across HumanEval and MBPP, compiledconstraints prompting produced no statistically significant improvement in passrates. However, the MBPP quality degradation procedure was methodologicallylimited, as benchmark tasks retained intact behavioural specifications and embedded examples despite degradation annotations. Consequently, benchmark findingsare interpreted cautiously and do not constitute evidence that requirement qualityis irrelevant to code generation correctness. Nonetheless, substantial behaviouralshifts emerged in LLM output behaviour when the framework was applied to realworld software requirements. On the PURE dataset, compiled prompts increaseddiagnostic outputs from near-zero levels under baseline prompting to between 63%and 73% across evaluated models, with Claude Sonnet 4.6 producing the strongestdiagnostic response (73.4%). Similar behavioural patterns were observed on the independent FarmBot SRS dataset, providing preliminary cross-domain corroborationof the PURE findings.Requirement-level behavioural correlations and consistency analyses indicate systematic relationships between requirement quality and LLM output behaviour, particularly the transition between implementation and diagnostic responses, with unambiguity showing the strongest observed association and verifiability appearingtheoretically important but less reliably assessed in the automated grading pipeline.Because the framework used pre-defined quality thresholds to determine whetherrequirements should be implemented or diagnosed, logistic regression is interpretedprimarily as a validation of the framework’s internal logic routing consistency, whilecross-model behavioural comparisons provide the main evidence for how requirementquality influences LLM output behaviour.The findings establish a clear distinction between correctness and behaviour in LLMbased code generation. While prompt design does not significantly improve correctness for well-specified benchmark tasks, it provides a powerful mechanism for controlling model behaviour when requirements are quality-defective. The proposedframework enables quality-aware interaction with LLMs by identifying defectiverequirements before implementation proceeds, thereby supporting safer and morereliable integration of LLMs into software engineering workflows.