Automating Behavior-Driven Development with Large Language Models : Exploring Test Case Generation for Fintech projects

Sammanfattning

Behavior-Driven Development is used to improve collaboration across technical and non-technical teams by having test scenarios written in natural language. The scenarios are written based on test requirements manually by a tester, which often demands specific domain knowledge and a significant amount of time. To address this challenge, this thesis aims to investigate the possibility of automatically generating the test scenarios from the test requirements written in natural language by leveraging large language models. This thesis was conducted at Nasdaq, a large financial technology company, and the test scenarios were generated for internal projects at the host company. The evaluated projects include a regulatory reporting system, a management platform, and one open-source project (Flask). Atesting toolwas implemented using models from Amazon Bedrock to generate behavior-driven development Gherkin-formatted test scenarios directly from requirements and source code from the projects. The source code was fetched by using Retrieval augmented generation, where the codebases had been processed and stored in a vector database. The scenarios were evaluated with quantitative metrics and qualitatively by gathering feedback from domain experts at Nasdaq. The quantitative metrics focus on syntax correctness, execution success rates, and semantic similarity between requirements, while the feedback mostly involved rating the adequacy, readability, domain relevance, and maintainability of the tests. The results indicate that the generated Gherkin-formatted scenarios can adequately capture the intended behavior described in the requirements while also demonstrating high Gherkin syntax correctness, with 89% of scenarios being syntactically valid in the worst-performing project and 96% at best. However, the pass rates of the test execution are still relatively low across projects and vary significantly, with the highest pass rate achieved at 59 % steps passing and only 13 % in the worst project. This thesis shows that behavior-driven test scenarios are possible to generate and can support testers, managers, and non-technical stakeholders by reducing testing efforts and improving communication across teams.