Test-Driven Development Using LLM : A look into LLMs writing tests in a test-driven development workflow

Sammanfattning

This research aims to explore different Large Language Models (LLM) and drawconclusions as to how well they can generate edge cases and respective unit tests in aTest-Driven Development (TDD) workflow. The experiment is concluded with LLMsthat are of the type Generative Pre-trained Transformers (GPT). The different LLMsthat are used for this research are ChatGPT 4 (GPT-4), ChatGPT 3.5 (GPT3.5) andGithub Copilot (Codex/GPT-4). The experiment consists of two phases, one where theprompt goes through prompt engineering with different methods such as few-shot andChain of Thought (COT). The second phase is where the prompt is sent to thedifferent LLMs to generate edge cases and unit tests. The prompt and questions thatare sent and asked to the LLM are taken from Advent of Code (AoC) which is aprogramming event every year. The LLMs performed differently depending on thetype of question from AoC. ChatGPT 3.5 performed better during text heavyquestions, ChatGPT 4 had an average performance throughout all the questions andGithub Copilot performed better towards the end where the questions became morecomplex and more programmatically focused.