Augmented Large Language Models for Software Engineering

Sammanfattning

Since the release of the recent generation of Large Language Models (LLMs) such as GPT-4 (GPT standing for generative pre-trained transformer), companies and individuals are exploring opportunities to improve their effectiveness by automating or receiving assistance with knowledge-intensive tasks by using artificial intelligence (AI). Within software engineering, a plethora of commercial services promising impressive results has reached the market in the past year. This thesis explores the practical implications of applying LLMs in this software engineering context. Through a case study at Craft Docs, a software development company, we investigated the current use of LLMs and identified specific opportunities to leverage these models as a way of improving the efficacy of software engineers. Action research (AR) was chosen as the research methodology, carried out by devel- oping a prototype that was deployed and evaluated at the company. A literature study and interviews surveying the use of and sentiment towards LLMs at the case company guided the development, and the prototype was evaluated with practical tests and retrospective interviews. Findings from the project suggest that while LLMs can enhance certain aspects of software engineering, their effectiveness is highly dependent on the context of use and the specific tasks they are designed to support. The prototype developed in this study demonstrates a practical implementation of retrieval-augmented generation (RAG) to provide additional context to LLMs, showing both the strengths and limitations of these technologies in real-world settings. Most apparent was that even with access to relevant code context, tasks given to an LLM need to be carefully scoped to yield effective results.