A small-scale exploratory case study of maintainability-related metrics of LLM-generated C++ code

Sammanfattning

Context: LLM-generated code is increasingly common, and as such, maintaining such code is more important than ever before. Existing research on LLM-generated code mostly focuses on Python and rarely evaluates the maintainability of C++ code. Objective: This small-scale exploratory case study aims to evaluate maintainability-related metrics of LLM-generated code in C++ and compare it to human-written code. Additionally, investigate how different models and prompts may affect maintainability-related metrics of the code. Methods: We conducted an experiment where a task was designed from a GitHub pull request for the LLMs to solve. The code from the original pull request was used as a human-written baseline that was compared to the LLM-generated code. We evaluated several maintainability-related metrics proposed by several researchers. Results: The results of this thesis show that LLM-generated code can be outperformed by human-written code in certain metrics and can perform similarly or surpass human-written code in others. Gemini generated code with far more SonarQube code smells and with more complexity than ChatGPT. However, Gemini with the maintainability-focused prompt wrote fewer lines of code compared to ChatGPT. The basic prompt and maintainability-focused prompt yielded mixed results; for example, while WMC-MCabe and LOC had better scores for the maintainability-focused prompt, the LCOM2 score was worse. The different models for the maintainability-focused prompt also had mixed results with other metrics, where one scored better while the other scored worse compared to the attempt with the basic prompt. Conclusions: Our results suggest that it is preferred to have human experts write the code instead of the models evaluated in this study. It is also important that a user is aware of the capabilities of the model they are using, as it may have a noticeable impact on the maintainability-related metrics of the code. Additionally, LLM-generated code can be influenced by a maintainability-focused prompt; however, not necessarily in a way that is more maintainable across all metrics.