Understanding How Developers Use LLMGenerated Refactoring Suggestions

Sammanfattning

Large language models (LLMs) have gainedwidespread popularity and have steadily improved over time,enabling software developers to use them for coding tasks. Onecommon task is code refactoring, where the chatbot suggestschanges for the developer to apply. Although current researchfocuses on evaluating LLM-generated refactoring suggestions,there is a limited understanding of how developers implementthese suggestions in practice. To explore this, we analyze GitHubcommits where developers use LLMs along with their linkedChatGPT (GPT-3.5 or GPT-4) conversations. We found thatmost developers use ChatGPT’s suggestions without modification,indicating their trust in the technology. This study presentsmethods that future research can use to explore LLM-developerinteractions and evaluate the effectiveness of other LLMs inrefactoring and other code-related tasks. For evaluation we haveimplementations in our Github repository.Index Terms—Large Language Models (LLMs), ChatGPT,Code Refactoring, GitHub, Commit, AI-Assisted Development