Analyzing AI-Generated Code Descriptions: Evaluating Clarity and Accuracy

Sammanfattning

Artificial Intelligence (AI) and Large Language Model (LLM)s are increasingly used to generate code descriptions, with the potential to improve code documentation and comprehension. These tools promise efficiency in software development but raise concerns about the clarity and accuracy of their outputs. AI-generated code descriptions can be vague, inconsistent, or misleading, limiting their usefulness for developers. Inaccurate summaries may obscure functionality and hinder understanding, especially in collaborative or educational environments. This thesis investigates the effectiveness of AI-generated descriptions by comparing them to human-written references. The aim is to assess how well these outputs represent code functionality and whether the presence of comments in source code affects the quality of the generated text. ChatGPT-4o was used to generate descriptions for C code samples, both with and without comments. Evaluations combined semantic (Sentence-BERT (Bidirectional Encoder Representations from Transformers) (SBERT)) and lexical (Recall-Oriented Understudy for Gisting Evaluation (ROUGE)) similarity metrics, along with manual readability assessments. Results show that descriptions generated from uncommented code often scored higher in semantic similarity and were more readable. Although lexical similarity was generally low, no major inaccuracies were observed. These findings suggest that AI can produce reliable and useful code summaries, but improvements are needed inlinguistic alignment and consistency.