Evaluating Code Generation by Large Language Models : A Comparative Study on Accuracy, Efficiency, and Errors

Sammanfattning

Large language models are increasingly being used for generating programming code in educational and software development contexts. Using these models can potentially save time and resources, but it is crucial to understand their strengths, limitations, and reliability. This study compared the performance of three popular language models (GPT-4 Turbo, GPT-4o, and Claude 3.7 Sonnet) by evaluating their capability to solve a total of 100 programming tasks across three difficulty levels from the Kattis platform. The results indicated that the models generally performed well on simpler tasks, but accuracy significantly decreased with increasing complexity. Claude 3.7 Sonnet was the most effective for easier tasks, whereas GPT-4 Turbo provided the best balance of accuracy and efficiency on intermediate problems. For difficult tasks, all models faced significant challenges with correctness and efficiency. Common errors included logical mistakes, runtime problems, and exceeding time and memory limits. Clear differences in code readability and complexity were also observed, with Claude often generating better-commented code compared to the GPT models. We found that critical evaluation of AI-generated code is essential and suggest further research into advanced prompting techniques, model tuning, and improved integration of language models in educational and development environments to maximize the benefits of these tools.