A Comparative Study of GPT-4 vs. GitHub Copilot on Kattis Problems

Sammanfattning

The use of Artificial Intelligence (AI) in education, particularly in computer science, has increased recently. Among AI technologies, Large Language Models (LLMs) like GPT-4 have been increasingly utilized for tasks such as assessing assignments and supporting students’ learning. In contrast, AI-driven tools like GitHub Copilot are designed to assist developers by providing code suggestions and completions. Despite these advancements, the effectiveness of these technologies in solving programming tasks is still poorly understood. This study investigates and compares the performance of GitHub Copilot and GPT-4 in solving programming tasks provided and assessed by the online programming platform Kattis. GitHub Copilot and GPT-4 were given the problem description of 50 randomly selected programming problems of varying difficulty levels. The results reveal that text input for GPT-4 successfully solved 21 out of 50 problems, achieving a success rate of 42%, whilst image input for GPT-4 solved 17 out of 50 problems, achieving a success rate of 34%. While GPT-4 demonstrated a notable initial performance, it faced difficulties with more complex problems. Conversely, Copilot solved 14 out of 50 problems, resulting in a success rate of 28%. Like GPT-4, Copilot struggled to maintain consistent performance across multiple attempts, particularly with more challenging tasks. This study contributes to the understanding of LLMs’ and AI-driven tools’ capabilities and limitations in generating code and solving programming problems by analyzing the performance of Copilot and GPT-4. The findings provide a foundation for future research to enhance the reliability and efficiency of AI-driven tools and LLMs.