Evaluating Programming Proficiency of Large Language Models : Assessing Large Language Models’ Effectiveness in Function and Class Generation, Code Commenting, Robustness, and Security

Sammanfattning

From basic chatbots to sophisticated virtual assistants like Google Assistant and Apple’s Siri, the evolution of Artificial Intelligence (AI) and Large Language Models (LLMs) has transformed how we interact with technology in our daily lives. These advancements have not only reshaped how we communicate with machines but also fundamentally altered the software industry, enabling coding assistants to assist developers in their daily tasks. The increased complexity and capabilities of LLMs raise concerns about data privacy and potential biases, pushing companies keen on privacy to look into hosting their own local LLM. Selecting the most suitable model can pose a challenge, motivated by this, this work introduces a new framework designed to evaluate LLMs in programming tasks. This is done to simplify the process for companies considering the deployment of a local LLM for coding-related tasks. The framework examines a LLMs performance in programming tasks such as function and class generation, code commenting, and important aspects such as code robustness and security. In addition to the framework, this work contributes a comprehensive evaluation of 10 State-of-the-Art (SOTA) LLMs done through the framework. Results indicate the significance of model size, pre-training data, and the approach in prompting, with larger and more specialized models generally exhibiting better performance. Post-training strategies such as weight quantization show potential in enabling LLM deployment on less powerful hardware.