Merging Large Language Models for Enhanced Code Generation : A Comparative Study of Model Merging Techniques Across Programming Languages

Sammanfattning

The rapid advancement of LLMs has transformed software development by enabling automated code generation across a wide range of programming tasks. While fine-tuned LLMs tailored to specific languages and domains have become increasingly accessible, training or fine-tuning multi-domain models remains resource-intensive and environmentally costly. In this context, model merging has emerged as a scalable and efficient alternative, allowing multiple specialized models to be integrated with substantially lower resource demands. This thesis investigates the potential of direct parameter merging methods to improve the multilingual code generation capabilities of LLMs. We evaluate seven prominent techniques: Uniform Soup, Task Arithmetic, SLERP, TIES-Merging, DARE, DELLA, and Model Breadcrumbs, applied to fine-tuned models from three open-source families: CodeQwen1.5-7B, DeepSeek-Coder-6.7b-Base, and CodeLlama-34b. The merged models are assessed on code generation tasks in Python, Java, JavaScript, and C++. Our findings highlight TIES-Merging as the most effective and robust method, achieving the highest average performance across model families and benchmarks. Model Breadcrumbs, DARE and Task Arithmetic also performed well, though slightly behind TIES-Merging. In contrast, simpler methods like SLERP and Uniform Soup consistently underperformed, likely due to their inability to address parameter interference, a key challenge in model merging. Notably, merged models often outperformed the individual models they were derived from, suggesting that merging can improve performance by acting as a form of regularization. This study demonstrates that model merging can be used to effectively enhance the multilingual performance of code-generating LLMs and identifies several of the most promising merging methods. It establishes model merging as a practical and scalable strategy for building multi-task coding models, potentially supporting the development of more capable and efficient AI assistants for use in software development.