Comparative Analysis of Large Language Models for Implementation in IT Service Desk Environments

Sammanfattning

This study conducts a comparative evaluation of a commercial Large language model(LLM), Copilot, with two open-source large language models (OSLLM), Llama and Mistral, in the context of IT service desk (ITSD) operations. The study was done to fill the gap in research comparing open-source and commercial LLMs in ITSD. A literature review was undertaken to establish a better understanding of the area and what results have previously been found. The literature review provided thebasis for understanding LLMs functionality, how to prompt LLMs, the differences between OSLLMs and LLMs as well as LLM applicability in an ITSD environment. The commercial LLM required a prompt and a dataset and was otherwise ready for use and integration in the company’s IT environment, such as Teams and Sharepoint. The OSLLMs required a fullstack solution to allow the users to interact with the LLMs. Model performance was assessed both quantitatively and qualitatively through a structured query set. The commercial LLM performed best which ispartially attributed to the model’s sheer size. The study also noted that setting upan AI-powered chatbot was notably easier using a commercial provider, but came atthe cost of privacy and control.