Evaluating Trade-offs of Quantized LLMs for Requirements and Test Alignment

Sammanfattning

Large Language Models (LLMs) have shown impressivecapabilities in various domains due to their ability toprocess and interpret natural language. Meanwhile, as softwaresystems continue to expand in size and complexity, the numberof associated artifacts (e.g, Requirements and Test Cases) growsas well, leading to challenges in aligning REST (RequirementEngineering and System Test) efforts. There have been earlierinitiatives in using LLMs for REST alignment, as it is a widelyused measure for software quality assurance. However, the costsassociated with model deployment and execution have limitedtheir feasibility. There is a need for a faster yet reasonablesolution to cope with the rate at which software artifacts keepgrowing.In this paper, we investigate whether quantized LLMs canserve as a viable alternative, given their smaller size and lessdemanding hardware requirements. We choose Mistral, a widelyused open-weight LLM, and assess it in conjunction with threedifferent quantization techniques: AWQ, GPTQ, and AQLM—comparing these four versions of Mistral against each other.The experiment is performed with four requirement specificationdatasets encompassing 433 Requirements and 408 Tests in total.We offer insights into the feasibility of adopting quantizedLLMs for REST alignment, highlighting the efficacy, efficiency,and trade-offs of adopting such models, along with a actionableguidance for practitioners.Index Terms—Large Language Models, REST, Traceability,Quantization, Software Testing