Building Domain-Specific Sub-Models from Large Language Models using Pruning

Sammanfattning

Large Language Models (LLMs) represent the forefront of technological advancements, however, their widespread adoption is impeded by their substantial computational demands and resource-intensive nature. Pruning, as a compression technique, offers promises for mitigating these challenges. However, existing methods, focusing on general purpose models, often lack optimization for specific tasks, demanding more tailored approaches. This thesis investigates the feasibility of constructing domain-specific sub-models by pruning LLMs using Wanda pruning technique with task-specific calibration datasets. Wanda, a state-of-the-art technique, strategically prunes less critical areas of LLMs, thereby diminishing computational demands while preserving, to some extent, performance integrity. However, it primarily relies on a limited set of general English text as a calibration dataset to estimate input activations and “zero out” less important weights. This approach does not investigate the impact of different types of calibration samples on the post-pruning accuracy and structure of the models. This thesis aims to address this research gap by exploring the impact of employing different task-specific datasets as calibration sets in the pruning process. The evaluation results demonstrate overall improvements in accuracy and inference speed for domain-specific sub-models pruned with task-specific datasets, highlighting the practical utility of the approach. Moreover, interesting results are shown on the structural differences of the sub-models obtained. More specifically, the obtained results reveal that same-domain sub-models retain a higher proportion of similar weights compared to those derived from different domains.