Knowledge Distillation for Compact Language Models on Mathematical Reasoning Tasks

Sammanfattning

Large language models (LLMs) have demonstrated exceptional capabilities in complex mathematical reasoning; however, their massive parameter scales impose prohibitive computational costs, limiting their deployment in resource constrained or local environments. To address these challenges, this thesis investigates advanced knowledge distillation methodologies to enhance the mathematical problem-solving performance of lightweight student models (0.5 billion and 1.5 billion parameters). We propose a framework that integrates large-scale reasoning data synthesis with classic knowledge distillation. Leveraging a high-performance teacher model as the generator, we explicitly produce detailed reasoning trajectories to expand the training corpus while simultaneously utilizing distribution alignment to match the teacher’s soft probability targets. Through the design of four comparative training strategies, we systematically evaluate the individual and combined effects of data synthesis and soft-target distillation. Evaluations on standardized benchmarks (GSM8K and MATH) reveal that our approach effectively mitigates the capacity bottlenecks inherent in small architectures. The results demonstrate that the proposed framework delivers substantial performance gains over both the un-tuned base models and the supervised fine-tuning baselines on targeted reasoning tasks, particularly on highly demanding competitive mathematics. These findings contribute to the development of efficient, locally deployable mathematical AI assistants under strict resource constraints.