Efficient fine-tuning of Transformer Models Under Hardware Constraints : An Empirical Study of LoRA and Knowledge Distillation

Sammanfattning

Background Transformer-based language models dominate modern natural language processing but impose substantial computational costs during full fine-tuning, limiting accessibility on consumer-grade hardware. Knowledge Distillation and Low-Rank Adaptation (LoRA) independently address this challenge, yet their interaction under strict single GPU memory constraints remains insufficiently explored. Objectives This thesis investigates whether transformer models can be fine-tuned on a single consumer-grade GPU (NVIDIA T4 class) without performance degradation, and in some cases with improved generalization. The primary objective is to evaluate a hybrid framework combining LoRA’s parameter efficiency with KD’s supervisory signal, quantifying trade-offs in accuracy, trainable parameter count, and training time on the SST-2 benchmark. Methods A dual-phase experimental design was conducted under fixed hard-ware constraints. Four configurations were evaluated: (1) full fine-tuning of BERT-Large; (2) LoRA-based fine-tuning with a frozen backbone; (3) knowledge distillation using a DistilBERT student; and (4) a hybrid LoRA–KD strategy with restricted adapter placement. Experiments were single-run with a fixed random seed. Although this precludes formal statistical testing, observed performance gaps substantially exceed typical fine-tuning variance of 0.2–0.5% documented in prior BERT evaluation studies, indicating systematic architectural effects rather than stochastic initialization noise. Statistical Validity Consideration To contextualize the single-run experimental design, we reference established variance benchmarks from prior BERT fine-tuning studies. Dodge et al. report that across five random seeds on SST-2, BERT-Base exhibits a standard deviation of approximately 0.4% in validation accuracy. Mosbach et al. demonstrate that with appropriate learning rate warmup and sufficient training steps, this variance can be reduced to 0.2–0.3%. Given these benchmarks, the observed performance gaps in this study are note-worthy: • LoRA vs. FFT: 2.07% gap ≈ 5–10× typical variance • Hybrid vs. LoRA (1,000 samples): 3.53% gap ≈ 9–18× typical variance • Hybrid vs. LoRA: 1.38% gap ≈ 3–7× typical variance While these margins suggest systematic effects, we acknowledge that formal sta-tistical significance would require multi-seed validation. The resource constraints ofthis study precluded such validation.We therefore interpret findings as indicative rather than definitive, and recommendmulti-seed replication as a priority for future work. Results Three principal findings emerge from the SST-2 evaluation. First, theLoRA (4.02 GB vs. 4.11 GB), indicating that offline KD enables richer supervisionwithout increased memory cost. Second, LoRA reduces trainable parameters byover 95% while achieving higher accuracy than FFT in this evaluation (93.81% vs.91.74%), suggesting an implicit regularization effect under moderate data conditions. Third, the Hybrid LoRA–KD approach shows improved performance in low-data regimes relative to standalone LoRA, supporting the hypothesis that distillation acts as a regularizer when labeled data is scarce. Third, qualitative analysis revealed increased error sensitivity in compressed models for linguistically complex inputs involving compositional negation. Conclusions On SST-2, LoRA provides sufficient parameter efficiency under moderate data availability, while knowledge distillation yields complementary gains in data-scarce regimes; the Hybrid model achieves 92.43% accuracy using only 1,000 samples and 4.7M trainable parameters- approaching FFT performance (91.74% on 30,000 samples) with 97% less training data- demonstrating strong data efficiency and enabling environmentally sustainable NLP research on consumer-grade GPUs (NVIDIA T4).