Scalable Model Training with Ray on Hopsworks : Design, Implementation, and Evaluation of a Prototype System for Multi-node Model Training

Sammanfattning

The growing demand for scalable model training in modern MLOps pipelines, together with the rise of large language models (LLMs), highlights the need for robust integration of distributed computing frameworks with reliable storage systems. This thesis presents a prototype system that integrates the Ray framework into Hopsworks to support scalable model training on heterogeneous compute resources, while also providing periodic data backups in the persistent storage, HopsFS. The system is containerized by Docker and orchestrated by Kubernetes. To evaluate the system, two experiments were conducted: the first benchmarked distributed training performance on ResNet152; the second finetuned the Llama-3.1-8B-Instruct model, leveraging LoRA for parameter-efficient adaptation and DeepSpeed for memory-efficient distributed training. The results of the experiments provide several key findings. In small clusters, inter-node overhead was negligible, implying that communication and coordination between nodes did not significantly affect training performance. Scaling resources improved training speed, although efficiency gains were limited, with doubling resources yielding only an approximate improvement of 16%, measured in seconds per epoch. The experiments also showed that dataset loading throughput and checkpoint saving throughput remained consistent across different cluster formations, indicating stable system I/O performance. Furthermore, the system successfully completed multiple training runs without failures, demonstrating its reliability under prolonged, resource-intensive workloads. Testing in different workloads and environments also confirmed system stability and flexibility. Overall, the results show that the integration of Ray with Hopsworks enables scalable model training while meeting the expected functionality, reliability, and stability. This prototype system extends Hopsworks’ capabilities to support advanced training pipelines for modern machine learning, especially LLM finetuning.