NSGA-III Based Multi-Modal Middleware for Dynamic Orchestration of Distributed Data Processing Pipelines

Sammanfattning

The orchestration of multi-modal data processing pipelines in cloud-native environments is a growing challenge, where execution cost and end-to-end latency must be optimised simultaneously while accounting for task dependencies. Modern data systems increasingly integrate batch, streaming, and serving workloads on Kubernetes-based infrastructure. However, existing orchestration mechanisms typically rely on static heuristics or single-objective optimisation, which do not explicitly model the trade-off between conflicting performance objectives. This thesis investigates how a middleware system using the NSGA-III evolutionary algorithm can dynamically orchestrate DAG-based multi-modal data processing pipelines in Kubernetes/KubeVirt environments while balancing execution cost and end-to-end latency. The pipeline is modelled as a directed acyclic graph (DAG), enabling dependency-aware scheduling and critical-path-based latency estimation. The optimisation problem is formulated to minimise total execution cost and end-to-end latency by selecting appropriate virtual machine instances (VMIs) and resource allocations for each task. The proposed middleware is implemented as a three-layer architecture consisting of a Logical Pipeline Manager, an Optimisation Engine, and a KubeVirt Execution Adapter, enabling integration with Kubernetes/KubeVirt environments. The system is evaluated using a Lambda Architecture-inspired sentiment analysis pipeline under Low, Medium, and High workload conditions, with 30 independent runs per scenario. Experimental results show that NSGA-III consistently generates multiple Pareto-optimal solutions, providing flexible trade-off options between cost and latency. Compared to a weighted-sum genetic algorithm baseline, NSGA-III achieves approximately 31% higher hypervolume across all workload levels (p < 0.001), indicating more comprehensive coverage of the trade-off space. The optimisation overhead remains stable at approximately 2.2 seconds per run, demonstrating practical feasibility. These results indicate that Pareto-based multi-objective optimisation provides a more effective approach to pipeline orchestration than single-objective methods, particularly in environments with competing performance requirements.