Performance Implications of Inter-Broker Latency on Synchronously Replicated Kafka Stretch Clusters

Sammanfattning

Apache Kafka is a popular event-streaming platform built with fault tolerance and resilience in mind with respect to node and network failures. However, maintaining data consistency and availability in the event of a disastrous event requires careful planning and specialized geo-replication techniques. Asynchronous data replication across multiple sites offers clear advantages in performance but cannot guarantee zero data loss. Synchronous replication in Kafka with stretch clusters can guarantee zero data loss, even in a disaster scenario, but at a measurable cost to latency and throughput. This thesis investigates how the operational performance of a synchronously replicated stretch cluster is impacted by inter-broker latency. Latency was injected between the brokers to simulate physical distance. The performance impact was evaluated and modeled in terms of replication latency, producer-visible latency, and producer-visible throughput. Linear relationships were found regarding both replication latency and producer-visible latency. Producer-visible throughput follows an asymptotic relationship with inter-broker latency, approaching zero as latency increases. A model was developed that accurately predicts relative throughput, though absolute throughput prediction remains a limitation. This gives operators a practical tool for estimating the throughput cost of stretch cluster deployments before committing to a configuration.