Materialization Strategies for Multi-Source Stream Enrichment : Comparing Imperative Microservices with Streaming SQL

Sammanfattning

Microservice architectures that assemble derived views by re-querying base tables at request time face a fundamental scaling challenge: the per-event cost grows with data volume, regardless of how little data actually changed. Streaming SQL databases offer an alternative by maintaining derived views incrementally through continuously executing queries, but their practical trade-offs against established architectures are not well characterized. This thesis evaluates the migration of a production securities lending data pipeline from a Java microservice architecture with request-time materialization, using PostgreSQL common table expression (CTE) queries, to a streaming SQL implementation using Timeplus. The evaluation compares the two approaches through scenario-based maintainability analysis, quantitative performance benchmarking of end-to-end latency and sustainable throughput, and an assessment of the capabilities and constraints each approach introduces. The streaming SQL implementation reduces integration code by approximately 75% for the most complex data source and avoids several abstraction layers, but introduces deployment scenarios where schema changes require recreating the materialized views, causing a temporary period of incomplete data. The baseline deploys uniformly regardless of change type and benefits from compile-time safety. In performance, the streaming implementation exhibits stable latency independent of dataset size and load rate, while the baseline degrades progressively as full-table re-aggregation saturates available CPU. This performance advantage is specific to the event-propagation path and disappears when both systems must serialize full result sets. Continuous materialization additionally enables decoupled multi-consumer architectures and intraday historical granularity, but involves coarser observability and commercial vendor dependency. The results suggest that streaming SQL is a practical alternative for workloads where per-event cost grows with data volume, with the trade-off shifting from code complexity to operational complexity.