Charting the Properties of Data Lakehouse Based Synchronization

Sammanfattning

Modern data architecture often relies on centralized storage that replicates the data of different tools. This format improves performance and the use of analytics. In this thesis the lakehouse structure will be the one in focus. The lakehouse is viewed as a central database that stores and reflects the data of several external tools. Each tool represents a smaller system that consists of a database. This structure also comes with challenges related to reliability, synchronization and data staleness. This study reviews these challenges, where synchronization is achieved through the use of pushes and pulls. This thesis will investigate under what structures and execution schedules a system reaches a unique perfect state. To answer this, a set of simplified scenarios are reviewed using selected execution schedules and two randomized scheduling strategies. The randomized schedules are used to examine how a system functions without the use of a conformant fixed schedule. These scenarios will be classified based on whether they lead to a unique and perfect state, a unique but not perfect state, or if multiple final states can be achieved. The results shows how conformant schedules can lead to unique perfect states, while the use of stale data in the lakehouse may create different final states. A cyclic behaviour can also cause quiescence to not be achievable. Reaching perfection is desired, but this does not guarantee the expected behaviour from the system.