A Comparative Study of Static Rules, Clustering, and Classification for Deduplication Accuracy : A Comparative Analysis of Algorithmic Performance and Reliability

Sammanfattning

As computing efficiency and storage capacity become increasingly important to companies, modernizing database maintenance has emerged as a challenge. This thesis evaluates the feasibility of transitioning from a legacy static rule-based deduplication system to a machine learning approach to customer records. Utilizing real-world customer data from the UK and Belgium, the study compares supervised classification (Random Forest, XGBoost) and unsupervised clustering (DBSCAN, Graph-based transitive closure) against a legacy baseline. Given the extreme sparsity of the data and the high imbalance of duplicate records, performance was measured using Informedness and Markedness. The findings reveal that the primary challenges is utilizing real-world customer data are extreme sparsity and inconsistent formatting. Despite these hurdles, both models identified more suspected duplicates than the static rules and achieved high performance against the constructed labeled datasets with Random Forest and XGBoost emerging as the most reliable machine learning models. Unsupervised clustering, especially DBSCAN, offers a more scalable path by eliminating the need for labeled data, though it remains highly dependent on optimized parameters. However, the study also highlights that trained classifier models are region-specific, requiring localized training or a diverse global dataset for effective deployment. The findings suggest that while machine learning noticeably improves suspected duplicate detection. algorithms behave fundamentally different depending on regional data entries. Therefore the most reliable operational solution is a probabilistic framework integrated with a Human-in-the-Loop system, offering the best balance of technical accuracy and business reliability.