Time Series Anomaly Detection of SMS Messaging Behavior : A Machine Learning Study Using PatchTST Forecasting and Embedding Difference

Sammanfattning

With the unyielding perseverance of fraudsters looking to scam vulnerable individuals through phishing and pharming messages, a small fraction of malicious messages often make it past static spam filters that scan the contents without contextual information. This thesis explores the extraction of 43 messaging behavior features from SMS Teknik AB messaging API data (15,000,000 SMS) and the subsequent analysis to pre-train a machine learning pipeline that can efficiently detect anomalies in user behavior time series data. The ultimate goal is to prevent users from continuing to send messages after being compromised by an attacker. The anomaly detection is performed in a purely self-supervised manner where users are "compromised" by another user in the dataset, replacing their last few statistical data points with real statistical data points of a different user to simulate a strict anomaly. Additionally, soft anomalies are evaluated where fewer features are replaced, making the problem more challenging. A low false-positive rate is critical for anomaly detection methods in high-traffic systems like this one. Therefore, the True Positive Rate (TPR) at specific False Positive Rate (FPR) thresholds is evaluated. PatchTST Prediction Error (PPE), PatchTST Embedding Difference (PED), and LinReg Prediction Error (LPE) are compared. Two types of pipelines are evaluated for all methods: one that generalizes to unseen users (UI) and another that generalizes to unseen time steps of seen users (N-UI). PPE is shown to be significantly better than PED for detecting both strict and soft anomalies. When compared to LPE, it is significantly better for soft anomalies but shows no statistically significant difference for strict anomalies. On average, UI is significantly worse than N-UI for detecting strict and soft anomalies but can achieve satisfactory results for detecting strict anomalies when paired with PPE and has the added benefit of potentially generalizing to unseen users. PPE requires the longest pretraining time but achieves the fastest sequence inference speeds when trained. The best features in terms of feature importance for messaging behavior time series anomaly detection are the originator-based features origina- tor_noUpperCaseCharacters, originator_noCharacters, originator_noLetters, as well as the text embedding principal component features embed_pca_0, embed_pca_1, and embed_pca_2, with the addition of the geolocation-based features sender_latitude and sender_longitude.