Evaluating Deepfake Image Detectors Under User-Induced Image Distortions

Sammanfattning

AI-generated images have become increasingly realistic, creating a growing need for reliable deepfake detection in social media environments where images are often edited before being shared on social media platforms. While many detectors are evaluated on clean benchmark datasets, real-world content may be affected by user-induced image distortions such as text, filters, and interface overlays. This thesis investigates how such distortions affect the robustness of deepfake image detectors. The study was organized as a controlled experimental evaluation with an initial screening phase followed by a main experiment. The screening phase was used to test the evaluation setup and select suitable detector candidates, while the main experiment evaluated stronger spatial and frequency-based detectors on benchmark datasets under social media-style distortions. The results show that detector robustness depends on both the detector architecture and the distortion type. Detector performance was measured using the area under the receiver operating characteristic curve (AUC). Across all detector-dataset combinations, the TikTok-style UI overlay produced the largest average AUC decrease (0.0649), followed by Snapchat-style text overlays (0.0468), while Instagram-style filters had the smallest average decrease (0.0082). However, the effect of distortions varied across datasets. On DFDCP, Snapchat-style overlays produced a larger performance decrease than the TikTok-style condition, indicating that the most challenging distortion depended on the evaluation setting. Across the full-test datasets, F3Net achieved the highest mean AUC under distorted conditions, followed by UCF and SPSL. These findings suggest that benchmark performance on clean data alone is insufficient for assessing deepfake detector robustness under realistic user-edited conditions. They also show that the impact of image modifications is not uniform across datasets and detectors, highlighting the importance of evaluating robustness under multiple distortion conditions rather than relying solely on aggregate performance measures.