Evaluating BERT for Email Text Classification: A Comparative Study with Manual Text Pattern Matching

Sammanfattning

Automated email systems can be implemented as a text categorization task to deliver timely and relevant responses to email inquiries. Although manually crafted text patterns have been proven sufficient for this purpose, they remain limited by human ability to capture contextual nuances. This study investigates how the performance of a pre-trained deep language representation model, Bidirectional Encoder Representations from Transformers (BERT), compares to manual text pattern matching. A laboratory experiment was conducted as a text categorization task using the same dataset and evaluation metrics as in a previous study, with manual text patterns serving as the benchmark. Performance was assessed in terms of precision, recall and F1 score and differences were evaluated using descriptive statistics and a Mann-Whitney U test. Results show that BERT achieves slightly higher overall performance, with a statistically significant improvement in recall compared to text pattern matching. These findings suggest that BERT’s ability to manage linguistic variation can enhance the responsiveness of automated email systems.