Comparative Analysis of unsupervised and supervised topic models in IT ticket classification

Sammanfattning

This thesis seeks to address the question of how significantly ticket classification performance is affected in the absence of labels. This issue is critical because labeling all tickets is an extremely time consuming task, and a slight reduction in accuracy may be acceptable if it allows for the use of an unsupervised model. To explore this, we evaluate three state of the art methods: LDA and BERTopic as unsupervised models, and a BERT-based classifier as a supervised approach. All models operate on a synthetically generated dataset that mirrors real municipal ticket structures and is labeled with ten predefined categories. The BERT model is trained using supervised learning, while LDA and BERTopic follow unsupervised approaches. The BERT-based classifier achieves the highest macro F1-score. However, a key finding is that BERTopic performs within 9\% of BERT in terms of macro F1-score, despite operating without labeled data. This indicates that unsupervised topic modeling may serve as a viable alternative when annotated training data is limited or unavailable. The results show that transformer-based models are highly effective for structured IT support classification tasks. At the same time, interpretable unsupervised methods such as BERTopic offer promising performance in label-scarce environments, supporting scalable and cost effective automation.