Benchmarking Framework - Evaluating Corpora Search Systems Using a Spectrum of Complex Search Queries

Sammanfattning

This thesis presents a benchmark designed to evaluate corpora search systems indigital humanities, focusing on handling complex search queries. By analyzing usergeneratedqueries from Korp server logs, the research examines various query structuresand computational challenges. The study develops a two-level abstractionprocess to simplify query patterns, enabling clustering and feature extraction techniques.The Hierarchical Density-Based Spatial Clustering of Applications withNoise (HDBSCAN) algorithm is used to segment queries into clusters representingdifferent user behaviors. The performance of these clusters is evaluated by executingqueries on selected corpora using Corpus Workbench (CWB), with executiontimes as the main metric. The results indicate variability in query execution efficiency,influenced by corpus size and query complexity. To improve benchmarkclarity, clusters with similar structures were grouped to create a more focused evaluationframework. The findings show how query complexity and data volume affectsystem performance, highlighting the importance of scalability in corpora searchsystems. This benchmark provides a framework for evaluating the performance ofcorpora search systems in digital humanities, contributing to the development ofdigital research tools in this field.