Evaluating semantic search versus SharePoint KQL for Information Retrieval : A Use Case in Facility Management

Sammanfattning

This study was conducted in collaboration with Sweco Sverige AB, where one of its clients acts as a facility manager for a large hospital in Sweden. The client uses SharePoint Server version 2019 as their project portal, which contains a large amount of project-specific documentation primarily in PDF format. While SharePoint allows for easy document management and search functionality, its search primarily relies on traditional keyword matching, using the Keyword Query Language (KQL). This study investigates whether implementing semantic search based on existing PDF documents improves the ability to find relevant information compared to using traditional keyword-based matching. To answer this question, a proof-of-concept Python application implementing semantic search was developed. Its retrieval performance was evaluated against SharePoint by formulating 22 queries using two types of query formulations: natural language and keyword-based formulations. Each query was executed in both systems, and each response was evaluated using cosine similarity score. To identify possible false positives, a manual review of each query and its response were made. The results showed that implementing semantic search based on existing PDF documents improved the retrieval of relevant information compared to using SharePoint and traditional keyword-based search. This was observed for both natural language and keyword-based query formulations.