Designing Trustworthy Conversational User Interfaces for Reliable Information Retrieval in Quality Management Systems

Sammanfattning

Access to accurate and reliable information is critical in quality management systems (QMS), as organizations using QMS operate under compliance requirements. However, existing QMS systems often rely on folder-based structures and free-text search, which can make information retrieval inefficient and dependent on prior system knowledge. Conversational user interfaces (CUIs) offer an alternative approach by allowing users to retrieve information through natural language interaction. This thesis investigated how a CUI can be designed to support trustworthy and reliable information retrieval in a QMS. The thesis followed a user-centered design approach using the design thinking process and combined qualitative and quantitative research methods. Semi-structured interviews were conducted with end users and system experts to understand current workflows, challenges, and expectations toward a potential CUI. These insights informed the design of low- and high-fidelity prototypes. The prototypes were iteratively evaluated through moderated usability testing. Perceived usability and trust were measured using the System Usability Scale (SUS) and the Short Trust in Automation Scale (S-TIAS), complemented by qualitative insights collected through thinking aloud and interviews. The results indicate that the CUI prototype supported information retrieval by allowing users to locate existing documentation through natural language interaction. The prototype received a SUS score of 90, indicating excellent perceived usability, while trust measured using S-TIAS yielded a mean score of 5.50, indicating high perceived trust. Trust and perceived reliability were not primarily associated with the conversational interaction itself, but with transparency mechanisms such as explicit source references, document metadata, and access to original documents. The qualitative findings also highlighted the importance of user control and alignment with users' existing practices for assessing information reliability. The findings suggest that CUIs can support information retrieval in QMS when designed with an emphasis on transparency, usability, and mechanisms that allow users to verify and control retrieved information. This thesis contributes design-oriented insights to the development of trustworthy CUIs in compliance-driven domains.