Designing AI for decision making : The importance of trust, control and explainability within a government agency

Sammanfattning

As Artificial Intelligence (AI) grows, government agencies have begun implementing them to reap the rewards. Skatteverket, a Swedish government has begun the process of implementing an AI tool for decision making. However, not much is known about how to design these AI tools. In order to address this gap, a research through design project was conducted. This project included a pilot study, two workshops as well as provotypes. The research questions for this study were; (1) How can an AI-based decision making tool be designed to promote trust and control for case managers working in government services?, and (2) How can explanations (XAI) of AI-based decision making processes be designed to promote trust and control for case managers working in government services? The results showed one major design implication for the first RQ as well as three design implications for the second RQ. The first design implication suggested that there is a need to design to elevate human knowledge. Keeping the case managers in the loop and designing for building a good gut feeling. The results for designing for XAI showed that they firstly wanted the explanations to ‘speak the same language as them’, using the same phrasing and laws as they use. Secondly, they want to understand the ‘why and how’ of the predictions and reasoning. And lastly, they wanted to be able to have different explanations for different workflows. This study provides a good start for understanding the user needs for XAI in AI tools and how to use provotypes to understand this.