Design and Evaluation of a Locally Hosted AI Assistant : A Design Science Research Study in Microservice-Based Software Development

Sammanfattning

Background. Retrieval Augmented Generation (RAG) has emerged as a promising approach for supporting Large Language Model based systems using project specific knowledge and documentation. Although RAG based systems have been explored in code generation and developer support related tasks, the application of locally hosted RAG systems for supporting microservice API documentation within industrial environments still remains limited. Within organisations operating under strict data governance and privacy requirements, cloud hosted AI systems are often not considered suitable for handling internal documentation and developer related information. Objectives. The objective of this thesis is mainly directed towards designing, developing, and evaluating a locally hosted RAG based developer assistant for supporting microservice API documentation understanding within GKN Aerospace Sweden. The study mainly focuses on evaluating response correctness, practical usefulness, system limitations, and the impact of response latency on developer adoption behaviour. Methods. The research follows Design Science Research Methodology (DSRM). The developed system mainly consists of a Retrieval Augmented Generation pipeline using Gemma 3 4B through Ollama, all-MiniLM-L6-v2 embeddings, PostgreSQL with pgvector, Streamlit, VS Code extension, and CLI based interaction environments. The system was evaluated through semi-structured interviews and interaction sessions with six engineers from GKN Aerospace. The collected responses and observations were analysed using Reflexive Thematic Analysis (RTA). Results. The evaluation sessions indicated that the assistant generally produced re-sponses that were consistent with the retrieved documentation. Across the observed tasks, no fabricated API endpoints or unsupported implementation details were identified when responses were compared with the available documentation. While these findings suggest that the retrieval-based design helped reduce unsupported responses, they should not be interpreted as evidence that the system is completely free from hallucinations outside the evaluated scenarios. Conclusions. The study demonstrates that a locally hosted RAG based assistant can provide useful and trustworthy support for developers interacting with microservice API documentation within industrial software engineering environments operating under strict privacy and data governance requirements.