Human-AI Runtime Requirements Validation: A Feedback Driven Framework - An investigation into the design, implementation and evaluation of a Human-AI framework for runtime requirements validation with a feedback driven approach.

Sammanfattning

Requirements engineering is a vital process to ensure all stakeholder needs are metappropriately. However, stakeholder requirements are prone to constant change dueto changing stakeholder needs, lack of domain knowledge, and long feedback loops.Many companies struggle to capture continuously evolving requirements; this is alsotrue at the case company. The case company has developed Tool X, which requiresextensive requirements engineering to be performed before usage. However, therequirements at the company evolve often, with each user having a unique set ofrequirements that could be unclear due to the highly interdependent nature of thedata. To capture these ever-evolving requirements at runtime, this research applieda Design Science Research (DSR) approach to investigate the problem, and to design, implement, and evaluate a solution framework at the case company. Based oninsights extracted from 10 stakeholder interviews, the study designed a multi-agentHuman-AI runtime requirements validation framework. This framework utilizes aKnowledge Graph and three distinct Data Processing Pipelines (DPPs) to create acontinuous feedback-driven loop, ensuring the human-in-the-loop remains firmly incharge. The evaluation of the implemented framework demonstrated robust technical results: the system achieved a 78.4% error detection rate with zero false positives,and 88.6% of its technical suggestions were accurate. Furthermore, user testing confirmed the AI-assisted process was highly usable, useful for achieving deploymentgoals, and significantly less taxing than manual validation. These findings highlightthat shifting to continuous runtime validation drastically speeds up feedback loops,provides stakeholders with domain knowledge needed to validate as human in charge,practitioners prefer AI as a collaborative assistant rather than an autonomous actor.