Relationship between flaky tests and requirements

Sammanfattning

Flaky tests produce inconsistent results without code changes, representing a significantchallenge in software development by undermining developer confidence anddelaying continuous integration processes [33]. Despite their recognized impact onsoftware quality assurance, the relationship between requirements quality and testinstability remains underexplored in industrial contexts. This study investigates howindustry practitioners perceive the relationship between requirements quality andflaky tests and identifies specific requirements quality attributes that most significantlycontribute to this problem, providing evidence-based guidelines to mitigatethese issues. A mixed-methods approach was employed by combining quantitativesurvey data from 74 industrial practitioners with qualitative semi-structured interviewsfrom eight professionals across diverse domains, including automotive, banking,healthcare, and software development. The findings establish flaky tests as awidespread phenomenon, with 80% of practitioners reporting regular encounterswith test flakiness in their projects. The study reveals a notable awareness practicegap: while 80% of practitioners recognize that requirement completeness frequentlyimpacts test stability, only 56% consistently consult requirements during flaky testinvestigations. This selective behavior reflects sophisticated contextual reasoningrather than a lack of knowledge, with practitioners strategically assessing whenrequirements-related factors are likely contributors versus when technical diagnosticsare more appropriate. The critical requirements quality attributes emerge ascontributors to test flakiness: completeness deficiencies (including unspecified timeoutdurations affecting 77% of practitioners, missing acceptance criteria at 61%,and incomplete data specifications at 62%), and ambiguity issues (especially unclearsuccess and failure conditions reported by 65%). The research identifies timing specificationsas a critical blind spot in current requirements practices, often treated asimplementation details rather than explicit requirements, leading to cascading effectswhere independent timing assumptions by developers and testers create environmentsensitive test behavior. The study demonstrates significant domain-specific variationin the requirements-flakiness relationship, with practitioners reporting varying levelsof requirements related contributions across different sectors. Structured domainswith formal processes (banking, healthcare) show lower requirements related flakiness,while less regulated environments experience higher impacts. This contextualdependency reflects how organizational processes, regulatory constraints, and systemcriticality mediate the impact of requirements quality on test stability. Based onempirical findings, the research presents evidence-based guidelines organized aroundfundamental requirements quality attributes to systematically reduce requirementsrelated test flakiness. These guidelines provide systematic approaches for organizationsto reduce requirements-related test flakiness through targeted interventionsaddressing the identified quality gaps. The study establishes requirements quality asa measurable contributor to software system reliability, with practitioners estimating that 21-60% of flaky tests originate from requirements related issues, deservingsystematic attention alongside traditional technical factors in software quality assuranceefforts.