TeleTester: An Industrial Evaluation ofLLM-Based RAG Cypress End-to-End Test Generation : Evaluating RAG Agentic Architecture

Sammanfattning

Background: End-to-end (E2E) testing of modern Single-Page Applications (SPAs) re-mains challenging due to dynamic Document Object Model (DOM) structures, asyn-chronous rendering, and frequent user interface (UI) changes. These characteristics often result in brittle selectors, flaky test executions, and high maintenance effort, particularly in UI-driven automation frameworks such as Cypress. Although recent advances in Large Language Models (LLMs) show potential for automating parts of software testing, their application to UI test generation is limited by hallucinations and insufficient grounding in observable interface state. Objectives: The objective of this thesis is to investigate whether grounding LLM-basedtest generation in concrete UI context can improve the correctness and executability of generated Cypress E2E tests. Specifically, the study evaluates TeleTester, a retrieval-augmented, multi-agent pipeline, within a realistic industrial SPA setting. The work aims to assess practical feasibility and perceived usefulness without claiming general applica-bility beyond the evaluated context. Methods: A Retrieval-Augmented Generation (RAG) based pipeline was designed and implemented, combining DOM crawling, semantic chunking, vector-based retrieval, andagent-based task decomposition. TeleTester distributes responsibilities across planning, retrieval, and code generation agents, each operating on task-specific prompts and retrieved DOM context. The system was evaluated in collaboration with an industrial partner using a production administrative SPA. Its performance was compared against an LLM-only baseline using quantitative execution metrics and qualitative analysis of generated testscripts. Results: The results show that DOM-grounded generation improves test correctness, and executability compared to the baseline approach. Generated tests required less manual correction and demonstrated higher alignment with intended UI workflows. Survey responses further indicate positive perceptions of usability, accuracy, and productivity support, particularly among less experienced testers. Conclusions: The findings suggest that retrieval-augmented, context-grounded LLM pipelines can effectively support Cypress E2E test generation in dynamic SPA environments. While the evaluation is limited to a single industrial case, the results provide indicative evidence that grounding LLMs in observable UI state can reduce common failure modes in UI testautomation and support practical testing workflows.