Evaluation of Automated Vulnerability Detection Techniques in Modern Web Applications: A Comparative Analysis of Detection Accuracy, Performance, and Practical Effectiveness

Sammanfattning

This thesis evaluates the effectiveness of automated vulnerability detection tools in modern web applications, focusing on Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST). As web applications become increasingly complex and frequently targeted by attackers, reliable automated security tools are important for identifying common vulnerabilities such as cross-site scripting, access control weaknesses, business logic flaws, injection vulnerabilities, file exposure, and open redirect issues. The research question addressed in this thesis is: How effective are automated vulnerability detection tools in identifying common web application vulnerabilities in controlled experimental environments? To answer this question, a controlled experimental study was conducted using OWASP Juice Shop, an intentionally vulnerable web application. Semgrep was used as the SAST tool and OWASP ZAP was used as the DAST tool. The tools were evaluated in the same controlled test environment, while acknowledging that they had different forms of system access: Semgrep analyzed the source code, whereas OWASP ZAP interacted with the running application. The outputs were compared against a manually verified ground truth of confirmed vulnerabilities. The collected data included detection rates, false negatives, false positives, partial detections, and performance across different vulnerability categories. The results showed that neither tool detected all manually verified vulnerabilities. Semgrep achieved 33.3% strict recall and 41.7% weighted recall, while OWASP ZAP achieved 8.3% strict recall and 12.5% weighted recall. Semgrep achieved higher coverage overall, especially for source-code-visible vulnerabilities, while OWASP ZAP detected the open redirect vulnerability and produced several runtime-related warnings. Both tools missed several access-control and business-logic vulnerabilities, and both required manual validation of findings. The study concludes that automated vulnerability detection tools are valuable for supporting security testing but are insufficient as standalone solutions. Their effectiveness varies depending on vulnerability type, tool configuration, application coverage, and the need for runtime or business-logic understanding. Automated tools should therefore be complemented with manual security testing to achieve more comprehensive vulnerability coverage.