Protocol Fuzzing- A Comparison between LLM-Assisted and Mutation-Based Fuzzers

Sammanfattning

As the demand for secure software and hardware solutions increase in our modern digitalized society, the importance of reliable and cost-effective product testing is apparent. One such testing technique is fuzzing, which involves generating large amounts of malformed and randomized data to stress test system interfaces. This study aimed to compare traditional fuzzing methods to modern approaches assisted with Large Language Models (LLMs) for the data generation, focusing on efficiency and performance. The fuzzing targets were three consumer grade routers-the D-Link Eagle Pro AI G416, R03 and R15-which received the input data via a direct Ethernet LAN connection. The tested interface was each router's internal HTTP-server, being fuzzed with data packets aimed at triggering errors based on previously published CVEs. The traditional fuzzing model used in the comparison was a mutation-based fuzzer, while the LLM-assisted approach employed OpenAI's LLM GPT-4.1 to generate the fuzzing data. Both models operated without prior knowledge of the target systems, qualifying the experiments as black-box fuzzing. Each router underwent five test sessions, during which HTTP-requests, timestamps, and potential error states were logged, with Wireshark capturing background traffic. The collected data included benchmark performance metrics, counts of discovered vulnerabilities, request deviation matrices, and energy usage estimates. The outcome of the project showcased that the traditional fuzzer performed quantitively better with regards to the benchmark metrics and energy cost per generated byte, but did not perform as well as the LLM-fuzzer with regards to fuzz data variation, alteration dispersion and complexity. Since no errors or vulnerabilities were found during any of the tests, it was concluded that the published CVEs did not aid enough in the context of black-box fuzzing.