Compiler correctness is super Creal : An experimental study on the construction of cross-compiler test oracles for compiler fuzzing

Sammanfattning

Compilers should be correct. One of the most fundamental parts of the software engineer’s toolkit is the compiler, which translates the source code into its target representation. It is of tremendous importance that the compiler preserves the semantics of the programs during translation, since it casts doubt on the correctness of all software compiled with it if it does not. Compiler fuzzing, also called random testing, is a mature and effective way to test the correctness of compilers, and multiple test case generation strategies have been proposed and developed throughout the years. A problem that has received significantly less attention is the so-called Oracle problem, the problem of how you determine whether a given output is correct for a given input. Oracles used in previous research include various differential testing strategies, the usage of referee compilers and test case generation strategies that pre-compute the answer. However, there exists no research that evaluates the construction of these oracles. In this thesis, we present a novel method for testing the accuracy of test oracles for compiler fuzzing. This method uses a cross-compiler differential testing oracle consisting of nine open-source compilers to construct a baseline benchmark which we later compare seven alternative oracles against. This baseline benchmark consists of a data set of test cases we know should fail or not fail according to our voting differential testing oracle. The method is test case generation-agnostic, and we used the state-of-the-art tool Creal to generate the test cases. Our results conclude that four of our alternative oracles performed identically to the baseline benchmark, and the three other alternative oracles each failed a subset of the failed test cases in the benchmark. No alternative oracle failed a test case that was passed in the benchmark. When comparing the different oracles, we found no evidence that the number of voters in a voting oracle or whether the voters had previously been tested with the test case generation strategy would impact their accuracy as test oracles. However, our results indicate that the quality of the voters significantly impacted their accuracy.