Regression test selection (RTS) reduces regression testing costs by re-running only tests that can change behavior due to code changes. To deal with the rapidly growing costs of regression testing, several RTS tools were recently developed and adopted by researchers and large software organizations. As RTS tools gain adoption, it becomes critical to check that they are correct and efficient. Unfortunately, checking RTS tools currently relies solely on limited sets of tests that RTS tool developers write.

We present RTSCheck , the first framework for checking RTS tools. RTSCheck feeds evolving programs (i.e., sequences of program revisions) to an RTS tool and checks the output against rules inspired by existing RTS test suites; violations of these rules are likely to be due to deviations from expected RTS tool behavior. RTSCheck uses three components to obtain evolving programs: (1) AutoEP automatically generates evolving programs and corresponding tests; (2) DefectsEP uses buggy and fixed program revisions from bug databases; and (3) EvoEP uses sequences of program revisions from actual histories of open-source projects. We use RTSCheck to check three recently developed RTS tools for Java: Clover, Ekstazi, and STARTS. RTSCheck discovered 27 bugs in the tools. Each RTSCheck component helped discover different bugs in all three RTS tools.