Test Case Prioritization: A Family of Empirical Studies

To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more
important, by some measure, are run earlier in the regression testing process. One potential goal of such prioritization is to increase a
test suite’s rate of fault detection. Previous work reported results of studies that showed that prioritization techniques can significantly
improve rate of fault detection. Those studies, however, raised several additional questions: 1) Can prioritization techniques be
effective when targeted at specific modified versions; 2) what trade-offs exist between fine granularity and coarse granularity
prioritization techniques; 3) can the incorporation of measures of fault proneness into prioritization techniques improve their
effectiveness? To address these questions, we have performed several new studies in which we empirically compared prioritization
techniques using both controlled experiments and case studies. The results of these studies show that each of the prioritization
techniques considered can improve the rate of fault detection of test suites overall. Fine-granularity techniques typically outperformed
coarse-granularity techniques, but only by a relatively small margin overall; in other words, the relative imprecision in coarse-granularity
analysis did not dramatically reduce coarse-granularity techniques’ ability to improve rate of fault detection. Incorporation of
fault-proneness techniques produced relatively small improvements over other techniques in terms of rate of fault detection, a result
which ran contrary to our expectations. Our studies also show that the relative effectiveness of various techniques can vary
significantly across target programs. Furthermore, our analysis shows that whether the effectiveness differences observed will result in
savings in practice varies substantially with the cost factors associated with particular testing processes. Further work to understand
the sources of this variance and to incorporate such understanding into prioritization techniques and the choice of techniques would be
beneficial.
Go to Source