Detecting Faults vs. Revealing Failures: Exploring the Missing Link

Document Type

Conference Proceeding

Publication Date

1-1-2024

Abstract

When we quantify the effectiveness of a test suite by its mutation coverage, we are in fact equating test suite effectiveness with fault detection: to the extent that mutations are faithful proxies of actual faults, it is sensible to consider that the effectiveness of a test suite to kill mutants reflects its ability to detect faults. But there is another way to measure the effectiveness of a test suite: by its ability to expose the failures of an incorrect program (or, equivalently, its ability to give us confidence in the correctness of a correct program). The relationship between failures and faults is tenuous at best: a fault is the adjudged or hypothesized cause of a failure. Whereas a failure is an observable, verifiable, certifiable effect, a fault is someone's hypothesis about the possible cause of the observed effect. The same failure may be attributed to more than one fault or combination of faults. In this paper we raise two questions: is the ability to detect faults the same as the ability to reveal failures? If not, which is the better measure of test suite effectiveness? We do not give definite answers to these questions, but we use empirical data to challenge some assumptions and show why these questions are worth answering.

Identifier

85199504327 (Scopus)

ISBN

[9798350365634]

Publication Title

IEEE International Conference on Software Quality, Reliability and Security, QRS

External Full Text Location

https://doi.org/10.1109/QRS62785.2024.00021

ISSN

26939177

First Page

115

Last Page

126

Grant

DGE 2043104

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS