The Experts below are selected from a list of 6459 Experts worldwide ranked by ideXlab platform
Mark Harman - One of the best experts on this subject based on the ideXlab platform.
-
an empirical study on mutation statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
International Conference on Software Engineering, 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
-
ICSE - An empirical study on mutation, statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
2017 IEEE ACM 39th International Conference on Software Engineering (ICSE), 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
-
AUSTIN: A Tool for Search Based Software Testing for the C Language and Its Evaluation on Deployed Automotive Systems
2nd International Symposium on Search Based Software Engineering, 2010Co-Authors: Kiran Lakhotia, Mark Harman, Hamilton GrossAbstract:Despite the large number of publications on Search--Based Software Testing (SBST), there remain few publicly available tools. This paper introduces AUSTIN, a publicly available SBST tool for the C language. The paper validates the tool with an empirical study of its effectiveness and efficiency in achieving Branch Coverage compared to random testing and the Evolutionary Testing Framework (ETF), which is used in-house by Daimler and others for Evolutionary Testing. The programs used in the study consist of eight non--trivial, real-world C functions drawn from three embedded automotive software modules. For the majority of the functions, AUSTIN is at least as effective (in terms of achieved Branch Coverage) as the ETF, and is considerably more efficient.
-
GECCO - A multi-objective approach to search-based test data generation
Proceedings of the 9th annual conference on Genetic and evolutionary computation - GECCO '07, 2007Co-Authors: Kiran Lakhotia, Mark Harman, Phil McminnAbstract:There has been a considerable body of work on search-based test data generation for Branch Coverage. However, hitherto, there has been no work on multi-objective Branch Coverage. In many scenarios a single-objective formulation is unrealistic; testers will want to find test sets that meet several objectives simultaneously in order to maximize the value obtained from the inherently expensive process of running the test cases and examining the output they produce. This paper introduces multi-objective Branch Coverage.The paper presents results from a case study of the twin objectives of Branch Coverage and dynamic memory consumption for both real and synthetic programs. Several multi-objective evolutionary algorithms are applied. The results show that multi-objective evolutionary algorithms are suitable for this problem, and illustrates the way in which a Pareto optimal search can yield insights into the trade-offs between the two simultaneous objectives.
Thierry Titcheu Chekam - One of the best experts on this subject based on the ideXlab platform.
-
an empirical study on mutation statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
International Conference on Software Engineering, 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
-
ICSE - An empirical study on mutation, statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
2017 IEEE ACM 39th International Conference on Software Engineering (ICSE), 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
Mike Papadakis - One of the best experts on this subject based on the ideXlab platform.
-
an empirical study on mutation statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
International Conference on Software Engineering, 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
-
ICSE - An empirical study on mutation, statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
2017 IEEE ACM 39th International Conference on Software Engineering (ICSE), 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
Yves Le Traon - One of the best experts on this subject based on the ideXlab platform.
-
an empirical study on mutation statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
International Conference on Software Engineering, 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
-
ICSE - An empirical study on mutation, statement and Branch Coverage fault revelation that avoids the unreliable clean program assumption
2017 IEEE ACM 39th International Conference on Software Engineering (ICSE), 2017Co-Authors: Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, Mark HarmanAbstract:Many studies suggest using Coverage concepts, such as Branch Coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between Coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar Coverage for both faulty and fixed (`clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of Coverage are attained.
Stephen H Edwards - One of the best experts on this subject based on the ideXlab platform.
-
checked Coverage and object Branch Coverage new alternatives for assessing student written tests
Technical Symposium on Computer Science Education, 2015Co-Authors: Zalia Shams, Stephen H EdwardsAbstract:Many educators currently use code Coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or Branch Coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage metrics assess what percentage of code has been exercised, but do not depend on whether a test suite adequately checks that the expected behavior is achieved. This paper evaluates checked Coverage, an alternative measure of test thoroughness aimed at overcoming this limitation, along with object Branch Coverage, a structure code Coverage metric that has received little discussion in educational assessment. Checked Coverage works backwards from behavioral assertions in test cases, measuring the dynamic slice of the executed code that actually influences the outcome of each assertion. Object Branch Coverage (OBC) is a stronger Coverage criterion similar to weak variants of modified condition/decision Coverage. We experimentally compare checked Coverage and OBC against statement Coverage, Branch Coverage, mutation analysis, and all-pairs testing to evaluate which is the best predictor of how likely a test suite is to detect naturally occurring defects. While checked Coverage outperformed other Coverage measures in our experiment, followed closely by OBC, both were only weakly correlated with a test suite's ability to detect naturally occurring defects produced by students in the final versions of their programs. Still, OBC appears to be an improved and practical alternative to existing statement and Branch Coverage measures, while achieving nearly the same benefits as checked Coverage.
-
SIGCSE - Checked Coverage and Object Branch Coverage: New Alternatives for Assessing Student-Written Tests
Proceedings of the 46th ACM Technical Symposium on Computer Science Education, 2015Co-Authors: Zalia Shams, Stephen H EdwardsAbstract:Many educators currently use code Coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or Branch Coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage metrics assess what percentage of code has been exercised, but do not depend on whether a test suite adequately checks that the expected behavior is achieved. This paper evaluates checked Coverage, an alternative measure of test thoroughness aimed at overcoming this limitation, along with object Branch Coverage, a structure code Coverage metric that has received little discussion in educational assessment. Checked Coverage works backwards from behavioral assertions in test cases, measuring the dynamic slice of the executed code that actually influences the outcome of each assertion. Object Branch Coverage (OBC) is a stronger Coverage criterion similar to weak variants of modified condition/decision Coverage. We experimentally compare checked Coverage and OBC against statement Coverage, Branch Coverage, mutation analysis, and all-pairs testing to evaluate which is the best predictor of how likely a test suite is to detect naturally occurring defects. While checked Coverage outperformed other Coverage measures in our experiment, followed closely by OBC, both were only weakly correlated with a test suite's ability to detect naturally occurring defects produced by students in the final versions of their programs. Still, OBC appears to be an improved and practical alternative to existing statement and Branch Coverage measures, while achieving nearly the same benefits as checked Coverage.