Comparing the performance of fault prediction models which report multiple performance measures: recomputing the confusion matrix

David Bowes; Tracy Hall; David Gray

doi:10.1145/2365324.2365338

Comparing the performance of fault prediction models which report multiple performance measures: recomputing the confusion matrix

David Bowes, Tracy Hall, David Gray

Centre for Computer Science and Informatics Research

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

19 Citations (Scopus)

261 Downloads (Pure)

Abstract

There are many hundreds of fault prediction models published in the literature. The predictive performance of these models is often reported using a variety of different measures. Most performance measures are not directly comparable.
This lack of comparability means that it is often difficult to evaluate the performance of one model against another. Our aim is to present an approach that allows other researchers and practitioners to transform many performance
measures of categorical studies back into a confusion matrix.
Once performance is expressed in a confusion matrix alternative preferred performance measures can then be derived. Our approach has enabled us to compare the performance of 600 models published in 42 studies. We demonstrate the application of our approach on several case studies, and discuss
the advantages and implications of doing this.

Original language	English
Title of host publication	Procs of the 8th Int Conf on Predictive Models in Software Engineering
Subtitle of host publication	PROMISE'12
Place of Publication	New York, NY, USA
Publisher	ACM Press
Pages	109-118
Number of pages	10
ISBN (Print)	978-1-4503-1241-7
DOIs	https://doi.org/10.1145/2365324.2365338
Publication status	Published - 2012

Publication series

Name	PROMISE '12
Publisher	ACM

Access to Document

10.1145/2365324.2365338

Comparing the performance of fault prediction model: re-computing the confusion matrix
"© ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Procs of the 8th Int Conf on Predictive Models in Software Engineering http://doi.acm.org/10.1145/2365324.2365338"
Accepted author manuscript, 253 KB

Cite this

@inproceedings{5a644d93c12e48cea8479e5034205208,

title = "Comparing the performance of fault prediction models which report multiple performance measures: recomputing the confusion matrix",

abstract = "There are many hundreds of fault prediction models published in the literature. The predictive performance of these models is often reported using a variety of different measures. Most performance measures are not directly comparable. This lack of comparability means that it is often difficult to evaluate the performance of one model against another. Our aim is to present an approach that allows other researchers and practitioners to transform many performance measures of categorical studies back into a confusion matrix. Once performance is expressed in a confusion matrix alternative preferred performance measures can then be derived. Our approach has enabled us to compare the performance of 600 models published in 42 studies. We demonstrate the application of our approach on several case studies, and discuss the advantages and implications of doing this.",

author = "David Bowes and Tracy Hall and David Gray",

note = "Awarded Best Paper in Conference. {"}{\textcopyright} ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Procs of the 8th Int Conf on Predictive Models in Software Engineering http://doi.acm.org/10.1145/2365324.2365338{"}",

year = "2012",

doi = "10.1145/2365324.2365338",

language = "English",

isbn = "978-1-4503-1241-7",

series = "PROMISE '12",

publisher = "ACM Press",

pages = "109--118",

booktitle = "Procs of the 8th Int Conf on Predictive Models in Software Engineering",

address = "United States",

}

Comparing the performance of fault prediction models which report multiple performance measures: recomputing the confusion matrix. / Bowes, David; Hall, Tracy; Gray, David.
Procs of the 8th Int Conf on Predictive Models in Software Engineering: PROMISE'12. New York, NY, USA: ACM Press, 2012. p. 109-118 (PROMISE '12).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Comparing the performance of fault prediction models which report multiple performance measures

T2 - recomputing the confusion matrix

AU - Bowes, David

AU - Hall, Tracy

AU - Gray, David

N1 - Awarded Best Paper in Conference. "© ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Procs of the 8th Int Conf on Predictive Models in Software Engineering http://doi.acm.org/10.1145/2365324.2365338"

PY - 2012

Y1 - 2012

N2 - There are many hundreds of fault prediction models published in the literature. The predictive performance of these models is often reported using a variety of different measures. Most performance measures are not directly comparable. This lack of comparability means that it is often difficult to evaluate the performance of one model against another. Our aim is to present an approach that allows other researchers and practitioners to transform many performance measures of categorical studies back into a confusion matrix. Once performance is expressed in a confusion matrix alternative preferred performance measures can then be derived. Our approach has enabled us to compare the performance of 600 models published in 42 studies. We demonstrate the application of our approach on several case studies, and discuss the advantages and implications of doing this.

AB - There are many hundreds of fault prediction models published in the literature. The predictive performance of these models is often reported using a variety of different measures. Most performance measures are not directly comparable. This lack of comparability means that it is often difficult to evaluate the performance of one model against another. Our aim is to present an approach that allows other researchers and practitioners to transform many performance measures of categorical studies back into a confusion matrix. Once performance is expressed in a confusion matrix alternative preferred performance measures can then be derived. Our approach has enabled us to compare the performance of 600 models published in 42 studies. We demonstrate the application of our approach on several case studies, and discuss the advantages and implications of doing this.

U2 - 10.1145/2365324.2365338

DO - 10.1145/2365324.2365338

M3 - Conference contribution

SN - 978-1-4503-1241-7

T3 - PROMISE '12

SP - 109

EP - 118

BT - Procs of the 8th Int Conf on Predictive Models in Software Engineering

PB - ACM Press

CY - New York, NY, USA

ER -

Comparing the performance of fault prediction models which report multiple performance measures: recomputing the confusion matrix

Abstract

Publication series

Access to Document

Fingerprint

Cite this