Authors' Reply to “Comments on 'Researcher Bias:: The Use of Machine Learning in Software Defect Prediction

Martin Shepperd; Tracy Hall; David Bowes

doi:10.1109/TSE.2017.2731308

Authors' Reply to “Comments on 'Researcher Bias: The Use of Machine Learning in Software Defect Prediction

Martin Shepperd, Tracy Hall, David Bowes

University of Hertfordshire

Research output: Contribution to journal › Article › peer-review

1 Citation (Scopus)

12 Downloads (Pure)

Abstract

In 2014 we published a meta-analysis of software defect prediction studies [1]. This suggested that the most important factor in determining results was Research Group i.e., who conducts the experiment is more important than the classifier algorithms being investigated. A recent re-analysis [2] sought to argue that the effect is less strong than originally claimed since there is a relationship between Research Group and Dataset. In this response we show (i) the re-analysis is based on a small (21%) subset of our original data, (ii) using the same re-analysis approach with a larger subset shows that Research Group is more important than type of Classifier and (iii) however the data are analysed there is compelling evidence that who conducts the research has an effect on the results. This means that the problem of researcher bias remains. Addressing it should be seen as a matter of priority amongst those of us who conduct and publish experiments comparing the performance of competing software defect prediction systems.

Original language	English
Pages (from-to)	1129-1131
Number of pages	3
Journal	IEEE Transactions on Software Engineering
Volume	44
Issue number	11
Early online date	24 Jul 2017
DOIs	https://doi.org/10.1109/TSE.2017.2731308
Publication status	Published - Nov 2018

Keywords

Analysis of variance
Analytical models
Data models
defect prediction
Measurement
NASA
Predictive models
researcher bias
Software
Software quality assurance

Access to Document

10.1109/TSE.2017.2731308Licence: Other

07990255
© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Accepted author manuscript, 261 KBLicence: Other

Cite this

@article{c770a10f0a5047a59b14407801645439,

title = "Authors' Reply to “Comments on 'Researcher Bias:: The Use of Machine Learning in Software Defect Prediction",

abstract = "In 2014 we published a meta-analysis of software defect prediction studies [1]. This suggested that the most important factor in determining results was Research Group i.e., who conducts the experiment is more important than the classifier algorithms being investigated. A recent re-analysis [2] sought to argue that the effect is less strong than originally claimed since there is a relationship between Research Group and Dataset. In this response we show (i) the re-analysis is based on a small (21%) subset of our original data, (ii) using the same re-analysis approach with a larger subset shows that Research Group is more important than type of Classifier and (iii) however the data are analysed there is compelling evidence that who conducts the research has an effect on the results. This means that the problem of researcher bias remains. Addressing it should be seen as a matter of priority amongst those of us who conduct and publish experiments comparing the performance of competing software defect prediction systems.",

keywords = "Analysis of variance, Analytical models, Data models, defect prediction, Measurement, NASA, Predictive models, researcher bias, Software, Software quality assurance",

author = "Martin Shepperd and Tracy Hall and David Bowes",

note = "{\textcopyright} 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.",

year = "2018",

month = nov,

doi = "10.1109/TSE.2017.2731308",

language = "English",

volume = "44",

pages = "1129--1131",

journal = "IEEE Transactions on Software Engineering",

issn = "0098-5589",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

number = "11",

}

TY - JOUR

T1 - Authors' Reply to “Comments on 'Researcher Bias:

T2 - The Use of Machine Learning in Software Defect Prediction

AU - Shepperd, Martin

AU - Hall, Tracy

AU - Bowes, David

N1 - © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

PY - 2018/11

Y1 - 2018/11

N2 - In 2014 we published a meta-analysis of software defect prediction studies [1]. This suggested that the most important factor in determining results was Research Group i.e., who conducts the experiment is more important than the classifier algorithms being investigated. A recent re-analysis [2] sought to argue that the effect is less strong than originally claimed since there is a relationship between Research Group and Dataset. In this response we show (i) the re-analysis is based on a small (21%) subset of our original data, (ii) using the same re-analysis approach with a larger subset shows that Research Group is more important than type of Classifier and (iii) however the data are analysed there is compelling evidence that who conducts the research has an effect on the results. This means that the problem of researcher bias remains. Addressing it should be seen as a matter of priority amongst those of us who conduct and publish experiments comparing the performance of competing software defect prediction systems.

AB - In 2014 we published a meta-analysis of software defect prediction studies [1]. This suggested that the most important factor in determining results was Research Group i.e., who conducts the experiment is more important than the classifier algorithms being investigated. A recent re-analysis [2] sought to argue that the effect is less strong than originally claimed since there is a relationship between Research Group and Dataset. In this response we show (i) the re-analysis is based on a small (21%) subset of our original data, (ii) using the same re-analysis approach with a larger subset shows that Research Group is more important than type of Classifier and (iii) however the data are analysed there is compelling evidence that who conducts the research has an effect on the results. This means that the problem of researcher bias remains. Addressing it should be seen as a matter of priority amongst those of us who conduct and publish experiments comparing the performance of competing software defect prediction systems.

KW - Analysis of variance

KW - Analytical models

KW - Data models

KW - defect prediction

KW - Measurement

KW - NASA

KW - Predictive models

KW - researcher bias

KW - Software

KW - Software quality assurance

UR - http://www.scopus.com/inward/record.url?scp=85028922080&partnerID=8YFLogxK

U2 - 10.1109/TSE.2017.2731308

DO - 10.1109/TSE.2017.2731308

M3 - Article

AN - SCOPUS:85028922080

SN - 0098-5589

VL - 44

SP - 1129

EP - 1131

JO - IEEE Transactions on Software Engineering

JF - IEEE Transactions on Software Engineering

IS - 11

ER -

Authors' Reply to “Comments on 'Researcher Bias: The Use of Machine Learning in Software Defect Prediction

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this