On the utilization of non-quality assessed literature in software engineering research

Quality of research and knowledge sources can be a critical factor to judge the quality of the research citing them. The prevailing online dissemination of research via blogs, websites, experience reports, and white papers can be helpful as it minimizes potential publication biases. In this paper, we have (i) identified the body of software engineering (SE) literature that is non-indexed by Web-of-Science but has been used in published SE systematic literature review (SLR) studies (as primary studies); (ii) proposed a checklist-based method for quality assessment of non-indexed and grey literature; (iii) empirically evaluated the effectiveness of the proposed method; (iv) searched primary studies retrieved from the SLRs in DBLP and Google Scholar (GS) to verify whether DBLP or Google Scholar can act as viable options to search for non-indexed Web-of-Science (non-WOS-indexed) literature. Besides proposing our synthesized checklist method, we found that (i) SE literature reviews use significantly non-indexed Web-of-Science sources; (ii) DBLP only covers a portion of the non-WOS-indexed reference, whereas GS has a more complete coverage of the non-indexed reference in comparison. Preliminary evaluation of the proposed quality assessment point system-based and checklist-based methods shows it to be a viable way to assess quality of the non-indexed and grey literature.