Can Large Language Models be Used as an Alternative for Human Annotation: A Case Study of Emotion Classification

For market-based software evolution, user feedback has become a primary source utilized by various machine learning (ML) algorithms to identify insightful information. However, it heavily employs human subjects to complete such experimental tasks, particularly for data annotation. It is reported in software engineering (SE) literature that human subjects are challenging to find, prone to errors, and can have a second guess in identifying the correct annotation type, resulting in possible bias. In contrast, large language models (LLMs) have recently demonstrated comparatively equal or better performances in various complex SE tasks, making them a good alternative for data annotation tasks. For this purpose, the proposed approach investigated and experimented with the performance of LLMs, particularly ChatGPT, to annotate end-user feedback for ML classification tasks. We experimented with two datasets, i.e., human and ChatGPT API annotated, to explore whether ChatGPT can be used as an alternative to human annotators when preparing labeled datasets for ML experiments. For this purpose, we identify the efficacy of various deep learning (DL) classifiers in detecting associated emotions, including anger, confusion, distrust, sadness, disappointment, frustration, disgust, and fear, with end-user reviews. We obtained satisfactory results with BILSTM, GRU, CNN, LSTM, BiGRU, and RNN algorithms using the ChatGPT-generated dataset compared to the human-annotated data set. We obtained an average accuracy of 92%, 92%, 91%, 90%, 91%, and 91% compared to the manually annotated data set, 75%, 75%, 79%,48%, 73%, and 85%, with CNN, LSTM, BILSTM, GRU, BiGRU, and RNN Classifiers, respectively. The study results show that LLMs can be an alternative source for annotating datasets for ML classification experiments. However, the results need to be validated by human experts for improved generalizability and trust.