Toward explaining value concerns in software artifacts: A human-in-the-loop method

Context: Violations of human values in software can lead to user dissatisfaction, economic harm, and loss of trust. Existing software engineering research has focused on detecting such violations through manual labeling or automated classification, without explaining the mechanisms through which they occur. This lack of explanation makes reported violations difficult to verify and offers limited guidance for addressing them. Objectives: This study introduces a human-in-the-loop method that explains potential violations of human values in software artifacts—value concerns—by articulating the conditions under which they may arise. Methods: The proposed human-in-the-loop method integrates the automated reasoning capabilities of a large language model (ChatGPT) with human assessment to identify and articulate value concerns in software artifacts. We evaluate the method through a case study of 1200 Android and iOS APIs, assessing its ability to produce traceable accounts of potential value violations. Results: The results show that ChatGPT’s inferences about value concerns are reasonably accurate but incomplete. Errors arise from unfounded explanations and overextended reasoning, while the persuasive presentation of LLM outputs can make these errors difficult to recognize. The human-in-the-loop method mitigates these issues through human review. Conclusion: Our findings support the use of LLMs as reasoning partners that, when combined with human oversight, can effectively explain value concerns in software artifacts.