TY - JOUR
T1 - Evaluating academic answers generated using ChatGPT
AU - Fergus, Suzanne
AU - Botha, Michelle
AU - Ostovar, Mehrnoosh
PY - 2023/3/20
Y1 - 2023/3/20
N2 - The integration of technology in education has become ever more prioritized since the COVID-19 pandemic. Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence technology that generates conversational interactions to user prompts. The trained model can answer follow up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. The functionality of ChatGPT in answering chemistry assessment questions required investigation to ascertain its potential impact on learning and assessment. Two chemistry-focused modules in Year 1 and Year 2 of a Pharmaceutical Science program were used to study and evaluate ChatGPT generated responses in relation to the end of year exam assessments. For questions that focused on knowledge and understanding with “describe” and “discuss” verbs, the ChatGPT generated responses. For questions that focused on application of knowledge and interpretation with non-text information, the ChatGPT technology reached a limitation. A further analysis of the quality of responses is reported in this study. ChatGPT is not considered a high-risk technology tool in relation to cheating. Similar to the COVID-19 disruption, ChatGPT is expected to provide a catalyst for educational discussions on academic integrity and assessment design.
AB - The integration of technology in education has become ever more prioritized since the COVID-19 pandemic. Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence technology that generates conversational interactions to user prompts. The trained model can answer follow up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. The functionality of ChatGPT in answering chemistry assessment questions required investigation to ascertain its potential impact on learning and assessment. Two chemistry-focused modules in Year 1 and Year 2 of a Pharmaceutical Science program were used to study and evaluate ChatGPT generated responses in relation to the end of year exam assessments. For questions that focused on knowledge and understanding with “describe” and “discuss” verbs, the ChatGPT generated responses. For questions that focused on application of knowledge and interpretation with non-text information, the ChatGPT technology reached a limitation. A further analysis of the quality of responses is reported in this study. ChatGPT is not considered a high-risk technology tool in relation to cheating. Similar to the COVID-19 disruption, ChatGPT is expected to provide a catalyst for educational discussions on academic integrity and assessment design.
U2 - 10.1021/acs.jchemed.3c00087
DO - 10.1021/acs.jchemed.3c00087
M3 - Article
JO - Journal of Chemical Education
JF - Journal of Chemical Education
SN - 0021-9584
ER -