Evaluating academic answers generated using ChatGPT

Suzanne Fergus; Michelle Botha; Mehrnoosh Ostovar

doi:10.1021/acs.jchemed.3c00087

Evaluating academic answers generated using ChatGPT

Suzanne Fergus, Michelle Botha, Mehrnoosh Ostovar

Research output: Contribution to journal › Article › peer-review

159 Downloads (Pure)

Abstract

The integration of technology in education has become ever more prioritized since the COVID-19 pandemic. Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence technology that generates conversational interactions to user prompts. The trained model can answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. The functionality of ChatGPT in answering chemistry assessment questions requires investigation to ascertain its potential impact on learning and assessment. Two chemistry-focused modules in year 1 and year 2 of a pharmaceutical science program are used to study and evaluate ChatGPT-generated responses in relation to the end-of-year exam assessments. For questions that focused on knowledge and understanding with “describe” and “discuss” verbs, the ChatGPT generated responses. For questions that focused on application of knowledge and interpretation with nontext information, the ChatGPT technology reached a limitation. A further analysis of the quality of responses is reported in this study. ChatGPT is not considered a high-risk technology tool in relation to cheating. Similar to the COVID-19 disruption, ChatGPT is expected to provide a catalyst for educational discussions on academic integrity and assessment design.

Original language	English
Pages (from-to)	1672-1675
Number of pages	4
Journal	Journal of Chemical Education
Volume	100
Issue number	4
Early online date	31 Mar 2023
DOIs	https://doi.org/10.1021/acs.jchemed.3c00087
Publication status	Published - 11 Apr 2023

Keywords

First-Year Undergraduate
Applications of Chemistry
Outreach
General
Public Understanding
Internet
Web-Based Learning
Communication

Access to Document

10.1021/acs.jchemed.3c00087Licence: CC BY

Final_Version
© 2023 The Authors. Published by American Chemical Society and Division of Chemical Education, Inc. This is an open access article distributed under the Creative Commons Attribution License (CC BY), https://creativecommons.org/licenses/by/4.0/
Final published version, 1.3 MBLicence: CC BY

Cite this

@article{60ddaf7501b5407fb700179e9efef883,

title = "Evaluating academic answers generated using ChatGPT",

abstract = "The integration of technology in education has become ever more prioritized since the COVID-19 pandemic. Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence technology that generates conversational interactions to user prompts. The trained model can answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. The functionality of ChatGPT in answering chemistry assessment questions requires investigation to ascertain its potential impact on learning and assessment. Two chemistry-focused modules in year 1 and year 2 of a pharmaceutical science program are used to study and evaluate ChatGPT-generated responses in relation to the end-of-year exam assessments. For questions that focused on knowledge and understanding with “describe” and “discuss” verbs, the ChatGPT generated responses. For questions that focused on application of knowledge and interpretation with nontext information, the ChatGPT technology reached a limitation. A further analysis of the quality of responses is reported in this study. ChatGPT is not considered a high-risk technology tool in relation to cheating. Similar to the COVID-19 disruption, ChatGPT is expected to provide a catalyst for educational discussions on academic integrity and assessment design.",

keywords = "First-Year Undergraduate, Applications of Chemistry, Outreach, General, Public Understanding, Internet, Web-Based Learning, Communication",

author = "Suzanne Fergus and Michelle Botha and Mehrnoosh Ostovar",

note = "{\textcopyright} 2023 The Authors. Published by American Chemical Society and Division of Chemical Education, Inc. This is an open access article distributed under the Creative Commons Attribution License (CC BY), https://creativecommons.org/licenses/by/4.0/",

year = "2023",

month = apr,

day = "11",

doi = "10.1021/acs.jchemed.3c00087",

language = "English",

volume = "100",

pages = "1672--1675",

journal = "Journal of Chemical Education",

issn = "0021-9584",

publisher = "American Chemical Society",

number = "4",

}

TY - JOUR

T1 - Evaluating academic answers generated using ChatGPT

AU - Fergus, Suzanne

AU - Botha, Michelle

AU - Ostovar, Mehrnoosh

N1 - © 2023 The Authors. Published by American Chemical Society and Division of Chemical Education, Inc. This is an open access article distributed under the Creative Commons Attribution License (CC BY), https://creativecommons.org/licenses/by/4.0/

PY - 2023/4/11

Y1 - 2023/4/11

N2 - The integration of technology in education has become ever more prioritized since the COVID-19 pandemic. Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence technology that generates conversational interactions to user prompts. The trained model can answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. The functionality of ChatGPT in answering chemistry assessment questions requires investigation to ascertain its potential impact on learning and assessment. Two chemistry-focused modules in year 1 and year 2 of a pharmaceutical science program are used to study and evaluate ChatGPT-generated responses in relation to the end-of-year exam assessments. For questions that focused on knowledge and understanding with “describe” and “discuss” verbs, the ChatGPT generated responses. For questions that focused on application of knowledge and interpretation with nontext information, the ChatGPT technology reached a limitation. A further analysis of the quality of responses is reported in this study. ChatGPT is not considered a high-risk technology tool in relation to cheating. Similar to the COVID-19 disruption, ChatGPT is expected to provide a catalyst for educational discussions on academic integrity and assessment design.

AB - The integration of technology in education has become ever more prioritized since the COVID-19 pandemic. Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence technology that generates conversational interactions to user prompts. The trained model can answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. The functionality of ChatGPT in answering chemistry assessment questions requires investigation to ascertain its potential impact on learning and assessment. Two chemistry-focused modules in year 1 and year 2 of a pharmaceutical science program are used to study and evaluate ChatGPT-generated responses in relation to the end-of-year exam assessments. For questions that focused on knowledge and understanding with “describe” and “discuss” verbs, the ChatGPT generated responses. For questions that focused on application of knowledge and interpretation with nontext information, the ChatGPT technology reached a limitation. A further analysis of the quality of responses is reported in this study. ChatGPT is not considered a high-risk technology tool in relation to cheating. Similar to the COVID-19 disruption, ChatGPT is expected to provide a catalyst for educational discussions on academic integrity and assessment design.

KW - First-Year Undergraduate

KW - Applications of Chemistry

KW - Outreach

KW - General

KW - Public Understanding

KW - Internet

KW - Web-Based Learning

KW - Communication

UR - http://www.scopus.com/inward/record.url?scp=85151320966&partnerID=8YFLogxK

U2 - 10.1021/acs.jchemed.3c00087

DO - 10.1021/acs.jchemed.3c00087

M3 - Article

SN - 0021-9584

VL - 100

SP - 1672

EP - 1675

JO - Journal of Chemical Education

JF - Journal of Chemical Education

IS - 4

ER -

Evaluating academic answers generated using ChatGPT

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this