The Performance of GPT-3.5 in Summarizing Scientific and News Articles

Sabkat Arshad; Muhammad Yaqoob; Tahir Mehmood

doi:10.1007/978-981-97-0293-0_4

The Performance of GPT-3.5 in Summarizing Scientific and News Articles

Sabkat Arshad, Muhammad Yaqoob, Tahir Mehmood

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In the age of information, we are overwhelmed with large amounts of data. The quest to know more in less time has increased the need for efficient text summarization models that convert information into precise summaries such that essential details are not overlooked. Recently, GPT-3.5 has demonstrated impressive performance in text completion, generation, and question answering. However, its effectiveness in generating concise and coherent summaries for scientific articles and news reports remains under-explored. This work evaluates the performance of GPT-3.5 in summarizing scientific research articles and news data. Scientific articles were collected from arXiv STEM dataset, whereas news articles were sampled from the CNN/DailyMail dataset. Using the GPT-3.5 OpenAI API, the pre-trained model is prompted to generate summaries of the scientific and news articles. In the next step, the ROUGE score is computed for the generated summaries against the reference summaries to analyse the performance of the model. Our results show that GPT-3.5 performs slightly better in summarizing scientific articles as compared to news articles with an average ROUGE score of 0.35 and 0.31, respectively. Moreover, in agreement with the literature, we show that the ROUGE is not the best measure for evaluating text similarity as it heavily relies on similar vocabulary rather than semantics.

Original language	English
Title of host publication	Data Science and Emerging Technologies
Subtitle of host publication	Proceedings of DaSET 2023
Editors	Yap Bee Wah, Dhiya Al-Jumeily OBE, Michael W. Berry
Place of Publication	Singapore
Publisher	Springer Nature Link
Pages	49-61
Number of pages	13
ISBN (Electronic)	978-981-97-0293-0, 978-981-97-0293-0
ISBN (Print)	978-981-97-0292-3
DOIs	https://doi.org/10.1007/978-981-97-0293-0_4
Publication status	E-pub ahead of print - 27 Apr 2024
Event	The International Conference on Data Science and Emerging Technologies DaSET 2023 - Virtual conference at UNITAR International University, Malaysia Duration: 4 Dec 2023 → 5 Dec 2023 Conference number: 2 https://icdaset.com/daset2023/

Publication series

Name	Lecture Notes on Data Engineering and Communications Technologies
Publisher	Springer
Volume	191
ISSN (Print)	2367-4512
ISSN (Electronic)	2367-4520

Conference

Conference	The International Conference on Data Science and Emerging Technologies DaSET 2023
Abbreviated title	DaSET 2023
Country/Territory	Malaysia
Period	4/12/23 → 5/12/23
Internet address	https://icdaset.com/daset2023/

Keywords

ChatGPT
Large language model
Natural language processing
Scientific papers
Text summarization

Access to Document

10.1007/978-981-97-0293-0_4Licence: Unspecified

Cite this

Arshad, S., Yaqoob, M., & Mehmood, T. (2024). The Performance of GPT-3.5 in Summarizing Scientific and News Articles. In Y. B. Wah, D. A.-J. OBE, & M. W. Berry (Eds.), Data Science and Emerging Technologies: Proceedings of DaSET 2023 (pp. 49-61). (Lecture Notes on Data Engineering and Communications Technologies; Vol. 191). Springer Nature Link. Advance online publication. https://doi.org/10.1007/978-981-97-0293-0_4

Arshad, Sabkat ; Yaqoob, Muhammad ; Mehmood, Tahir. / The Performance of GPT-3.5 in Summarizing Scientific and News Articles. Data Science and Emerging Technologies: Proceedings of DaSET 2023. editor / Yap Bee Wah ; Dhiya Al-Jumeily OBE ; Michael W. Berry. Singapore : Springer Nature Link, 2024. pp. 49-61 (Lecture Notes on Data Engineering and Communications Technologies).

@inproceedings{487ffc1e2c0f41799815a4eb81538131,

title = "The Performance of GPT-3.5 in Summarizing Scientific and News Articles",

abstract = "In the age of information, we are overwhelmed with large amounts of data. The quest to know more in less time has increased the need for efficient text summarization models that convert information into precise summaries such that essential details are not overlooked. Recently, GPT-3.5 has demonstrated impressive performance in text completion, generation, and question answering. However, its effectiveness in generating concise and coherent summaries for scientific articles and news reports remains under-explored. This work evaluates the performance of GPT-3.5 in summarizing scientific research articles and news data. Scientific articles were collected from arXiv STEM dataset, whereas news articles were sampled from the CNN/DailyMail dataset. Using the GPT-3.5 OpenAI API, the pre-trained model is prompted to generate summaries of the scientific and news articles. In the next step, the ROUGE score is computed for the generated summaries against the reference summaries to analyse the performance of the model. Our results show that GPT-3.5 performs slightly better in summarizing scientific articles as compared to news articles with an average ROUGE score of 0.35 and 0.31, respectively. Moreover, in agreement with the literature, we show that the ROUGE is not the best measure for evaluating text similarity as it heavily relies on similar vocabulary rather than semantics.",

keywords = "ChatGPT, Large language model, Natural language processing, Scientific papers, Text summarization",

author = "Sabkat Arshad and Muhammad Yaqoob and Tahir Mehmood",

note = "{\textcopyright} 2024 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.; The International Conference on Data Science and Emerging Technologies DaSET 2023, DaSET 2023 ; Conference date: 04-12-2023 Through 05-12-2023",

year = "2024",

month = apr,

day = "27",

doi = "10.1007/978-981-97-0293-0_4",

language = "English",

isbn = "978-981-97-0292-3",

series = "Lecture Notes on Data Engineering and Communications Technologies",

publisher = "Springer Nature Link",

pages = "49--61",

editor = "Wah, {Yap Bee} and OBE, {Dhiya Al-Jumeily} and Berry, {Michael W.}",

booktitle = "Data Science and Emerging Technologies",

address = "Netherlands",

url = "https://icdaset.com/daset2023/",

}

Arshad, S, Yaqoob, M & Mehmood, T 2024, The Performance of GPT-3.5 in Summarizing Scientific and News Articles. in YB Wah, DA-J OBE & MW Berry (eds), Data Science and Emerging Technologies: Proceedings of DaSET 2023. Lecture Notes on Data Engineering and Communications Technologies, vol. 191, Springer Nature Link, Singapore, pp. 49-61, The International Conference on Data Science and Emerging Technologies DaSET 2023, Malaysia, 4/12/23. https://doi.org/10.1007/978-981-97-0293-0_4

The Performance of GPT-3.5 in Summarizing Scientific and News Articles. / Arshad, Sabkat; Yaqoob, Muhammad; Mehmood, Tahir.
Data Science and Emerging Technologies: Proceedings of DaSET 2023. ed. / Yap Bee Wah; Dhiya Al-Jumeily OBE; Michael W. Berry. Singapore: Springer Nature Link, 2024. p. 49-61 (Lecture Notes on Data Engineering and Communications Technologies; Vol. 191).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - The Performance of GPT-3.5 in Summarizing Scientific and News Articles

AU - Arshad, Sabkat

AU - Yaqoob, Muhammad

AU - Mehmood, Tahir

N1 - Conference code: 2

PY - 2024/4/27

Y1 - 2024/4/27

N2 - In the age of information, we are overwhelmed with large amounts of data. The quest to know more in less time has increased the need for efficient text summarization models that convert information into precise summaries such that essential details are not overlooked. Recently, GPT-3.5 has demonstrated impressive performance in text completion, generation, and question answering. However, its effectiveness in generating concise and coherent summaries for scientific articles and news reports remains under-explored. This work evaluates the performance of GPT-3.5 in summarizing scientific research articles and news data. Scientific articles were collected from arXiv STEM dataset, whereas news articles were sampled from the CNN/DailyMail dataset. Using the GPT-3.5 OpenAI API, the pre-trained model is prompted to generate summaries of the scientific and news articles. In the next step, the ROUGE score is computed for the generated summaries against the reference summaries to analyse the performance of the model. Our results show that GPT-3.5 performs slightly better in summarizing scientific articles as compared to news articles with an average ROUGE score of 0.35 and 0.31, respectively. Moreover, in agreement with the literature, we show that the ROUGE is not the best measure for evaluating text similarity as it heavily relies on similar vocabulary rather than semantics.

AB - In the age of information, we are overwhelmed with large amounts of data. The quest to know more in less time has increased the need for efficient text summarization models that convert information into precise summaries such that essential details are not overlooked. Recently, GPT-3.5 has demonstrated impressive performance in text completion, generation, and question answering. However, its effectiveness in generating concise and coherent summaries for scientific articles and news reports remains under-explored. This work evaluates the performance of GPT-3.5 in summarizing scientific research articles and news data. Scientific articles were collected from arXiv STEM dataset, whereas news articles were sampled from the CNN/DailyMail dataset. Using the GPT-3.5 OpenAI API, the pre-trained model is prompted to generate summaries of the scientific and news articles. In the next step, the ROUGE score is computed for the generated summaries against the reference summaries to analyse the performance of the model. Our results show that GPT-3.5 performs slightly better in summarizing scientific articles as compared to news articles with an average ROUGE score of 0.35 and 0.31, respectively. Moreover, in agreement with the literature, we show that the ROUGE is not the best measure for evaluating text similarity as it heavily relies on similar vocabulary rather than semantics.

KW - ChatGPT

KW - Large language model

KW - Natural language processing

KW - Scientific papers

KW - Text summarization

UR - http://www.scopus.com/inward/record.url?scp=85192768243&partnerID=8YFLogxK

U2 - 10.1007/978-981-97-0293-0_4

DO - 10.1007/978-981-97-0293-0_4

M3 - Conference contribution

SN - 978-981-97-0292-3

T3 - Lecture Notes on Data Engineering and Communications Technologies

SP - 49

EP - 61

BT - Data Science and Emerging Technologies

A2 - Wah, Yap Bee

A2 - OBE, Dhiya Al-Jumeily

A2 - Berry, Michael W.

PB - Springer Nature Link

CY - Singapore

T2 - The International Conference on Data Science and Emerging Technologies DaSET 2023

Y2 - 4 December 2023 through 5 December 2023

ER -

Arshad S, Yaqoob M, Mehmood T. The Performance of GPT-3.5 in Summarizing Scientific and News Articles. In Wah YB, OBE DAJ, Berry MW, editors, Data Science and Emerging Technologies: Proceedings of DaSET 2023. Singapore: Springer Nature Link. 2024. p. 49-61. (Lecture Notes on Data Engineering and Communications Technologies). Epub 2024 Apr 27. doi: 10.1007/978-981-97-0293-0_4

The Performance of GPT-3.5 in Summarizing Scientific and News Articles

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this