Artificial intelligence (AI) and ChatGPT involvement in scientific and medical writing, a new concern for researchers. A scoping review

The scientific research process, including data analysis, manuscript writing and publication, is essential for individual researchers and the scientific community throughout various disciplines and specialties, eventually leading to knowledge advancement and contributing to individual researchers’ career development (Iskander, Wolicki, Leeb, & Siegel, 2018; Kennedy, 2018; Khalifa, 2022; Pittman, Stahre, Tomedi, & Wurster, 2017). However, this process could be exhausting, time-consuming and requiring teamwork, making researchers eager to find and develop research and writing assisting tools and applications (Pittman et al., 2017; Salvagno, Taccone, & Gerli, 2023).

Artificial intelligence (AI) has been involved in different aspects of our life for a considerable time, and scientific research is not an exception (Haenlein & Kaplan, 2019; Salvagno et al., 2023); however, starting from November 2022, the idea and usage of AI was transformed enormously after the introduction of Generative Pretrained Transformer “ChatGPT” (OpenAI, San Francisco, CA, USA), which is an open AI platform trained on a massive amount of online data (including media, books and articles), enabling this tool to perform sophisticated tasks (Gao et al., 2023; Sallam, 2023).

AI and ChatGPT soon found their way to be extensively involved in nearly all aspects of scientific research, including data collection, analysis, drafting manuscripts and final formatting (Alattar & McDowell, 2023; Alkaissi & McFarlane, 2023; Qasem, 2023), which created fears and controversies among the scientific communities, where some researchers are very enthusiastic for these new tools (Chen, 2023; O'Connor, 2023). In contrast, others reported considerable concerns and worries related mainly to misusing these tools (Kunze, Jang, Fullerton, Vigdorchik, & Haddad, 2023; Leopold, Haddad, Sandell, & Swiontkowski, 2023; Majovsky, Cerny, Kasal, Komarc, & Netuka, 2023; Stokel-Walker, 2023).

The primary objective of the current review was to evaluate the recent PubMed publications on ChatGPT or AI involvement in scientific or medical writing. The secondary objectives were to determine whether ChatGPT or AI was used in creating any of these articles, and, if yes, whether it was listed as an author or acknowledged by the human authors. Furthermore, the aim was to shed some light on the advantages, disadvantages and future perspectives reported in the included articles.

Design: We used the scoping review methodology according to Arksey and O'Malley (Arksey & O'Malley, 2005). The review was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines (Tricco et al., 2018) ( Supplementary file, Table S1). Per Arksey and O'Malley (Arksey & O'Malley, 2005), we followed their recommended six steps for conducting the current scoping review; however, they reported that the last is optional.

This was mentioned earlier.

After performing a preliminary screening of the literature to obtain and formulate an idea about the search terms to be used, the two authors independently created search terms, which were discussed among the authors to agree on a final search strategy. The Title/Abstract filed of the PubMed database was searched from January 1, 2023 to November 29, 2023, using the following search strategy: ((((ChatGPT[Title/Abstract]) OR (AI[Title/Abstract])) OR (Artificial intelligence[Title/Abstract])) AND (medical writing[Title/Abstract])) OR (scientific writing[Title/Abstract]). The search was limited to articles on humans and published in the English language.

We decided to include all articles (regardless of type) discussing AI or ChatGPT involvement in scientific writing. Furthermore, excluded articles reported on the role of AI or ChatGPT in medical education, disease diagnosis, radiograph assessment, writing medical reports and if the article was not centered on scientific writing and its related aspects.

All the results were downloaded to the Endnote program in reference form for evaluation against the predetermined inclusion and exclusion criteria. Both authors independently performed traditional screening (first by title and abstract, followed by full-text screening); any disagreement was resolved by discussion between the authors until a consensus was reached.

After agreeing on the finally included articles, each author independently extracted data into a predesigned Excel sheet. The following data were collected: first, study characteristics: Authors, country of origin, journal where the article was published and type of the article; second, evaluating the content of the articles and the central theme the authors discussed; and, finally, to document the role of AI or ChatGPT in creating the articles (if it was used or not? If used, was it listed as an author or not? If its use was mentioned in the acknowledgment section or not?). The data extracted by the authors were compared to resolve any discrepancies by discussion and agreement among the authors.

As we aimed to formulate a scoping review, the collected findings were used to provide an overview of the subject in concern rather than an individual study quality assessment. The results were grouped and presented in a table. A brief qualitative assessment was performed, and a descriptive summary was reported. Both authors discussed the final results and agreed on the final presentation.

Arksey and O'Malley suggested an optional final step, which is consultation with stakeholders and experts to provide ideas beyond what is collected from the literature; however, this step was not needed for the current review and was not performed.

The initial search revealed 127 articles, of which 41 were eligible for final analysis (Abuyaman, 2023; Alkaissi & McFarlane, 2023; Altmae, Sola-Leyva, & Salumets, 2023; Athaluri et al., 2023; Babl & Babl, 2023; Buholayka, Zouabi, & Tadinada, 2023; Chen, 2023; Currie, 2023; Daugirdas, 2023; Dave, Athaluri, & Singh, 2023; Doyal, Sender, Nanda, & Serrano, 2023; Dunn et al., 2023; Gandhi & Gandhi, 2023; Gao et al., 2023; Giglio & Costa, 2023; Gilat & Cole, 2023; Homolak, 2023; Huang & Tan, 2023; Khlaif et al., 2023; Koga, 2023; Lee, Salim, Abdullah, & Teo, 2023; Levin, Brezinov, & Meyer, 2023; Macdonald, Adeloye, Sheikh, & Rudan, 2023; Majovsky et al., 2023; Marescotti, 2023; Mavrogenis & Scarlat, 2023; Misra & Chandwar, 2023; Ocampo, Silva, Alencar-Palha, Haiter-Neto, & Oliveira, 2023; Pal, Bhattacharya, Islam, & Chakraborty, 2023; Rozencwajg & Kantor, 2023; Sallam, 2023; Salvagno et al., 2023; Shafiee, 2023; Svab, Klemenc-Ketis, & Zupanic, 2023; Temsah et al., 2023; Verhoeven, Wendling, & Prati, 2023; Wohlfarth, Streit, & Guttormsen, 2023; Wu & Dang, 2023; Xia & Wang, 2023; Yasin & Al-Hamad, 2023; Zheng & Zhan, 2023). The PRISMA flow diagram illustrates the search results (Figure 1).

Articles were published in 34 different journals. The Cureus journal had the highest contributions, with six (14.6%) articles, and editorials were the most common article type, with 15 (36.6%) (Figure 2). Authors originated from 27 countries, and authors from the USA contributed the most, in 14 (34.1%) articles (Figure 3).

The included articles discussed various aspects of AI and ChatGPT involvement in scientific writing; however, the three main topics (themes) discussed were, first, AI tools and writing capabilities in 19 (46.3%) articles; second, the advantages and disadvantages in 12 (29.3%) articles; and, lastly, 11 (26.8%) articles discussed ethical issues and concerns. As reported by the authors, AI or ChatGPT was involved in various aspects of article preparation (manuscript drafting, study or experiment formulation and language corrections) in 31 (75.6%) articles. None of the articles listed AI or ChatGPT as an author, and in 19 (46.3%) articles, the authors mentioned utilizing AI or ChatGPT in the acknowledgments. Details and characteristics of the included articles are shown in Table 1.

Although various AI-assistive scientific writing tools are present and being increasingly used over the past few years (Chen, 2023; Katsnelson, 2022), it is evident that the introduction of ChatGPT as a new tool involved in various aspects of scientific research and writing occurred rapidly and grasped the attention and concern of many researchers and scientists. The current review showed that articles discussing AI and ChatGPT involvement in scientific writing were published by authors from different parts of the world in various journals; furthermore, a lot of these authors themselves tried ChatGPT and its ability to formulate scientific text, besides reporting its advantages, disadvantages and the ethical concerns related to its use.

Machine learning (ML) is considered an advancement of AI, where a decision could be made using ML algorithms based on data collected from large data sets (Sallam, 2023). Furthermore, neural networks (NN) are considered a specific type of ML algorithm capable of recognizing even more complex data sets patterns, which was ultimately advanced to form the large language models (LLMs) trained on a vast amount of data enabling these models to generate content-defined text and predict the best following text elements, thus leading to the production of naturally read texts (Choi, Coyner, Kalpathy-Cramer, Chiang, & Campbell, 2020; Gao et al., 2023; Luitse & Denkena, 2021).

ChatGPT recently became one of the most popular LLMs used by personnel among different disciplines; soon, it found its way to being deeply involved in scientific research and academic writing, owing to the large amount of data it was trained on and its human-like writing capabilities (Buholayka et al., 2023; Sallam, 2023). However, there is still an ongoing debate and hassle among journals, publishers and even individual researchers regarding the optimum ChatGPT use in scientific research activities and ethical regulations controlling its involvement in the scientific writing process (Stokel-Walker, 2023; Thorp, 2023; Tools such as ChatGPT threaten transparent science; here are our ground rules for their use, 2023).

In a bibliometric analysis by Levin et al., the authors reported 42 articles published in PubMed after they used the search term “ChatGPT” on February 15, 2023 (Levin et al., 2023). At the time of preparing the current review, a new search was performed using the same search terms used by Levin et al., which revealed 1809 results, meaning that in about ten months, the publications increased by about 43 folds, indicating a plethora of increasing concerns of this topic. On the contrary, in the current review, the initial search revealed 127 articles, which is considerably few compared to the results obtained from using the search terms mentioned in the previous study; however, this could be attributed to the high selectivity of the articles using strict search terms.

Moreover, in the study by Levin et al., the most common publication type was editorials, constituting 52%, and authors from the UK contributed the most (33%). In the current review, we reported the same results regarding the article type where editorials represent the commonly published article type (36.6%); however, authors from the USA were the highest contributors (34.1%). The predominance of editorials, which are mostly written by eminent researchers, reflects the concern regarding this considerably new phenomenon; furthermore, contributing authors are from countries across the world, indicating the global interest in evaluating this issue.

The articles included in the current review handled various themes and topics, showing the authors’ interest in discussing and evaluating various aspects of AI or ChatGPT involvement in scientific writing; furthermore, in some articles, more than one theme was discussed (Table 1). The most commonly evaluated issue was the role of AI or ChatGPT as an assistant during different aspects of manuscript writing (language corrections, especially for non-English speakers, abstract formulation, building a structured manuscript and obtaining related references). Then, various articles reported the advantages and disadvantages of these tools’ involvement in the scientific writing process, and few provided some recommendations and future perspectives for their use. Lastly, the ethical issues and increasing concerns regarding the misuse of these tools were raised in some articles (Table 2).

The current review has some limitations. First, the search was performed in one database, which could lead to missing some articles published in other databases. Second, using restricted search terms and focusing on a specific research point led to excluding articles discussing other aspects of scientific research rather than manuscript writing and preparation. Third, a full bibliometric analysis (including the number of citations of each article) was not performed. Fourth, as this was a scoping review focusing on a few points, only qualitative data synthesis and descriptive presentation were performed. Further, more detailed systematic reviews, including quantitative analysis (metanalysis), could be performed to provide more insight into AI and ChatGPT’s role in scientific research and overcome the current review’s drawbacks. Last, we could not offer recommendations for the proper utilization of AI or ChatGPT; however, we believe the current review could work as a foundation for formulating solid recommendations and regulations based on the data we reported.

Interests and worries regarding using AI or ChatGPT in different aspects of scientific research, including specifically the writing process, pose an increasing concern among researchers in different specialties. It is too early to judge these new tools; however, it is apparent that these are here to stay, and the main burden is on the shoulders of individual researchers and their moral and ethical consciousness to control their judicious use of these tools and applications. We believe that precise and mature regulations will be developed soon by journals, publishers and editors, which will pave the way for the best usage of these tools.

As Ridley Scott said, “Life is not black and white. It is a million gray areas, don’t you find?” there is no absolute white (right) or black (wrong) regarding AI or ChatGPT involvement and usage in scientific research and writing; however, we are waiting for millions of gray areas, which possibly will appear shortly.

ChatGPT or other AI tools were not used to generate any aspect of the current article. The authors are sincerely grateful to the reviewers for their time and effort in reviewing the manuscript, which provided insightful and valuable comments that helped improve the final manuscript’s quality.

The supplementary material for this article can be found online.