Examples of recent studies on GPLLMs
| No | References | GPLLMs | Specific domain | Test datasets | ||
|---|---|---|---|---|---|---|
| Question sources | Question types | Number of questions | ||||
| 1 | Alan et al. (2024) | GPT-3.5 turbo | Islam understanding | Designed by experts | Open-ended | 3 (mentioned by the author) |
| 2 | Hou and Zhang (2024) | GPT-3.5 and GPT-4.0 | Dietary supplement | Information on the MSKCC website | Closed-ended (MSQs and True/False) | 2000 |
| 3 | Mansurova et al. (2024) | Llama-2-7b and Llama-2-13b | General | TriviaQA open-domain dataset | Closed-ended (Filling in the blank) | 500 |
| 4 | Rasool et al. (2024) | GPT-3.5-turbo and GPT-4 | Healthcare | CogTale dataset | Closed-ended (MMQs, MSQs, True/False, and number extraction) | 337 |
| 5 | Rizzo et al. (2024) | GPT-3.5 turbo and GPT-4 | Orthopaedics | OITE in the 2020, 2021, and 2022 | Closed-ended (MSQs) | 207 |
| 6 | Sahin et al. (2024) | GPT-4 | Neurosurgery | The latest six written TNSPBE | Closed-ended (MSQs) | 523 |
| 7 | Schoch et al. (2024) | GPT-3.5 and GPT-4 | Urology | A test book published by the FEBU association | Closed-ended (MSQs) | Around 600 |
| 8 | Su et al. (2024) | GPT-4 | Nursing | Taiwan’s 2022 Nursing Licensing Exam | Closed-ended (MSQs) | 400 |
| 9 | Tsoutsanis and Tsoutsanis (2024) | Llama-2, Google Bard, Bing Chat, and GPT-3.5 | Clinical help | Commercial question banks (i.e. Qbank) for the MSRA exam | Closed-ended (MSQs) | 100 |
| 10 | Antaki et al. (2023) | GPT-3.5 turbo and GPT-4 | Ophthalmology | Basic and Clinical Science Course Self-Assessment Program and an online question bank (i.e. OphthoQuestions) | Closed-ended (MSQs) | 520 |
| 11 | Choi et al. (2023) | ChatGPT | Laws | Exams for law school courses at the University of Minnesota | Closed-ended (MSQs) and open-ended (essay writing) | 107 |
| 12 | Gencer and Aydin (2023) | GPT-3.5 and GPT-4 | Thoracic surgery | Turkish-language thoracic surgery exam questions | Closed-ended (MSQs) | 105 |
| 13 | Gilson et al. (2023) | InstructGPT, GPT-3.5, and ChatGPT | Medicine | A question bank for medical students and the NBME | Closed-ended (MSQs) | 220 |
| 14 | Oh et al. (2023) | GPT-3.5 and GPT-4 | Surgery | The KGSBE in 2020, 2021, and 2022 | Closed-ended (MSQs) | 280 |
| 15 | Pursnani et al. (2023) | GPT-3.5-Legacy, GPT-3.5-Turbo, and GPT-4 | Engineering fundamental knowledge | An unpublished practice exam | Closed-ended (MSQs, MMQs, and filling in the blank) | 134 |
| 16 | Rosól et al. (2023) | GPT-3.5 and GPT-4 | Medicine | 3 versions of PMFE | Closed-ended (MSQs) | 600 |
| 17 | Saad et al. (2023) | GPT-4 | Orthopedics | Mock FRCS Orth Part A | Closed-ended (MSQs) | 240 |
| No | References | GPLLMs | Specific domain | Test datasets | ||
|---|---|---|---|---|---|---|
| Question sources | Question types | Number of questions | ||||
| 1 | GPT-3.5 turbo | Islam understanding | Designed by experts | Open-ended | 3 (mentioned by the author) | |
| 2 | GPT-3.5 and GPT-4.0 | Dietary supplement | Information on the MSKCC website | Closed-ended (MSQs and True/False) | 2000 | |
| 3 | Llama-2-7b and Llama-2-13b | General | TriviaQA open-domain dataset | Closed-ended (Filling in the blank) | 500 | |
| 4 | GPT-3.5-turbo and GPT-4 | Healthcare | CogTale dataset | Closed-ended (MMQs, MSQs, True/False, and number extraction) | 337 | |
| 5 | GPT-3.5 turbo and GPT-4 | Orthopaedics | OITE in the 2020, 2021, and 2022 | Closed-ended (MSQs) | 207 | |
| 6 | GPT-4 | Neurosurgery | The latest six written TNSPBE | Closed-ended (MSQs) | 523 | |
| 7 | GPT-3.5 and GPT-4 | Urology | A test book published by the FEBU association | Closed-ended (MSQs) | Around 600 | |
| 8 | GPT-4 | Nursing | Taiwan’s 2022 Nursing Licensing Exam | Closed-ended (MSQs) | 400 | |
| 9 | Llama-2, Google Bard, Bing Chat, and GPT-3.5 | Clinical help | Commercial question banks (i.e. Qbank) for the MSRA exam | Closed-ended (MSQs) | 100 | |
| 10 | GPT-3.5 turbo and GPT-4 | Ophthalmology | Basic and Clinical Science Course Self-Assessment Program and an online question bank (i.e. OphthoQuestions) | Closed-ended (MSQs) | 520 | |
| 11 | ChatGPT | Laws | Exams for law school courses at the University of Minnesota | Closed-ended (MSQs) and open-ended (essay writing) | 107 | |
| 12 | GPT-3.5 and GPT-4 | Thoracic surgery | Turkish-language thoracic surgery exam questions | Closed-ended (MSQs) | 105 | |
| 13 | InstructGPT, GPT-3.5, and ChatGPT | Medicine | A question bank for medical students and the NBME | Closed-ended (MSQs) | 220 | |
| 14 | GPT-3.5 and GPT-4 | Surgery | The KGSBE in 2020, 2021, and 2022 | Closed-ended (MSQs) | 280 | |
| 15 | GPT-3.5-Legacy, GPT-3.5-Turbo, and GPT-4 | Engineering fundamental knowledge | An unpublished practice exam | Closed-ended (MSQs, MMQs, and filling in the blank) | 134 | |
| 16 | GPT-3.5 and GPT-4 | Medicine | 3 versions of PMFE | Closed-ended (MSQs) | 600 | |
| 17 | GPT-4 | Orthopedics | Mock FRCS Orth Part A | Closed-ended (MSQs) | 240 | |
Note(s): CogTale: Cognitive Treatments Article Library and Evaluation FEBU: Fellow of the European Board of Urology; FRCS Orth: Orthopedic fellow of the Royal College of Surgeons; iDISK: International Dietary Supplement Knowledgebase; KGSBE: Korean General Surgery Board Exams; MSKCC: Memorial Sloan Kettering Cancer Center; MSRA: Multi-Specialty Recruitment Assessment; NBME: National Board of Medical Examiners; OITE: Orthopedic In-Training Examination; PMFE: Polish Medical Final Examination; TNSPBE: Turkish Neurosurgical Society Proficiency Board Exams
Source(s): Authors’ own work