There is an increase in the use of large language models (LLMs) in information science, including evaluating academic journal articles. Despite this, it is unclear whether they “know” about articles in the sense of being able to answer simple questions about individual papers without web searches.
In this study, 4 questions were asked of ChatGPT 4o-mini about 64,055 academic journal articles (excluding reviews) from 2021, identified by their titles and abstracts, with uncited and highly cited articles also assessed by ChatGPT 4.1 and 5 open weight LLMs.
The results were mostly incorrect, even for the most cited articles from that year. In particular, ChatGPT 4o-mini and the open weights LLMs had almost no knowledge of an article’s first author affiliation, rarely knew the publishing journal and usually guessed the publication year wrong, although ChatGPT 4o-mini was 42% correct for Physical Review B. Even ChatGPT 4.1 could only identify a small majority of the journals for the top cited papers of the year.
Smaller LLMs’ lack of basic knowledge about articles suggests that when they are asked to evaluate them without web searches, they will rarely cheat by eliciting citation information or journal reputation but will instead answer based on the article text because they may not associate online criticisms with individual articles.
This is the first investigation of the ability of LLMs to recall basic facts about journal articles.
