Large Language Models (LLMs) can be helpful for literature search and summarization but retracted articles can confuse them. It is therefore important to know how aware LLMs are of retractions.
Three open-weight (offline) LLMs were asked whether 161 high-profile retracted articles had been retracted, with a similar check for a benchmark multidisciplinary set of 34,070 non-retracted articles.
From titles and abstracts, in over 80% of cases the LLMs claimed that a retracted article had not been retracted (GPT OSS 120 B: 82%; Gemma 3 27 B: 84%; DeepSeek R1 72 B: 88%). Reasons for a correct retraction declaration were often wrong. For the benchmark test, there were only 55 false retraction claims from 34,070 non-retracted full-text articles, and 28 false claims when only the title and abstract were entered, suggesting that there is only a small chance that LLMs discount valid studies. When retractions are erroneously claimed, this does not seem to be due to mistakes in the article.
The results confirm for the first time that offline LLMs have little ability to distinguish between valid and retracted studies, unless they are allowed to, and do, check online. The results therefore give new evidence, based on LLM knowledge of retractions, to be cautious about LLM claims about academic findings.
