Human intelligence versus artificial intelligence in classifying economics research articles: exploratory evidence

The Journal of Economic Literature (JEL) classification codes system maintained and published by the American Economic Association is the de facto standard for classifying research papers in economics (Cherrier, 2017; Heikkilä, 2021, 2022; Bornmann and Wohlrabe, 2024). Data on JEL classification codes has been utilized to analyze the evolution of published economics papers by fields and styles (e.g. Card and DellaVigna, 2013; Angrist et al., 2017; Bornmann and Wohlrabe, 2024). Kosnik (2018) has documented that there can be differences in the author-assigned and editor-assigned JEL codes (in the American Economic Review journal) and Heikkilä (2022) further illustrated that JEL codes of economics working papers can differ from those of the final peer-reviewed and published articles. Concurrently, the micro task of classifying economics research papers has become increasingly complex. For instance, Bornmann and Wohlrabe (2024) report that the average number of JEL codes per paper has increased steadily from about 1.9 in 1991 to 4.3 in 2021.

Using “human intelligence” may lead to subjective and boundedly rational choices (Artinger et al., 2022) and different researchers choose different JEL codes that in their subjective opinion given their evolving beliefs and expectations are the most appropriate ones. It is an empirical question, to what extent researchers in reality familiarize themselves with the JEL Classification Codes guide when choosing JEL codes (Heikkilä, 2022) for their articles and how much there is rational inattention (Maćkowiak et al., 2023). The guide is available online ^[1] and it has quite detailed instructions for the use of specific JEL codes so that it requires a costly and time-consuming effort to learn and follow the guide’s recommendations. Presumably, there is a non-negligible amount of rational inattention around the use of JEL codes: researchers may think that the expected benefits of choosing appropriate JEL codes may not exceed the costs of learning the detailed instructions provided by the JEL codes guide.

It is important to keep in mind that originally, JEL codes classification system and its predecessors were developed to decrease search costs in the “paper era” in the 20th century (Cherrier, 2017), but now that we live in the era of digitized information, researchers can easily conduct their searches using online search engines that enable keyword searches from full texts of articles which has tremendously decreased search costs (cf. Goldfarb and Tucker, 2019). As in recent years, we have seen significant progress in the field of large language models (LLM) ^[2] and generative AI ^[3], we are expecting an increasing number of tasks to be automated (Brynjolfsson et al., 2023; Eloundou et al., 2024). Presumably, search costs will further decrease and time savings related to literature reviews increase as continuously improving AI systems can review and process whole databases of research papers. Recently, Korinek (2023) argued that “economists can reap significant productivity gains by taking advantage of generative AI to automate micro-tasks” and it seems that choosing JEL codes for research papers is one such task. Thus, the simple research question that is explored in this paper is the following: Can generative AI systems augment human intelligence in the choice of appropriate JEL codes? The answer is positive and this is demonstrated with a simple experiment next.

Do researchers study AEA’s JEL codes guide when submission systems of journals require them to assign appropriate JEL codes to their articles? Presumably, in this micro task, many researchers do not optimize but rather search JEL codes until they are satisfied (meeting satisficing aspiration level, cf. Simon, 1955; Artinger et al., 2022) with their “good enough” JEL code choices and stop searching (cf. Caplin et al., 2011). Due to technological progress, AI can nowadays be utilized either to replace human choice of JEL codes altogether or complement and augment human intelligence in such a classification task as illustrated in Figure 1. On the one hand, AI systems can base their choices on a huge and accumulating amount of training data which processing is beyond human capacity. On the other hand, AI systems may not be able to adapt to changes (e.g. new classification codes) in the JEL codes classification system as flexibly as humans, for instance when no existing articles in the training data are associated with novel JEL codes. As both human intelligence and artificial intelligence have their pros and cons, augmented intelligence might be the preferred option – at least for now.

However, there is no agreed concept of “optimal choice of JEL codes” – as there is no clear and consistent guidelines for the choice of keywords for articles (cf. Lu et al., 2020). It is not unreasonable to assume that there is no common knowledge of JEL code choice criteria. Therefore, we intentionally do not define the concept “optimal” (or appropriate) here - that is, what and whose preferences ought to define what should be optimized in the choice of JEL codes. However, we prompted selected AI systems (OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini) to explain what they consider to be important using the following prompt:

Table A1 in the  Appendix summarizes the answers of AI systems. To summarize, they generally note the goal of the JEL code choice to be to maximize visibility, discoverability and impact (ChatGPT 3.5 and 4 refer explicitly to citations) among appropriate and intended academic audiences. All selected AI systems list relevance and accuracy - JEL codes should accurately reflect the content (topics, fields, themes, focus, scope, methodologies) of the articles - as key factors to consider. Specificity is also listed, but its definition is slightly ambiguous. ChatGPT 4 highlights that choosing specific JEL codes “can increase the visibility of the article among researchers who are working on the same niche” while Gemini recommends to “avoid overly general secondary codes”. Copilot and Gemini recommend consulting AEA’s JEL codes guide. While Copilot advises not to overuse JEL codes, a bit surprisingly Gemini recommends limiting the choice to a maximum of two JEL codes (however, it recommends more codes itself as demonstrated in Table 1a and 2a below). ChatGPT 3.5 considers consistency with the JEL codes used in the existing literature and maintaining coherence within the academic discourse important and Gemini recommends considering journal audience and look at the typical JEL codes used in your target journal (even discuss the JEL code choice with colleagues). Copilot links the choice of JEL codes to the choice of keywords by noting that “look for keywords and phrases within the article that match the JEL code descriptions”. Other factors listed include interdisciplinarity, current trends and emerging topics.

In order to compare the JEL code choices between human intelligence (by researchers) and artificial intelligence, we chose a set of articles, the latest published issue of Journal of Economic Literature as of 12 April 2024. This is the March 2024 issue, issue 1 of volume 62. It includes seven articles that are shown in Table 1. The Journal of Economic Literature journal is particularly appropriate case to study the choice of JEL codes as it is the outlet where the JEL classification system was introduced in 1969 and has been publishing the official classification system ever since ^[4]. We proceeded by prompting three generative AI systems (OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini) to assign JEL codes to the articles using the following prompt ^[5]:

Table 1 compares the JEL codes assigned to the articles (shaded columns) to the JEL codes that AI systems assigned to the articles based on their abstracts only (Table 1a) and based on full texts (Table 1b) of earlier working paper versions of the same articles as well as the final versions. The option to provide the AI system with the full text is not available for all the used AI systems and the full texts of the articles are mainly available for subscribers only. Thus, to enable comparisons between different AI systems we used only abstracts in Table 1a since the abstracts are publicly available online for anyone.

Since there are often multiple versions of earlier working papers, in Table 1b we chose the earliest working paper versions that we found from IDEAS RePEc and Google Scholar and prioritized major established working paper series in economics (e.g. NBER, CEPR, cf. Baumann and Wohlrabe, 2020) ^[6]. This illustrates how the set of JEL codes assigned to a working paper – presumably, typically by authors themselves – can be significantly different from the ones assigned to the peer-reviewed final paper.

Several patterns can be observed even from this limited set of articles. First, both AI systems and authors assign typically systematically less JEL codes compared to the final set and authors often assign less JEL codes than AI. In the case of AI systems, this could be explained by the fact that we prompted the AIs to suggest JEL codes based on the abstract only and not based on the whole article. For instance, Gemini typically suggests three JEL codes, one primary and two secondary ones. The justification for the choice of the specific JEL codes by the AIs is reasonable and Gemini even provides “justification for excluding other JEL codes” and explains why some selected JEL codes would not be appropriate.

Second, the overlap between the ones suggested by AI systems based on abstracts and final JEL codes varies in the range of 10%–30% (Table 1a). This may seem low, but it should be noted that AI systems (as well as authors) suggest systematically less JEL codes. Unlike other selected AI systems, Copilot provides by default the information sources underlying its reasoning for the JEL codes. The investigation of these sources reveals that in multiple cases Copilot refers to the publicly available working papers or the website of the final version of the underlying article where the abstract is available. Sometimes Copilot ends its answer: “For more details on the paper, you can refer to the [link to the article here].”

Third, we also tested which JEL codes ChatGPT 4 would assign to the selected working paper version and the final peer-reviewed articles based on the text of the whole article (Table 1b). Despite the fact that the differences between the content, focus and scope of the working paper version and the final article are relatively minor, ChatGPT 4 assigns quite different JEL codes to them. It seems that in most cases ChatGPT 4 recommends exactly the JEL codes listed in the articles and argues why they are appropriate.

To conclude, generative AI systems can augment human intelligence in choosing the JEL codes by providing reasoned suggestions based on the article abstracts only.

Next, in order to test the robustness of our observation regarding discrepancies of JEL code choices between human and artificial intelligence in the Journal of Economic Literature, we applied the same method to the nine articles published in the latest (as of April 2024) issue of the American Economic Review (AER).

Again, as Table 2 presents, we find that the generative AI can suggest reasonable JEL codes and provide the reasoning why these could be the optimal set of JEL codes (see Supplementary material). Again, in line with Kosnik’s (2018) and Heikkilä’s (2022) observations, we find that author-assigned JEL codes to the working paper versions (Table 2b) often differ from the final assigned JEL codes.

Again, authors (Table 2b) and AI systems assign typically less JEL codes and Table 2a indicates that the overlap between the JEL codes assigned by AI systems based on abstract and the final JEL codes ranges between ca. 18% (Gemini) and 65% (Copilot). For both ChatGPT 3.5 and 4 the overlap is about 40%. While Copilot sometimes refers to the websites where the abstract of the article - and related JEL codes - are publicly available, it often does not pick those exact JEL codes but rather assigns a smaller number of JEL codes. For this set of articles, the recommended JEL codes by ChatGPT4 based on the whole articles have lower overlap (ca. 70%) with the final ones compared to the overlap reported in Table 1b (ca. 93%).

In the prior literature it is common to focus on more aggregated levels of JEL codes instead of the most granular ones as we did in Tables 1 and 2. For instance, Card and Della Vigna (2013) classified articles based on their own classification where JEL codes were aggregated into 14 field categories and recently Bornmann and Wohlrabe (2024) focused in their analyses on the level of 20 primary JEL code categories (see also Kosnik, 2018). In Tables A2 and A3 in the  Appendix, we apply this aggregated approach based on the 20 primary JEL code categories to the articles presented in Tables 1 and 2. These further analyses show a higher level of overlap as expected (ranging between 50–100%) indicating that while the AI systems may not recommend exactly the same JEL codes, they at least recommend JEL codes from the same JEL code categories in most of the cases. It seems that the imperfect overlap stems mainly from the fact that AI system as well as authors select fewer JEL codes compared to the final editor-assigned JEL codes of the articles.

The exploratory evidence presented here indicates that generative AI may help researchers and augment human intelligence in the choice of appropriate JEL codes for their articles. At minimum, generative AI can be used to cross-check the choices of researchers if not to fully automate the micro task. Thus, augmented intelligence may make the use of JEL classification codes more efficient and consistent and save researchers’ time.

Since learning the nuances of the AEA’s JEL codes guide requires costly and time-consuming effort, it seems economic to utilize AI systems in (partially) automating the micro-task of choosing JEL codes. Our simple experiment indicates that generative AI may help researchers and augment their boundedly rational human intelligence in choosing appropriate JEL codes for their articles based on abstracts. However, we also documented that there are significant discrepancies between the chosen JEL codes by humans and the selected generative AI systems as well as between the AI systems.

If the scientific community and research publishers want to continue classifying research papers using JEL codes, then the use of AI may help make the human choice of JEL codes less boundedly rational and more consistent (cf. Kosnik, 2018). While it remains an open question how to define “appropriate” or “optimal” choice of JEL codes, AI can save time and help in finding more “satisficing” (Simon, 1955; Caplin et al., 2011; Artinger et al., 2022) sets of JEL codes. More consistent use of JEL codes improves the training data of AI systems. When this is complemented with automated recommendation systems that suggest JEL codes best describing research content, it could further decrease the search costs of the audiences as well as promote the analysis of research trends based on JEL codes (cf. Card and DellaVigna, 2013; Angrist et al., 2017; Bornmann and Wohlrabe, 2024).

We acknowledge that the presented preliminary observations have several limitations. First, the analysis focuses on only two recent issues of leading economics journals, so the external validity of the observations is limited. More extensive analyses of larger numbers of articles across a more diverse set of journals would lead to more credible and generalizable observations.

Second, there is a hallucination problem with large language models – that is, they may generate text that is not true (Zhai, 2024). In this analysis we did not try to detect hallucination, but a more rigorous analysis of JEL code choices with larger sets of articles should be accompanied with the check of reasoning for each selected JEL code (to confirm that the AI systems do not come up with any hallucinated JEL codes).

Third, “model collapse” (Shumailov et al., 2024) is another detrimental phenomenon which refers to the degenerative recursive process where AI systems trained with polluted (incl. hallucinated) data end up training the next generation of AI systems with model-generated polluted data. Similarly, in the context of JEL codes, if AI systems are again and again trained with data were inappropriate or hallucinated JEL codes are statistically linked to specific articles, this probably compounds the biases dynamically.

Fourth, the analysis used only a very limited set of AI systems. As the development of AI systems continues, authors can consult an increasing number of continuously improving AI systems to cross-check their recommendations of optimal JEL codes.

While the preliminary findings presented here will become quickly obsolete as AI systems (incl. training data) improve and are increasingly utilized, future studies may apply the presented approach to analyze whether the JEL code choices between authors, editors and AI systems converge and become more consistent over time.

I thank two anonymous reviewers for their helpful comments. Financial support from the Päijät-Häme Regional Fund of the Finnish Cultural Foundation is gratefully acknowledged.

The supplementary material for this article can be found online.