Using Generative AI to Calculate Party Positions: A Comparison of Human Experts and Large Language Models

The spatial theory of voting, first introduced by Downs (1957), posits that voters choose political parties based on their positions on the political spectrum, from left to right. Traditionally, left-leaning voters favor progressive policies and government intervention, while right-leaning voters support conservative values and free markets. Centrists seek a balance. This framework helps voters find parties that align with their beliefs and preferences. One important tool that voters may use to decide spatially is the Manifesto Project, which analyzes political parties’ election manifestos to study their policy preferences and estimate their positions. This paper investigates the feasibility of replacing human intervention in estimating party positions with intelligent tools, such as LLMs. By examining the party positions estimated by five different LLMs (ChatGPT 3.5 Turbo, Cohere Command, Gemini 1, Llama 2, and Llama 3), our work provides a renewed look at party positions in three major democracies with two-party and multi-party systems (Germany, the United Kingdom, and the United States) and discusses the feasibility of employing AI tools to assist human experts. Results show that, on average, LLMs perceive leftist and rightist manifestos as 73.64% less extreme than they truly are. Given that CMP-trained human experts are likely performing their tasks accurately, this significant bias implies that replacing human experts with LLMs for the calculation of party positions may not be feasible in the near future.