Divergence or Dysfunction? Using LLMs to Measure Disagreement on the U.S. Supreme Court

For institutional decision-making bodies like the U.S.Supreme Court, interpersonal dynamics are central to the decision-making process, and the ability of even minimally empowered leaders is crucial in managing those dynamics. Prior research has extensively explored these questions, but measurement difficulties have limited scholarly consensus. Here, we propose a novel methodological approach to these questions, leveraging large language models (LLMs) to score the sentiment of judicial opinions, with a specific focus on differentiating normal jurisprudential disagreements from more personal/targeted criticisms within the Court’s opinions. Employing LLMs, we are able to differentiate between general and targeted criticism, then employ targeted criticism as a variant of aspect-level sentiment analysis over all majority, concurring, and dissenting opinions issued by the Court between 1954 and 2010. In doing so, we develop a measure of the degree of consensual and discordant content for each opinion, justice, and term of the Court. Our approach addresses the limitations of prior methods in distinguishing between what are generally regarded as healthy and unhealthy disagreements, and examining data at the opinion level provides further leverage on uncovering leadership effects. Our results provide new evidence of the chief justice’s role in building and maintaining consensus on the Court and suggest a historically unique modern Court increasingly willing to publicly express disagreement, with implications for understanding institutional debates more generally.