Prompt Engineering Meets “Definition of the Situation” and Identity Theory: Using ChatGPT to Study Big Social Media Datasets From a Qualitative Symbolic Interactionist Perspective

In this research note, we evaluate the performance and potential of generative pre-trained transformers (GPTs) in the collection and qualitative analysis of naturalistic social media data from the platform Reddit. We focus on, first, how generative artificial intelligences (GAIs) interpret or make meaning out of data and how well they communicate analytical insights when acting one-on-one with a human researcher, and with other GPT agents in a semi-autonomous context. Second, we review how instructions regarding definitions of the situation and role identities shape their analytical actions. The authors use a small sample set of 15 Reddit posts to conduct a comparative analysis of two GPT platforms: ChatGPT-4o, the latest personal subscription version available at the time of writing, and Microsoft AutoGen, a powerful API version capable of organizing multiple GPT agents to complete research tasks semi-autonomously. We describe and evaluate the structure and content of each model’s output, including how the models engage with both prompts and raw data thematically, empirically, and conceptually. We highlight observations concerning multiple response options, the organization and labeling of key themes, selected extracts from the data, and displays of humanlike agency. Our preliminary discussion is meant to provoke theoretical work on the similarities and differences between how humans and GAIs such as ChatGPT interpret and communicate meaning.