This study aims to investigate the perception of automated versus traditionally produced news content among general readers and professional journalists. It specifically examines the ability of these groups to distinguish between machine-generated and human-written articles and evaluates their assessments of content quality across various dimensions.
The study utilized automated content created with certain parameters by large language models, specifically OpenAI’s ChatGPT, and compared it to content produced by traditional means via a questionnaire. Two different groups were examined: university students, representing general readers and professional journalists. Non-parametric statistical methods were applied to analyze the results, focusing on the ability of participants to differentiate between the two types of content and their evaluations of various quality dimensions.
Results indicate that neither journalists nor university students could reliably identify the origin of an article with any statistically significant accuracy. There were, however, distinct differences regarding how each group perceived the individual characteristics of each text, suggesting a divergence between the evaluation criteria for each of the two groups.
This study contributes to the growing body of research on the integration of AI in journalism by providing empirical insights into how both readers and professional journalists perceive automated content. It recreates a familiar experimental structure but modernizes the way automated journalistic content is produced for the purposes of the experiment, with the utilization of large language models, as opposed to more antiquated algorithmic solutions.
