Table 1 Chosen assessment strategies...

Table 1

Chosen assessment strategies present in gameplay research literature: their advantages and disadvantages in the context of communicative process evaluation

Advantages	Disadvantages	References
Post-game questionnaire
-Cost-effective -Time effective -Scalable -Enables data collection standardization -Allows for statistical analysis -The most common evaluation method to assess simulation games	-Self-reporting: participants may not accurately recall or interpret their communicative behaviors -Potential lack of objectivity: psychological and social biases may take place, like researcher-pleasing answers -Lack of insight depth: the questionnaire does not capture the nuanced dynamics of communication as it happens in real time	Chin and Gamson (2009). Assessment in Simulation and Gaming: A Review of the Last 40 Years. Simulation & Gaming 40(4). 553–568
		Gorsic, Clapp, Darzi, and Novak (2019). Brief Measure of Interpersonal Interaction for 2-Player Serious Games: Questionnaire Validation. JMIR Serious Games 2019; 7(3)
		Faizan, Löffler, Heininger, Utesch, and Krcmar (2019). Classification of Evaluation Methods for the Effective Assessment of Simulation Games: Results from a Literature Review. International Journal of Engineering Pedagogy, 9(1)
Direct observation
-Real-time data collection -Contextual insights can allow understanding of the context -When conducted in-filed provides a high level of ecological validity	-Potential observer bias: observers' presence may influence the teams behavior -Potential subjectivity of observer when not enough methodological discipline is implemented and/or the observer is not trained/experienced	Ulmer et al. (2021).Communication Patterns During Routine Patient Care in a Pediatric Intensive Care Unit: The Behavioral Impact of In Situ Simulation. Journal of Patient Safety
		Wideman et al. (2007). Unpacking the potential of educational gaming: A new tool for gaming research. Simulation & Gaming - Simulat Gaming, 38, 10–30
Video and/or audio recording analysis
-In-depth analysis allowing for deep insight into communication patterns -Rich data on team dynamics -Video + audio recording allows for disciplined observation -Repeated review of recordings when needed	-Possible discomfort of being recorded in research subjects -Time-costly -Resource-consuming, possible data overload	Sharritt, Aune, and Suthers (2011). Gamer Talk: Becoming Impenetrably Efficient. Business, Technological, and Social Dimensions of Computer Games: Multidisciplinary Developments. 252–270
		Kuznekoff and Rose (2012). Communication in multiplayer gaming: Examining player responses to gender cues. New Media & Society, 15(4), 541–556
Focus groups
-Interactive -Rich data on team dynamics	-Logistics -Subjectivity -Possible group dynamics bias	Tidbury, Jarvis, and Bridge (2019). Initial evaluation of a virtual reality bomb-defusing simulator for development of undergraduate healthcare student communication and teamwork skills. BMJ Simulation & Technology Enhanced Learning, 6, 229–231
-Interactive -Rich data on team dynamics	-Logistics -Subjectivity -Possible group dynamics bias	Verkuyl et al. (2017). Virtual Gaming Simulation in Nursing Education: A Focus Group Study. Journal of Nursing Education, 56(5), 274–280
Interviews
-Rich data with potential to deepen the insight while conducting an interview -Flexibility – allows for clarification and deepening of chosen areas of interest	-Resource-intensive -Possible interviewer bias: interviewer behavior can influence subjects' responses	Mettler and Pinto (2015). Serious Games as a Means for Scientific Knowledge Transfer—A Case From Engineering Management Education,” In: IEEE Transactions on Engineering Management, 62(2), pp. 256–265, May 2015
		Wilson et al. (2016). Serious Games: An Evaluation Framework and Case Study. System Sciences (HICSS), 49th Hawaii International Conference, IEEE. 638–647
Simulation game artifacts analysis
-Objective approach to communication data when analyzing artifacts -Facilitates pattern recognition	-Possible context loss (and without the context, incomplete or misleading analysis) -Ethical concerns, e.g. when analyzing communication artifacts like private chat messages, etc.	Palomo-Duarte et al. (2016). Assessing foreign language learning through mobile game-based learning environments. International Journal of Human Capital and Information Technology Professionals, 70.964–981
		Berns, Palomo-Duarte, Dodero, and Valero-Franco (2013). Using a 3D Online Game to Assess Students' Foreign Language Acquisition and Communicative Competence. In: D. Hernández-Leo, T. Ley, R. Klamma, A. Harrer (eds), Scaling up Learning for Sustained Impact. EC-TEL 2013. Lecture Notes in Computer Science, vol 8095. Springer, Berlin, Heidelberg
Automated analysis (AI)
-Efficiency -Scalability -Data-driven pattern identification (that might not be apparent to a human observer	-Complexity – a potential barrier as it requires specialized knowledge to implement -“Blindness” of the algorithm to subtleties in communication that a human observer would detect (i.e., sarcasm detection)	Thompson, Leung, Blair, and Taboada (2017). Sentiment analysis of player chat messaging in the video game StarCraft 2: Extending a lexicon-based model. Knowledge-Based Systems, 137, 149–162 Madge, Chamberlain, Fort, Kruschwitz, and Lukin (2024), May. Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024. ELRA & ICCL
Practice-led approach
-Targeting isolated communicative behaviors by designing game scenarios exclusively to test and develop communication skills -Real-time engagement in communicative tasks and feedback -Customizable and Adaptive Scenarios -Iterative Learning Cycles	-Resource-intensive design and execution of the game -Complex (and often subjective) measurement of the outcomes -Possible participant bias: as the game is designed solely for communication research purposes, it may cause altering behavior in participants	Buidze, Sommer, Zhao, Fu, and Gläscher (2025). Expectation violations signal goals in novel human communication. Nature Communications, 16(1), Article 1989
		Zadilska, Zaveriushchenko, Horlachova, Zhukevych, and Tsymbal (2024). The role of simulation games in preparing students for communicative foreign language teaching. Revista EDaPECI

Advantages	Disadvantages	References
Post-game questionnaire
-Cost-effective -Time effective -Scalable -Enables data collection standardization -Allows for statistical analysis -The most common evaluation method to assess simulation games	-Self-reporting: participants may not accurately recall or interpret their communicative behaviors -Potential lack of objectivity: psychological and social biases may take place, like researcher-pleasing answers -Lack of insight depth: the questionnaire does not capture the nuanced dynamics of communication as it happens in real time	Chin and Gamson (2009). Assessment in Simulation and Gaming: A Review of the Last 40 Years. Simulation & Gaming 40(4). 553–568
		Gorsic, Clapp, Darzi, and Novak (2019). Brief Measure of Interpersonal Interaction for 2-Player Serious Games: Questionnaire Validation. JMIR Serious Games 2019; 7(3)
		Faizan, Löffler, Heininger, Utesch, and Krcmar (2019). Classification of Evaluation Methods for the Effective Assessment of Simulation Games: Results from a Literature Review. International Journal of Engineering Pedagogy, 9(1)
Direct observation
-Real-time data collection -Contextual insights can allow understanding of the context -When conducted in-filed provides a high level of ecological validity	-Potential observer bias: observers' presence may influence the teams behavior -Potential subjectivity of observer when not enough methodological discipline is implemented and/or the observer is not trained/experienced	Ulmer et al. (2021).Communication Patterns During Routine Patient Care in a Pediatric Intensive Care Unit: The Behavioral Impact of In Situ Simulation. Journal of Patient Safety
		Wideman et al. (2007). Unpacking the potential of educational gaming: A new tool for gaming research. Simulation & Gaming - Simulat Gaming, 38, 10–30
Video and/or audio recording analysis
-In-depth analysis allowing for deep insight into communication patterns -Rich data on team dynamics -Video + audio recording allows for disciplined observation -Repeated review of recordings when needed	-Possible discomfort of being recorded in research subjects -Time-costly -Resource-consuming, possible data overload	Sharritt, Aune, and Suthers (2011). Gamer Talk: Becoming Impenetrably Efficient. Business, Technological, and Social Dimensions of Computer Games: Multidisciplinary Developments. 252–270
		Kuznekoff and Rose (2012). Communication in multiplayer gaming: Examining player responses to gender cues. New Media & Society, 15(4), 541–556
Focus groups
-Interactive -Rich data on team dynamics	-Logistics -Subjectivity -Possible group dynamics bias	Tidbury, Jarvis, and Bridge (2019). Initial evaluation of a virtual reality bomb-defusing simulator for development of undergraduate healthcare student communication and teamwork skills. BMJ Simulation & Technology Enhanced Learning, 6, 229–231
-Interactive -Rich data on team dynamics	-Logistics -Subjectivity -Possible group dynamics bias	Verkuyl et al. (2017). Virtual Gaming Simulation in Nursing Education: A Focus Group Study. Journal of Nursing Education, 56(5), 274–280
Interviews
-Rich data with potential to deepen the insight while conducting an interview -Flexibility – allows for clarification and deepening of chosen areas of interest	-Resource-intensive -Possible interviewer bias: interviewer behavior can influence subjects' responses	Mettler and Pinto (2015). Serious Games as a Means for Scientific Knowledge Transfer—A Case From Engineering Management Education,” In: IEEE Transactions on Engineering Management, 62(2), pp. 256–265, May 2015
		Wilson et al. (2016). Serious Games: An Evaluation Framework and Case Study. System Sciences (HICSS), 49th Hawaii International Conference, IEEE. 638–647
Simulation game artifacts analysis
-Objective approach to communication data when analyzing artifacts -Facilitates pattern recognition	-Possible context loss (and without the context, incomplete or misleading analysis) -Ethical concerns, e.g. when analyzing communication artifacts like private chat messages, etc.	Palomo-Duarte et al. (2016). Assessing foreign language learning through mobile game-based learning environments. International Journal of Human Capital and Information Technology Professionals, 70.964–981
		Berns, Palomo-Duarte, Dodero, and Valero-Franco (2013). Using a 3D Online Game to Assess Students' Foreign Language Acquisition and Communicative Competence. In: D. Hernández-Leo, T. Ley, R. Klamma, A. Harrer (eds), Scaling up Learning for Sustained Impact. EC-TEL 2013. Lecture Notes in Computer Science, vol 8095. Springer, Berlin, Heidelberg
Automated analysis (AI)
-Efficiency -Scalability -Data-driven pattern identification (that might not be apparent to a human observer	-Complexity – a potential barrier as it requires specialized knowledge to implement -“Blindness” of the algorithm to subtleties in communication that a human observer would detect (i.e., sarcasm detection)	Thompson, Leung, Blair, and Taboada (2017). Sentiment analysis of player chat messaging in the video game StarCraft 2: Extending a lexicon-based model. Knowledge-Based Systems, 137, 149–162Madge, Chamberlain, Fort, Kruschwitz, and Lukin (2024), May. Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024. ELRA & ICCL
Practice-led approach
-Targeting isolated communicative behaviors by designing game scenarios exclusively to test and develop communication skills -Real-time engagement in communicative tasks and feedback -Customizable and Adaptive Scenarios -Iterative Learning Cycles	-Resource-intensive design and execution of the game -Complex (and often subjective) measurement of the outcomes -Possible participant bias: as the game is designed solely for communication research purposes, it may cause altering behavior in participants	Buidze, Sommer, Zhao, Fu, and Gläscher (2025). Expectation violations signal goals in novel human communication. Nature Communications, 16(1), Article 1989
		Zadilska, Zaveriushchenko, Horlachova, Zhukevych, and Tsymbal (2024). The role of simulation games in preparing students for communicative foreign language teaching. Revista EDaPECI

Source(s): Own elaboration

[ViewLarge]

Sharing Unavailable