Online labor pools continue to grow in popularity, serving as an inexpensive, readily available source of research data. Despite early skepticism, accounting research has generally found evidence that supports the use of these labor pools. However, one important distinction unique to online labor markets is the pre-screening process that qualifies participants for future studies. As the identity of online participants are generally unknown, researchers rely on participants’ self-reported identities to establish a pool of qualified respondents. This paper aims to provide evidence of the reliability of online participants’ answers to pre-screening questions.
Following the current literature’s recommendations on pre-screening candidates, I employ a multi-stage design using two similar surveys that are taken by each participant. I compare participants' answers on the first survey and the second survey to provide evidence on the consistency of their responses.
My results indicate that online participants are responding with substantial inconsistency to survey questions related to their social identity at a rate that may not be tolerable for many researchers. This has implications for researchers who use these online labor markets to represent a particular population of interest.
This study is the first to provide evidence on the consistency of online labor market participant responses. Additionally, it is the first to test the efficacy of current recommended guidelines for identifying populations of interest in the literature.
