Chapter 5: Kappa and Alpha and PI, Oh My: Beyond Traditional Inter-rater Reliability Using Gwet’s AC₁ Statistic

The aggregation of research results is an important and useful mechanism for better understanding a variety of phenomena, including those relevant to human resource management (HRM). For decades, aggregate studies in the form of meta-analyses have provided an overarching examination and synthesis of results. Such studies provide conclusions based on the amalgamation of previous works, enabling scholars to determine the consistency of findings and the magnitude of an effect, as well as affording more precise recommendations than may be offered based on the results of a solitary study (Aguinis, Dalton, Bosco, Pierce, & Dalton, 2011; Borenstein, Hedges, Higgins, & Rothstein, 2011). Similarly, content analyses are another mechanism by which to identify trends through examining the text, themes, or concepts present across existing studies. These aggregate studies can have real and significant implications for better understanding HRM issues (e.g., Allen, Hancock, Vardaman, & McKee, 2014; Barrick & Mount, 1991; Eby, Casper, Lockwood, Bordeaux, & Brinley, 2005; Hancock, Allen, Bosco, McDaniel, & Pierce, 2013; Pindek, Kessler, & Spector, 2017). However, in order to deduce meaningful conclusions, the data must be reliable and demonstrate construct validity.