Chapter 6: Measuring Rater Effectiveness: New Uses of Value-Added Modeling in Competency-Based Education

Raters are effective to the extent that they are unbiased yet helpful. This chapter demonstrates a novel way for institutions to monitor the levels of bias and helpfulness among raters without requiring multiple ratings of task submissions. The ability to work with single ratings is crucial in settings where the volume and velocity of task submissions is high. While bias and helpfulness require separate models and different outcome variables they can be measured using the same basic approach: a variation of value-added modeling where the “teacher effect” is translated into the “rater effect.” Understood within this framework a rater’s bias is the extent to which her pass rate tends to be higher or lower than expected (given other factors), and a rater’s helpfulness is the extent to which his students’ growth rates tend to be faster or slower than expected (given other factors). The result of each model is a measure of effectiveness that competency-based institutions can use to evaluate their raters. My goal in this chapter is to provide an overview of the methods and outputs of this approach at a conceptual rather than technical level. To this end, the chapter uses examples from fictional raters and students at Western Governors University (WGU). After the conceptual models are established, analyses of over 800 actual raters at WGU (whose actual Evaluation Department rates thousands of task submissions every day) across 2 months demonstrates: (a) the rater bias model, trained on 1 month of data, correctly predicts about three-fourths (AUC = 0.749) of the following month’s testing outcomes; (b) the rater helpfulness model, also trained on 1 month of data, explains about two-thirds of the variance (R-sqd = 0.64) in the following month’s score increases; and (c) bias and helpfulness measures are uncorrelated among the raters which suggests that they indicate distinct abilities.