Chapter 9: Cross-Classified Random Effects Models for Assessing Rater Severity and Differential Rater Functioning

This study demonstrates how a conventional multilevel model, a cross-classified multiple membership random effects model (CCMMrem) and a cross-classified random effects model (CCrem) can be used as multilevel measurement models (MMMs) to estimate rater effects. The three versions of these MMMs are presented along with extensions permitting impact assessment, differential item functioning, and differential rater functioning. The formal equivalence among some parameters in the models is presented. Distinctions are also made to highlight scenarios when more complex parameterizations might be needed and to delineate the types of differential rater functioning that each of the three models can assess. Using a simulated dataset, the equivalence of item difficulty, person ability, and rater severity estimation is empirically demonstrated for the CCrem and CCMMrem versions of the rater effects MMM. The chapter concludes with a discussion of limitations and directions for future research.