Chapter 14: Common Measurement Issues in a Multilevel Framework

In this chapter, we provide a general introduction to issues in measurement and psychometrics that are particularly salient within a multilevel modeling context. It is important to understand potential problems with ignoring nested data, particularly in the examination of validity evidence. Specifically, in this chapter, we examine the impact of multilevel data structure on common psychometric questions concerning internal structure, internal consistency, and fairness. These ideas are situated in a conversation about what to consider in developing a measure intended to be used in a multilevel framework, as well as examining a measure that is used in such a framework but was not originally designed with multilevel data in mind. Our primary focus is on multilevel factor analysis, and multilevel differential item functioning (MDIF). We also touch on multilevel reliability estimates. Throughout the chapter, we refer to an example instrument that is used in a higher education context. It was originally designed for individual assessment, but we discuss how it is used at an aggregate level and whether a multilevel construct could be defensible from a multilevel theoretical perspective.