Chapter 2: Value-Added to What?: The Paradox of Multidimensionality

The motivation for the present study comes from a dimensionality paradox that arises when teachers and schools are evaluated on the basis of the implied value they add to multiple measures of student achievement. To unearth the paradox, consider the following scenario. The results from a state’s large-scale assessments of mathematics and reading are modeled separately using item response theory (IRT) and reported on distinct score scales. A fundamental assumption of the IRT modeling of such tests is that a student’s item responses are independent, conditional on the presence of a continuous unidimensional latent variable. To check this assumption, a test contractor will typically perform some form of exploratory factor analysis in which the magnitudes of the eigenvalues associated with the first two extracted factors is compared to the magnitude of the eigenvalues associated with the second and third extracted factors. When this ratio is above some threshold (i.e., > 3), one treats this as support for the assumption of unidimensionality. Naturally, invoking this rationale simplifies the scaling and reporting of test scores immensely. For example, even if a test of mathematics was written to ask questions that would appear to come from conceptually distinct domains (algebra vs. geometry), or invoke conceptually distinct thinking processes (recall of procedures vs. multi-step reasoning), a single number is sufficient to adequately summarize each student’s performance. Now, an economist interested in estimating teacher value-added stumbles onto the scene and, seeing distinct scores being reported for students in math and reading, decides to average the scores together as an estimate of a student’s combined academic achievement. This simplifies matters even more, because now, instead of needing to compute two estimates of a teacher’s value-added, the economist need only compute one.