Chapter 7: Using Auxiliary Teacher Data to Improve Value-Added: An Application of Small Area Estimation to Middle School Mathematics Teachers

In the literature on value-added (VA) modeling, much of the modeling effort has been devoted to reducing the bias of VA estimates to provide valid estimates of causal effects (cf., Harris, Sass, & Semykina, 2010; Lockwood and McCaffrey, 2012; Lockwood, McCaffrey, Mariano, & Setodji, 2007; Mariano, McCaffrey, & Lockwood, 2010, Rothstein, 2010). Less research has been dedicated to increasing the precision of VA estimates. However, the lack of precision in VA estimates can greatly limit their utility in education evaluation systems. Figure 7.1 plots VA estimates and their variances as a function of the number of students contributing to the estimates, for middle school mathematics teachers from a large urban school district. In the figure, the extreme sampling variability of the estimates among teachers with small numbers of students is clear. Such imprecise VA estimates can lead to great difficulties in decision making. For instance, education policy promulgated by the federal government through Race to the Top suggests awarding tenure or pay on the basis of performance measures that rely heavily on VA estimates (Federal Register, 2009). Such decisions may inappropriately favor or penalize teachers with small classes simply due to the inflated errors in their VA estimates. For example, the right frame of Figure 7.1 shows that, if no correction is made for the error in the estimates, a decision rule to award teachers with VA estimates greater than.5 will recognize almost exclusively teachers having fewer than 15 students. On the other hand, if the bonus/penalty decision is based on the z-score (estimate divided by standard error) or a similar statistic, teachers with small classes will have considerable disadvantages due to their large estimation errors. This suggests the need to improve the precision of VA estimates, particularly for teachers teaching relatively small numbers of students.