3: Reliability

In Chapter 1 we briefly reviewed the basic ideas underlying classical test theory (CTT). The fundamental equation underlying CTT is

where X is the observed score on a test or measure for an individual, T is the individual’s true score (i.e., true ability) on the construct of interest, and E is random error. This random error encompasses all factors that might influence the observed score, other than an examinee’s true ability. In Chapter 1 we also provided examples of what might cause error, including examinee fatigue and mood, as well as environmental variation such as the temperature and noise in the room where the test is administered, and finally factors specific to the test such as the items selected for inclusion. The random nature of error leads to a number of truisms about the relationships among X, T, and E, concluding in the definition of reliability as the ratio of true score to observed score variance, or