Selection bias in evaluating treatment effects: Some formal illustrations

Regression analyses of compensatory educational programs have been criticized on the grounds that the pupils were not randomly selected. Specifically, it has been argued that a spurious deleterious effect of the treatment will be observed when the selection procedure systematically puts lower-ability students into the treatment group and higher-ability students into the control group.

We evaluate this argument via a simple test score model: pretest score and posttest score are fallible measures of underlying true ability and the true treatment effect is zero. Posttest is regressed on pretest and a treatment dummy. The spurious effect arises when selection of subjects for treatment is explicit on the basis of true ability, but not when it is explicit on the basis of pretest score.