Commentary: Next Steps for Research on SACD Programs: Embracing Complexity

The research design used in the SACD field trial reflects both an independent analysis on a core set of measures (Haegerich & Metz, this volume; Kaminski, Battistich, & David-Ferdon, this volume) as well as local evaluations related to each programs’ core goals (Flay, Berkowitz, & Bier, this volume). It seems unlikely, however, that the primary research questions for the multiprogram evaluation (i.e., the effect of the SACD programs on school climate and student behavior, across school and student characteristics) can be addressed adequately by summarizing data across studies given that assignment of program to schools is confounded by the different methods and goals across programs.

Similarly, the within-study analyses, which are tailored to the core features of each program, can illustrate the strengths of each program but will be difficult to interpret given the heterogeneity across programs. Invoking a standard metric across programs, such as effect size analyses, is also unlikely to yield interpretable results given the different foci of the programs (Snyder & Lawson, 1993). At best, we may get a “horse race” of which program best met their self-determined goals, which is difficult to evaluate and subject to possible experimenter bias (Eisner, 2009).

Furthermore, it may be challenging to determine the nuanced effect of these seven SACD programs on students’ social competence and learning not only because of the heterogeneity of program methods, but because of the imprecision related to large-scale field trials. For example, a recent study of a program to enhance social competence for urban low-income third graders (Elias & Haynes, 2008) nicely illustrated the nuanced evaluation needed to disentangle the effects of teacher and peer support on children’s learning. Teacher support appeared more strongly related to learning early in the year for highly competent children, but later in the year for less competent students, whereas peer social support demonstrated effects only late in the school year, and only for the most socially competent students.

Establishing the unique contributions of SACD programs over and above the effects of standard academic programs is an important goal for policy and practice given the opportunity costs associated with selecting programs. Opportunity costs refer to the degree to which the benefits derived from one program are offset by the lack of time spent on a second program. This is especially relevant in the current era of using test scores and other indicators of student achievement as defining features of effective schools, particularly in low-income communities where resources to support learning are strained (Cappella et al., 2008).

To consider fully opportunity costs in implementing SACD programs, it is important to note that academic programs actually have been shown to have strong effects on SACD skills. For example, classwide peer tutoring, reciprocal peer tutoring, and task modifications (e.g., increasing structure or enhancing task salience) have been shown to improve children’s socialization with adults (DuPaul & Eckert, 1998; DuPaul, Ervin, Hook, & McGoey, 1998) and peers (Heller & Fantuzzo, 1993). In a seminal study, academic tutoring had larger positive effects on peer social preference scores than did a social skills intervention (Coie & Krehbiel, 1984). Moreover, in an evaluation of a school counseling program, counseling was no more related to student gains on teacher and peer relations when compared to academic tutoring without counseling (Catron, Harris, & Weiss, 1998).

Nevertheless, one area of great promise for SACD programs is evidence that programs with components that support effective teaching, such as classroom management (e.g., Academic and Behavioral Competencies Program, Competence Support Program in this volume) and positive teacher-student relations (e.g., Rimm-Kaufman, Fan, Chiu, & You, 2007), are also likely to enhance children’s academic success, especially among low-SES students (Connors & Epstein, 1995). Parent involvement activities, such as school-home notes (ABC Program; Kelley, 1990), improving rapport at parent-teacher conferences (Epstein Munk, Bursuck, Polloway, & Jayanthi, 1999), and involvement in school-based activities (e.g., attending school events and PTA meetings), and home-based activities (e.g., reading, homework support), are also associated with improved learning (Connors & Epstein, 1995; Stone & McKay, 2000). Thus, to the extent that an SACD program is focused on these proximal indicators of student learning, there would be an expected effect on students’ academic achievement, particularly in schools with high percentages of low achieving students (Brophy & Good, 1986).

Therefore, in addition to the analyses proposed by the SACD consortium (i.e., estimating program impact on primary student and school outcomes and subgroup analyses by student risk factors), we suggest one additional analytic approach that includes clustering programs based on those that included at least one proximal indicator of classroom performance (i.e., consultation and/or coaching for teachers around classroom management, altering the classroom and school environment to support academic learning) versus those programs focused exclusively on enhancing students’ SACD skill development (i.e., enhancing prosocial behavior and reducing antisocial behavior). When we clustered the SACD programs in that manner, three programs (i.e., Academic and Behavioral Competencies Program, Competence Support Program, and Reading Writing Respect and Resolution) included at least one proximal indicator of classroom performance plus student SACD enhancement, while the remaining four (i.e., Love in a Big World, Positive Action, PATHS, and Second Step) did not include proximal indicators of academic learning.

Pooling the SACD programs by proximal indicators of classroom performance provides incremental utility over and above within-study analyses of distal child and school outcomes (i.e., school climate, social/emotional competence) for a number of reasons. First, the former provides a mechanism for advancing SACD practice by shedding light on those programs whose goals are most clearly integrated and aligned with the ongoing work of schools and teachers. Establishing this empirical link enhances the face validity of SACD programs and may ultimately enhance implementation as there is some evidence to suggest that interventions perceived as more acceptable are implemented with greater fidelity (Brown et al., 1997). Second, given the slow progression of knowledge regarding how interventions look and function in schools and the time it takes for schools to adopt comprehensive programs and derive any benefits, it may be necessary to attend to the proximal indicators of students’ learning, such as teacher and parent predictors of learning, to develop an ecological context for students’ school success (Cappella et al., 2008). Third, focusing on the proximal indicators of learning advances knowledge by examining the theoretical mechanisms of change inherent in the SACD program models; specifically, the degree to which a focus on proximal indicators of academic achievement and children’s social and emotional skill development each provide incremental benefit to children’s school success.

The studies included in the current issue are designed to evaluate how the SACD programs “worked” relative to standard educational practice. However, the inherent complexities of conducting research in schools necessitates moving beyond the big questions and getting out of the black box that largely characterizes intervention efforts in education. At the culmination of this extensive field trial, if the answer is “Yes, all seven programs work,” school administrators will be faced with the task of identifying which program is best suited to their students and families. Thus, achieving even the optimal outcome for this field trial of SACD programs, the results may fall short of improving policy and practice. We suggest that a more compelling question to consider is how and why these programs worked and what can be learned from identified barriers and facilitators to program implementation.

Given the enormous number of variables that are free to vary in effectiveness trials, we fully appreciate the obstacles that the SACD consortium encountered around implementation. For example, by the end of year three, only two of the seven schools implementing the Positive Action program were rated as high implementers (i.e., meeting between 60–70% of benchmarks). Of the five remaining schools, four were moderate implementers (i.e., meeting 50–60% of the benchmarks) and one school was designated as a low implementer (i.e., less than 50% across all benchmarks). In the Academic and Behavioral Competencies (ABC) program, there was notable variability in teacher implementation of different program components, ranging from 98% implementation of a rule tracking system to 25% implementation of a daily social skills program. In addition, in the ABC program, as Bickman et al. (this volume) note, high implementation of content components did not equate with an implementation process that was high-quality, suggesting that use of specific components does not assure high-quality implementation.

These implementation challenges suggest that consumers of interventions (e.g., teachers), by incorporating program components into their practice, are also reinterpreting SACD programs for their unique context. In the latest revision of Rogers’ (2003) classic diffusion of innovation model, consumers or adopters of interventions are viewed as active users and interpreters of programs, and that such adaptations and reinventions are done to better meet their needs. This suggests that fidelity and adaptation are best understood as interactive processes, with program developers and users each contributing to the development of an effective and sustainable program (Atkins et al., 2003).

The difference between implementation fidelity achieved in efficacy studies versus effectiveness studies is often explained by contextual factors, which are typically treated as “noise” or “nuisance” and are greatly reduced in efficacy trials (Hohmann & Shear, 2002). However, in effectiveness trials, ignoring the noise that cannot be controlled may result in erroneous conclusions about treatment fidelity, outcomes, and consumer buy-in. Bickman et al. (this volume) note: (The fact that) “programs could not be implemented as intended also suggests that there is a mismatch between the program characteristics and the context within which it is being implemented” (p. 100). The authors conclude that the “measurement of implementation to be idiographic to the structure and content of the specific program evaluated” (Bickman et al., this volume, p. 100); to which we add that program implementation may need to be ideographic to the local context as well.

Given the reality that adopters of innovations also adapt them (either by omission or commission), rather than interpreting these adaptations as threats to program integrity, an alternate interpretation is that adaptations happen more often than not and rather than ignoring those adaptations they should be studied at all stages of a research project (Chafouleas & Riley-Tillman, 2005; Dusenbury, Brannigan, Falco, & Hansen, 2003; Lau, 2006). Thus, we suggest that in future data analyses, the SACD consortium use findings of low implementation to more fully understand how schools and teachers used the different SACD programs and what differential fidelity to implementation across components of a program teaches us about the need for modifications of program components. To fully realize the potential for further modifying SACD programs to enhance their effectiveness and sustainability will require flexible research designs and hybrid research models (e.g., Atkins, Frazier, & Cappella, 2006).

Hybrid or mixed-method research models can take many forms, such as combining qualitative methods with between-group analyses (Cook & Reichardt, 1979), iterative and formative research designs (National Research Council and Institute of Medicine, 2002), or utilizing flexible research designs that adjust to the exigencies of the setting of interest (e.g., Connor-Smith & Weisz, 2003). In each case, the goal of the research is to utilize over-lapping, if not contrasting, research designs to model more effectively the vagaries of real-world practice (Atkins, Frazier, Birman et al., 2006). This type of research is ideal for school-based studies given the unique characteristics of schools and the often-urgent need for change.

Hoagwood, Burns, and Weisz (2002) propose a model for bridging research and practice, the clinic/community intervention development model (CID), which can provide direction to the SACD consortium as they move forward. The CID model involves eight steps in research development, starting with manual development in practice settings, followed by highly controlled efficacy trials, and single case applications in practice settings to adapt the program manual to real-world circumstances. These steps are followed by a series of large-scale effectiveness trials in practice settings to determine goodness-of-fit, culminating in dissemination research once the value of the program has been established.

For the programs in this series, we suggest attending to the recursive use of single case examples to inform program development highlighted in the CID model. The studies in this series appear well-situated for the application of the CID model given that they have been manualized and studied using rigorous research methods (e.g., random assignment to conditions, adequate control group). Because this series represents the largest effectiveness trial of SACD programs to date and given the varying goals and methods employed, it would appear that a hybrid use of single-case and between-group analyses would be most helpful in elucidating program effects across individual and setting contexts. Morgan and Morgan (2001) noted the relevance of single-subject designs to advance practice research in managed care settings. We suggest that their arguments are equally prescient for school-based research where decisions are made for students and teachers in classrooms and for which large group analyses often translate poorly (Horner et al., 2005).

For example, examining individual program outcomes by classroom, by teacher, or by student helps elucidate, with greater precision, trajectories of change for individuals and the temporal link between what transpired within that unique context and changes in outcomes (Goldfried & Wolf, 1996). Assessing when and for whom program outcomes were less effective also helps illustrate specific factors that may have impeded program implementation (e.g., dosage, intensity, adaptation). We suggest that subsequent studies of SACD programs would be enriched by single-case designs that balance individualized information specific to setting or person with experimental rigor (see Barlow, Nock, & Hersen, 2009; Horner et al., 2005). Such designs help illustrate factors associated with teachers’ effective use of practices or a child’s mastery of an SACD skill, and can highlight the component training modules associated with effective program implementation. Most importantly, for the consumers of SACD programs, these local illustrations help elucidate the extent to which programs can work in their context and under conditions specific to their daily activities.

Schools have long valued the social and emotional development of students; and the work of the SACD consortium can represent an important advance in the field’s understanding of the core features of SACD programs that can impact positive outcomes for students. We believe that the SACD consortium is in a unique position to influence the public policy implications for their work by asking additional questions to establish the extent to which SACD programs are aligned with the core mission of schools and are sensitive to the unique characteristics of schools, classrooms, and students. We suggest that much can be learned from adaptation, from embracing the complexity of real world practice, and from the idiographic experiences of students and teachers, that will inform the development of effective and sustainable SACD programs.