Implementation of A School-Wide Adolescent Character Education and Prevention Program: Evaluating the Relationships Between Principal Support, Faculty Implementation, and Student Outcomes

Components of the Capturing Kids’ Hearts Campus by Design model have been widely disseminated throughout the United States. By the end of 2009, these programs had been implemented in more than 5,500 schools in 47 states. The model is based upon a multifactorial causal model of youth risk behaviors and is closely aligned with Social Learning Theory (Bandura, 1986).

Character education programs encourage schools to create learning environments that foster the development of ethical, responsible students who demonstrate caring concern for others (What Works Clearinghouse, 2007). Capturing Kids’ Hearts has been used in conjunction with other educational programs to meet the goals of federal initiatives such as Smaller Learning Communities (Flippen Group, 2009; U.S. Dept. Ed., 2009) in order to promote a healthy school climate. A positive school culture is dependent upon a number of parameters that include developing a safe, caring environment, establishing rules that hold all school members to a high standard of behavior, and building healthy relationships that create a professional culture and increase student connectedness (Lickona & Davidson, 2005; Character Education Partnership, 2010).

Resnick et al. (1997) reported that this connectedness to a parent or school-based role model (teacher) is a protective factor that protects youth against every measure of health risk behavior except history of pregnancy. A subsequent study (Resnick Ireland, & Borowsky, 2004) reported that the percentage of youth involved in violence is reduced in the presence of protective factors, despite the continued presence of significant risk factors. In addition, the Office of Juvenile Justice and Delinquency Prevention (2004) states that programs that have the ability to increase students’ attachment to their school are often highlighted as models for violence prevention. Results from the National Longitudinal Study on Adolescent Health demonstrated that student connectedness reduced negative behaviors in four domains of adolescent health and morbidity: emotional health, violence, substance use, and sexuality. Thus, the Capturing Kids’ Hearts Campus by Design model is considered to be both a character education and violence prevention intervention because it is designed to strengthen students’ connectedness to school through enhancing protective factors (i.e., strong bonds with teachers; clear rules of conduct that are consistently enforced) and targeting modifiable risk factors (i.e., inappropriate behavior; poor social coping skills).

Capturing Kids’ Hearts Campus by Design is a skill intensive, systemic process designed to develop high-performing school cultures, align organizational and individual behaviors to outcomes, and increase school connectedness. As a result of implementing this integrated approach, (1) faculty members build intentional cultures that emphasize connectedness with students and with each other, (2) negative behaviors are minimized while learning is maximized, and (3) students acquire communication and conflict resolution skills that help them succeed both in school and after graduation.

Evidence-based longitudinal research studies and comprehensive literature reviews have demonstrated that connectedness to an appropriate role model (teacher) and effective classroom management exert a positive impact on students’ behavior (Resnick et al., 1997) and academic achievement (Wang, Haertel, & Walberg, 1993). In a meta-analysis of more than 100 studies, Marzano (2003) reported that the keystone for all aspects of effective classroom management was the quality of teacher-student relationships.

The goal of Capturing Kids’ Hearts Campus by Design is to develop leaders who have the knowledge and skill-sets to develop healthy relationships with students, colleagues, and members of the community. Through these relationships, adults model the character traits that students must acquire in order to reach their full potential. Capturing Kids’ Hearts not only identifies a process designed to build high-quality relationships, but also provides an opportunity for participants to learn and practice skills that exert a positive impact on classroom management.

The theoretical framework underlying the Capturing Kids’ Hearts Campus by Design learning approach is the Social Cognitive Theory developed by Bandura (1986). A central premise of Social Cognitive Theory is that behavior is dynamic, dependent upon both personal constructs and environmental factors that influence each other simultaneously (Perry, Baranowski, & Parcel, 1991). Social Cognitive Theory asserts that the likelihood that a behavior will be repeated is a function of the individual’s expectancy that the behavior will lead to a particular outcome and of the extent to which that outcome is valued (Norman, 1991).

In addition, a person must have self-efficacy in order for the behavioral change to occur: individuals must first have the conviction that they can execute the behavior successfully before the desired behavior is acquired. This conviction is acquired through observing others and implementing self-regulatory mechanisms (Bandura, 1986). Self-regulation involves self-assessing behavioral performance, keeping track of actions, comparing actions to a standard of behavior, and self-directing acquired behaviors. An outstanding summary of the assumptions, statements, and concepts of Social Cognitive Theory that help explain how people develop and maintain behaviors was developed by Glanz, Rimer, and Lewis (2002). The 11 factors that affect behavioral change, the intervention strategies that promote positive changes in behavior by addressing these factors, and the program inputs in the Capturing Kids’ Hearts process designed to implement the intervention strategies are provided in Table 1. For instance, “observational learning” (factor 7) is a factor that impacts behavior by helping the learner acquire new behavior by observing the behavior of someone else. The intervention strategy includes providing credible role models (teachers and student leaders) of the targeted behavior. The modules “Leadership Skills,” “EXCEL Model,” and “Values/Building Healthy Relationships” specifically address how to implement this intervention strategy. Implementation measures (surveys and observational tools) are then used to determine whether or not the program components within these modules were delivered with fidelity (i.e., adherence).

Previously reported experimental (Castro Johnson, & Smith, 2008; Danaher, 2006) and quasi-experimental (Cirillo-Teverbaugh, 1994; Cirillo-Teverbaugh & Colwell, 1993; Sherwood, 2003) studies, as well as numerous longitudinal case studies (The Flippen Group, 2006), have highlighted the positive effects that components of this comprehensive process exert on behavioral and academic outcomes. The magnitude of the effect is impacted by a number of variables, including administrative support for the process and level of program implementation by school faculty members.

Comprehensive interventions usually are not implemented fully in the first year, and even in subsequent years, not all schools implement all of the core components of interventions as prescribed by the developer (Berends, Bodilly, & Kirby, 2002; Glennan, 1998; St. Pierre & Kaltreider, 2004) due to conflicts or issues that inevitably arise when attempting to disseminate a school-wide program. As a result of this variability in implementation, Zhang, Shkolnik, and Fashola, (2005) report that student outcomes such as academic achievement may lag years behind initial program implementation. In addition, the relationship between faculty implementation and student outcomes is often weak, and becomes weaker when implementation stagnates or declines over time.

Interventions that impact an entire school community must be implemented fully and with fidelity in order for schools to obtain maximum effect. A number of studies (Domitrovich & Greenberg, 2000; Durlack & Wells, 1998; Gresham, Gansle, Noell, & Cohen, 1993) report that few programs provide data on levels of implementation or consider whether implementation quality is related to intervention outcomes. Hallfors and Godette (2002) found that while most school districts selected research-based programs, many programs were implemented with low fidelity. Other studies (Kam, Greenberg, & Walls, 2003, and references therein) demonstrate that implementation quality impacts student outcomes; thus, even an empirically-validated, school-based program does not produce consistent student outcomes if it is not implemented with fidelity.

Greenberg, Domitrovich, Graczyk, and Zins (2005) identify a number of reasons for including implementation assessment when evaluating interventions. For instance, lack of implementation data may result in researchers incorrectly determining that a program is ineffective when, in fact, the program was implemented so poorly that no analyses can provide meaningful conclusions. Additionally, program evaluators may be faced with considerable unexplained variance in program effects. Obtaining implementation information can help researchers with both types of research difficulties by providing data that allows them to draw valid conclusions and identify reasons for variability in program outcomes.

In this study, we report the first-year results obtained after implementation of a school-based character education program, Capturing Kids’ Hearts Campus by Design. The primary questions of the study include: (1) How well do principals in intervention schools support the program as measured by personally modeling the skills taught in the process and providing a systemic mechanism by which program progress can be discussed? (2) How well do teachers in intervention schools implement Capturing Kids’ Hearts skills in the classroom? (3) What is the relationship between principal support and teacher implementation? (4) How well do students acquire the skills that are modeled by teachers? (5) What is the relationship between teacher implementation of the process and student acquisition of skills? (6) What specific changes occur in student behaviors (i.e., changes in the number of discipline referrals or increases in specific prosocial behaviors) as a result of program implementation? The relationship between fidelity of implementation and changes in student outcomes is discussed.

School leadership teams from the Oneida-Herkimer-Madison Board of Cooperative Educational Services (BOCES) in New York participated in a state-wide meeting that led to the directive for districts to improve student outcomes. In response to this directive, the leadership team developed a BOCES-wide vision specifically addressing student behaviors and skills that would prepare students to be successful in the workplace beyond high school graduation. The findings from the National Longitudinal Study on Adolescent Health (Resnick et al., 1997) indicated that the leading factor supporting positive outcomes and preventing youth participation in negative behaviors is connectedness to an adult role model.

Oneida-Herkimer-Madison BOCES implemented a strategic plan to increase student connectedness. A Professional Development Council was convened and involved a committee of 20 superintendents, assistant superintendents, and other district leadership members tasked with examining program options. The team met for a number of months, reviewing regional needs assessments, interviewing principals and staff, and engaging in extensive committee discussions.

All members of the Professional Development Council were asked to consider professional development programs that would help connect administrators, teachers, and students. Eight research-based programs (as evidenced by inclusion of the program on a “model program” list or by publication of research articles in peer-reviewed journals) were identified that would address relational connectedness. The Capturing Kids’ Hearts process, a comprehensive, school-based program designed to improve prosocial competencies and reduce negative behaviors in students, was the program that was chosen. The committee then developed a test group of individuals to participate in the training and provide feedback.

The test group, consisting of 52 teachers and administrators, attended a Capturing Kids’ Hearts 3-day workshop. A BOCES-wide meeting was then convened to discuss outcomes of the training, determine the district’s options, and develop next steps. A unanimous vote was provided by the test group indicating that the program would be of great benefit for the students and would help the schools meet state-directed, district-wide goals.

Before committing a significant amount of funding to implement the training district-wide, the BOCES leadership team decided to invest time and resources in a pilot study that would be capable of measuring the effectiveness of the program in local high schools. Principals within the region that were willing to participate in the study were asked to submit school archival data from the preceding three years describing demographic, behavioral, and academic trends. Due to limited funding, BOCES was able to include only six schools in the research study. Statistical modeling (Holtzapple, 2009; Spybrook Raudenbush, Liu, & Congdon, 2006) indicated that for a six-school study using randomization at the school level, researchers could encounter significant difficulties in measuring outcomes. Therefore, a partner district was considered in order to increase the number of schools in the study, thus improving the likelihood that changes in measurable outcomes could be detected if they occur.

The Riverside County Office of Education (RCOE) in California also sought to document the effectiveness of this program in local schools. The leadership had previously developed a strategic initiative to increase school connectedness within the county and had implemented the Capturing Kids’ Hearts process for a number of years, but had not participated in a randomized, controlled research trial. The RCOE leadership team asked principals interested in participating in a study to submit the same data as that submitted by the BOCES schools.

In order to meet federal and state guidelines for using evidence-driven research, an experimental (randomized, controlled), blocked, research design was used, with schools chosen as the unit of randomization. Randomization at the school level was necessary because there would be a significant “spillover” effect between intervention and control participants within a school that would tend to underestimate the apparent effectiveness of the intervention. Eighteen public high schools served by BOCES and RCOE provided academic and behavioral data from the 3 years preceding the study (SY2005, 2006, and 2007). After determining the level of variability in the information derived from the 3-year discipline referral data (Holtzapple, 2011), eight schools exhibiting the lowest variability (< 25%) in the 3-year data were paired according to school type (grades 7-12 or grades 9-12) and then randomly assigned to intervention or control groups. Schools with greater levels of variability in the data were not eligible for participation in the study.

The study sample included 8,350 students in Grades 7-12 and 469 teachers from 6 high schools located in districts served by the Oneida-Herkimer-Madison Board of Cooperative Educational Services (BOCES) in New York and 2 high schools served by the Riverside County Office of Education (RCOE). In the intervention group, 4% of the students were African American, 33% were Hispanic, and 57% were Caucasian. In the control group, 5% of the students were African American, 26% were Hispanic, and 65% were Caucasian. Overall, 4% of the students were African American, 28% were Hispanic, and 62% were Caucasian, with 30% of the students eligible for free or reduced lunch.

Schools assigned to the treatment condition received all of the programs and processes in the school-wide model, Capturing Kids’ Hearts Campus by Design, including Capturing Kids’ Hearts, Process Champions, Executive Coaching, and Phone Consulting. These trainings address the mechanisms and processes of social skills instruction (SSI) that lead to improved student behavior by promoting skills acquisition (i.e., modeling, coaching, and behavioral rehearsal), enhancing skills performance, removing competing behavior, and facilitating maintenance of social skills (Gresham & O’Shaughnessy, 2002).

Capturing Kids’ Hearts is a 3-day teacher and administrator training program that provides school faculty with the skills they need to model and teach relational skills, communicative competencies, problem-solving skills, citizenship, and consequential thinking. This program provides the foundation on which teachers build effective classroom management strategies including engaging students at the beginning of class, developing classroom expectations for behavior, providing effective feedback, and identifying and addressing conflict.

Process Champions is a 2-day teacher and administrator training program that reinforces the Capturing Kids’ Hearts training and develops teachers and administrators who act as on-site process mentors, helping their colleagues apply and master all aspects of the training. Process Champions mentors receive special additional instruction that prepares them to assist and support their peers in implementing the process with fidelity.

Campus Consulting consists of two, 2-day sessions that provide the opportunity for campus administrators and Process Champions (teacher mentors) to work one-on-one with a Project Consultant from the Flippen Group who helps improve implementation of the process. One session is offered in each school term (fall and spring).

Phone Consulting consists of six, 1-hour phone calls that provide support for campus administrators and Process Champions as they implement the Campus by Design process. During the consultations, the Project Consultant may assist with setting implementation expectations, dealing with conflict, establishing an effective communication system, or addressing other issues that require attention. The consultation phone calls are conducted on a monthly basis during the school year.

Survey development included intensive discussions to identify research objectives and design survey questions that address specific outcomes. An eight-elementary school, field pretest of an initial survey measured responses to behavioral and psychometric survey items. Analysis of the survey responses and evaluation of individual teacher comments indicated that the items measuring specific behaviors would be most effective in measuring changes that result from program implementation, and that psychometric survey items derived from previously validated surveys were insufficiently specific given the number of schools used and the variability in how items can be interpreted by respondents. For example, in response to “Adults at this school really care about every student,” teachers responded that they can only report how they personally feel, and that they didn’t know which aspects of “care” are involved in the statement.

Although it is preferable to use questions from previously validated surveys so that study results can be compared with those reported in the literature, using poor measures of the program being evaluated, regardless of the historical value of the analytical tool, will not provide an effective means of measuring outcomes. As noted by Fowler (1995), there is a large body of research in the social and medical sciences that repeatedly incorporates poor outcome measures because of their historical significance; however, he points out that science is best served when researchers use survey questions that have been systematically evaluated using procedures that ensure that the questions are effective. In addition, Fowler comments that although multi-item surveys can often improve measurement, 20- to 30- item scales can be effective tools if the items are chosen carefully.

Therefore, although the Capturing Kids’ Hearts survey initially contained approximately 100 items in the field pretest, in order to minimize respondent burden, the final survey contained 19 items. When developing survey tools evaluating factual events (observable behaviors), key principles involved in good survey design include (1) asking questions that respondents can answer, (2) clearly defining key terms so that respondents are all answering the same question, and (3) providing adequate context so that respondents will see the value of answering the questions accurately (Fowler, 1995). In order to follow these key principles, the final survey and observational form only included items measuring specific behavior.

Table 2 demonstrates how key outcomes taught in the Capturing Kids’ Hearts process correspond to items described in the taxonomy of variable outcomes developed by Berkowitz and Bier (2005). The table further links these outcomes to items within an established psychometric survey (WestEd, 2008), and provides examples of how psychometric survey items were converted to measures of observable behaviors in the Capturing Kids’ Hearts Survey or Observational Form. For instance, to measure the leadership skill of engagement (first item in Table 2) the more general terms used in the California Healthy Kids Survey (“supportive” and “inviting”) was converted to a specific behavior that can be observed (“the teacher is at the door greeting students”). Other behavioral measures of prosocial skills are provided in Table 2.

Both the Capturing Kids’ Hearts survey and observational form contain similar questions, but are specific to the person filling out the tool (i.e. “The teacher [is at the door]” is an example of the beginning of a sentence used in the observational form whereas “I [am at the door]” is an example of the corresponding item used in the self-report surveys). A review of these items by former school administrators and teachers, current teachers, and program trainers with knowledge of the subject matter established the content validity (Litwin, 1995) of the survey and observational form.

The proximal outcomes of the training (teacher and principal implementation) were assessed using the Capturing Kids’ Hearts Observational Form at the classroom level through observation of 40 randomly selected classrooms (using a web-based number randomizer; 5 teachers per school), and at the individual-level (all teachers) using the Capturing Kids’ Hearts Survey. Each wave of classroom-level observational data was aggregated to represent a school-level construct (Tseng & Seidman, 2007). A data triangulation approach was used in that each wave of individual-level (survey) data was also aggregated and used to determine whether or not teachers’ perspectives of how well they model Capturing Kids’ Hearts behaviors correspond with the implementation ratings obtained from observing teachers in the classroom. School implementation ratings were generated by trained members of the program developer’s research staff, each of whom had taught the process and had been part of the evaluative process for more than 5 years. Implementation analyses were based upon two, on-site observational sessions for each school during the school year. The first wave of observational data (midyear data point) was obtained approximately 3 to 5 months after teachers attended Capturing Kids’ Hearts training, and the second wave was obtained at the end of the school year. The evaluation measured proximal outcomes in three areas: (1) principal support for the process (i.e., (s)he personally modeled Capturing Kids’ Hearts skills and established procedures for following up with on-site mentors), (2) teacher implementation of the Capturing Kids’ Hearts relational skills, and (3) teacher implementation of specific classroom management techniques. All three areas were rated based upon a 3-point Likert-type scale with 0 indicating no implementation, 1 indicating partial implementation, and 2 indicating full implementation. Fractional points were not used in the evaluation of individual teachers. An aggregated implementation score (reported as a percentage of full implementation) was calculated for each school.

Observational measures of students’ acquisition of prosocial skills (distal outcomes) specific to the intervention were obtained during the two on-site days for each school in which teacher and principal implementation measures were acquired. Selected items from the California Healthy Kids School Climate Survey (2008) were matched to specific behaviors/skills taught in the Campus by Design model, and these behaviors/skills were matched to the taxonomy of outcomes described by Berkowitz and Bier (2005). These outcomes were divided into subscales that measured modeling of specific skills demonstrating personal morality (sense of justice/fairness, respect), prosocial behaviors (caring concern for others, teamwork, helping others, sharing), communicative competency (communication skills, attentive listening), citizenship (democratic values), and problem-solving skills (consequential thinking, behavioral adjustment, conflict resolution). An aggregated prosocial skills outcome score was also calculated, giving equal weight to each subscale.

Schools were provided a prestudy data form requesting demographic, academic, and behavioral data for the 3 school years preceding initiation of the study. Principals from each of the schools provided the data sheet and submitted it to their respective leadership teams. For Capturing Kids’ Hearts Campus by Design (CKH CBD), a process that impacts school climate through strengthening teacher-student connectedness, the BOCES and RCOE leadership teams were primarily interested in measuring differences in behavioral outcomes (i.e. reduction of risk factors such as discipline referrals and acquisition of protective factors such as prosocial relational skills).

Using discipline referral data as a measurable outcome of program implementation can be problematic because the application of referral rules and policies can vary greatly between teachers within a school and between schools within a district. Additionally, new policies may be implemented in one or more of the intervention or control schools involved in a research study such that the number of referrals reported at the end of the study is not indicative of what actually occurred in the school. For instance, a principal in a control school could implement stricter discipline rules during the year such that it would seem that referrals increased relative to the previous years’ data, but in fact, the increase would only reflect changes in policies, not true increases in disruptive student behaviors. Thus, although discipline referral records continue to be used to evaluate behavioral changes within schools (Irvin, Tobin, Sprague, Sugai, & Vincent, 2004), the integrity of the data may be compromised when consistency in the discipline referral system is not maintained (Wright & Dusek, 1998).

Sprague, Sugai, Horner, and Walker, (1999) attempted to overcome potential variability in schools’ discipline referral systems by selecting schools that had an established discipline referral system and that were in the initial stages of a plan to improve school-wide discipline procedures. These two strategies were utilized in the present study. In addition, a “stratification of schools” strategy based upon the variability in the discipline referral data in the 3 years preceding the study was implemented so that changes in referrals could be detected if they occurred.

Stratification of schools to meet statistical demands of research trials (Holtzapple, 2011) is particularly critical to the success of studies involving fewer than 10 schools. In the present study, funding was available for only 8 schools; therefore, given the statistical difficulties involved in measuring outcomes when using fewer than 10 schools, the variability in the number of discipline referrals in the 3 years preceding the study was determined for each school that applied to be a study participant. The variability in the data ([standard deviation/3-yr mean number of discipline referrals] expressed as a percentage) ranged from 1% to 42% (see  Appendix). Because previous studies had demonstrated that the Capturing Kids’ Hearts process can reduce discipline referrals in schools 25% to 75%, and because school leadership teams wanted to be able to detect changes in this outcome, the 8 schools whose school-level variability in discipline referral data was less than 25% were chosen for the study.

Statistical power analyses (Holtzapple, 2009) were conducted using the Optimal Design software (initially version 1.76 and then the updated version 2.0) for cluster randomized trials (Spybrook et al., 2006; Spybrook, Raudenbush, Liu, & Congdon, 2009). The significance level (a) was held constant at 0.05 for all calculations. Univariate ANOVA analyses were conducted using SPSS version 16.0 software.

School-level prestudy, midyear, and end of year implementation outcomes are highlighted in Figure 1. The prestudy survey data provides the baseline level of Capturing Kids’ Hearts Campus by Design (CKH-CBD) prosocial behaviors self-reported by school principals and faculty for each of the eight schools. The presurveys were used as a surrogate for prestudy direct observation of behaviors because scheduling conflicts prevented completion of classroom observations in the intervention schools prior to the training. The pre- and postsurvey (end of year) means (23.25 ± 5.68 and 25.25 ± 5.44, respectively; p [two-tailed] = 0.6222) from the control schools provides a test-retest reliability measure of the survey instrument. The presurvey and mid-year observation means (23.25 ± 5.68 and 18.75 ± 5.5, respectively; p [two-tailed] = 0.2803) in the control schools demonstrates that the results from prestudy self-report surveys are a reliable substitute for the prestudy observational instrument. A sampling of 5 randomly selected teachers from each school provides good information about faculty follow-through and the pervasiveness of implementation. Therefore, the presurvey results served as the initial baseline for all schools, whereas midyear, and end-of-year 1 data sets were obtained using direct observation of behaviors and skills.

Initially, intervention and control schools demonstrated similar baseline levels of specific behaviors and skills associated with the Capturing Kids’ Hearts training. By the end-of-year 1 evaluation, control schools exhibited levels of CKH-CBD skills that were similar to baseline levels; however, the intervention schools exhibited increased, but varying levels of CKH-CBD skills, with implementation ranging from 32% to 72% as determined by observation of CKH-CBD skills exhibited by teachers and principals. In order to determine which factors affect program implementation and, ultimately, student outcomes, we investigated: (1) the level of principal support of the intervention, (2) the relationship between principal support and teacher implementation of the intervention in the classroom, and (3) the relationship between teacher implementation and student acquisition of specific prosocial skills.

Figure 2 demonstrates the end-of-year 1 level of principal support for the intervention for all 8 schools as measured by direct observation of principal-initiated skills related to Campus By Design. Note that even in the control schools (represented by the bar graphs located in the small box in Figure 2), the baseline level of principal support/principal leadership is not zero. Posttest evaluation of principal support/principal leadership was based upon evaluator-observed use of specific behaviors and skills by the principals. Thus, the levels of Campus By Design skills observed in principals from control schools would be skills that may be expected in individuals as a result of natural skills, college training, or postcollegiate principal development trainings whose objectives overlap those of Capturing Kids’ Hearts Campus By Design. Levels of these skills in principals from control schools ranged from 10% to 33%.

Initially, the principals of the intervention schools demonstrated a similar level of program-specific skills; however, by the end of the school year, the principals from 3 of the intervention schools had implemented and were modeling for their teaching faculty between 72% and 87% of the skill set taught by the intervention. In contrast, the principal from the fourth intervention school (I4) implemented less than 30% of the skills.

Given the importance of principal leadership when implementing interventions, we investigated how principal support for the intervention is related to teacher implementation of CKH-CBD skills in the classroom. As can be observed in Figure 3, there is a direct, linear relationship between the level of support provided by the principal and the degree to which the process is implemented by teachers. In the intervention schools, when principal support is low (school I4), there is a correspondingly low level of teacher implementation. In schools in which principal modeling of the intervention skills is high (schools I1, I5, and I7), there is a correspondingly high level of teacher implementation of the intervention’s skills in the classroom.

The variability in the level of implementation of the process in the classroom in intervention schools suggests that there will be similar variability in student outcomes. Figure 4 demonstrates that this is indeed the case. In intervention school I4, low implementation of the CKH-CBD skills by teachers resulted in a low level of student acquisition of specific prosocial skills.

The information in the preceding figures demonstrate that one of the intervention schools implemented less than 35% of the skills taught in the Campus By Design model and may therefore be considered a nonadhering participant. A “withdrawal for nonadherence” approach was considered for this intervention school exhibiting < 60% implementation of the intervention because doing so would allow the trial to provide a comparison of student outcomes resulting from optimal intervention versus control condition. Including the results from the nonadhering school would “contaminate” the intervention group and reduce the observed outcomes of the trial, thus making the intervention appear to be less effective. In clinical trials, it has been proposed that if the rules for withdrawing participants are identified prior to study initiation, then it is legitimate to remove nonadhering participants (Sackett & Gent, 1979). However, Friedman, Furberg, and DeMets (1998) argue that an “intention-to-treat” approach should be used in research studies because adherence, or lack thereof, is itself a response to the intervention. They state that the potential for bias cannot be avoided when nonadhering participants are dropped from the trial. In addition, Chaudhry and McDermott (2008) point out that nonadherence is not only a problem in research studies, but is also a major barrier to obtaining desirable outcomes in nontrial scenarios. Low adherence to the intervention in trials of educational interventions may indicate that widespread use of the program may not be feasible unless developers address the reasons for non-compliance (i.e., participant knowledge of what is required is lacking or follow-up strategies are ineffective). For the following evaluation of student outcomes, an “intention-to-treat” approach was used; therefore, the outcomes of all eight schools were included in the analyses.

Student prosocial outcomes were classified according to the taxonomy of outcomes presented by Berkowitz and Bier (2005) in a research guide for educators. Direct observation of behaviors was used to detect changes in personal morality (PM; respect, sense of justice and fairness), prosocial behaviors (Pr-S; caring concern for others, teamwork, helping others, sharing), communicative competencies (CC; communication skills, attentive listening), citizenship (CZ; democratic values), and problem solving (PS; consequential thinking, behavioral adjustment, conflict resolution).

As can be seen in Figure 5, prosocial outcomes increased from 24% (pretest) to 57% (posttest) in the intervention schools whereas they decreased from 23% (prestudy) to 12% (poststudy) in the control schools. The p - values for these outcomes were 0.8254 at pretest (indicating no significant difference between the control and intervention groups at pretest) and 0.0070 at posttest (indicating statistically significant differences between the two groups). These prosocial outcomes were divided into subcategories, and the scores for these subcategory outcomes are also provided in Figure 5. (P -values: PM = 0.0167; PrS = 0.0276; CC = 0.0103; CZ = 0.0185; PS = 0.0472).

All schools were asked to provide the number of discipline referrals for the 3 years preceding the study as well as for the 2008-2009 school year. Discipline referral data was analyzed by univariate ANOVA. The effect size, reported as the standardized mean difference (Hedge’s g; df = 7) was computed to be -2.1 (reduction in discipline referrals) with p = 0.0189. This outcome represents an average 11% increase in discipline referrals in the 4 control schools; in contrast, a 22% decrease in referrals occurred in the 4 intervention schools.

The Centers for Disease Control and Prevention (2009) provides strategies that will help create an environment that facilitates student health and academic achievement. The strategies addressed by the Capturing Kids’ Hearts—Campus by Design model include (1) providing students with social skills necessary to be actively engaged in school, (2) using effective classroom management methods to foster a positive learning environment, (3) providing professional development for teachers to enable them to meet the emotional needs of adolescents, and (4) creating trusting and caring relationships that promote open communication. Implementing the EXCEL Leadership Model through the Capturing Kids’ Hearts Campus by Design process results in a culture in which students and staff model leadership qualities, show respect for others, share and celebrate each others’ successes, demonstrate caring concern for others, communicate effectively, listen attentively to others, have a sense of justice and fairness, help others, resolve conflicts, and work together as a team.

The limitations of the study are directly linked to the statistical constraints placed on school districts when attempting to conduct a randomized, controlled trial using schools as the unit of randomization. In order to minimize costs, (1) a limited number of schools were used, and as a result of the associated statistical constraints, (2) principals had to be willing to participate, (3) only schools with stable baseline data were eligible for the study, and (4) schools were recruited from only two states, New York and California. The results do not necessarily demonstrate outcomes that would occur in schools in which principals are disinclined to participate or for which the baseline data is unstable. Next steps in research include measuring outcomes in elementary and middle school students, and working with a different unit of analysis (using both intervention and control participants within the same school) despite the possibility of a spillover effect between participants.

When implemented with fidelity, the Capturing Kids’ Hearts Campus by Design model increases prosocial behaviors (protective factors) in students and decreases the number of discipline referrals (risk factors) in intervention schools. Continued support for the process will help sustain the behaviors that enhance the learning environment and equip students for success in school and after graduation.

The author is grateful for invaluable research discussions with Drs. Thomas Lickona, Matt Davidson, and Vlad Khmelkov (SUNY-Cortland); Vic Battistich (late) and Marvin Berkowitz (U. Missouri-St. Louis); Jessaca Spybrook (U. Western Michigan); and Victor Willson (Texas A&M University).