Commentary: A Commentary for the Special Issue on the SACD Research Program

The New Jersey Center for Character Education (NJCCE) was established in 2002 through a grant proposal drafted for the New Jersey Department of Education under the Partnerships in Character Education (PCE) provisions of the No Child Left Behind Act. During the first 4 years of our work we focused on the evaluation of evidenced-based character education programs in 10 diverse school districts in New Jersey.¹ An Expert Evaluation Panel of five national and state experts external to NJCCE was established to serve as a resource for ideas and direction at critical scientific choice points, and to provide an additional source of objectivity, as NJCCE staff was involved in facilitating relationships between the program providers and the school sites in addition to conducting the evaluation.

With guidance from the panel, the project chose a single self-report instrument to measure changes in student behaviors and attitudes over the 3 years of program implementation across school sites. The Institute for Education Sciences (IES) had put extensive effort into the construction of measures that could be used across multiple sites with a variety of interventions to test the impacts of social and character development. IES and its contractor for the SACD project cross-site evaluation, Mathematica Policy Research (MPR), combined measures from a large number of separate research studies on a wide range of domains into one children’s survey, the SACD Child Report. It was only after we were committed to this choice of instruments that we discovered that the SACD Child Report had not been field tested, nor had there been a separate factor analysis for the newly constructed measure. While the project was successful in its program implementation goals and execution of the evaluation plan, as investigators new to the rigorous requirements of IES we learned a considerable amount. Two issues stand out that are germane to the topics of this special issue:

1. The “blunt instrument” issue. The project offered a view and numerous insights into one of the most important challenges in social and character development intervention and outcome research: The lack of broadly appropriate and feasible assessment tools. The evaluation effort also revealed flaws in the assumption that student surveys would enable us to capture the most important changes that we observed occurring in some of the participating schools over the three years of the project. It is what Mary Utne O’Brien, the Expert Evaluation Panel chairperson, characterized as the “blunt instrument” issue in the following remarks submitted to the U.S. Department of Education as part of the project’s final report:

   Specifically, even when program staff witnessed, for example, consistently high levels of positive school climate over three years and positive teachers interview data confirming changes in student behavior for which they credited program involvement, the instrument did not appear to be sensitive to these changes or to the comparisons made by the teachers. Perhaps additional analyses, such as more extensive factor analysis to assure appropriate scale construction and use of the information contained in these responses would have revealed more promising and face-valid descriptions from the data.²

2. Need for technical assistance support to grantees. It takes enormous effort to adequately conduct evaluation research at the level expected by IES, when local school districts bear the primary responsibility in consort with chosen program evaluators. Sufficiently defined expectations, planning time and initial and sustained technical assistance resources are required if grant programs structured like PCE are to yield results commensurate with the level of effort required to initiate and sustain multiple-year research efforts to implement evidence-based character education programming. As the project neared its end, the Character Education and Civic Engagement Technical Assistance Center (CETAC) was beginning to provide meaningful but very modest technical assistance resources in evaluation to grantee projects, but our view is that CETAC was insufficiently resourced to accomplish what was required by the structure of the grant program.

Each of the papers in this special issue address a different aspect of the effort to examine research program design and implementation challenges in the attempt to better understand the use of universal prevention programs in elementary level school settings, where most programs are targeted for maximum developmental impact. The following reflections are not intended to be a critique of the SACD program or the individual papers, but to offer comments that might serve to highlight some important aspects of this work from the consultative perspective which lies intermediate between the researcher and the school site. The first paper in the series (Haegerich et al., this issue) lays out the history and intent of the SACD program quite clearly. Of special interest to the practitioner and policy-maker alike is the fact that IES and CDC have joined together in a mutual understanding of the importance of doing evaluation research of this scale and complexity, seeking to provide new knowledge that will have real implications for future lines of inquiry and research programs, and with potential for guidance to educators seeking to make hard choices about use of scarce prevention resources. This is a heartening sign, as fragmentation of program efforts is such a common and significant problem in making sense of what educators should do from the federal government, straight through the state and community decision making levels.

In summarizing the seven evidence-based programs chosen to be studied, the second paper has constructed examples of logic-theoretical models of expected effects that could be useful teaching tools for how to understand program intent and outcomes (Flay et al., this volume). Over the past roughly 10 years the logic model framework has become a marker for those who want to be taken seriously in their level of sophistication in the planning of program design and choice. Logic models do not tell the whole story, but they are a significant enough advance in linking research-grounded assumptions, identified needs and program objectives that more should be made of this as a common tool and expectation of practitioners as well as university-based consultants. As noted in the paper, programs with a more constructivist model and those that view the child as meaning-maker and rely on relationship building between children and adults as the heart of the prosocial change process are underrepresented in the SACD program group. This is unfortunate, as programs such as the Responsive Classroom and Community of Caring are not only commonly used, but tend to be less curricular based, and therefore more easily adopted in the eyes of many educators. They are deserving of the same level of evaluation and treatment as those in the SACD program, and one would hope that a similar research effort could study these more climate-change oriented programs.

There was much of interest in the paper on the development and validation of outcome measures for the SACD program (Kaminiski et al., this volume). First, the evidence and acknowledgment that “measures of child behavior were affected by both the behavior constructs being assessed and by the person reporting on that behavior” (e.g., parents, vs. teachers, vs. self-reports). This is social science common sense to experienced educators and program developers on one hand, but deserving of much greater attention by serious researchers on the other. The situations in which much survey data are collected are so rife with nuanced social influences that interpretation of results, both positive and negative, is often a very tenuous affair. Practitioners drawn into evaluation research need to be educated regarding these hazards, but also need to be more involved in documenting research conditions and historical context which can easily overwhelm both the interventions being studied as well as produce spurious results. I remember, for example, talking to a principal who was puzzling over data we had collected which seemed to indicate an uptick in student reported perceptions of aggression and bullying behaviors, when she thought an effective program was in place. Then she looked at the date the data had been collected, checked her calendar and realized that by chance the established date for data collection was a day after a special assembly program on bullying prevention had taken place.

The Kaminksi paper suggests that the outcome measures identified, such as children’s problem behaviors, altruism, pro-social behavior, impulsivity, and school connectedness, may be of interest to school administrators and other program providers. This is both a tantalizing and aggravating suggestion. It points to the very great divide that still separates basic research findings and professional practice. While our means of disseminating and sharing such information have grown exponentially in the last decade with the multiple tools available through the internet and world wide web, most educators do not have the professional training, context, time, or institutional structures to know how to practically grapple with the implications of these kinds of findings. While research-based programs are designed to fill this void, unless programs become embedded in professional practice and school culture, they rarely survive.

The Bickman et al. article (this volume) is an articulate review of the understudied and critically important issues involved in implementation hazards in evaluation research of SACD school programs. It goes beyond describing the fidelity problems inherent in implementing programs as they were designed, and begins to identify and describe the wealth of measurement issues involved in school-based research. In saying that implementers should be accountable for the quality of implementation and noting the resistance that teachers may have to being honest in their self-reports of their own implementation (which has been confirmed by comparing objective observations with self-reports) another layer of responsibility needs to be identified. Implementation is a district and school level responsibility as well as a responsibility of the classroom teacher, and the program coordinators and administrators involved set the expectations, level of priority and tone associated with time and effort devoted to implementation. I am reminded of our panic during the second year of implementation of an evidence-based program in a large urban school when we learned in a planning meeting that most of the fourth grade teachers and some of the fifth grade teachers who had been trained the previous year had been assigned to other schools, and we were about to spend considerable resources trying to measure implementation in classrooms where the teachers had not even been oriented to the program, let alone received adequate training. The district level program coordinators, who tended to be overwhelmed with substance abuse and violence crises on a regular basis, had not even thought to bring this to our attention until we met, close to the beginning of implementation.

The Bickman article also highlights the fact that there are no standard procedures or a clear framework for accounting for the multiple threats to the validity of program outcomes when dealing with the complexities of confounding variables that appear at every turn in doing school research on SACD programs. My view is that more time should be spent looking squarely at these issues and the implications for the best research strategies to gain broad knowledge about the place of SACD in student growth and school change, rather than assuming that the proffered bio-medical gold standard of randomized controlled trials is what we should be striving for.

Likewise, the Massetti et al. article (this volume) notes contamination issues such as diffusion of treatment effects in relation to using classrooms or students in study designs, which represents just the tip of the methodological quagmire in testing the effectiveness of social competence and climate improvement programs. The set of papers in this special issue are a significant step in unwrapping the complexities of assessing SACD programs. The field has certainly come a long way in the last twenty years in defining relevant constructs rooted in sound developmental and social change research and in developing programs grounded in theory. However, schools are not static institutions where even the semblance of controlled conditions hold over the period of time needed to do multiyear research studies.

The on-the-ground evidence that well-implemented SACD programs can result in significant changes in student behaviors in comportment, engagement and academic outcomes is very convincing. Witnessing first hand the efforts that many schools have made to implement SACD programs, efforts that administrators, teachers and stakeholders such as parents and school board members all agree have made an important difference in their schools and student’s lives, should be treated as more than just anecdotal evidence. My sobering and enlightening conclusion from observing these changes at close range over the past 30 years is that no evidence-based program sufficiently accounts for these changes. Programs such as the seven studied in the SACD research program can serve to provide needed skill structures and professional development to teachers, and ignite the conditions leading to broader change. But sustainable change requires significant alterations of policies and practices at the school level over a period of 3 to 5 years. Furthermore, research studies rarely describe the wonderfully complex conditions that result in meaningful change, precisely because the traditional methods used by necessity focus on what can be controlled.

It should be the responsibility of those in public policy positions to assist the SACD field, both researchers and practitioners, to come up with innovative solutions to the problem of how to better document and unwrap what we know seems to work. What is called for, I believe, is more mixed-methods research, where advances in qualitative approaches based on a social-ecological model of change are combined with measures of individual change.

Schools function at the level of complexity of a living organism; they are not laboratories for learning. I tend to trust the face validity of a principal’s or teacher’s account of positive changes they have seen in themselves, their colleagues and students in accounting for measurable changes in a school’s culture and climate over a period of years resulting from SACD-related efforts more that I do the results of discrete survey measures. However, this simply points to the need for better evidence of the effectiveness of these approaches to improve learning and developmental outcomes for students than we have available now. This is the challenge that the SACD program took on. It may be that our theories of change are not yet up to the task of describing in measurable terms how the web of human relationships devoted to standard educational outcomes in the institution we call schools can assist in optimizing the development of the whole child. As the articles in this special issue show clearly, SACD research challenges abound. For the good of our civilization they are very worth pursuing, because if the moral and social development of our children are not as worthy of attention as our drive for academic success in the service of economic productivity, we will foster the kind of cultural impoverishment and ethical misconduct that undermines our ability to thrive as a nation.