The Importance of Multidimensional Baseline Measurements to Assessment of Integrated Character Education Models

The mere fact that we continue to discuss and research the development of the construct called character some 2,400 years after Aristotle (1984) introduced his heuristic insights on the excellence of character, goes to show that instilling a level of greater character in mankind is no easy task; yet it is perceived as a worthwhile endeavor that many generations have and continue to embrace. As a result of this ongoing pursuit to increase the moral excellence of our youth, character education continues to persevere and search for evidence supportive of a more definitive role within the modern American education system. Thus, one might define character education today as a continuation of the means of teaching and instilling character within the citizens of a culture. Yet given that many of the current efforts and lessons often shared by caring educators work toward such goals, it might be easier in some cases to state which educational practices are not part of a character education process rather than those which are part of character education. Regardless of one’s definition or position on the role of character education, as Damon (2005) points out, “It is an odd mark of our time that the first question people ask about character education is whether public schools should be doing it at all. The question is odd because it invites us to imagine that schooling … somehow could be arranged to play no role in the formation of a child’s character” (p. 2).

In reality, when one considers character education as part of the natural developmental process that education has embraced for centuries as a foundational goal or responsibility, we all practice character education to some degree by default. Those of us who teach are role models, and our actions (good or bad) as educators are scrutinized by our students. Classrooms and schools are moral/ethical climates, for better or worse. Students observe teachers and the greater school environment, and these observed individuals and influential environments theoretically impact character development. From within these moral climates are behaviors and actions that children observe and contemplate adopting. If one is a teacher or administrator, one often is obligated to provide consequences (negative or positive) for whichever behavior a student decides to embrace.

Social learning and reinforcement theories (Bandura, 1977; Skinner, 1969) suggest that the future probability of a behavior will be influenced by such consequences provided by educators. Therefore, social learning theories, as well as sociocultural and other development theories (Bronfenbrenner & Evans, 2000; Dewey as cited in Cruz, 1987; Erickson, 1950; Kohlberg, Levine, & Hewer, 1983; Piaget, 1969; Vygotsky as cited in Wertsch, 1985), provide a lens to see how character education is rooted within theory supporting the learning process and development of the whole child. Some practitioners are unaware of these more complex psychosocial views of character development, however, and what passes for character education may consist of a few words (e.g., respect, responsibility) posted on a bulletin board or in the hallway. Character education, for some, rests on the adoption of positive character traits and is to be taught directly to students as plug-in programs on a daily, weekly, or monthly basis. The variety of approaches to character education makes it difficult to make generalized statements about the field regarding either intent or effectiveness.

With only a few studies on character education programs showing evidence of effectiveness (IES What Works Clearinghouse, n.d.), hundreds of companies are marketing expensive character education programs or products (similar to the above mentioned approach) to schools struggling to stay fiscally afloat. To be fair, the possibility exists that the assessment of such programs could be flawed (or nonexistent) and some of these programs do compliment the instructional or developmental process. To be fair, one might question as to why educators would expect a simple program implemented over a short time of a child’s life to increase moral reasoning or academic achievement. Unfortunately, to a good number of the educators that were interviewed or took part in focus groups during the qualitative research efforts of this grant, the marketed product side of character education is one of the most significant shortcomings and/or challenges of character education today.

According to some of the less motivated participating teachers at the beginning of this study, the marketing of such programs feeds a connotation that character education is a trendy, temporary thing that will go away soon. To others, the marketed programs and products suggest that character education can be accomplished in a mere 20 minutes sandwiched between lunch period and the afternoon reading block, leaving the doubting Mr. and Ms. Thomases of education questioning the efficacy of character education. Additionally, some feel that such a direct approach to teaching character education traits might not be as effective as an indirect approach. Meanwhile, individuals in charge of some schools continue to make character education the sole responsibility of a few teachers and/or the counselors who lack the legitimate power to successfully navigate the innovation process. To some, such programs appear to be the “flavor of the day” and just another thing added to one’s already overloaded plate.

However, as our colleague Dr. Philip Fitch Vincent stated in the keynote address of the 2006 Character Education Partnership Annual Conference, “Character education is not another thing on your plate, it is your plate.” In other words, Vincent views character education as a foundational model or conduit to compliment the character development process while simultaneously improving the school/learning climate, and increasing success of other educational efforts already underway within the schools (Vincent, Wangaard, & Weimer, 2004). The Child Development Project practiced a similar approach with success in their comprehensive, whole-school interventions to foster social, ethical, and intellectual development in students and caring communities of learners (Battistich, Schaps, & Wilson, 2004). Therefore, the efforts (educational interventions) that we hypothesize will work best (especially for rural schools struggling with budget and staffing issues) are character education models that are fully integrated from dawn to dusk (e.g., from the classroom to the cafeteria to the playground, all day and everyday), and basically cost nothing.

Considering the abundance of educational research, philosophical views, and theory supporting the promising efficacy of a character education process model, this study challenged the participating educators to embrace character education as an educational foundation and new instructional lens. Yet as Berkowitz (2002) explains, character education for the most part has been viewed as a practice and not a science. “Whereas character education is not new, scientific study of its effectiveness has been only sporadically implemented … [and] therefore not informed by a scientific knowledge base” (Berkowitz & Bier, 2004, p. 72). Thus, the importance of approaching recent government-funded exploration of character education with sound methods, as well as reliable and more importantly, valid measures, is essential to discovering the true role that character education can play in the educational process. With a process-based definition of character education proposed, and a multitude of research and theory that begins to identify multiple components of the character education process, the following dimensional model was created to assist with a more strategic approach for guiding our data collection procedures.

At a recent research symposium sponsored by the U.S. Department of Education’s Office of Safe and Drug Free Schools, outside evaluators of character education grants were asked to submit a list of the core measures that they currently use for assessing the effectiveness of their character education experimental stimuli. Upon compiling the list submitted by more than 30 grantees, approximately 60 uniquely different core measures were identified. With so many evaluators using such a vast array of differing core measurements focusing on a broad host of variables, the ability to conceptualize a character education process model as well as compile and compare future findings becomes quite challenging. Therefore, though the main focus of this research study is to highlight the relationship between random selection and pretest/posttest designs, before addressing the methods and outcomes of this study, let us explore how such variables that need to be measured at the beginning and end of the study might actually collapse into more generic or common dimensions.

With a plethora of character education products, programs, frameworks, and process models at work in our schools and more than 60 core measurements at the federal level being used to assess quantitatively the possible effects of character-based education on academic achievement, one might say that research into character education is quite diverse. In fact, some might say that research efforts are too diverse. In order to collect a definitive body of evidence on the efficacy of character education, or possibly some day actually create a meta-analysis built on effect sizes collected through reliable, valid, and sound measures and methods, numerous researchers would need to focus more often on a similar variable or set of variables. Such efforts to compare or compile findings could be even more fruitful if the same reliable and validated scales or statistics were used to collect such data across numerous studies. Unfortunately, to agree on specific dimensions or scales that work across a majority of the ongoing research efforts appears to not be possible at this juncture. Beyond the differing opinions as to what constitutes character education, to some degree each research project demands a specific dimensional model that fits the focus and goals of the individual studies; and at this point there appears to be no single agency or catalyst capable of providing such a conduit between character education research efforts.

Yet when one peruses the decades of qualitative and quantitative research amassed on character education (or other related fields such as moral education, civic education, values-based, etc.), one begins to see a few thematic threads begin to surface. When researching new and uncharted waters, such thematic analysis is beneficial to determining which components are theoretically and qualitatively supported in order to focus quantitative efforts accordingly. Therefore, in order to help organize our ongoing research efforts we: (a) reviewed a massive amount of literature, (b) used the constant comparative method (Glaser, 1965) (later known as grounded theory) to complete a thorough qualitative analysis of the literature as well as our participating sample’s insights, and (c) also consulted with character education specialists and fellow researchers to determine a rough outline of such thematic components. Upon reaching consensus that there are many thematic threads that appear to create a systemic model of character education, we created a dimensional model that fit the identified qualitative themes (e.g., clusters and/or dimensions), and adopted or developed scales to measure the variables within the dimensions that we seek to measure for our ongoing federal grant research efforts.

Figure 1 highlights the dimensions identified and other components that the grant guidelines request to be measured. The figure is comprised of categories of experimental stimuli, independent variables, covariates, and dependent variables. Although the proposed thematic dimensions that fall under the independent variables category in Figure 1 are the main focus of these data, we have included the other components to assist in better visualizing how all of the parts theoretically work together and how Figure 1 might create a possible logic model in the future that might be supported statistically when all is said and done. In other words, we find a linear assumption that x (character education/experimental stimuli) causes y (increased academic achievement/ dependent variable) to be slightly over zealous. If the Harry Potter series is unable to have a significant impact on the reading habits of children (Rich, 2007), it seems somewhat illogical to believe that by putting a character education intervention (or for that matter product or program) into place one will instantly see academic achievement increase. Therefore, our systemic logic hypothesizes that the experimental stimulus theoretically could impact the independent variable dimensions resulting possibly in proximal or short-term outcomes. These proximal outcomes then possibly could create enough synergy over a period of time to ideally impact more distal outcomes related to academic achievement. However, before possibly claiming such hypothesized impacts, possible causal relationships, or providing quantitative evidence of a hierarchical linear model; the quasi-experimental process must first be completed, extensive data must be collected on the numerous variables and covariates addressed in Figure 1, and figured into the analyses. To initiate such a process, the baseline pretest data collected on our study’s sample was organized according to the dimensional model in Figure 1.

At the bottom of Figure 1 is the experimental stimuli section. The character education intervention should serve as the foundation for change. When dealing with experimental or quasi-experimental designs, it is essential to measure the type, quality, and duration of implementation or integration taking place. Such process measurements are crucial to delineate the differences between the control samples and the experimental samples. Therefore, for example, in our studies we use a character education process evaluation rubric that is completed regularly by the teachers and administrators in all of our participating schools as one of the tools to quantitatively assess how much character education is being integrated into the curricula and school, what type of character education, and numerous other details. Please note that we are always monitoring for additional variables relating to experimental stimuli specifics found at the school level, community level, as well as individual efforts taking place within the classrooms. Furthermore, we collect such qualitative and quantitative self-report insights from school staff, faculty, parents, and students for a multiple triangulation effect.

At the next level (coming up from experimental stimulus in Figure 1) one finds six dimensions that are supported theoretically as well as through decades of qualitative and mostly non-experimental-based quantitative studies. These dimensions are hypothesized to be the locus of change during a character education intervention. The dimensions consist of character development, community engagement, school climate, professional development, school leadership, and educational attitudes. Theoretically, when character education is effectively put into practice children will develop character (Berger, 2003; Lickona, 1991), the school climate improves (Lickona & Davidson, 2005; Vincent et al., 2004), teachers experience improvement in efficacy and instructional success (Grove & Schneider, 2006), leadership finds new direction (Williams & Taylor, 2003), and children embrace education with more positive attitudes (Berkowitz & Bier, 2005). To begin to measure such dimensions we utilize a number of self-report scales. These scales will be further addressed in the Method section. Note, however, that in order to show that change does take place in dimensions where archival or schoolwide data are unavailable, one must use reliable and (when possible) valid scales or measurements for each category so that quantitative evidence is observable in a meaningful fashion. This way we can feel more confident that we are consistently measuring the same thing over and over (e.g., reliability), and that we actually are measuring that which we say we are measuring (e.g., validity).

The next two levels at the top of Figure 1 designate the need for controlling statistically for extraneous variables (covariates) and deciding on dependent variables that suit the needs of one’s research focus. Covariates are an important part of this dimensional model, because before one can confidently state that character education was responsible for change, one must control for other measures that might erase such findings when added to the equation (e.g., socioeconomic status, SES). In relation to the character education grants awarded through the Office of Safe and Drug Free Schools, the dependent variables for this study’s dimensional model are more aligned with showing how character education impacts academic achievement. Thus, the use of standardized test scores, grade point average (GPA), adequate yearly progress (AYP), and other related variables are expected and are part of future studies related to this research. Before comprehensively discussing the methods and results of the pretest data collection, let us now consider some challenges associated with random assignment.

Exercising random assignment in educational settings can be very difficult (Cook, 2001; Kemple, 1998). Issues that exist with experimental studies in educational settings often deal with the complexity of random assignment to the said intervention. This complexity often is intensified when considering the random assignment of participants to a character education intervention. The term participants may refer to students, teachers, classrooms, or schools. As is the case in many educational settings it is difficult to randomly assign individual students to the intervention (while controlling for contamination) and the approach more often is to assign randomly selected schools or classrooms to the intervention.

The underlying assumption of random assignment often presumes, however, that it will in fact produce equivalent groups. The difficulty that researchers of character education studies face with random assignment of schools is the potential for participant selection bias where “intervention participants might differ from comparison participants in important ways that may affect the ability to detect the intended interventions effect” (U.S. Department of Education, 2007, p. 15). In a study where the school represents the participant, there are many intervention factors that can cause selection bias. In such studies, often it is not within the scope of the researcher to be able to eliminate the preexisting effects of character education interventions that are already in play; therefore increasing the need to statistically control for pretest levels. Some schools in the comparison or experimental sample may already be doing some form of character education, whether formal or informal. School climate and leadership styles also can differ greatly and impact morale and motivation to learn. As a result, issues such as trust and safety can be affected (Corrigan & Chapman, in press) and suddenly schools that appear to be similar in size, age, and other contextual settings differ considerably when measured quantitatively in relation to attitudes of students, staff, and parents. Therefore, even given the assumption of random selection, researchers must carefully depend on baseline data to empirically determine the equivalence of control and experimental schools (or statistically control for differences) when investigating possible changes on variables that in the future might be attributed to the character education intervention. In other words, there is more to measuring the homogeneity of schools than collecting archival data on student body demographics, academic scores and behavioral reports.

When selecting a sample of schools (or in other situations where data are collected from existing groups), the potential for preexisting group effects is present. One way to address this challenge is to treat the school as the unit of analysis. By selecting an equal number of comparison and experimental participants (schools), this allows for analyses of group equivalence in the entire sample of schools, as well as the individual schools themselves (Roberts, Francois, Batista-Foguet, & King, 2000). The current funding of studies in the field of character education and the demands for providing “rigorous evaluation” bring to the forefront the need for researchers to further investigate (at baseline) group equivalency and produce those findings in light of future findings on the impact of the character education intervention.

According to Lickona and Davidson’s (2005) report on smart and good high schools, character education is a promising practice in our schools. Centuries of educational philosophy and more recently developed theory also offer substance to the theoretical foundations and possible efficacy of character education. But similar to many research efforts across numerous social or behavioral science endeavors, a body of definitive quantitative evidence gathered through sound experimental designs is lacking. According to What Works Clearinghouse (2005), “The current nationwide emphasis on ensuring that all students and schools meet high standards has increased the demand for evidence of ‘what works’ in education.”

Due to the demands and expectations of the Government Performance and Results Act of 1993, such shortcomings related to methodology designs and a lack of reliable and valid quantitative evidence (collected through sound methodologies) threaten to affect the future funding of numerous federal lines of research and programmatic implementation. Therefore, to begin collecting data that possibly could offer evidence to support a multidimensional approach to measuring character education efforts and further identify the need for performing baseline measurements on randomly selected participants, the following research questions are proposed:

• Research Question 1 (RQ1): Are there significant differences between the control and experimental groups (of participating students) in relation to self-reported levels of character (Dimension 1)?

• Research Question 2 (RQ2): Are there significant differences between the control and experimental groups (of participating students) in relation to self-perceived levels of community engagement (Dimension 2)?

• Research Question 3 (RQ3): Are there significant differences between the control and experimental groups (of participating students) in relation to perceived school climate (Dimension 3)?

• Research Question 4 (RQ4): Are there significant differences between the control and experimental groups (of participating students) in relation to self-reported educational attitudes (Dimension 6)?

• Research Question 5 (RQ5): Are there significant differences between the control and experimental groups (of participating students) in relation to academic achievement (dependent variables)?

• Research Question 6 (RQ6): What are the relationships shared between the proposed dimensions of character education and academic achievement?

The selection process for the participants in this quasi-experimental study began by selecting four rural counties from the 55 counties in the participating Appalachian-based state. Prior to selecting the four representative rural counties, however, all counties considered urban or more heavily populated were removed from the selection pool. This resulted in the removal of only four counties, given that the Appalachian-based state is predominantly rural and is the only state located completely within the Appalachian Regional Commission (ARC) territory. A purposive sampling procedure was then utilized to identify rural school systems that would: (a) be representative of the challenges experienced by rural school systems in the Appalachian-based state; (b) insure an acceptable level of homogeneity of variance; (c) be willing to participate in this longitudinal study; and (d) provide enough participants to employ a matched random sampling selection of schools and an adequate number of students supporting statistical power.

Once the four counties were selected and secured, we then employed a matched random sampling technique to select the schools. Matched random sampling is a method of assigning participants to groups in which pairs of participants are first matched on some characteristic and then individually assigned randomly to groups (Brown, Cozby, Kee, & Worden, 1999). As part of the selection process the schools were matched by grade (elementary, middle, and high schools), assigned a random number, and then randomly selected to be either an experimental or control group school. This approach allowed all schools in the four counties an equal chance of selection (by group designation), and an adequate number of participating schools affordable within the constraints of the grant award. The result was the selection of two middle schools (from a total of six middle schools in the four counties), two high schools (from a total of six high schools in the four counties), and four elementary schools (from a total of 19 elementary schools in the four counties). Thus, both of the experimental and control samples consist of one high school, one middle school, and two elementary schools. Furthermore, a priori examination of the four counties chosen suggests that the counties are strongly similar to each other and representative of the average rural county in the participating Appalachian-based state. Please note that not all of the schools selected were meeting AYP at the beginning of the project. However, all of the schools were nearly identical in socioeconomic status and students with special needs.

At the initiation of this study all selected schools were assessed to be very similar in the amount of character education being utilized (experimental stimuli). Note that the level of character education taking place at all of the schools was identified as minimal during the development phase and more associated with the program approach to character education. As a control measure against contamination, all control schools agreed to not introduce any new programs or processes related or similar to character education until after the study was completed.

The individual participants from this study then were recruited from the selected schools. Please note that consent rates obtained during the first year of the grant (project development phase) by the participating schools range from 75% to 98% participation. Although the complete sample of participants consists of parents, teachers, administrators, school staff, and students, the focus of this study is on student baseline measurements to be used as pretest assessments in a longitudinal study. The student participants at the middle and high school level consist of 151 males (42%) and 199 females (55%) for a combined N = 366. The high school students at the time of this survey (beginning of phase two, intervention stage,) had recently begun 10th grade, and the middle school students were starting seventh grade. The participants at the elementary level consist of 61 males (52%) and 56 females (48%) for a combined N = 124 (note: 7 nonreports). At the time of this measurement the elementary students recently began fourth grade.

Overall, the Appalachian-based state has a low minority rate when compared to other states. However, data show that the portion of the population that is minority lives in impoverished conditions similar to those of third world nations. The economic insecurity that haunts the Appalachian-based state families is evident in this Appalachian-based state having the lowest median income (as reported by the Census Bureau for 2006), and one of the highest child poverty rates in the country. According to the state’s department of education, approximately 51% of the children in the Appalachian-based state qualify for free or reduced lunch. Furthermore, as of March 16, 2004 the Appalachian-based state had the highest rate of children with disabilities in the nation.

In addition to the economical, behavioral, and physical challenges faced by children in the participating Appalachian-based state, there also are environmental factors to consider in regard to the student participant. According to a Youth Risk Behavior Report (Centers for Disease Control, 2004), collected in the Appalachian-based state in 2003 (collected on high school students): (a) a total of 20.7% of students carried a weapon in the past 30 days; (b) a total of 9.3% of students attempted suicide in the past 12 months; (c) a total of 44.4% drank alcohol in the last 30 days, (4) a total of 52% of students have had sexual intercourse; and d) a total of 29% of current high school students are smokers.

In the four counties we are studying approximately 10% of the births in the counties are to unmarried teens, 20% of births are to mothers with less than a 12th grade education, and dropout rates average about 17%. In the participating elementary schools 60% of the students receive free or reduced lunch and 21% are designated as having special needs. In the participating middle and high schools 53% of the students receive free or reduced lunch, and 14% are designated as having special needs.

Experimental and control schools reported levels of students with special needs and SES challenges were quite similar in Phase 1 of the grant. To study the affect of character education on school systems experiencing such challenges and threats to child development, all of the student participants and their parents agreed to take part in the study by supporting the character efforts of the school, and agreeing to take a survey twice a year as well as provide access to personal data such as standardized test scores, attendance, special needs, SES, and other education-related statistics.

Similar to many rural school settings across the Appalachian-based state, the participating schools are surrounded by a sometime surreal contrast of rich and poor, almost devoid of a middle class. One often will find new luxurious homes built in close proximity to poverty-ridden trailer parks or dilapidated residential structures. School consolidation is not uncommon to the region. Some of the participating schools are new and some are very old. Yet although, the schools might differ qualitatively in description, and in some categories differ slightly quantitatively (to be discussed shortly), they all seem equally to provide a home away from home; and unfortunately for some children a place that is much warmer and loving than the home they live in.

As discussed earlier, how one defines character education will impact the character education that one practices. Thus for this study we defined character education as a process; not a program, product, or a practice, but more of an educational foundation supported by theory and approached as a model or framework. Given that the level of character education practiced at both the control and experimental schools was minimal at best at the beginning of the grant, a good part of the first year of the grant was dedicated to helping the experimental schools become educated on character education. To begin this process, character development teams (CDTs) similar to Lickona and Davidson’s (2005) ethical learning communities (ELCs) were formed at each experimental school. These teams consisted of teachers, administrators, counselors, service personnel, parents, and community members. The CDTs were charged with researching and developing unique character education models for their specific schools. Once a school model of character education was created and agreed on for the schoolwide initiative by the service staff, teachers, and administrators, the teachers were then trained to integrate character education into the classroom. Thematic instruction was encouraged to be a part of all subjects in order to create more inspirational, moral-based applicable themes to learning. Furthermore, teachers were provided a publication that examined how a large number of state content standards crossed over to character education. The experimental participants were asked to wait until after the baseline measurements to begin implementing the newly designed models. The control participants were asked not to introduce anything new to the school or curriculum that was similar to character education for the first three years of the grant. The experimental and control schools (as well as the CDT members in the experimental schools) were provided stipends for their participation in the study.

Although this study focuses on pretest data collected specifically from participating students, a multiple triangulation methodology guides this grant project’s mixed methods research effort. Multiple triangulation is defined as when the researchers combine in one investigation: multiple observers, theoretical perspectives, sources of data, and methodologies. Researchers triangulate by using different data sources to confirm one another (Gay & Airasian, 2003). During the first and last month of the school year (for the next 3 years), student participants (as well as parents and staff) will complete a multiple-scale survey. The measurements are taken twice a year in order to account for changes that might take place during the summer months away from the experimental settings. Given that this was a multiple site collection process, settings for the school’s data collection differ from place to place. However, the surveys for this pretest assessment were administered in either the cafeteria or auditorium by outside evaluators, and no school staff or teachers were present at the time. Furthermore, student identification numbers were used on the surveys and answer sheets to provide a greater level of confidentiality. Participants were asked to respond truthfully to the survey questions by reflecting on previous experiences relating to the past year in their neighborhood, community, and school. The surveys were administered in accordance with guidelines for research with human participants (American Psychological Association and the institutions involved). As part of an agreement with the Appalachian-based state’s department of education, additional data were provided for indicators of academic achievement, special needs, and other academic related statistics for use in analyses. Thus, the student identification number also provides a mechanism to track the youth and their parents longitudinally in relation to character education received, related dimensional variables, and education-related dependent variables. Confidential code numbers also are assigned to participating parents, educators and school staff for longitudinal assessment efforts.

The survey contains demographic questions that record the participant’s sex, age, involvement in community or school-related activities, and questions relating to television use, video games, homework, and family structure. Additional data were provided by the state and participating local school systems on academic achievement (standardized test scores, GPA), special needs, and free/reduced lunch (SES). In addition to the demographic questions and archival data collected, the survey used multiple questions and self-report type scales described in the following paragraphs to begin a more thorough assessment of the proximal and distal outcomes possibly related to character education. The scales on this survey were selected after several pilot runs administered during Phase 1 of the study. The scales have been organized by the dimension they fall under in relation to Figure 1. See Tables 1, 2, 3, and 4 for the characteristics of all scales.

Concern for others scale (CFOS). This 9-item Likert-type scale (1 = strongly disagree to 5 = strongly agree) was developed by the Developmental Studies Center to measure the level of concern that students have for others. Previous research utilizing this scale reports past internal reliabilities ranging from .78 to .81 (Battistich et al., 2004; Battistich, Solomon, Kim, Watson, & Schaps, 1995; Solomon, Battistich, Watson, Schaps, & Lewis, 2000). No formal studies supporting the construct validity of this scale have been identified. A sample item is “People should look after themselves and not try to solve other people’s problems.”

Assessment of student moral and performance character scale (MORC & PERC). Both of these 12-item Likert-type scales (1 = strongly disagree to 5 = strongly agree) were adapted from a part of the Khmelkov and Davidson (2006) Collective Responsibility for Excellence and Ethnics (CREE) assessment survey that focused on performance and moral character development. Coefficient alpha reliabilities (internal consistency) for these scales have been reported in the .80 ranges. A sample item for the moral character scale is “I admit if I do something wrong.” A sample item for the performance character scale is “I think about my school work and consider whether I need to work harder.”

Self-perceived character elements scale (CHAR). This Likert-type scale was developed for this study by combining the most widely published/used character traits/values/virtues/ elements (e.g., honesty, respect) that are touted throughout character education and collapsing them down to 19 one-word items. For example, respondents can answer as to how much they understand the term “Diligence” on a scale from 1 to 5; with 1 being “I do not know what this is” to 3 being “I know what this is, I think about it often, and practice it occasionally” to 5 being “I know what this is, I think about it often, and practice it constantly.” These items will be studied through an educational psychology based lens that considers the process of learning through the cognitive, affective, and behavioral levels. At this time no reliabilities or validity tests are available. Please note that this scale is considered too advanced for elementary students and therefore was only utilized in the middle/high school survey.

Misconduct at school (MS). This 4-item Likert-type scale (1 = never to 5 = 10 plus times) was developed by the Developmental Studies Center to measure the level of misconduct that students practice in school. Research utilizing this scale reports internal reliabilities ranging from .74 to .79 (Battistich et al., 1995; Battistich et al., 2004; Solomon et al., 2000). No formal studies supporting the construct validity of this scale have been identified. A sample item is “Have you ever cheated on a test?”

Altruism scale (ALTS). This 9-item Likert-type scale (1 = never to 5 = 10 plus times) is a scale that was modified after Rushton, Chrisjohn, and Fekken (1981) by the Developmental Studies Center to measure the level of altruism that students practice in general. Research utilizing this scale reports internal reliabilities ranging from .80 to .85 (Battistich et al., 1995; Battistich et al., 2004; Solomon et al., 2000). No formal studies supporting the construct validity of this scale have been identified. A sample item is “Have you stood up for someone who was being picked on?”

Interpersonal community engagement scale (ICE). The ICE scale is a 20-item Likert-type scale that was developed to provide a measurement for the degree to which one is communicatively engaged within one’s community or neighborhood (Corrigan & Walls, 2007). For this study, participants were asked to consider the neighborhood in which they live and answer each item (1 = strongly disagree to 5 = strongly agree). The scale was designed to capture the level of community communication and involvement to better gauge the connection one feels to one’s neighborhood, and to empirically measure if interpersonal community-based dynamics have a relationship to youth behaviors and educational attitudes. Research utilizing this scale reports internal reliabilities ranging from .89 to .93, and some initial support for construct validity (Corrigan, 2004). A sample item is “My relationships with my neighbors have helped me to be a better person.”

It is noteworth to mention that in addition to the ICE scale, involvement in community or school activities at the middle/high school level was measured by several 5-point Likert-type items. The questions asked the participant to “please indicate your level of activity in the following activities based on your involvement with 1 being the least active and 5 being the most active.” The activities included (1) band, (2) church, (3) family, (4) school (extracurricular), (5) sports teams, (6) youth organizations, and (7) neighborhood.

Sense of school community (SSAI & SSUP). Both of the 9-item Likert-type subscales (1 = strongly disagree to 5 = strongly agree) were developed by the Developmental Studies Center to measure a sense of school community. The SSAI focuses on the autonomy and influence experienced by the student participant, while SSUP focuses on school supportiveness. Research utilizing this scale reports internal reliabilities ranging from .80 to .82 (Battistich et al., 1995). Numerous studies utilizing this scale offer support for the construct validity of this scale (Battistich et al., 1995; Battistich et al., 2004; Solomon, Battistich, Kim, & Watson, 1997; Solomon et al., 2000). A sample item for SSAI is “Students help to decide what goes on at this school.” A sample item for SSUP is “Students at this school are willing to go out of their way to help someone.” Please note that these subscales were only used on the middle/high school survey.

Victimization at school (VICS). This 6-item Likert-type scale (1 = never to 5 = 10 plus times) was developed by the Developmental Studies Center to measure the level of victimization students experience at school. Research utilizing this scale reports internal reliabilities ranging from .75 to .79 (Battistich et al., 1995; Battistich et al., 2004; Solomon et al., 2000). No formal studies supporting the construct validity of this scale have been identified. A sample item is “Did someone threaten to hurt you but did not actually hurt you?”

Liking for school (LFS). This 6-item Likert-type scale (1 = strongly disagree to 5 = strongly agree) was developed by the Developmental Studies Center to measure how much students like their school. Research utilizing this scale reports internal reliabilities ranging from .81 to .83 (Battistich et al., 1995; Battistich et al., 2004; Solomon et al., 2000). No formal studies supporting the construct validity of this scale have been identified. A sample item is “My school is a fun place to be.” Please note that due to the length of the middle/high school survey, this scale was used only in the elementary survey.

Loneliness at school (LSS). This 8-item Likert-type scale (1 = strongly disagree to 5 = strongly agree) was developed by Asher, Hymel, and Renshaw (1984) to measure how lonely students feel at their school. Research utilizing this scale reports internal reliabilities ranging from .83 to .85 (Battistich et al., 1995). No formal studies supporting the construct validity of this scale have been identified. A sample item is “I don’t have anyone to hang around with at school.” Please note that due to the length of the middle/high school survey, this scale was used only in the elementary survey.

Student motivation scale (SMS).Brophy (1987) defined student motivation to learn as “a student tendency to find academic activities meaningful and worthwhile and to try to derive the intended academic benefits from them” (p. 205). This study used the Richmond (1990) Student Motivation Scale (SMS) to measure the state of a student’s motivation to learn, which is believed to be intrinsic. The scale consists of five, five-step bipolar adjectives (5-point semantic differential scale) with reported previous alpha coefficients of .94 and considerable evidence of construct validity (Rubin, Palmgreen, & Sypher, 1994). Responses were given to “immediate feelings about school work.” Examples of the bipolar adjectives are “motivated-unmotivated” and “interested-uninterested.” For this study, SMS also was used to identify the possible relationship between one’s level of engagement in the community and education-related views.

Academic Self-Esteem (ASE). This 4-item Likert-type scale (1 = strongly disagree to 5 = strongly agree) was developed by the Developmental Studies Center to measure a student’s level of academic self-esteem. Research utilizing this scale reports internal reliabilities ranging from .82 to .84 (Battistich et al., 1995; Battistich et al., 2004; Solomon, Battistich, Watson, Schaps, & Lewis, 2000). No formal studies supporting the construct validity of this scale have been identified. A sample item is “I am doing a good job in school.”

Trust in teachers (TTS) and trust in principals (TPS). The trust in teachers scale was originally a 6-item scale that measured the level of trust a student has in one’s teachers. Due to requests from one of the participating school systems, two negatively worded items were cut from the scale. Also, with permission from the Developmental Studies Center the trust in teachers then was adapted to also measure trust in principals. Research utilizing the trust in teachers scale (as a 6-item measure) reports internal reliabilities ranging from .75 to .84 (Battistich et al., 1995; Battistich et al.; Solomon et al., 2000). No formal studies supporting the construct validity of the trust in teachers scale have been identified. The trust in principals’ scale is a new adaptation of the trust in teachers scale and will need further research to support construct validity. Both have been edited down to 4-item scales that measure the level of trust a student has for teachers or principals based upon a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree. Sample items for these two scales are “The teachers in my classes really care about me” and “The principal in my school really cares about me.”

Student perceived teacher efficacy scale (SPTES). The SPTES is based on Milson’s (2003) Character Education Efficacy Belief Instrument (CEEBI). SPTES consists of 24 statements on a 5-point Likert-type scale adapted to measure a student’s perception of their teachers’ ability to teach and develop character, rather than teacher’s view of themselves. The original instrument has two scales: 12 items measuring personal teacher efficacy (PTE) and 12 items measuring general teacher efficacy (GTE). The CEEBI scales of PTE demonstrate alpha scores of .80 and GTE scales demonstrate alpha scores of .66. Bivariate correlations between the scales have been significant and strong at .69. For the first test of the new scale all of the 24 items were rewritten to fit the student perceived view and will be considered in future efforts to factor analyze a more succinct measure. A sample item is “My teachers know how to use strategies that might lead to positive changes in students’ character.” (Please note that Dimensions 4 and 5 are more related to data collected through or on teachers and administrators and will be discussed further in future reports related to the adult participants in the study.)

The first part of the analyses focuses on assessing the reliabilities of the scales utilized for this study. The second part examines the similarities and/or possible significant differences between the students’ self-reported survey data from the control and experimental schools; as well as data provided by the state and local school systems on academic achievement, special needs, and free/reduced lunch. Therefore, MANOVA will be conducted. Please note that given the age difference between the elementary sample and middle/high school sample, a slightly different survey was administered to collect the data related to the six dimensions. Thus, for the pretest assessment, the elementary and middle/high school students will be analyzed separately. Lastly, to begin considering the possible relationships between character, academic achievement, and educational-related variables, post hoc correlational analyses will be conducted. The initial baseline post hoc analyses also will be performed separately on the middle/high school sample and the elementary sample. Such an approach will allow us to go beyond examining differences between the control and experimental groups, and help to further (or more specifically) assess the relationships between proposed variables and the baseline homogeneity of the like age groups.

Reliability (coefficient alpha) was computed for all of the scales discussed in the Method section. Past and present reliabilities (as well as validity support) for each scale can be found in Tables 1-4. Most scales showed adequate alpha coefficient reliability (.70 to .93) for the baseline measurements. However, a few of the scales utilized at the elementary level performed less reliably in comparison to the middle/high school data. Such results might suggest that a couple of the scales utilized were not age-level appropriate to the elementary setting. Therefore, special attention to the elementary results should be directed toward specific scale reliabilities of the variables measured and compared.

Research Question 1 (RQ1) examined if there were significant differences between the control and experimental groups (of participating students) in relation to the students’ self-perceived character (Dimension 1). To examine overall findings and to control for Type 1 error, a one-way MANOVA was computed. According to Wilks’ Lambda, control and experimental groups at the middle/high school level did not differ significantly [F(6, 328) = 1.48, p = .184], but the elementary schools did differ significantly [F(5, 81 ) = 2.639, p = .03]. See Table 5 for a comparison of mean scores and significant differences identified for Dimension 1.

Research Question 2 (RQ2) examined if there were significant differences between the control and experimental groups (of participating students) in relation to the students’ self-perceived community engagement (Dimension 2). To examine overall findings and to control for Type 1 error, a one-way MANOVA was computed. According to Wilks’ Lambda, control and experimental groups at the middle/high school level did not differ significantly [F(5, 349 ) = .448, p = .815]. At the elementary level only one variable was used to assess community engagement, therefore no MANOVA was conducted. See Table 6 for a comparison of mean scores for Dimension 2.

Research Question 3 (RQ3) examined if there were significant differences between the control and experimental groups (of participating students) in relation to the students’ self-reported perceptions of school climate (Dimension 3). To examine overall findings and to control for Type 1 error, a one-way MANOVA was computed. According to Wilks’ Lambda, control and experimental groups at the middle/high school level did not differ significantly [F(3, 351) = .767, p = .513], but the elementary schools did differ significantly [F(3, 89) = 5.839, p = .001]. See Table 7 for a comparison of mean scores and significant differences identified for Dimension 3.

Research Question 4 (RQ4) examined if there were significant differences between the control and experimental groups (of participating students) in relation to the students’ self-reported educational attitudes (Dimension 6). To examine overall findings and to control for Type 1 error, a one-way MANOVA was computed. According to Wilks’ Lambda control and experimental groups at the middle/high school level did differ significantly [F(5, 347) = 8.537, p = .001] and the elementary schools differed significantly [F(4, 90) = 6.027, p = .001] as well. See Table 8 for a comparison of mean scores and significant differences identified for Dimension 6.

Research Question 5 (RQ5) examined if there were significant differences between the control and experimental groups (of participating students) in relation to academic achieve-ment. To examine overall findings and to control for Type 1 error, a one-way Multivari-ate Analysis of Variance (MANOVA) was computed. According to Wilks’ Lambda, control and experimental groups at the middle/high school level did differ significantly [F(2, 286) = 3.969, p = .02], but the elementary schools did not differ significantly [F(2, 97) = 1.235, p = .295]. See Table 9 for a comparison of mean scores and significant differences identified for academic achievement.

As part of a post hoc analysis effort to go beyond accounting for baseline measurements and to begin exploring how the components of the dimensional model work together and relate to academic achievement, Research Question 6 (RQ6) examined the relationships shared between the dimensions and academic achievement. These correlations were per-formed separately on the elementary and middle/high school samples in regard to each of the six dimensions and how they relate to aca-demic achievement. Pearson correlations iden-tified numerous significant relationships between academic achievement scores and the measurements or scales used to assess all of the proposed dimensions. Basically, the results from the correlational analyses offer support to the hypothesis that a multidimensional model of character education relates positively to academic achievement. Therefore, the post hoc analyses provide support for the belief that when a student’s self-reported levels of character, community engagement, educational attitudes, as well as views of school climate increase, theoretically, so will that student’s academic achievement. See Tables 10A through 13B for the complete listings of the Pearson correlations and p values.

The results of the pretest assessment highlight that at the beginning of this grant’s character education intervention, for the most part (on the data provided by the participating state department of education) the control and experimental schools seem quite similar. Yet when we move deeper into a multi-dimen-sional analysis that encompasses more than the typical archival behavior and attendance reports, or test data readily available, we begin to see that not all students feel the same way about one’s school. Such feelings, if signifi-cantly different, can provide extra challenges when trying to improve schools through character education. Often assessment of schools strictly relies upon academic scores, behavioral reports, attendance, and other archival data collected county or statewide.

However, such a strong reliance on school reports or archival data can be misleading. Grade point averages are often inflated (Woo-druff & Ziomek, 2003). State based standard-ized test scores often only provide a snap shot of a classroom or school’s mental acuity and rarely ever track the progress or improvement made by individual students from year to year. If lack of attendance is assumed to rep-resent a lack of character, a student who is at home with a life threatening medical issue might be labeled incorrectly low in character. And the unfortunate part of behavioral reports is that many principals in the worse schools tend to not fill out the reports as readily as those principals who have schools that shine with excellence. Therefore, when beginning research into the efficacy of a movement such as character education, we believe that is important to assess the opinions of all of the participants involved in order to more thoroughly quantitatively assess the bigger picture of the statistics that exists. Theoretically, the six dimensions proposed hold great promise for improving education and research would suggest that deficiencies in such areas might impact one’s efforts to improve a learning environment.

For instance, some significant differences that should catch one’s attention are that the students at the control schools (at both the elementary and middle/high school level) seem to trust their principals more than the students at the experimental schools. Also, at the elementary level control school students (in comparison to the experimental school students) appear to like their school better, feel more motivated to learn, and yet feel more victimized at school. Overall, the pretest data collected from student participants on the six dimensions should provide an organized, thorough assessment of the preexisting conditions relevant to this research, as well as an interesting reflection to the same data collected from parents and school staff.

However, future assessment of the intervention will seek to utilize these data as a benchmark and more deeply explore how such an intervention possibly influences academic achievement, attendance, and behavior issues; as well as change of attitudes within the school’s children, educators, and parents. One of the goals of the grant is to work toward determining hierarchical linear models supported by quantitative evidence. In other words, it is our intention to try to use the proposed dimensional model (Figure 1) to serve as a rough hypothesized draft of a possible logic model. The possible impact of the intervention as well as the interaction between the proposed components of the dimensional model should provide more detailed quantitative evidence of how the variables potentially relate together and provide a role within a complex system associated with a character education process model. Thus the post hoc analyses provided a sneak peek into the window of what might materialize.

Post hoc correlational analysis provides reliable evidence that character, as well as school climate and educational attitudes, are significantly and positively related to academic achievement. A positive relationship suggests that when one variable goes up or down, the related variable goes up or down accordingly. Therefore, we gather from this sample and study that when one’s character is more positively self-perceived, one’s academic achievement shows higher levels of success. The variables most significantly and positively related to standardized test scores were student motivation to learn, liking for school, and character elements. The question to be answered from future research on this grant, however, is whether or not the intervention employing a character education model (process) can increase levels of perceived character, school climate, community engagement, and educational attitudes, and if such increases do lead to increasing academic achievement.

The pretest data also highlight that there are a few limitations that should be addressed. First, Tables 1 through 4 show that some of the scales did not reach a high enough reliability at the elementary level. Efforts to monitor such reliabilities or change/replace scales accordingly should be put forth. Tables 1 through 4 also illustrate that very few of the scales utilized within this study (and also within character education in general) have gone through a formal construct validation effort. This we feel is paramount to truly supporting findings at the end of the study. Therefore, major efforts to perform construct and criterion related validation studies on scales related to each dimension are mandatory. Also, the elementary samples will provide a few challenges when it comes to statistical power and cell sizes, given the smaller populations associated with rural elementary schools and differing class sizes in the control and experimental samples.

Research states that schools that are outperforming the norm in relation to meeting No Child Left Behind tend to follow three practices: (a) they require more hours of class time than typical public schools (e.g., time on task), (b) they strongly support teachers in an effort to enthusiastically treat instruction and lesson planning more like a science rather than an art, and (c) they make a conscious effort to guide behavior, and even values, of their students by teaching what they call character (Tough, 2006). With character education, the hope lies in creating students who are responsible, diligent, and more motivated to learn the essential skills for meeting No Child Left Behind demands as well as for becoming contributing members of society. A more Skinnerian-based view might propose that we could simply increase motivation through positive reinforcement for learning to take place at a greater rate and deeper level. Theoretically, children are more motivated to perform to their highest ability when they are inspired and feel a need to make one’s mentors proud. Theoretically, children can accomplish such goals more often in learning environments that offer a supportive, safe, caring, and constructive avenue to academic success. If current efforts to increase proficiency rates across all children are falling short of expectations, is it possible for schools practicing integrated character education models to increase academic performance through enhancing a positive learning environment and increasing motivation to succeed? Future research into the efficacy of character education should closely examine this possibility, as well as the methods employed to measure such multidimensional educational goals.

The funding for this longitudinal quasi-experimental study came through the U.S. Department of Education’s Partnerships in Character Education.