Holding Schools Accountable for Equity in Character Measurement: Bridging Theory and Practice Free

Although character education has been a foundational part of American schooling since its origins (Campbell, 2012; Dewey, 1916/1966, 1938; Johanek, 2012; McClellan, 1999; Seider, 2012), recent years have seen an explosion of a new focus on character and social emotional learning frameworks, education, and measurement (e.g., CASEL, Character Lab, 21st Century Skills, etc.; Assessment Work Group, 2019; EASEL Lab, 2019). In fact, one study found that over 71% of American principals are currently implementing, or plan to implement, socioemotional learning (SEL) programs in their schools (Atwell & Bridgeland, 2019); moreover, schools and school districts are estimated to spend approximately $640 million on such education annually (Assessment Work Group, 2019; Krachman & LaRocca, 2017). This emphasis on character education and SEL in schools is unsurprising, given that studies continually demonstrate associations between character or SEL and positive life outcomes; for example, a 2011 meta-analysis of 213 studies and more than 270,000 students indicated that social and emotional programming yields significant positive effects on prosocial behaviors, self-esteem, and academic performance, as well as reduced behavioral incidents and reduced emotional distress such as depression and anxiety (Durlak et al., 2011).

Despite this growing emphasis on character and social emotional learning throughout educational settings, how to best measure character in schools and evaluate the effectiveness of character interventions has remained a complex question. Concerns around reliability (e.g., will results differ depending on a student’s particular mood when completing a survey? Might students and teachers misinterpret a survey item or inaccurately report on internal emotions?), face validity (e.g., does a particular survey on student empathy actually measure how a school is defining empathy? How can we account for students who will rate themselves higher to appear more desirable?) and reference bias (e.g., does a student’s frame of reference impact how they interpret elements of character) abound. Questions around what constitutes possessing a certain character strength remain as well: for example, should we trust student self-reports on their own beliefs and dispositions, or must we rely on measuring only external manifestations such as tangible behaviors that can be seen and documented?

Perhaps most importantly, there is also a growing call for a need to be attuned to issues of character assessment and equity, with equity referring to the concept that “each child receives what he or she needs to develop to his or her full academic and social potential” (nationalequityproject.org). For example, to date, research has found that students of color score lower on traditional summative surveys of character, signaling a need to shift the way that schools conceptualize and evaluate these constructs (e.g., Fiarman et al., 2021). Similarly, character and SEL education has not been found to benefit all students in a fair and equitable manner. For example, in a recent review of 136 SEL competency frameworks, fewer than 20% took culturally or linguistically diverse samples into consideration, fewer than 20% considered youth with disabilities’ experiences, and fewer than 6% considered students’ potential experience with trauma (Berg et al., 2017). A study by RAND has likewise found that schools that implement SEL use such frameworks to control and manage students’ individual behaviors rather than to foster student belonging or to explore contextual aspects of students’ lives (The Education Trust, 2020; Hamilton et al., 2019). In fact, in focus groups with students of color, Black families, and Latinx families, The Education Trust (2020) found that these participants did not trust their educational institutions to teach the SEL skills and strengths that they value, such as a knowledge of one’s self and one’s cultural identity, and did not believe that these institutions were set up to create safe, unbiased spaces where students of color could explore their authentic selves.

One specific equity concern relates to the continued focus on the need for individual students to “fix” themselves and their own character rather than for schools to provide adequate environments and relationships to help nurture student character (e.g., Fisher, 2020; Kaler-Jones, 2020; Kirshner, 2015; Love, 2019; Simmons, 2019; Soutter, 2020; Zaino, 2016). For example, Fisher (2020) noted that in one recent prominent character report the word “skills” appeared 140 times whereas the words “relationship” and “network” appeared only 23 and 14 times, respectively. As Fisher stated,

In a similar critique of the preeminent SEL focus on teaching students of all backgrounds to persist or “have grit” despite whatever systemic challenges they might encounter, one scholar noted, “can you imagine the outcry if, let’s say, an old toxic dump was discovered near Scarsdale or Beverly Hills and the National Institutes of Health undertook a program to teach kids strategies to lessen the impact of toxins but didn’t do anything to address the toxic dump itself?” (Rose, 2013). Clearly, a “one size fits all” approach to character and SEL measurement will not adequately capture the experiences and development of all students; measuring character and SEL equitably means looking outwards towards our environments and contexts and understanding how they, too, can impact student development.

A second equity concern relates to how character measurements are used by teachers and institutions. Character education measurement traditionally grew out of trait-based personality measures (e.g., the Big 5 trait taxonomy; Costa & McCrae, 1992), that originally aimed to assess how stable an individual’s personality remained over time (Lerner et al., 2015). As such, these tools, although often used to measure student character (Lerner et al., 2015), were not optimally designed to be responsive to teachers’ need for developmentally appropriate measures that could both assess students’ character change and growth and subsequently inform teacher practice (Assessment Work Group, 2019). Based on this history, current character summative assessments—those that attempt to measure student or teacher results or progress after a period of time—tend to be higher stake (e.g., student or teacher outcomes, such as performance ratings, might rely on the results) and/or they do not provide enough information from which to make actionable, real-time adjustments in the classroom (Assessment Work Group, 2019; Marsh et al., 2018; McKown, 2019). Given the rise in character education in schools across the United States and elsewhere, a new focus on low-stakes, formative character measurement in schools is now becoming increasingly widespread (Bryk et al., 2015; McKown, 2019). That is, more and more educators and administrators want character and social emotional learning measures that can be used for “continuous improvement” efforts in classrooms in order to help better guide instruction and respond to students’ character and social emotional learning needs (Assessment Work Group, 2019; Fiarman et al., 2021; Lee & Riordan, 2018; Schlund et al., 2020). Still, the question of how character measurements are interpreted and used remains an important concern. If a student receives a low mark on an end-of-year character report card, for example, that score might become internalized (by themselves and others) as a final judgment of that student’s character. If, on the other hand, measurements are explicitly used to gather information so as to make instructional and institutional improvements, these measures have the potential to be interpreted with a more strengths-based lens (e.g., what can I as the teacher do to support students in these ways) and to serve a very different purpose.

Thus, if we care about finding ways to measure character equitably in schools it behooves us to find ways to holistically and formatively assess the character and context of all students. Below, we delve further into these two areas related to character measurement and equity, followed by a description of the ways in which one organization—EL Education—has approached character measurement around these two areas of focus through their co-construction and initial implementation of their walkthrough observation tool. This approach looks beyond the use of survey tools alone and works to create an equitable school environment through assessing students’ learning contexts and using these data for the purpose of continuous improvement. We are hopeful that such a case study might prove useful to others interested in creating formative measures for assessing student character development, but, perhaps more importantly, for those working with or in schools attempting to understand new ways to think about equitable character assessment and implementation.

Given these circumstances and the concerns outlined above, scholars have outlined several ways that the field can address these two overarching equity concerns regarding contextual considerations and summative vs. formative measurement. We review the various ways these concerns have been addressed in the sections below.

Individual Versus Contextual Conceptualizations. Many scholars have advocated for a reconceptualization of character and SEL. For example, Jagers and colleagues (2018) have argued that there is a need for “transformative SEL,” a form of SEL that acknowledges that students can have vastly different experiences in the classroom due to racial/ethnic or gender discrimination, economic status, or past traumatic events. As such, they noted that transformative SEL “build[s] strong, respectful relationships founded on an appreciation of similarities and differences; learn[s] to critically examine root causes of inequity; and develop[s] collaborative solutions to community and social problems” (p. 2). Similarly, The Education Trust (2020) champions a vision of SEL education that similarly focuses on recognizing contextual and cultural differences across individuals, challenges students to reach their full potential, aims to reduce bias, works to foster student belonging, and focuses on the whole student.

Within the character education field scholars have similarly criticized the primary focus on developing students’ individual character strengths to the detriment of acknowledging other contextual and systemic factors that hinder or promote character growth. For example, Seider and Graves (2020) argue that focusing on cultivating character strengths like grit or growth mindset in students “can have powerful and positive effects, but doing so can also unintentionally convey that students’ ability to thrive is entirely dependent upon their own internal qualities” (para. 2). They argue, instead, for a critically conscious framework for character education—one that includes teaching students about structural inequities and how to challenge such inequities while drawing on moral, intellectual, and civic character strengths (Seider et al., 2017). Kirshner (2015) has similarly argued that youth development, and youth character development, needs to be understood from a sociocultural perspective wherein youth development is understood holistically within its ecologies and contexts. As others have noted, equitable social and emotional learning needs to move away from focusing on how to “fix kids” and more towards focusing on the contexts within which children live and develop, including their cultural, familial, socioeconomic, educational, individual, and societal contexts (The Education Trust, 2020). This shift is complex, of course, as character development is most commonly understood as resting neither solely with an individual nor solely within the influence of one’s contextual surroundings (e.g., Lerner & Callina, 2014; Search Institute, 1997). Indeed, consistent with ecological systems theory (Bronfenbrenner, 1979, 1993) and relational developmental systems metatheory (Lerner & Schmid Callina, 2014), character development is most recently understood as resulting from reciprocal interactions between the individual and their culture(s), peers, schools, families, sociopolitical climate, and other and contexts (Osher et al., 2020). So while we cannot attribute socioemotional development to context alone, it is nonetheless critically important to recognize these factors to avoid an overly individualistic vision of character that ignores external and systemic influences.

What does transformative SEL (Jagers et al., 2018) or a critically conscious approach to character (Seider et al., 2017) mean for character measurement? How can we measure character equitably? As Carter and colleagues professed (2017): “You can’t fix what you don’t look at” (p. 207); if part of equitable character education is being attuned to students’ contexts and how they impact student development (e.g., The Education Trust, 2020), then, in order to build more equitable learning environments we need to assess and analyze these learning contexts and environments rather than only students’ thoughts, attitudes, and behaviors themselves (Schlund et al., 2020).

Measuring Contextual Elements. One example of this focus on contextual elements comes from Paunesku and Farrington’s (2020) work with practitioners to develop an educational and developmental framework focused on students’ socioemotional and academic development (SEAD). In this project, Paunesku and Farrington (2020) examined how contextual, “upstream,” elements, such as feelings of classroom belonging, could influence “downstream” elements such as students’ social emotional outcomes and levels of academic attainment. Accordingly, these scholars argued that it is critical that scholars and practitioners move away from the more typical focus of accountability measures (such as test scores) and focus more on measuring the ways that students feel about the “upstream” elements that are related to the development of their social emotional and academic strengths (Paunesku & Farrington, 2020). Importantly, Paunesku and Farrington (2020) pointed out that such strengths tend to not be something that an individual “has” or does not “have,” but rather are something that one displays depending upon the surrounding context, which, they argued, further indicates the importance of measuring educational contexts and how they are affecting students.

Notice that when speaking about character and SEL measurements in schools we often refer almost exclusively to surveys. In fact, in a recent report on the state of K–12 SEL education, Tyton Partners (2020) reported that 52% of school districts have engaged in some form of SEL measurement, with the Panorama Student Survey, Panorama Teacher and Staff Survey, Panorama Family-School Relationships Survey, and Panorama SEL survey being the most popular forms of measurement in schools. Other core SEL measurement providers and assessments identified in the report include the CORE district surveys, YouthTruth, Perts, Emote, and the Pearson Social Skills Improvement System Rating Scales (SSIS), and xSEL Labs. Of these, xSEL Labs and the Pearson SSIS are the only non-survey options; xSEL labs offers behavioral measures (those that directly measure observable skills) in addition to teacher and peer rating forms, while the Pearson SSIS offers a way for teachers to directly rate their students.

Indeed, many questions exist about whether surveys are the most appropriate way to measure character, particularly from an equity perspective. Many, for example, worry that character surveys are reductive of nonacademic outcomes (or, in one school leader’s words, “suck the joy out of character”). Others fear that character assessment may be used to further alienate and penalize stakeholders (leaders, teachers, and children) in a system already rife with inequitable and punitive consequences connected with data (Soutter, 2020). Self-report surveys also suffer from being prone to a variety of biases; for example, reference bias (King & Wand, 2007), where individuals use different frames of reference to measure themselves or others when making a judgment; social desirability bias (answering in ways that the participant thinks the researcher wants to see or hear; Arthur et al., 2016; Duckworth & Yeager, 2015; Park & Peterson, 2005); or, when rating others, implicit or personal bias (Peterson, Rubie-Davies, Osborne, & Sibley, 2016).

Given this measurement landscape, some organizations and researchers have attempted to move towards observation measures as a form of character and SEL assessment in educational settings in order to capture what is happening contextually in the classroom or school more broadly. Observation measures focus on classroom or school climate rather than student-level thoughts or behaviors, and often ask an administrator or teams to rate constructs such as teacher-student relationships or elements of classroom management. Holahan and Batey (2019) noted that classroom observations have been effectively used as a means of evaluating and improving adult-student relationships in out-of-school time activities, and that such tools are beneficial because they elevate “the importance of context, the learning environment, and adult-student relationships” (p. 18). To date, several character-oriented classroom observation tools have emerged from out-of-school time research or school-climate focused observation questions that are part of larger teacher assessment frameworks (Holahan & Batey, 2019). For example, the Weikart Center for Youth Program Quality (n.d.) has created an SEL program quality assessment for out-of-school time programs that assesses elements such as creation of safe spaces, scaffolded learning, and growth mindset ethos. Teacher frameworks that include school climate observational elements include the Danielson Framework for Teaching as well as the Marzano Observation Protocol (Yoder, 2014). More school specific SEL observation protocols include The Classroom Assessment Scoring System (CLASS) as well as CASEL’s recent walkthrough protocol. CLASS specifically looks at elements such as the emotional climate of the classroom as well as behavior management (Allen et al., n.d.). CASEL’s SEL walkthrough protocol offers a 4-point rubric that assesses classroom climate and practices (e.g., teacher-student relationships; cultural responsiveness) as well as school-wide climate and practices (e.g., youth voice and engagement; authentic family partnerships; CASEL, 2019). The walkthrough tool is specifically designed to help SEL teams or observers assess areas for improvement and to help schools focus on areas for “feedback and development” (CASEL, 2019, p. 1).

Importantly, classroom observations, too, suffer from drawbacks. Since observations are aggregated at the classroom or school level, this means that their results cannot be disaggregated by student group identities or demographics for either purposes of formative improvement or federal data reporting. Hola- han and Batey (2019) have also noted the importance of using trained observers in order to ensure valid, reliable data, which, they note, can sometimes be a challenging bar to achieve for some school systems. In addition, as noted earlier, it is important to not measure contextual factors alone when measuring character as these cannot be used as proxies for character development; indeed, without a clear theoretical framework connecting contextual influences and character strengths, these kinds of observations can also be problematic (e.g., bias or noise, see Kahneman et al., 2021). Ultimately, triangulation between different forms of measurement—drawing upon several forms of measurement in order to draw patterns of interpretation between them—is one of the primary ways of dealing with the limitations of character measures while simultaneously allowing educators to capture and examine variations in student outcomes (Arthur et al., 2017, p. 112).

Delineating Between Summative Versus Formative Measures. Beyond focusing on context, evidence also suggests that placing learning at the center of assessment practices—through formative assessment and a focus on improving context and classroom practices—is associated with more equitable outcomes for students (Black & Williams, 2010; Kalinec-Craig, 2017; Pryor & Crossouard, 2008; Shepard & Penuel, 2018).

Indeed, although there are numerous types of character and SEL measures and assessments, here we are most concerned with the differences between formative and summative assessments and between high-stakes and low-stakes assessments (McKown, 2019). Summative and high-stakes measures are often longer, tend to come at the end of a defined period of learning (e.g., the end of the semester or school year) and may be used in an evaluative manner either by researchers or school systems to assess students’ character strengths or character development. Formative and low-stakes measures are often shorter, may be used at any time throughout the learning period, are used to guide and improve instruction, and are meant to be authentic to the setting or behaviors in question (Duckworth & Yeager, 2015; McKown, 2017a, 2017b, 2019; Yeager et al., 2013). Importantly, there is no clear delineation or groupings of types of character measurements (e.g., surveys, teacher ratings) by whether or not they are high or low stakes, formative or summative. In addition, the terms high stakes and summative are not necessarily synonymous, nor are low stakes and formative. As such, in this manuscript we are focused on the goals for the use of the assessment rather than type of assessment or whether or not it is summative or formative. For example, McKown (2019) points out that assessing student character negatively on report cards, a summative assessment, could be interpreted as a form of high-stakes assessment should parents feel this could become a long-lasting stain on a students’ permanent record. On the other hand, assessing students’ character negatively on report cards could be used as a form of formative assessment if it is used to help encourage student growth throughout the school year and as a way to guide teachers in thinking about ways to improve instructions for student character growth. That is, the same form of assessment, character report cards, can be used for both high-stakes, summative and low-stakes, formative purposes, depending on the way it is used by the educator.

The improvement science movement in education is an example of the ways in which formative assessment principles are being used to create more equitable character education and SEL learning environments (The Carnegie Foundation, n.d.). Park et al. (2013) have described improvement science as “dis- cern[ing what works for addressing a particular problem, for whom, and under what set of specific conditions” (p. 4). The six core principles of improvement science include: (1) make the problem specific and user centered; (2) address the variation in the problem; (3) see the system producing the outcomes; (4) “we cannot improve at scale what we cannot measure”; (5) engage in inquiry cycles (e.g., Plan, Do, Study, Act cycles) to improve; and (6) use networked communities to help accelerate improvements (Bryk et al., 2010). Continuous improvement then means making these principles of improvement regular and constant within one’s organization, making them deeply interwoven into the daily work of the individuals of the organization, and working with individuals to continually situate their work within systems and contexts (Park et al., 2013). The California CORE educational districts specifically draw on the ideas of continuous improvement, using a nine-step continuous improvement cycle and drawing on networked communities to spread effective practices throughout the districts (Gallagher et al., 2019; Gallagher Cottingham, 2019). The CORE districts began measuring four SEL competencies via student self-report survey in 2016 (growth mindset, self-efficacy, self-management, and social awareness) and additionally measured areas of school climate and belonging via student self-report survey (core- districts.org). However, these CORE measures were used not only as measures of high-stakes accountability under the Every Student Succeeds Act, but, as noted here, also as formative measures to help teachers think about ways to continuously improve their practice. For example, one CORE principal noted that they would look at the survey data “at the end of the year to ‘do a cycle of continuous improvement, saying where did we rank, what can we do to improve participation and such’” (Marsh et al., 2018, p. 57).

Using Summative Versus Formative Measures. Thus, a school’s character and SEL assessment journey should not end with collecting and analyzing data. Collecting and analyzing data is one element of the continuous improvement cycle and should serve to inform conversations about how to improve instruction and school climate and ethos (Annie E. Casey Foundation, 2014; Bryk et al., 2015; Schlund et al., 2020). For example, in their work with educators, Paunesku and Farrington (2020) developed an online survey system where educators can deploy surveys to their students regarding their SEL learning experiences; teachers then receive rapid feedback from the online system regarding their students’ experiences surrounding feeling cared for, receiving feedback, and feeling engaged in their work. The system subsequently recommends appropriate strategies to help the teacher improve their students’ experiences. Once the teacher tries out the new strategies the students can then be later assessed via the same online survey system to see if any changes occur. Thus, Paunesku and Farrington’s (2020) model serves as an example of using continuous improvement strate- gies—via a formative survey assessment and rapid feedback—to help build more equitable classrooms. Carter and colleagues (2017) likewise recommend that data analysis in schools lead to continuous improvement and further conversation related to equity concerns. They noted that data should

Apart from attending to the contextual aspects of the school environment and using data formatively to improve learning and development, other data collection equity concerns arise when measuring character and SEL in schools. For example, assessments often use biased language, or the assessment method itself may have been found to be biased for or against some demographic groups (Jagers et al., 2018). Similarly, teachers may have implicit or explicit biases that emerge when rating or otherwise assessing students (Schlund et al., 2020). Accordingly, it is important to consider power dynamics within a school, classroom, or community, and to take into account who is collecting data from whom and to whom data will be shared (Schlund et al., 2020). In light of this concern, some schools and districts engage students, families, and/or their local communities in the data development and/or collection processes. For example, the Minneapolis Public Schools’ Youth Participatory Evaluation teams, made up of middle and high school students, are involved in making recommendations regarding school climate and student engagement measures in their district (Schlund et al., 2020).

In addition, there are ways to analyze character and SEL data in a more equitable manner. For example, data should be disaggregated by race, gender, disability and other relevant demographics in educational environments in order to help identify which additional resources might be needed for which student groups (Annie E. Casey Foundation, 2014; Carter et al., 2017; Fiarman et al., 2021; National Forum on Education Statistics, 2016). For example, in Washoe County, Nevada, principals are provided with “data books” that guide educators regarding how to interpret data related to school climate, engagement, discipline, and academic achievement “with an equity lens” (Schlund et al., 2020, p. 22). Accordingly, educators in Washoe County are asked to think about their data in a disaggregated manner and to consider whether both their educational strategies and their assessments are being applied in an equitable manner. Another example: over the 2015–2016 and 2016–2017 school years, California’s Oakland School Unified District developed a school performance framework that drew upon an end of year school climate survey and other behavioral indicators (e.g., absences) in order to assess student growth and performance. This data was disaggregated by specific groups (e.g., ELL learners) and schools were given a guide in order to help them use the data to set priorities to lessen the achievement gap and battle inequities in their schools. In disaggregating data, however, it is of utmost importance to caution that disparities are not viewed in discriminatory ways (e.g., seeing a particular subgroup as lacking or less-than) but rather as a recognition of contextual changes that must be made to support these differences. For example, if data show that a certain group of students report a lower sense of belonging, this should not be seen as a deficit of that group of students, but rather as an area that the school or classroom itself must attend to (e.g., Kaler-Jones, 2020; Simmons, 2019; Soutter, 2020; Zaino, 2016).

Like others in the field of character education and social emotional learning, EL Education—a national network of primary and secondary schools—has become increasingly interested in finding and creating measures that assess the impact of their educational model on students’ character change and growth while informing school improvement efforts; to date they have pursued both summative, higher stakes character assessments as well as lower stakes, formative assessments. However, intertwined within EL Education’s character assessment model is a deep commitment to developing equitable measures of character that not only assess the student and their character development, but also the contexts within which EL Education students are studying in order to improve upon them.

Below, we describe EL Education’s character framework followed by its approach to measurement as a way of illustrating the ways in which this organization is using an equity-driven, socioculturally grounded, practitioner-informed approach to character measurement for equity and continuous improvement. We first outline EL Education’s conceptualization of character education and equity, and then describe their character measurement goals. Subsequently, we describe the ways in which EL Education’s approach to measurement is aligned with a commitment to equity through its prioritization of contextual factors and continuous improvement cycles, and a co-construction of measures that values practitioner input.

It is important to note in the sections below that the process embarked upon by EL Education is grounded in the basic principles of measurement design, but was intended for practical use. That is, their primary goal was to compile and create tools that were streamlined and actually usable in schools. So while their process was rigorous, our aim in outlining aspects of their undertaking is not necessarily to focus on the particulars of their psychometric design, but rather to focus on how school leaders might embark on a similar process of creating equitable, formative measurement tools that are actually practical and not prohibitively time consuming.

In addition, it is not necessarily the uniqueness of EL Education’s measures that we aim to highlight here; though we emphasize their use of a walkthrough tool, we recognize (as described above) that other scholars have aimed to similarly take stock of students’ classroom contexts (e.g., CASEL, 2020; Paunesku & Farrington, 2020). Rather it is the explicit commitment to measuring character in such a way that not only prioritizes character as a critical indicator of success, but that aims to do so in an equitable way. EL Education’s efforts to account for systemic and contextual factors, and to use disaggregated data for continuous improvement is a relatively new endeavor, and one that centers equity concerns in a way that so many other efforts do not. Our hope is that practitioners and school leaders alike can learn from this case study when conceptualizing their own approach to equitable character measurement.

Grounded in a belief that the purpose of school is to prepare students not to perform on tests but to be critical thinkers with a strong sense of character, EL Education champions a broader definition of student achievement that prioritizes character as a critical, interlinking dimension. More specifically, successful achievement is defined as a combination of mastery of knowledge and skills, high quality student work, and character (see Figure 1).

In order to paint a clearer picture of what they mean by character, this piece is expanded upon in EL Education’s Character Framework that is grounded in three core dimensions (effective learners, ethical people, contributing to a better world) surrounded by the necessary conditions for these areas to thrive (see Figure 2). As evidenced below, EL Education maintains an explicit focus on equity in each foundational piece of their conceptualization of character, a commitment that remains a driving force of their measurement goals as well.

At its core, EL Education supports students to become effective learners and ethical people who contribute to a better, more equitable world. They define effective learners as those who “do more than they think possible, developing the mindsets and skills for success in college, career, and life” and as being comprised of character strengths such as initiative, responsibility, perseverance, and collaboration. Ethical people are described as those who “treat others with respect and compassion, and stand up for what is right” and who personify character strengths such as empathy, integrity, respect, and compassion. Finally, at the center of the framework is the message that EL Education students work to contribute to a better world “putting their learning to use as active citizens, working for social justice, environmental stewardship, and healthy, equitable communities.” This central goal is inherently focused on equity and the greater good, including character strengths such as civic responsibility and service, civic empowerment, long-term purpose, social justice, and environmental stewardship.

The next layer of EL Education’s framework focuses on fostering a positive student identity by emphasizing belonging, purpose, and agency to create the necessary conditions for students to become effective learners and ethical people who contribute to a better world. Each of these surrounding layers are explicitly focused on ensuring that all students are supported in these ways, and, as illustrated below, when measuring these constructs, equitable outcomes for all students is a central goal.

In order to truly prioritize character as an equally important part of school, EL Education embeds character education through all aspects of the day. Their schools implement a compelling curriculum where teachers use protocols to create opportunities for all voices to be heard, and activities and texts that honor the knowledge, languages, beliefs, and skills from the cultures and backgrounds of students and their families. EL Education also leverages student-engaged assessment practices to promote character and equity, tracking progress toward not only standards-based learning, but toward character targets as well. Students set goals, reflect on growth and challenges, and present evidence of their achievement and growth to a range of audiences. Finally, teachers in EL Education schools engage students in deeper instruction that challenges and empowers students in meaningful work in order to build authentic understanding.

All of the character framework components described above rest on a foundation of crew culture and empowering leadership, which are inextricably linked to EL Education’s character and equity goals. Crew culture is a defining feature of EL schools and is organized around activities designed to facilitate community-building, a sense of belonging, and character development. These activities range from advisory-style check-ins and discussions, to group games designed to build teamwork and foster reflective discussions, to explicit conversations about students’ character. Finally, this kind of environment is bolstered by EL Education’s approach to empowering leadership where school leaders (including teacher-leaders) work collaboratively with families, students, and staff to make evidence-based decisions that enable all students to achieve and that build a schoolwide community of collaboration and trust.

EL Education’s framework serves as a guiding force not only for their approach to teaching, but also for the way they measure success. As noted above, our goal in detailing this particular institution is to highlight two overarching tenets of their measurement approach that we see as critical considerations for the field: their commitments to equity through contextual factors and continuous improvement. Below we detail the ways in which they center these two focal points in thoughtful and impactful ways in the design, development, and implementation of their measurement tools. The organization ultimately uses two forms of measurement (student self-report surveys and school leader administered walkthrough tools); however, we focus primarily on their walkthrough tools as they are an approach that is more directly and intentionally used as a way of addressing contextual considerations and fostering conversations around continuous improvement.

Consistent with their strengths-based, holistic view of character, EL Education wanted to design a measurement tool that could take context into account. Instead of relying on measurement tools that could further entrench an individualistic, immutable vision of character (e.g., where the onus is solely on the student to improve), they pursued a form of measurement that acknowledges character as an ever-changing element of each student’s identities evolving within their sociocultural contexts.

More specifically, when designing measurement tools, EL Education was particularly sensitive to the ways in which character has been conceptualized and measured historically and how that presented significant equity concerns; as detailed earlier, when character is viewed as innate and unchangeable, and measured accordingly through survey tools, then there is no need to create classroom environments in which students’ character can grow and be nurtured. If, on the other hand, character is seen as a set of malleable skills, dispositions, and mindsets that depend in part on contextual factors (e.g., student experiences outside the classroom, their feelings about certain school subjects, their relationships with peers and teachers) as well as developmental opportunities for growth, then educators can take on a strengths-based (vs. deficit-based) approach to teaching and cultivating character.

Aligned with this vision of character, EL Education wanted to look beyond survey tools alone since they can be limited in this regard. As one way to keep contextual influences at the forefront of their vision of character and to design measurement tools that could address equity concerns, EL Education developed a character observation tool, called a walkthrough. Walkthroughs allow observers to take context into account in a way that survey tools typically do not. For example, if one student is having a challenging moment in class, the observer might record not only that child’s behavior, but also the ways in which the rest of the class responded to and supported that student (or, potentially fell short in doing so). In this way, the walkthrough tool attends to the ecosystem of the classroom and school community and allows for the examination of the extent to which students are able to demonstrate their character strengths and support one another’s development as well as the extent to which the teacher is providing meaningful support and opportunities for character development.

Walkthroughs were also specifically designed because they provided information about the extent to which character could be observed across the entire school (e.g., the extent to which teachers were promoting and supporting habits of character during all parts of the school day, and the extent to which students were developing these strengths), and because it positioned school leaders to get into classrooms to be able to provide formative feedback and appropriate supports to their staff around character development.

To construct character walkthroughs, schools first solidified their central character goals. That is, before any measurement tools could be considered, a clear definition of core values was decided upon at each participating school. Leaders collaborated with faculty, students, and families to name and define what habits of character best exemplify the values and skills that they aspire to build in their school community, and how those align with the core EL Education principles. For example (as seen in Table 1), they came to understand the constructs of ‘empathy and perspective taking’ as “the ability to consider the perspectives and share the feelings of others.” However, as is also described in the following section, arriving at the understanding of what these terms mean was an iterative, collaborative process that allowed each school to interpret these constructs to fit their own contexts. In other words, the instrument each school uses is the same, but the interpretation of each character strength and how they will be rated may vary across schools.

EL Education then conducted a review of currently existing and robust observation tools and adult reports (e.g., teacher observations). Existing observation tools often require users to observe for long periods of time, require retrospective judgment, or cannot be used across a variety of contexts; in contrast, the EL Education walkthrough is meant to be quick (typically about 7–10 minutes), frequent, and should allow for many observations in many classes over the course of a day across a variety of contexts (e.g., not just in the classrooms, but in the cafeteria, in the hallways at the end of the day, in physical education, art, etc.) and developmental stages. In light of these constraints, EL Education referenced existing observation instruments, but also weighed what behaviors could feasibly and consistently be observed in school contexts within a short time window. Ultimately, they designed walkthrough tools aligned with two of EL Education’s central tenets: becoming effective learners and becoming ethical people. (The third central tenet, contribute to a better world, is embodied by character strengths that are more difficult to capture in discrete behaviors in the classroom and so is currently only measured through survey items, and is more holistically on display during their yearly “Better World Day”). Table 1 shows a sample of two of the constructs that are measured through the walkthrough tool that are aligned with becoming ethical people, and Figure 3 shows the corresponding walkthrough tools.

One of the ways that EL Education evaluated the ability of these walkthrough tools to consider contextual factors was through a collaborative process with several EL Education schools: EL Education staff provided feedback on initial drafts for face validity (meaning, did the tool actually measure the character strengths it was aiming to capture), clarity, and the ability to complete the walkthroughs in a short time window. Subsequently the tools were user-tested in two pilot schools with the respective school leadership teams who identified that the indicators needed to be better clarified and that some indicators were difficult to observe consistently or that they varied developmentally. For example, regarding the crew walkthrough (looking at an EL Education school structure focused on weekly meetings between groups of students and an educator), educators noted that terms such as “circling up” and greeting structures, as well as language surrounding other particular rituals, traditions, and crew norms were missing from the initial walkthrough. In addition, when first testing out the effective learners walkthrough, educators at Amana Academy pushed EL Education to continue to clarify their terms. For instance, one section asked educators to assess whether students are sticking “to a tough problem, even when confronted with mistakes, uncertainty, or challenge”; the Amana Academy educators wondered in their piloting notes “What is a tough problem?” In other words, actually having the opportunity to test this tool allowed teachers to see which character constructs were perhaps more difficult to identify or measure, which then allowed researchers to reconsider how items were presented. After another round of revisions, the walkthrough tool was again piloted in eight additional schools and feedback was received via survey and through school team documents, which informed a final round of revisions. For example, in order to further clarify for educators what they should look for regarding student persistence through difficulty, the language “students vocalizing frustration or difficulty, but staying engaged and on-task” was added to the walkthrough tool.

Another example of calibration and practitioner feedback on contextual factors occurred while piloting the walkthrough tool in an elementary classroom. During the walkthrough, observers noted that the majority of the students were sitting on the carpet analyzing a poem together. After they left the classroom, the group circled up to debrief and calibrate on their ratings for whether the instruction was engaging, challenging, and empowering in inclusive and equitable ways. Of the five observers who engaged in the walkthrough, three noted that equitable participation was not evident, but two noted that it was evident. The team then went around to explain their evidence ratings. The three observers that noted “not evident” explained that they noticed three African American boys sitting in chairs apart from the rest of the crew who were not engaged in any meaningful learning activity or paying attention to what the rest of the group was doing. “Why were they separated, and why weren’t they engaged in any kind of learning activity?” one observer asked. The school leaders noted that the three boys were not engaged in the whole group because they were waiting for a specialist to come pick them up for scheduled pull-out time. The ensuing discussion was especially productive. The team raised important questions about how the boys might experience the waiting while class was going on in front of them. Was there a missed opportunity for inclusion in meaningful learning during this wait time? How might this have felt for the children, who were physically separated and disengaged from the rest of the group? And what alternatives might be introduced by the teacher and instructional guide that can ensure that students who are waiting for pull-outs are still supported and included physically, socially, and intellectually as part of the larger group?

While this particular conversation did not serve to recalibrate the instrument itself, it did serve as an example of one of the ways that the walkthrough could be used to recognize contextual disparities and then have meaningful discussions on the roots of these inequalities and what changes could be made to address these. What is important about these conversations as an outcome of the walkthrough instrument is that the tool itself acts as a catalyst for recognizing, identifying, reflecting on, and enacting change.

In addition, EL Education school leaders received training on recognizing issues of equity. Educators trained to engage in the walkthroughs were urged to consider questions such as, “who is the teacher calling on?,” “If students are doing group work, is every child meaningfully involved?”, and to “pay attention to how [students] are given access to the content and curriculum, how their specific language needs are met, and perhaps most importantly, whether they speak at all” (Aguilar, 2015).

These kinds of discussions not only centered equitable support for all students, but they also allowed teachers to further solidify the nuances of inclusion and belonging and how it was envisioned at their school. It is also critically important to note that these walkthrough tools are designed to be ways of starting conversations about character development in the classroom, not as high-stakes evaluations for the students or for the teachers themselves. This emphasis on formative measurement for continuous improvement is expanded upon in greater detail below.

EL Education’s measures were designed not only to demonstrate proof of impact, but also to be used formatively in the classroom to better understand and track character growth in order to drive instructional change. That is, instead of using measurement tools as a high-stakes evaluation of students’ character, these tools are meant to be used as a way to support a reflective process where teachers and students alike are considering how they can continuously grow and improve as individuals and as a community. Below we describe the ways in which EL Education ultimately uses these measurement tools in practical, data-driven ways that center equity in their application and interpretation.

First, EL Education uses the walkthrough tool as an opportunity for staff and students to further solidify a shared vision of what character looks like in their schools. For example, similar to the conversations that ensued when providing feedback on the walkthrough tool, when calibrating on a walkthrough indicator (for example, an indicator might ask observers to look for examples of students being supported in being persistent), faculty can unpack “what does this look like and sound like for us?” This vision-setting can be supportive for buy-in to both the tool and the data it produces, help set high expectations for teacher practice and student outcomes, and concretize the expectations for what every classroom should feel like for every student. For example, Cher- isse Campbell, principal of Amana Academy in Georgia, described performing a walkthrough in her school community as offering an opportunity to inform her staff of “who we are or where we are, right now, and this is how we can support each other going forward.” She noted that walkthroughs are a particularly powerful tool for her educators because they offer a glimpse into the contextual supports in the school community that can be further leveraged to help promote positive student outcomes. She commented:

After analyzing data, EL Education educators are asked to engage in inquiry to create, test, and measure the impact of a new idea. Based on their root cause analyses, teachers engage in an inquiry cycle and decide on a new change that they are going to test out in the educational setting in a frequent and structured format. Their inquiry teams then meet frequently in order to discuss how this new change is functioning and whether changes should be made or if the impact is positive and as expected; the change is then continued, revised, or other new practices are implemented. For example, in an inquiry meeting, Cherisse Campbell at Amana Academy and her team of teachers were able to discuss data from a recent walkthrough and discuss possible pedagogical implications:

Thus, through looking at the walkthrough data, the principal and her team were able to pinpoint the area of reflections as an area for growth, as well as a strategy to better support their faculty to plan for high-leverage strategies such as debrief conversations. Similarly, Laina Cox, principal at Capital City in Washington, DC, described the process of examining data as, “It is just what do you notice about the data? Where is it working better and how can we then leverage that across other grade levels?”

In the end, change is then spread by other schools and networks adhering to these new ideas and improvements brought about through these plan-do-study-act cycles. As part of their 2017 study on crew belonging, EL Education created a spring convening so that each participating school could share their own poster and Powerpoint presentation regarding what they learned from their data, challenges, and their new plans based on their plan-do-study-act cycle. EL Education has continued to host webinars for their network to help build their educators ability to apply new ideas and practice in the learning context. For example, as part of their work with the Raikes Foundation and their Building Equitable Learning Environments study during the 2018–2019 school year, EL Education hosted webinars such as “using survey data with students,” “equitable teaching practices,” and “belonging, identity, and community.” Moreover, they developed professional learning communities across their network, with each school forming a pair with another participating network school. The aim of such professional learning communities is for each to be able to learn from others’ practice, engage in shared problems of practice, and exchange improvement ideas and structures.

In addition to taking into account practitioner feedback as a part of their commitment to continuous improvement, EL Education also incorporated student participation and feedback as part of their development process. While student voice was more a part of the development of survey measures, we include it here as an example of the ways in which students were included in a meaningful way.

First, in addition to assessing the appropriate age range for each survey by checking lex- ile levels for each survey, students in Grades 4–7 were also interviewed. The purpose of these interviews was not only to see if the survey levels were developmentally appropriate, but also to see if they accurately captured students’ contextualized, holistic character strengths. When these kinds of contextual, socio-cultural considerations are not prioritized and asked about, survey outcome measures can be misinterpreted, often with subgroups scoring lower because they are being compared to White, Eurocentric expectations of behavior and character (Kaler-Jones, 2020; Simmons, 2019; Soutter, 2020; Zaino, 2016).

These interviews consisted of students being read aloud the survey items and asked what each one meant to them, resulting in important considerations for survey dissemination. For example, one of the original Ethical People survey items read, “I try to understand rather than judge people who are strangers to me,” and only 41.6% of students were accurately able to interpret this item: one fourth grade student incorrectly described this item as “Judging someone means to tell them what to do and you are trying to tell them to do the right thing” whereas a seventh-grade student correctly noted “I try to be considerate and not think bad thoughts if I don't know the person.” Similarly, one of the questions in the Contributing to a Better World survey, “It is important to fight against social and economic inequality,” was not able to be accurately interpreted by any of the students interviewed: one Grade 5 student, for example, did not know the terms economic or inequality and described it as “Like if people are giving you a hard time” and a sixth-grade student similarly struggled with the term economic and said that inequality meant “to try hard.” Given such findings, EL Education currently recommends that their surveys are most appropriate for students in the sixth-grade and above, and that school staff should review key terms and concepts with students prior to distributing the survey.

These student interviews reinforce the tension between psychometric and practitioner approaches to measurement. Despite passing psychometric reliability analyses with datasets including younger students, it was clear upon closer inspection that statistical methods alone do not sufficiently address whether measures are developmentally appropriate for linguistically, developmentally, culturally, and academically diverse groups. This round of piloting, coupled with the solicitation of student feedback, signals a true valuing of student comprehension and perspectives, cultural considerations, and an active commitment to equity and student voice.

During piloting, teachers also asked for more demographic breakdowns by schools, grades, classes, and subjects in order to be able to analyze the data more precisely. The purpose of gathering these demographics was of course not to reinforce stereotyped assumptions about subgroup behaviors, but instead to see where the teachers themselves and the school environment could adapt to better support students’ needs. For example, if a certain group was reporting a lower sense of belonging, this would be important data for schools to reflect upon to see what steps need to be taken to foster increased inclusion and trust to bolster a true sense of belonging. EL Education ultimately designed a data system that allows them to disaggregate character measurement data by subgroups in order to evaluate whether their approach was serving all students equally well, and, if not, to make adjustments accordingly. This database has been a key piece in allowing for the kind of continuous improvement that is central to its primary goals of character measurement as described above. For example, in a recent investigation into the impact of Crew on equity in EL Education schools, they were able to use disaggregated data to track an increase in sense of belonging for students of color; while there continues to be much work to be done to ensure that all students feel a strong sense of community in schools, these data provided useful information to begin tracking their own progress (EL Education, 2018).

The question of character measurement is not just a question of how to do it accurately (though this is certainly an endlessly complex question rife with challenges as outlined above), but how to hold ourselves accountable as educators to ensure that character and equity are centered in our work. Indeed, measuring character is not something that should be taken on lightly or without a firm understanding of the complexity of the ways in which the benefits, drawbacks, and contextual factors intersect. We must enter this work with a clear recognition of the ways in which measurement can be misinterpreted if not conceptualized within a sociocultural, race-conscious, equity-based lens (Kaler-Jones, 2020; Simmons, 2019; Soutter, 2020; Zaino, 2016).

Our goal in highlighting this case study of EL Education is not to lift up one example of perfect measurement, but rather to illustrate the ways in which they have approached this feat through an intentional, collaborative process that prioritized equity, contextual considerations, and continuous improvement at every turn. Rather than seeing character measurement as a summative, high stakes way of getting students to conform to benchmarks grounded in White, Eurocentric moral and behavioral expectations, EL Education’s approach supports teachers in seeing their individual students’ strengths, to examine how their classroom community can shift to better support them, and to truly hold character as a critical, interlinking piece of student achievement. Certainly, their approach does not encompass all equity considerations nor does it answer the many outstanding questions and critiques related to character measurement as a field. Indeed, schools and school leaders must be painfully careful to ensure that character measurement does not become weaponized or used to idealize a singular way to behave, act, or feel.

However, we also must recognize that if we continue to measure academic outcomes alone, then these become our sole indicators of success. If we as educators truly care about centering holistic student well-being, we must critically examine our pedagogy, ask ourselves difficult questions, engage in inquiry cycles, reflect on our successes and failures, and work collectively to hold ourselves accountable to centering character in just, holistic, equitable ways.