An Evaluation of the Developmental Designs Approach and Professional Development Model on Classroom Management in 22 Middle Schools in a Large, Midwestern School District

Developmental Designs provides two levels of professional development over a 2-year period of time. Known as DD1 and DD2, the approach prepares teachers and school administrators to use positive, proactive student behavior management techniques in classrooms and schools. DD1 is a year-long beginning level approach that provides program understanding and initial skill acquisition; DD2 provides more in-depth training during a second year after educators have had an opportunity to implement basic program strategies for a full academic school year. While both levels of training may or may not include classroom-based “coaching” as a value-added professional development component, (when incorporated in tandem) both DD1 and DD2 have been found to be integral components of ongoing, site-based, job-embedded professional development for middle grades teachers. Throughout the 2-year preparation and implementation phases, DD1 and DD2 prepare educators in young adolescent behavior management approaches designed to promote self monitoring student behaviors and positive character traits that foster self respect and consideration of others’ rights and privileges.

To achieve this overarching goal, classroom teachers, school administrators, and support personnel apply a number of equal and equitable reinforcements while also modeling appropriate behaviors and enforcement approaches. As a result, it is expected that positive character traits will, in turn, be linked to positive student outcomes including but not limited to improved academic achievement. In efforts to facilitate key features of the DD1 and DD2 approaches a third level, DD3, has recently been developed, as well. This latter phase is not addressed as part of this evaluation.

Prior to the beginning of the 2009-2010 academic school year, more than 1,400 teachers in over 200 schools across 25 states had received DD1 and/or DD2 training. All professional development was provided during week-long workshops or institutes held during summer months and facilitated by expert practioners carefully recruited and prepared in the Development Designs approach. While some institutes were comprised of educators from different schools across different districts from throughout the United States, many workshops focusing on school wide reform were developed specifically for individual schools within the same district. This latter approach was taken during the summer of 2009 in a large, urban Midwestern school district and is the basis for this study.

Four questions guided this program evaluation: (1) How do teachers perceive the quality of DD1 and DD2 professional development training provided by Development Designs facilitators during summer workshops held in July-August 2009? (2) How confident are teachers in implementing the various young adolescent behavior management strategies presented during the 1-week summer workshops they attended? (3) How extensively are teachers implementing DD1 and DD2 strategies in their classrooms 6 months after having completed the summer workshops? (4) What is the relationship between DD1 and DD2 program training and classroom implementation and student attendance, behavior, and academic achievement as measured by state assessments.

Eleven different Development Designs professional development facilitators conducted 15 five-day professional development workshops held July 6-10, July 13-17, and August 3-7, 2009, in the school district studied. During this time facilitators demonstrated as many as 17 DD1 and 12 DD2 middle grades behavior management strategies. A total of 360 educators, mostly teachers, participated in the weeklong workshops—317 in DD1 and 43 in DD2. On the last day of each summer workshop, Developmental Designs professional development facilitators administered a survey questionnaire that was completed by all participants in attendance. These questionnaires were returned to facilitators who submitted them to Origins staff. Hard copy questionnaires were provided by Origins to the external evaluator who entered all data into SPSS 17.0 for analysis.

In January-February 2010, school district personnel administered follow-up program implementation questionnaires to the teachers, counselors, principals, and others who had completed the DD1 and DD2 training in the summer 2009. These questionnaires were designed to generate perceptual data pertaining to the extent to which teachers, in particular, had been implementing the various strategies addressed in the summer workshops, as well as the degree of confidence teachers have with regard to implementing these strategies. Both of these data sources were imported into SPSS 17.0 along with the summer workshop questionnaires for analyses.

Table 1 shows the number of teachers in each middle grades school, the number of DD1 and DD2 participants by school, the number of students enrolled at each school during the 2008-2009 academic school year, the percent of teachers trained at each school, and student-to-trained-teacher ratios. Because the range of teacher participation per school is from 1 to 45, with 15 or fewer participants in 17 schools / sites, comparisons are not made between schools. Also, because the number of DD1 and DD2 participants, n = 317 and n = 43, respectively, varies greatly, comparisons between the two groups are made only by random selection of 30 participants from each group. Comparisons between the two levels of professional development serve to exemplify what subsequent analyses might entail over time as levels and degrees of DD1 and DD2 implementation increase along with the number of teachers trained at each school.

Most all middle grades schools are organized with sixth, seventh, and eighth grade spans; exceptions include one school with Grades 4-8, and one with Grades 6-12. One alternative school that serves students who enter and exit intermittently throughout the academic school year was not included in the evaluation. Two schools with students in Grades K-12 and 6-12 had no teachers who completed DD1 and/or DD2 professional development workshops in the summer 2009 were not included in the study.

Six middle grades schools had a significant number of teachers trained in DD1, with a 30 (or fewer) to 1 ratio of students per DD1 trained teacher schoolwide. This represents more than a third of all teachers in each school and two thirds of all teachers in three of the schools among this group. One school with 80% of their teachers trained (i.e., 44 of its 55 teachers having participated in DD1 and/or DD2) produced a ratio of 22 students-per-trained-teacher. A second school’s participation rate was 71% with 45 of its 63 teachers trained, representing a 23 to 1 student-to-trained-teacher ratio; and a third’s participation rate was 68% with 26 of its 38 teachers trained, producing an 18 to 1 student-to-trained-teacher ratio. Subsequent years’ analyses should include these baseline data when examining the extent to which the rate of students-to-trained-teachers and/or the percent of trained teachers per school are examined in relationship to student outcomes.

Table 2 provides data reported by school district personnel who participated in the summer 2009 workshops and includes their role as an educator, gender, highest level of education, number of years as a professional educator, number of years these educators have been at their current school, and the number of years they have been in their current positions. While all DD1 and DD2 participants’ data are aggregated (n = 360), this total number represents 317 DD1 participants and 43 DD2 participants, among which 318 (88.3%) are teachers. Just over 3% (n = 11) reported being principals, .6% as being counselors (n = 2), and 8.1% (n = 29) as being something other than a teacher, principal, or counselor, 9 of these individuals reported their role to be that of an assistant principal, making the total number of school administrators who participated (n = 20) representative of 5.5% of the total. Most others reported their role to be that of “resource specialists.”

Just over 78% of the summer 2009 participants (n = 282) are female, and 288 (80%) hold master’s degree or master’s plus additional college hours. Participants’ years in education range from 1 to over 31 with most 52.5% (n = 189) having 10 or fewer and 10.3% having 26 or more years’ experience in education. A greater number (n = 311) 86.9% have been at their current school for 10 or fewer years with just over 13% having been at their current school for more than 10 years. Similar numbers are evident in terms of the number of years summer 2009 participants had been in their current positions at their schools with 322 (89.9%) having been in those positions for 10 or fewer years and 10% reporting having been in their current positions for 11 or more years.

Table 3 displays data pertaining to summer 2009 participants’ grade levels and subjects taught, indicating that most participants teach sixth grade (n = 105) 29.2%, and most teach language arts (n = 70) 20.4%. If one adds the number of participants who teach sixth, seventh, or eighth (n = 224) to the number of participants who teach all three grade levels (n = 73), this represents 82.5% of all participants. Approximately two thirds of the participants teach one of the core subjects; for example, if one adds the number of participants who teach language arts, mathematics, science, and social studies (n = 227), this represents 66.2% of the total.

Using descriptive data from Tables 1, 2, and 3 it is possible to create a profile of the most common type of school educator who completed Developmental Designs training during the summer of 2009: a female teacher with less than 10 years’ experience in education who holds a master’s degree and has been teaching a core subject to sixth-eighth grade students for fewer than 5 years at her current school. Using discriminate analysis statistical techniques it is possible to accurately identify such a participant 81% of the time. It should be noted that technical presentation of discriminant analysis techniques are not included in this report, but may be detailed in subsequent analyses both to describe and predict teacher characteristics and student outcomes, respectively, should the evaluation remain ongoing.

DD1 practices and strategies are arranged in descending order from highest to lowest combined group means in Table 4. While group sizes make statistical comparisons between the 317 school district educators who completed DD1 training and the 43 educators who completed DD2 training spurious, one can readily see from the data in Table 4 that DD2 participants are more confident implementing DD1 practices and strategies on each of the 14 DD1 practices and strategies, even though these were not the subject of direct focus during DD2 training. Both groups were most confident implementing the “redirecting for small things” M = 4.24 for DD1 participants and M = 4.40 for DD2 participants.

To create a balanced design from which an appropriate comparison can be made, 30 DD1 and 30 DD2 teacher participants were selected at random using a table of random numbers. An independent samples t test was then used to examine differences between the two groups with regard to their levels of confidence in implementing the 14 DD1 practices and strategies listed in Table 4. In lieu of the “All DD1 combined” variable, a scaled score mean was developed for each group by combining all 14 practices and strategies. This procedure produced a confidence mean of M = 3.81, s = .69 for the DD1 group and M = 4.29, s = .65 for the DD2 group; this (difference = .48) was found to be significant, t (53) = 4.96, p < .05, with an effect size of .51, using Cohen’s d, indicating a relatively large effect.

Considering DD1 participants were expressing their degree of confidence in implementing strategies and practices that they just learned during a 1-week workshop and had yet to use them in actual classrooms settings, the range in means of 4.24 to 3.87 is exceeding high, given the 5-point Likert-type level of confidence scale used in the questionnaire. A review of the frequency distributions of responses indicates that on average over 84% of DDMS1 participants rated their level of confidence a “4” or “5.” Even the two lowest ratings, “Student need for autonomy” and “All DD1 strategies and practices combined” (M = 3.87) for each, is well above the scale’s midpoint of 2.50, and 7 of the 14 items have means greater than 4.00.

Table 5 provides summary data regarding a number of questions pertaining to the value of the DD1 and DD2 professional development workshops, perceptions regarding support that has been or might be provided through coaching, and the degree to which participants were involved in follow-up professional development at their schools. While the 317 DD1 participants would not have yet had any opportunities to participate in DD1 coaching or professional development follow-up at their schools, the 43 DD2 participants who had completed DD1 training the prior year noted that almost 50% (n = 22) had participated in school-based DD1 professional development the prior year on average 6 times (ranging from 2 to 12 times). In addition, 33 DD2 participants noted they had received coaching with an average of 5 classroom visits by a coach during the 2008-2009 academic school year (with a range of 3 to 9 visits). Five respondents reported having received coaching from Development Designs facilitators, 19 reported having received coaching from school district staff, and 7 reported having received coaching from both Developmental Designs facilitators and school district personnel.

After having completed one of the week-long summer 2009 DD1 or DD2 professional development workshops, the 360 school district participants had an opportunity to implement strategies and practices during the first half of the 2009-2010 academic school year, roughly between August 2009 and January 2010. In January-February 2010, school district central office personnel e-mailed DD1 and DD2 follow-up implementation questionnaires to the summer participants. Because the follow-up questionnaires varied slightly with regard to the descriptions of some Development Designs practices and strategies, a few inconsistencies between the two are evident throughout Tables 9, 10, and 11.

Of the 360 original participants, school district personnel identified 334 DD1 and 41 DD2 participants to whom follow-up implementation questionnaires were e-mailed. Completed questionnaires were returned by 229 DD1 participants and 25 DD2 participants, representing return rates of 69% and 61%, respectively, when based upon the 334 participants identified. Overall, the total number of spring 2010 respondents (n = 254) represents a 71% response rate when based upon the 360 participants from the summer 2009 workshops and 76% when based upon the 334 participants identified by school district personnel. How ever, it should be noted that in six schools, there were more spring 2010 respondents than the number of school personnel who had attended the summer 2009 DD1 and DD2 professional development workshops. School district officials acknowledge that in some cases, individuals may have forwarded the link to others (nonparticipants) and that at least one school may have sent the link to the entire staff. Table 6 shows individual response rates by school based upon the actual number of summer 2009 DD1 and DD2 workshop participants (360). Again, this overall response rate of 71% is based on no elimination of data. Because all questionnaire responses are anonymous, it would be impractical to attempt to determine which questionnaires should not be included.

Table 7 shows a number of descriptive characteristics of respondents: role in school, gender, highest level of education, number of years as a professional educator, number of years in their current school, and number of years in their current position. In general, spring 2010 respondent characteristics are similar to those of the summer 2009 workshop participants. Most are female (81%) teachers (86%) who hold a masters or higher degree (85%) and have been in their current position and at their current school for fewer than 5 years (74%) and (64%), respectively, and have been a professional educator for 15 years or fewer (65%).

Likewise, Table 8 indicates that spring 2010 respondents’ teaching assignments are similar to those reported by the summer 2009 participants in terms of grade levels and subjects they teach. Approximately 88% of spring respondents teach Grade 6, 7, or 8 or 6-8, as do approximately 89% of the summer 2009 participants. The “core” subjects, that is, math, science, language arts, and social studies are taught by 65% of the summer participants and 55% of the spring respondents. Part of this difference may be due to the manner in which the questions were asked on the respective survey questionnaires, however.

Table 9 compares means and standard deviations, along with mean differences and effect sizes of DD1 means between the summer 2009 DD1 participants and the spring DD1 respondents. Negative mean differences reflect lower confidence levels in the spring after teachers had approximately 5 months to implement the DD1 practices and strategies they learned during the summer 2009 workshops. In all but one of the 12 categories of practices and strategies, “circle of power and respect,” common to both questionnaires (and including “all DD1 practices and strategies combined”), teachers’ confidence levels decreased. Because utilizing t test statistical approaches to determine statistical significance for each difference in levels of confidence in implementing the practices and strategies would artificially inflate the significance levels, an effect size was calculated to determine the degree of statistical “power” among the different variables. As seen in Table 9, Cohen’s d effect sizes ranged from .03 to .77. (The following scale was used for interpreting the relative strength of each effect size: 0.0-.20 = a small effect size; .21-.50 = a medium effect size; .51+ = a large effect size.) Using this scale, one effect size is large, “plan -work-reflect (PWR)” d = .77, and eight are medium effect sizes, indicating that the differences between DD1 participants’ confidence levels from summer to spring are generally far enough apart to warrant attention.

Table 10 displays the means and standard deviations of DD1 and DD2 spring 2010 respondents’ levels of implementation of DD1 practices and strategies in order of highest to lowest level of implementation by DD1 respondents. It should be noted that the second group of 12 practices and strategies are specific to DD2 only and as such do not apply to DD1 respondents and also have no prior basis for comparison, given that DD2 respondents would be implementing these for the first time.

Again, caution must be taken when attempting to make comparisons between the groups because of the size differences; however, both groups rate “circle of power and respect” as the practice they implement most: DD1 respondents M = 4.06, s = 1.30 and DD2 respondents M = 4.83, s = .64; with a mean difference of .77. DD2 respondents rate their levels of implementation higher on each of the 15 DD1 practices and strategies than do DD1 respondents. These mean differences range from .21 to 1.25, with 12 mean differences at .50 or higher. DD1 respondents’ implementation means also include much greater variability than do DD2 respondents’.

Table 11 depicts the degrees of confidence in implementing both DD1 and DD2 practices and strategies for the DD2 summer participants and spring 2010 respondents. Confidence levels increased in all but one area, i.e., the DDMS1 practice identified as “Plan-Work-Reflect (PWR).” Confidence in utilizing the “Buddy Room” and implementing the “Circle of Power and Respect,” were the two DD1 strategies where confidence rose the most with mean differences of .55 and .54, respectively. The greatest gain in confidence among the DD2 practices and strategies is in the area of utilizing “Social interaction” with a summer 2009 mean of 4.03 and a spring 2010 mean of 4.57, producing a mean difference of .54. Similar findings can be seen for “Exploratory modeling,” and “Reflection.”

Table 12 provides summary data regarding a number of questions pertaining to the value of the DD1 and DD2 professional development workshops, perceptions regarding support that has been or might be provided through coaching, and the degree to which participants were involved in follow-up professional development at their schools. The 25 DD2 participant respondents would have had an opportunity to participate in DD1 and or DD2 coaching as well as professional development at their schools over an 18-month period of time, since first having completed DD1 workshops in the summer of 2008. Of these 25 DD2 participant respondents, 22 had participated in school-based DD1 professional development, on average, 4+ times. In addition, 18 had participated in DD2 school-based professional development, on average, 4-times between August 2009 and January 2010. Nine indicated they would like support implementing DD1 via coaching and 18 indicated they had support implementing DD2 via coaching. Among those who did participate in coaching, DD1 coaches visited respondents’ classrooms, on average, 7 times and DD2 coaches visited, on average, 2+ times. For DD1 participant respondents 161 (%) reported having participated in DD1 school-based professional development, on average, 3+ times, and 33 (%) reported having received coaching with the average number of coaches’ visits being 3+ times.

DD2 respondents rated the value of their DD2 training relatively high (M = 4.46) s = .88 in the spring 2010 and (M = 4.47) s = .52 back in the summer 2009. DD1 respondents rated the value of their DD1 training in the summer 2009 higher (M = 4.38) s = .63 immediately after the training than 5-6 months later in the spring 2010 (M = 4.02), s = .92.

While data collected in the summer 2009 and spring 2010 via survey questionnaires completed by school district educators [DD1 (n = 317) & DD2 (n = 43) = 360] and [DD1 (n = 229) & DD2 (n = 25) = 254], respectively, have been analyzed and presented in this study as aggregated summary data, ultimate usefulness can only be fully determined when built upon over time as additional information is obtained. Hence, this study serves as a baseline to which future data should be added in subsequent years, allowing for more sophisticated analyses to be conducted that can address a number of research questions pertaining to the relationship of the Developmental Designs approach to teaching and learning outcomes. For instance, it should be of interest in the future to examine student outcomes for those taught by DD1-and DD2-trained teachers compared to students taught by others to examine the following questions: Is there a “threshold” level or percent of DD1-and/or DD2-trained teachers that is associated with given outcomes such as proficiency on state assessments, attendance, and behavior? What is the value-added nature of ongoing professional development via DD2 and DD3? Findings related to these “etc.” questions could impact future policy and practice.

The school district educators participating in this study [teachers (n = 318), principals and assistant principals (n = 20), and support personnel, i.e., counselors, resource teachers, etc. (n = 22)] who took part in the DD1 and DD2 professional development workshops during the summer 2009 reported a high level of satisfaction for the training they received as well as high levels of confidence in implementing DD1 strategies and practices about which they learned. Even so, DD1 participants’ confidence levels decreased from the summer 2009 to spring 2010 while DD2 participants’ confidence levels increased over the same period of time. Because most all ratings from teacher participants are relatively high, it would be of value to determine the specific reasons. One might assume that DD1 is of only limited benefit until the second year of DD2 training and implementation. If the high level of satisfaction is due, in part, to extended practical application of strategies and practices that can be used in proactive ways in classrooms to improve student behavior and learning out comes while instilling positive character traits, then policies supporting continual implementation should be addressed.

Clearly, teachers found their DD1 and DD2 training to be of great value and left their respective summer workshops feeling confident they could implement what they had learned in their classrooms throughout the 2009-2010 academic school year. Levels of implementation were high for DD1 and DD2 but noticeably more for the DD2 participants. As one might expect, confidence and implementation were generally related; however, only after DD2 training and a second year of implementation did DD1 confidence increase. Both DD1 and DD2 participants viewed follow-up professional development as potentially helpful and recognized coaching as a key component that could assist. It is fair to say, then, that the DD1 and DD2 summer 2009 training provided a confident beginning for teachers to initiate implementation in the fall 2009 but that follow-up participation in DD2 and a second year’s implementation is necessary to increase both confidence and implementation.

Data regarding the value of ongoing, job-embedded, site-based professional development are limited in this initial phase of the evaluation process; however, additional data related to the value of follow-up DD2 training and coaching are included in the addendum to this initial report. Those data, in tandem with findings outlined herein, might be useful in determining how best to support long-term implementation of the Developmental Designs approach to teaching and learning.