ENSURING TEACHER EDUCATION PROGRAM SUCCESS THROUGH FORMATIVE ASSESSMENTS: An Overview of the Wisconsin K-12 Energy Education Program Free

KEEP began with a three-phase program design:

• Phase 1 (1995-97) entailed developing a conceptual framework that identified important energy concepts energy literate citizens should know and understand and creating an activity guide that included teaching strategies to actualize these energy concepts in K-12 classrooms.

• Phase 2 (1997-99) involved designing and offering a university accredited course that disseminated the activity guide to teachers throughout Wisconsin.

• Phase 3 (1999-present) focused on evolving KEEP into a sustainable capacity-building program.

Since 1998, KEEP has been in Phase 3 and currently strives to promote energy literacy in Wisconsin through five program areas: professional development, resources, networking and outreach, student involvement, and helping schools secure funds for special projects in energy education. Each phase of KEEP has had various assessments which have helped steer the program’s design and have encouraged stakeholders to contribute to the success of the program. The timeline in Figure 1 summarizes the assessments which are described below.

Steering committees and advisory committees have played a critical role in all three phases of KEEP, monitoring KEEP’s progress and reviewing KEEP’s products. Comments, suggestions, and recommendations are recorded and used to revise educational materials and to assess the effectiveness of completed products. The committees consist of representatives from a variety of stakeholders, including utilities, state education agencies, energy resource managers, teachers and teacher educators.

A team of teacher educators within the Wisconsin Center for Environmental Education monitored the first two phases of KEEP. These educators were also full time faculty with the College of Natural Resources at the University of Wisconsin-Stevens Point. KEEP staff consisted of a curriculum developer who was experienced in developing national curriculum, a resource specialist with an electrical engineering background, and an office manager who was a trained environmental educator.

The instruments developed for internal reviews and for formative assessments were designed by KEEP staff based on literature reviews and consultations with other teacher education programs. The primary objective of the instruments was to gain relevant information needed to develop and revise the KEEP instructional materials and education services. The accuracy and usefulness of the instruments have been substantiated by reviews of the teacher education team, research projects (Ziolkowski, 2007), and the researchers at the Energy Center of Wisconsin.

Focus groups were used to identify concepts for the framework and to review drafts of the framework. Two different groups composed of representatives from a variety of energy and educational fields met separately to provide input into the conceptual framework development process. Both groups contained steering committee members. The meetings took place in early 1996. For the next seven months, KEEP staff worked on creating a draft of the framework. The draft was sent to both focus groups to review; they were provided with a review form on which to record their comments. While group members were welcome to write comments directly on the document, the review form aided KEEP staff in compiling the comments and revising the framework. Steering committee members were requested to complete a comprehensive review of the final draft to ensure the comprehensiveness, thoroughness, and accuracy of the concepts and educational theories included in the KEEP conceptual framework. They too were provided with a review form to document their comments. The comprehensive review of the conceptual framework was used to make final changes before publication near the end of 1996.

Also in 1996, teachers were invited to the WCEE Resources Library to review activities and lessons from existing resources and help determine how these activities related to the conceptual framework being developed. KEEP staff created a resource review form and an activity review form for teachers to record their comments. Teachers helped determine the best fit for activities and learning experi ences for their respective grade levels and subject areas.

To ensure conceptual accuracy of the materials, professionals in energy-related fields were contacted in 1996 to conduct a content review of the activities. KEEP staff created a content review form to assist reviewers in organizing and recording their comments and suggestions. The results of the reviews were used to revise the activities.

The lessons were reviewed or piloted to ensure that the objectives of the activity were achieved and that they were appropriate for the grade levels indicated. An activity pilot evaluation form was created for teachers to record their observations and reviews. Based on their comments and experiences, the activities were revised again as needed.

During the spring of 1996 the steering committee as well as the energy education committees from the Energy Center of Wisconsin and the Wisconsin Utility Association received a draft of the activity guide. They used a comprehensive review form created by KEEP to record their comments and suggestions regarding the overall cohesiveness and projected effectiveness of the activity guide.

During the 1997 pilot year of administering the activity guide through KEEP’s teacher education course, teachers were requested to use at least two activities with their students and complete an activity review form. Each KEEP activity was reviewed by at least 10 teachers. The reviews were used to draft the second edition of the KEEP Activity Guide.

In March 1998, the KEEP Activity Guide was evaluated by the National Consortium for Product Quality. They used the Curriculum Quality Standards for School-to-Work: Curriculum Review Instrument. Two outside reviewers and two ad hoc instructors used the instrument to review the guide. The result of the evaluation concluded the KEEP Activity Guide was a sound resource that could be effectively used to build and foster school-to-career skills.

The KEEP Activity Guide continues to go through updates and revisions. It is currently going on its third edition, which will primarily be online. The guide was submitted for review using the Environmental Education Materials: Guidelines for Excellence produced by the North American Association for Environmental Education (2004), a national organization for professionals and research in environmental education. The Environmental Education Materials are part of North American Association for Environmental Education’s National Project for Excellence in Environmental Education. They include criteria for six key characteristics of high quality environmental education materials with indicators to discern if the materials being evaluated meet the guidelines for excellence. When the KEEP Activity Guide was submitted for review through these guidelines in 2009, it scored the highest ranking in all six key characteristics.

Energy educators and experienced teacher educators met in the spring of 1996 to share strategies on effectively educating teachers about energy. Comments provided by participants were recorded and used to develop an energy education course that incorporates the conceptual framework and KEEP Activity Guide.

Professionals experienced in developing and conducting teacher education courses met with the KEEP staff in the spring of 1997. They reviewed draft agendas of the energy education course and ad hoc instructor training workshop. Comments provided by these reviewers were recorded and used to enhance the effectiveness of the course and the workshop.

After the review, the training workshop for the ad hoc instructors was conducted. During the workshop, the instructors were taught energy education concepts and learned about the purpose and implementation of the KEEP Activity Guide. Because the participants were experienced in teacher education, they were asked to review the proposed agenda for the teacher course. Their comments were recorded and used to revise the course. The workshop itself was evaluated by the participants who completed a Workshop Evaluation Form. The evaluation results concluded the workshop was effective in training the ad hoc faculty to teach KEEP courses.

During the pilot year of 1997, teachers who attended KEEP courses completed a precourse survey and a postcourse survey 6 months after the course. The survey was developed with the assistance of researchers from the University of Wisconsin-Stevens Point and the Energy Center of Wisconsin. Over 500 surveys were collected and pre-and postresponses were compared for significant changes. The study did find significant changes in teachers’ perceptions of their preparation to teach about energy, including their ability to provide students with fundamental knowledge about energy. Furthermore, the study showed that the amount of time teachers spent teaching about energy increased and the respondents reported more confidence in obtaining resources to teach about energy. A report of the results was presented to the Energy Center of Wisconsin.

A KEEP ad hoc instructor survey was conducted in March 1998. By this time, each of the 20 ad hoc instructors had taught at least one course. They commented on the design of the course, KEEP materials, and how effectively the course increased teacher energy literacy. Their comments were used to revise the KEEP course.

Over the years, based on teacher interest and stakeholder support, several other energy education courses have been developed by KEEP using the same process. These include a renewable energy course and a school building energy efficiency course. Assessment of KEEP courses continues after each course, with participants completing a course evaluation form. The evaluation was adapted from the standard university course evaluation to include questions about perceptions of gained knowledge and experiences related to energy. There are also items about the effectiveness of the course instructor. Each year, the evaluations for all courses are compiled and reported. There are annual reports from 1998 to the present.

As KEEP approached its fifth year, the advisory committee and staff decided that KEEP should gain insight from Wisconsin teachers to improve its program offerings. Between September 2000 and September 2001, just over 100 K-12 teachers throughout Wisconsin participated in the KEEP nominal group assessment project. Through the project, participants provided responses to the following question:

• What programs, projects, and/or resources should KEEP develop to make energy education effective and fun?

The nominal group assessment technique, based on the process developed by McReynolds (n.d.), which he derived from Byrd (2006), involves group members providing responses to the question and then working together to prioritize results. Rather than quantifying responses as in traditional surveys, the outcomes were derived from a collaborative and guided group decision process.

KEEP staff trained University of Wisconsin-Stevens Point ad hoc instructors on the nominal group assessment technique in September 2000. These ad hoc instructors were then requested to conduct a session in their region of the state. They were responsible for promoting their session and sending the results to the KEEP office and were encouraged to recruit teachers who were and were not familiar with KEEP. Participation in the sessions was voluntary, with a $25 stipend provided as incentive. Fourteen sessions were conducted: the initial training, twelve with teachers, and one with a student group. The student group consisted of nine undergraduates in environmental education who planned to teach K-12 students. This student group session was conducted to gain a student perspective on making energy education effective in K-12 schools.

At the end of 12 months, 107 educators had participated in the nominal group sessions. Middle school teachers represented the largest population (N = 59) of the teacher participants, along with 20 teachers of elementary grades, 20 high school, and the rest teaching all grade levels. There were 50 teachers who had some knowledge of KEEP as they had completed a KEEP course. A wide variety of subject areas was represented, with around five teachers from each; however, there were a larger number of science teachers (N = 29) and teachers of family and consumer science (N = 20).

KEEP staff received the outcomes from the 14 sessions and compiled and reviewed results to look for themes and emphasis of responses. The information gained was used to assist immediate programming decisions. The analysis revealed that teachers wanted hands-on and minds-on experiences in energy education. They needed resources and the time to explore how to use them and how to integrate them into their curriculum. While some of the outcomes could be applied directly to program planning, others were beyond the current staffing and financial means of the program. Nonetheless, staff retained the outcomes and continued to refer to them as the program evolved. Perhaps one of the most beneficial outcomes of the assessment was defining KEEP’s five program areas (professional development, resources, networking and outreach, student involvement, and funding opportunities). Having these program areas has helped KEEP ensure a comprehensive approach to developing and implementing teacher support services and materials.

One especially notable outcome of the nominal group process was the recommendation for hands-on instructional resources to accompany the activity guide. Consequently, KEEP created and piloted energy education trunks that contained various types of energy-related activities, tools and demonstration devices that could be used in the classroom. The assessments for this pilot included both teacher and student surveys (Estes, 2003).

• In 2002, the Energy Center of Wisconsin hired the evaluation consultants, Primen, to conduct an independent evaluation of KEEP. They found that KEEP teachers reported teaching greater quantities and perceived teaching greater quality of energy education. The results also showed that KEEP teachers outdo non-KEEP teachers in several metrics:

• frequency of student exposure to energy concepts;

• amount of student exposure (time) to energy education;

• teacher perception of student learning and attitudes about energy;

• self-reported preparedness to teach about energy.

There was interest among stakeholders for this study to include student assessments to determine program success. The evaluators cautioned against this approach for a variety of reasons that are corroborated by Darling-Hammond (2006):

In rare cases, programs have developed evidence of teachers’ “impact” based on analyses of changes in their pupils’ learning gauged through measures of student attitudes or behavior, work samples, performance assessments, or scores on standardized tests.

The impact or “effectiveness” data increasingly demanded by policy makers are, of course, the most difficult to collect and interpret for several reasons: First is the difficulty of developing or obtaining comparable pre-measures and postmeasures of student learning that can gauge change in valid ways that educators feel appropriately reflect genuine learning; second is the difficulty of attributing changes in student attitudes or performances to an individual teacher, given all of the other factors influencing children, including other teachers past and present; third is the difficulty of attributing what the teacher knows or does to the influence of teacher education. Complex and costly research designs are needed to deal with these issues. (p. 121)

In 2008, KEEP completed a customer satisfaction survey that was part of a larger graduate student research project at the University of Wisconsin-Stevens Point (Ford, 2009). The project was designed to assess teachers’ perception of the quality of a professional development experience in energy education, including the extent to which it helped them increase students’ understanding of energy concepts.

Data for the study was collected using a paper survey of all teachers who had graduated from the original KEEP course since it was first offered (N = 2,562) along with face-to-face interviews of 20 of those teachers. Both the survey and interview questions were adapted from the previous baseline study through the Energy Center of Wisconsin and the independent evaluation conducted by Primen. The survey instrument was reviewed by research consultants to ensure the integrity and content validity.

Nearly 400 teachers responded to the survey, a 16% response rate. Teachers responded positively about the quality of the course. They felt that the course goals were met and that the course helped them to address the themes and subthemes outlined in the KEEP conceptual framework.

The interview was provided to provide supplemental information. Interviewees represented a variety of subject areas, grade levels, and course dates. Teachers reported using course materials and information in and out of classroom settings. All responded that they include the topic of energy in their curriculum because it is an important, timely and relevant topic for teachers and students. There is the sense that KEEP is filling a pertinent demand for energy education professional development.

The overall conclusion of the study was that KEEP effectively responds to the needs of teachers who desire to learn more about and teach energy concepts and that the program should continue to operate as it currently does - being responsive.

In 2011, stakeholders became interested in understanding teachers’ energy use behaviors and the role KEEP could play in promoting behaviors that save energy. Accordingly, the Wisconsin Center for Environmental Education conducted an assessment of KEEP teachers’ energy use practices. The survey was drafted by KEEP staff based on a review of the literature and reviewed by consultants in the field energy resource management.

The electronic survey asking about energy behaviors in the classroom and in at home was sent to 5,110 teachers. These teachers included randomly selected teachers from the KEEP database and a comparable number of randomly selected teachers who had not participate in KEEP courses (provided through the Wisconsin Department of Public Instruction teacher database). More than 500 teachers responded (10% response rate). Among the participants, 302 teachers indicated they have some experience with KEEP.

The teachers who responded reported a variety of positive energy use behaviors. For example, 96% of the respondents of this survey reported that they completely turn off their classroom lights always or most of the time. Similarly, 92% turn their lights completely off at home when they leave a room always or most of the time, and 82% of all teachers reported that half or more of their light fixtures at home have energy efficient bulbs. There were some interesting findings about home energy use compared to school energy use, such as 74% reporting always or almost always turning off their computer monitors while not in use at home, while only 31% reported the same behavior at school. Of the teachers who have had some experience with KEEP, 13% indicated having a Home Performance with ENERGY STAR audit while 8% of the teachers who had not participated in a KEEP course reported having their homes audited.

The feedback gathered from this survey is assisting the development of future KEEP programs. For example, the disparity between energy behaviors at home and at school is motivation to customize energy education lessons to address the unique learning needs of these different locations.

Over the years, KEEP continued to respond to teacher requests for addressing their energy education needs. One request was to develop online courses and courses focused on specific energy topics such as renewable energy and school building energy efficiency. Currently, KEEP offers six different courses on energy. Teachers who completed at least three of the various KEEP courses are referred to as “KEEPtacular” since their participation indicates strong interest in energy education.

On October 26, 2011 an electronic survey was sent to 270 KEEPtacular teachers to gain insight into their motivations and practices and to gain advice for future programming. Of the teachers surveyed, 79 respondents provided their perspective regarding energy education in the classroom, a 29% response rate. Respondents offered their views on the extent that KEEP has affected their ability to transfer their energy knowledge to students. Here are a few of the highlights:

• ninety-four percent agree or strongly agree that they have an improved ability to increase student knowledge about energy.

• eighty-seven percent agree or strongly agree that they have an improved ability to encourage students to use energy more efficiently.

In the survey, teachers were asked to list any additional materials or support services that they felt would improve their ability to make teaching about energy more effective and fun. The following figure summarizes the findings:

The Wisconsin K-12 Energy Education Program continues using formative assessments to help improve its services to teachers and to address stakeholder interests. With the goal in mind of improving and increasing energy education, the staff is planning a variety of assessment ideas including, but not limited to, pre-and postcourse surveys that measure behavior change occurring after participating in a KEEP course.

As mentioned previously, measuring student learning directly is cost prohibitive with results that may or may not be attributed to program activities. Therefore, one alternative is to explore how teachers currently assess student understanding of energy concepts, and to create mechanisms by which KEEP can help teachers gain better insights into student learn-To begin to gain an understanding of teacher assessment practices, KEEP staff sent an email to middle school teachers who have graduated from their courses asking them to share their strategies. Following is one response:

This testimony implies that this teacher believes KEEP materials help improve student interest in the topic, but more constructive ways to assess learning are needed. KEEP staff are currently working to deploy more thorough surveys to gather additional information and develop strategies to improve teacher assessment practices.

As KEEP enters its 18th year, assessments and reviews continue to play a vital role in the program’s success. KEEP is carrying out the objectives of myriad stakeholders; therefore, assessments are critical to ensuring the needs and interests of all stakeholders are being met.

First and foremost among the stakeholders are the teachers and their students. Since KEEP is a teacher education program and teachers are the target of its support services and materials, they will always be the primary assessment audience. Teachers are the primary source for learning about changes in student knowledge, attitudes, and behaviors regarding energy use decisions and practices. The recommendation concerning this stakeholder population is to check in with the target audience, which includes middle school teachers, to assess interests and needs in relation to the field or topic.

Another set of stakeholders is comprised of professionals in the fields of environmental education and energy resource management. For these supporters, a recommendation is to create a conceptual framework that encompasses their perspectives and that provides a cohesive and comprehensive overview of the topic. Make sure all aspects of the conceptual framework are addressed through various support services and materials and in turn, all materials and services are relevant to the framework. Furthermore, conduct rigorous content reviews and pilot testing of all services and materials.

Finally, there are the financial supporters who have a stake in the program’s success. By forming an advisory committee that includes representatives from these stakeholders, program designers create a venue to share the results of formative assessments. The inclusion of these stakeholders is vital to the sustainability of the program.