A self-study using data to inform student-teacher supports in a transformed teacher preparation program Open Access

The urban northeastern university enrolls a diverse student body of over 14,000 students and 81,000 alumni and offers more than 90 undergraduate and graduate programs. Particularly, the School of Education encompasses three departmental levels of study: (1) early and childhood (EC); (2) middle and high school (MHS); and (3) special education (SE). Students enrolled in the School of Education during the 2018–2019 AY were (N = 1,389). About 207 of these students were enrolled within the School of Education, both full-time and part-time as undergraduate and graduate students. The School of Education comprises more than 25 nationally recognized and state-approved teacher certification programs. The School of Education was re-accredited by the Council for Accreditation of Educator Preparation (CAEP) in 2021.

SOEIEP incorporated a three-year phase method of improvement approaches designed to aid in a transformed teacher preparation program. The School of Education developed a constructive plan to work with specific educational departments within each phase of implementation. Within phase one, the School of Education began implementation with the EC program. More specifically, phase one (Year 1) embodied the early childhood program, phase two (Year 2) the special education program, and phase three (Year 3) all three programs, including EC, SE and middle and high school (MHS).

According to the National Council on Teacher Quality (NCTQ), the US employs three million teachers and relies on 27,000 teacher preparation programs in 2,000 separate higher education institutions for training those teachers (NCTQ, 2020). Data and analysis from the January 2020 Teacher Prep Review: Program Performance in Early Reading Instruction finds 51% of 1,000 traditional elementary teacher preparation programs earning a passing grade of B or better, our School of Education did not attain this grade systematically throughout its programs. Figure 1 details the Teacher Prep Review distribution of grades for graduate programs, 2013–2020.

Self-studies in teacher preparation is a growing body of research confirming teacher education itself needs to change (Darling-Hammond, 1997; Zeichner et al., 2024). A self-study in teacher education preparation programs constitutes educators analyzing their own teaching methods and practices through the scope of intense research. Many universities find various challenges with implementing continuous improvement, such as training faculty on data-use, progress monitoring and problem-solving (Noell and Gansle, 2018). The purpose of this study is to establish continuous improvement; the challenge becomes integrating change to represent improvement (Lougharan and Russell, 2005).

During the Obama administration, the US Department of Education instituted a plan that supported states with data systems that would “identify high- and low-performing teacher preparation programs” as well as “provide all programs with information to help them improve, while holding them accountable for how well they prepare teachers to succeed in today’s classrooms and throughout their careers” (White House, 2014). In the Every Student Succeeds Act (ESSA), such a program includes funding for Title II, Part A, the Supporting Effective Instruction State Grants program.

Title II was composed to inspire districts to use effective evidence-based strategies and to examine teacher practices and professional development most effective for improving teacher preparation and school partnerships. According to Darling-Hammond (2018), this includes research exploring sustained approaches that allow teachers to work with faculty on curriculum and teaching strategies they develop and refine over time. It also includes strategies for schools to create collegial partnerships that support teachers and course content knowledge of teacher preparation programs.

The University–School Partnerships for the Renewal of Educator Preparation (US PREP, 2020) National Center provides on-the-ground support and services to a coalition of university-based teacher preparation programs. As a coalition member to US PREP, the School of Education is committed to building teacher candidate competencies, using data to support continuous improvement efforts. We also offer mentor teachers with preparing candidates to work with students with diverse needs and abilities and strive to build strong partnerships with schools in order to meet the needs of PK-12 students.

Faculty in the School of Education have collaborated to improve the field-based approaches to teacher education. Seminal research from Labaree (2010) suggested field-based programs place primary emphasis on classroom experiences in educational settings for teacher candidates. These programs depend on strong, continuously evolving university–school partnerships. Additionally, effective field-based experiences for teacher candidates prepare them to be work-ready from the start of their teaching career and, equally important, these experiences are more effective in increasing student outcomes (Boyd et al., 2008; Hays et al., 2023). The power of culturally relevant and responsive clinical practice generates strong teachers who thrive in the classroom (Hays et al., 2023). Teacher preparation programs that adopt these improved teacher education programs are frequently evaluated based on teacher candidates’ amount of actual time spent in classrooms (Ryan et al., 2014).

There is a wealth of research aimed at improving teacher preparation, including national standards for practice such as the Interstate New Teacher Assessment and Support Consortium (INTASC, 2013) and the Council for the Accreditation of Educator Preparation (CAEP, 2023). Additionally, state and federal policy mandates, such as the new CAEP accreditation standards, have realigned to make teacher education programs become more data-driven (Peck et al., 2014). Peck et al. suggests “at the same time, it is easy to be overwhelmed by the intensifying policy requirements around ‘data use’ and easy to experience these as mandates for accountability rather than opportunities for inquiry, learning and program improvement” (p. 2).

Many researchers attest that the most powerful teacher preparation programs incorporate an extensive field-experience residency, examine and apply course content knowledge with faculty who implement culturally responsive teaching practices to learners (Darling-Hammond, 2006; Hays et al., 2023). Additionally, researchers have determined that rich clinical field experiences place emphasis on educational settings for teacher candidates and in strong, effective university–school partnerships (Chang et al., 2016).

Faculty leadership in the School of Education has collaborated with US PREP to redesign their teacher education preparation program. The reformed program is an improvement on the traditional model (i.e. focusing on theoretical principles of practice) to a rich clinical practice with an emphasis on content-focused pedagogical methods, data-use practices and university–school partnerships.

Mentor teachers support teacher candidates through effective methods such as mentoring, coaching and restorative feedback. The teacher candidates co-teach with their mentor teacher, experiencing expert teacher assessment, participation in parent–teacher sessions and professional development conferences for both mentor teachers and teacher candidates. Previous studies have found teacher candidates who receive these supports ultimately remain in schools, teaching at rates higher than those who receive deficient preservice teacher supports.

The Institutional Review Board (IRB,  Appendix A) of the urban university approved a qualitative analysis of participants’ expression of a genuine theory of belief systems. Data collection occurred for eight weeks using open-ended research questions during one-hour interviews. Interviews were recorded and transcribed using transcription and audio dictation software. Recordings were reviewed with interviewees for clarification and modification purposes.

This study is organized around three major questions that address key aspects of teacher preparation. These questions are commonly asked by policymakers, teacher educators and other stakeholders who are interested in the quality of the teaching force and how it may be improved.

• (1)

  What ideas and perceptions do stakeholders hold regarding data use in the School of Education Intensive Education Program (SOEIEP)?

• (2)

  What data are collected, and how is it archived and analyzed in the School of Education Intensive Education Program (SOEIEP)?

• (3)

  What program policies and practices affect data use?

Participants in the study reflected with the research lead on different aspects of the SOEIEP teacher preparation program. Each participant responded to pre-prepared open-ended questions related to their program expertise. Participants included were:

• (1)

  Site coordinator (1): The site coordinator is responsible for training clinical faculty supervisors and mentor teachers. The coordinator collaborates with all stakeholders involved in the new SOEIEP program, facilitates all meetings with leadership and local school partners, including governance meetings and analyzes all data and teacher evaluations to share with stakeholders at governance. Additionally, the coordinator guides and evaluates teacher candidates.

• (2)

  SOEIEP clinical faculty supervisors (4): The supervisors guide and evaluate teacher candidates.

• (3)

  PK-12 participating mentor teachers (17): The mentor teachers are selected by the city’s department of education and the local PK-12 school principals. These mentors are trained by the coordinator, and they work one semester with the teacher candidates in planning, modeling, and teaching lesson plans.

• (4)

  Participating teacher candidates (4): Teacher candidates are expected to engage students in learning content through activities, assignments, group work, supplemental resources, structure and pacing. They participate in the evaluation and documentation of student progress and attendance.

Five study recruitment guidelines were repeatedly rationalized throughout the research inquiry process. The five concepts were (1) enabling study participants to understand they are part of the research team, allowing participants to give feedback in multiple modes (i.e. candidate observation/evaluation rubrics and teacher progress reports); (2) keeping the participants engaged over time, a monthly newsletter was emailed to keep participants engaged and aware of updates on events; (3) the study was given a positive connotation, creating a positive experience for participants; (4) consent forms were created for easy understanding, and the participants were knowledgeable about the study and (5) the recruitment strategy was matched to our population of teachers. Face-to-face interviews were scheduled for most participants. For the mentor teacher participants, a reflection survey was created ( Appendix B).

A random assignment procedure was drawn for each teacher candidate to participate in independently assigned groups. There were (N = 17) teacher candidates assigned to (N = 5) district elementary schools. One teacher candidate was selected as a participant from each school. A table of random numbers was assigned to the 17 teacher candidates for the selection of participants in the study; see Table 1.

Purposive sampling was used to recruit the selected mentor teachers (N = 17), clinical faculty supervisors (N = 4), deans (N = 2) and chair (N = 1) from the EC program. This sample frame was purposive because each of these participants were exclusively involved in the pilot study.

Interviews were conducted through a direct method of inquiry using transcription and audio dictation software of recordings. Interviewee participants (N = 24) were recorded during one-hour sessions, and each participant responded to open-ended research questions. Recordings were reviewed with interviewees for clarification, accuracy and modification purposes. A reflection questionnaire was presented to mentor teachers (N = 17) as a way to dig deeper into teacher preparation and to offer an opportunity to analyze authentic classroom experiences.

Governance data, facilitated by the site coordinator, assessed and monitored teacher candidate progress, including the implementation of co-teaching, evaluated performance assessments from walkthrough coaching visits and shared teacher candidate data with faculty; using data to shape programming.

The School of Education’s candidate performance data and Council for the Accreditation of Educator Preparation (CAEP) annual reporting measures included the Educative Teacher Performance Assessment (edTPA) and content specialty tests (CST) pass rates. CAEP’s mission is to create educator preparation program, assuring continuous improvement at the faculty level to demonstrate student achievement through CAEP’s seven strategic standards (CAEP, 2023). The edTPA is a performance-based, subject-specific assessment and support system used by teacher preparation programs throughout the United States to emphasize, measure and support the skills and knowledge that all teachers need from Day 1 in the classroom (Pearson Education, Inc., 2024).

Title II reports are prepared in accordance with the Title II Higher Education Amendment of 2023. The US Department of Education requires higher education institutions with teacher preparation programs to submit data on candidate performance to their respective states. The SOEIEP provided test results for program completers on the state’s teacher certification examinations. An annual institutional report is generated for the SOEIEP and the state education department.

The deans for impact (DFI,  Appendix C) diagnostic tool was used to examine the SOEIEP program. This tool outlines the trajectory of the improved teacher preparation program to become more data-informed. Using this tool provides constructive analysis of a program’s current use of data, where its areas of strength are, and where areas for growth exist.

The tool is divided into four focus sections that our work with teacher preparation programs across the country that suggests are important for creating positive conditions to use data:

• (1)

  Developing shared understanding.

• (2)

  Collecting, organizing and analyzing data.

• (3)

  Organizing people to learn.

• (4)

  Using data for program improvement.

After interviews were transcribed, transcriptions were reviewed using low inference coding. The research lead captured high-quality notes, ensuring best practices for aligning the dictated needed areas of improvement. Low-inference coding included evidence gathered from transcribed interviews and reflection questionnaires, without bias. Governance data and CAEP reporting were analyzed using a correlation quantitative data analysis method. After which, several findings were summarized into four major themes. Finally, assessment of the use of data for continuous improvement using the DFI tool yielded areas of strength, areas of growth and next actionable steps.

Four major themes emerged using an inductive approach. After data collection through a series of interviews, the transcribed data was coded to describe the beliefs and perceptions expressed throughout the interview. All data were gathered into groups identified by code. These codes allowed the research lead to gain a condensed overview of the main points and common meanings that recur throughout the data. Several codes were combined into four themes. Figure 2 illustrates the four established themes.

Several major findings were extracted from the interviews. The findings were supported with direct quotes from the participants in their interviews.

First, the research lead probed how program members perceived data use as either an inquiry process or as a compliance process. Findings specify that stakeholders hold different ideas and perceptions regarding data use. Most of the participants interviewed see data use as an inquiry process meant to support improvement measures. There were specific stakeholders who noted the “compliance nature” of data collection and use in the teacher education program. Notably, one participant stated, “With all the data that is shared with them, teacher candidates see the bigger picture and are more aware of how they are doing in the classroom” and “Students want to know how to improve.” Second, the research lead extended question #1 to examine how people in the SOEIEP program see their work primarily through the lens of either individual practice (coursework, field experiences) or as a collective practice (the program as a whole).

Findings identify that participants hold differentiated views on their work with data use. Some see the work as individual practice; others see the work as collective. Ideas from participants stated, “Program improvement should be a collective effort; an action enabled by the dean or the faculty governance – curriculum change is through faculty.” Third, question #1 established how common goals and values were shared among program members and how strongly and how concretely. Findings determine that participants who hold leadership roles see the need to ground program improvement efforts in data. Thus, the need to share data is essential to faculty input and buy-in. Other participants felt that “data sharing” is critical to the growth of teacher candidates in the field. The overall finding of research question #1 suggests participants hold differentiated ideas and beliefs regarding data use that appear to be related to their roles and responsibilities within the SOEIEP program. As stated by an interviewee, “program improvement should be a collective effort through faculty governance.”

The research lead dissected question #2 with probing points such as what data is collected and of the data sources available, which are viewed as most useful or not useful. Findings suggest that considerable data are being collected in the SOEIEP program. Many participants feel “too much data” is being collected, specifically in the SOEIEP program, with the requirements of edTPA. There appears to be a large focus on the need to revise the observation rubric ( Appendix D). Participants feel the most helpful data were qualitative versus quantitative in nature. The investigation found no real consensus on what data is most important and most helpful. As one participant pointed out, “Many data points, too much data collected, too many tools such as the city’s department of education mentor teacher progress reports, the SOEIEP observation evaluation rubric, mentor teacher reflection questionnaires, observational coaching visits and teacher evaluations utilizing a structured process known as POP cyles, which involves pre-conferences, recorded observations, and post-conferences.” Another participant implied, “rubric scores are not helpful at all – feedback and comments (next steps) would have been more helpful” and “most helpful data are video recording of lessons and self-reflection journals.” Next, the research lead examined how accessible the data are either to individual program members or to role-alike groups (e.g. field supervisors, full-time faculty). Data appear to be shared between supervisors and teacher candidates, such as the web-based electronic portfolio and assessment teacher candidate management system, “TaskStream data.” For the most part, data does not appear to be readily accessible to all stakeholders. Participants revealed, “data from mentor teachers was not shared” and observational evaluative data is available only to TaskStream users: candidates and faculty.

Finally, students’ access to data related to their progress in the program was assessed. Findings indicated that students have access to their observation scores via TaskStream and Google documents that are shared with them by their supervisors. Data collected by the city department of education is not available to students as it is not shared with them. Quoted remarks from participants included “a lot of communication and sharing between teacher candidates and supervisor,” “access their observation scores through TaskStream” and “mentor teacher data not shared with teacher candidates.” The overarching finding implies considerable data are being collected in the SOEIEP program. Participants appeared to prefer qualitative data over quantitative data. Data collected by supervisors appear to be somewhat available to those connected to the work, including students. Program data collected for compliance purposes was not mentioned in interviews and may not be readily accessible to stakeholders. As quoted from one participant, “The informals” [coaching visits], and even the formals [formal evaluation/observations], were recorded on special sheets [and] helped to inform me because it was concrete. [They] informed me… about what direction I need to go [with the Teacher Candidate].

For the final question #3, the research lead investigated the ongoing structures in place to support collection, analysis and action planning related to program data. Findings showed a process in place for the collection and sharing of data by the supervisors to inform their work with teacher candidates. Participants noted various concerns and needs regarding ongoing data use practices. Participants verbalized feedback included, “lots of communication and sharing between teacher candidate and supervisor” and “use the data to inform how to support teacher candidates.” Also, “issues with students not being able to be filmed – much of the observation were with small groups for that reason – couldn’t be with the whole class – how can we use technology to collect data that’s authentically reflecting the classroom?” Lastly, one participant added, “Lehman’s rubric needs work – qualitative not quantitative – not sure what the numbers really mean – it is a grade – prefer the reflection – need to understand and learn more about the rubric.”

Second, the research lead examined how these practices were integrated into regular faculty/staff/partner meetings. Findings indicated there were no policies regarding data sharing, but participants noted a need to share data through governance and data meetings. One participant noted, “no policy around data sharing (i.e. with the DOE) – it is better through USPREP and the collaboration due to the grant.” A third examination of research question 3 assessed how the program is connecting data review or analysis to specific program improvement actions. Findings revealed the interviews did not divulge ways the program is connecting data analysis to specific program improvement efforts. However, some participants did note a need for governance and data meetings as well as a communication process, to share data. As previously mentioned, one participant said, “Need a communication process to share data; need to share data – governance – data meetings.” A fourth investigation of research question 3 analyzed whether the roles and responsibilities related to data use were well defined and coordinated vs assigned ad hoc. Findings showed the role of the supervisor appeared to be well defined regarding data use, although there were areas for improvement. One participant noted, “data from mentor teachers is not shared; there’s a lot of communication and sharing between teacher candidate and supervisor like coaching and POP data.” Last, the research lead attempted to examine expectations related to data usage identified in program personnel policies (e.g. job descriptions, training criteria and compensation). However, findings indicated nothing came up in the interviews regarding data usage and program personnel policies.

The overarching finding to research question 3 illustrates no program policies for data use practices, with the exception of the role of the clinical field supervisor. However, there was a belief that data use practices need to be established and implemented. One participant replied, “the DOE [city department of education] and the university does not have an MOU [Memorandum of Understanding] about data sharing [but the] partnership facilitates [a] collaboration in data sharing.” Another participant suggested, “whenever we talk about the data, there should be… implications you can draw from. So [our] next focus may be to draw some implications for the curriculum change and for the course realignment… for our own practice as educators. That should enlighten our faculty to do a better job to prepare TCs [teacher candidates].”

Governance data and CAEP reporting were analyzed using a correlated qualitative data analysis method. According to CAEP Standard 5: Quality Assurance System and Continuous Improvement, the SOEIEP adhered to the following:

• (1)

  R5.1 The SOEIPE developed, implemented and modified, as needed, a functioning quality assurance system that ensured a sustainable process to document operational effectiveness.

• (2)

  R5.2 The SOEIEPs quality assurance system relied on relevant, verifiable, representative, cumulative and actionable measures to ensure interpretations of data are valid and consistent.

• (3)

  R5.3 The SOEIEP included relevant internal (i.e. administrators, faculty, teacher candidates) and external (e.g. mentor teachers, school administrators, stakeholders in program evaluation and continuous improvement processes).

• (4)

  R5.4 The SOEIEP regularly, systematically and continuously assessed performance against its goals and relevant standards, tracked results over time, documents modifications and their effects on their transformed program.

• (5)

  Table 2 displays the edTPA pass rates summary for both early and childhood education programs. Table 3 displays teacher certification examination results, and Table 4 displays SOEIEP Annual CAEP EPP Report.

The self-study report is more than an amalgamation of reports prepared by the research lead. Rather, the study represents the results of SOEIEP’s careful analysis and assessment of the sufficiency and effectiveness of its policies, procedures, practices, programs, activities, resources, structures and outcomes relative to CAEP accreditation standards. Careful assessment is made to examine areas that warrant improvement. This critical self-assessment occurred through the DFI tool. The DFI is a significant internal assessment of the self-study process to which the SOEIEP team paid particular attention, as these judgments provide considerable insight into internal planning and progress monitoring of the SOEIEP teacher educator resources to achieve mission, goals, objectives and student learning outcomes.

Examination of the SOEIEP program’s strengths and areas of growth warrants further development and assists the SOEIEP program with analyzing and assessing its program development and continuous improvement through data use. The SOEIEP areas of strength and growth are listed below.

• (1)

  Shared data with all stakeholders sharing common goal and vision for continuous program improvement.

• (2)

  Data collection assessments measure student performance and teacher effectiveness.

• (3)

  State teacher competencies are systematically and collectively evaluated using the Danielson framework.

• (4)

  Stakeholders engaged in data use processes.

• (1)

  The SOEIEP Program has acquired too many different assessments, measuring the same data.

• (2)

  Two institutions (City DOE and university) are enacting two similar Danielson evaluative rubrics assessing similar data, interpreting similar outcomes. How can we align the two?

• (3)

  Stakeholders state there is not enough time to conduct authentic observations and evaluations with the edTPA assessment within a 15-week semester.

• (4)

  Coursework needs to be aligned to field work in support of teacher candidates’ growth.

• (5)

  Program-wide data use plan needs to be developed.

• (6)

  No collection of longitudinal data from graduates.

Having analyzed Lehman’s SOEIEP program policies, procedures, practices, resources and outcomes, the self-study has addressed plans and recommendations for future development. The recommendations are part of SOEIEP’s overall planning process, representing a continuous action plan to improve the quality of its educational services over the next years. The recommendations are linked directly to the specific findings identified in the study.

• (1)

  Need a commitment to action as program faculty to engage the issues identified in the self-study.

• (2)

  Administration needs to lead faculty in setting priorities and making choices.

  • Streamline data collection in SOEIEP.

  • Revision of the SOEIEP observation rubric to address concerns.

  • Align coursework to fieldwork to support teacher candidates in making critical connections between theory and practice.

  • Devise a data use plan that provides accessibility and manageability, time-sensitive and collective engagement for program improvement.

• (3)

  Develop a survey to support longitudinal data gathered from graduates to inform program improvement efforts and to fulfill CAEP requirements.

The primary goal of the study was to develop a transformed organizational structure, understanding data-use through state policies and effective teacher practices supported by evidence-based decision-making and continuous program improvement in our teacher education program. The results of the study were used to develop resources and actionable steps focused on the improvement of the SOEIEP program.

This self-study represents Phase 1 and yielded rich qualitative data derived from interviews, focus groups and an examination of pertinent artifacts referred to within interviews (assessments, state exams, observations, etc.). The interviews were conducted by the research lead/site coordinator, faculty, teacher candidates, university leadership, partner school mentor teachers and principals. The goal of the study was to understand the kinds of organizational policies and practices necessary to support evidence-based decision-making and program improvement in teacher education (Peck and MacDonald, 2014). The data-use findings revealed four general themes within these data:

• (1)

  Data-informed training practices provide a deeper understanding of analysis and effective practice, building a faculty culture that engages in “inquiry-based” data use practices, as opposed to one of compliance.

• (2)

  Differentiated perceptions of data collection, analysis and use are needed for effective teaching practices within faculty and teacher candidate responsibilities.

• (3)

  Context matters; quantitative data is made more palatable and meaningful when accompanied by the qualitative story through interviews. A collective vision regarding data collection, analysis and use should be the foundation for qualitative data use practices.

• (4)

  Data-informed approach must continuously be revised for program improvement.

Based on the findings yielded by the self-study regarding the data-informed practices within the SOEIEP Program developed the following actionable next steps:

• (1)

  Developed an improved observation and evaluation plan focused on data collection, analysis and improvement.

• (2)

  Collaboratively constructed programmatic outcomes with all stakeholders: university leadership, faculty, teacher candidates and partner school district partners.

• (3)

  Governance committee co-constructed a data plan with stakeholders:“What data should guide programmatic decisions?”

• (4)

  Use the collaboratively constructed programmatic outcomes and the data plan to inform continuous programmatic improvement efforts each semester.

We have become much more intentional around data use for continuous improvement as a result of the self-study.

The supplementary material for this article can be found online.